AM571x memory issues in v4l capture with Ubuntu RFS

John Lusby

Experiencing memory corruption issues when running camera capture on our custom
am5718 board. This board is the alpha version of a custom board our company is
developing using the AM5718 processor. We're using the kernel provided in the
2.00.02 version of the sdk plus a driver for the imagers we're using and the
necessary device tree changes to get it working. Overall the board's design is
fairly similar to the AM572X evm. We're using an Ubuntu 14.04 root filesystem.

Here are the tests we've done to pinpoint the problem and their results followed
by some example runs and their output.

Our board + Ubuntu rfs: Running our camera application + memtester - Failure
Our board + Ubuntu rfs: Running yavta + memtester - Failure
Our board + Ubuntu rfs: Running memtester by itself - Success
Evm + Arago rfs (all files and kernel are unchanged vanilla versions from the ti 2.00.02 sdk)
: Running yavta + memtester - Success
Our board + Arago rfs : Running yavta + memtester - Success
Evm + Kernel for our board + arago linux + normal evm dts
: Running yavta + memtester - Success

Our original theory was that for our board we had somehow missconfigured the
memory / CMA / DMA / video subsystem so that when we tried to use it it was
clobbering the rest of the memory for other applications. The testing with the
Arago filesystem seems to imply that its probably something missing in our
ubuntu root fileystem, some additional configuration needed to get everything
to work.

Here is an example run that triggers the problem. On our board + Ubuntu RFS

root@arm:~# ps aux | grep yavta
root 1528 0.3 0.1 11756 1088 ? S 17:26 0:00 yavta --capture /dev/video1
root 1601 0.0 0.1 3308 1596 pts/1 S+ 17:29 0:00 grep --color=auto yavta
root@arm:~# memtester 512M
memtester version 4.3.0 (32-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffff000
want 512MB (536870912 bytes)
got 512MB (536870912 bytes), trying mlock ...locked.
Loop 1:
Stuck Address : testing 0FAILURE: possible bad address line at offset 0x19c3a000.
Skipping to next test...
Random Value : /
.
.
.
FAILURE: 0x1deb84cb != 0x00000000 at offset 0x09dd7cd0.
FAILURE: 0x12ea8de5 != 0x00000000 at offset 0x09dd7cd4.
FAILURE: 0x61e9ceea != 0x00000000 at offset 0x09dd7cd8.
FAILURE: 0x4ecf6c79 != 0x00000000 at offset 0x09dd7cdc.
FAILURE: 0x0f42e110 != 0x00000000 at offset 0x09dd7ce0.
FAILURE: 0xc6d237c2 != 0x00000000 at offset 0x09dd7ce4.
FAILURE: 0x11e9b6ab != 0x00000000 at offset 0x09dd7ce8.
FAILURE: 0xaec9220d != 0x00000000 at offset 0x09dd7cec.
FAILURE: 0x91eb5445 != 0x00000000 at offset 0x09dd7cf0.
FAILURE: 0x85c91dbb != 0x00000000 at offset 0x09dd7cf4.
FAILURE: 0x8e78e872 != 0x00000000 at offset 0x09dd7cf8.
Compare SUB : FAILURE: 0xec9760a2 != 0x00000000 at offset 0x096d57fc.
FAILURE: 0xed89df5e != 0x00000000 at offset 0x096d5800.
FAILURE: 0xa5fdb3f6 != 0x00000000 at offset 0x096d5804.
FAILURE: 0xa9698292 != 0x00000000 at offset 0x096d5808.
FAILURE: 0x3bcbae2a != 0x00000000 at offset 0x096d580c.
FAILURE: 0x23d82f80 != 0x00000000 at offset 0x096d5810.
FAILURE: 0xe18b90da != 0x00000000 at offset 0x096d5814.
.
.
.

The theres so many of those failures that it just goes on and on and scrolls
the buffer so I cant show where they first start without logging the output.

Without yavta or our application which is what we'd normally be running to
interface with the camera the memtester looks much more normal

root@arm:~# pkill yavta
root@arm:~# ps aux | grep yavta
root 1659 0.0 0.1 3308 1528 pts/1 S+ 17:31 0:00 grep --color=auto yavta
root@arm:~# memtester 512M
memtester version 4.3.0 (32-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffff000
want 512MB (536870912 bytes)
got 512MB (536870912 bytes), trying mlock ...locked.
Loop 1:
Stuck Address : ok
Random Value : ok
Compare XOR : ok
Compare SUB : ok
Compare MUL : ok
Compare DIV : ok
Compare OR : ok
Compare AND : ok
Sequential Increment: ok
Solid Bits : testing 6

Incase its important this is the current CMA configuration on our kernel

3 #
2 # Default contiguous memory area size:
1 #
1233 CONFIG_CMA_SIZE_MBYTES=128
1 CONFIG_CMA_SIZE_SEL_MBYTES=y
2 # CONFIG_CMA_SIZE_SEL_PERCENTAGE is not set
3 # CONFIG_CMA_SIZE_SEL_MIN is not set
4 # CONFIG_CMA_SIZE_SEL_MAX is not set
5 CONFIG_CMA_ALIGNMENT=8

over 9 years ago

0 Biser Gatchev-XID over 9 years ago

TI__Guru**** 393215 points

I will ask the software team to comment.

0 Yordan Kovachev over 9 years ago

TI__Guru**** 161600 points

Hi John,

John Lusby said:
We're using the kernel provided in the 2.00.02 version of the sdk plus a driver for the imagers we're using and the necessary device tree changes to get it working.

Have in mind that the kernel from SDK02.00.02 is modified to work with wayland, while Ubuntu rfs uses X server. I am not that familiar with Ubuntu file system, so what I can suggest is to cross check the related drm, fbdev & dss kernel sources from some Ubuntu repository against the TISDK kernel drivers. There may be some differences in the handles provided to the filesystem.

Best Regards,
Yordan

0 John Lusby over 9 years ago in reply to Yordan Kovachev

Prodigy 130 points

X Server hopefully shouldn't be involved. This is a headless install and I don't think X is installed at all.

Also further testing found that we were able to get the failure in Arago. The only difference between Arago and Ubuntu appears to be the fact that Arago uses 100MB less ram on startup, so in order to get the failure we needed to run memtester 612MB, similarly if we decreased the size on the ubuntu memtester allocation amount it would not fail.

We also noticed some non deterministic behaviour surrounding the failure. Arago didn't fail at first @ 512MB, then we used 100MB more since we observed that the Ubuntu boot uses about 100 more mb right out of the gate than arago. Then Arago showed the memory error too, then we went back to using 512mb and it still had memory errors, so it was the amount of ram locked and wether it would fail that was nondeterministic.

0 John Lusby over 9 years ago

Prodigy 130 points

As a shot in the dark test we tried seeing if it was possibly related to the fact that our imagers are 10 bit imagers being read in as 8bit imagers when from what we could tell the TI support for 8 bit imagers was limited (we had to hack it a bunch to get it to work properly).

We set up video1 as 16 and video2 as 8b... so we could test on the same kernel... started batch yavta on 16b imager for 100000 frames.. then start memtester at 512MB and runs fine. Repeat that on the 8b one and it fails the stuck addr test almost immediately every time

0 Yordan Kovachev over 9 years ago in reply to John Lusby

TI__Guru**** 161600 points

Hi,

Are you using a custom app on your board to perform the test?

I tried the following on my am57xx board, running SDK02.00.02.11 (kernel 4.1.18):

root@am57xx-evm:~# memtester 612M > logmem.txt &

Then start the yavta:

root@am57xx-evm:~# yavta --capture /dev/video1

Wait for ~ 10 mins and terminate both memtester & yavta. The result from cat logmem.txt is:

root@am57xx-evm:~# cat logmem.txt

memtester version 4.3.0 (32-bit)

Licensed under the GNU General Public License version 2 (only).

pagesize is 4096

pagesizemask is 0xfffff000

want 612MB (641728512 bytes)

got 612MB (641728512 bytes), trying mlock ...locked.

Loop 1:

Stuck Address : ok

Random Value : ok

Compare XOR : ok

Compare SUB : ok

Compare MUL : ok

Compare DIV : ok

Compare OR : ok

Compare AND : ok

Sequential Increment: ok

Solid Bits : ok

Block Sequential : ok

Checkerboard : ok

Bit Spread : ok

Bit Flip : ok

Walking Ones : ok

Walking Zeroes : ok

Loop 2:

Stuck Address : ok

Random Value : ok

Compare XOR : ok

Compare SUB : ok

Compare MUL : ok

Compare DIV : ok

Compare OR : ok

Compare AND : ok

Sequential Increment: ok

Solid Bits : ok

Block Sequential : testing 240

As you can see there is no fail.

I use the default tisdk_am57xx-evm_defconfig:

# Default contiguous memory area size:

CONFIG_CMA_SIZE_MBYTES=24

CONFIG_CMA_SIZE_SEL_MBYTES=y

# CONFIG_CMA_SIZE_SEL_PERCENTAGE is not set

# CONFIG_CMA_SIZE_SEL_MIN is not set

# CONFIG_CMA_SIZE_SEL_MAX is not set

CONFIG_CMA_ALIGNMENT=8

Best Regards,
Yordan

0 John Lusby over 9 years ago in reply to Yordan Kovachev

Prodigy 130 points

Yorden,

That is consistent with our fourth test. We've been able to pinpoint the issue to the fact that we're using our imagers in 8b mode. Our imager is a 10b grayscale imager where we ignore the 2 lsb and treat it like an 8b imager. We weren't ever able to find an official so we hacked our own support for it and that seems to have introduced a memory corruption issue. What we need is to figure out how to properly configure support for an 8b grayscale imager

Here is at least part of how we hacked it into 8bit mode

commit ff40fe29a591ede7dfb65a34052c861b9058baf1
Author: Justin Beghtol
Date: Wed Feb 17 08:07:23 2016 -0800

Modified e2v and ti-vpe to support an 8-bit image encoding.

diff --git a/drivers/media/platform/ti-vpe/vip.c b/drivers/media/platform/ti-vpe/vip.c
index 10ee2c6..03a14a1 100644
--- a/drivers/media/platform/ti-vpe/vip.c
+++ b/drivers/media/platform/ti-vpe/vip.c
@@ -127,6 +127,15 @@ static struct vip_srce_info srce_info[5] = {

static struct vip_fmt vip_formats[] = {
{
+ .name = "SGRBG8 8-bit grayscale",
+ .fourcc = V4L2_PIX_FMT_SGRBG8,
+ .code = MEDIA_BUS_FMT_SGRBG8_1X8,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .coplanar = 0,
+ .vpdma_fmt = { &vpdma_rgb_fmts[VPDMA_DATA_FMT_SGRBG8]
+ },
+ },
+ {
.name = "NV24 YUV 444 co-planar",
.fourcc = V4L2_PIX_FMT_NV24,
.code = MEDIA_BUS_FMT_UYVY8_2X8,
diff --git a/drivers/media/platform/ti-vpe/vpdma.c b/drivers/media/platform/ti-vpe/vpdma.c
index e8ed6ba..467e90c 100644
--- a/drivers/media/platform/ti-vpe/vpdma.c
+++ b/drivers/media/platform/ti-vpe/vpdma.c
@@ -88,6 +88,11 @@ const struct vpdma_data_format vpdma_yuv_fmts[] = {
EXPORT_SYMBOL(vpdma_yuv_fmts);

const struct vpdma_data_format vpdma_rgb_fmts[] = {
+ [VPDMA_DATA_FMT_SGRBG8] = {
+ .type = VPDMA_DATA_FMT_TYPE_RGB,
+ .data_type = DATA_TYPE_SGRBG8_2222,
+ .depth = 8,
+ },
[VPDMA_DATA_FMT_RGB565] = {
.type = VPDMA_DATA_FMT_TYPE_RGB,
.data_type = DATA_TYPE_RGB16_565,
diff --git a/drivers/media/platform/ti-vpe/vpdma.h b/drivers/media/platform/ti-vpe/vpdma.h
index 0df156b..73fb352 100644
--- a/drivers/media/platform/ti-vpe/vpdma.h
+++ b/drivers/media/platform/ti-vpe/vpdma.h
@@ -102,6 +102,7 @@ enum vpdma_rgb_formats {
VPDMA_DATA_FMT_ABGR32,
VPDMA_DATA_FMT_BGRA24,
VPDMA_DATA_FMT_BGRA32,
+ VPDMA_DATA_FMT_SGRBG8,
};

enum vpdma_misc_formats {
diff --git a/drivers/media/platform/ti-vpe/vpdma_priv.h b/drivers/media/platform/ti-vpe/vpdma_priv.h
index 150336f..73e9120 100644
--- a/drivers/media/platform/ti-vpe/vpdma_priv.h
+++ b/drivers/media/platform/ti-vpe/vpdma_priv.h
@@ -130,6 +130,8 @@
#define DATA_TYPE_ABGR32_8888 0x7
#define DATA_TYPE_BGRA24_6666 0x5
#define DATA_TYPE_BGRA32_8888 0x9
+/* Justin: This data type added without knowing what the magic number does */
+#define DATA_TYPE_SGRBG8_2222 0x23

0 Justin Beghtol over 9 years ago in reply to John Lusby

Prodigy 30 points

To clarify, the DATA_TYPE_SGRBG8_2222 value is a made up number. Previously, for a time, we were using :

.vpdma_fmt = { &vpdma_yuv_fmts[VPDMA_DATA_FMT_Y444]

This worked similar to our hacked version, only VIP was swapping every 16-bit word as if converting for endianness, only we're in 8-bit mode and it shouldn't do any byte swapping at all. The mode described in the patch above has been the only method we've discovered that doesn't cause the VPE manipulate the order of the byte stream from our imager.

In both of the SGRBG8 and Y444 cases using 8-bit mode we get 1.3MB frame buffers (correct) and memory corruption (bad).

0 Yordan Kovachev over 9 years ago in reply to Justin Beghtol

TI__Guru**** 161600 points

Hi,

+/* Justin: This data type added without knowing what the magic number does */

I think you've correctly added your custom data type. However, if you have a look at the VPDMA descriptor (AM57xx TRM, Section 9.4.8.6.1 Data Transfer Descriptors) there isn't a type 0x23 => hence no rules how to process such data type is described.

Previously, for a time, we were using :

.vpdma_fmt = { &vpdma_yuv_fmts[VPDMA_DATA_FMT_Y444]

This worked similar to our hacked version, only VIP was swapping every 16-bit word as if converting for endianness

This is explained in Section 9.4.8.6.1.1.1 Data Type, the text after the Setting RGB Data Types:
"The commonly used RGB format identifiers require the color components to be stored in a little-endian
style, where the left most component is the LSB component."
"VPDMA specifies its component ordering in the big-endian style, which requires the data to be stored in
the reversed order. Example with ARGB data type is shown in Table 9-36 and Table 9-37. The VPDMA
ordering for ARGB data type matches the common BGRA data format.
..........
..........
In order color components to be mapped correctly and to avoid swapping, the reversal must be taken into
consideration when configuring the Data Type in the VPDMA transfer descriptor.
Table 9-38 shows the proper settings required for RGB data types for both storage schemes."

Maybe if you try and process your custom SGRBG8 format as described in Section 9.4.8.6.1.1.1 Data Type, Table 9-38. VPDMA Descriptor RGB Data Type Mapping, will resolve the memory corruption you're facing. However this can be quite a difficult task.

Best Regards,
Yordan

0 Justin Beghtol over 9 years ago in reply to Yordan Kovachev

Prodigy 30 points

Hi Yordan,

Is it perhaps possible for a VPE expert at TI to suggest how an 8-bit imager should be used? I appreciate the references to the TRM, but it makes no attempt to address 8-bit imagers whatsoever. Every entry in table 9-38 is a 16, 24 or 32 bit encoding.

Inside your own driver we see:

case 16:
iface = SINGLE_16B_INTERFACE;
/*
* See TRM p1893 for Repack values:
* 0 - straight, no reordering
* 3 - center/right swap, reverse 16b endian
*/
repack_sel = 0;
break;
case 8:
default:
iface = DUAL_8B_INTERFACE;
repack_sel = 0;
}

Which clearly makes it seem in the 8-bit case repack_sel is set to 0, which by the comment above means "STRAIGHT, NO REORDERING". But the bytes are still reordered. Also there's no SINGLE_8B_INTERFACE to use. I've wondered if the DUAL_8B interface is trying to fill in two 8b images into one DMA buffer because there is no single mode, but that's pure conjecture.

Since the TRM offers no recommendations for 8-bit grayscale (or even 8-bit color) encodings, are we supposed to guess which will make the VPE do the right thing? Should we try every single option, recompiling our kernel every time, until one of them works?

Our imager has a test mode to output predefined patterns of pixels. I use this to validate byte ordering.

I tried using RGB16-565 (DATA_TYPE_RGB16_565 from vpdma_priv.h): This yields still 1.3MP images (right), but corrupts memory (wrong) and reorders bytes (wrong). Sample of the first few bytes (should be increasing): 01 02 03 00 05 06 07 04

I tried the opposite, BGR16-565 (DATA_TYPE_BGR16_565 from vpdma_priv,h): Thie yields still 1.3MP images (right), but corrupts memory (wrong) and reorders bytes (wrong). Sample of the first few bytes (should be increasing): 01 00 03 02 05 04 07 06

Going back to my hacked value of 0x23: This yields 1.3MP images (right), but corrupts memory (wrong), but at least it doesn't reorder the bytes: 00 01 02 03 04 05 06 07

I can systematically try every single encoding in the header file, but I would appreciate some guidance. Maybe I'm adjusting the wrong value? Does TI have ANY other customer using an 8-bit imager??

Regardless of whatever crazy byte ordering the VPE does, I feel like this memory corruption is a fundamental issue that shouldn't be possible using VPE.

0 Yordan Kovachev over 9 years ago in reply to Justin Beghtol

TI__Guru**** 161600 points

Hi,

Justin Beghtol said:
Is it perhaps possible for a VPE expert at TI to suggest how an 8-bit imager should be used?

I've forwarded this to the designers. Their feedback will be posted directly here, when available.

Best Regards,

Yordan

Processors

Processors forum

AM571x memory issues in v4l capture with Ubuntu RFS