AM5718: USB3 camera issue when IPU is running

Marcus,

Can you please get an AM571x_IDK evm to test the same usecase? We need to first check if this is board related.

Unfortunately the AM571x_IDK does not have any USB3 ports on the board. Only USB2 which is not possible to use with the USB3 vision camera.

Sorry, I missed that. We can only focus on your custom board.

I think we first need to check if this is DDR bandwidth issue, because AM571x and less EMIF bandwidth than that on AM572x.
Can you please share more information of your project?
- Do you stream video from the USB3.0 camera or just grab still images? What is the resolution? How much DDR bandwidth does it use?
- You mentioned it is irrelevant to the sw running on IPU, so you have tried to not do "anything" in IPU sw?

Ok,

1. At the moment we are only using frame grabbing functionality, we are grabbing an 4024px by 3036px. We are only running at 1 Hz at the moment, but we have been testing with slower speeds and it does not seem to make any difference.

2. Actually, it seems like when NOT starting TI-RTOS (BIOS) it seems to work. As soon as I run any IPU with TI-RTOS without any tasks started the result is the same.

Marcus

What application do you use to grab still images? Have to checked if the USB bus still stream video even when application doesn't take pictures? USB bus analyzer can tell it.

What Processor SDK version do you use? I don't know much about IPU, but I think IPU1 doesn't anything related to IVA-HD, so if you only load TI-RTOS to IPU1, IPU shouldn't use any resource, right? Do you have any customized sw running on IPU1?

Have you tried to reduce the image resolution down to a point which will not have such issue?

We are using the manufacturers API to grab the images.

We are using SDK ti-processor-sdk-linux-am57xx-evm-04.00.00.04-Linux-x86-Install.bin

It seems that when we are lowering the resolution to 640x480 is works fine. Obviously the size of the image has an impact on the issue. I was wondering if we are accessing any memory regions from the IPU that is different between AM572x and AM571x.

The IPU code uses IPC package to communicate with A15. In the memory map the following memory allocations are made, which is identical to IPC samples for AM572x. However since we are using AM571x this may be incorrect?

In addition to this I have tried running the ex02 messageq examples with the exact same result. It seems that when starting TI-RTOS the USB3 camera stops working

/* Memory Map for ti.platforms.evmDRA7XX:ipu1
*
* --- External Memory ---
* Virtual Physical Size Comment
* ------------------------------------------------------------------------
* 0000_4000 ????_???? F_C000 ( ~1 MB) EXT_CODE
* 8000_0000 ????_???? 20_0000 ( 2 MB) EXT_DATA
* 8020_0000 ????_???? 30_0000 ( 3 MB) EXT_HEAP
* 9F00_0000 9F00_0000 6_0000 ( 384 kB) TRACE_BUF
* 9F06_0000 9F06_0000 1_0000 ( 64 kB) EXC_DATA
* 9F07_0000 9F07_0000 2_0000 ( 128 kB) PM_DATA (Power mgmt)
*/

Best regards

It seems to be an IPC issue between A15 and M4.

When ANY of IPU1 or IPU2 are loaded with ex02_messageq IPC-examples, the issue is seen.

When IPU1 or IPU2 is unbinded, the issue is gone.

It seems to be some kind of memory mapping/allocation issue. We could really use some support in reviewing the resource table which is taken from this example,

processors.wiki.ti.com/.../File:UART_BasicExample_evmAM572x_m4ExampleProject_with_ipc.zip

I am trying to figure out if either "rsc_table_vayu_ipu.c" or "IpuAmmu.cfg" needs to be re-configured somehow to fit AM5718 with 2GB of DDR3 instead of 3GB DDR3 found on EVM_AM572x

Please advice!

Thanks

Marcus Berglund

It will take our expert some time to dig into this. Meanwhile, if you doubt the memory region conflict, please refer to this wiki page that may help you to dig further into this. Please also check the dts files.

Thanks for the link. As far as we can see ther are no conflicts in the configuration. Although it is not easy to penetrate the complete memory management.

We still believe that the answer lies in the difference between AM571x and AM572x regarding the IPC-package.

However, when testing ex02_messageq built in release mode instead of debug mode, the issue seems almost disappeared. The optimization of IPU1 code is a bit difficult to understand how it can affect the performance of a user space application in linux. 

Hi Marcus,

Can you try setting the environment variable IPC_DEBUG=1 and running the IPC example again and posting the output here?

Hi Sahin,

Actually, the read error from the camera can be seen as soon as IPU1 or IPU2 are loaded using "bind". If both are loaded fail rate is like 1/5 but if only one is loaded the fail rate is more like 2/3. This is before I even run "./app_host IPU1" or "./app_host IPU2". But if I run them using the IPC_DEBUG=1. This is the output.

___________________________________________________
--> main:
MultiProc_getConfig: got LAD response for client 0, status=0
NameServer_setup: got LAD response for client 0, status=1
MessageQ_getConfig: got LAD response for client 0, status=0
MessageQ_setup: LAD response for client 0, status=1
NameServer_attach: LAD response, status=0
GateMP_isSetup: got LAD response for client 0, status=0
GateMP_attach: LAD response, status=-5
NameServer_attach: LAD response, status=0
GateMP_isSetup: got LAD response for client 0, status=0
GateMP_attach: LAD response, status=-5
NameServer_attach: LAD response, status=-1
NameServer_attach: LAD response, status=0
GateMP_isSetup: got LAD response for client 0, status=0
GateMP_attach: LAD response, status=-5
GateMP_isSetup: got LAD response for client 0, status=0
GateHWSpinlock_getConfig: got LAD response for client 0, status=0
GateMP_start: got LAD response for client 0, status=0
GateMP_getNumResources: got LAD response for client 0, status=0
NameServer_get: got LAD response for client 0
GateMP_start: could not open default gate, status=-5
GateMP_stop: entered
--> Main_main:
--> App_create:
MessageQ_create: got LAD response for client 0, status=0
MessageQ_create: creating endpoints for 'HOST:MsgQ:01' queueIndex 0
MessageQ_create: got LAD response for client 0, status=0
NameServer_getUInt32: got LAD response for client 0
App_create: Host is ready
<-- App_create:
--> App_exec:
App_exec: sending message 1
App_exec: sending message 2
App_exec: sending message 3
App_exec: message received, sending message 4
App_exec: message received, sending message 5
App_exec: message received, sending message 6
App_exec: message received, sending message 7
App_exec: message received, sending message 8
App_exec: message received, sending message 9
App_exec: message received, sending message 10
App_exec: message received, sending message 11
App_exec: message received, sending message 12
App_exec: message received, sending message 13
App_exec: message received, sending message 14
App_exec: message received, sending message 15
App_exec: message received
App_exec: message received
App_exec: message received
<-- App_exec: 0
--> App_delete:
MessageQ_delete: got LAD response for client 0, status=0
<-- App_delete:
<-- Main_main:
GateMP_isSetup: got LAD response for client 0, status=0
GateMP_detach: LAD response, status=-5
NameServer_detach: LAD response, status=0
GateMP_isSetup: got LAD response for client 0, status=0
GateMP_detach: LAD response, status=-5
NameServer_detach: LAD response, status=0
GateMP_isSetup: got LAD response for client 0, status=0
GateMP_detach: LAD response, status=-5
NameServer_detach: LAD response, status=0
MessageQ_destroy: got LAD response for client 0, status=0
NameServer_destroy: entered
NameServer_destroy: got LAD response for client 0, status=0
<-- main:
________________________________________________


However, if I unbind both the IPU1 and IPU2 and bind the DSP1. I have no issus AT ALL when binding the same "ex02_messageq" example. Even when I stress test and run the application over and over I am not getting any errors at all.

Whatever IPU1 and IPU2 is interfering the linux host with, DSP1 is NOT doing it. When running the exact same IPC/TI-RTOS based example application. Any ideas?

Best regards

Marcus Berglund

Hi Marcus,

I understand now, sorry for the confusion. There aren't any differences in IPC between AM571x and AM572x that I'm aware of. They both use the DRA7XX platform so it's interesting that it would work on one and not the other. I've reached out to the development team to confirm this.

Can you provide your cfg files, rsc table, and dts file? As you mentioned, this could be an issue with memory allocation.

Hi Sahin,

It turns out that the unicache is somehow causing a major conflict in AM5718.

in IpuAmmu.cfg when

var Cache = xdc.useModule('ti.sysbios.hal.unicache.Cache');
Cache.enableCache = true;

is replaced with

var Cache = xdc.useModule('ti.sysbios.hal.unicache.Cache');
Cache.enableCache = false;

Then the IPU1 and IPU2 examples work flawlessly with USB3 camera. This is however not seen in AM5728.

Can someone please explain?

Best regards

Marcus Berglund

Hi Marcus,

As Sahin pointed out there aren't any differences in IPC between AM571x and AM572x. In order to understand the memory allocation/cacheability issue on the two platforms with different DDR memory (2GB DDR3 on AM571x and 3GB DDR3 on EVM_AM572x), the memory info outlined in the wiki would be helpful.

• Overall Linux Memory Map
• CMA Carveouts
• IPU IOMMU page table

Also instead of disabling the cache for all the memory, have you tried to pinpoint the conflicted memory page using AMMU.CachePolicy_NONE_CACHEABLE, for example:

/* Private, Shared and IPC Data regions: Large page (512M); non cacheable */

/* config large page[2] to map 512MB VA 0x80000000 to L3 0x80000000 */

AMMU.largePages[2].pageEnabled = AMMU.Enable_YES;

AMMU.largePages[2].logicalAddress = 0x80000000;

AMMU.largePages[2].translationEnabled = AMMU.Enable_NO;

AMMU.largePages[2].size = AMMU.Large_512M;

AMMU.largePages[2].L1_cacheable = AMMU.CachePolicy_NONE_CACHEABLE;

AMMU.largePages[2].L1_posted = AMMU.PostedPolicy_POSTED;

>>when testing ex02_messageq built in release mode instead of debug mode, the issue seems almost disappeared

Were you able to confirm ex02_messageq release mode has no impact to your application?

>>This is before I even run "./app_host IPU1" or "./app_host IPU2"

Some logs, e.g.Log_print0(Diags_ENTRY, "--> main:")), from IPU is sent to host through messaging transport trace buffer once the IPU image is bound and booted up. Can you try to disable the logs to see if it behaves differently?

Regards,

Garrett