Codec Engine universal_copy example is unstable with HDVICP&HDVPSS firmware loaded

As a supplement, the above tests have been made with a DM8168 EVM, software built with EZSDK 5.05.01.04 (apart from the updated Codec Engine package).

I can reconstruct the same problem too, but at an DM8148 EVM - additionally I tested with CMEM_MODPARAMS phys_start=0x96C00000 and phys_end=0x97ffffff, although at the mmoment I don't see a memory conflict for the memory range given above.

After a couple of starts of the test program universal_copy_remote.xv5T starting DSP fails, and afterwards I have to reboot to get it run again.

Thanks in advance for any hint,
Joern.

Some further testing: while calling the CE test application multiple times with and without HDVICP and HDVPSS firmware loaded, to ensure that the effect only occures when the video firmware has been loaded, I observed a significant difference in timing.

root@dm814x-evm:/anywhere/yat# CE_DEBUG=1 ./universal_copy_remote.xv5T
[t=0x00000005] [tid=0x40020000] ti.sdo.ce.Engine: [+6] Engine_init> CE debugging on (CE_DEBUG=1; allowed CE_DEBUG levels: 1=min, 2=good, 3=max)
[t=0x000002e7] [tid=0x40020000] xdc.runtime.Main: main> ti.sdo.ce.examples.apps.universal_copy
[t=0x0000068c] [tid=0x40020000] xdc.runtime.Main: [+1] App-> Application started.engineName remote_copy_dsp input-file ./in.dat output-file ./out.dat.
[DSP] [t=0x00154a86] [tid=0x154a86] xdc.runtime.Main: main-> Adding alg to server...
[DSP] [t=0x0017ca60] [tid=0x17ca60] xdc.runtime.Main: main-> Calling BIOS_start()...
[t=0x0001b2c9] [tid=0x40020000] xdc.runtime.Main: [+1] Alg version:  1.00.00.00
[t=0x0001b429] [tid=0x40020000] xdc.runtime.Main: [+1] App-> Processing frame 0...
[t=0x0001b8ae] [tid=0x40020000] xdc.runtime.Main: [+2] App-> Alg frame 0 process returned - 0x0
[t=0x0001b9d4] [tid=0x40020000] xdc.runtime.Main: [+1] App-> Processing frame 1...
[t=0x0001be26] [tid=0x40020000] xdc.runtime.Main: [+2] App-> Alg frame 1 process returned - 0x0
[t=0x0001bf11] [tid=0x40020000] xdc.runtime.Main: [+1] App-> Processing frame 2...
[t=0x0001c36b] [tid=0x40020000] xdc.runtime.Main: [+2] App-> Alg frame 2 process returned - 0x0
[t=0x0001c456] [tid=0x40020000] xdc.runtime.Main: [+1] App-> Processing frame 3...
[t=0x0001c8af] [tid=0x40020000] xdc.runtime.Main: [+2] App-> Alg frame 3 process returned - 0x0
[t=0x0001c96e] [tid=0x40020000] xdc.runtime.Main: [+1] 4 frames processed
[t=0x0001d880] [tid=0x400a6490] ti.sdo.ce.ipc.Processor: [+2] Processor_delete_d> defaultHeapH: 0x139170, proc->heapH: 0x139170
[t=0x0002bd23] [tid=0x40020000] xdc.runtime.Main: [+1] app done.
root@dm814x-evm:/anywhere/yat#

After the line with "Processor_delete_d" the behaviour is differently:

• if video firmware is loaded: immediately the next output line follows
• if video firmware is NOT loaded, then there it needs some seconds (around 5 form me) until proceeding

May that be that with loaded firmware some CE or DSP component is not correctly shut down by the CE framework? How to hinder that?

Regards,
Joern.

P.S.

At least a time out of 8 seconds before each next starting of the universal_copy test program made not really a difference - so it seems not to be a simple question of timing.

I'm hoping someone who knows more about the HDVICP & HDVPSS firmware will respond. But meanwhile, have you looked at these suggestions:-

http://processors.wiki.ti.com/index.php/SysLink_FAQs#Runtime_questions

Gunjan said:

I'm hoping someone who knows more about the HDVICP & HDVPSS firmware will respond. But meanwhile, have you looked at these suggestions:-

http://processors.wiki.ti.com/index.php/SysLink_FAQs#Runtime_questions

Thanks for your input, Gunjan. I have tried to add the DM timer into the server cfg file before, and the sample universal_copy will always fail. The error message from debug log is the same (i.e. Ipc_control(STARTCALLBACK) status: -1)

Hi Gunjan, thanks for your hint, from me too. From my experience, if that timer wasn't adjusted corrrectly, then the application would not have been started at all.

Additional observations, maybe help to find the decisive hint:

For me it seems a bit that the amount of successful starts might be dependent on the CE_DEBUG setting - the higher it is, and the more output I get, the faster the effect occures, that hinders to load the DSP software oncemore.

And my colleague found and pointed out to me, that sys_top shows a memory use value increasing with each call to our little CE test program:

Initially I see

SRIn 0 :PhyA 0x9f700000 Virt 0x0        Size 0x200000   
 SRHeap:Size 2095488    Used 16896      MaxU 16896      Free 2078592    LarF 2078592   

for HDVPSS and HDVICP, and some CE test program calls later it looks like that, f.e.:

SRIn 0 :PhyA 0x9f700000 Virt 0x0        Size 0x200000   
 SRHeap:Size 2095488    Used 1374080    MaxU 1374080    Free 721408     LarF 721408 

The video firmware for the HDVPSS and the HDVICP is just loaded, I made no further use of them while that test. Other values changed, too, but went back to their initial values after end of the CE test program.

The red marked, increasing value at the first glance increases faster when CE_DEBUG has a higher value (and more debug output is generated). It never reached the maximum, the effect sometimes occured with smaller values, too. But in general, if that increasing memory use value has to do with the ugly effect or not - isn't that something that better should not happen?

By the way, the firmware version for both video components is 05_02_00_46 at the moment.

Hoping for a helpful hint,
best regards,
Joern.

Just trying to understand that last observation regarding the SRHeap values. When exactly are those two measurements made ?

Is it for before and after your CE test program is run or during ? Do you observe different behavior when firmware is loaded and when it isn't ?

Gunjan said:

Just trying to understand that last observation regarding the SRHeap values. When exactly are those two measurements made ?

Is it for before and after your CE test program is run or during ? Do you observe different behavior when firmware is loaded and when it isn't ?

Hi Gunjan,

In Joern's test, the first values should be before any CE test and the second is after running CE test for some times. If the hdvicp&hdvpss firmware is not loaded, there's no memory data (i.e. SR0/SR1) to display, for sys_top seems not to be integrated into DSP.

Hi Gunjan,

it was exactly like Robert explains, the first measurement was taken after loading video firmware, the second after having the CE test program started a couple of times.

To be clear: the "used" value for SRHeap increased each time when I started the CE test program, but never decreased after the CE test program came to an end.

Regards,
Joern.

Supplementary information:

Obviously the probability of the issue occurence (CE example program cannot be loaded oncemore) is dependent on the firmware load order:

1. load-hd-firmware.sh start, loadmodules.sh
2. load-hd-firmware.sh start, loadmodules.sh, load-hd-firmware.sh stop, load-hd-firmware.sh start
3. load-hd-firmware.sh start, load-hd-firmware.sh stop, load-hd-firmware.sh start, loadmodules.sh
4. loadmodules.sh load-hd-firmware.sh start

For 1) commonly the issue occures after below 100 restarts of the CE example program, often only around 10 or 20

For 2-4) the probability of multiply possible restarts of the CE sample app seems to be indeed high, that I got at least near to 100, often more, and sometimes above 200 - in the latter case the stop obviously was directly bound to the above explained memory leak problem - there was simply no more remaining SRHeap memory. Each call of the CE example app left an around 10.000 bytes wide memory leak.

Two times I observed for 4) a much lower memory leak of "only" 128 bytes for each call of the CE example program (one test is still running, with over 800 restarts in the meanwhile). Interestingly in exact these cases I observed that difference in timing, which I already reported here. But this time the video firmware was (or from sys_top seemed to be) loaded: some seconds wait before the end of the CE example application. If that wait does not occure, I always see around 10.000 bytes memory leak.

I think, here are two issues, the memory leak on one side, and additionally another issue, maybe concerning use of syslink, which leads to our issue, too. Both of them at the moment for me look situated within the TI's firmware, within the drivers around, or maybe within the CE framework. I'd be happy if someone could spend a deeper look into that. If you need any additional information, please let us know.

Hopefully,
best regards,
Joern.

P.S. All tests have been made after soft reboots only - and I don't know how much of the concerning hardware actually is reset by this, so I cannot be sure if they have been influenced from each other.

A team member pointed out this note in an old (er) EZSDK user guide:- 

Note! The Codec Engine examples cannot be run out with graphics. Please execute the following steps to
teardown the graphics plane and ensure that no firmware is running...

Maybe someone from the EZSDK team can corroborate (or do you  have the user guide to look it up and confirm ?)

Gunjan said:

A team member pointed out this note in an old (er) EZSDK user guide:- 

Note! The Codec Engine examples cannot be run out with graphics. Please execute the following steps to
teardown the graphics plane and ensure that no firmware is running...

Maybe someone from the EZSDK team can corroborate (or do you  have the user guide to look it up and confirm ?)

Thanks for your help, Gunjan.

Yes, this statement is included in the EZSDK 5.05 user guide. I guess one purpose of mentioned steps is to avoid the memory conflict.  There is a todo in EZSDK5.03 user guide (http://processors.wiki.ti.com/index.php/DM814x_EZ_5.03_Software_Developers_Guide).

TODO: Which steps are necessary to get firmware started by load-hd-firmware.sh running parallel to Codex Engine examples...

As this TODO entry had been done by myself (:-| the typo inclusively), I should add some information here concerning this:

When I placed this little TODO entry I did this just in the hope that some of the TI people would see it and at least think of this matter. It didn't happen. At least not there. (But already before some hints waited to be explored within the EZSDK Memory Map page: see below EZSDK_Memory_Map#Modifying_Memory_Map.)

For some time we started the developed components only separately from each other because of this matter.

Then we restarted investigating the reasons of that constraint on the codec engine examples. The happy ending (at least partially) you find within this thread, which also had been mentioned by Robert in his first entry here. The rest main problem concerning memory mapping had been the differences in naming of the several memory ranges (EZSDK Memory Map vs. names in serverplatform.xs and configuration files like that).

Now we have this status:

• DM8148 (same effects on DM8168), EZSDK 5.05.01.04
• /etc/init.d/pvr-init and /etc/init.d/matrix-gui-e are not run at all, as we don't need display features - so we don't have to call them with stop parameter
• /etc/init.d/load-hd-firmware.sh is (at least at my platform) still disabled, too, but for my tests I run it by hand (with the "start" parameter, of course)
• Aftere correctly changing the memory map settings fitting to what is set for the video firmware, probably more or less all the CE examples may be run in parallel to the loaded video firmware (loaded by load-hd-firmware.sh), but mostly I test with the ex01_universalcopy_remote example from codec_engine_3_23_00_07, and the above observations are done with that sample.
• But two obscure effects remain: after multiple starts and ends of the CE sample app (that means loading and unloading the DSP server application) more and more the memory of a shared region gets lost, and (not in each case obviously dependent) earlier or later it is no longer possible to load the DSP server application oncemore.

The last point of this status list is yet open, we have decribed it above in detail. Also we made the observation, that the effect does somehow depend on the load order of the firmware, see here, being hopefully that this might help someone to get an idea of where the reason for the problems is.

If you need any further test details, please let us know.

Hopefully,
Joern.

Hi, any further comments? Shall we yet test anything, do you need some additional information?

Regards,
Joern.

Can you confirm the version of SysLink you're using?  In the EZSDK you mention (5.05.01.04) I think that's 2.10.03.20.

If that's the case, there have been several fixes over the last few months.  Here are the SysLink releases:
     http://software-dl.ti.com/dsps/dsps_public_sw/sdo_sb/targetcontent/syslink/index.html

Looking at the latest in your 2.10.* stream, I see 2.10.08.35.  In the 2.10.08.35 Release Notes, I see SDOCM00094670, which fixes a loadCallback/stopCallback leak, which I think you're exposed to when using Codec Engine.

If you stick within a "Major.minor" version stream (e.g. 2.10.*), upgrading should be easy (going outside the Major.minor stream, upgrades may be possible, but may require updates to dependencies which could ripple through the stack).  I guess I'm suggesting you try upgrading to SysLink 2.10.08.35.

Chris

Hi Chris,

thanks for your hint to the syslink version - our EZSDK 5.05.01.04 came with syslink 2.20.00.14, and all above obvservations were done with this driver version (and that syslink version is also reported while loading of the syslink module).

The EZSDK Software BOM announces for EZSDK 5.05.01.04 for both platforms, DM8148 and DM8168 that it would come with syslink 2.10.03.20 (as you feared) - but that seems just to be an error and is maybe somehow connected to that these lists are headed "EZSDK 5.05.00 DM814x Software BOM" and "EZSDK 5.05.00 DM816x Software BOM" - I never saw this maybe early canceled 5.05 version of EZSDK.

My colleague already made some tries (at dm8168 platform), and he reported this:

(Where "hangs" always means: DSP firmware cannot be loaded again.)

I'll try to look deeper into that with my platform here (dm8148), especially concerning the most current syslink version...

Until soon,
Joern.

Thank you for that hint, I'll try it out.

My current IPC is 1.24.03.32, as it came with EZSDK 5.05.01.04. For the IPC you suggest concerning this remaining little memory leak it is recommened to use SYS/BIOS 6.34.03.19 (I have 6.33.05.46) and XDCtools 3.24.05.48(I have 3.23.03.53)... probably simply because tested with these versions - but maybe for more serious reasons. At another thread I already had been suggested to update Codec Engine (and I did).

More and more it looks for me as if I should better from time to time gather my own EZSDK from the current components (especially as it is never clear when the next EZSDK will come). Not to be misunderstood: I absolutely welcome that these components obviously are maintained actively. But maybe within the TI's Wiki we should start a little "Howto make EZSDK being up to date". Besides step-by-step lists especially a dependency tree was of interest, and maybe some hints which features at least should be tested after combining new components. I'd be happy if the TI's EZSDK builders might provide some first, non-binding thoughts concerning that.

Thanks again,
Joern.

Thanks a lot for your help, Chris.

I have verified on my DM8168 with IPC 1.25 and the memory leak is gone. 

Here are my software packages to solve the CE stability and memory leak issues:

ipc 1.25.01.09

syslink 2.21.00.03

codec_engine 3.23.00.07

BTW, I agree with Joern that a wiki page would be very helpful for the latest software packages.

Hello

i read your posts and they are helpful for me but i still see a strange problem. I modified the memory map in .cfg of the CE server. I run the universal_copy and it works. But if i run the capture_encode example and in the same time i run universal_copy, when capture_encode end i see this error:

http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/717/t/260901.aspx

The final program that i want to make is a capture_encode example with one (or more) algorithms that runs on DSP at the same time on the images grabbed from the input. Is this possible with CE and OMX?

Can anyone help me on this problem?

Thanks.