Crash in algActivate on C6678

Hi Oleg, the scenario is not completely clear for us, it would be good to know how you are Act/Deac the algorithms if you do cache invalidation before and after, how you sync mixer output and encoders? etc. Also did this hangs happens with different video content or only with the clip you mentioned? you mentioned that if you remove any component it works so if you use directly output YUV from H.264 dec as input it works?

No sure if it is related to http://e2e.ti.com/support/embedded/multimedia_software_codecs/f/356/p/292415/1051035.aspx but if you use another decoder (for example Mpeg2) everything works ok?

thank you,

Paula

Hi Paula,
- I checked cache operations and added cache wb/inv before and after each algActivate and algDeactivate.
But it does not solve the crash.
- The YUV is passed from decoder to mixer and from mixer to encoder through shared memory, you have this code in artifacts issue simulation that was resolved earlier.
- If I pass the yuv directly from decoder to encoder, the crash disappers
- If I use another decoder the crash also disappers.
- The crash can be reproduced only (currently) with a stream that is coming from a specific client. It disappears if I use stream with the same parameters (resolution and fps) generated for example by ffmpeg.

As a result I think that the main suspect is h264hpvdec library.

You can use me as a tester for your debug versions if you want
You can save in memory all required info and I'll reproduce the crash and read the info for you.

Thanks,
Oleg

Hi Oleg could you please share with us the clip with which you can reproduce the issue? Also which core crashes (enc/dec/mixer)? When it crashes?, inmidiately or after transcoding several frames? could you please share your project .cfg, cmd, .map? if there is any concern about sharing here these files you can send them to me through a conversation.

thank you,

Paula

Hi Oleg,

Can you please help us understand the hang issue you are encountering? Below I will try to describe your use case. Please make corrections and answer questions as needed.

Core 1: 1 channel of H264HP Decoder

Core 3: 1 channel of H263 encoder (30fps), 1 channel of H264 encoder (5fps), 1 channel of H263 encoder (5fps)

Core 7: 1 channel of video mixer, which receives YUV from Core 1 and sends mixed YUV to Core 3

Call flow:

1) Create codec/mixer channels on Core 1, Core 3, and Core 7 as above

2) Input H264 stream is sent to Core 1, which decodes the bit stream to YUV data

3) Decoded YUV data is sent to Core 7, which does post processing of YUV data, and creates two different copies of YUV data

4) Mixed YUV data 1 is sent to H263 encoder (30fps) on Core 3, Mixed YUV data 2 is sent to H263 encoder (5fps)  and H264 encoder (5fps) on Core 3. Encoders does the encoding and sends the encoded bit streams out.

Is the above sequence correct for your use case? If not, please provide similar step by step description of your use case.

Assuming the above call flow, when is crash happening? On Step 1), 2), 3), or 4)? When the crash happens, which core(s) and/or codec/mixer channels(s) hang?

In your use case, core 3 is special as it has three codec channels. Since all the codec channels are sharing the same set of EDMA resources, please ensure only one encoding channel is doing the encoding at one particular time point. Concurrent encoding process calls are not allowed on the same core.

Thanks,

Hongmei

Hi,
To Paula:
Attached is the h.264 stream that I use to reproduce the crash

8867.aver_content.h264.txt

The core crashes after several frames (in my scenario it crashes after 33 frames)
In my current scenario the crash occurs on H.264 HP decoder core.
It seems like this core, after it begins executing algAlloc, start to corrupt memory, and after about 2-3 minutes 2 other cores also crash (because their memory was corrupted)
I can be sure that algAlloc corrupts memory because I copy the frame buffer to a debug buffer before algAlloc is called;
after the crash I read the memory and can see that these 2 buffers differs and there is junk in the original buffer.

map, cfg and cmd files are attached:
6136.app.cfg

4452.lnk.cmd.txt

2465.C6678.map.txt

To Hongmei:
Core configuration is correct
Call flow:
1) Create codec/mixer channels on cores 1,3,7
2) Decoder runs state machine that received input from RTP and when whole frame is available feeds it to the h.264 hp decoder library. When there is ready yuv from the decoder, it is passed to the mixer on core 7.
3) The mixer creates 2 mixer yuv streams (30fps and 5fps) and passes it to encoders on core 3.
Generation of these yuv is synchronized to timer, so if there is no new frame from the decoder, the output yuv frame will be duplicated.
4) 3 encoders on core#3 run (sequentially, our scheduler runs it one after another, it is running inside single sys/bios task) when it receive new yuv frame from mixer, generate new encoded streams and sends it through rtp to the client.

The crash happens on step 2, when executing algActivate.
If you look at the attached stream, the crash occurs before starting decoding frame with offset 0x9913E

Regarding the EDMA resources on core#3, I confirm that there is only one encoder running at any particular time point because it is run sequentually.

Thanks,
Oleg

Hi Oleg, it is pretty interesting that the crash happens at 0x9913E, which is an I-frame (frame 50, 62 IDR Nal units) after several small P-frames. I think you locate input H.264 bitsteam at DDR_NONCACHE, and no sure who else is sharing this section. Just wondering if this might be a little bit tight in size. We will continue investigating.

thank you,

Paula

Hi Paula,

I also find it interesting. Moreover, earlier it crashed on another offset that is also start of an I-frame.

On the other hand, the core crashes before this I-frame is fed into the decoder, on algActivate, before the process method was called with this I-frame.

The input frame it located in cached DDR and the input buffer is about 320K size that is more than enough for a single frame. Every time I get a marker bit from RTP (end of frame) this frame is fed to the decoder so only one frame at a time is saved in this buffer.

thank you and have a pleasant holiday!

Oleg

Hi Oleg,

Thanks for the clarification and the files.

From your .cfg file and .map file, it looks like there are two stacks of size 0x200. Can you please check if there is stack overflow  (using CCS ROV) when crash happens? In ROV, please also check if there is heap overflow.

In your map file, .cio and .args sections are placed in DDR. Please move them to local L2.

b20efd80 b20efd80 00000120 00000000 rw- .cio

b1077ef0 b1077ef0 00000008 00000008 rw- .args

Thanks,

Hongmei

Hi Hongmei,

I moved .cio and .args sections to IRAM

I also checked stack overflows with memory read, the stack is filled with 0xbebebebe values, so I can see what's the usage of the stack.

There is no stack overflow.

How can I check if the heap overflows?

Thanks,

Oleg

Hi Oleg,

Attached please find the screen capture for how to use CCS-ROV to check heap usage and stack status.

7455.ROV.zip

Can you please also check if there is packet loss for the incoming RTP packets to the decoder?

Thanks,

Hongmei

Hi Oleg, wondering how your debug is going. Please let us know

thanks,

Paula

Hi,

Sorry for the delay, I was not in the office for the last week

I didn't check the heap overflow because of some difficulties with JTAG connection on our board.

But I'm pretty sure that this is not the problem since I tried to increase all heaps by factor of 3 and the crash still happens.

Can you proceed without this test? Because it will be very hard for me to do that...

Thanks,

Oleg

Hi Oleg, is incoming RTP packet loss rule out?

Thanks,

Paula

Hi Paula,

There is no packet loss in RTP

Oleg

Oleg, I guess the question is more if the decoder input buffer is always OK (rule out any overflow in the RTP stack). Also in the other e2e thread I uploaded an H.264 Dec fixed lib.

Thank you,

Paula

Hi Oleg, first at all, happy holidays. On the other hand I would like to touch bases with you and see if you have had a chance to check latest H.264 Dec lib.

thank you,

Paula 

Hi Paula,

I wish you a happy and joyful new year!

I checked the H.264 decoder, but only in my small tests, it still was not checked by our QA team.

In my minor tests it looks, no aftifacts were observed.

In these days we are releasing stable version after many QA cycles, so we don't want to change the library now, we still use the fixed lib that you gave me, not this official version.

So, in my tests it looks good, but I don't have any official results from our QA.

Thank you and have a happy holidays!

Oleg

Hi Oleg, thanks for the update, quick question how is going the debug for the dec + mixer + enc issue?

thank you,

Paula

I stopped debugging this issue, we are trying to avoid this specific scenario in the system.

Oleg, thanks for the update

Paula