Artifacts when using 2 pairs of h.264 HP decoder + encoder on C6678

Oleg, could you attach some pictures of the artifacts that you are facing? how you rule out is not the encoder? Just in case latest H264 HP enc 1_0_1_4 at: http://software-dl.ti.com/dsps/dsps_public_sw/codecs/C6678/H264HP_E/latest/index_FDS.html 

Could you send us a project on which we can reproduce the issue?

Thank you,

Paula

Hi,
1) Screenshots of the artifacts are attached
2) I figured out the yuv is corrupted already in decoder in my more complicated scenarious on our system:
I used 2 different encoders with different resolutions (one stream was resized) that encoded streams coming from the same source.
Both output streams had exactly the same artifacts.
3) I tried both new version of h.264 hp encoder that you recommended (1.0.1.4) and decoder patch from the following post:
http://e2e.ti.com/support/embedded/multimedia_software_codecs/f/356/t/277030.aspx
it did not help
4) It took me some time, but I prepared a simulation of this issue, it is really critical for us right now.
I simplified my previous scenario to the following:
core 0 - decodes input stream and sends yuv frames to core 1 via shared memory
core 1 - received yuv frames from core 0 and encodes it (single core encoder)
core 2 - decodes input stream (the same stream as core 0) and sends yuv frames to core 3 via shared memory
core 3 - received yuv frames from core 2 and encodes it (single core encoder)

see README.txt inside the project zip file for exact simulation usage

h264_enc_app.zip - simulation
surf_platform.zip - platform that I used for this simulation, should be put under <xdc_path>/packages/surf
I used C6678 evm for this simulation

Thank you for your help

Oleg

P.S. please let me know if you can reproduce the issue with this simulation

8551.surf_platform.zip

0181.h264_enc_app.zip

Hi Oleg,

We have tried to reproduce the issue you mentioned from the provided zip packages.

With difficulty we are able to build the project(after changing the paths, getting required libraries, tools like pdk etc)

When we followed mentioned procedure in readme.txt file, we are observing some issues in core 0 & core 2. 

[C66xx_0] end of simulation
[C66xx_0] [t=0x00000002:d239881d] ti.sysbios.hal.Hwi: ERROR: line 121: E_stackOverflow: ISR stack overflow.
[C66xx_0] ti.sysbios.hal.Hwi: line 121: E_stackOverflow: ISR stack overflow.
[C66xx_0] xdc.runtime.Error.raise: terminating execution
[C66xx_1] encoding1 task finished
[C66xx_3] encoder 2 finished
[C66xx_2] end of simulation
[C66xx_2] [t=0x00000001:d251fc1b] ti.sysbios.hal.Hwi: ERROR: line 121: E_stackOverflow: ISR stack overflow.
[C66xx_2] ti.sysbios.hal.Hwi: line 121: E_stackOverflow: ISR stack overflow.
[C66xx_2] xdc.runtime.Error.raise: terminating execution

We are using below tool versions(Can you please build from your side and provide us executable?):

framework_components_3_23_02_16,xdctools_3_22_04_46,bios_6_32_05_54,edma3_lld_02_11_03_03,pdk_C6678_1_1_2_5,xdais_7_21_01_07

Can you please recheck once provided package?

Hi,

It seems that the simulation finished. Did you try to look at the results (located at encoding_buffer1 and encoding_buffer2 arrays)?

Actually I think that I get similar errors, but I ignored it because in my case it is printed only after the transcoding is finished (and I saved the results on my hard disk) as well as on your output.

I think you should get similar results to mine if you look at the encoded streams.

Thanks,

Oleg

Hi Oleg,

We are able to see the corrupted output. earlier I was not loading input data properly. 

It looks like issue is random. Please let us know if you have more details like how to make this consistent, possibility of pointing out exact issue with encoder or decoder etc

I am not able to attach any images to show the corruption...

Regards,

Rama

Hi Rama,

Actually I also noticed that every time I execute the simulation I get different artifacts.

I know that the issue is reproduced only with 2 pairs of decoder+encoder and does not happen when running single transcoding (1 decoder + 1 encoder only)

I also figured out that the YUV that I get from the decoder is already corrupted (I sent it to 2 different encoders - 1 with the same resolution and the other through a resizer and I still got the same artifacts on 2 encoded streams)

Unfortunately this is all info that I got. I worked on this issue for about 3 weeks full time and I checked every single line of code in our project to find the problem and I have not found it.

It seems  like the problem is complicated, may be related to EDMA or memory access issues...

I hope you will be able to solve this issue soon because it is really critical for us.

Thanks,

Oleg

Oleg, it is possible for you, as a test, to move enc/dec to other cores?

For example:

Test1:

cores 0 - 720p decoders (A)
cores 1 - 720p decoders (B)
cores 4,5 - 720p distributed encoder (C)
cores 6,7 - 720p distributed encoder (D)

Test2

cores 0 - 720p decoders (A)
cores 2 - 720p decoders (B)
cores 4,5 - 720p distributed encoder (C)
cores 6,7 - 720p distributed encoder (D)

and let us know if you still see the artifacts? Main idea is to see who is interfering with who

thanks,

Paula

Hi,

First of all I performed the requested tests and the result is the same: in any cores configuration I get artifacts on output streams.

Some more delails that we found I want to share with you:
- If there is only 1 decoder on dsp and its output is sent to 2 different encoders, there are still artifacts in the encoded streams. I think that this is the minimal scenario that can be created to reproduce this problem.
- In our system the artifacts appear usually (but not always) after we receive the I-frame. I'm not sure if it is related to I-frame itself or frame arrival timing:
Client that we test with has one characteristic: before sending an I-frame it does not generate any frame for about 0.5 sec. So in this scenario the decoder does not work during this period, but the encoders continue duplicating frames (that's what I tried to reproduce in my simulation). So it looks like the errors are "aggregated" during this 0.5 seconds and cause to the new artifacts after decoding next frame.

Waiting for any news on this issue,
Thank you,
Oleg

Oleg, a quick confirmation, 1 enc + 1 dec works always OK, right? artifacts only appears when another encoder is added and some time down is introduced to the decoder. is my understanding correct?

thank you,

Paula

1 dec + 1 enc works always right, that is correct.

Regarding the delays, there were delays in decoding on our system, so I simulated it in my simulation, I didn't check the results without it. If it helps, i will check it tomorrow morning when I come to the office.

Oleg

Oleg, please check 1 dec + 2 enc without delays and let us know if you still get artifacts or not

thanks,

Paula

Hi Paula,

I performed the test (1dec + 2 single core encoders without delay) and I still can see the artifacts,

Attached is surf_main.c file that was changes in order to perform this test. You can just overwrite surf_main.c file in my simulation with this one in order to reproduce this scenario.

Thanks,

Oleg

6076.surf_main.c

lost attachment from previous post...

Hi Oleg,

Today we have checked multiple times & analysed generated bit stream to get further inputs on corruption, but not much information is obtained.

We are suspecting either encoder or decoder is utilizing more buffer than allocated buffer size. Can you please look into that area by checking memory near to allocated buffers. Better you can check with Shared memory first and then DDR memory. No need to check for L2 memory as it is independent for each core.

We will also check tomorrow from encoder perspective.

Regards

Rama

Hi Rama,
I did the following test:
In shared memory I added padding of 0x2000 bytes before and after each memory block that is allocated.
Then I ran the same test as I performed before. The result - I can still see the artifacts.
Thanks,
Oleg

Oleg, I will start trying to reproduce this issue in my setup, I will come back to you shortly in case of questions/issues when reproducing/debugging it.

thank you,

Paula

Oleg, a quick update, I was able to reproduce the issue.

thanks,

Paula

Hi Paula,

This is good news

Thank you for the update

Do you have any estimation for fixing this bug?

Oleg

Hi Oleg, do you think it is possible for you to add a decoder YUV buffer in your project?. It would be ideal if we can double check that indeed the artifacts are already in the decoder output. This could be challanging because I have seen the artifacts appearing after hundreds of frames. But maybe if we can capture few hundreds frames after frame, says 400, we might be able to capture the issue. Also, it would be good if we can reproduce the issue (if possible) with a smaller resolution so it would be easier to save the decoder output. 

One more question, have you tried H.264 (non HP) decoder? if so, do you get the same results?

Thank you,

Paula

Hi Paula,

As I wrote before, I'm sure that the artifacts are already in the decoded stream because of the following: (see my second post)

when I send one decoded stream to 2 different encoders, I can see exactly the same artifacts on both of these encoders. This can be reproduced both on our system and on my simulation that I sent you.

Regarding adding a decoder buffer it can be problematically: it already takes too much time (about 2 hours) to run my simulation (load 8 cores, load input  - about 7mb, and save output data - 2x15mb). I don't know how can I possibly download 400Mb from the DSP, it will take about 2 days.

I thought that you have succeeded to reproduce the problem on your setup; your setup is much more comfortable to work with, isn't it? May be you can do it using your tools, I'm sure you have more convenient ways to perform this test than me.

Regarding your second question: I tried to reproduce it using MP decoder and the problem didn't happen. The problem that I must use HP decoder because we work with 720p resolutions.

Thanks,

Oleg

Oleg, Using your project in my setup, I saw that when we have 1 dec and 2 encoder we get the same artifacts on both encoders output. I agree this point us to decoder or decoder buffer errors. My thought here is that the artifacts could be generated by the decoder or introduced inside the decoder buffer before being encoding.. the second option is what I would like to rule out. If we have the same artifacts in a decode YUV copy, then that would be a strong statement that the decoder is introducing them, Contrary if there are not artifacts inside the decode YUV copy, then, we will need to check the rest of the framework and see where they are introduced. Anycase, I understand it could be challenge to get a copy of the decoder output.

About the MP decoder, no need to migrate back, I was just courious if you see the same issues there in order to get more clues.

thank you,

Paula

Oleg, we will start checking if there is any DMA issue with the decoder. 

thank you,

Paula

Hi Paula,

I want to share some additional information with you that may be will help you.

We did the following test:

2 decoder + 2 encoders

one of the decoders save its first frame in a separate buffer and only this frame is always encoded by the encoder. So we get a static image in video. (The decoder that sends static output is started several seconds after the first decoder - in this case we usually get artifacts on the second decoder right on the beginning if we run it on our system)

The interesting thing is that in this frame we already see artifacts and it remains without any changes for the whole video length.

Hi Oleg, due to some business traveling starting next week (and few days of current week) I will be working on DMA test by the end of next week. I will keep you post it of the results.

thanks,

Paula

Hi Paula,

We have another 2 problems, but I'm not sure if it can be related to this issue or may be I should open different post on it.

1) We saw a core crash in h.264 HP video encoder several times. The crash was somewhere in process function call. How can we address this? Do you have any tools/debug versions/traces in the codec so we can somehow understand what was the cause of the crash. We don't have an exact scenario to reproduce this.

2) Sometimes the bitrate of H.264 HP encoder drops drastically and after some time it returns to its normal values. During the low bitrate period the stream quality is awful. I saw it today on CIF 30fps with 256000 bitrate. Each p-frame frame was only 60-100 bytes long. As a result I observed extremely high qualtization.

Thanks,

Oleg

Hi Oleg,

We have feature to enable debug log inside codec to get status of last few frames before hang. To enable this we need to recompile the code. We can provide this library. Can we get the configuration used for encoding, which scenario crash is observed like multicore, only encode thread, or how many encode threads, any overclocking etc...

To see the bit rate issue can we get the generated bit stream?

Regards

Rama

Hi Rama,

It would be great if you can provide me the library with debug mode enabled. How can I get those traces? I saw some explanation about traces in the user guide doc, is this the same traces?

I'll try to obtain the encoder output with very-low-bitrate problem.

Thanks,

Oleg

Hi Oleg,

Please find attached library with debug trace enabled. Yes you need to follow user guide to get trace dump.

You can set  "DebugTraceLevel", "lastNFramesToLog" parameters while creating instance. Once at the end of frame process using control call with GET STATUS command you can get DDR pointer for debug trace buffer. Please send us the buffer dump to analyse when crash happens.

7536.h264hpvenc_ti.zip

Regards

Rama

Hi Oleg, I need to do further testing but so far it looks like the artifact issue is related to H.264 HP dec DMA linked transfers. Currently if I add "DMA_wait()" after all linked transfer are used, I don't see the artifacs (more testing is need it to confirm).

Next step is to narrow down which of the DMA linked function is introducing the artifacs (or if more than one) and finally, check inside of the decoder where linked transfer(s) is (are) used.

I will keep you post it

Thank you,

Paula 

Oleg, a quick update, I narrow down the issue to one type of DMA transfer (1D2D linked) currently locating inside the decoder where this type of transfer is used and see why the data gets corrupted.

I will continue keep you post it,

thanks,

Paula

Hi Paula,

We have been testing this new library today and currently there were no artifacts observed!

We tested it also with less complicated scenarios (similar to the simulation that I sent earlier) and also with the full system and for now it seems that the problem was solved. We will continue testing it.

Some additional questions:

-Do these changes have any influence on performance of the decoder?

-What are the next steps? Are you going to release an official release? 

-Is this version good enough to use it on a real system? (in terms of debug changes, etc.)

Paula, thanks a lot for your great effort! This is very important for us and I appreciate your contribution and very short response times

Oleg

Hi Oleg, Thanks for helping me to test it in your side. I will open an IR with the codec team, so they can check and incorporate a fix on the next official release. In other words, this was a temporary fix (proof of concept) and we need to take a closer look of the issue/fix. But, in the meantime it can be used.

Best Regards,

Paula

Oleg, the codec team shared a fix, I test it on your project and it seems it is OK. I am attaching you lib so you can test it on your side. Please let us know your findings.

6457.h264hpvdec_ti.le66.txt

Thank you,

Paula

Oleg, FYI, you can get latest H.264 HP dec (with DMA wait fix) at:

http://software-dl.ti.com/dsps/dsps_public_sw/codecs/C6678/H264HP_D/latest/index_FDS.html 

thank you,

Paula