This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: Color shift after decoding of TDA4's H264 stream

Part Number: TDA4VM
Other Parts Discussed in Thread: TDA4, TEST
  1.  1920x1080 nv12 color image encoding on tda4-sdk73(encoded parameter as follow):
    mm_enc_ctrl_params: {
        features = MM_ENC_FEATURE_CABAC | MM_ENC_FEATURE_8x8
    ​    rcmode = MM_ENC_VBR
        idr_period = 1
        i_period = 1
        bitrate = 40000000
        framerate = 30
        crop_left = 0
        crop_right = 0
        crop_top = 0
        crop_bottom = 8
        nslices = 1
        base_pipe = 0
        initial_qp_i = 0
        initial_qp_p = 0
        initial_qp_b = 0    
        min_qp = 0
        max_qp = 0
        min_blk_size = MM_ENC_BLK_SZ_DEFAULT
        intra_pred_modes = 0
    }
    mm_vid_create_params: {
        width = 1920
        height = 1088  /*ALIGN(1080, 64)*/
        in_pixel_format = MM_PIX_FMT_NV12
        ​out_pixel_format = MM_PIX_FMT_H264
    }
     
  2. Decode H264 encoded data of 1920x1080 raw data
    Experimental result
    Desired result
    experiment_1920_1088.png desired_image_1920_1088.png
                                                                                                                                                                                                                                                                             
  3. Result analysis
When the original width(1920) and height(1080) are not multiples of 16 or 64, the current sdk73 encoding causes color shift
  • Hello billy he,

    Could you please share the output images? It would be bit difficult to view the images and figure out if the entire UV is shifted, or just portion of the image. 

    Also it looks like only red light is moved, but color seems also changed, the blueish banner became red'ish. 

    Please share the images to analyze it further.

    I am in parallel checking with codec team to see if there is any restriction in image size.. 

    Regards,

    Brijesh

  • Hello Brijesh Jadav,

    please see the table below, Comparison before and after encoding of the same data frame
    :

                                  raw-frame                                                       H264-frame(ffmpeg translate H264 to jpg file)   
              
  • Hello billy he,

    I meant, can you please share nv12 image generated from decoder? 

    Regards,

    Brijesh

  • Hello Brijesh Jadav,

    I'm very sorry, I was busy with work and couldn't provide the data in time.

    The H264 code stream generated by TDA4 is decoded by TDA4, and the color will not be offset,
    but the color will be offset when decoded by ffmpeg.
    We believe that the H264 code stream
    obtained by TDA4 encoding is a general code stream that can be used on any platform that conforms to the H264 standard.
    But according to the current experimental results, H264 stream can only be parsed normally on TDA4.

    At the same time, I have detected that the PSNR value is only 36(the general value should be above 45), the distortion rate is high.
    please see the table below:
    raw-data decoded by tda4(r5f) decoded by ffmpeg in x86/arm
    frams.bin:raw_last_frame.bin.tar.gz decoded_last_frame_by_tda4.bin.tar.gz
    Regards,
    Billy
  • Hi Billy,

    Can you please help us to reproduce this issue? The below points would help:

    • SDK version which you used.
    • Where are you setting the encoder parameters?
    • What is the command used to decode the h.264 file using ffmpeg?
  • Hi Prashanth,

    1.SDK version is "ROCESSOR-SDK-RTOS-J721E_07.03"

    2.encoder parameters:

    1920x1080 nv12 color image encoding on tda4-sdk73(encoded parameter as follow):

    mm_enc_ctrl_params: {
        features = MM_ENC_FEATURE_CABAC | MM_ENC_FEATURE_8x8
    ​    rcmode = MM_ENC_VBR
        idr_period = 1
        i_period = 1
        bitrate = 40000000
        framerate = 30
        crop_left = 0
        crop_right = 0
        crop_top = 0
        crop_bottom = 8
        nslices = 1
        base_pipe = 0
        initial_qp_i = 0
        initial_qp_p = 0
        initial_qp_b = 0    
        min_qp = 0
        max_qp = 0
        min_blk_size = MM_ENC_BLK_SZ_DEFAULT
        intra_pred_modes = 0
    }
    mm_vid_create_params: {
        width = 1920
        height = 1088  /*ALIGN(1080, 64)*/
        in_pixel_format = MM_PIX_FMT_NV12
        ​out_pixel_format = MM_PIX_FMT_H264
    }
    3. command in x86  using ffmpeg:  ffplay  -i  H264.dat
    Regards,
    Billy
  • Hi Billy,

    Thank you for sharing the details.

    I tried to encode the input file shared by you. But I used the width and height(1920x1080).

    The encoded file can be played with ffplay also was able to decode on TDA4. There was no color sift or other issues seen.

    Can you please refer to

    • ti-processor-sdk-rtos-j721e-evm-07_03_00_07/tiovx/kernels_j7/hwa/test/test_video_encoder.c for the parameter settings(reads input file, width and height from the cfg file)?
    • ti-processor-sdk-rtos-j721e-evm-07_03_00_07/tiovx/conformance_tests/test_data/tivx/video_encoder/*.cfg for width and height settings?
    • Attaching the snap shot of ffplay   
  • Hi Prashanth,

    Actually, I did the same experiment as you on TDA4, WxH(1920x1080) is not aligned(1920x1088) by 64, and set into encoder on tda4.

    The encoded file can be played with ffplay also was able to decode on TDA4. There was no color sift or other issues seen.

    But part information is incorrect in H264 code stream file.

    could you please check your encoded file from tda4 with ffprobe tool.
    Use the command "ffprobe -show_streams  H264_file " to view the encoded file information:



    Regards.
    Billy

  • Hi Billy,

    I did not see the issue in my encoded file. Below is the screen shot.

    In the screen shot shared by you, I saw few warning messages, which is not seen in the file encoded at our side.

    Can you please check if any setting is different than the one suggested in the previous post?

  • Hi  Prashanth,

    Thanks for your information about encoded file.

    I have performed two experiments.
    Experiment 1: Encode parameters according to the values ​​I provided to you.
    Encountered color offset when decoding on other platforms (ffplay)

    Experiment 2: Only modify the encoding parameter height=1080, crop_bottom=0 (in experiment 1, the value is 1088, crop_bottom=8),
    the others remain unchanged, and the color shift will not be encountered when decoding on other platforms (ffplay).
    But weight x height value in encoded file is incorrect (1920x1072).


    Maybe your environment sdk has fixed this issue ?
    Can you please share your demo code. I run it on my tda4 platform?


    Regards,
    Billy
  • Hi Billy,

    I have used the SDK7.3 code and you can refer the below files where the parameter settings are shown.

    1. ti-processor-sdk-rtos-j721e-evm-07_03_00_07/tiovx/kernels_j7/hwa/test/test_video_encoder.c for the parameter settings(reads input file, width and height from the cfg file)?
    2. ti-processor-sdk-rtos-j721e-evm-07_03_00_07/tiovx/conformance_tests/test_data/tivx/video_encoder/*.cfg for width and height settings?

    On Experiment 2: According to  H.264 spec, the height and width in decoder are calculated as below:

    • Width = ((pic_width_in_mbs_minus1 +1)*16) - frame_crop_right_offset*2 - frame_crop_left_offset*2;
    • Height = ((2 - frame_mbs_only_flag)* (pic_height_in_map_units_minus1 +1) * 16) - (frame_crop_top_offset * 2) - (frame_crop_bottom_offset * 2);

    So, setting crop_bottom = 8 will evaluate the height as 1088 - (8*2) = 1072. Please set crop_bottom = 4, so the height will be 1088 - (4*2) = 1080.

  • Hi Prashanth,

    Thanks for your  suggestions.

    I will read the demo source code first.

    I will reply to you as soon as I have a conclusion

    Regards.
    Billy

  • Hi Prashanth,

    according to your opinion, I set crop_bottom = 4,The results of the two coding experiments are as follows:

    experiment NO.1 experiment NO.2
    encoder parameter
    information from encoded file
    analysis height = 1088 - crop_bottom(4) = 1084

    height = 1072 - crop_bottom(4) = 1068

    1072 is a number less than 1080 aligned by 16.

    In MM_Create_Enc, 1080 should be rounded down by 16 and passed to the encoder

    Regards,

    Billy

  • Hi Billy,

    Yes, your interpretation is correct. Sorry for the confusion created.

    • The reason I suggested to set the crop_bottom to 4 is, hardware will evaluate it as 4*2
    • Here in the RTOS driver, we should set the exact lines of bottom crop i.e, 8 (which is used by you previously).
    • The driver will take care the crop_bottom internally and send it to hardware as per h.264 spec. You can see in video_codec/ti-img-encode-decode/timmlib/encoder/mm_enc_create.c:    ctx->crop_params.bottom_crop_offset = parm->crop_bottom / 2;
    • Please refer to the comments section in tiovx/kernels_j7/include/TI/j7_video_encoder.h for crop_bottom. It states:
      •     Crop settings set how much to crop off of each side from the original image. \n\n
        NOTE: Width and Height of the video buffers must be 16-aligned to work with the driver.
      • By default if a non-16-aligned value is input for buffer width/height, the driver will round down to 16-align.
      • In order to get a full width/height from a non-16-aligned stream size the application must read the data into the buffer starting at the "upper left" corner, making sure lines are width aligned. Then an appropriate crop must be set for the
             *  encoder to crop out the empty data along the edges.
             *  ex. 1920x1080
             *  Set to 1920x1088 width/height. Upon filling the input data this will leave 8 lines of empty data in the bottom of the buffer.
             *    Set crop_bottom = 8 to crop the empty 8 lines at the bottom of the buffer to avoid green lines.
             *  NOTE: crop values can only be set in multiples of 2. Any odd numbered crop will be rounded down in the driver.
             *  These should be set to 0 by default.
  • Hi Prashanth,

    I have also read the codec user manual and followed the given advice to set the width and height to 16 alignment to the encoder.

    experiment NO.1 experiment NO.2 experiment NO.3
    encoder parameter
    H264 code stream on tda4
      
    
    
    information from encoded H264 file

     w = 1920

     h = 1080

     w = 1920

     h = 1064

     w = 1920

     h = 1072

  • Hi Billy,

    Unfortunately this behavior is not seen in the SDK 7.3 and everything works fine.

    Why is the crop_bottom set only when height >= 1088?

    There is another sample application in "../ti-processor-sdk-rtos-j721e-evm-07_03_00_07/video_codec/examples/apps/encoder/app_encoder_test.c" which is similar to the tiovx testapp.

  • Hi  Prashanth,

    Thanks for your support !

    BR