TDA4VM: Camera Output Format Mismatch with v4l2h264enc Input - Conversion Needed?

LingHui Zhang

Prodigy 20 points

Part Number: TDA4VM

Tool/software:

I've found a format mismatch between my camera's output and the v4l2h264enc encoder's input requirements:

Camera Output (v4l2-ctl):

YUV formats (YUYV, UYVY, etc.)
Bayer formats (BA81, GBRG, etc.)
Greyscale (GREY)
RGB formats (RGBR, XR24, etc.)

v4l2h264enc Input (gst-inspect-1.0):

Accepts: video/x-raw
Supported formats: Y42B, NV16, NV61, NV21, NV12, I420
Supports resolutions 1x1 to 32768x32768 and various frame rates.

Key Issue: The camera's primary outputs (like YUYV/UYVY) aren't directly compatible with the encoder's required formats (like NV12/I420). This means we can't feed the camera output directly into v4l2h264enc.

Question: Given this mismatch, is format conversion necessary before encoding? Is my understanding correct? If not, what's the proper approach?

2 months ago

0 LingHui Zhang 2 months ago

Prodigy 20 points

root@j784s4-evm:~# gst-inspect-1.0 v4l2h264enc
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                V4L2 H.264 Encoder
  Klass                    Codec/Encoder/Video/Hardware
  Description              Encode H.264 video streams via V4L2 API
  Author                   ayaka <ayaka@soulik.info>
  Documentation            https://gstreamer.freedesktop.org/documentation/video4linux2/v4l2h264enc.html

Plugin Details:
  Name                     video4linux2
  Description              elements for Video 4 Linux
  Filename                 /usr/lib/gstreamer-1.0/libgstvideo4linux2.so
  Version                  1.22.12
  License                  LGPL
  Source module            gst-plugins-good
  Documentation            https://gstreamer.freedesktop.org/documentation/video4linux2/
  Source release date      2024-04-29
  Binary package           GStreamer Good Plug-ins source release
  Origin URL               Unknown package origin

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstVideoEncoder
                         +----GstV4l2VideoEnc
                               +----GstV4l2H264Enc
                                     +----v4l2h264enc

Implemented Interfaces:
  GstPreset

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)Y42B, (string)NV16, (string)NV61, (string)NV21, (string)NV12, (string)I420 }
                  width: [ 1, 32768 ]
                 height: [ 1, 32768 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h264
          stream-format: byte-stream
              alignment: au
                  level: { (string)1, (string)1b, (string)1.1, (string)1.2, (string)1.3, (string)2, (string)2.1, (string)
(string)3.1, (string)3.2, (string)4, (string)4.1, (string)4.2, (string)5, (string)5.1 }
                profile: { (string)baseline, (string)constrained-baseline, (string)main, (string)extended, (string)high, 
 (string)high-4:2:2, (string)high-4:4:4 }

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:

  capture-io-mode     : Capture I/O mode (matches src pad)
                        flags: readable, writable
                        Enum "GstV4l2IOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (1): rw               - GST_V4L2_IO_RW
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
                           (4): dmabuf           - GST_V4L2_IO_DMABUF
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
  
  device              : Device location
                        flags: readable
                        String. Default: "/dev/video1"
  
  device-fd           : File descriptor of the device
                        flags: readable
                        Integer. Range: -1 - 2147483647 Default: -1 
  
  device-name         : Name of the device
                        flags: readable
                        String. Default: null
  
  extra-controls      : Extra v4l2 controls (CIDs) for the device
                        flags: readable, writable
                        Boxed pointer of type "GstStructure"
  
  min-force-key-unit-interval: Minimum interval between force-keyunit requests in nanoseconds
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "v4l2h264enc0"
  
  output-io-mode      : Output side I/O mode (matches sink pad)
                        flags: readable, writable
                        Enum "GstV4l2IOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (1): rw               - GST_V4L2_IO_RW
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
                           (4): dmabuf           - GST_V4L2_IO_DMABUF
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
  
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  
  qos                 : Handle Quality-of-Service events from downstream
                        flags: readable, writable
                        Boolean. Default: false
  
 ESC[D
                        Integer. Range: -1 - 2147483647 Default: -1 
  
  device-name         : Name of the device
                        flags: readable
                        String. Default: null
  
  extra-controls      : Extra v4l2 controls (CIDs) for the device
                        flags: readable, writable
                        Boxed pointer of type "GstStructure"
  
  min-force-key-unit-interval: Minimum interval between force-keyunit requests in nanoseconds
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "v4l2h264enc0"
  
  output-io-mode      : Output side I/O mode (matches sink pad)
                        flags: readable, writable
                        Enum "GstV4l2IOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (1): rw               - GST_V4L2_IO_RW
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
                           (4): dmabuf           - GST_V4L2_IO_DMABUF
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
  
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  
  qos                 : Handle Quality-of-Service events from downstream
                        flags: readable, writable
                        Boolean. Default: false

0 LingHui Zhang 2 months ago in reply to LingHui Zhang

Prodigy 20 points

root@j784s4-evm:~# v4l2-ctl -d /dev/video4 --list-formats-ex
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'YUYV' (YUYV 4:2:2)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [1]: 'UYVY' (UYVY 4:2:2)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [2]: 'YVYU' (YVYU 4:2:2)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [3]: 'VYUY' (VYUY 4:2:2)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [4]: 'BA81' (8-bit Bayer BGBG/GRGR)
                Size: Stepwise 16x1 - 16384x16384 with step 16/1
        [5]: 'GBRG' (8-bit Bayer GBGB/RGRG)
                Size: Stepwise 16x1 - 16384x16384 with step 16/1
        [6]: 'GRBG' (8-bit Bayer GRGR/BGBG)
                Size: Stepwise 16x1 - 16384x16384 with step 16/1
        [7]: 'RGGB' (8-bit Bayer RGRG/GBGB)
                Size: Stepwise 16x1 - 16384x16384 with step 16/1
        [8]: 'GREY' (8-bit Greyscale)
                Size: Stepwise 16x1 - 16384x16384 with step 16/1
        [9]: 'BG10' (10-bit Bayer BGBG/GRGR)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [10]: 'GB10' (10-bit Bayer GBGB/RGRG)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [11]: 'BA10' (10-bit Bayer GRGR/BGBG)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [12]: 'RG10' (10-bit Bayer RGRG/GBGB)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [13]: 'RGI0' (10-bit Bayer RGBG/GIrGIr)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [14]: 'GRI0' (10-bit Bayer GRGB/IrGIrG)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [15]: 'BGI0' (10-bit Bayer BGRG/GIrGIr)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [16]: 'GBI0' (10-bit Bayer GBGR/IrGIrG)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [17]: 'GIR0' (10-bit Bayer GIrGIr/RGBG)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [18]: 'IGR0' (10-bit Bayer IrGIrG/GRGB)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [19]: 'GIB0' (10-bit Bayer GIrGIr/BGRG)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [20]: 'IGB0' (10-bit Bayer IrGIrG/GBGR)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [21]: 'BG12' (12-bit Bayer BGBG/GRGR)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [22]: 'GB12' (12-bit Bayer GBGB/RGRG)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [23]: 'BA12' (12-bit Bayer GRGR/BGBG)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [24]: 'RG12' (12-bit Bayer RGRG/GBGB)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [25]: 'RGBR' (16-bit RGB 5-6-5 BE)
                Size: Stepwise 8x1 - 8192x16384 with step 8/1
        [26]: 'XR24' (32-bit BGRX 8-8-8-8)
                Size: Stepwise 4x1 - 4096x16384 with step 4/1
        [27]: 'XB24' (32-bit RGBX 8-8-8-8)
                Size: Stepwise 4x1 - 4096x16384 with step 4/1

0 Josiitaa RL 2 months ago in reply to LingHui Zhang

TI__Mastermind 40966 points

Hi,

Thank you for the detailed question. Please expect a delay in responses due to a holiday today in the US.

Regards,

Josiitaa

0 Sarabesh Srinivasan 2 months ago in reply to LingHui Zhang

TI__Genius 16610 points

Hello,

What SDK version are you using? I will do some background research on what plugins you may need to incorporate to assist with the conversion. However, the encoder does support using YUV420 input and YUV422 (gets downsampled internally to YUV420).

Thanks,
Sarabesh S.

0 LingHui Zhang 2 months ago in reply to Sarabesh Srinivasan

Prodigy 20 points

Hello,

The SDK version we are using is 10.00.00.

Thanks,

LingHui.

0 Sarabesh Srinivasan 2 months ago in reply to LingHui Zhang

TI__Genius 16610 points

Hi Linghui,

Thanks for the information. I shall have an update for you tomorrow.

Regards,
Sarabesh S.

0 Sarabesh Srinivasan 1 month ago in reply to Sarabesh Srinivasan

TI__Genius 16610 points

Hi Linghui,

Apologies for the delay, bandwidth was limited. After taking a look at the spec the VPU only supports native H264 and H265 encoding/decoding of YUV420. However, YUV422 input is an acceptable encoding input and will get downsampled to 420 for internal processing by the PRP hardware module.

Therefore your use-case is supported and you can feed the camera output to the encoder without an external gstreamer plugin.

example pipeline:

gst-launch-1.0 filesrc location=/example.yuv! rawvideoparse width=1920 height=1080 format=YVYU framerate=30/1 colorimetry=bt709 ! v4l2h264enc ! filesink location=/example_out.h264

Thank you,
Sarabesh S

0 LingHui Zhang 1 month ago in reply to Sarabesh Srinivasan

Prodigy 20 points

Hi Sarabesh,

When we run the capture pipeline:

gst-launch-1.0 v4l2src device=/dev/video4 num-buffers=100 ! \
video/x-raw,format=UYVY,width=1920,height=1080 ! \
filesink location=test.uyvy

…it completes without error and produces `test.uyvy`.

However, when we then feed that file into the encoder pipeline:

gst-launch-1.0 filesrc location=/test.uyvy ! \
rawvideoparse width=1920 height=1080 format=UYVY framerate=30/1 colorimetry=bt709 ! \
v4l2h264enc ! filesink location=/example_out.h264

the launcher aborts with:

WARNING: erroneous pipeline: could not set property "format" in element "rawvideoparse" to "UYVY"

Regards,
Linghui

0 Sarabesh Srinivasan 1 month ago in reply to LingHui Zhang

TI__Genius 16610 points

Hi Linghui,

I will provide an update tomorrow.

Thanks,
Sarabesh S.

0 Sarabesh Srinivasan 1 month ago in reply to Sarabesh Srinivasan

TI__Genius 16610 points

Hi Linghui, apologies for the delay B/w has been limited. Still working to verify the correct pipeline for this use-case. Thanks for your patience.

Regards,
Sarabesh S.

0 Kevin Peng 1 month ago in reply to Sarabesh Srinivasan

TI__Expert 3772 points

Hi Sarabesh,

May I know any stage updates you could provide to customer in advance please?

Customer is still waiting for it.

Thanks,

Kevin

0 Sarabesh Srinivasan 1 month ago in reply to Kevin Peng

TI__Genius 16610 points

Hi Kevin,

Apologies, been occupied on some other topics that took up bandwidth. I verified the the YUV422 encode with the following pipeline:

gst-launch-1.0 filesrc location=/squirrel-720x576-422P.yuv ! rawvideoparse width=720 height=576 format=y42b framerate=30/1 colorimetry=bt601 ! v4l2h264enc ! filesink location=/squirrel.h264

For your particular usecase try the following:

gst-launch-1.0 filesrc location=/example.yuv! rawvideoparse width=1920 height=1080 format=y42b framerate=30/1 colorimetry=bt601 ! v4l2h264enc ! filesink location=/example_out.h264

If this doesn't work for you please share the stream input so I can replicate on my environment.

Thank you,
Sarabesh S.

0 LingHui Zhang 1 month ago in reply to Sarabesh Srinivasan

Prodigy 20 points

Hi Sarabesh,

Thanks for the pipeline sample!
I only changed the file name:

gst-launch-1.0 filesrc location=./raw_video_600.yuv ! \
rawvideoparse width=1920 height=1080 format=y42b framerate=30/1 colorimetry=bt601 ! \
v4l2h264enc ! \
filesink location=./example_out.h264

The resulting colours are off.
Our source is actually UYVY, so I tried:

gst-launch-1.0 filesrc location=./raw_video_600.yuv ! \
rawvideoparse width=1920 height=1080 format=uyvy framerate=30/1 ! \
videoconvert ! \
v4l2h264enc ! \
filesink location=./example_out.h264

…but it fails during preroll:

ERROR: from element /GstPipeline:pipeline0/GstRawVideoParse:rawvideoparse0: Internal data stream error.

Could you share any pointers? The original video stream is provided below.
Thanks again!

Best regards,
LingHui

4555.raw_video_10.rar

0 Sarabesh Srinivasan 30 days ago in reply to LingHui Zhang

TI__Genius 16610 points

Hi LingHui,

I tested the provided video stream and got a successful output with this command:

gst-launch-1.0 filesrc location=/raw_video_10.yuv ! rawvideoparse width=1920 height=1080 format=uyvy framerate=30/1 colorimetry=bt601 ! v4l2h264enc ! filesink location=/raw_video_10_uyvy.h264

Please try and let me know your results. This is with the latest default 11.0 SDK release.

Thank you,
Sarabesh S.

Processors

Processors forum

TDA4VM: Camera Output Format Mismatch with v4l2h264enc Input - Conversion Needed?