TDA4VH-Q1: v4l2h264enc CPU load optimization

We have made the following progress at present.

1.From the source code of gstreamer, the reason for the error "Got 2 dmabuf but needed 1" is that the n_mem of the src buffer is 2, but the n_mem of the dest buffer is 1.

Furthermore, we find that the reason why dest n_mem is 1 is most likely because in gst_v4l2_object_get_caps_info, v4l2object->prefered_non_contiguous is always false. Therefore, it leads to the n_v4l_planes in gst_v4l2_object_set_format_full always being 1.

2.Meanwhile, through the v4l2-ctl command, we found that video1 corresponding to v4l2h264enc did not show support for non_contiguous, but video0 of v4l2h264dec showed "[8]: 'NM21' (Y/VU 4:2:0 (N-C)) (Does it indicate that v4l2h264dec supports non_contiguous?)

root@j784s4-evm:~# v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'YU12' (Planar YUV 4:2:0)
                Size: Continuous 8x8 - 8192x4320
        [1]: 'NV12' (Y/UV 4:2:0)
                Size: Continuous 8x8 - 8192x4320
        [2]: 'NV21' (Y/VU 4:2:0)
                Size: Continuous 8x8 - 8192x4320
        [3]: '422P' (Planar YUV 4:2:2)
                Size: Continuous 8x8 - 8192x4320
        [4]: 'NV16' (Y/UV 4:2:2)
                Size: Continuous 8x8 - 8192x4320
        [5]: 'NV61' (Y/VU 4:2:2)
                Size: Continuous 8x8 - 8192x4320
        [6]: 'YM12' (Planar YUV 4:2:0 (N-C))
                Size: Continuous 8x8 - 8192x4320
        [7]: 'NM12' (Y/UV 4:2:0 (N-C))
                Size: Continuous 8x8 - 8192x4320
        [8]: 'NM21' (Y/VU 4:2:0 (N-C))
                Size: Continuous 8x8 - 8192x4320
        [9]: 'YM16' (Planar YUV 4:2:2 (N-C))
                Size: Continuous 8x8 - 8192x4320
        [10]: 'NM16' (Y/UV 4:2:2 (N-C))
                Size: Continuous 8x8 - 8192x4320
        [11]: 'NM61' (Y/VU 4:2:2 (N-C))
                Size: Continuous 8x8 - 8192x4320
root@j784s4-evm:~# v4l2-ctl -d /dev/video1 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'HEVC' (HEVC, compressed)
                Size: Stepwise 256x128 - 8192x8192 with step 8/8
        [1]: 'H264' (H.264, compressed)
                Size: Stepwise 256x128 - 8192x8192 with step 8/8
root@j784s4-evm:~# v4l2-ctl -d /dev/video2 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'YU12' (Planar YUV 4:2:0)
                Size: Continuous 8x8 - 8192x4320
        [1]: 'NV12' (Y/UV 4:2:0)
                Size: Continuous 8x8 - 8192x4320
        [2]: 'NV21' (Y/VU 4:2:0)
                Size: Continuous 8x8 - 8192x4320
        [3]: '422P' (Planar YUV 4:2:2)
                Size: Continuous 8x8 - 8192x4320
        [4]: 'NV16' (Y/UV 4:2:2)
                Size: Continuous 8x8 - 8192x4320
        [5]: 'NV61' (Y/VU 4:2:2)
                Size: Continuous 8x8 - 8192x4320
        [6]: 'YM12' (Planar YUV 4:2:0 (N-C))
                Size: Continuous 8x8 - 8192x4320
        [7]: 'NM12' (Y/UV 4:2:0 (N-C))
                Size: Continuous 8x8 - 8192x4320
        [8]: 'NM21' (Y/VU 4:2:0 (N-C))
                Size: Continuous 8x8 - 8192x4320
        [9]: 'YM16' (Planar YUV 4:2:2 (N-C))
                Size: Continuous 8x8 - 8192x4320
        [10]: 'NM16' (Y/UV 4:2:2 (N-C))
                Size: Continuous 8x8 - 8192x4320
        [11]: 'NM61' (Y/VU 4:2:2 (N-C))
                Size: Continuous 8x8 - 8192x4320
root@j784s4-evm:~# v4l2-ctl -d /dev/video3 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'HEVC' (HEVC, compressed)
                Size: Stepwise 256x128 - 8192x8192 with step 8/8
        [1]: 'H264' (H.264, compressed)
                Size: Stepwise 256x128 - 8192x8192 with step 8/8
root@j784s4-evm:~#
root@j784s4-evm:~# gst-inspect-1.0 | grep v4l2
video4linux2:  v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2:  v4l2h264dec: V4L2 H264 Decoder
video4linux2:  v4l2h264enc: V4L2 H.264 Encoder
video4linux2:  v4l2h265dec: V4L2 H265 Decoder
video4linux2:  v4l2h265enc: V4L2 H.265 Encoder
video4linux2:  v4l2radio: Radio (video4linux2) Tuner
video4linux2:  v4l2sink: Video (video4linux2) Sink
video4linux2:  v4l2src: Video (video4linux2) Source
video4linux2:  v4l2video2h264dec: V4L2 H264 Decoder
video4linux2:  v4l2video2h265dec: V4L2 H265 Decoder
video4linux2:  v4l2video3h264enc: V4L2 H.264 Encoder
video4linux2:  v4l2video3h265enc: V4L2 H.265 Encoder
root@j784s4-evm:~# v4l2h264enc

So for the current usage of gstreamer mentioned above, the current driver does not support the option output-io-mode=dmabuf-import, right?

Hi Sarabesh,

can you help check this?

BR,

Biao

Hi

io-mode 5 means dma-import. It means some other upstream plugin will provide dma buf for the encoder to import from. In your cases, the upstream element ‘appsrc ’ does not necessarily provide the dma buf required. 

Regards,

Adam

hi

In our case, can appsrc provide DMA buffers through certain configurations?If possible, what should we do?

Hi Hongyao,

Could you try change the image size to 1920x1080? This maybe similar with an issue I saw before.

Regards,

Adam

Hello, 

Please let me know if there is any update on this issue. Otherwise I can mark this thread closed.

Thanks,
Sarabesh S.

hi

We saw the following use case in the Linux SDK documentation.

target # gst-launch-1.0 v4l2src device=/dev/video2 io-mode=dmabuf ! video/x-bayer,width=1920,height=1080, framerate=30/1, format=bggr ! tiovxisp sensor-name=SENSOR_SONY_IMX219_RPI dcc-isp-file=/opt/imaging/imx219/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a.bin sink_0::device=/dev/v4l-subdev2 ! video/x-raw,format=NV12 ! v4l2h264enc output-io-mode=dmabuf-import extra-controls="controls,h264_i_frame_period=60" ! rtph264pay ! udpsink port=5000 host=<ip_address>

In our use case, can the appsrc be configured with the io-mode=dmabuf option?

Hello, 

Yes, you should be able to reference buffers for the encoder with the buffers in the camera src using dma-buf. Let me look into any resources we may have on this topic. I will provide an updat early next-week.

Thanks,
Sarabesh S.

Hi Sarabesh S,

When can you  provide the demo ?

BR

Yunjie Li

Hi Yunjie, apologies, bandwidth was limited last week so I haven't had time to review our dma-buf examples. Could you let me know if you're using Linux standalone SDK or the Linux+RTOS (Vision Apps) SDK?

Thanks,
Sarabesh S.

Hi Sarabesh S,

I use TDA4VH  SDK 11.0 Linux+Freertos, the specific version is: 11.00.00.06.

Sarabesh Srinivasan said:

Yes, you should be able to reference buffers for the encoder with the buffers in the camera src using dma-buf.

How to reference buffer of appsrc as the input of encoder is critical for my architecture,  the current demo of TDA4VH which costs more CPU usage,  about a 30 fps 2M video taking 30% of a single core's CPU.  My requirement is to record the video of 4*3M+8M 30fps, so I wonder whether referencing buffers makes sense, could you give me a conclusion in advance?  Then, the demo is provided later.

Thanks

Yunjie Li 

Hi Yunjie, 

Yes that does make sense however in the past we have seen issues with using appsrc along with dma-buf. What is the reason for using appsrc instead of v4l2src?

Thanks,

Sarabesh S

Hi Sarabesh S,

Thanks for you confirmation.

The backbone of my pipeline as below:

For TDA4VH, my application is developed base on tiovx, image is  duplicated by scaler, one for DL model, the other for H264.

Thanks!

Yunjie Li

Hi Yunjie, 

To clarify is your Appsrc input coming directly from a CSI camera capture or is it just a file recording captured by the CSI camera?

Regards,
Sarabesh S.

Hi Sarabesh S,

The more detailed architecture as below

The images are captured from CSI, then forward their descriptions to the other processes. So it is more similar to reading data from a file.

Thanks

Yunjie Li

Hi Yunjie, 

We have a similar Vision-Apps demo application that similarly fits your use-case. Have you looked at referencing the app-multi-cam-codec demo (here)?

Thanks,
sarabesh S.

Hi Sarabesh S.

I refer to that demo to organize my pipeline,  but  gst takes more CPU, about details, you can look up the link below.

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1505491/tda4vh-q1-how-to-reduce-the-cpu-usage-of-the-app_multi_cam_codec-sample/5839091?tisearch=e2e-sitesearch&keymatch=%252520user%25253A533595#

Thanks

Yunjie Li

Hi Yunjie, 

The development team has provided feedback with regards to appsrc and appsink's utilization. Essentially, the GStreamer framework w/ appsrc and appsink is quite restrictive and doesn't allow for easy buffer handling between the app and gstreamer plugins. 

It looks like your appsrc is just multiple camera feeds and your appsink is a file write. Is this correct? Could you also extend on what exactly you are using the VPAC for? Are you trying to mosaic the camera inputs like in app_multi_cam_codec?

Thanks,
Sarabesh S.

Hi Sarabesh S.,

Sarabesh Srinivasan said:

It looks like your appsrc is just multiple camera feeds and your appsink is a file write. Is this correct?

Yes, you are right.

Sarabesh Srinivasan said:

Could you also extend on what exactly you are using the VPAC for?

The main function:

1. The format convert of image data

2. Mesh warp  as below

3.Dupilcate image and resize them

Sarabesh Srinivasan said:

Are you trying to mosaic the camera inputs like in app_multi_cam_codec?

Yes,  I will mosaic the resized image into an image for data recording.

Thanks

Yunjie Li

Thanks for the information. I'll continue looking into this but please expect some delays this week due to some other ongoing tasks taking place.

Regards,
Sarabesh S.

Hi Sarabesh S.,

Any update ?

Thanks

Yunjie Li

Hi Yunjie, 

Still looking into what other demos we have to assist with this usecase.