TDA4VH-Q1: Video encoder delay

Can you send the command or application  you are using? I am unaware of this issue and will have to investigate some things.

Thanks,
Sarabesh S.

Sure.

Here is the code for codec initialization:

const AVCodec* codec = avcodec_find_encoder_by_name("h264_v4l2m2m");
if (!codec)
{
    throw std::runtime_error("Codec h264_v4l2m2m not found");
}

m_codecContext = avcodec_alloc_context3(codec);
if (!m_codecContext)
{
    throw std::runtime_error("Could not allocate video codec context");
}

m_pkt = av_packet_alloc();
if (!m_pkt)
{
    throw std::runtime_error("Could not allocate AVPacket");
}

m_codecContext->bit_rate = 4000000;
m_codecContext->width = width;
m_codecContext->height = height;

m_codecContext->time_base = {
    1, m_framerate
};
m_codecContext->framerate = {
    m_framerate, 1
};

m_codecContext->gop_size = 10;
m_codecContext->max_b_frames = 0;
m_codecContext->pix_fmt = AV_PIX_FMT_NV12;


int ret = avcodec_open2(m_codecContext, codec, nullptr);
if (ret < 0)
{
    throw std::runtime_error("Could not open codec " + CODEC_NAME);
}

m_frame = av_frame_alloc();
if (!m_frame)
{
    throw std::runtime_error("Could not allocate video frame");
}
m_frame->format = m_codecContext->pix_fmt;
m_frame->width = m_codecContext->width;
m_frame->height = m_codecContext->height;

ret = av_frame_get_buffer(m_frame, 0);
if (ret < 0)
{
    throw std::runtime_error("Could not allocate the video frame data");
}

And here is the frameEncode function that is called for each input frame:

ret = avcodec_send_frame(m_codecContext, m_frame);
if (ret < 0)
{
    LOG_ERROR("Error sending a frame for encoding");
    success = false;
}

while (ret >= 0)
{
    ret = avcodec_receive_packet(m_codecContext, m_pkt);
    if (ret == AVERROR(EAGAIN) || ret == AVERROR(EOF))
    {
        LOG_INFO2("End of stream or packet not ready");
    }
    else if (ret < 0)
    {
        LOG_ERROR("Error during encoding");
        success = false;
    }
    else
    {
        int offset = (int)result.size();
        result.resize(result.size() + m_pkt->size);
        memcpy(result.data() + offset, m_pkt->data, m_pkt->size);
        av_packet_unref(m_pkt);
    }
}

Is there a reason you are not using GStreamer for your application? Are you able to try this on the latest 10.1 SDK and let me know your results?

Thanks,
Sarabesh S.

Hi Sarabesh, 

I'm helping Matthieu on this issue. 

We used ffmpeg(libavcodec) as we were already using it in our application. 
We could be able to try 10.1, but our product is running on 9.2 SDK so we are required to have a workaround for that version. 
Is there some patches already ported to SDK 9.2 for the wave5 encoder?

Thanks, 

If you could try it on SDK 10.1 and that way we can see what patches may or may not be needed for this to work. I will have to discuss internally to determine the support level here.

Thanks,
Sarabesh S,

Hi Sarabesh, 

I'm trying on SDK 10.1 (Linux j784s4-evm 6.6.44-ti-01478-g541c20281af7-dirty #1 SMP PREEMPT Thu Nov 14 19:20:24 UTC 2024 aarch64 GNU/Linux) and with the same code I run on SDK 9.2 the codec seem to get stuck. 

I did this test to encode a dummy frame using h264_v4l2m2m encoder:

ffmpeg -f lavfi -i testsrc=size=1920x1080:rate=30 -pix_fmt yuv420p -c:v h264_v4l2m2m -b:v 4M -t 2 output.mp4

root@j784s4-ecu:~# ffmpeg -f lavfi -i testsrc=size=1920x1080:rate=30 -pix_fmt yuv420p -c:v h264_v4l2m2m -b:v 4M -t 2 output.mp4
ffmpeg version 5.0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11.4.0 (GCC)
  configuration: --cross-prefix=aarch64-oe-linux- --ld='aarch64-oe-linux-gcc -mbranch-protection=standard --sysroot=/recipe-sysroot' --cc='aarch64-oe-linux-gcc -mbranch-protection=standard --sysroot=/recipe-sysroot' --cxx='aarch64-oe-linux-g++ -mbranch-protection=standard --sysroot=/recipe-sysroot' --arch=aarch64 --target-os=linux --enable-cross-compile --extra-cflags=' -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map==/usr/src/debug/ffmpeg/5.0.1-r0.arago0 -fdebug-prefix-map==/usr/src/debug/ffmpeg/5.0.1-r0.arago0 -fdebug-prefix-map=/recipe-sysroot= -fdebug-prefix-map=/recipe-sysroot-native= -mbranch-protection=standard --sysroot=/recipe-sysroot' --extra-ldflags='-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fmacro-prefix-map==/usr/src/debug/ffmpeg/5.0.1-r0.arago0 -fdebug-prefix-map==/usr/src/debug/ffmpeg/5.0.1-r0.arago0 -fdebug-prefix-map=/recipe-sysroot= -fdebug-prefix-map=/recipe-sysroot-native= ' --sysroot=/recipe-sysroot --libdir=/usr/lib --shlibdir=/usr/lib --datadir=/usr/share/ffmpeg --cpu=generic --pkg-config=pkg-config --enable-alsa --disable-altivec --enable-avcodec --enable-avdevice --enable-avfilter --enable-avformat --enable-bzlib --disable-libfdk-aac --disable-gpl --disable-libgsm --disable-indev=jack --disable-libopus --disable-libvorbis --enable-lzma --disable-libmfx --disable-mipsdsp --disable-mipsdspr2 --disable-libmp3lame --disable-openssl --enable-pic --enable-postproc --enable-pthreads --disable-sdl2 --enable-shared --disable-libspeex --disable-libsrt --disable-stripping --enable-swresample --enable-swscale --enable-libtheora --disable-vaapi --disable-vdpau --disable-libvpx --disable-libx264 --disable-libx265 --disable-libxcb --disable-outdev=xv --enable-zlib
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
Input #0, lavfi, from 'testsrc=size=1920x1080:rate=30':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1920x1080 [SAR 1:1 DAR 16:9], 30 tbr, 30 tbn
File 'output.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0xa885330] Using device /dev/video3
[h264_v4l2m2m @ 0xa885330] driver 'wave5-enc' on card 'wave5-enc' in mplane mode
[h264_v4l2m2m @ 0xa885330] requesting formats: output=YU12 capture=H264
Output #0, mp4, to 'output.mp4':
  Metadata:
    encoder         : Lavf59.16.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 4000 kb/s, 30 fps, 15360 tbn
    Metadata:
      encoder         : Lavc59.18.100 h264_v4l2m2m
frame=   60 fps= 21 q=-0.0 Lsize=     898kB time=00:00:01.96 bitrate=3738.7kbits/s speed= 0.7x    
video:897kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.116442%
root@j784s4-ecu:~# 

On SDK 10.1 the dummy output is not created and ffmpeg fail with a segmentation fault

root@j784s4-evm:~# ffmpeg -f lavfi -i testsrc=size=1920x1080:rate=30 -pix_fmt yuv420p -c:v h264_v4l2m2m -b:v 4M -t 2 output.mp4
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.3.0 (GCC)
  configuration: --disable-stripping --enable-pic --enable-shared --enable-pthreads --cross-prefix=aarch64-oe-linux- --ld='aarch64-oe-linux-gcc -mbranch-protection=standard --sysroot=/recipe-sysroot' --cc='aarch64-oe-linux-gcc -mbranch-protection=standard --sysroot=/recipe-sysroot' --cxx='aarch64-oe-linux-g++ -mbranch-protection=standard --sysroot=/recipe-sysroot' --arch=aarch64 --target-os=linux --enable-cross-compile --extra-cflags=' -O2 -pipe -g -feliminate-unused-debug-types -fcanon-prefix-map -fmacro-prefix-map=/ffmpeg-6.1.1=/usr/src/debug/ffmpeg/6.1.1 -fdebug-prefix-map=/ffmpeg-6.1.1=/usr/src/debug/ffmpeg/6.1.1 -fmacro-prefix-map=/build=/usr/src/debug/ffmpeg/6.1.1 -fdebug-prefix-map=/build=/usr/src/debug/ffmpeg/6.1.1 -fdebug-prefix-map=/recipe-sysroot= -fmacro-prefix-map=/recipe-sysroot= -fdebug-prefix-map=/recipe-sysroot-native= -mbranch-protection=standard --sysroot=/recipe-sysroot' --extra-ldflags='-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fcanon-prefix-map -fmacro-prefix-map=/ffmpeg-6.1.1=/usr/src/debug/ffmpeg/6.1.1 -fdebug-prefix-map=/ffmpeg-6.1.1=/usr/src/debug/ffmpeg/6.1.1 -fmacro-prefix-map=/build=/usr/src/debug/ffmpeg/6.1.1 -fdebug-prefix-map=/build=/usr/src/debug/ffmpeg/6.1.1 -fdebug-prefix-map=/recipe-sysroot= -fmacro-prefix-map=/recipe-sysroot= -fdebug-prefix-map=/recipe-sysroot-native= ' --sysroot=/recipe-sysroot --libdir=/usr/lib --shlibdir=/usr/lib --datadir=/usr/share/ffmpeg --disable-mipsdsp --disable-mipsdspr2 --cpu=generic --pkg-config=pkg-config --enable-alsa --disable-altivec --enable-avcodec --enable-avdevice --enable-avfilter --enable-avformat --enable-bzlib --disable-libfdk-aac --disable-gpl --disable-libgsm --disable-indev=jack --disable-libopus --disable-libvorbis --enable-lzma --disable-libmfx --disable-libmp3lame --disable-openssl --enable-postproc --disable-sdl2 --disable-libspeex --disable-libsrt --enable-swresample --enable-swscale --enable-libtheora --disable-libv4l2 --disable-vaapi --disable-vdpau --disable-libvpx --disable-libx264 --disable-libx265 --disable-libxcb --disable-outdev=xv --enable-zlib
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
Input #0, lavfi, from 'testsrc=size=1920x1080:rate=30':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, rgb24, 1920x1080 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 30 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0x8d1e3f0] Using device /dev/video1
[h264_v4l2m2m @ 0x8d1e3f0] driver 'wave5-enc' on card 'wave5-enc' in mplane mode
[h264_v4l2m2m @ 0x8d1e3f0] requesting formats: output=YU12/yuv420p capture=H264/none
Output #0, mp4, to 'output.mp4':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 4000 kb/s, 30 fps, 15360 tbn
    Metadata:
      encoder         : Lavc60.31.102 h264_v4l2m2m
Segmentation fault (core dumped)
root@j784s4-evm:~#

Hi David, 

Apologies for the delay. I am going to attempt to replicate this today. I am currently in discussion with our development team, so I should have an update shortly.

Thanks,
Sarabesh S.

Hi David, 

Thanks for checking this on 10.1. On 9.2 we don't claim ffmpeg support and it shouldn't even be enabled in the default filesystem. For 10.0 SDK, have you referred to the SDK docs to confirm you are enabling all of the necessary parameters?

You can review our ffmpeg examples and parameters here: https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-j784s4/10_01_00_05/exports/docs/linux/Foundational_Components_Multimedia_wave5.html#ffmpeg-plugins-for-multimedia

Please try running one of our example pipelines on 10.x. Additionally, for some better debug logs you can add strace in front of the command. Please share your results when available.

Regards,
Sarabesh S.

Hi David, 

That's great news! Glad to hear, let me know if you need anything else.

Best,
Sarabesh S.