• State Resolved
  • Locked Locked
  • Replies 3 replies
  • Subscribers 98 subscribers
  • Views 185 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM62A7: Is there any plugins can overlay OSD directly on NV12 streams?

Jason Zeng
Intellectual 315 points
Part Number: AM62A7

Tool/software:

SDK ver : 10_00_00_08

v4l2src -> tiovxisp -> tiovxmultiscaler -> tiovxmosaic -> kmssink

I have a GStreamer pipeline that ultimately renders to KMSSINK. The entire video stream is in NV12 format, and I need to add OSD elements—such as rectangles, icons, and text—to the video stream before it's rendered to the screen.

 

I've found a cairooverlay plugin, but it only supports ​BGRx, BGRA, and RGB16​ for both input and output. I need to add two colorconvert nodes (before and after) to make it work.

 

But the hardware acclerate color convert element like tiovxdlcolorconvert not support BGRx, BGRA or RGB16.

I also have a question: In edgeai-tiovx-apps, the AM62A can normally use both tiovx color convert and tiovx dl color convert nodes. However, in GStreamer, only gsttiovxdlcolorconvert is available, and I noticed that tiovxcolorconvert is not included for AM62A in ext/tiovx/meson.build.

So I want to know if there are alternative methods to meet my requirements. Perhaps modifying colorconvert could be a solution?

  • 0
    TI__Genius 15616 points

    Hello Jason,

    Right, using Cairooverlay will require additional color conversions. The compliant pixel formats for cairooverlay are not supported by our color-conversion TIOVX modules. Modifying those to support an addition format is one option.

    • Note that on AM62A, no special hardware is being used to accelerate tiovxdlcolorconvert -- it is using Arm Neon floating-point support. 
      • I believe the tiovxcolorconvert (not dl version) is using hardware acceleration, and is not available on 62A
        • This version also does not support the pixel formats of cairooverlay
    • However, TIOVX plugins are more memory-optimized since they use shared DMA buffer pools. Otherwise, there may be memory-copies between some plugins, which would slow the pipeline. 

    Another option (but less efficient) would be to use a generic conversion plugin like 'videoconvert'. This can handle the formats you request.

    If the overlay is static, there may be simpler options like gdkpixbufoverlay. Yet, if those OSD elements are more dynamic, that plugin is probably less helpful. 

    BR,
    Reese

  • 0 in reply to Reese Grimsley
    Intellectual 315 points

    Hello Reese.

    Reese Grimsley said:
    Modifying those to support an addition format is one option.

    Is there any existing color convert patch to add support for additional color format?

    From my understanding, in TI's GStreamer plugins, those prefixed with ​**tiovx​ are hardware-accelerated, while ones starting with ​ti**​ (e.g., tidl) are software-based (e.g., NEON-optimized).

    Because when I validated using the edgeai-tiovx-apps repository on AM62A7 , I tested the color convert module (not dl version) and saw calls to OpenVX APIs like vxColorConvertNode(). That’s why I assumed it was hardware-accelerated. Are you suggesting this interface internally uses NEON acceleration instead?

    I did try using videoconvert, but the actual frame rate dropped from 60fps to 15fps, which is unacceptable for our use case. Additionally, we need more dynamic OSD elements to overlay algorithm detection results (e.g., bounding boxes) onto the video stream.

    Best Regards,

    Jason

  • +1 in reply to Jason Zeng
    TI__Genius 15616 points

    Hi Jason,

    I am not aware of any such patch for additional color formats, but I will check internally.

    TIOVX plugins are not necessarily hardware accelerated, but they will at least use the TIOVX framework. This allows efficient passing of data using DMA-based buffer-pools. The non-TIOVX plugins may require a memory copy or not able to use such buffer pools, and this often limits performance.

    AM62A is not using any explicit hardware acceleration for color conversion to my knowledge, but they may have recompiled the C66x implementation for C7x. I am certain the TIOVX-DL-Color-convert is using Neon. 

    • I performed a few runtime tests for that module, it does look like the TIOVXColorConvert is using C7x, but we do not (by default) create the gstreamer plugin for this.
      • Thus, this kernel is not using Neon. 

    videoconvert is generally slow, yes. I suppose too slow for your application then. 

    I think what you're trying to achieve will require some software development, either to have a more efficient color-conversion or to draw OSD elements on NV12 encoded images. Without hardware-acceleration, I am doubtful color-conversion is the right approach.

    We have a few plugins that are drawing elements to the screen live for NV12 images -- please see tiperfoverlay (boxes with performance indicators) and tiovxdlpostproc (draw TIDL outputs to the screen).

    • My suggestion is to extend one of these. I predict tiperfoverlay being easier, since it does not have additional TIOVX interfaces to consider)

    BR,

    Reese