• State Resolved
  • Locked Locked
  • Replies 21 replies
  • Subscribers 100 subscribers
  • Views 884 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-AM62A-LP: Gstreamer streaming video overlay for date, current time (with milliseconds), camera name

Steve T
Intellectual 310 points
Part Number: SK-AM62A-LP

Tool/software:

Hello all,

I've been trying for a couple of days to come up with a solution to add date, time and camera name as an overlay on video streams. It seemed like adding "clockoverlay" and "textoverlay" gstreamer plugins to my pipe would do the trick but it is dropping my framerate from ~15 to  ~3 FPS. It does seem to perform better by reducing the font size but going lower than 10 font size/pixels starts to be an eyechart test for the average user. clockoverlay also won't display milliseconds due to using the strftime() function. I think these plugins are also creating a bottle neck as they apply the overlay using library pango. I've tried using an OTB font to see if it would improve performance but that hasn't noticeably helped. Tens of millisecond resolution would be a workable solution for my requirements.

Does anyone know of a better way to add/overlay this information into a video stream?

Thank you,

Steve T

  • 0
    TI__Mastermind 29275 points

    Hi Steve,

    Apologies for the delay.

    Where in the pipeline are you trying to add these elements for embedding the date and time? The position of it might also cause latencies as these are SW plugins and switching between SW and HW based plugins can add some latencies and thereby frame drops are seen. 

    Please share us the pipeline. 

    Best Regards,

    Suren

  • 0 in reply to Suren Porwar
    Intellectual 310 points

    Hello Suren,

    Sorry, I was out of the office yesterday. Also, I should mention this is using SDK-10.1 Here is the pipeline:

    gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! video/x-bayer, width=2600, height=1952, format=grbg10, framerate=45/1 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name=SENSOR_ONSEMI_AR0521 dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 sink_0::pool-size=8 src_0::pool-size=8 ! queue ! textoverlay halignment=right valignment=bottom shaded-background=false shaded-background=true draw-outline=false draw-shadow=false font-desc="Mono, 10" text="CAM2-UpperLeft" ! queue ! clockoverlay halignment=left valignment=bottom shaded-background=false shaded-background=true draw-outline=false draw-shadow=false time-format="%D %H:%M:%S" font-desc="Mono, 10" ! queue ! v4l2h264enc extra-controls=controls,h264_i_frame_period=45,prepend_sps_and_pps_to_idr=1,video_gop_size=45 ! mpegtsmux ! udpsink host=xxx.xxx.xxx.xxx port=550xx

    I can see using the TI tracers python script (with the DEBUG environment variables added to the command above) that there is a latency increase of about 10 (so ~77) in the tiovxisp0 element when the FPS drops down to ~35. It is much lower (~20) without the textoverlay and clockoverlay elements but it seems like this is where Tracers is reporting the latency even though it is actually caused by those other pipeline plug-ins/elements..

    Thank you,

    Steve

  • 0 in reply to Steve T
    TI__Mastermind 29275 points

    Hi Steve,

    Can you share the tracers screenshot to understand the FPS and latency aspect of the pipeline you are running.

    Also, can you configure the camera for a lower resolution and see if the behavior changes?

    Best Regards,

    Suren

  • 0 in reply to Suren Porwar
    Intellectual 310 points

    Here are the tracers for the gstreamer pipeline provided above which are close to 45 FPS:

    Without the overlays I can see a constant 45 FPS and the latency numbers are much lower for tiovxisp:

    Here are tracers from the gstreamer pipeline provided with the font size changed from 10 to 16 for both overlays:

    I can't easily change to other resolutions at the current time as it would involve some driver work and I need it to be usable at this resolution anyhow.

    Best Regards,

    Steve

  • 0 in reply to Steve T
    TI__Mastermind 29275 points

    Hi Steve,

    Apologies for the delay.

    I have routed your query to our expert, who is out of office. Please allow a day or two to respond back on why tiovxisp latencies differ when adding overlay components in the pipeline.

    Best Regards,

    Suren

  • 0 in reply to Suren Porwar
    TI__Guru 59285 points

    Hi Steve,

    I tried overlaying time and then streaming to a display using a 8M@30fps camera, and didn't see any fps drop:

    GST_DEBUG_FILE=/run/trace.log GST_DEBUG_NO_COLOR=1 GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=element)" \
gst-launch-1.0 -v v4l2src device=/dev/video-imx728-cam0 io-mode=dmabuf-import ! \
video/x-bayer, width=3856, height=2176, format=rggb12, framerate=30/1 ! \
tiovxisp sink_0::pool-size=2 sink_0::device=/dev/v4l-imx728-subdev0 sensor-name=SENSOR_SONY_IMX728_UB971_D3 \
dcc-isp-file=/opt/imaging/imx728/wdr/dcc_viss_wdr.bin sink_0::dcc-2a-file=/opt/imaging/imx728/wdr/dcc_2a_wdr.bin format-msb=11 wdr-enabled=true ! \
queue ! video/x-raw, format=NV12 ! tiovxmultiscaler target=0 ! video/x-raw, format=NV12, width=1920, height=1080 ! queue ! \
mosaic_0. tiovxmosaic name=mosaic_0 target=1 src::pool-size=2 sink_0::startx="<200>" sink_0::starty="<0>" sink_0::widths="<1520>" sink_0::heights="<850>" ! \
video/x-raw,format=NV12, width=1920, height=1080 ! queue ! \
textoverlay halignment=right valignment=bottom shaded-background=false \
shaded-background=true draw-outline=false draw-shadow=false font-desc="Mono, 10" \
text="CAM2-UpperLeft" ! queue ! \
clockoverlay halignment=left valignment=bottom shaded-background=false shaded-background=true \
draw-outline=false draw-shadow=false time-format="%D %H:%M:%S" font-desc="Mono, 10" ! queue ! \
kmssink driver-name=tidss sync=false skip-vsync=true

    Can you try streaming to a display instead of encoding and sending through the Ethernet? This can help us identify where the bottleneck is.

    Thanks,

    Jianzhong

  • 0 in reply to Jianzhong Xu
    Intellectual 310 points

    Hello Jianzhong,

    Thank you for looking into this. I ran the following command which excluded the mosaic because I'm not sure what that does exactly or quite how to use it (let me know if you would like me to try it and perhaps provide a gstreamer command, please):

    GST_DEBUG_FILE=/run/trace.log GST_DEBUG_NO_COLOR=1 GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=element)" gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! video/x-bayer, width=2600, height=1952, format=grbg10, framerate=45/1 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name=SENSOR_ONSEMI_AR0521 dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 sink_0::pool-size=8 src_0::pool-size=8 ! tiovxmultiscaler target=0 ! video/x-raw, format=NV12, width=1600, height=900 ! queue ! textoverlay halignment=right valignment=bottom shaded-background=false shaded-background=true draw-outline=false draw-shadow=false font-desc="Mono, 10" text="CAM2-UpperLeft" ! queue ! clockoverlay halignment=left valignment=bottom shaded-background=false shaded-background=true draw-outline=false draw-shadow=false time-format="%D %H:%M:%S" font-desc="Mono, 10" ! queue ! kmssink driver-name=tidss sync=false skip-vsync=true

    My monitor native resolution is 1600x900. Here are the tracers from the running the command above:

    Here are the tracer results using a fakesink instead of a kmssink and the command above but without tiovxmultiscaler:

    Thank you,

    Steve

  • 0 in reply to Steve T
    TI__Guru 59285 points

    Hi Steve,

    The multiscaler might be the bottleneck. Can you try two more things?

    1. add a "queue" element before tiovxmultiscaler

    2. keep tiovxmultiscaler and use fakesink

    Thank you.

    Jianzhong

  • 0 in reply to Jianzhong Xu
    Intellectual 310 points

    Hi Jianzhong,

    Per your requests:

    Jianzhong Xu said:
    1. add a "queue" element before tiovxmultiscaler

    Jianzhong Xu said:
    2. keep tiovxmultiscaler and use fakesink

    I did find that I'm not having higher latency in tiovxisp trying this with a different image sensor running at 1920x1200. Is a resolution of this size a strong contributing factor to the latency?

    Thank you,

    Steve

  • 0 in reply to Steve T
    TI__Guru 59285 points

    Hi Steve,

    Steve T said:
    Is a resolution of this size a strong contributing factor to the latency?

    Yes, it can be. 

    The test results of having a fakesink, with and without tiovxmultiscaler, show the multiscaler may be the bottleneck. Can you please try the patch I provided in the following thread?

    AM62A7-Q1: 3840x2160: Unable to run VPAC with VISS and LDC at 30 fps - Processors forum - Processors - TI E2E support forums

    Please find the patch at the following post of the above thread:

    Regards,

    Jianzhong

  • 0 in reply to Jianzhong Xu
    Intellectual 310 points

    Here are some tracer results after applying the patches and using tiovxmultiscaler:

    1. Using fakesink

    2. Using kmssink instead of fakesink

    Best Regards,

    Steve

  • 0 in reply to Steve T
    TI__Guru 59285 points

    Hi Steve,

    Using fakesink shows 45fps, which means the multiscaler patch helps.

    Can you share the pipeline of using kmssink?

    Thanks,

    Jianzhong

  • 0 in reply to Jianzhong Xu
    Intellectual 310 points

    Here is the pipeline I used for kmssink:

    GST_DEBUG_FILE=/run/trace.log GST_DEBUG_NO_COLOR=1 GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=element)" gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! video/x-bayer, width=2600, height=1952, format=grbg10, framerate=45/1 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name=SENSOR_ONSEMI_AR0521 dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 sink_0::pool-size=8 src_0::pool-size=8 ! queue ! tiovxmultiscaler target=0 ! video/x-raw, format=NV12, width=1600, height=900 ! queue ! textoverlay halignment=right valignment=bottom shaded-background=false shaded-background=true draw-outline=false draw-shadow=false font-desc="Mono, 10" text="CAM2-UpperLeft" ! queue ! clockoverlay halignment=left valignment=bottom shaded-background=false shaded-background=true draw-outline=false draw-shadow=false time-format="%D %H:%M:%S" font-desc="Mono, 10" ! queue ! kmssink driver-name=tidss sync=false skip-vsync=true

    Thanks,

    Steve

  • 0 in reply to Steve T
    Intellectual 310 points

    I didn't get an email when I posted the above reply and I've been experience some issues with login and posting today... so posting this to see if it gets through the system this time and sends notifications/email for an update to the thread.

  • 0 in reply to Steve T
    TI__Guru 59285 points

    Hi Steve,

    Does your display support 45fps?

    What if you remove textoverlay and clockoverlay? Would that bring the fps to 45?

    Thanks,

    Jianzhong

  • 0 in reply to Jianzhong Xu
    Intellectual 310 points

    Hi Jianzhong,

    The display native resolution runs at 60 Hz. It seems to support a video stream at 45 Hz as best I can tell. Removing textoverlay and clockoverlay produced these tracers:

    Here was the pipeline I used:

    GST_DEBUG_FILE=/run/trace.log GST_DEBUG_NO_COLOR=1 GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=element)" gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! video/x-bayer, width=2600, height=1952, format=grbg10, framerate=45/1 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name=SENSOR_ONSEMI_AR0521 dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 sink_0::pool-size=8 src_0::pool-size=8 ! queue ! tiovxmultiscaler target=0 ! video/x-raw, format=NV12, width=1600, height=900 ! queue ! kmssink driver-name=tidss sync=false skip-vsync=true

    Thanks,

    Steve

  • +1 in reply to Steve T
    TI__Guru 59285 points

    Hi Steve,

    Ok. So this is similar to your original question:

    Steve T said:
    It seemed like adding "clockoverlay" and "textoverlay" gstreamer plugins to my pipe would do the trick but it is dropping my framerate from ~15 to  ~3 FPS.

    Now, with "clockoverlay" and "textoverlay", the fps dropped from 45 to 30.

    I think these are generic Gstreamer questions. You'll probably need to implement some kind of accelerated Gstreamer plugin for those overlays. A quick search found me something like this: https://shop.ridgerun.com/products/gstreamer-fast-text-graphics-overlay-1?variant=16668448751687.

    Regards,

    Jianzhong

  • 0 in reply to Jianzhong Xu
    Intellectual 310 points

    Hey Jianzhong,

    Would you be able to provide any insight as to why the latency is being reported in the tiovxisop piece of the pipeline in the tracer report instead of the overlays?

    I had looked at that RidgeRun plugin but it doesn't seem to support time in milliseconds. I've been experimenting with a custom gstreamer application  that uses the textoverlay plugin to update the text parameter with the current date and time (during a callback from an identity plugin) but textoverlay is very similar in the way it does the overalay to the other two plugins and still observe some frames dropping and higher latency in tiovxisp in the tracers.

    Thanks,

    Steve

  • 0 in reply to Steve T
    TI__Guru 59285 points

    Hi Steve,

    Steve T said:
    Would you be able to provide any insight as to why the latency is being reported in the tiovxisop piece of the pipeline in the tracer report instead of the overlays?

    I have this question as well. I don't fully understand how Gstreamer calculates the latency for each element.

    Regards,

    Jianzhong

  • 0 in reply to Jianzhong Xu
    Intellectual 310 points

    Hello Jianzhong,

    I've still been working to hopefully provide a good solution here for the latency and dropped frames using the gstreamer overlay plugins. I've found that reducing the size of the overlay box (i.e. number of characters used to reduce the length, font size for height) can greatly decrease the latency. I can get a latency starting around 55 (although I see it increase up to 68 while is was running and then started going back down ~1/100 at a time) using 24 total characters in the text string using a font of "Mono, 8px". I think this is basically the limit for this configuration to not drop frames. If I use a text string of the same font and size but only 19 total characters then the latency is around 16-18. Unfortunately this only leaves a few characters for say a camera name when using a full date and time format (i.e. YYYY-MM-DD HH:MM:SS.mm being 22 characters) so it seems that it will be necessary to find a balance there of reducing the date/timestamp in order to use the textoverlay plugin the way I want.

    I'm using my gstreamer app to update the textoverlay plugin text field every frame to get the millisecond resolution functionality I require but I suspect that the  above character size and length impact/limitation is applicable when using clockoverlay with a gst-launch pipeline for certain configurations especially if additionally using its "text" element property.

    From some additional research, it seems to me that the latency is likely related to memory management with Pango, the image resolution size and the text to render all fitting in CPU cache efficiently in order to be processed as quickly as possible. Please correct or clarify this theory, if possible.

    I've found this solution from another company's product that uses custom overlay plugins to interact with an on-chip GPU so it may not be highly applicable to the AM62Ax (but perhaps for other TI processors it would make sense).

    https://github.com/Freescale/gstreamer-imx/pull/147


    Is a solution like in the link above something TI would be interested in providing?

    Best Regards,
    Steve T

  • 0 in reply to Steve T
    TI__Guru 59285 points

    Hi Steve,

    Thanks for sharing the additional information. It's unlikely TI will provide something like the customer plugin you mentioned.

    Regards,

    Jianzhong