• State Resolved
  • Locked Locked
  • Replies 6 replies
  • Subscribers 98 subscribers
  • Views 573 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-AM62A-LP: Video latency streaming H.264 encoded video from a IMX219

Steve T
Intellectual 310 points
Part Number: SK-AM62A-LP

I'm trying to determine what the cause of about a 2 second video latency using the AM62A EVM with the provided 08.06.00.45 SDK EdgeAI image from an attached Sony IMX219 camera streaming H.264 encoded video over RTP to a PC running VLC attached through a Gigabit ethernet switch and DHCP router.

I am using the following commands to start the stream (along with capturing gstreamer tracers):

root@am62axx-evm: media-ctl --set-v4l2 "'imx219 4-0010':0[fmt:SRGGB10_1X10/1920x1080 field:none]"

root@am62axx-evm: GST_DEBUG_FILE=/run/trace.log GST_DEBUG_NO_COLOR=1 GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=element)" gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video2 ! video/x-bayer, width=1920, height=1080, format=rggb10 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name=SENSOR_SONY_IMX219_RPI dcc-isp-file=/opt/imaging/imx219/dcc_viss_10b_1920x1080.bin sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a_10b_1920x1080.bin format-msb=9 ! v4l2h264enc output-io-mode=dmabuf-import ! rtph264pay config-interval=1 pt=96 ! udpsink host=XXX.XXX.XXX.XXX port=5000 -e 
(Note: XXX.XXX.XXX.XXX is replaced with an actual IP address of the receiver PC)

The gst-traceres script reports the following on my setup:

+-----------------------------------------------------------------------------------+
|element latency out-latancy out-fps frames |
+-----------------------------------------------------------------------------------+
|capsfilter0 0.10 33.31 30 3201 |
|tiovxisp0 9.05 33.30 30 3200 |
|v4l2h264enc0 8.41 33.28 30 3201 |
|rtph264pay0 0.24 33.27 30 3202 |
+-----------------------------------------------------------------------------------+

And from the perf_stats tool:


Summary of CPU load,
====================

CPU: mpu1_0: TOTAL LOAD = 5.70 % ( HWI = 0.24 %, SWI = 0.12 % )
CPU: mcu1_0: TOTAL LOAD = 2. 0 % ( HWI = 0. 0 %, SWI = 0. 0 % )
CPU: c7x_1: TOTAL LOAD = 0. 0 % ( HWI = 0. 0 %, SWI = 0. 0 % )


HWA performance statistics,
===========================

HWA: VISS: LOAD = 17.45 % ( 62 MP/s )


DDR performance statistics,
===========================

DDR: READ BW: AVG = 770 MB/s, PEAK = 2855 MB/s
DDR: WRITE BW: AVG = 319 MB/s, PEAK = 1190 MB/s
DDR: TOTAL BW: AVG = 1089 MB/s, PEAK = 4045 MB/s


My goal is to have a much lower video latency that is practically unnoticeable to human perception, in this setup at least. Any suggestions or help with areas to inspect would be appreciated.

Thank You

  • 0
    TI__Mastermind 27995 points

    Hi Steve,

    From the tracers that you have attached, I see that encoder is taking around 8ms (8.41) which is what is expected for encoding latency to be. Also, the element latency would not help us understand the latency of the pipeline. Appreciate if you could run full tracer to get pipeline latency.  

    GST_TRACERS="latency(flags=element+pipeline)" to see overall latency. 

    Even if we add the overall latency that you have shared it doesn't account to 2sec. 

    Element latency would be helpful in understanding which element in the pipeline is taking how much time. 

    Since, we are video streaming, we also need to understand the network delay and jitter in consideration. 

    Hope this helps.

    Best Regards

    Suren

  • 0 in reply to Suren Porwar
    Intellectual 310 points

    Hello Suren,

    Thank you for your assistance. I have gathered the Tracers with the added pipeline flag:

    I have also run a ping test from the AM62A to the receiving PC to characterize the network delay and jitter and it is under 1ms since the topology is just a LAN and pretty much as small as you can make it:

    root@am62axx-evm:/opt/edgeai-gst-apps# ping XXX.XXX.XXX.XXX
    PING XXX.XXX.XXX.XXX(XXX.XXX.XXX.XXX): 56 data bytes
    64 bytes from XXX.XXX.XXX.XXX: seq=0 ttl=128 time=0.573 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=1 ttl=128 time=0.475 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=2 ttl=128 time=0.471 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=3 ttl=128 time=0.593 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=4 ttl=128 time=0.734 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=5 ttl=128 time=0.982 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=6 ttl=128 time=0.466 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=7 ttl=128 time=0.477 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=8 ttl=128 time=0.455 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=9 ttl=128 time=0.476 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=10 ttl=128 time=0.491 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=11 ttl=128 time=0.521 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=12 ttl=128 time=0.510 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=13 ttl=128 time=0.504 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=14 ttl=128 time=0.482 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=15 ttl=128 time=0.638 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=16 ttl=128 time=0.482 ms
    64 bytes from XXX.XXX.XXX.XXX: seq=17 ttl=128 time=0.520 ms
    ^C
    --- XXX.XXX.XXX.XXX ping statistics ---
    18 packets transmitted, 18 packets received, 0% packet loss
    round-trip min/avg/max = 0.455/0.547/0.982 ms

    The IMX219 camera is from ArduCam and is V2.2 as marked on the PCB.

    Can any areas be investigated further?

    Thank you

  • 0 in reply to Steve T
    TI__Mastermind 27995 points

    Thanks Steve,

    While I investigate it further on my side, How are you observing 2 second video latency?

    Are you also trying to decode on the client side? or just trying to play the udp stream on VLC?

    Best Regards,

    Suren

  • 0 in reply to Suren Porwar
    Intellectual 310 points

    I was just removing my hand from the view of the camera and clocking how long it takes to disappear from the video on the receiver PC. It is admittedly more of a manual process but it is a very noticeable delay between them. Here is a picture of the clock along with the video stream pointed at the clock (running on the receiver PC). So the delay seems to actually be closer to 1.25-1.5 seconds.

     

    I have only been using VLC to receive the stream so far. I have created a local file called "stream.sdp" with the following contents and added it to the VLC playlist:

    v=0
    m=video 5000 RTP/AVP 96
    c=IN IP4 127.0.0.1
    a=rtpmap:96 H264/90000

    I have to hit the play button in VLC before starting the stream on the AM62A board currently to get it to start capturing on the receiver PC.

    Best Regards,

    Steve

  • +1 in reply to Steve T
    Intellectual 310 points

    Your comment made me realize that I hadn't tried this using Gstreamer to receive the stream instead of VLC. I opted not to go through attempting to install it on my Windows 10 system (which was using VLC) and used an Ubuntu 18.04 Virtual Machine that I already had with gstreamer installed. It turned out that this significantly reduced the video latency to a very acceptable level. I used the following command to receive the stream:

    gst-launch-1.0 udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! autovideosink sync=false

    And captured another pic of the video latency:

    So the issue seems to be related to VLC in Windows 10 in my setup.

    Best Regards,

    Steve True

  • 0 in reply to Steve T
    TI__Mastermind 27995 points

    Hi Steve,

    Happy to be of help. Since, the latency was caused by VLC and with decode and rendering it on videosink the latency is negligible, I will go ahead and close this thread. 

    Please don't hesitate to contact for any further assistance.

    Best Regards,

    Suren