AM62A7-Q1: 3840x2160: Unable to run VPAC with VISS and LDC at 30 fps

Part Number: AM62A7-Q1

Tool/software:

Hi,

When I run the following script to stream video from an AR0823 to ethernet, with VPAC using VISS and LDC, I am only able to get 22 fps, and the tiperfoverlay shows VISS and LDC using 48-49% each.

#!/bin/bash

MULTICASTADDR=224.1.1.1
PORT=5004

HEVC_PROFILE="main"
HEVC_LEVEL="1"
VIDEO_BITRATE=6000000
ENC_EXTRA_CONTROLS="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5,frame_level_rate_control_enable=1,video_bitrate_mode=0,vbv_buffer_size=3000,video_bitrate=${VIDEO_BITRATE}"

TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:' -n '$0' -- "$@")

if [ $? -ne 0 ]; then
	echo 'Terminating...' >&2
	exit 1
fi

eval set -- "$TEMP"
unset TEMP

while true; do
	case "$1" in
		'-h'|'--help')
			echo "$0 - H.265 compress a JPEG file into a one second stream"
			echo "Parameters:"
			echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
			echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
			echo "  -b 0..700000000 (Video Bitrate)"
			echo "  --mc <multicast ip address>"
			echo "  --port <port number> (default 5000)"
			echo "  -h This help"
			exit 0
			shift
			continue
		;;
		'-p'|'--profile')
			case "$2" in
				'main'|'Main'|'0')
					HEVC_PROFILE="main"
					;;
				'main-still-picture'|'1')
					HEVC_PROFILE="main-still-picture"
					;;
				'main-10'|'2')
					HEVC_PROFILE="main-10"
					;;
				*)
					HEVC_PROFILE="main"
					;;
			esac
			echo "HEVC Profile: '$HEVC_PROFILE'"
			shift 2
			continue
		;;
		'-l'|'--level')
			case "$2" in
				'1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
					HEVC_LEVEL="$2"
					;;
				*)
					HEVC_LEVEL="1"
					;;
			esac
			echo "HEVC Level: '$HEVC_LEVEL'"
			shift 2
			continue
		;;
		'-b'|'--bitrate')
			echo "Video Bitrate: '$2'"
			VIDEO_BITRATE=$2
			shift 2
			continue
		;;
		'mc')
			MULTICASTADDR=$2
			shift 2
			continue
		;;
		'port')
			PORT=$2
			shift 2
			continue
		;;
		'--')
			shift
			break
		;;
		*)
			echo 'Internal error!' >&2
			exit 1
		;;
	esac
done

#media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
#media-ctl -V '"ar0521 1-0036":0 [fmt:SGRBG8_1X8/3840x2160 field:none]'
media-ctl -V '"ar0823 1-0010":0 [fmt:SGRBG12_1X12/3840x2160 field:none]'
gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
	! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
	! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
		dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
		sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
	! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
	! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
	! queue max-size-buffers=1 leaky=0 \
	! tiperfoverlay title="Camera 1" \
	! v4l2h265enc extra-controls=${ENC_EXTRA_CONTROLS} \
	! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
	! rtph265pay config-interval=1 pt=96 \
	! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT} 

When I remove the LDC, tiperfoverlay shows a CPU usage for the VISS of 68%.

#!/bin/bash

MULTICASTADDR=224.1.1.1
PORT=5004

HEVC_PROFILE="main"
HEVC_LEVEL="1"
VIDEO_BITRATE=6000000
ENC_EXTRA_CONTROLS="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5,frame_level_rate_control_enable=1,video_bitrate_mode=0,vbv_buffer_size=3000,video_bitrate=${VIDEO_BITRATE}"

TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:' -n '$0' -- "$@")

if [ $? -ne 0 ]; then
	echo 'Terminating...' >&2
	exit 1
fi

eval set -- "$TEMP"
unset TEMP

while true; do
	case "$1" in
		'-h'|'--help')
			echo "$0 - H.265 compress a JPEG file into a one second stream"
			echo "Parameters:"
			echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
			echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
			echo "  -b 0..700000000 (Video Bitrate)"
			echo "  --mc <multicast ip address>"
			echo "  --port <port number> (default 5000)"
			echo "  -h This help"
			exit 0
			shift
			continue
		;;
		'-p'|'--profile')
			case "$2" in
				'main'|'Main'|'0')
					HEVC_PROFILE="main"
					;;
				'main-still-picture'|'1')
					HEVC_PROFILE="main-still-picture"
					;;
				'main-10'|'2')
					HEVC_PROFILE="main-10"
					;;
				*)
					HEVC_PROFILE="main"
					;;
			esac
			echo "HEVC Profile: '$HEVC_PROFILE'"
			shift 2
			continue
		;;
		'-l'|'--level')
			case "$2" in
				'1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
					HEVC_LEVEL="$2"
					;;
				*)
					HEVC_LEVEL="1"
					;;
			esac
			echo "HEVC Level: '$HEVC_LEVEL'"
			shift 2
			continue
		;;
		'-b'|'--bitrate')
			echo "Video Bitrate: '$2'"
			VIDEO_BITRATE=$2
			shift 2
			continue
		;;
		'mc')
			MULTICASTADDR=$2
			shift 2
			continue
		;;
		'port')
			PORT=$2
			shift 2
			continue
		;;
		'--')
			shift
			break
		;;
		*)
			echo 'Internal error!' >&2
			exit 1
		;;
	esac
done

#media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
#media-ctl -V '"ar0521 1-0036":0 [fmt:SGRBG8_1X8/3840x2160 field:none]'
media-ctl -V '"ar0823 1-0010":0 [fmt:SGRBG12_1X12/3840x2160 field:none]'
gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
	! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
	! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
		dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
		sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
	! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
	! queue max-size-buffers=1 leaky=0 \
	! tiperfoverlay title="Camera 1" \
	! v4l2h265enc extra-controls=${ENC_EXTRA_CONTROLS} \
	! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
	! rtph265pay config-interval=1 pt=96 \
	! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT} 

Is 3840x2160 @ 30 fps supported for the VPAC? If yes, how can I improve the frame rate of the first pipeline?

Regards,

Bas Vermeulen

  • Hello Bas,

    Is 3840x2160 @ 30 fps supported for the VPAC? If yes, how can I improve the frame rate of the first pipeline?

    Yes, AM62A VPAC can support up to 315MPixel per second. Your sensor is 3840x2160 @ 30fps = 249MPixel/s. It can be supported by AM62A VPAC.

    If yes, how can I improve the frame rate of the first pipeline?

    You can try to increase the buffer pool size, for example:

          gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
    	! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
    	! tiovxisp sink_0::pool-size=4 sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
    		dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
    		sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
    	! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin sink_0::pool-size=4 src::pool-size=4 \
    

    Regards,

    Jianzhong

  • Hi Jianzhong,

    Increasing the buffer-pool doesn't increase the frames per second. It's still at 22 fps instead of 30 fps.

    Are there any other things I can try? Simply streaming with v4l2-ctl -d 3 --stream-mmap gives me 30.02 fps, so the sensor/MIPI is able to sustain the required frame rate (and as said before, removing the LDC gets us back to an acceptable 29 fps according to the tiperfoverlay).

    Regards,

    Bas Vermeulen

  • Adding a queue element before tiovxldc should help:

          gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
    	! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
    	! tiovxisp sink_0::pool-size=4 sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
    		dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
    		sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
    	! queue ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin sink_0::pool-size=4 src::pool-size=4 \

  • Hi Jianzhong,

    Adding a queue element gets me to 25 fps (with or without the pool-size of 4).

    Removing the tiperfoverlay element gets me to 26 fps, but still not getting to ~30.

    Regards,

    Bas Vermeulen

  • Hi Bas,

    Can you try to add a queue before v4l2h265enc and rtph265pay?

    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
    	! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
    	! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
    		dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
    		sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
    	! queue ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
    	! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
    	! queue max-size-buffers=1 leaky=0 \
    	! tiperfoverlay title="Camera 1" \
    	! queue ! v4l2h265enc extra-controls=${ENC_EXTRA_CONTROLS} \
    	! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
    	! queue ! rtph265pay config-interval=1 pt=96 \
    	! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT} 

  • Adding a queue there drops the framerate to 23 fps.

  • Can you stream the camera data to a display instead of sending to the network, or use a fakesink after tiovxldc? That can help us find where the bottleneck is.

  • The following pipeline gives me 30 fps:

    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
                    dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
                    sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
            ! queue \
            ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
            ! queue max-size-buffers=1 leaky=0 \
            ! fakesink

    If I add a convert line and tiperfoverlay, that drops to 21:

    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
                    dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
                    sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
            ! queue \
            ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
            ! queue max-size-buffers=1 leaky=0 \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! tiperfoverlay title="Camera 1" \
            ! fakesink

    Removing the tiperfoverlay from there gets me back to 30 fps.

    If I add the v4lh265enc line, the fps drops to 26 again. I can add the rtp payloader and the fps stays 26.

  • If I add a convert line and tiperfoverlay, that drops to 21:

    Can you try to add a queue before tiperfoverlay? For example:

            ! queue max-size-buffers=1 leaky=0 \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! queue ! tiperfoverlay title="Camera 1" \
    

  • We came out at the following gstreamer pipeline that gets full performance (30 fps):

    #!/bin/bash
    
    MULTICASTADDR=224.1.1.1
    PORT=5004
    
    HEVC_PROFILE="main"
    HEVC_LEVEL="1"
    VIDEO_BITRATE=6000000
    ENC_EXTRA_CONTROLS="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5,frame_level_rate_control_enable=1,video_bitrate_mode=0,vbv_buffer_size=3000,video_bitrate=${VIDEO_BITRATE}"
    
    TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:' -n '$0' -- "$@")
    
    if [ $? -ne 0 ]; then
            echo 'Terminating...' >&2
            exit 1
    fi
    
    eval set -- "$TEMP"
    unset TEMP
    
    while true; do
            case "$1" in
                    '-h'|'--help')
                            echo "$0 - H.265 compress a JPEG file into a one second stream"
                            echo "Parameters:"
                            echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
                            echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
                            echo "  -b 0..700000000 (Video Bitrate)"
                            echo "  --mc <multicast ip address>"
                            echo "  --port <port number> (default 5000)"
                            echo "  -h This help"
                            exit 0
                            shift
                            continue
                    ;;
                    '-p'|'--profile')
                            case "$2" in
                                    'main'|'Main'|'0')
                                            HEVC_PROFILE="main"
                                            ;;
                                    'main-still-picture'|'1')
                                            HEVC_PROFILE="main-still-picture"
                                            ;;
                                    'main-10'|'2')
                                            HEVC_PROFILE="main-10"
                                            ;;
                                    *)
                                            HEVC_PROFILE="main"
                                            ;;
                            esac
                            echo "HEVC Profile: '$HEVC_PROFILE'"
                            shift 2
                            continue
                    ;;
                    '-l'|'--level')
                            case "$2" in
                                    '1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
                                            HEVC_LEVEL="$2"
                                            ;;
                                    *)
                                            HEVC_LEVEL="1"
                                            ;;
                            esac
                            echo "HEVC Level: '$HEVC_LEVEL'"
                            shift 2
                            continue
                    ;;
                    '-b'|'--bitrate')
                            echo "Video Bitrate: '$2'"
                            VIDEO_BITRATE=$2
                            shift 2
                            continue
                    ;;
                    'mc')
                            MULTICASTADDR=$2
                            shift 2
                            continue
                    ;;
                    'port')
                            PORT=$2
                            shift 2
                            continue
                    ;;
                    '--')
                            shift
                            break
                    ;;
                    *)
                            echo 'Internal error!' >&2
                            exit 1
                    ;;
            esac
    done
    
    #media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
    #media-ctl -V '"ar0521 1-0036":0 [fmt:SGRBG8_1X8/3840x2160 field:none]'
    media-ctl -V '"ar0823 1-0010":0 [fmt:SGRBG12_1X12/3840x2160 field:none]'
    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
                    dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
                    sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
                    sink_0::pool-size=4 \
            ! queue max-size-buffers=1 leaky=0 \
            ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
                    sink_0::pool-size=4 src::pool-size=4 \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! queue max-size-buffers=1 leaky=0 \
            ! v4l2h265enc output-io-mode=dmabuf-import extra-controls=${ENC_EXTRA_CONTROLS} \
            ! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
            ! rtph265pay config-interval=1 pt=96 \
            ! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT}
    

    The main modifications are adding a queue between tiovxisp and tiovxldc, moving the other queue down below the capsfilter line, and adding output-io-mode=dmabuf-import to the vhl2h265enc line.

    With that, I get the following statistics:

    perf_stats output:

    Summary of CPU load,
    ====================
    
    CPU: mpu1_0: TOTAL LOAD =  27.81 % ( HWI =   1.50 %, SWI =   1.25 % )
    CPU:  c7x_1: TOTAL LOAD =   0. 2 % ( HWI =   0. 0 %, SWI =   0. 0 % )
    
    HWA performance statistics,
    ===========================
    
    HWA:   VISS: LOAD =  69.50 % ( 255 MP/s )
    HWA:   LDC : LOAD =  72.49 % ( 256 MP/s )
    
    DDR performance statistics,
    ===========================
    
    DDR: READ  BW: AVG =   2905 MB/s
    DDR: WRITE BW: AVG =   1775 MB/s
    DDR: TOTAL BW: AVG =   4680 MB/s
    
    SoC temperature statistics
    ==========================
    
    thermal_zone0(DDR):     72.94 degree Celsius
    thermal_zone1(CPU):     71.33 degree Celsius
    thermal_zone2(C7x):     72.74 degree Celsius

    parse_gst_tracers.py:

    +-----------------------------------------------------------------------------------+
    |element                       latency      out-latancy      out-fps     frames     |
    +-----------------------------------------------------------------------------------+
    |capsfilter0                   0.51         33.31            30          17184      |
    |tiovxisp0                     25.16        33.31            30          17183      |
    |queue0                        0.91         33.31            30          17183      |
    |tiovxldc0                     25.92        33.30            30          17182      |
    |capsfilter1                   0.65         33.30            30          17182      |
    |queue1                        0.59         33.30            30          17182      |
    |v4l2h265enc0                  31.15        33.30            30          17181      |
    |capsfilter2                   0.65         33.30            30          17181      |
    |v4l2src0                      86.80        33.30            30          17181      |
    |rtph265pay0                   1.27         33.30            30          17181      |
    +-----------------------------------------------------------------------------------+

    Thank you very much for the help! This gets me to 30 frames per second, and a latency that's really nice.

    Regards,

    Bas Vermeulen

  • I need to follow up on this; I am trying to use a tee to have two streams running from the same video source at the same time. When I do this, my framerate drops to 25 (when using tee) or 15 (when using tiovxmultiscaler).

    The script I use to test:

    #!/bin/bash
    
    MULTICASTADDR1=224.1.1.1
    MULTICASTADDR2=224.1.1.2
    PORT=5004
    
    TEE=tee
    HEVC_PROFILE="main"
    HEVC_LEVEL="1"
    VIDEO_BITRATE=6000000
    ENC_EXTRA_CONTROLS="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5,frame_level_rate_control_enable=1,video_bitrate_mode=0,vbv_buffer_size=3000,video_bitrate=${VIDEO_BITRATE}"
    
    TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:' -n '$0' -- "$@")
    
    if [ $? -ne 0 ]; then
            echo 'Terminating...' >&2
            exit 1
    fi
    
    eval set -- "$TEMP"
    unset TEMP
    
    while true; do
            case "$1" in
                    '-h'|'--help')
                            echo "$0 - H.265 compress a JPEG file into a one second stream"
                            echo "Parameters:"
                            echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
                            echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
                            echo "  -b 0..700000000 (Video Bitrate)"
                            echo "  --mc <multicast ip address>"
                            echo "  --port <port number> (default 5000)"
                            echo "  -h This help"
                            exit 0
                            shift
                            continue
                    ;;
                    '-p'|'--profile')
                            case "$2" in
                                    'main'|'Main'|'0')
                                            HEVC_PROFILE="main"
                                            ;;
                                    'main-still-picture'|'1')
                                            HEVC_PROFILE="main-still-picture"
                                            ;;
                                    'main-10'|'2')
                                            HEVC_PROFILE="main-10"
                                            ;;
                                    *)
                                            HEVC_PROFILE="main"
                                            ;;
                            esac
                            echo "HEVC Profile: '$HEVC_PROFILE'"
                            shift 2
                            continue
                    ;;
                    '-l'|'--level')
                            case "$2" in
                                    '1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
                                            HEVC_LEVEL="$2"
                                            ;;
                                    *)
                                            HEVC_LEVEL="1"
                                            ;;
                            esac
                            echo "HEVC Level: '$HEVC_LEVEL'"
                            shift 2
                            continue
                    ;;
                    '-b'|'--bitrate')
                            echo "Video Bitrate: '$2'"
                            VIDEO_BITRATE=$2
                            shift 2
                            continue
                    ;;
                    'mc')
                            MULTICASTADDR=$2
                            shift 2
                            continue
                    ;;
                    'port')
                            PORT=$2
                            shift 2
                            continue
                    ;;
                    '--')
                            shift
                            break
                    ;;
                    *)
                            echo 'Internal error!' >&2
                            exit 1
                    ;;
            esac
    done
    
    #media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
    #media-ctl -V '"ar0521 1-0036":0 [fmt:SGRBG8_1X8/3840x2160 field:none]'
    media-ctl -V '"ar0823 1-0010":0 [fmt:SGRBG12_1X12/3840x2160 field:none]'
    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
                    dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
                    sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
            ! queue max-size-buffers=1 leaky=0 \
            ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! ${TEE} name=multi \
            multi. \
            ! queue max-size-buffers=1 leaky=0 name=qstream1 \
            ! v4l2h265enc output-io-mode=dmabuf-import extra-controls=${ENC_EXTRA_CONTROLS} \
            ! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
            ! rtph265pay config-interval=1 pt=96 \
            ! udpsink host=${MULTICASTADDR1} auto-multicast=true port=${PORT} \
            multi. \
            ! queue max-size-buffers=1 leaky=0 name=qstream2 \
            ! v4l2h265enc output-io-mode=dmabuf-import extra-controls=${ENC_EXTRA_CONTROLS} \
            ! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
            ! rtph265pay config-interval=1 pt=96 \
            ! udpsink host=${MULTICASTADDR2} auto-multicast=true port=${PORT}

    If you set the TEE variable in the script to tiovxmultiscaler, it will use the multiscaler, when set to tee it uses the tee element.

    My end goal is being able to dynamically change my pipeline from streaming to multiple fakesinks, to streaming to one or two network (UDP) sinks and a screen (all running at 30 fps).

    Removing the second branch of the tee will make the pipeline run at 30 fps.

    Any idea how to get this running at 30 fps with two or more branches?

    Regards,

    Bas Vermeulen

  • Hi Bas,

    If I understand correctly, what you are seeing is when you have a tee element added to the output to tiovxldc and stream two encoded streams you are seeing the FPS to be 25fps and if you don't use tee element and use tiovxmultiscaler, you are seeing the FPS to be 15. 

    Also can you share the tracers that you are seeing when using the script with tee element and multiscaler to see what element is causing the fps to drop?

    Best Regards,

    Suren

  • Hi Suren,

    Correct. If I drop the second encoder (and replace with a fakesink), I get back to 30 fps. If I use two encoders, I get 25 (tee) and 15 (tiovxmultiscaler).

    Output of parse_gst_tracers.py for the script with TEE=tee:

    +-----------------------------------------------------------------------------------+
    |element                       latency      out-latancy      out-fps     frames     |
    +-----------------------------------------------------------------------------------+
    |capsfilter0                   0.39         39.19            25          1033       |
    |tiovxisp0                     65.18        39.15            25          1031       |
    |queue0                        15.02        39.17            25          1031       |
    |tiovxldc0                     36.67        39.15            25          1030       |
    |capsfilter1                   0.55         39.15            25          1030       |
    |multi                         0.72         19.57            51          2060       |
    |qstream1                      0.53         39.15            25          1030       |
    |qstream2                      0.64         39.14            25          1030       |
    |v4l2h265enc1                  51.01        39.04            25          1028       |
    |capsfilter3                   0.65         39.04            25          1028       |
    |v4l2src0                      163.60       19.52            51          2057       |
    |rtph265pay1                   1.23         39.04            25          1028       |
    |v4l2h265enc0                  34.20        39.00            25          1029       |
    |capsfilter2                   0.64         39.00            25          1029       |
    |rtph265pay0                   1.23         39.00            25          1029       |
    +-----------------------------------------------------------------------------------+

    Output of parse_gst_tracers.py for the script with TEE=tiovxmultiscaler:

    +-----------------------------------------------------------------------------------+
    |element                       latency      out-latancy      out-fps     frames     |
    +-----------------------------------------------------------------------------------+
    |capsfilter0                   0.52         64.00            15          862        |
    |tiovxisp0                     111.86       63.98            15          860        |
    |queue0                        39.77        64.03            15          860        |
    |tiovxldc0                     25.57        64.00            15          859        |
    |capsfilter1                   0.72         64.00            15          859        |
    |multi                         36.01        31.97            31          1718       |
    |qstream1                      0.80         63.94            15          859        |
    |qstream2                      0.71         63.94            15          859        |
    |v4l2h265enc0                  32.39        63.82            15          858        |
    |capsfilter2                   0.70         63.82            15          858        |
    |v4l2src0                      258.53       31.92            31          1716       |
    |rtph265pay0                   1.46         63.82            15          858        |
    |v4l2h265enc1                  50.01        63.82            15          858        |
    |capsfilter3                   0.69         63.82            15          858        |
    |rtph265pay1                   1.39         63.82            15          858        |
    +-----------------------------------------------------------------------------------+

    I always find it difficult to judge what element is causing the extra latency.

    Regards,

    Bas Vermeulen

  • Hi Bas,

    Is it okay if you were to try the same experiment with 1080 resolution? I am suspecting two 4K stream encode would be an overkill on the system and causing frame drops.

    Best Regards,

    Suren

  • Hi Bas,

    Is it okay if you were to try the same experiment with 1080 resolution? I am suspecting two 4K stream encode would be an overkill on the system and causing frame drops.

    Best Regards,

    Suren

  • I have a pipeline running on the EVM (with the imx219 camera, and without LDC) where I have a 4K stream and a full HD stream running at the same time.

    #!/bin/bash
    
    MULTICASTADDR=224.1.1.1
    PORT=5000
    MULTICASTADDR1=224.2.2.1
    PORT1=5000
    
    HEVC_PROFILE="main"
    HEVC_LEVEL="1"
    VIDEO_BITRATE=0
    
    TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:,mc1:,port1:' -n '$0' -- "$@")
    
    if [ $? -ne 0 ]; then
            echo 'Terminating...' >&2
            exit 1
    fi
    
    eval set -- "$TEMP"
    unset TEMP
    
    while true; do
            case "$1" in
                    '-h'|'--help')
                            echo "$0 - H.265 compress a JPEG file into a one second stream"
                            echo "Parameters:"
                            echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
                            echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
                            echo "  -b 0..700000000 (Video Bitrate)"
                            echo "  --mc <multicast ip address>"
                            echo "  --port <port number> (default 5000)"
                            echo "  --mc1 <multicast ip address>"
                            echo "  --port1 <port number> (default 5000)"
                            echo "  -h This help"
                            exit 0
                            shift
                            continue
                    ;;
                    '-p'|'--profile')
                            case "$2" in
                                    'main'|'Main'|'0')
                                            HEVC_PROFILE="main"
                                            ;;
                                    'main-still-picture'|'1')
                                            HEVC_PROFILE="main-still-picture"
                                            ;;
                                    'main-10'|'2')
                                            HEVC_PROFILE="main-10"
                                            ;;
                                    *)
                                            HEVC_PROFILE="main"
                                            ;;
                            esac
                            echo "HEVC Profile: '$HEVC_PROFILE'"
                            shift 2
                            continue
                    ;;
                    '-l'|'--level')
                            case "$2" in
                                    '1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
                                            HEVC_LEVEL="$2"
                                            ;;
                                    *)
                                            HEVC_LEVEL="1"
                                            ;;
                            esac
                            echo "HEVC Level: '$HEVC_LEVEL'"
                            shift 2
                            continue
                    ;;
                    '-b'|'--bitrate')
                            echo "Video Bitrate: '$2'"
                            VIDEO_BITRATE=$2
                            shift 2
                            continue
                    ;;
                    'mc')
                            MULTICASTADDR=$2
                            shift 2
                            continue
                    ;;
                    'port')
                            PORT=$2
                            shift 2
                            continue
                    ;;
                    'mc1')
                            MULTICASTADDR1=$2
                            shift 2
                            continue
                    ;;
                    'port1')
                            PORT1=$2
                            shift 2
                            continue
                    ;;
                    '--')
                            shift
                            break
                    ;;
                    *)
                            echo 'Internal error!' >&2
                            exit 1
                    ;;
            esac
    done
    
    media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
    gst-launch-1.0 -q v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb10 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_SONY_IMX219_RPI" \
                    dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss_10b.bin \
                    sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a_10b.bin format-msb=9 \
            ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 \
            ! tiovxmultiscaler name=multi target=0 \
            multi.src_0 \
            ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 \
            ! mosaic.sink_0 \
            multi.src_1 \
            ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
            ! mosaic.sink_1 \
            multi.src_2 \
            ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
            ! mosaic.sink_2 \
            multi.src_3 \
            ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
            ! mosaic.sink_3 \
            multi.src_4 \
            ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
            ! queue max-size-buffers=1 leaky=0 \
            ! v4l2h265enc extra-controls="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5" \
            ! rtph265pay config-interval=1 pt=96 \
            ! udpsink host=${MULTICASTADDR1} auto-multicast=true port=${PORT1} \
            tiovxmosaic name=mosaic target=2 \
            sink_0::startx="<0>" sink_0::starty="<0>" \
            sink_1::startx="<1920>" sink_1::starty="<1080>" \
            sink_2::startx="<0>" sink_2::starty="<1080>" \
            sink_3::startx="<1920>" sink_3::starty="<0>" \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! queue max-size-buffers=1 leaky=0 \
            ! v4l2h265enc extra-controls="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5" \
            ! rtph265pay config-interval=1 pt=96 \
            ! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT}

    parse_gst_tracers.py output:

    +-----------------------------------------------------------------------------------+
    |element                       latency      out-latancy      out-fps     frames     |
    +-----------------------------------------------------------------------------------+
    |capsfilter0                   0.27         33.33            30          6594       |
    |tiovxisp0                     8.55         33.32            30          6601       |
    |capsfilter1                   0.38         33.32            30          6605       |
    |multi                         11.87        6.66             150         33040      |
    |capsfilter2                   0.37         33.32            30          6612       |
    |capsfilter3                   0.23         33.32            30          6618       |
    |capsfilter4                   0.21         33.32            30          6618       |
    |capsfilter5                   0.20         33.32            30          6618       |
    |capsfilter6                   0.21         33.32            30          6618       |
    |queue0                        0.27         33.32            30          6618       |
    |v4l2h265enc0                  27.46        33.31            30          6617       |
    |v4l2src0                      79.19        16.66            60          13232      |
    |rtph265pay0                   0.53         33.31            30          6617       |
    |mosaic                        40.56        33.31            30          6616       |
    |capsfilter7                   0.35         33.31            30          6620       |
    |queue1                        0.25         33.31            30          6621       |
    |v4l2h265enc1                  44.34        33.30            30          6622       |
    |rtph265pay1                   0.73         33.30            30          6623       |
    +-----------------------------------------------------------------------------------+

    The only difference with what I am trying now is that this pipeline doesn't have the LDC, and I am using the tiovxmultiscaler + tiovxmosaic to generate a 4K stream.

    That's why I don't really understand why everything is so slow when using the larger camera.

    The encoding should be fine. I just don't understand why I can't translate this to our phyboard with the 4K camera.

    Regards,

    Bas

  • Hi Bas,

    Let me try and run some pipelines on my AM62A board early next week and get back to you with the observations on my end.

    Have a great weekend!

    Best Regards,

    Suren

  • Hi ,

    Were you able to run some pipelines, and what were your observations?

    Regards,

    Bas Vermeulen

  • Hi Bas,

    I tried the below pipeline with 1080p and with videotestsrc. 

    [2024-10-09 16:32:43.589] root@am62axx-evm:/opt/edgeai-gst-apps# GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG=GST_TRACER:7 GST_DEBUG_FILE="/run/latency_4k.txt" \
    [2024-10-09 16:32:46.483] > gst-launch-1.0 -v videotestsrc ! \
    [2024-10-09 16:32:46.535] > video/x-raw, width=1920, height=1080, framerate=60/1, format=NV12 ! \
    [2024-10-09 16:32:46.635] > tee name=tee_split0 \
    [2024-10-09 16:32:46.687] > tee_split0. ! queue ! v4l2h265enc ! rtph265pay ! udpsink host=127.0.0.1 port=5001 \
    [2024-10-09 16:32:46.845] > tee_split0. ! queue ! v4l2h265enc ! rtph265pay ! udpsink host=127.0.0.1 port=6001 
    [2024-10-09 16:32:48.405] Setting pipeline to PAUSED ...
    +-----------------------------------------------------------------------------------+
    [2024-10-09 17:08:49.619] |element                       latency      out-latancy      out-fps     frames     |
    [2024-10-09 17:08:49.635] +-----------------------------------------------------------------------------------+
    [2024-10-09 17:08:49.636] |capsfilter0                   0.21         22.12            45          302        |
    [2024-10-09 17:08:49.651] |tee_split0                    0.88         11.06            90          604        |
    [2024-10-09 17:08:49.651] |queue0                        19.00        22.12            45          302        |
    [2024-10-09 17:08:49.667] |queue1                        21.27        22.12            45          302        |
    [2024-10-09 17:08:49.667] |v4l2h265enc1                  39.21        22.04            45          301        |
    [2024-10-09 17:08:49.683] |videotestsrc0                 61.15        11.02            90          601        |
    [2024-10-09 17:08:49.683] |rtph265pay1                   0.46         22.01            45          300        |
    [2024-10-09 17:08:49.699] |v4l2h265enc0                  39.47        21.98            45          301        |
    [2024-10-09 17:08:49.699] |rtph265pay0                   0.44         21.97            45          301        |
    [2024-10-09 17:08:49.700] +-----------------------------------------------------------------------------------+
    

    Not sure, why the tee element is receiving half of frames. Will try and experiment more on this. 

    Best Regards,

    Suren

  • Hi Bas,

    Continuing on running IMX219 with 4K resolution:

    Below pipeline:

    GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG=GST_TRACER:7 GST_DEBUG_FILE="/run/latency_4k-enc-1.txt" \
    gst-launch-1.0 v4l2src device=/dev/video-imx219-cam0 io-mode=dmabuf-import ! queue max-size-buffers=1 leaky=2 ! \
    video/x-bayer, width=3280, height=2464, framerate=15/1, format=rggb10 ! \
    tiovxisp sink_0::pool-size=4 sink_0::device=/dev/v4l-imx219-subdev0 sensor-name="SENSOR_SONY_IMX219_RPI" \
    dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss_10b.bin \
    sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a_10b.bin format-msb=9 ! \
    video/x-raw, format=NV12, width=3280, height=2464, framerate=15/1 ! queue ! tiovxmultiscaler ! queue ! \
    video/x-raw, format=NV12, width=1920, height=1080, framerate=15/1 ! \
    v4l2h264enc ! rtph264pay ! tee name=t1 \
    t1. ! queue ! udpsink host=127.0.0.1 port=5001 \
    t1. ! queue ! udpsink host=127.0.0.1 port=6001

    Best Regards,

    Suren