AM62A7-Q1: 3840x2160: Unable to run VPAC with VISS and LDC at 30 fps

Part Number: AM62A7-Q1

Tool/software:

Hi,

When I run the following script to stream video from an AR0823 to ethernet, with VPAC using VISS and LDC, I am only able to get 22 fps, and the tiperfoverlay shows VISS and LDC using 48-49% each.

#!/bin/bash

MULTICASTADDR=224.1.1.1
PORT=5004

HEVC_PROFILE="main"
HEVC_LEVEL="1"
VIDEO_BITRATE=6000000
ENC_EXTRA_CONTROLS="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5,frame_level_rate_control_enable=1,video_bitrate_mode=0,vbv_buffer_size=3000,video_bitrate=${VIDEO_BITRATE}"

TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:' -n '$0' -- "$@")

if [ $? -ne 0 ]; then
	echo 'Terminating...' >&2
	exit 1
fi

eval set -- "$TEMP"
unset TEMP

while true; do
	case "$1" in
		'-h'|'--help')
			echo "$0 - H.265 compress a JPEG file into a one second stream"
			echo "Parameters:"
			echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
			echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
			echo "  -b 0..700000000 (Video Bitrate)"
			echo "  --mc <multicast ip address>"
			echo "  --port <port number> (default 5000)"
			echo "  -h This help"
			exit 0
			shift
			continue
		;;
		'-p'|'--profile')
			case "$2" in
				'main'|'Main'|'0')
					HEVC_PROFILE="main"
					;;
				'main-still-picture'|'1')
					HEVC_PROFILE="main-still-picture"
					;;
				'main-10'|'2')
					HEVC_PROFILE="main-10"
					;;
				*)
					HEVC_PROFILE="main"
					;;
			esac
			echo "HEVC Profile: '$HEVC_PROFILE'"
			shift 2
			continue
		;;
		'-l'|'--level')
			case "$2" in
				'1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
					HEVC_LEVEL="$2"
					;;
				*)
					HEVC_LEVEL="1"
					;;
			esac
			echo "HEVC Level: '$HEVC_LEVEL'"
			shift 2
			continue
		;;
		'-b'|'--bitrate')
			echo "Video Bitrate: '$2'"
			VIDEO_BITRATE=$2
			shift 2
			continue
		;;
		'mc')
			MULTICASTADDR=$2
			shift 2
			continue
		;;
		'port')
			PORT=$2
			shift 2
			continue
		;;
		'--')
			shift
			break
		;;
		*)
			echo 'Internal error!' >&2
			exit 1
		;;
	esac
done

#media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
#media-ctl -V '"ar0521 1-0036":0 [fmt:SGRBG8_1X8/3840x2160 field:none]'
media-ctl -V '"ar0823 1-0010":0 [fmt:SGRBG12_1X12/3840x2160 field:none]'
gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
	! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
	! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
		dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
		sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
	! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
	! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
	! queue max-size-buffers=1 leaky=0 \
	! tiperfoverlay title="Camera 1" \
	! v4l2h265enc extra-controls=${ENC_EXTRA_CONTROLS} \
	! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
	! rtph265pay config-interval=1 pt=96 \
	! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT} 

When I remove the LDC, tiperfoverlay shows a CPU usage for the VISS of 68%.

#!/bin/bash

MULTICASTADDR=224.1.1.1
PORT=5004

HEVC_PROFILE="main"
HEVC_LEVEL="1"
VIDEO_BITRATE=6000000
ENC_EXTRA_CONTROLS="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5,frame_level_rate_control_enable=1,video_bitrate_mode=0,vbv_buffer_size=3000,video_bitrate=${VIDEO_BITRATE}"

TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:' -n '$0' -- "$@")

if [ $? -ne 0 ]; then
	echo 'Terminating...' >&2
	exit 1
fi

eval set -- "$TEMP"
unset TEMP

while true; do
	case "$1" in
		'-h'|'--help')
			echo "$0 - H.265 compress a JPEG file into a one second stream"
			echo "Parameters:"
			echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
			echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
			echo "  -b 0..700000000 (Video Bitrate)"
			echo "  --mc <multicast ip address>"
			echo "  --port <port number> (default 5000)"
			echo "  -h This help"
			exit 0
			shift
			continue
		;;
		'-p'|'--profile')
			case "$2" in
				'main'|'Main'|'0')
					HEVC_PROFILE="main"
					;;
				'main-still-picture'|'1')
					HEVC_PROFILE="main-still-picture"
					;;
				'main-10'|'2')
					HEVC_PROFILE="main-10"
					;;
				*)
					HEVC_PROFILE="main"
					;;
			esac
			echo "HEVC Profile: '$HEVC_PROFILE'"
			shift 2
			continue
		;;
		'-l'|'--level')
			case "$2" in
				'1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
					HEVC_LEVEL="$2"
					;;
				*)
					HEVC_LEVEL="1"
					;;
			esac
			echo "HEVC Level: '$HEVC_LEVEL'"
			shift 2
			continue
		;;
		'-b'|'--bitrate')
			echo "Video Bitrate: '$2'"
			VIDEO_BITRATE=$2
			shift 2
			continue
		;;
		'mc')
			MULTICASTADDR=$2
			shift 2
			continue
		;;
		'port')
			PORT=$2
			shift 2
			continue
		;;
		'--')
			shift
			break
		;;
		*)
			echo 'Internal error!' >&2
			exit 1
		;;
	esac
done

#media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
#media-ctl -V '"ar0521 1-0036":0 [fmt:SGRBG8_1X8/3840x2160 field:none]'
media-ctl -V '"ar0823 1-0010":0 [fmt:SGRBG12_1X12/3840x2160 field:none]'
gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
	! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
	! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
		dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
		sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
	! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
	! queue max-size-buffers=1 leaky=0 \
	! tiperfoverlay title="Camera 1" \
	! v4l2h265enc extra-controls=${ENC_EXTRA_CONTROLS} \
	! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
	! rtph265pay config-interval=1 pt=96 \
	! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT} 

Is 3840x2160 @ 30 fps supported for the VPAC? If yes, how can I improve the frame rate of the first pipeline?

Regards,

Bas Vermeulen

  • Hello Bas,

    Is 3840x2160 @ 30 fps supported for the VPAC? If yes, how can I improve the frame rate of the first pipeline?

    Yes, AM62A VPAC can support up to 315MPixel per second. Your sensor is 3840x2160 @ 30fps = 249MPixel/s. It can be supported by AM62A VPAC.

    If yes, how can I improve the frame rate of the first pipeline?

    You can try to increase the buffer pool size, for example:

          gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
    	! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
    	! tiovxisp sink_0::pool-size=4 sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
    		dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
    		sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
    	! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin sink_0::pool-size=4 src::pool-size=4 \
    

    Regards,

    Jianzhong

  • Hi Jianzhong,

    Increasing the buffer-pool doesn't increase the frames per second. It's still at 22 fps instead of 30 fps.

    Are there any other things I can try? Simply streaming with v4l2-ctl -d 3 --stream-mmap gives me 30.02 fps, so the sensor/MIPI is able to sustain the required frame rate (and as said before, removing the LDC gets us back to an acceptable 29 fps according to the tiperfoverlay).

    Regards,

    Bas Vermeulen

  • Adding a queue element before tiovxldc should help:

          gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
    	! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
    	! tiovxisp sink_0::pool-size=4 sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
    		dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
    		sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
    	! queue ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin sink_0::pool-size=4 src::pool-size=4 \

  • Hi Jianzhong,

    Adding a queue element gets me to 25 fps (with or without the pool-size of 4).

    Removing the tiperfoverlay element gets me to 26 fps, but still not getting to ~30.

    Regards,

    Bas Vermeulen

  • Hi Bas,

    Can you try to add a queue before v4l2h265enc and rtph265pay?

    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
    	! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
    	! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
    		dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
    		sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
    	! queue ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
    	! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
    	! queue max-size-buffers=1 leaky=0 \
    	! tiperfoverlay title="Camera 1" \
    	! queue ! v4l2h265enc extra-controls=${ENC_EXTRA_CONTROLS} \
    	! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
    	! queue ! rtph265pay config-interval=1 pt=96 \
    	! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT} 

  • Adding a queue there drops the framerate to 23 fps.

  • Can you stream the camera data to a display instead of sending to the network, or use a fakesink after tiovxldc? That can help us find where the bottleneck is.

  • The following pipeline gives me 30 fps:

    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
                    dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
                    sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
            ! queue \
            ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
            ! queue max-size-buffers=1 leaky=0 \
            ! fakesink

    If I add a convert line and tiperfoverlay, that drops to 21:

    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
                    dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
                    sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
            ! queue \
            ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
            ! queue max-size-buffers=1 leaky=0 \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! tiperfoverlay title="Camera 1" \
            ! fakesink

    Removing the tiperfoverlay from there gets me back to 30 fps.

    If I add the v4lh265enc line, the fps drops to 26 again. I can add the rtp payloader and the fps stays 26.

  • If I add a convert line and tiperfoverlay, that drops to 21:

    Can you try to add a queue before tiperfoverlay? For example:

            ! queue max-size-buffers=1 leaky=0 \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! queue ! tiperfoverlay title="Camera 1" \
    

  • We came out at the following gstreamer pipeline that gets full performance (30 fps):

    #!/bin/bash
    
    MULTICASTADDR=224.1.1.1
    PORT=5004
    
    HEVC_PROFILE="main"
    HEVC_LEVEL="1"
    VIDEO_BITRATE=6000000
    ENC_EXTRA_CONTROLS="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5,frame_level_rate_control_enable=1,video_bitrate_mode=0,vbv_buffer_size=3000,video_bitrate=${VIDEO_BITRATE}"
    
    TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:' -n '$0' -- "$@")
    
    if [ $? -ne 0 ]; then
            echo 'Terminating...' >&2
            exit 1
    fi
    
    eval set -- "$TEMP"
    unset TEMP
    
    while true; do
            case "$1" in
                    '-h'|'--help')
                            echo "$0 - H.265 compress a JPEG file into a one second stream"
                            echo "Parameters:"
                            echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
                            echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
                            echo "  -b 0..700000000 (Video Bitrate)"
                            echo "  --mc <multicast ip address>"
                            echo "  --port <port number> (default 5000)"
                            echo "  -h This help"
                            exit 0
                            shift
                            continue
                    ;;
                    '-p'|'--profile')
                            case "$2" in
                                    'main'|'Main'|'0')
                                            HEVC_PROFILE="main"
                                            ;;
                                    'main-still-picture'|'1')
                                            HEVC_PROFILE="main-still-picture"
                                            ;;
                                    'main-10'|'2')
                                            HEVC_PROFILE="main-10"
                                            ;;
                                    *)
                                            HEVC_PROFILE="main"
                                            ;;
                            esac
                            echo "HEVC Profile: '$HEVC_PROFILE'"
                            shift 2
                            continue
                    ;;
                    '-l'|'--level')
                            case "$2" in
                                    '1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
                                            HEVC_LEVEL="$2"
                                            ;;
                                    *)
                                            HEVC_LEVEL="1"
                                            ;;
                            esac
                            echo "HEVC Level: '$HEVC_LEVEL'"
                            shift 2
                            continue
                    ;;
                    '-b'|'--bitrate')
                            echo "Video Bitrate: '$2'"
                            VIDEO_BITRATE=$2
                            shift 2
                            continue
                    ;;
                    'mc')
                            MULTICASTADDR=$2
                            shift 2
                            continue
                    ;;
                    'port')
                            PORT=$2
                            shift 2
                            continue
                    ;;
                    '--')
                            shift
                            break
                    ;;
                    *)
                            echo 'Internal error!' >&2
                            exit 1
                    ;;
            esac
    done
    
    #media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
    #media-ctl -V '"ar0521 1-0036":0 [fmt:SGRBG8_1X8/3840x2160 field:none]'
    media-ctl -V '"ar0823 1-0010":0 [fmt:SGRBG12_1X12/3840x2160 field:none]'
    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
                    dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
                    sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
                    sink_0::pool-size=4 \
            ! queue max-size-buffers=1 leaky=0 \
            ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
                    sink_0::pool-size=4 src::pool-size=4 \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! queue max-size-buffers=1 leaky=0 \
            ! v4l2h265enc output-io-mode=dmabuf-import extra-controls=${ENC_EXTRA_CONTROLS} \
            ! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
            ! rtph265pay config-interval=1 pt=96 \
            ! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT}
    

    The main modifications are adding a queue between tiovxisp and tiovxldc, moving the other queue down below the capsfilter line, and adding output-io-mode=dmabuf-import to the vhl2h265enc line.

    With that, I get the following statistics:

    perf_stats output:

    Summary of CPU load,
    ====================
    
    CPU: mpu1_0: TOTAL LOAD =  27.81 % ( HWI =   1.50 %, SWI =   1.25 % )
    CPU:  c7x_1: TOTAL LOAD =   0. 2 % ( HWI =   0. 0 %, SWI =   0. 0 % )
    
    HWA performance statistics,
    ===========================
    
    HWA:   VISS: LOAD =  69.50 % ( 255 MP/s )
    HWA:   LDC : LOAD =  72.49 % ( 256 MP/s )
    
    DDR performance statistics,
    ===========================
    
    DDR: READ  BW: AVG =   2905 MB/s
    DDR: WRITE BW: AVG =   1775 MB/s
    DDR: TOTAL BW: AVG =   4680 MB/s
    
    SoC temperature statistics
    ==========================
    
    thermal_zone0(DDR):     72.94 degree Celsius
    thermal_zone1(CPU):     71.33 degree Celsius
    thermal_zone2(C7x):     72.74 degree Celsius

    parse_gst_tracers.py:

    +-----------------------------------------------------------------------------------+
    |element                       latency      out-latancy      out-fps     frames     |
    +-----------------------------------------------------------------------------------+
    |capsfilter0                   0.51         33.31            30          17184      |
    |tiovxisp0                     25.16        33.31            30          17183      |
    |queue0                        0.91         33.31            30          17183      |
    |tiovxldc0                     25.92        33.30            30          17182      |
    |capsfilter1                   0.65         33.30            30          17182      |
    |queue1                        0.59         33.30            30          17182      |
    |v4l2h265enc0                  31.15        33.30            30          17181      |
    |capsfilter2                   0.65         33.30            30          17181      |
    |v4l2src0                      86.80        33.30            30          17181      |
    |rtph265pay0                   1.27         33.30            30          17181      |
    +-----------------------------------------------------------------------------------+

    Thank you very much for the help! This gets me to 30 frames per second, and a latency that's really nice.

    Regards,

    Bas Vermeulen

  • I need to follow up on this; I am trying to use a tee to have two streams running from the same video source at the same time. When I do this, my framerate drops to 25 (when using tee) or 15 (when using tiovxmultiscaler).

    The script I use to test:

    #!/bin/bash
    
    MULTICASTADDR1=224.1.1.1
    MULTICASTADDR2=224.1.1.2
    PORT=5004
    
    TEE=tee
    HEVC_PROFILE="main"
    HEVC_LEVEL="1"
    VIDEO_BITRATE=6000000
    ENC_EXTRA_CONTROLS="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5,frame_level_rate_control_enable=1,video_bitrate_mode=0,vbv_buffer_size=3000,video_bitrate=${VIDEO_BITRATE}"
    
    TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:' -n '$0' -- "$@")
    
    if [ $? -ne 0 ]; then
            echo 'Terminating...' >&2
            exit 1
    fi
    
    eval set -- "$TEMP"
    unset TEMP
    
    while true; do
            case "$1" in
                    '-h'|'--help')
                            echo "$0 - H.265 compress a JPEG file into a one second stream"
                            echo "Parameters:"
                            echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
                            echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
                            echo "  -b 0..700000000 (Video Bitrate)"
                            echo "  --mc <multicast ip address>"
                            echo "  --port <port number> (default 5000)"
                            echo "  -h This help"
                            exit 0
                            shift
                            continue
                    ;;
                    '-p'|'--profile')
                            case "$2" in
                                    'main'|'Main'|'0')
                                            HEVC_PROFILE="main"
                                            ;;
                                    'main-still-picture'|'1')
                                            HEVC_PROFILE="main-still-picture"
                                            ;;
                                    'main-10'|'2')
                                            HEVC_PROFILE="main-10"
                                            ;;
                                    *)
                                            HEVC_PROFILE="main"
                                            ;;
                            esac
                            echo "HEVC Profile: '$HEVC_PROFILE'"
                            shift 2
                            continue
                    ;;
                    '-l'|'--level')
                            case "$2" in
                                    '1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
                                            HEVC_LEVEL="$2"
                                            ;;
                                    *)
                                            HEVC_LEVEL="1"
                                            ;;
                            esac
                            echo "HEVC Level: '$HEVC_LEVEL'"
                            shift 2
                            continue
                    ;;
                    '-b'|'--bitrate')
                            echo "Video Bitrate: '$2'"
                            VIDEO_BITRATE=$2
                            shift 2
                            continue
                    ;;
                    'mc')
                            MULTICASTADDR=$2
                            shift 2
                            continue
                    ;;
                    'port')
                            PORT=$2
                            shift 2
                            continue
                    ;;
                    '--')
                            shift
                            break
                    ;;
                    *)
                            echo 'Internal error!' >&2
                            exit 1
                    ;;
            esac
    done
    
    #media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
    #media-ctl -V '"ar0521 1-0036":0 [fmt:SGRBG8_1X8/3840x2160 field:none]'
    media-ctl -V '"ar0823 1-0010":0 [fmt:SGRBG12_1X12/3840x2160 field:none]'
    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
                    dcc-isp-file=/opt/imaging/ar0823/linear/dcc_viss.bin \
                    sink_0::dcc-2a-file=/opt/imaging/ar0823/linear/dcc_2a.bin format-msb=11 \
            ! queue max-size-buffers=1 leaky=0 \
            ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=/opt/imaging/ar0823/linear/dcc_ldc.bin \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! ${TEE} name=multi \
            multi. \
            ! queue max-size-buffers=1 leaky=0 name=qstream1 \
            ! v4l2h265enc output-io-mode=dmabuf-import extra-controls=${ENC_EXTRA_CONTROLS} \
            ! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
            ! rtph265pay config-interval=1 pt=96 \
            ! udpsink host=${MULTICASTADDR1} auto-multicast=true port=${PORT} \
            multi. \
            ! queue max-size-buffers=1 leaky=0 name=qstream2 \
            ! v4l2h265enc output-io-mode=dmabuf-import extra-controls=${ENC_EXTRA_CONTROLS} \
            ! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
            ! rtph265pay config-interval=1 pt=96 \
            ! udpsink host=${MULTICASTADDR2} auto-multicast=true port=${PORT}

    If you set the TEE variable in the script to tiovxmultiscaler, it will use the multiscaler, when set to tee it uses the tee element.

    My end goal is being able to dynamically change my pipeline from streaming to multiple fakesinks, to streaming to one or two network (UDP) sinks and a screen (all running at 30 fps).

    Removing the second branch of the tee will make the pipeline run at 30 fps.

    Any idea how to get this running at 30 fps with two or more branches?

    Regards,

    Bas Vermeulen

  • Hi Bas,

    If I understand correctly, what you are seeing is when you have a tee element added to the output to tiovxldc and stream two encoded streams you are seeing the FPS to be 25fps and if you don't use tee element and use tiovxmultiscaler, you are seeing the FPS to be 15. 

    Also can you share the tracers that you are seeing when using the script with tee element and multiscaler to see what element is causing the fps to drop?

    Best Regards,

    Suren

  • Hi Suren,

    Correct. If I drop the second encoder (and replace with a fakesink), I get back to 30 fps. If I use two encoders, I get 25 (tee) and 15 (tiovxmultiscaler).

    Output of parse_gst_tracers.py for the script with TEE=tee:

    +-----------------------------------------------------------------------------------+
    |element                       latency      out-latancy      out-fps     frames     |
    +-----------------------------------------------------------------------------------+
    |capsfilter0                   0.39         39.19            25          1033       |
    |tiovxisp0                     65.18        39.15            25          1031       |
    |queue0                        15.02        39.17            25          1031       |
    |tiovxldc0                     36.67        39.15            25          1030       |
    |capsfilter1                   0.55         39.15            25          1030       |
    |multi                         0.72         19.57            51          2060       |
    |qstream1                      0.53         39.15            25          1030       |
    |qstream2                      0.64         39.14            25          1030       |
    |v4l2h265enc1                  51.01        39.04            25          1028       |
    |capsfilter3                   0.65         39.04            25          1028       |
    |v4l2src0                      163.60       19.52            51          2057       |
    |rtph265pay1                   1.23         39.04            25          1028       |
    |v4l2h265enc0                  34.20        39.00            25          1029       |
    |capsfilter2                   0.64         39.00            25          1029       |
    |rtph265pay0                   1.23         39.00            25          1029       |
    +-----------------------------------------------------------------------------------+

    Output of parse_gst_tracers.py for the script with TEE=tiovxmultiscaler:

    +-----------------------------------------------------------------------------------+
    |element                       latency      out-latancy      out-fps     frames     |
    +-----------------------------------------------------------------------------------+
    |capsfilter0                   0.52         64.00            15          862        |
    |tiovxisp0                     111.86       63.98            15          860        |
    |queue0                        39.77        64.03            15          860        |
    |tiovxldc0                     25.57        64.00            15          859        |
    |capsfilter1                   0.72         64.00            15          859        |
    |multi                         36.01        31.97            31          1718       |
    |qstream1                      0.80         63.94            15          859        |
    |qstream2                      0.71         63.94            15          859        |
    |v4l2h265enc0                  32.39        63.82            15          858        |
    |capsfilter2                   0.70         63.82            15          858        |
    |v4l2src0                      258.53       31.92            31          1716       |
    |rtph265pay0                   1.46         63.82            15          858        |
    |v4l2h265enc1                  50.01        63.82            15          858        |
    |capsfilter3                   0.69         63.82            15          858        |
    |rtph265pay1                   1.39         63.82            15          858        |
    +-----------------------------------------------------------------------------------+

    I always find it difficult to judge what element is causing the extra latency.

    Regards,

    Bas Vermeulen

  • Hi Bas,

    Is it okay if you were to try the same experiment with 1080 resolution? I am suspecting two 4K stream encode would be an overkill on the system and causing frame drops.

    Best Regards,

    Suren

  • Hi Bas,

    Is it okay if you were to try the same experiment with 1080 resolution? I am suspecting two 4K stream encode would be an overkill on the system and causing frame drops.

    Best Regards,

    Suren

  • I have a pipeline running on the EVM (with the imx219 camera, and without LDC) where I have a 4K stream and a full HD stream running at the same time.

    #!/bin/bash
    
    MULTICASTADDR=224.1.1.1
    PORT=5000
    MULTICASTADDR1=224.2.2.1
    PORT1=5000
    
    HEVC_PROFILE="main"
    HEVC_LEVEL="1"
    VIDEO_BITRATE=0
    
    TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:,mc1:,port1:' -n '$0' -- "$@")
    
    if [ $? -ne 0 ]; then
            echo 'Terminating...' >&2
            exit 1
    fi
    
    eval set -- "$TEMP"
    unset TEMP
    
    while true; do
            case "$1" in
                    '-h'|'--help')
                            echo "$0 - H.265 compress a JPEG file into a one second stream"
                            echo "Parameters:"
                            echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
                            echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
                            echo "  -b 0..700000000 (Video Bitrate)"
                            echo "  --mc <multicast ip address>"
                            echo "  --port <port number> (default 5000)"
                            echo "  --mc1 <multicast ip address>"
                            echo "  --port1 <port number> (default 5000)"
                            echo "  -h This help"
                            exit 0
                            shift
                            continue
                    ;;
                    '-p'|'--profile')
                            case "$2" in
                                    'main'|'Main'|'0')
                                            HEVC_PROFILE="main"
                                            ;;
                                    'main-still-picture'|'1')
                                            HEVC_PROFILE="main-still-picture"
                                            ;;
                                    'main-10'|'2')
                                            HEVC_PROFILE="main-10"
                                            ;;
                                    *)
                                            HEVC_PROFILE="main"
                                            ;;
                            esac
                            echo "HEVC Profile: '$HEVC_PROFILE'"
                            shift 2
                            continue
                    ;;
                    '-l'|'--level')
                            case "$2" in
                                    '1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
                                            HEVC_LEVEL="$2"
                                            ;;
                                    *)
                                            HEVC_LEVEL="1"
                                            ;;
                            esac
                            echo "HEVC Level: '$HEVC_LEVEL'"
                            shift 2
                            continue
                    ;;
                    '-b'|'--bitrate')
                            echo "Video Bitrate: '$2'"
                            VIDEO_BITRATE=$2
                            shift 2
                            continue
                    ;;
                    'mc')
                            MULTICASTADDR=$2
                            shift 2
                            continue
                    ;;
                    'port')
                            PORT=$2
                            shift 2
                            continue
                    ;;
                    'mc1')
                            MULTICASTADDR1=$2
                            shift 2
                            continue
                    ;;
                    'port1')
                            PORT1=$2
                            shift 2
                            continue
                    ;;
                    '--')
                            shift
                            break
                    ;;
                    *)
                            echo 'Internal error!' >&2
                            exit 1
                    ;;
            esac
    done
    
    media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
    gst-launch-1.0 -q v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb10 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_SONY_IMX219_RPI" \
                    dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss_10b.bin \
                    sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a_10b.bin format-msb=9 \
            ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 \
            ! tiovxmultiscaler name=multi target=0 \
            multi.src_0 \
            ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 \
            ! mosaic.sink_0 \
            multi.src_1 \
            ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
            ! mosaic.sink_1 \
            multi.src_2 \
            ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
            ! mosaic.sink_2 \
            multi.src_3 \
            ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
            ! mosaic.sink_3 \
            multi.src_4 \
            ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
            ! queue max-size-buffers=1 leaky=0 \
            ! v4l2h265enc extra-controls="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5" \
            ! rtph265pay config-interval=1 pt=96 \
            ! udpsink host=${MULTICASTADDR1} auto-multicast=true port=${PORT1} \
            tiovxmosaic name=mosaic target=2 \
            sink_0::startx="<0>" sink_0::starty="<0>" \
            sink_1::startx="<1920>" sink_1::starty="<1080>" \
            sink_2::startx="<0>" sink_2::starty="<1080>" \
            sink_3::startx="<1920>" sink_3::starty="<0>" \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! queue max-size-buffers=1 leaky=0 \
            ! v4l2h265enc extra-controls="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5" \
            ! rtph265pay config-interval=1 pt=96 \
            ! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT}

    parse_gst_tracers.py output:

    +-----------------------------------------------------------------------------------+
    |element                       latency      out-latancy      out-fps     frames     |
    +-----------------------------------------------------------------------------------+
    |capsfilter0                   0.27         33.33            30          6594       |
    |tiovxisp0                     8.55         33.32            30          6601       |
    |capsfilter1                   0.38         33.32            30          6605       |
    |multi                         11.87        6.66             150         33040      |
    |capsfilter2                   0.37         33.32            30          6612       |
    |capsfilter3                   0.23         33.32            30          6618       |
    |capsfilter4                   0.21         33.32            30          6618       |
    |capsfilter5                   0.20         33.32            30          6618       |
    |capsfilter6                   0.21         33.32            30          6618       |
    |queue0                        0.27         33.32            30          6618       |
    |v4l2h265enc0                  27.46        33.31            30          6617       |
    |v4l2src0                      79.19        16.66            60          13232      |
    |rtph265pay0                   0.53         33.31            30          6617       |
    |mosaic                        40.56        33.31            30          6616       |
    |capsfilter7                   0.35         33.31            30          6620       |
    |queue1                        0.25         33.31            30          6621       |
    |v4l2h265enc1                  44.34        33.30            30          6622       |
    |rtph265pay1                   0.73         33.30            30          6623       |
    +-----------------------------------------------------------------------------------+

    The only difference with what I am trying now is that this pipeline doesn't have the LDC, and I am using the tiovxmultiscaler + tiovxmosaic to generate a 4K stream.

    That's why I don't really understand why everything is so slow when using the larger camera.

    The encoding should be fine. I just don't understand why I can't translate this to our phyboard with the 4K camera.

    Regards,

    Bas

  • Hi Bas,

    Let me try and run some pipelines on my AM62A board early next week and get back to you with the observations on my end.

    Have a great weekend!

    Best Regards,

    Suren

  • Hi ,

    Were you able to run some pipelines, and what were your observations?

    Regards,

    Bas Vermeulen

  • Hi Bas,

    I tried the below pipeline with 1080p and with videotestsrc. 

    [2024-10-09 16:32:43.589] root@am62axx-evm:/opt/edgeai-gst-apps# GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG=GST_TRACER:7 GST_DEBUG_FILE="/run/latency_4k.txt" \
    [2024-10-09 16:32:46.483] > gst-launch-1.0 -v videotestsrc ! \
    [2024-10-09 16:32:46.535] > video/x-raw, width=1920, height=1080, framerate=60/1, format=NV12 ! \
    [2024-10-09 16:32:46.635] > tee name=tee_split0 \
    [2024-10-09 16:32:46.687] > tee_split0. ! queue ! v4l2h265enc ! rtph265pay ! udpsink host=127.0.0.1 port=5001 \
    [2024-10-09 16:32:46.845] > tee_split0. ! queue ! v4l2h265enc ! rtph265pay ! udpsink host=127.0.0.1 port=6001 
    [2024-10-09 16:32:48.405] Setting pipeline to PAUSED ...
    +-----------------------------------------------------------------------------------+
    [2024-10-09 17:08:49.619] |element                       latency      out-latancy      out-fps     frames     |
    [2024-10-09 17:08:49.635] +-----------------------------------------------------------------------------------+
    [2024-10-09 17:08:49.636] |capsfilter0                   0.21         22.12            45          302        |
    [2024-10-09 17:08:49.651] |tee_split0                    0.88         11.06            90          604        |
    [2024-10-09 17:08:49.651] |queue0                        19.00        22.12            45          302        |
    [2024-10-09 17:08:49.667] |queue1                        21.27        22.12            45          302        |
    [2024-10-09 17:08:49.667] |v4l2h265enc1                  39.21        22.04            45          301        |
    [2024-10-09 17:08:49.683] |videotestsrc0                 61.15        11.02            90          601        |
    [2024-10-09 17:08:49.683] |rtph265pay1                   0.46         22.01            45          300        |
    [2024-10-09 17:08:49.699] |v4l2h265enc0                  39.47        21.98            45          301        |
    [2024-10-09 17:08:49.699] |rtph265pay0                   0.44         21.97            45          301        |
    [2024-10-09 17:08:49.700] +-----------------------------------------------------------------------------------+
    

    Not sure, why the tee element is receiving half of frames. Will try and experiment more on this. 

    Best Regards,

    Suren

  • Hi Bas,

    Continuing on running IMX219 with 4K resolution:

    Below pipeline:

    GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG=GST_TRACER:7 GST_DEBUG_FILE="/run/latency_4k-enc-1.txt" \
    gst-launch-1.0 v4l2src device=/dev/video-imx219-cam0 io-mode=dmabuf-import ! queue max-size-buffers=1 leaky=2 ! \
    video/x-bayer, width=3280, height=2464, framerate=15/1, format=rggb10 ! \
    tiovxisp sink_0::pool-size=4 sink_0::device=/dev/v4l-imx219-subdev0 sensor-name="SENSOR_SONY_IMX219_RPI" \
    dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss_10b.bin \
    sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a_10b.bin format-msb=9 ! \
    video/x-raw, format=NV12, width=3280, height=2464, framerate=15/1 ! queue ! tiovxmultiscaler ! queue ! \
    video/x-raw, format=NV12, width=1920, height=1080, framerate=15/1 ! \
    v4l2h264enc ! rtph264pay ! tee name=t1 \
    t1. ! queue ! udpsink host=127.0.0.1 port=5001 \
    t1. ! queue ! udpsink host=127.0.0.1 port=6001

    Best Regards,

    Suren

  • Hi Suren,

    Unfortunately the problem isn't visible when using 4K @ 15 fps. Do you have a possibility to test with 4K @ 30 fps?

    Regards,

    Bas

  • Hi Bas,

    Unfortunately, I don't have a camera that can run 4K@30fps. 

    Best Regards,

    Suren

  • Hi ,

    Did you get any insights from your team after our discussion?

    Regards,

    Bas

  • Hi Bas,

    Its Diwali week and most of the folks are on vacation. Will have an update next week, when they will be back in office.

    Apologies for the delay.

    Best Regards,

    Suren

  • Hi ,

    I hope you and your team had a nice Diwali week. Were you able to get any insights from your team after our discussion?

    Regards,

    Bas Vermeulen

  • Hi Bas,

    I briefly had a discussion and the expert thinks this could be due to networking.

    Can we try to dump the encoded streams into file instead of being streamed on network and do we see the same issue? 

    AFAIK, fakesink shows 30fps. is that correct?

    Best Regards,

    Suren

  • When using the fakesink I am not using the encoder. I can test with the fakesink after the RTP Payloader.

  • I've replaced udpsink on both paths with fakesink, and then I get 30 fps.

    Any idea what the best way is to tune the performance of the udpsink?

  • Hi Bas,

    We might have to do trial and error on the properties of udpsink. 

    Can you try putting sync=false with udpsink? Play with buffer-size parameter,   

    Also we can try playing with max-size-buffers, min-threshold-bytes of the queue element.

    Let me know how it goes.

    Best Regards,

    Suren

  • Hi Suren, sorry for the delay.

    I'm trying to recreate the two full paths with fakesink @ 30 fps, and seem to be failing there. The moment I reintroduce the encoder, the fps drops to 25, so it's possible that that is the limiting factor. I've experimented with adding a tiovxmultiscaler on the second stream, and this drops the latency for the encoder element down to below 33 ms, which should be fine. The framerate still drops to 25 though.

    I currently have scripts for the following scenarios:

    1. Pipeline with tee to two streams, one with udpsink and one with fakesink
      Sensor -> ISP -> LDC -> tee -> queue -> encoder -> rtp payloader -> udpsink to multicast address
                                                     |-> queue -> fakesink
      This pipeline gets 30 fps or close to it.
    2. Pipeline with tee to two streams, both with udpsink
      Sensor -> ISP -> LDC -> tee -> queue -> encoder -> rtp payloader -> udpsink to multicast address 1
                                                    |-> queue -> tiovxmultiscaler to 1920x1080 -> encoder -> rtp payloader -> udpsink to multicast address 2
      This pipeline gets 25-26 fps
    3. Pipeline with tee to two streams, with encoder and rtp payloader, to fakesinks
      Sensor -> ISP -> LDC -> tee -> queue -> encoder -> rtp payloader -> fakesink
                                                    |-> queue -> tiovxmultiscaler to 1920x1080 -> encoder -> rtp payloader -> fakesink
      This pipeline gets 25-26 fps
    4. Pipeline with tee to two streams, one with udpsink and one with fakesink
      Sensor -> ISP -> LDC -> tee -> queue -> encoder -> rtp payloader -> udpsink to multicast address
                                                     |-> queue -> tiovxmultiscaler to 1920x1080 -> fakesink
      This pipeline gets 26 fps or close to it.
    5. Pipeline with tee to two streams, both with udpsink
      Sensor -> ISP -> LDC -> tee -> queue -> encoder -> rtp payloader -> udpsink to multicast address 1
                                                    |-> queue -> encoder -> rtp payloader -> udpsink to multicast address 2
      This pipeline gets 25 fps

    I'm unsure what causes the drop in framerate. What annoys me is that I'm sure I tried things correctly before, but apparently that was wrong. The moment I use a serious element behind the second streams queue, my framerate drops to around 25 fps.

    Using a single multiscaler to lower the resolution of the second stream helps a little bit.

    I've tried increasing the max-size-buffers on the queue, but that didn't help (from 1 to 4). I don't want to increase those if possible, as that will add to the latency of my system.

    Regards,

    Bas Vermeulen

  • Hi Bas

    Sensor -> ISP -> LDC -> tee -> queue -> encoder -> rtp payloader -> udpsink to multicast address 1
                                                  |-> queue -> tiovxmultiscaler to 1920x1080 -> encoder -> rtp payloader -> udpsink to multicast address 2

    In the above scenario, you are streaming a 4K encoded stream  and the other one FHD correct? 

    What if you use the output of LDC-> Multiscaler to FHD-> Encode -> tee -> queue -> udpstream1

                                                                                                                         -> queue -> udpstream2 

    Does this work with no drops in FPS? I am suspecting 4K + FHD both @30fps would be a stretch.. I will try to run the pipeline with 5MP resolution and FHD next week and update you with the results.

    Best Regards,

    Suren

  • That gets me 24 frames per second.

    +-----------------------------------------------------------------------------------+
    |element                       latency      out-latancy      out-fps     frames     |
    +-----------------------------------------------------------------------------------+
    |capsfilter0                   0.41         41.05            24          749        |
    |tiovxisp0                     68.63        40.98            24          747        |
    |queue0                        16.84        41.00            24          746        |
    |tiovxldc0                     39.73        40.98            24          745        |
    |capsfilter1                   0.54         40.98            24          745        |
    |queue1                        17.07        41.00            24          745        |
    |tiovxmultiscaler0             39.37        40.98            24          744        |
    |capsfilter2                   0.68         40.98            24          744        |
    |queue2                        0.53         40.98            24          744        |
    |multi                         1.07         20.49            48          1488       |
    |qstream1                      0.68         40.98            24          744        |
    |qstream2                      0.72         40.98            24          744        |
    |v4l2h265enc0                  19.65        40.87            24          744        |
    |capsfilter3                   0.58         40.87            24          744        |
    |v4l2src0                      210.66       20.45            48          1487       |
    |rtph265pay0                   1.08         40.87            24          744        |
    |v4l2h265enc1                  27.26        40.87            24          743        |
    |capsfilter4                   0.52         40.87            24          743        |
    |rtph265pay1                   1.05         40.87            24          743        |
    +-----------------------------------------------------------------------------------+

    The script I used:

    #!/bin/bash
    
    DCC_ISP_FILE=/opt/imaging/ar0823/linear/dcc_viss.bin
    DCC_2A_FILE=/opt/imaging/ar0823/linear/dcc_2a.bin
    LDC_DCC_FILE=/opt/imaging/ar0823/linear/dcc_ldc.bin
    
    MULTICASTADDR1=224.1.1.1
    MULTICASTADDR2=224.1.1.2
    PORT=5004
    
    if [ x$TEE == x ]; then
        TEE=tee
    fi
    HEVC_PROFILE="main"
    HEVC_LEVEL="1"
    VIDEO_BITRATE=6000000
    ENC_EXTRA_CONTROLS="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5,frame_level_rate_control_enable=1,video_bitrate_mode=0,vbv_buffer_size=3000,video_bitrate=${VIDEO_BITRATE}"
    
    TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:' -n '$0' -- "$@")
    
    if [ $? -ne 0 ]; then
            echo 'Terminating...' >&2
            exit 1
    fi
    
    eval set -- "$TEMP"
    unset TEMP
    
    while true; do
            case "$1" in
                    '-h'|'--help')
                            echo "$0 - H.265 compress a JPEG file into a one second stream"
                            echo "Parameters:"
                            echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
                            echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
                            echo "  -b 0..700000000 (Video Bitrate)"
                            echo "  --mc <multicast ip address>"
                            echo "  --port <port number> (default 5000)"
                            echo "  -h This help"
                            exit 0
                            shift
                            continue
                    ;;
                    '-p'|'--profile')
                            case "$2" in
                                    'main'|'Main'|'0')
                                            HEVC_PROFILE="main"
                                            ;;
                                    'main-still-picture'|'1')
                                            HEVC_PROFILE="main-still-picture"
                                            ;;
                                    'main-10'|'2')
                                            HEVC_PROFILE="main-10"
                                            ;;
                                    *)
                                            HEVC_PROFILE="main"
                                            ;;
                            esac
                            echo "HEVC Profile: '$HEVC_PROFILE'"
                            shift 2
                            continue
                    ;;
                    '-l'|'--level')
                            case "$2" in
                                    '1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
                                            HEVC_LEVEL="$2"
                                            ;;
                                    *)
                                            HEVC_LEVEL="1"
                                            ;;
                            esac
                            echo "HEVC Level: '$HEVC_LEVEL'"
                            shift 2
                            continue
                    ;;
                    '-b'|'--bitrate')
                            echo "Video Bitrate: '$2'"
                            VIDEO_BITRATE=$2
                            shift 2
                            continue
                    ;;
                    'mc')
                            MULTICASTADDR=$2
                            shift 2
                            continue
                    ;;
                    'port')
                            PORT=$2
                            shift 2
                            continue
                    ;;
                    '--')
                            shift
                            break
                    ;;
                    *)
                            echo 'Internal error!' >&2
                            exit 1
                    ;;
            esac
    done
    
    #media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
    #media-ctl -V '"ar0521 1-0036":0 [fmt:SGRBG8_1X8/3840x2160 field:none]'
    media-ctl -V '"ar0823 1-0010":0 [fmt:SGRBG12_1X12/3840x2160 field:none]'
    gst-launch-1.0 -v -e v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
            ! video/x-bayer, width=3840, height=2160, framerate=30/1, format=grbg12 \
            ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0823" \
                    dcc-isp-file=${DCC_ISP_FILE} \
                    sink_0::dcc-2a-file=${DCC_2A_FILE} format-msb=11 \
            ! queue max-size-buffers=1 leaky=0 \
            ! tiovxldc sensor-name="SENSOR_ONSEMI_AR0823" dcc-file=${LDC_DCC_FILE} \
            ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
            ! queue max-size-buffers=1 leaky=0 \
            ! tiovxmultiscaler \
            ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 \
            ! queue max-size-buffers=1 leaky=0 \
            ! ${TEE} name=multi \
            multi.src_0 \
            ! queue max-size-buffers=1 leaky=0 name=qstream1 \
            ! v4l2h265enc output-io-mode=dmabuf-import extra-controls=${ENC_EXTRA_CONTROLS} \
            ! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
            ! rtph265pay config-interval=1 pt=96 mtu=1400 \
            ! udpsink host=${MULTICASTADDR1} auto-multicast=true port=${PORT} \
            multi.src_1 \
            ! queue max-size-buffers=1 leaky=0 name=qstream2 \
            ! v4l2h265enc output-io-mode=dmabuf-import extra-controls=${ENC_EXTRA_CONTROLS} \
            ! "video/x-h265, profile=(string)${HEVC_PROFILE}, level=(string)${HEVC_LEVEL}" \
            ! rtph265pay config-interval=1 pt=96 mtu=1400 \
            ! udpsink host=${MULTICASTADDR2} auto-multicast=true port=${PORT}
    

    Regards,

    Bas Vermeulen

  • Hi Suren,

    Would it help if we send you one of our camera setups? That would be a phyboard + am62a SOM, a camera board and an assembly to hold everything together. Add ethernet, a micro usb cable for serial port access and a USB C power cable for power.

    The software is based on SDK 9.2.0, and would be included.

    Let me know if this would help you in your investigation of the issue.

    Regards,

    Bas Vermeulen

  • On another note, with the imx219 4K + FHD stream, I stream both streams at the same time with the udpsink. So the networking should not have a problem with the amount of data that is being sent.

    The same pipeline also shows that the encoder is able to encode 3840x2160 and 1920x1080 at the same time; that's part of the reason I wanted that tested, so that I could be sure there wasn't a bottleneck in that area.

    I'm just not sure where the bottleneck is currently.

  • Hi Bas,

    Sent you a private message. 

    Best Regards,

    Suren

  • Hi Suren,

    Any progress on this, and is there anything I can do to help your testing? This is a high priority at our end, and if possible we would like to resolve before the end of the year.

    Regards,

    Bas Vermeulen

  • Hi Bas,

    Since it was a Thanksgiving week, had no chance to work on this. Just wanted to let you know we have received the package at our dock. 

    Jianzhong, my colleague will try and run the setup and will reach out to you soon.

    Best Regards,

    Suren