AM62A7-Q1: Encoder limitation

Hi Biao,

I used the following script to test this with the imx219 RPi camera:

#!/bin/bash

MULTICASTADDR=224.1.1.1
PORT=5000
MULTICASTADDR1=224.2.2.1
PORT1=5000

HEVC_PROFILE="main"
HEVC_LEVEL="1"
VIDEO_BITRATE=0

TEMP=$(getopt -o 'p:l:b:h' --long 'profile:,level:,bitrate:,input:,output:,help,mc:,port:,mc1:,port1:' -n '$0' -- "$@")

if [ $? -ne 0 ]; then
        echo 'Terminating...' >&2
        exit 1
fi

eval set -- "$TEMP"
unset TEMP

while true; do
        case "$1" in
                '-h'|'--help')
                        echo "$0 - H.265 compress a JPEG file into a one second stream"
                        echo "Parameters:"
                        echo "  -p main|main-still-picture|main-10 (HEVC Profile)"
                        echo "  -l 1|2|2.1|3|3.1|4|4.1|5|5.1 (HEVC Level)"
                        echo "  -b 0..700000000 (Video Bitrate)"
                        echo "  --mc <multicast ip address>"
                        echo "  --port <port number> (default 5000)"
                        echo "  --mc1 <multicast ip address>"
                        echo "  --port1 <port number> (default 5000)"
                        echo "  -h This help"
                        exit 0
                        shift
                        continue
                ;;
                '-p'|'--profile')
                        case "$2" in
                                'main'|'Main'|'0')
                                        HEVC_PROFILE="main"
                                        ;;
                                'main-still-picture'|'1')
                                        HEVC_PROFILE="main-still-picture"
                                        ;;
                                'main-10'|'2')
                                        HEVC_PROFILE="main-10"
                                        ;;
                                *)
                                        HEVC_PROFILE="main"
                                        ;;
                        esac
                        echo "HEVC Profile: '$HEVC_PROFILE'"
                        shift 2
                        continue
                ;;
                '-l'|'--level')
                        case "$2" in
                                '1'|'2'|'2.1'|'3'|'3.1'|'4'|'4.1'|'5'|'5.1')
                                        HEVC_LEVEL="$2"
                                        ;;
                                *)
                                        HEVC_LEVEL="1"
                                        ;;
                        esac
                        echo "HEVC Level: '$HEVC_LEVEL'"
                        shift 2
                        continue
                ;;
                '-b'|'--bitrate')
                        echo "Video Bitrate: '$2'"
                        VIDEO_BITRATE=$2
                        shift 2
                        continue
                ;;
                'mc')
                        MULTICASTADDR=$2
                        shift 2
                        continue
                ;;
                'port')
                        PORT=$2
                        shift 2
                        continue
                ;;
                'mc1')
                        MULTICASTADDR1=$2
                        shift 2
                        continue
                ;;
                'port1')
                        PORT1=$2
                        shift 2
                        continue
                ;;
                '--')
                        shift
                        break
                ;;
                *)
                        echo 'Internal error!' >&2
                        exit 1
                ;;
        esac
done

media-ctl -V '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
gst-launch-1.0 -q v4l2src device=/dev/video3 io-mode=dmabuf-import do-timestamp=true \
        ! video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb10 \
        ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_SONY_IMX219_RPI" \
                dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss_10b.bin \
                sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a_10b.bin format-msb=9 \
        ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 \
        ! tiovxmultiscaler name=multi target=0 \
        multi.src_0 \
        ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 \
        ! mosaic.sink_0 \
        multi.src_1 \
        ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
        ! mosaic.sink_1 \
        multi.src_2 \
        ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
        ! mosaic.sink_2 \
        multi.src_3 \
        ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
        ! mosaic.sink_3 \
        multi.src_4 \
        ! video/x-raw, format=NV12,width=1920, height=1080, framerate=30/1 \
        ! queue max-size-buffers=1 leaky=0 \
        ! v4l2h265enc extra-controls="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5" \
        ! rtph265pay config-interval=1 pt=96 \
        ! udpsink host=${MULTICASTADDR1} auto-multicast=true port=${PORT1} \
        tiovxmosaic name=mosaic target=2 \
        sink_0::startx="<0>" sink_0::starty="<0>" \
        sink_1::startx="<1920>" sink_1::starty="<1080>" \
        sink_2::startx="<0>" sink_2::starty="<1080>" \
        sink_3::startx="<1920>" sink_3::starty="<0>" \
        ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 \
        ! queue max-size-buffers=1 leaky=0 \
        ! v4l2h265enc extra-controls="enc,prepend_sps_and_pps_to_idr=1,video_gop_size=5" \
        ! rtph265pay config-interval=1 pt=96 \
        ! udpsink host=${MULTICASTADDR} auto-multicast=true port=${PORT}

This script receives a 2MP (Full HD) video from the camera, runs it through the ISP, passes it to the multiscaler, and then uses the multiscaler to generate 5 streams with the same resolution. Four of those streams are passed to the tiovxmosaic element, and are combined into a single 3840x2160 (8 MP) stream. The fifth stream is used as a second Full HD stream. Both streams are passed into the encoder, and then passed to the RTP payloader and an udp sink.

This runs at 30 fps at full speed, and in my opinion shows that the encoder can handle a 8M*30fps and a 2M*30fps stream.

Please note however that I am running into trouble when running a similar setup with an 8MP sensor, where I use a tee to duplicate the 8M stream, and scale the second one down to full HD. I'm not sure if you would run into the same bottlenecks that I do (and I'm not sure yet what the bottlenecks are).

Hi Bas,

Thanks for the test, I said this is bottleneck because of the theory calculation, 8M*30+2M*30 require 300MP/s encoder bandwidth, but AM62A only support 240MP/s, correct me if my theory calculation is wrong.

BR,

Biao 

Hi Bas,

Sure, thanks for support.

BR,

Biao

Hi Bas, Suren,

Thanks for your strong support, customer new requirement for the encode is 4K@60fps, from the TRM, we can support, but from the datasheet we can't, Previously Bas help run 5*2M@30fps, I can confirm we can support 8+2m@30FPS. can you help double confirm we can support 4K@60fps?

BR,

Biao 

Hi Biao Li, 

If its AM62A, the clock at which VPU is operating is 400MHz and with this we should only be able to do 4K@30fps reliably.You can definitely stretch the encoder to do 4K@60fps, but can't guarantee if there are frame drops/latency or any other system related issues.

Best Regards,

Suren

Hi Suren,

can you help just run some test to double confirm this? just like Bas, run the 8*2M*30fps to instead.

BR,

Biao 

To give the VPU some additional oomph, you can change the clock frequency to 500 MHz. I believe the TRM is using 500 MHz to do it's calculations regarding 4K@60fps. You can set the clock speed with k3conf for a quick check, and you can change the clock speed in the device tree if this is something you want to have at every boot.

Regards,

Bas

Hi Bas,

can you help run this test 8*2M*30fps again? I don't have the camera to do this test. Thanks a lot for your strong support.

BR,

Biao 

Hi Biao Li,

I ran this below pipeline on my AM62A board (1920x1080 @ 30fps - 5 Encode streams ) 

Transmitter:

gst-launch-1.0 -v \
v4l2src device=/dev/video-imx219-cam0 io-mode=5 ! queue leaky=2 ! video/x-bayer, width=1920, height=1080, format=rggb ! \
tiovxisp sensor-name=SENSOR_SONY_IMX219_RPI dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss.bin format-msb=7 sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a.bin sink_0::device=/dev/v4l-imx219-subdev0 ! \
video/x-raw, format=NV12 ! tee name=t1 \
t1. ! queue ! v4l2h264enc ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5000 sync=false \
t1. ! queue ! v4l2h264enc ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5001 sync=false \
t1. ! queue ! v4l2h264enc ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5002 sync=false \
t1. ! queue ! v4l2h264enc ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5003 sync=false \
t1. ! queue ! v4l2h264enc ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5004 sync=false 

Receiver:

gst-launch-1.0 -v udpsrc port=5000 ! 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96' ! \
rtph264depay ! h264parse ! v4l2h264dec ! kmssink driver-name=tidss sync=false
gst-launch-1.0 -v udpsrc port=5001 ! 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96' ! \
rtph264depay ! h264parse ! v4l2h264dec ! kmssink driver-name=tidss sync=false
gst-launch-1.0 -v udpsrc port=5002 ! 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96' ! \
rtph264depay ! h264parse ! v4l2h264dec ! kmssink driver-name=tidss sync=false
gst-launch-1.0 -v udpsrc port=5003 ! 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96' ! \
rtph264depay ! h264parse ! v4l2h264dec ! kmssink driver-name=tidss sync=false
gst-launch-1.0 -v udpsrc port=5004 ! 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96' ! \
rtph264depay ! h264parse ! v4l2h264dec ! kmssink driver-name=tidss sync=false

Had no issues running 5 streams of encoding with VPU running at  400MHz

Hope this helps

Best Regards,

Suren

Hi Suren,

can you help try 8*1920*1080*30fps? customer support need is 4k@60fps.

BR,

Biao 

Hi Biao Li, 

Did the customer try changing the clock frequency to 500MHz using k3conf and give it a try to encode 4K@60FPS?

Best Regards,

Suren

Hi Suren,

I think this is mainly relate in spec or overs spec issue. if customer change the frequency to 500MHz, it can work even 4k@60fps, can we support customer to Mass production? I want to get some confirmation from BU team (may be system and design team or validation team), what is the enc/dec max limitation we can provide to customer? 

BR,

Biao

Hi Biao Li,

Can you start an email conversation with the BU team?

Best Regards,

Suren