• State TI Thinks Resolved
  • Locked Locked
  • Replies 5 replies
  • Answers 2 answers
  • Subscribers 98 subscribers
  • Views 436 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Original question:

AM5728: Streaming video using gstreamer

AM5728: Multiple pipelines in gstreamer

Vishal Maheshwari
Expert 1020 points
Part Number: AM5728

Tool/software:

Hi,

I have a custom board based on AM5728 SOC with omnivision 0v5640 camera installed on it running TI Linux SDK -  6.03 .

I want to support maximum 4 party conference streams using gstreamer.

POC done for three party conference streaming is as follows:-

Three custom boards namely A, B and C were taken.

On A, a gstreamer pipeline was run to open self camera feed and its self video and audio streams were sent to both B and C. B and C were able to receive the video and audio streams as receiving pipelines were run on each.
Also, on B and C pipelines were run to send their respective video and audio streams to A.

It is needed to display the feeds of both B and C at A. How can that be achieved?

The gstreamer pipelines used are as below:-

######################
Sender Pipeline at A
######################

B=192.168.8.139 #(IP address of party B)
C=192.168.8.140 #(IP address of party C)

A_BVidSendPort=5000
A_BAudSendPort=5001
A_CVidSendPort=5002
A_CAudSendPort=5003

gst-launch-1.0 -e v4l2src device=/dev/video1 io-mode=4 ! \
'video/x-raw,format=(string)YUY2,width=1280,height=720,framerate=(fraction)30/1' ! \
vpe num-input-buffers=8 ! \
queue ! ducatih264enc intra-interval=1 ! \
h264parse ! \
rtph264pay mtu=200 ! \
multiudpsink clients=$B:$A_BVidSendPort,$C:$A_CVidSendPort alsasrc device='hw:0,0' ! \
audioconvert ! \
'audio/x-raw,channels=1,depth=16,width=16,rate=16000,encoding-name=(string)L16' ! \
rtpL16pay ! \
multiudpsink clients=$B:$A_BAudSendPort,$C:$A_CAudSendPort sync=false

######################
Receiver Pipeline at A
######################

#TBD

#####################
Sender Pipeline at B
#####################

B=192.168.8.138 # (IP address of party A)
A=192.168.8.139 # (IP address of party B)

A_BVidSendPort=5004
A_BAudSendPort=5005

gst-launch-1.0 -e v4l2src device=/dev/video1 io-mode=4 ! \
'video/x-raw,format=(string)YUY2,width=1280,height=720,framerate=(fraction)30/1' ! \
vpe num-input-buffers=8 ! \
queue ! ducatih264enc intra-interval=1 ! \
h264parse ! \
rtph264pay mtu=200 ! \
udpsink host=$B port=$A_BVidSendPort alsasrc device='hw:0,0' ! \
audioconvert ! \
'audio/x-raw,channels=1,depth=16,width=16,rate=16000,encoding-name=(string)L16' ! \
rtpL16pay ! \
udpsink host=$B port=$A_BAudSendPort sync=false

########################
#Receiver Pipeline at B
########################

B_AVidSendPort=5000
B_AAudSendPort=5001

gst-launch-1.0 -v udpsrc port=$B_AVidSendPort ! \
'application/x-rtp,media=(string)video, clock-rate=(int)90000, payload=(int)96' ! \
rtph264depay ! \
h264parse ! \
ducatih264dec ! \
vpe ! \
'video/x-raw,format=(string)NV12, width=1280,height=1232' ! \
videoscale sharpen=1 ! \
kmssink sync=false \
udpsrc port=$B_AAudSendPort ! \
"application/x-rtp,media=(string)audio,clock-rate=(int)16000, width=16, height=16,\
encoding-name=(string)L16,\
encoding-params=(string)1, channels=(int)1, channel-positions=(int)1,\
payload=(int)96" !\
rtpL16depay ! audioconvert ! alsasink sync=false

#######################
#Sender pipeline at C
######################

B=192.168.8.138 # (IP address of party A)
A=192.168.8.140 # (IP address of party C)

A_BVidSendPort=5006
A_BAudSendPort=5007

gst-launch-1.0 -e v4l2src device=/dev/video1 io-mode=4 ! \
'video/x-raw,format=(string)YUY2,width=1280,height=720,framerate=(fraction)30/1' ! \
vpe num-input-buffers=8 ! \
queue ! ducatih264enc intra-interval=1 ! \
h264parse ! \
rtph264pay mtu=200 ! \
udpsink host=$B port=$A_BVidSendPort alsasrc device='hw:0,0' ! \
audioconvert ! \
'audio/x-raw,channels=1,depth=16,width=16,rate=16000,encoding-name=(string)L16' ! \
rtpL16pay ! \
udpsink host=$B port=$A_BAudSendPort sync=false

########################
#Receiver pipeline at C
#######################

B_AVidSendPort=5002
B_AAudSendPort=5003

gst-launch-1.0 -v udpsrc port=$B_AVidSendPort ! \
'application/x-rtp,media=(string)video, clock-rate=(int)90000, payload=(int)96' ! \
rtph264depay ! \
h264parse ! \
ducatih264dec ! \
vpe ! \
'video/x-raw,format=(string)NV12, width=1280,height=1232' ! \
videoscale sharpen=1 ! \
kmssink sync=false \
udpsrc port=$B_AAudSendPort ! \
"application/x-rtp,media=(string)audio,clock-rate=(int)16000, width=16, height=16,\
encoding-name=(string)L16,\
encoding-params=(string)1, channels=(int)1, channel-positions=(int)1,\
payload=(int)96" !\
rtpL16depay ! audioconvert ! alsasink sync=false

This is the POC for 3 party conference streaming. Since there are multiple piplelines already in 3 party POC, 4 party conference will have furthermore pipelines added to these.
Wouldn’t it affect the system performance as the number of pipelines are increased?
Also can the sender and receiver pipelines be improved using gstreamer’s mux feature?

  • 0
    TI__Mastermind 19680 points

    Hi Vishal,

    As mentioned previously, support for SDK versions this old is very limited, especially for multimedia since it has since been de-scoped and the engineers that worked on this are no longer on the team. Unfortunately, we have no tutorials on how to achieve multicam stream via UDP for this system so I would recommend looking through GStreamer tutorials on how to achieve this.

    Tutorials for performance:

    https://gstreamer.freedesktop.org/documentation/additional/design/latency.html?gi-language=c

    https://gstreamer.freedesktop.org/documentation/additional/design/tracing.html?gi-language=c

    If you would like more TI support on this project, I recommend taking a look at our newer devices such as AM68A since we have several out of box demos that achieve the tasks you are looking for with TIOVX Streamer plugins such as tiovxmosiac: https://github.com/TexasInstruments/edgeai-gst-plugins/wiki/tiovxmosaic#pipeline-examples

    Thank you,

    Fabiana

  • 0 in reply to Fabiana Jaimes
    Expert 1020 points

    Hi,

    For three party streaming we ran the following pipelines on Ubuntu PCs (running on linux shell):-

    #sender pipeline at PC - B
    ########################

    gst-launch-1.0 \
    v4l2src ! video/x-raw,framerate=30/1 ! videoconvert ! x264enc ! rtph264pay ! udpsink host=192.168.21.26 port=5000

    #sender pipeline at PC - C
    #########################

    gst-launch-1.0 \
    v4l2src ! video/x-raw,framerate=30/1 ! videoconvert ! x264enc ! rtph264pay ! udpsink host=192.168.21.26 port=5001

    #Receiver pipeline at PC - A (IP Address :- 192.168.21.26)

    ############################

    gst-launch-1.0 compositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=640 sink_1::ypos=0 ! videoconvert ! autovideosink udpsrc port=5000 ! application/x-rtp ! rtph264depay ! avdec_h264 ! videoconvert ! video/x-raw ! queue ! comp.sink_0 udpsrc port=5001 ! application/x-rtp ! rtph264depay ! avdec_h264 ! videoconvert ! video/x-raw ! queue ! comp.sink_1

    "PC - A" was able to successfully receive the video streams of "PC - B" and "PC - C".

    Similarly, we tried to achieve the same result on our custom boards with the following pipelines (running on linux shell):-

    #sender pipeline at custom board B
    ###################################

    B=192.168.21.159

    A_BVidSendPort=5000

    gst-launch-1.0 -e v4l2src device=/dev/video1 io-mode=4 ! 'video/x-raw,format=(string)YUY2,width=1280,height=720,framerate=(fraction)30/1' ! vpe num-input-buffers=8 ! queue ! ducatih264enc intra-interval=1 ! h264parse ! rtph264pay mtu=200 ! udpsink host=$B port=$A_BVidSendPort

    #sender pipeline at custom board C
    ##################################

    B=192.168.21.159

    A_BVidSendPort=5001

    gst-launch-1.0 -e v4l2src device=/dev/video1 io-mode=4 ! 'video/x-raw,format=(string)YUY2,width=1280,height=720,framerate=(fraction)30/1' ! vpe num-input-buffers=8 ! queue ! ducatih264enc intra-interval=1 ! h264parse ! rtph264pay mtu=200 ! udpsink host=$B port=$A_BVidSendPort


    #Receiver pipeline at custom board A (IP Address :- 192.168.21.159)
    ##################################################################

    GST_DEBUG=3 gst-launch-1.0 compositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=640 sink_1::ypos=0 ! videoconvert ! autovideosink udpsrc port=5000 ! application/x-rtp ! rtph264depay ! ducatih264dec ! videoconvert ! video/x-raw ! queue ! comp.sink_0 udpsrc port=5001 ! application/x-rtp ! rtph264depay ! ducatih264dec ! videoconvert ! video/x-raw ! queue ! comp.sink_1

    Sender pipelines on both custom board "B" and "C" ran fine, but the receiver pipeline on custom board "A" failed with the error as shown below:-

    Setting pipeline to PAUSED ...
    0:00:00.744068265 1527 0x1cc320 ERROR ducati gstducatividdec.c:1645:gst_ducati_viddec_change_state:<ducatih264dec1> Failed to create codec ivahd_h264dec, not supported
    ERROR: Pipeline doesn't want to pause.
    Setting pipeline to NULL ...
    Freeing pipeline ...

    So basically the ducatih264dec is having issues with compositor.

    How can this be solved?

  • 0 in reply to Vishal Maheshwari
    TI__Mastermind 19680 points

    Are you using the same GStreamer versions between Ubuntu PC and custom board?

    Thank you,

    Fabiana

  • 0 in reply to Fabiana Jaimes
    Expert 1020 points

    Hi ,

    On PC the gstreamer version is - GStreamer 1.24.2

    On 6.03 Linux SDK for AM5728, version is GStreamer 1.14.4

    But clearly, the issue seems to be with IVAHD h264 decoder!

    Basically, I want to use IVAHD core for video encoding and decoding. If h264 is not supported on it then what other encoder can be used?

  • 0 in reply to Vishal Maheshwari
    TI__Mastermind 19680 points

    Hi Vishal,

    Unfortunately, the SDK version 6.03 is very old and is not actively supported. If there are any bugs or issues with multimedia any AM57x SDK release, there is nothing we can do to debug or patch due to this being de-scoped as previously stated.

    Thank you for understanding,

    Fabiana