AM62A7: gstreamer pipeline causing issue after few iteration

Part Number: AM62A7

Tool/software:

Hi,

We are using AM62A7 processor with our Camera sensor. When we are using the pipeline with tiovxmultiscaler plugin, pipeline stops when closing the it after few start and stop of pipeline. Below is pipeline:

gst-launch-1.0 v4l2src device=/dev/video-rpi-cam0 io-mode=dmabuf-import ! video/x-bayer, width=1920, height=1200, format=rggb ! tiovxisp sink_0::device=/dev/v4l-rpi-subdev0 dcc-isp-file=/opt/imaging/ar0235/dcc_viss_8bit_1920x1200.bin sink_0::dcc-2a-file=/opt/imaging/ar0235/dcc_2a_8bit_1920x1200.bin format-msb=9 sink_0::ae-mode=2 ! video/x-raw, format=NV12, width=1920, height=1200 ! tiovxmultiscaler name=split split. ! video/x-raw, format=NV12, width=960, height=600 ! queue ! textoverlay text="foo" ! videorate ! video/x-raw,framerate=30/1 ! v4l2h264enc extra-controls=\"c,video_gop_size=30,h264_profile=4,video_bitrate=90000,h264_level=9,h264_i_frame_period=512,prepend_sps_and_pps_to_idr=1\" ! video/x-h264,profile=high !  mpegtsmux ! udpsink host=127.0.0.1 port=6310 split. ! video/x-raw, format=NV12, width=1920, height=1200 ! queue ! textoverlay text="bar" ! videorate ! video/x-raw,framerate=30/1 ! v4l2h264enc extra-controls=\"c,video_gop_size=30,h264_profile=4,video_bitrate=90000,h264_level=9,h264_i_frame_period=512,prepend_sps_and_pps_to_idr=1\" ! video/x-h264,profile=high !  mpegtsmux ! udpsink host=127.0.0.1 port=6320 split. ! video/x-raw, format=NV12, width=1920, height=1200 ! queue ! textoverlay text="baz" ! v4l2jpegenc ! rtpjpegpay ! udpsink host=127.0.0.1 port=6330
 

After few iteration, below error is received and pipeline hangs at this point. 

APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
 69920.128897 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
 69920.129015 s:  VX_ZONE_INIT:Enabled
 69920.129029 s:  VX_ZONE_ERROR:Enabled
 69920.129041 s:  VX_ZONE_WARNING:Enabled
 69920.129992 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
 69920.130152 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
Redistribute latency...
Redistribute latency...
IttCtrl_registerHandler: command echo registered at location 0
IttCtrl_registerHandler: command iss_read_2a_params registered at location 1
IttCtrl_registerHandler: command iss_write_2a_params registered at location 2
IttCtrl_registerHandler: command iss_raw_save registered at location 3
IttCtrl_registerHandler: command iss_yuv_save registered at location 4
IttCtrl_registerHandler: command iss_read_sensor_reg registered at location 5
IttCtrl_registerHandler: command iss_write_sensor_reg registered at location 6
IttCtrl_registerHandler: command dev_ctrl registered at location 7
IttCtrl_registerHandler: command iss_send_dcc_file registered at location 8
Error: failed to open i2c bus at /dev/i2c-9
Warning: Failed to initialize i2c bus. Register read/write will not work !!!
 NETWORK: Opened at IP Addr = 172.25.3.37, socket port=5000!!!
Redistribute latency...
handling interrupt.9.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:09.752114548
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0: Failed to allocate required memory.
Additional debug info:
../gst-plugins-good-1.20.7/sys/v4l2/gstv4l2videoenc.c(849): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
Buffer pool activation failed
An error happened while waiting for EOS
ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc1: Failed to allocate required memory.
Additional debug info:
../gst-plugins-good-1.20.7/sys/v4l2/gstv4l2videoenc.c(849): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc1:
Buffer pool activation failed
An error happened while waiting for EOS

Looks like pipeline didn't get EOS signal and still waiting for EOS signal. When this error occurs, RAM and CMA memory are enough to run another pipeline.

please let us know how can we correct this?

Regards,

Jay

  • Hi Jay,

    Please write a gstreamer application that handles EOS signal before starting the pipeline and you also need to verify the states of the elements/pipelines are maintained correctly. 

    Command line usage is not reliable in testing the scenario. 

    Also a simple pipeline with Camera capture and Video Encode scenario without any tiovxmultiscaler, are you seeing the issue?

    Best Regards,

    Suren

  • Hi Suren,

    Thank you for your reply.

    Without tiovxmultiscaler, Simple pipeline works fine without problem for any number of start and stop of pipeline

    gst-launch-1.0 -e -v v4l2src device=/dev/video-rpi-cam0 io-mode=dmabuf-import !
      video/x-bayer, width=1920, height=1200, framerate=30/1, format=rggb !
      tiovxisp sink_0::device=/dev/v4l-rpi-subdev0 dcc-isp-file=/opt/imaging/ar0235/dcc_viss_8bit_1920x1200_wider.bin sink_0::dcc-2a-file=/opt/imaging/ar0235/dcc_2a_8bit_1920x1200_wider.bin format-msb=9 sink_0::ae-mode=2 !
      tiovxldc dcc-file=/opt/imaging/ar0235/dcc_ldc_8bit_1920x1200_wider.bin sensor-name=SENSOR_SONY_IMX219_RPI sink_0::ae-mode=2 !
      video/x-raw, format=NV12, width=1920, height=1200, framerate=30/1 !
      v4l2jpegenc output-io-mode=dmabuf-import ! tcpserversink host = 172.25.3.37 port=5000 buffers-soft-max=2 recover-policy=1

    above simple pipeline works fine with n number of start and stop of the pipeline.

    Regards,

    Jay

  • Hi Jay,

    I have routed your query to my colleague. He will respond to you tomorrow.

    Best Regards,

    Suren

  • Hi Jay,

    I saw the following messages in your log:

    IttCtrl_registerHandler: command echo registered at location 0
    IttCtrl_registerHandler: command iss_read_2a_params registered at location 1
    IttCtrl_registerHandler: command iss_write_2a_params registered at location 2
    IttCtrl_registerHandler: command iss_raw_save registered at location 3
    IttCtrl_registerHandler: command iss_yuv_save registered at location 4
    IttCtrl_registerHandler: command iss_read_sensor_reg registered at location 5
    IttCtrl_registerHandler: command iss_write_sensor_reg registered at location 6
    IttCtrl_registerHandler: command dev_ctrl registered at location 7
    IttCtrl_registerHandler: command iss_send_dcc_file registered at location 8
    Error: failed to open i2c bus at /dev/i2c-9

    Did you turn on the live tuning?

    Did you change I2C_BUS to 9 at this line: https://git.ti.com/cgit/processor-sdk/imaging/tree/utils/itt_server/src/itt_ctrl_handle_sensor.c?h=main#n71?

    Looks like after several start and stop of the pipeline, I2C bus 9 is not accessible.

    Regards,

    Jianzhong

  • Hi Jianzhong,

    We haven't change the I2C_BUS to 9. i2c bus error was seen in the working case as well. And I suspect this error was seen to the image tuning related changes. So, i have removed the tuning changes and checked again. I have observed the pipeline hang issue in this as well. Below are the logs for the same:

    APP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=6) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
      2327.406492 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
      2327.406602 s:  VX_ZONE_INIT:Enabled
      2327.406618 s:  VX_ZONE_ERROR:Enabled
      2327.406629 s:  VX_ZONE_WARNING:Enabled
      2327.407615 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
      2327.407815 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
    Setting pipeline to PAUSED ...
    Pipeline is live and does not need PREROLL ...
    Pipeline is PREROLLED ...
    Setting pipeline to PLAYING ...
    New clock: GstSystemClock
    Redistribute latency...
    Redistribute latency...
    Redistribute latency...
    Redistribute latency...
    Redistribute latency...
    Redistribute latency...
    handling interrupt.9.
    Interrupt: Stopping pipeline ...
    EOS on shutdown enabled -- Forcing EOS on the pipeline
    Waiting for EOS...

    So, Hang issue has nothing to do with i2c-9 error. I suspect that one of the element in the pipeline is not freeing  resource and application EOS signal from that element and waits forever.

    Regards,

    Jay

  • Hi Jianzhong,

    Let us know if there is any update on this thread.

    Regards,

    Jay

  • Hi Jianzhong,

    Did you get chance to check this issue? This issue seems critical for our product. Let us know if there is anything else is required from our side.

    Regards,

    Jay

  • Hi Jianzhong, 

    Hint: This issue is observed in the SDK-9. But issue is not observed in the SDK-10. But Currently Our production release will be done with SDK-9 and we want to fix this issue in the SDK-9 as  well. Can you raise this to particular  Development team if they can provide any hint on this issue. Prompt reply will be helpful here.

    Regards,

    Jay

  • Hi Jay,

    SDK 9.x was a pretty old release and I don't think we can support it. Please upgrade to SDK 10.x or 11.1.

    Regards,

    Jianzhong