SK-AM62A-LP: Troubleshooting AR0521 Image Sensor to stream h264 encoded video smoothly at 60 FPS

Steve T

Part Number: SK-AM62A-LP

I'm working with SDK 9.0.0.8 and trying to get an OnSemi AR0521 sensor to stream clearly at 60 FPS. I have written my own driver for the AR0521 that is very similar to the OV5647 driver since I found issue with the native linux driver and have yet to get it to stream even with the little I have tried using it. I'm set up for a resolution of 2600x1952 and believe there to be 440 Horizontal Blanking and 316 Vertical Blanking. I have measured the frame time with a oscilloscope and found it to be 16.6 ms as expected for 60FPS. The MIPI clock is running at ~515MHz measured by the oscilloscope. The max exposure time (Course Integration Time for the sensor) in the driver and gsttiovxisp.c is set to 2260. I'm using bin files created using Imaging project version 9.0.0.5. I'm currently seeing choppy video which leads me to believe that my configuration is dropping frames and I also don't believe that the frame rate is at 60, as expected.

I'm using the following Gstreamer command to stream H264 packets:

GST_DEBUG_FILE=/run/trace.log GST_DEBUG_NO_COLOR=1 GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=element+pipeline)" gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! video/x-bayer, width=2600, height=1952, format=grbg10, framerate=60/1 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0521" dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 ! queue max-size-buffers=1 leaky=0 ! v4l2h264enc ! rtph264pay config-interval=1 pt=96 ! udpsink host=xxx.xxx.xxx.xxx port=5000

And here is the output from the TI gst_tracers utility:

I see the same/similar results in gst_traces if I use a fakesink instead after the v4l2h264enc plugin.

I have produced smooth streams at with the AR0521 configured at 27.5 FPS (an FPS provided by the sensor configuration software).

I have tried using SDK 9.1.0.0 but currently am seeing 35 out-fps across the board. Also, I would highly prefer a solution using the 9.0 SDK as I have a custom image for my custom board that I don't want to migrate at this point.

I'm hoping to get some assistance on where to look to see what piece is causing the bottleneck or dropping frames at this point since I've done a bit of troubleshooting and haven't found the root cause yet.

Thank you,
Steve T

over 1 year ago

0 Jianzhong Xu over 1 year ago

TI__Guru 59445 points

Hello Steve,

Steve T said:
The max exposure time (Course Integration Time for the sensor) in the driver and gsttiovxisp.c is set to 2260

Can you share the source code of gsttiovxisp.c? Did you mean you fixed the exposure time to 2260? What's the unit of 2260, e.g. number of line periods?

Can you try a few things:

1. measure the fps without ISP:

gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! \
video/x-bayer, width=2600, height=1952, format=grbg10, framerate=60/1 ! fpsdisplaysink text-overlay=false

2. measure the fps with ISP

gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! \
video/x-bayer, width=2600, height=1952, format=grbg10, framerate=60/1 ! \
tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0521" \
dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin \
sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 ! \
fpsdisplaysink text-overlay=false

3. add some optimization to the GStreamer pipeline:

gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! \
video/x-bayer, width=2600, height=1952, format=grbg10, framerate=60/1 ! queue leaky=2 ! \
tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0521" \
dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin \
sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 ! \
sink_0::pool-size=8 src_0::pool-size=8 ! queue ! \
fpsdisplaysink text-overlay=false

Regards,

Jianzhong

0 Steve T over 1 year ago in reply to Jianzhong Xu

Intellectual 310 points

Hello Jianzhong,

Thank you once again for your assistance! Here are my current gsttiovxisp.c settings:

static int32_t
get_ar0521_ae_dyn_params (IssAeDynamicParams * p_ae_dynPrms)
{
  int32_t status = -1;
  uint8_t count = 0;

  g_return_val_if_fail (p_ae_dynPrms, status);

  p_ae_dynPrms->targetBrightnessRange.min = 40;
  p_ae_dynPrms->targetBrightnessRange.max = 50;
  p_ae_dynPrms->targetBrightness = 45;
  p_ae_dynPrms->threshold = 1;
  p_ae_dynPrms->enableBlc = 1;
  p_ae_dynPrms->exposureTimeStepSize = 1;

  p_ae_dynPrms->exposureTimeRange[count].min = 16;
  p_ae_dynPrms->exposureTimeRange[count].max = 2260;
  p_ae_dynPrms->analogGainRange[count].min = 1024;    // 1.0x Global Gain
  p_ae_dynPrms->analogGainRange[count].max = 61440;   // 60*1024, 60 gain increments
  p_ae_dynPrms->digitalGainRange[count].min = 256;
  p_ae_dynPrms->digitalGainRange[count].max = 256;
  count++;

  p_ae_dynPrms->numAeDynParams = count;
  status = 0;
  return status;
}

...
...
...

  } else if (g_strcmp0 (self->sensor_name, "SENSOR_ONSEMI_AR0521") == 0) {
    // double multiplier = 0;
    uint16_t gain_increment = 0;

    // *exposure_time_mapped = (4952 * exposure_time / 36363) + 0.5; // 27.5 FPS
    *exposure_time_mapped = exposure_time;

    // multiplier = analog_gain / 1024.0;
    // *analog_gain_mapped = 256.0 - 256.0 / multiplier;
    gain_increment = analog_gain / 1024;
    *analog_gain_mapped = gain_increment;

I realize that I will need to adjust exposure time to make it work with the DCC tool and will do when I need to.

I have been testing some things out using your provided commands but I did find that none of them would run on my system as is, most likely due to not having a monitor attached that supports the high resolution. Also, I don't think 1) would work due to not doing any kind of Bayer conversion.

So for example, "2) measure the fps with ISP"

root@am62axx-evm:~# gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! video/x-bayer, width=2600, height=1952, format=grbg10, framerate=60/1 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0521" dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 ! fpsdisplaysink text-overlay=false
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   433.002253 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   433.002382 s:  VX_ZONE_INIT:Enabled
   433.002440 s:  VX_ZONE_ERROR:Enabled
   433.002473 s:  VX_ZONE_WARNING:Enabled
   433.003871 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
   433.004777 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Got context from element 'fps-display-video_sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayWayland\)\ gldisplaywayland0";
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...

I did some researching and believe I got what your were requesting to work in a "headless" mode:

data-mode="text">GST_DEBUG=fpsdisplaysink:6 gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! video/x-bayer, width=2600, height=1952, format=grbg10, framerate=60/1 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0521" dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 ! fpsdisplaysink text-overlay=false video-sink=fakesink signal-fps-measurements=true s: GTC Frequency = 200 MHz VX_ZONE_WARNING:Enabled VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!! VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!! 0xf6e7c30 DEBUG fpsdisplaysink fpsdisplaysink.c:441:fps_display_sink_start:<fpsdisplaysink0> Use text-overlay? 0 and does not need PREROLL ... 0xf70c860 DEBUG fpsdisplaysink fpsdisplaysink.c:373:display_current_fps:<fpsdisplaysink0> Updated max-fps to 60.665701 0xf70c860 DEBUG fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 60.665701 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.665701 droprate:0.000000 avg-fps:60.665701 0xf70c860 DEBUG fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 57.882034 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:57.882034 droprate:0.000000 avg-fps:59.287590 0xf70c860 DEBUG fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 56.275148 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:56.275148 droprate:0.000000 avg-fps:58.271192 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.928728 droprate:0.000000 avg-fps:58.690542 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.136888 droprate:0.000000 avg-fps:58.981770 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.945555 droprate:0.000000 avg-fps:59.143736 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.153668 droprate:0.000000 avg-fps:59.288607 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.954884 droprate:0.000000 avg-fps:59.372435 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.140642 droprate:0.000000 avg-fps:59.458050 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.948423 droprate:0.000000 avg-fps:59.507363 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.153829 droprate:0.000000 avg-fps:59.566249 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.930613 droprate:0.000000 avg-fps:59.596772 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.157098 droprate:0.000000 avg-fps:59.639932 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.901253 droprate:0.000000 avg-fps:59.658695 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.162599 droprate:0.000000 avg-fps:59.692315 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.951881 droprate:0.000000 avg-fps:59.708603 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.166012 droprate:0.000000 avg-fps:59.735522 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.921551 droprate:0.000000 avg-fps:59.745901 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.151999 droprate:0.000000 avg-fps:59.767283 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.956276 droprate:0.000000 avg-fps:59.776765 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.145775 droprate:0.000000 avg-fps:59.794343 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.948863 droprate:0.000000 avg-fps:59.801390 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.149189 droprate:0.000000 avg-fps:59.816513 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.938584 droprate:0.000000 avg-fps:59.821617 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.168210 droprate:0.000000 avg-fps:59.835476 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.842892 droprate:0.000000 avg-fps:59.835762 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.222464 droprate:0.000000 avg-fps:59.850064 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.940328 droprate:0.000000 avg-fps:59.853298 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.140461 droprate:0.000000 avg-fps:59.863199 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.935066 droprate:0.000000 avg-fps:59.865602 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.169064 droprate:0.000000 avg-fps:59.875384 0xf70c860 LOG fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.940559 droprate:0.000000 avg-fps:59.877427 VX_ZONE_INIT:[tivxHostDeInitLocal:110] De-Initialization Done for HOST !!! VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!! Alloc's: 23 alloc's of 74105306 bytes Free's : 23 free's of 74105306 bytes Open's : 0 allocs of 0 bytes

Rates were the same for "3) add some optimization to the GStreamer pipeline":

root@am62axx-evm:/opt/edgeai-gst-apps/scripts/gst_tracers# GST_DEBUG=fpsdisplaysink:6 gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! video/x-bayer, width=2600, height=1952, format=grbg10, framerate=60/1 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0521" dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 sink_0::pool-size=8 src_0::pool-size=8 ! queue ! fpsdisplaysink text-overlay=false video-sink=fakesink signal-fps-measurements=true
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  1329.389379 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  1329.389618 s:  VX_ZONE_INIT:Enabled
  1329.389645 s:  VX_ZONE_ERROR:Enabled
  1329.389685 s:  VX_ZONE_WARNING:Enabled
  1329.390895 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  1329.391778 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
Setting pipeline to PAUSED ...
0:00:00.177855989  1162      0x633ad60 DEBUG         fpsdisplaysink fpsdisplaysink.c:441:fps_display_sink_start:<fpsdisplaysink0> Use text-overlay? 0
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
0:00:00.947106664  1162      0x632a860 DEBUG         fpsdisplaysink fpsdisplaysink.c:373:display_current_fps:<fpsdisplaysink0> Updated max-fps to 60.612395
0:00:00.947234044  1162      0x632a860 DEBUG         fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 60.612395
0:00:00.947297984  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.612395 droprate:0.000000 avg-fps:60.612395
0:00:01.464467387  1162      0x632a860 DEBUG         fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 59.925338
0:00:01.464826967  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.925338 droprate:0.000000 avg-fps:60.266908
0:00:01.979268491  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.222750 droprate:0.000000 avg-fps:60.252182
0:00:02.496431011  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.942085 droprate:0.000000 avg-fps:60.174357
0:00:03.011739847  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.156856 droprate:0.000000 avg-fps:60.170856
0:00:03.528887838  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.945150 droprate:0.000000 avg-fps:60.133120
0:00:04.044145572  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.163978 droprate:0.000000 avg-fps:60.137527
0:00:04.561615115  1162      0x632a860 DEBUG         fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 59.902076
0:00:04.561983305  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.902076 droprate:0.000000 avg-fps:60.107994
0:00:05.076797786  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.190252 droprate:0.000000 avg-fps:60.117123
0:00:05.593985250  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.928194 droprate:0.000000 avg-fps:60.098176
0:00:06.109281074  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.159438 droprate:0.000000 avg-fps:60.103741
0:00:06.626575770  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.925762 droprate:0.000000 avg-fps:60.088869
0:00:07.141911206  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.155919 droprate:0.000000 avg-fps:60.094021
0:00:07.659201354  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.928057 droprate:0.000000 avg-fps:60.082136
0:00:08.174409122  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.169954 droprate:0.000000 avg-fps:60.087983
0:00:08.691910037  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.903852 droprate:0.000000 avg-fps:60.076441
0:00:09.207278573  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:60.151321 droprate:0.000000 avg-fps:60.080841
0:00:09.724273700  1162      0x632a860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:59.962142 droprate:0.000000 avg-fps:60.074234
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:10.024286297
Setting pipeline to NULL ...
Freeing pipeline ...
  1339.601102 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:110] De-Initialization Done for HOST !!!
  1339.605766 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 29 alloc's of 135007706 bytes
DDR_SHARED_MEM: Free's : 29 free's  of 135007706 bytes
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

However it dropped to ~45 FPS adding in the v4l2h264enc piece to the pipeline:

root@am62axx-evm:~# GST_DEBUG=fpsdisplaysink:6 gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! video/x-bayer, width=2600, height=1952, format=grbg10, framerate=60/1 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0521" dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 sink_0::pool-size=8 src_0::pool-size=8 ! queue ! v4l2h264enc ! fpsdisplaysink text-overlay=false video-sink=fakesink signal-fps-measurements=true
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  2557.932420 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  2557.932544 s:  VX_ZONE_INIT:Enabled
  2557.932557 s:  VX_ZONE_ERROR:Enabled
  2557.932567 s:  VX_ZONE_WARNING:Enabled
  2557.933782 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  2557.934258 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
Setting pipeline to PAUSED ...
0:00:00.177754892  1179     0x19850760 DEBUG         fpsdisplaysink fpsdisplaysink.c:441:fps_display_sink_start:<fpsdisplaysink0> Use text-overlay? 0
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 2552.179282] cdns-csi2rx: Number of lanes: 4
[ 2552.183513] cdns-csi2rx 30101000.csi-bridge: Link Frequency: 128000000
[ 2552.190057] cdns-mipi-dphy-rx 30110000.phy: DPHY lanes: 4
Redistribute latency...
Redistribute latency...
0:00:01.029331878  1179     0x19876860 DEBUG         fpsdisplaysink fpsdisplaysink.c:373:display_current_fps:<fpsdisplaysink0> Updated max-fps to 49.974239
0:00:01.029498219  1179     0x19876860 DEBUG         fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 49.974239
0:00:01.029576135  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:49.974239 droprate:0.000000 avg-fps:49.974239
0:00:01.534461497  1179     0x19876860 DEBUG         fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 45.530353
0:00:01.534666949  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:45.530353 droprate:0.000000 avg-fps:47.785038
0:00:02.053835297  1179     0x19876860 DEBUG         fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 44.283319
0:00:02.054072569  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:44.283319 droprate:0.000000 avg-fps:46.607719
0:00:02.553904300  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:45.996669 droprate:0.000000 avg-fps:46.458296
0:00:03.073839663  1179     0x19876860 DEBUG         fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 44.233442
0:00:03.073993084  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:44.233442 droprate:0.000000 avg-fps:46.007248
0:00:03.573903226  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:45.998850 droprate:0.000000 avg-fps:46.005878
0:00:04.078932113  1179     0x19876860 DEBUG         fpsdisplaysink fpsdisplaysink.c:377:display_current_fps:<fpsdisplaysink0> Updated min-fps to 43.558377
0:00:04.079126559  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:43.558377 droprate:0.000000 avg-fps:45.659605
0:00:04.598832657  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:44.239915 droprate:0.000000 avg-fps:45.479134
0:00:05.109126105  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:45.073923 droprate:0.000000 avg-fps:45.434185
0:00:05.624495706  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:44.641892 droprate:0.000000 avg-fps:45.354385
0:00:06.134201585  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:45.108276 droprate:0.000000 avg-fps:45.332077
0:00:06.639330414  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:45.535258 droprate:0.000000 avg-fps:45.348818
0:00:07.159030408  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:44.253265 droprate:0.000000 avg-fps:45.263194
0:00:07.689068568  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:45.279558 droprate:0.000000 avg-fps:45.264402
0:00:08.219209671  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:45.272194 droprate:0.000000 avg-fps:45.264938
0:00:08.729339559  1179     0x19876860 LOG           fpsdisplaysink fpsdisplaysink.c:381:display_current_fps:<fpsdisplaysink0> Signaling measurements: fps:45.089171 droprate:0.000000 avg-fps:45.254031
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:08.849581543
Setting pipeline to NULL ...
[ 2561.023841] cdns-csi2rx 30101000.csi-bridge: Stream stopped, Register 0x30101060 = 0x10000000
Freeing pipeline ...
  2566.916023 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:110] De-Initialization Done for HOST !!!
  2566.920583 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 29 alloc's of 135007706 bytes
DDR_SHARED_MEM: Free's : 29 free's  of 135007706 bytes
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

1.) Does this definitely point to the encoder slowing the framerate down then at this point in the pipeline?

I generated some mp4 files and VLC other tools are saying that they are also ~45FPS.

gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! video/x-bayer, width=2600, height=1952, format=grbg10, framerate=60/1 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0521" dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 sink_0::pool-size=8 src_0::pool-size=8 ! queue ! v4l2h264enc ! h264parse config-interval=3 ! qtmux ! filesink location=ar521_60fps_capture4.mp4 -e

Perhaps I'm off the mark troubleshooting here but I tried to set some of the extra-controls in the GStreamer original posted pipeline for the v4l2h264enc but I never saw that their "value" changed issuing a "v4l2-ctl -d 1 --list-ctrls-menu" command (while streaming or not and even setting them with "v4l2-ctl -d 1 --set-ctrl h264_profile=4")

Here is the pipeline with the H264 extra-controls added:

gst-launch-1.0 v4l2src io-mode=dmabuf-import device=/dev/video3 ! \
video/x-bayer, width=2600, height=1952, format=grbg10, framerate=60/1 ! \
tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_ONSEMI_AR0521" dcc-isp-file=/opt/imaging/ar0521/linear/dcc_viss_GRBG_10b_2600x1952.bin sink_0::dcc-2a-file=/opt/imaging/ar0521/linear/dcc_2a_GRBG_10b_2600x1952.bin format-msb=9 sink_0::pool-size=8 src_0::pool-size=8 ! \
queue ! v4l2h264enc extra-controls="controls, h264_profile=4, h264_level=15" ! \
rtph264pay config-interval=1 pt=96 ! udpsink host=xxx.xxx.xxx.xxx port=5000

2.) Are these extra-controls supported even? How would I set them if so? (assuming the driver doesn't override them...)

I see in the TRM that the H264 Encoder is "capable" of baseline, compatible baseline, main and high profiles and 5.2 H264 Level. It seems to me currently having it configured to use a 5.2 Level and the High profile would be best.

Best Regards,

Steve T.

+1 Jianzhong Xu over 1 year ago in reply to Steve T

TI__Guru 59445 points

Hi Steve,

Steve T said:
However it dropped to ~45 FPS adding in the v4l2h264enc piece to the pipeline:

The AM62A supports up to 240MPixels/s, as per the data sheet:

AM62Ax Sitara Processors Data Sheet datasheet (Rev. A)

Your sensor resolution is 5MP, so the maximum frame rate that the encoder can support is 240/5 = 48fps. That is approximately what you observed.

Regards,

Jianzhong

0 Steve T over 1 year ago in reply to Jianzhong Xu

Intellectual 310 points

Ohhhh! I see I wasn't interpreting that correctly. Alright, I will work on reconfiguring the driver/image sensor accordingly and give an update hopefully by end of the day.

Thank you!

0 Steve T over 1 year ago in reply to Steve T

Intellectual 310 points

Configuring the driver to work at 45 FPS stopped the jumpy/choppy video and my software receiving the stream is reporting 45 FPS as expected. Thanks for the help troubleshooting to get the configuration correct.

Processors

Processors forum

SK-AM62A-LP: Troubleshooting AR0521 Image Sensor to stream h264 encoded video smoothly at 60 FPS