AM62A7: GStreamer error "vdec 30210000.video-codec: Too many simultaneous instances: 32 (max: 32)"

Hi Christoph,

Can you share the c application that you are trying to run to reproduce the issue on my end. 

Also, have you tried to use the gst_bin_remove() to remove the encoder element to see if it helps.

Best Regards,

Suren

Hi Suren, thank you for your reply.

We are using a Leopard Imaging Camera with an Omnivision OX05B Sensor and the SK-AM62A-LP Eval Board running the current TI Linux Processor SDK Image Version 10.00.00.08. We use the camera driver that is provided in the SDK.

I did some additional tests now and also added gst_bin_remove() as you suggested. Unfortunately this did not solve the issue.

I put together a C example application that reproduces the error (see my next post).

The application creates a RTP Stream using a gstreamer pipeline, streams for a few seconds, then stops and unrefs the pipeline and creates a new one with different parameters to switch between for example RGB/IR mode or to change the resolution. See Line 259/260 in the example code:

I also added a counter that counts the number of switches (RGB/IR or resolution changes) and displays it in an overlay.

I currently see the following 2 errors:

1. When i set the application to switch between RGB and IR, the application crashes after around 62 switches showing first an allocation error and this "vdec 30210000.video-codec: Too many simultaneous instances" error afterwards:

APP: Init ... Done !!!
  1122.755475 s:  VX_ZONE_INIT:Enabled
  1122.755485 s:  VX_ZONE_ERROR:Enabled
  1122.755494 s:  VX_ZONE_WARNING:Enabled
  1122.755859 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-0
  1122.755983 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-1
  1122.756063 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-2
  1122.756253 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-3
  1122.756278 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
  1122.756311 s:  VX_ZONE_INIT:[tivxHostInitLocal:106] Initialization Done for HOST !!!
Pipeline created.
Started RGB Stream with resolution: 1920x1080 | Switch count: 62
[  232.229167] cma: cma_alloc: linux,cma: alloc failed, req-size: 765 pages, ret: -12
[  232.237755] cma: cma_alloc: linux,cma: alloc failed, req-size: 765 pages, ret: -12
[  232.246751] cma: cma_alloc: linux,cma: alloc failed, req-size: 2693 pages, ret: -12
[  232.254486] ------------[ cut here ]------------
[  232.259097] WARNING: CPU: 2 PID: 2168 at /mm/page_alloc.c:4403 __alloc_pages+0x644/0xdb0
[  232.267189] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cdns_csi2rx v4l2_fwnode rpmsg_ctrl snd_soc_hdmi_codec rpmsg_char snd_soc_simple_card crct10dif_ce snd_soc_simple_card_utils e5010_jpeg_enc display_connector v4l2_jpeg rti_wdt k3_j72xx_bandgap ox05b1s rtc_ti_k3 ti_k3_r5_remoteproc tidss snd_soc_tlv320aic3x_i2c sii902x snd_soc_tlv320aic3x drm_dma_helper wave5 j721e_csi2rx drm_kms_helper videobuf2_dma_contig ti_k3_dsp_remoteproc v4l2_async snd_soc_davinci_mcasp tps6598x videobuf2_memops v4l2_mem2mem mcrc64 videobuf2_v4l2 cdns_dphy_rx sa2ul videobuf2_common videodev snd_soc_ti_udma snd_soc_ti_edma typec snd_soc_ti_sdma mc cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[  232.330369] CPU: 2 PID: 2168 Comm: queue220:src Tainted: G           O       6.6.32-ti-gdb8871293143-dirty #1
[  232.340265] Hardware name: Texas Instruments AM62A7 SK (DT)
[  232.345822] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  232.352769] pc : __alloc_pages+0x644/0xdb0
[  232.356860] lr : __dma_direct_alloc_pages.isra.0+0x158/0x284
[  232.362510] sp : ffff800083ffb620
[  232.365812] x29: ffff800083ffb700 x28: 000000000000000c x27: ffff0008061a5480
[  232.372937] x26: 0000000000011000 x25: 0000000000000002 x24: ffff8000800d19f4
[  232.380061] x23: 0000000000000cc0 x22: ffff00080015d410 x21: 0000000000000a84
[  232.387184] x20: 0000000000a84fff x19: 0000000000000000 x18: 0000000000000006
[  232.394306] x17: 72202c7365676170 x16: 2033393632203a65 x15: 7a69732d71657220
[  232.401430] x14: 2c64656c69616620 x13: ffff80008127f178 x12: 00000000000005f7
[  232.408554] x11: 00000000000001fd x10: ffff8000812d7178 x9 : ffff80008127f178
[  232.415677] x8 : 00000000ffffefff x7 : 0000000000000000 x6 : 80000000fffff000
[  232.422800] x5 : 0000000000000003 x4 : 0000000000000000 x3 : 0000000000000000
[  232.429923] x2 : ffff00080e2c8e40 x1 : 0000000000000001 x0 : ffff800081385000
[  232.437048] Call trace:
[  232.439484]  __alloc_pages+0x644/0xdb0
[  232.443226]  __dma_direct_alloc_pages.isra.0+0x158/0x284
[  232.448527]  dma_direct_alloc+0x80/0x328
[  232.452439]  dma_alloc_attrs+0x88/0xfc
[  232.456177]  wave5_vdi_allocate_dma_memory+0x48/0xb4 [wave5]
[  232.461846]  wave5_vpu_enc_register_framebuffer+0x274/0x638 [wave5]
[  232.468110]  wave5_vpu_enc_register_frame_buffer+0x108/0x168 [wave5]
[  232.474460]  wave5_vpu_enc_start_streaming+0x3e0/0x4b4 [wave5]
[  232.480290]  vb2_start_streaming+0x68/0x15c [videobuf2_common]
[  232.486127]  vb2_core_streamon+0xf8/0x1bc [videobuf2_common]
[  232.491785]  vb2_streamon+0x18/0x64 [videobuf2_v4l2]
[  232.496753]  v4l2_m2m_ioctl_streamon+0x38/0x98 [v4l2_mem2mem]
[  232.502508]  v4l_streamon+0x24/0x30 [videodev]
[  232.507012]  __video_do_ioctl+0x330/0x3fc [videodev]
[  232.512014]  video_usercopy+0x2c8/0x658 [videodev]
[  232.516842]  video_ioctl2+0x18/0x24 [videodev]
[  232.521325]  v4l2_ioctl+0x40/0x60 [videodev]
[  232.525634]  __arm64_sys_ioctl+0xac/0xf0
[  232.529548]  invoke_syscall+0x48/0x114
[  232.533291]  el0_svc_common.constprop.0+0xc0/0xe0
[  232.537984]  do_el0_svc+0x1c/0x28
[  232.541290]  el0_svc+0x2c/0x84
[  232.544337]  el0t_64_sync_handler+0x120/0x12c
[  232.548683]  el0t_64_sync+0x190/0x194
[  232.552335] ---[ end trace 0000000000000000 ]---
[  232.557092] vdec 30210000.video-codec: prepare_fb: wave5_vpu_enc_register_frame_buffer, fail: -12
[  232.566126] vdec 30210000.video-codec: Framebuffer preparation, fail: -12
[  234.889612] ------------[ cut here ]------------
[  234.894261] Trying to vunmap() nonexistent vm area (000000001b3f1bd0)
[  234.900722] WARNING: CPU: 3 PID: 1557 at /mm/vmalloc.c:2871 vunmap+0x58/0x60
[  234.907767] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cdns_csi2rx v4l2_fwnode rpmsg_ctrl snd_soc_hdmi_codec rpmsg_char snd_soc_simple_card crct10dif_ce snd_soc_simple_card_utils e5010_jpeg_enc display_connector v4l2_jpeg rti_wdt k3_j72xx_bandgap ox05b1s rtc_ti_k3 ti_k3_r5_remoteproc tidss snd_soc_tlv320aic3x_i2c sii902x snd_soc_tlv320aic3x drm_dma_helper wave5 j721e_csi2rx drm_kms_helper videobuf2_dma_contig ti_k3_dsp_remoteproc v4l2_async snd_soc_davinci_mcasp tps6598x videobuf2_memops v4l2_mem2mem mcrc64 videobuf2_v4l2 cdns_dphy_rx sa2ul videobuf2_common videodev snd_soc_ti_udma snd_soc_ti_edma typec snd_soc_ti_sdma mc cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[  234.970933] CPU: 3 PID: 1557 Comm: app_poc Tainted: G        W  O       6.6.32-ti-gdb8871293143-dirty #1
[  234.980481] Hardware name: Texas Instruments AM62A7 SK (DT)
[  234.986037] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  234.992983] pc : vunmap+0x58/0x60
[  234.996292] lr : vunmap+0x58/0x60
[  234.999596] sp : ffff800082d9bc20
[  235.002899] x29: ffff800082d9bc20 x28: ffff00080b323900 x27: 0000000000000000
[  235.010023] x26: 0000000000000000 x25: ffff0008150ac000 x24: 0000000000000000
[  235.017146] x23: 00000000c1700000 x22: ffff800087467000 x21: 0000000000041000
[  235.024268] x20: ffff00080015d410 x19: ffff800087467000 x18: 0000000000000006
[  235.031392] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800082d9b640
[  235.038515] x14: 0000000000000000 x13: ffff80008127f178 x12: 000000000000068a
[  235.045638] x11: 000000000000022e x10: ffff8000812d7178 x9 : ffff80008127f178
[  235.052761] x8 : 00000000ffffefff x7 : ffff8000812d7178 x6 : 80000000fffff000
[  235.059884] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[  235.067007] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00080b323900
[  235.074131] Call trace:
[  235.076568]  vunmap+0x58/0x60
[  235.079526]  dma_direct_free+0x10c/0x130
[  235.083441]  dma_free_attrs+0x7c/0xc0
[  235.087093]  wave5_vdi_free_dma_memory+0x34/0x70 [wave5]
[  235.092411]  wave5_vpu_enc_close+0x1d0/0x240 [wave5]
[  235.097375]  wave5_vpu_release_device+0x74/0x13c [wave5]
[  235.102684]  wave5_vpu_enc_release+0x20/0x2c [wave5]
[  235.107647]  v4l2_release+0xe4/0xec [videodev]
[  235.112150]  __fput+0xb4/0x274
[  235.115198]  __fput_sync+0x50/0x5c
[  235.118593]  __arm64_sys_close+0x38/0x7c
[  235.122505]  invoke_syscall+0x48/0x114
[  235.126247]  el0_svc_common.constprop.0+0xc0/0xe0
[  235.130940]  do_el0_svc+0x1c/0x28
[  235.134246]  el0_svc+0x2c/0x84
[  235.137293]  el0t_64_sync_handler+0x120/0x12c
[  235.141638]  el0t_64_sync+0x190/0x194
[  235.145291] ---[ end trace 0000000000000000 ]---
[  235.150122] ------------[ cut here ]------------
[  235.154784] 65 pages are still in use!
[  235.158564] WARNING: CPU: 1 PID: 1557 at /mm/page_alloc.c:6345 free_contig_range+0xa0/0xbc
[  235.166823] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cdns_csi2rx v4l2_fwnode rpmsg_ctrl snd_soc_hdmi_codec rpmsg_char snd_soc_simple_card crct10dif_ce snd_soc_simple_card_utils e5010_jpeg_enc display_connector v4l2_jpeg rti_wdt k3_j72xx_bandgap ox05b1s rtc_ti_k3 ti_k3_r5_remoteproc tidss snd_soc_tlv320aic3x_i2c sii902x snd_soc_tlv320aic3x drm_dma_helper wave5 j721e_csi2rx drm_kms_helper videobuf2_dma_contig ti_k3_dsp_remoteproc v4l2_async snd_soc_davinci_mcasp tps6598x videobuf2_memops v4l2_mem2mem mcrc64 videobuf2_v4l2 cdns_dphy_rx sa2ul videobuf2_common videodev snd_soc_ti_udma snd_soc_ti_edma typec snd_soc_ti_sdma mc cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[  235.229989] CPU: 1 PID: 1557 Comm: app_poc Tainted: G        W  O       6.6.32-ti-gdb8871293143-dirty #1
[  235.239539] Hardware name: Texas Instruments AM62A7 SK (DT)
[  235.245095] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  235.252040] pc : free_contig_range+0xa0/0xbc
[  235.256304] lr : free_contig_range+0xa0/0xbc
[  235.260564] sp : ffff800082d9bbb0
[  235.263866] x29: ffff800082d9bbb0 x28: ffff00080b323900 x27: 0000000000000000
[  235.270991] x26: 0000000000000000 x25: ffff0008150ac000 x24: 0000000000000000
[  235.278115] x23: ffff800080f7f140 x22: fffffc0000000000 x21: 00000000000c1741
[  235.285238] x20: 0000000000000041 x19: 00000000000c1741 x18: 0000000000000006
[  235.292361] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800082d9b5d0
[  235.299485] x14: 0000000000000000 x13: ffff80008127f178 x12: 00000000000006ff
[  235.306610] x11: 0000000000000255 x10: ffff8000812d7178 x9 : ffff80008127f178
[  235.313735] x8 : 00000000ffffefff x7 : ffff8000812d7178 x6 : 80000000fffff000
[  235.320859] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[  235.327981] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00080b323900
[  235.335104] Call trace:
[  235.337542]  free_contig_range+0xa0/0xbc
[  235.341458]  cma_release+0x7c/0xcc
[  235.344852]  dma_free_contiguous+0x2c/0x70
[  235.348940]  dma_direct_free+0xe0/0x130
[  235.352767]  dma_free_attrs+0x7c/0xc0
[  235.356418]  wave5_vdi_free_dma_memory+0x34/0x70 [wave5]
[  235.361737]  wave5_vpu_enc_close+0x1d0/0x240 [wave5]
[  235.366701]  wave5_vpu_release_device+0x74/0x13c [wave5]
[  235.372011]  wave5_vpu_enc_release+0x20/0x2c [wave5]
[  235.376975]  v4l2_release+0xe4/0xec [videodev]
[  235.381476]  __fput+0xb4/0x274
[  235.384525]  __fput_sync+0x50/0x5c
[  235.387918]  __arm64_sys_close+0x38/0x7c
[  235.391831]  invoke_syscall+0x48/0x114
[  235.395574]  el0_svc_common.constprop.0+0xc0/0xe0
[  235.400268]  do_el0_svc+0x1c/0x28
[  235.403573]  el0_svc+0x2c/0x84
[  235.406620]  el0t_64_sync_handler+0x120/0x12c
[  235.410966]  el0t_64_sync+0x190/0x194
[  235.414619] ---[ end trace 0000000000000000 ]---
[  235.419293] ------------[ cut here ]------------
[  235.423935] Trying to vunmap() nonexistent vm area (0000000089bfe046)
[  235.430398] WARNING: CPU: 1 PID: 1557 at /mm/vmalloc.c:2871 vunmap+0x58/0x60
[  235.437441] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cdns_csi2rx v4l2_fwnode rpmsg_ctrl snd_soc_hdmi_codec rpmsg_char snd_soc_simple_card crct10dif_ce snd_soc_simple_card_utils e5010_jpeg_enc display_connector v4l2_jpeg rti_wdt k3_j72xx_bandgap ox05b1s rtc_ti_k3 ti_k3_r5_remoteproc tidss snd_soc_tlv320aic3x_i2c sii902x snd_soc_tlv320aic3x drm_dma_helper wave5 j721e_csi2rx drm_kms_helper videobuf2_dma_contig ti_k3_dsp_remoteproc v4l2_async snd_soc_davinci_mcasp tps6598x videobuf2_memops v4l2_mem2mem mcrc64 videobuf2_v4l2 cdns_dphy_rx sa2ul videobuf2_common videodev snd_soc_ti_udma snd_soc_ti_edma typec snd_soc_ti_sdma mc cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[  235.500599] CPU: 1 PID: 1557 Comm: app_poc Tainted: G        W  O       6.6.32-ti-gdb8871293143-dirty #1
[  235.510146] Hardware name: Texas Instruments AM62A7 SK (DT)
[  235.515702] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  235.522649] pc : vunmap+0x58/0x60
[  235.525954] lr : vunmap+0x58/0x60
[  235.529259] sp : ffff800082d9bc20
[  235.532561] x29: ffff800082d9bc20 x28: ffff00080b323900 x27: 0000000000000000
[  235.539684] x26: 0000000000000000 x25: ffff0008150ac000 x24: 0000000000000000
[  235.546808] x23: 00000000c1640000 x22: ffff800087421000 x21: 0000000000021000
[  235.553931] x20: ffff00080015d410 x19: ffff800087421000 x18: 0000000000000006
[  235.561054] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800082d9b640
[  235.568178] x14: 0000000000000000 x13: ffff80008127f178 x12: 000000000000077a
[  235.575301] x11: 000000000000027e x10: ffff8000812d7178 x9 : ffff80008127f178
[  235.582425] x8 : 00000000ffffefff x7 : ffff8000812d7178 x6 : 80000000fffff000
[  235.589547] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[  235.596670] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00080b323900
[  235.603793] Call trace:
[  235.606229]  vunmap+0x58/0x60
[  235.609188]  dma_direct_free+0x10c/0x130
[  235.613102]  dma_free_attrs+0x7c/0xc0
[  235.616754]  wave5_vdi_free_dma_memory+0x34/0x70 [wave5]
[  235.622074]  wave5_vpu_enc_close+0x1dc/0x240 [wave5]
[  235.627037]  wave5_vpu_release_device+0x74/0x13c [wave5]
[  235.632348]  wave5_vpu_enc_release+0x20/0x2c [wave5]
[  235.637312]  v4l2_release+0xe4/0xec [videodev]
[  235.641809]  __fput+0xb4/0x274
[  235.644858]  __fput_sync+0x50/0x5c
[  235.648251]  __arm64_sys_close+0x38/0x7c
[  235.652164]  invoke_syscall+0x48/0x114
[  235.655905]  el0_svc_common.constprop.0+0xc0/0xe0
[  235.660598]  do_el0_svc+0x1c/0x28
[  235.663904]  el0_svc+0x2c/0x84
[  235.666950]  el0t_64_sync_handler+0x120/0x12c
[  235.671297]  el0t_64_sync+0x190/0x194
[  235.674950] ---[ end trace 0000000000000000 ]---
[  235.679592] ------------[ cut here ]------------
[  235.684217] 33 pages are still in use!
[  235.687972] WARNING: CPU: 1 PID: 1557 at /mm/page_alloc.c:6345 free_contig_range+0xa0/0xbc
[  235.696226] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cdns_csi2rx v4l2_fwnode rpmsg_ctrl snd_soc_hdmi_codec rpmsg_char snd_soc_simple_card crct10dif_ce snd_soc_simple_card_utils e5010_jpeg_enc display_connector v4l2_jpeg rti_wdt k3_j72xx_bandgap ox05b1s rtc_ti_k3 ti_k3_r5_remoteproc tidss snd_soc_tlv320aic3x_i2c sii902x snd_soc_tlv320aic3x drm_dma_helper wave5 j721e_csi2rx drm_kms_helper videobuf2_dma_contig ti_k3_dsp_remoteproc v4l2_async snd_soc_davinci_mcasp tps6598x videobuf2_memops v4l2_mem2mem mcrc64 videobuf2_v4l2 cdns_dphy_rx sa2ul videobuf2_common videodev snd_soc_ti_udma snd_soc_ti_edma typec snd_soc_ti_sdma mc cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[  235.759383] CPU: 1 PID: 1557 Comm: app_poc Tainted: G        W  O       6.6.32-ti-gdb8871293143-dirty #1
[  235.768930] Hardware name: Texas Instruments AM62A7 SK (DT)
[  235.774485] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  235.781432] pc : free_contig_range+0xa0/0xbc
[  235.785692] lr : free_contig_range+0xa0/0xbc
[  235.789952] sp : ffff800082d9bbb0
[  235.793254] x29: ffff800082d9bbb0 x28: ffff00080b323900 x27: 0000000000000000
[  235.800378] x26: 0000000000000000 x25: ffff0008150ac000 x24: 0000000000000000
[  235.807501] x23: ffff800080f7f140 x22: fffffc0000000000 x21: 00000000000c1661
[  235.814625] x20: 0000000000000021 x19: 00000000000c1661 x18: 0000000000000006
[  235.821748] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800082d9b5d0
[  235.828870] x14: 0000000000000000 x13: ffff80008127f178 x12: 00000000000007ef
[  235.835994] x11: 00000000000002a5 x10: ffff8000812d7178 x9 : ffff80008127f178
[  235.843118] x8 : 00000000ffffefff x7 : ffff8000812d7178 x6 : 80000000fffff000
[  235.850241] x5 : ffff00087f83db88 x4 : 0000000000000000 x3 : 0000000000000027
[  235.857364] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00080b323900
[  235.864487] Call trace:
[  235.866923]  free_contig_range+0xa0/0xbc
[  235.870835]  cma_release+0x7c/0xcc
[  235.874227]  dma_free_contiguous+0x2c/0x70
[  235.878315]  dma_direct_free+0xe0/0x130
[  235.882140]  dma_free_attrs+0x7c/0xc0
[  235.885791]  wave5_vdi_free_dma_memory+0x34/0x70 [wave5]
[  235.891111]  wave5_vpu_enc_close+0x1dc/0x240 [wave5]
[  235.896075]  wave5_vpu_release_device+0x74/0x13c [wave5]
[  235.901385]  wave5_vpu_enc_release+0x20/0x2c [wave5]
[  235.906349]  v4l2_release+0xe4/0xec [videodev]
[  235.910846]  __fput+0xb4/0x274
[  235.913894]  __fput_sync+0x50/0x5c
[  235.917287]  __arm64_sys_close+0x38/0x7c
[  235.921200]  invoke_syscall+0x48/0x114
[  235.924940]  el0_svc_common.constprop.0+0xc0/0xe0
[  235.929633]  do_el0_svc+0x1c/0x28
[  235.932938]  el0_svc+0x2c/0x84
[  235.935984]  el0t_64_sync_handler+0x120/0x12c
[  235.940330]  el0t_64_sync+0x190/0x194
[  235.943982] ---[ end trace 0000000000000000 ]---
[  235.948638] ------------[ cut here ]------------
[  235.953270] Trying to vunmap() nonexistent vm area (000000000e0ff614)
[  235.959730] WARNING: CPU: 1 PID: 1557 at /mm/vmalloc.c:2871 vunmap+0x58/0x60
[  235.966774] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cdns_csi2rx v4l2_fwnode rpmsg_ctrl snd_soc_hdmi_codec rpmsg_char snd_soc_simple_card crct10dif_ce snd_soc_simple_card_utils e5010_jpeg_enc display_connector v4l2_jpeg rti_wdt k3_j72xx_bandgap ox05b1s rtc_ti_k3 ti_k3_r5_remoteproc tidss snd_soc_tlv320aic3x_i2c sii902x snd_soc_tlv320aic3x drm_dma_helper wave5 j721e_csi2rx drm_kms_helper videobuf2_dma_contig ti_k3_dsp_remoteproc v4l2_async snd_soc_davinci_mcasp tps6598x videobuf2_memops v4l2_mem2mem mcrc64 videobuf2_v4l2 cdns_dphy_rx sa2ul videobuf2_common videodev snd_soc_ti_udma snd_soc_ti_edma typec snd_soc_ti_sdma mc cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[  236.029932] CPU: 1 PID: 1557 Comm: app_poc Tainted: G        W  O       6.6.32-ti-gdb8871293143-dirty #1
[  236.039479] Hardware name: Texas Instruments AM62A7 SK (DT)
[  236.045036] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  236.051982] pc : vunmap+0x58/0x60
[  236.055287] lr : vunmap+0x58/0x60
[  236.058592] sp : ffff800082d9bc20
[  236.061893] x29: ffff800082d9bc20 x28: ffff00080b323900 x27: 0000000000000000
[  236.069017] x26: 0000000000000000 x25: ffff0008150ac000 x24: 0000000000000000
[  236.076139] x23: 00000000c1680000 x22: ffff800087443000 x21: 0000000000023000
[  236.083262] x20: ffff00080015d410 x19: ffff800087443000 x18: 0000000000000006
[  236.090386] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800082d9b640
[  236.097509] x14: 0000000000000000 x13: ffff80008127f178 x12: 000000000000086a
[  236.104633] x11: 00000000000002ce x10: ffff8000812d7178 x9 : ffff80008127f178
[  236.111757] x8 : 00000000ffffefff x7 : ffff8000812d7178 x6 : 80000000fffff000
[  236.118880] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[  236.126003] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00080b323900
[  236.133126] Call trace:
[  236.135561]  vunmap+0x58/0x60
[  236.138520]  dma_direct_free+0x10c/0x130
[  236.142435]  dma_free_attrs+0x7c/0xc0
[  236.146087]  wave5_vdi_free_dma_memory+0x34/0x70 [wave5]
[  236.151405]  wave5_vpu_enc_close+0x1e8/0x240 [wave5]
[  236.156369]  wave5_vpu_release_device+0x74/0x13c [wave5]
[  236.161679]  wave5_vpu_enc_release+0x20/0x2c [wave5]
[  236.166642]  v4l2_release+0xe4/0xec [videodev]
[  236.171141]  __fput+0xb4/0x274
[  236.174189]  __fput_sync+0x50/0x5c
[  236.177583]  __arm64_sys_close+0x38/0x7c
[  236.181495]  invoke_syscall+0x48/0x114
[  236.185236]  el0_svc_common.constprop.0+0xc0/0xe0
[  236.189930]  do_el0_svc+0x1c/0x28
[  236.193235]  el0_svc+0x2c/0x84
[  236.196282]  el0t_64_sync_handler+0x120/0x12c
[  236.200628]  el0t_64_sync+0x190/0x194
[  236.204280] ---[ end trace 0000000000000000 ]---
[  236.208929] ------------[ cut here ]------------
[  236.213560] 35 pages are still in use!
[  236.217329] WARNING: CPU: 1 PID: 1557 at /mm/page_alloc.c:6345 free_contig_range+0xa0/0xbc
[  236.225585] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cdns_csi2rx v4l2_fwnode rpmsg_ctrl snd_soc_hdmi_codec rpmsg_char snd_soc_simple_card crct10dif_ce snd_soc_simple_card_utils e5010_jpeg_enc display_connector v4l2_jpeg rti_wdt k3_j72xx_bandgap ox05b1s rtc_ti_k3 ti_k3_r5_remoteproc tidss snd_soc_tlv320aic3x_i2c sii902x snd_soc_tlv320aic3x drm_dma_helper wave5 j721e_csi2rx drm_kms_helper videobuf2_dma_contig ti_k3_dsp_remoteproc v4l2_async snd_soc_davinci_mcasp tps6598x videobuf2_memops v4l2_mem2mem mcrc64 videobuf2_v4l2 cdns_dphy_rx sa2ul videobuf2_common videodev snd_soc_ti_udma snd_soc_ti_edma typec snd_soc_ti_sdma mc cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[  236.288741] CPU: 1 PID: 1557 Comm: app_poc Tainted: G        W  O       6.6.32-ti-gdb8871293143-dirty #1
[  236.298287] Hardware name: Texas Instruments AM62A7 SK (DT)
[  236.303843] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  236.310790] pc : free_contig_range+0xa0/0xbc
[  236.315051] lr : free_contig_range+0xa0/0xbc
[  236.319309] sp : ffff800082d9bbb0
[  236.322612] x29: ffff800082d9bbb0 x28: ffff00080b323900 x27: 0000000000000000
[  236.329736] x26: 0000000000000000 x25: ffff0008150ac000 x24: 0000000000000000
[  236.336859] x23: ffff800080f7f140 x22: fffffc0000000000 x21: 00000000000c16a3
[  236.343983] x20: 0000000000000023 x19: 00000000000c16a3 x18: 0000000000000006
[  236.351107] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800082d9b5d0
[  236.358230] x14: 0000000000000000 x13: ffff80008127f178 x12: 00000000000008df
[  236.365353] x11: 00000000000002f5 x10: ffff8000812d7178 x9 : ffff80008127f178
[  236.372478] x8 : 00000000ffffefff x7 : ffff8000812d7178 x6 : 80000000fffff000
[  236.379602] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[  236.386724] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00080b323900
[  236.393847] Call trace:
[  236.396282]  free_contig_range+0xa0/0xbc
[  236.400196]  cma_release+0x7c/0xcc
[  236.403589]  dma_free_contiguous+0x2c/0x70
[  236.407677]  dma_direct_free+0xe0/0x130
[  236.411504]  dma_free_attrs+0x7c/0xc0
[  236.415156]  wave5_vdi_free_dma_memory+0x34/0x70 [wave5]
[  236.420473]  wave5_vpu_enc_close+0x1e8/0x240 [wave5]
[  236.425437]  wave5_vpu_release_device+0x74/0x13c [wave5]
[  236.430747]  wave5_vpu_enc_release+0x20/0x2c [wave5]
[  236.435710]  v4l2_release+0xe4/0xec [videodev]
[  236.440206]  __fput+0xb4/0x274
[  236.443255]  __fput_sync+0x50/0x5c
[  236.446649]  __arm64_sys_close+0x38/0x7c
[  236.450562]  invoke_syscall+0x48/0x114
[  236.454302]  el0_svc_common.constprop.0+0xc0/0xe0
[  236.458995]  do_el0_svc+0x1c/0x28
[  236.462301]  el0_svc+0x2c/0x84
[  236.465347]  el0t_64_sync_handler+0x120/0x12c
[  236.469692]  el0t_64_sync+0x190/0x194
[  236.473345] ---[ end trace 0000000000000000 ]---
[  236.477995] ------------[ cut here ]------------
[  236.482622] Trying to vunmap() nonexistent vm area (000000004f58d11e)
[  236.489079] WARNING: CPU: 1 PID: 1557 at /mm/vmalloc.c:2871 vunmap+0x58/0x60
[  236.496120] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cdns_csi2rx v4l2_fwnode rpmsg_ctrl snd_soc_hdmi_codec rpmsg_char snd_soc_simple_card crct10dif_ce snd_soc_simple_card_utils e5010_jpeg_enc display_connector v4l2_jpeg rti_wdt k3_j72xx_bandgap ox05b1s rtc_ti_k3 ti_k3_r5_remoteproc tidss snd_soc_tlv320aic3x_i2c sii902x snd_soc_tlv320aic3x drm_dma_helper wave5 j721e_csi2rx drm_kms_helper videobuf2_dma_contig ti_k3_dsp_remoteproc v4l2_async snd_soc_davinci_mcasp tps6598x videobuf2_memops v4l2_mem2mem mcrc64 videobuf2_v4l2 cdns_dphy_rx sa2ul videobuf2_common videodev snd_soc_ti_udma snd_soc_ti_edma typec snd_soc_ti_sdma mc cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[  236.559276] CPU: 1 PID: 1557 Comm: app_poc Tainted: G        W  O       6.6.32-ti-gdb8871293143-dirty #1
[  236.568823] Hardware name: Texas Instruments AM62A7 SK (DT)
[  236.574379] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  236.581326] pc : vunmap+0x58/0x60
[  236.584632] lr : vunmap+0x58/0x60
[  236.587937] sp : ffff800082d9bc20
[  236.591238] x29: ffff800082d9bc20 x28: ffff00080b323900 x27: 0000000000000000
[  236.598361] x26: 0000000000000000 x25: ffff0008150ac000 x24: 0000000000000000
[  236.605484] x23: 00000000c1620000 x22: ffff800083bbe000 x21: 0000000000012000
[  236.612607] x20: ffff00080015d410 x19: ffff800083bbe000 x18: 0000000000000006
[  236.619730] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800082d9b640
[  236.626854] x14: 0000000000000000 x13: ffff80008127f178 x12: 000000000000095a
[  236.633977] x11: 000000000000031e x10: ffff8000812d7178 x9 : ffff80008127f178
[  236.641101] x8 : 00000000ffffefff x7 : ffff8000812d7178 x6 : 80000000fffff000
[  236.648224] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[  236.655346] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00080b323900
[  236.662469] Call trace:
[  236.664905]  vunmap+0x58/0x60
[  236.667864]  dma_direct_free+0x10c/0x130
[  236.671778]  dma_free_attrs+0x7c/0xc0
[  236.675429]  wave5_vdi_free_dma_memory+0x34/0x70 [wave5]
[  236.680747]  wave5_vpu_enc_close+0x1f4/0x240 [wave5]
[  236.685712]  wave5_vpu_release_device+0x74/0x13c [wave5]
[  236.691022]  wave5_vpu_enc_release+0x20/0x2c [wave5]
[  236.695985]  v4l2_release+0xe4/0xec [videodev]
[  236.700482]  __fput+0xb4/0x274
[  236.703530]  __fput_sync+0x50/0x5c
[  236.706924]  __arm64_sys_close+0x38/0x7c
[  236.710836]  invoke_syscall+0x48/0x114
[  236.714577]  el0_svc_common.constprop.0+0xc0/0xe0
[  236.719270]  do_el0_svc+0x1c/0x28
[  236.722575]  el0_svc+0x2c/0x84
[  236.725621]  el0t_64_sync_handler+0x120/0x12c
[  236.729967]  el0t_64_sync+0x190/0x194
[  236.733620] ---[ end trace 0000000000000000 ]---
[  236.738260] ------------[ cut here ]------------
[  236.742886] 18 pages are still in use!
[  236.746652] WARNING: CPU: 1 PID: 1557 at /mm/page_alloc.c:6345 free_contig_range+0xa0/0xbc
[  236.754907] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cdns_csi2rx v4l2_fwnode rpmsg_ctrl snd_soc_hdmi_codec rpmsg_char snd_soc_simple_card crct10dif_ce snd_soc_simple_card_utils e5010_jpeg_enc display_connector v4l2_jpeg rti_wdt k3_j72xx_bandgap ox05b1s rtc_ti_k3 ti_k3_r5_remoteproc tidss snd_soc_tlv320aic3x_i2c sii902x snd_soc_tlv320aic3x drm_dma_helper wave5 j721e_csi2rx drm_kms_helper videobuf2_dma_contig ti_k3_dsp_remoteproc v4l2_async snd_soc_davinci_mcasp tps6598x videobuf2_memops v4l2_mem2mem mcrc64 videobuf2_v4l2 cdns_dphy_rx sa2ul videobuf2_common videodev snd_soc_ti_udma snd_soc_ti_edma typec snd_soc_ti_sdma mc cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[  236.818062] CPU: 1 PID: 1557 Comm: app_poc Tainted: G        W  O       6.6.32-ti-gdb8871293143-dirty #1
[  236.827608] Hardware name: Texas Instruments AM62A7 SK (DT)
[  236.833164] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  236.840110] pc : free_contig_range+0xa0/0xbc
[  236.844371] lr : free_contig_range+0xa0/0xbc
[  236.848630] sp : ffff800082d9bbb0
[  236.851931] x29: ffff800082d9bbb0 x28: ffff00080b323900 x27: 0000000000000000
[  236.859054] x26: 0000000000000000 x25: ffff0008150ac000 x24: 0000000000000000
[  236.866177] x23: ffff800080f7f140 x22: fffffc0000000000 x21: 00000000000c1632
[  236.873300] x20: 0000000000000012 x19: 00000000000c1632 x18: 0000000000000006
[  236.880424] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800082d9b5d0
[  236.887546] x14: 0000000000000000 x13: ffff80008127f178 x12: 00000000000009cf
[  236.894670] x11: 0000000000000345 x10: ffff8000812d7178 x9 : ffff80008127f178
[  236.901793] x8 : 00000000ffffefff x7 : ffff8000812d7178 x6 : 80000000fffff000
[  236.908915] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[  236.916038] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00080b323900
[  236.923160] Call trace:
[  236.925595]  free_contig_range+0xa0/0xbc
[  236.929509]  cma_release+0x7c/0xcc
[  236.932902]  dma_free_contiguous+0x2c/0x70
[  236.936989]  dma_direct_free+0xe0/0x130
[  236.940815]  dma_free_attrs+0x7c/0xc0
[  236.944466]  wave5_vdi_free_dma_memory+0x34/0x70 [wave5]
[  236.949784]  wave5_vpu_enc_close+0x1f4/0x240 [wave5]
[  236.954748]  wave5_vpu_release_device+0x74/0x13c [wave5]
[  236.960058]  wave5_vpu_enc_release+0x20/0x2c [wave5]
[  236.965022]  v4l2_release+0xe4/0xec [videodev]
[  236.969515]  __fput+0xb4/0x274
[  236.972564]  __fput_sync+0x50/0x5c
[  236.975957]  __arm64_sys_close+0x38/0x7c
[  236.979869]  invoke_syscall+0x48/0x114
[  236.983610]  el0_svc_common.constprop.0+0xc0/0xe0
[  236.988303]  do_el0_svc+0x1c/0x28
[  236.991609]  el0_svc+0x2c/0x84
[  236.994655]  el0t_64_sync_handler+0x120/0x12c
[  236.999001]  el0t_64_sync+0x190/0x194
[  237.002653] ---[ end trace 0000000000000000 ]---
  1128.009429 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:120] De-Initialization Done for HOST !!!
  1128.014152 s:  VX_ZONE_INIT:[tivxDeInitLocal:206] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
  1128.014636 s: IPC: Deinit ... !!!
  1128.015071 s: IPC: DeInit ... Done !!!
  1128.015105 s: MEM: Deinit ... !!!
  1128.015178 s: DDR_SHARED_MEM: Alloc's: 41 alloc's of 162268544 bytes
  1128.015192 s: DDR_SHARED_MEM: Free's : 41 free's  of 162268544 bytes
  1128.015202 s: DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes
  1128.015216 s: MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
Pipeline and elements stopped and unrefed.
Creating the pipeline with video resolution 1920x1080...
APP: Init ... !!!
  1128.018169 s: MEM: Init ... !!!
  1128.018224 s: MEM: Initialized DMA HEAP (fd=8) !!!
  1128.018344 s: MEM: Init ... Done !!!
  1128.018358 s: IPC: Init ... !!!
  1128.035488 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  1128.038970 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  1128.039026 s:  VX_ZONE_INIT:Enabled
  1128.039036 s:  VX_ZONE_ERROR:Enabled
  1128.039046 s:  VX_ZONE_WARNING:Enabled
  1128.039372 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-0
  1128.039506 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-1
  1128.039593 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-2
  1128.039768 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-3
  1128.039791 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
  1128.039822 s:  VX_ZONE_INIT:[tivxHostInitLocal:106] Initialization Done for HOST !!!
Pipeline created.
Started IR Stream with resolution: 1920x1080 | Switch count: 63
[  237.429154] cma: cma_alloc: linux,cma: alloc failed, req-size: 765 pages, ret: -12
[  237.437415] cma: cma_alloc: linux,cma: alloc failed, req-size: 765 pages, ret: -12
[  237.446324] cma: cma_alloc: linux,cma: alloc failed, req-size: 2693 pages, ret: -12
[  237.454142] vdec 30210000.video-codec: prepare_fb: wave5_vpu_enc_register_frame_buffer, fail: -12
[  237.463168] vdec 30210000.video-codec: Framebuffer preparation, fail: -12
  1131.151868 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:120] De-Initialization Done for HOST !!!
  1131.156472 s:  VX_ZONE_INIT:[tivxDeInitLocal:206] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
  1131.156928 s: IPC: Deinit ... !!!
  1131.159162 s: IPC: DeInit ... Done !!!
  1131.159215 s: MEM: Deinit ... !!!
  1131.159308 s: DDR_SHARED_MEM: Alloc's: 31 alloc's of 97426940 bytes
  1131.159322 s: DDR_SHARED_MEM: Free's : 31 free's  of 97426940 bytes
  1131.159333 s: DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes
  1131.159347 s: MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
Pipeline and elements stopped and unrefed.
Creating the pipeline with video resolution 1920x1080...
APP: Init ... !!!
  1131.162434 s: MEM: Init ... !!!
  1131.162596 s: MEM: Initialized DMA HEAP (fd=8) !!!
  1131.162761 s: MEM: Init ... Done !!!
  1131.162787 s: IPC: Init ... !!!
  1131.179832 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  1131.183170 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  1131.183224 s:  VX_ZONE_INIT:Enabled
  1131.183234 s:  VX_ZONE_ERROR:Enabled
  1131.183243 s:  VX_ZONE_WARNING:Enabled
  1131.183597 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-0
  1131.183732 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-1
  1131.183816 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-2
  1131.184017 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-3
  1131.184045 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
  1131.184077 s:  VX_ZONE_INIT:[tivxHostInitLocal:106] Initialization Done for HOST !!!
Pipeline created.
Started RGB Stream with resolution: 1920x1080 | Switch count: 64
[  240.627522] vdec 30210000.video-codec: Too many simultaneous instances: 32 (max: 32)
  1134.318865 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:120] De-Initialization Done for HOST !!!
  1134.323461 s:  VX_ZONE_INIT:[tivxDeInitLocal:206] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
  1134.323892 s: IPC: Deinit ... !!!
  1134.326046 s: IPC: DeInit ... Done !!!
  1134.326099 s: MEM: Deinit ... !!!
  1134.326192 s: DDR_SHARED_MEM: Alloc's: 41 alloc's of 162268544 bytes
  1134.326207 s: DDR_SHARED_MEM: Free's : 41 free's  of 162268544 bytes
  1134.326217 s: DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes
  1134.326231 s: MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
Pipeline and elements stopped and unrefed.
Creating the pipeline with video resolution 1920x1080...
APP: Init ... !!!
  1134.329231 s: MEM: Init ... !!!
  1134.329314 s: MEM: Initialized DMA HEAP (fd=8) !!!
  1134.329455 s: MEM: Init ... Done !!!
  1134.329477 s: IPC: Init ... !!!
  1134.346751 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  1134.350066 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  1134.350112 s:  VX_ZONE_INIT:Enabled
  1134.350122 s:  VX_ZONE_ERROR:Enabled
  1134.350131 s:  VX_ZONE_WARNING:Enabled
  1134.350506 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-0
  1134.350659 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-1
  1134.350760 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-2
  1134.350941 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-3
  1134.350960 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
  1134.350989 s:  VX_ZONE_INIT:[tivxHostInitLocal:106] Initialization Done for HOST !!!
Pipeline created.
Started IR Stream with resolution: 1920x1080 | Switch count: 65
[  243.735836] vdec 30210000.video-codec: Too many simultaneous instances: 32 (max: 32)
  1137.459871 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:120] De-Initialization Done for HOST !!!
  1137.464479 s:  VX_ZONE_INIT:[tivxDeInitLocal:206] De-Initialization Done !!!

2. When running the application to switch between resolutions only, the application crashes after exactly 366 iterations with a segmentation fault:

Pipeline created.
Pipeline started with resolution: 800x600 | Switch count: 365
 2775.474804 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
 2775.479385 s:  VX_ZONE_INIT:[tivxDeInitLocal:204] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 39 alloc's of 148876544 bytes
DDR_SHARED_MEM: Free's : 39 free's  of 148876544 bytes
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
Pipeline and elements stopped and unrefed.
Creating the pipeline with video resolution 1920x1080...
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
 2775.515572 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
 2775.515623 s:  VX_ZONE_INIT:Enabled
 2775.515635 s:  VX_ZONE_ERROR:Enabled
 2775.515646 s:  VX_ZONE_WARNING:Enabled
 2775.516061 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-0
 2775.516223 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-1
 2775.516324 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-2
 2775.516808 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-3
 2775.516835 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
 2775.516870 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
Pipeline created.
Pipeline started with resolution: 1920x1080 | Switch count: 366
 2778.664922 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Init[ 1284.481120] kapptd_printk_skb: 6 callbacks suppressed
ialization Done for HOST !!!
 2778.669501 s:  VX_ZONE_INIT:[ti[ 1284.481135] appt: type=1701 appt(1725332949.264:20): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=1618 comm="app_poc" exe="/home/root/app_poc" sig=11 res=1
vxDeInitLocal:204] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
[ 1284.604527] appt: type=1334 appt(1725332949.388:21): prog-id=15 op=LOAD
[ 1284.611372] appt: type=1334 appt(1725332949.392:22): prog-id=16 op=LOAD
Segmentation fault (core dumped)

What am i doing wrong in my code? Is there anything additional i can add to free up the resources before creating and starting a new pipeline? Or is there an issue with a TIOVX Plugin?

Thank you and regards,

Christoph

OK "Inserting" the code as "Code" here does not seem to work for me. So here is my example application as a simple paste:

#include <gst/gst.h>
#include <iostream>
#include <string>
#include <glib.h>  // For g_usleep

/**
 * @brief Enum for color schemes used in the video stream.
 */
enum GSM_ColorScheme_t
{
    GSM_ColorScheme_NV12, /**< NV12 color scheme (YUV 4:2:0). */
    GSM_ColorScheme_GRAY8 /**< GRAY8 color scheme (grayscale). */
};

// Global GStreamer elements and variables
static GstElement *pipeline = nullptr;
static GstElement *text_overlay = nullptr;
static GstElement *video_converter = nullptr;
static GstElement *video_encoder = nullptr;
static GstElement *isp = nullptr;
static GMainContext *context = nullptr;

static std::string Color = "";  // Current stream color (RGB or IR)

static int RTPStreamWidth = 1920;
static int RTPStreamHeight = 1080;
static bool isHighResolution = true;  // Toggle between high and low resolution
static GSM_ColorScheme_t ColorMode = GSM_ColorScheme_NV12;  // Default color scheme
static int switch_count = 0;  // Counter for how many times the resolution has been switched
static int iteration_count = 0;  // Counter for main loop iterations

/**
 * @brief Sets up the GStreamer pipeline for video processing.
 *
 * This function constructs the GStreamer pipeline according to the specified
 * width, height, and color scheme (RGB or IR).
 *
 * @param width The width of the video stream.
 * @param height The height of the video stream.
 * @param RTPStreamColorScheme The color scheme for the RTP stream (e.g., NV12 or GRAY8).
 */
static void GSM_CreatePipeline(int width, int height, GSM_ColorScheme_t RTPStreamColorScheme)
{
    GError *error = nullptr;
    std::string pipeline_desc;

    std::cout << "Creating the pipeline with video resolution " << width << "x" << height << "..." << std::endl;

    // Choose the pipeline description based on the color scheme.
    switch (RTPStreamColorScheme)
    {
        case GSM_ColorScheme_GRAY8:
            // IR grayscale processing.
            pipeline_desc =
                "v4l2src device=/dev/video3 io-mode=dmabuf-import ! "
                "video/x-bayer, width=2592, height=1944, framerate=30/1, format=bggi10 ! "
                "queue ! tiovxisp name=isp sink_0::pool-size=2 sensor-name=SENSOR_OX05B1S "
                "dcc-isp-file=/opt/imaging/ox05b1s/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ox05b1s/linear/dcc_2a.bin format-msb=9 ! "
                "video/x-raw, format=GRAY8, width=2592, height=1944 ! tee name=t "
                "t. ! queue ! tiovxmultiscaler src_0::pool-size=2 target=1 ! video/x-raw, format=GRAY8, width=" + std::to_string(width) + ", height=" + std::to_string(height) + " ! "
                "queue ! textoverlay name=overlay ! videoconvert name=videoconverter ! v4l2h264enc name=videoencoder ! rtph264pay ! udpsink host=192.168.1.10 port=5000";
            break;

        case GSM_ColorScheme_NV12:
        default:
            // Default NV12 (YUV) color processing.
            pipeline_desc =
                "v4l2src device=/dev/video4 io-mode=dmabuf-import ! "
                "video/x-bayer, width=2592, height=1944, framerate=30/1, format=bggi10 ! "
                "queue ! tiovxisp name=isp sink_0::pool-size=2 sink_0::device=/dev/v4l-subdev2 sensor-name=SENSOR_OX05B1S "
                "dcc-isp-file=/opt/imaging/ox05b1s/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ox05b1s/linear/dcc_2a.bin format-msb=9 ! "
                "queue ! tiovxldc dcc-file=/opt/imaging/ox05b1s/linear/dcc_ldc.bin sensor-name=SENSOR_OX05B1S ! "
                "video/x-raw, format=NV12, width=2592, height=1944 ! queue ! tiovxmultiscaler src_0::pool-size=4 target=1 ! video/x-raw, format=NV12, width=" + std::to_string(width) + ", height=" + std::to_string(height) + " ! "
                "queue ! textoverlay name=overlay ! videoconvert name=videoconverter ! v4l2h264enc name=videoencoder ! rtph264pay ! udpsink host=192.168.1.10 port=5000";
            break;
    }

    // Parse and create the pipeline from the description.
    pipeline = gst_parse_launch(pipeline_desc.c_str(), &error);
    if (error != nullptr)
    {
        std::cerr << "Error creating pipeline: " << error->message << std::endl;
        g_clear_error(&error);
        return;
    }

    // Retrieve key elements from the pipeline.
    text_overlay = gst_bin_get_by_name(GST_BIN(pipeline), "overlay");
    video_converter = gst_bin_get_by_name(GST_BIN(pipeline), "videoconverter");
    video_encoder = gst_bin_get_by_name(GST_BIN(pipeline), "videoencoder");
    isp = gst_bin_get_by_name(GST_BIN(pipeline), "isp");

    std::cout << "Pipeline created." << std::endl;
}

/**
 * @brief Sets the text overlay with the specified string.
 *
 * @param TextOverlay The text to be displayed on the overlay.
 */
int GSM_SetTextOverlay(const std::string& TextOverlay) {
    if (text_overlay != nullptr) {
        g_object_set(text_overlay, "text", TextOverlay.c_str(), nullptr);
    } else {
        std::cerr << "text_overlay element is nullptr" << std::endl;
    }

    return 0;
}

/**
 * @brief Stops and deallocates the GStreamer pipeline.
 *
 * This function ensures all pipeline elements are properly cleaned up and the pipeline
 * is set to the NULL state.
 */
void GSM_StopPipeline()
{
    if (pipeline)
    {
        // Set the pipeline to NULL state to ensure it stops.
        gst_element_set_state(pipeline, GST_STATE_NULL);
        
        // Safely unref all individual elements.
        if (isp) {
            gst_bin_remove(GST_BIN(pipeline), isp);
            gst_object_unref(isp);
            isp = nullptr;
        }
        if (text_overlay) {
            gst_bin_remove(GST_BIN(pipeline), text_overlay);
            gst_object_unref(text_overlay);
            text_overlay = nullptr;
        }
        if (video_converter) {
            gst_bin_remove(GST_BIN(pipeline), video_converter);
            gst_object_unref(video_converter);
            video_converter = nullptr;
        }
        if (video_encoder) {
            gst_bin_remove(GST_BIN(pipeline), video_encoder);
            gst_object_unref(video_encoder);
            video_encoder = nullptr;
        }

        // Finally, unref the entire pipeline.
        gst_object_unref(pipeline);
        pipeline = nullptr;
    }
    std::cout << "Pipeline and elements stopped and unrefed." << std::endl;
}

/**
 * @brief Toggles the video resolution between 1920x1080 and 800x600.
 */
void GSM_ToggleResolution()
{
    // Stop the current pipeline.
    GSM_StopPipeline();

    // Toggle the resolution between high and low.
    if (isHighResolution)
    {
        RTPStreamWidth = 800;
        RTPStreamHeight = 600;
    }
    else
    {
        RTPStreamWidth = 1920;
        RTPStreamHeight = 1080;
    }

    isHighResolution = !isHighResolution;
    switch_count++;  // Increment the switch counter.

    // Create and start the pipeline with the new resolution.
    GSM_CreatePipeline(RTPStreamWidth, RTPStreamHeight, GSM_ColorScheme_NV12);
    if (pipeline)
    {
        gst_element_set_state(pipeline, GST_STATE_PLAYING);

        // Update the text overlay with the current resolution, switch count, and color mode.
        std::string overlay_text = "Resolution: " + std::to_string(RTPStreamWidth) + "x" + std::to_string(RTPStreamHeight) +
                                   " | Switch count: " + std::to_string(switch_count) +
                                   " | Color: " + Color;
        GSM_SetTextOverlay(overlay_text);

        std::cout << "Started " << Color << " Stream with resolution: " << RTPStreamWidth << "x" << RTPStreamHeight << " | Switch count: " << switch_count << std::endl;
    }
}

/**
 * @brief Toggles between color modes (RGB and IR).
 */
void GSM_ToggleColorMode()
{
    // Stop the current pipeline.
    GSM_StopPipeline();
    
    // Toggle between NV12 (RGB) and GRAY8 (IR).
    if (ColorMode == GSM_ColorScheme_NV12)
    {
        ColorMode = GSM_ColorScheme_GRAY8;
        Color = "IR";
    }
    else
    {
        ColorMode = GSM_ColorScheme_NV12;
        Color = "RGB";
    }

    switch_count++;  // Increment the switch counter.

    // Create and start the pipeline with the new color mode.
    GSM_CreatePipeline(RTPStreamWidth, RTPStreamHeight, ColorMode);
    if (pipeline)
    {
        gst_element_set_state(pipeline, GST_STATE_PLAYING);

        // Update the text overlay with the current resolution, switch count, and color mode.
        std::string overlay_text = "Resolution: " + std::to_string(RTPStreamWidth) + "x" + std::to_string(RTPStreamHeight) +
                                   " | Switch count: " + std::to_string(switch_count) +
                                   " | Color: " + Color;
        GSM_SetTextOverlay(overlay_text);

        std::cout << "Started " << Color << " Stream with resolution: " << RTPStreamWidth << "x" << RTPStreamHeight << " | Switch count: " << switch_count << std::endl;
    }
}

/**
 * @brief Main function to initialize GStreamer and control the main loop.
 */
int main(int argc, char *argv[])
{
    std::cout << "Initializing GStreamer..." << std::endl;

    // Initialize GStreamer.
    gst_init(&argc, &argv);

    // Create a default GStreamer context.
    context = g_main_context_default();

    // Create and start the initial pipeline.
    GSM_CreatePipeline(RTPStreamWidth, RTPStreamHeight, ColorMode);
    gst_element_set_state(pipeline, GST_STATE_PLAYING);

    int switch_after_iterations = 30;  // Change resolution/color every 30 iterations.

    // Main loop to handle GStreamer events and toggle modes.
    while (true) {
        // Process GStreamer events in the main loop.
        g_main_context_iteration(context, FALSE);

        // Increment the iteration counter.
        iteration_count++;

        // Toggle color mode or resolution after a set number of iterations (~ 3 seconds).
        if (iteration_count % switch_after_iterations == 0) {
            GSM_ToggleColorMode();
            //GSM_ToggleResolution();
        }

        // Sleep for a short time between iterations (100 ms).
        g_usleep(100000);
    }

    return 0;
}

Regards,

Christoph

Thanks Christoph.

Will give it a try early next week and update you with the result/provide resolution.

Best regards,

Suren

Hi Suren,

die you get the chance to test the demo code on your site?

Regards,

Christoph

Hi Christoph,

I didn't have the OX05B Camera sensor. Now I have it. Will be able to verify and update you by tomorrow.

Apologies for the delay.

Best Regards,

Suren

Hi Christoph,

I was able to reproduce the issue. I am working with the SW team to root cause the issue.

Will keep you posted on the developments.

Best Regards,

Suren

Thank you very much Suren.

Hi Christoph, 

We suspect the application is not able to close the pipeline correctly. As a result, the CMA memory is getting exhausted and you end having the errors.

We also tried to play with the below script and not seeing any issues.

#!/bin/bash

x=1
while true
do
gst-launch-1.0 videotestsrc is-live=true do-timestamp=true num-buffers=90 ! video/x-raw, width=1920, height=1080, framerate=30/1, format=NV12 ! v4l2h264enc ! filesink location=test.h264
echo "Run $x"
cat /proc/meminfo | grep Cma
x=$(( $x + 1 ))
done

Looks like the application need to handle the EOS correctly upon changing the resolution or toggling the mode and remove/add encoder elements.

Best Regards,

Suren

Hi Suren, thank you for your tests and help. Do you have any suggestions what i could try to handle the EOS correctly?

Could the error also be caused by one of the tiovx plugins that im using since the test you did is not using the ISP, multiscaler etc...?

Best Regards,

Christoph

Hi Christoph,

Please refer the below thread on handling EOS. In actual application creating pad probes and issuing EOS is relevant.

https://stackoverflow.com/questions/50417045/gstreamer-change-source-element-dynamically

Also try to unreference the tiovx elements as well in the pipeline and see if it helps.

Best Regards,

Suren

Hi Christoph,

An alternative approach could be to have two virtual channels from the sensor, one for RGB and one for IR. This will create two video nodes in user space. Then you can stream from both video nodes simultaneously. For example, below Gstreamer pipeline takes input from each video node and send it to the encoder. Just replace the “fakesink” with the your RTP sink. Then on the receive side, these 2 streams can be combined together.

gst-launch-1.0 -v \
v4l2src device=/dev/video4 io-mode=dmabuf-import ! \
video/x-bayer, width=2592, height=1944, framerate=30/1, format=bggi10 ! queue ! \
tiovxisp sink_0::pool-size=2 sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_OX05B1S" \
dcc-isp-file=/opt/imaging/ox05b1s/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ox05b1s/linear/dcc_2a.bin format-msb=9 ! \
video/x-raw, format=NV12, width=2592, height=1944 ! queue ! \
tiovxmultiscaler src_0::pool-size=3 target=1 ! video/x-raw, format=NV12, width=1600, height=1200 ! queue ! v4l2h264enc ! \
fakesink \
v4l2src device=/dev/video3 io-mode=dmabuf-import ! \
video/x-bayer, width=2592, height=1944, framerate=30/1, format=bggi10 ! queue ! \
tiovxisp sink_0::pool-size=2 sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_OX05B1S" \
dcc-isp-file=/opt/imaging/ox05b1s/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ox05b1s/linear/dcc_2a.bin format-msb=9 ! \
video/x-raw, format=GRAY8, width=2592, height=1944 ! queue ! \
tiovxmultiscaler src_0::pool-size=3 target=0 ! video/x-raw, format=GRAY8, width=1600, height=1200 ! \
videoconvert ! video/x-raw, format=NV12, width=1600, height=1200 ! queue ! v4l2h264enc ! \
fakesink

Regards,

Jianzhong

Hi Jianzhong, thank you for your reply.

Is there also a way to have only one RTP Stream output and decide dynamically to send IR or RGB data to this stream?

We already tried with "input-selector" but were not able to get this to run.

Currently our only working approach is to stop and unref the current pipeline and then start a new pipeline.

But then we run into memory issues after a while regardless of unrefing all elements, sending EOS etc.

This memory issue could also be related to this gstreamer bug:

https://discourse.gstreamer.org/t/memory-leaking-after-restarting-pipelines/607

Our current pipelines look like this:

for IR RTP stream (while simultaneously receiving IR and RGB images in full res):

pipeline_desc =
"v4l2src device=/dev/video4 io-mode=dmabuf-import ! "
"video/x-bayer, width=2592, height=1944, framerate=30/1, format=bggi10 ! "
"queue ! tiovxisp sink_0::pool-size=2 sink_0::device=/dev/v4l-subdev2 sensor-name=SENSOR_OX05B1S "
"dcc-isp-file=/opt/imaging/ox05b1s/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ox05b1s/linear/dcc_2a.bin format-msb=9 ! "
"queue ! tiovxldc dcc-file=/opt/imaging/ox05b1s/linear/dcc_ldc.bin sensor-name=SENSOR_OX05B1S ! "
"video/x-raw, format=NV12, width=2592, height=1944 ! appsink name=bgr_sink drop=true "
"v4l2src device=/dev/video3 io-mode=dmabuf-import ! "
"video/x-bayer, width=2592, height=1944, framerate=30/1, format=bggi10 ! "
"queue ! tiovxisp sink_0::pool-size=2 sensor-name=SENSOR_OX05B1S "
"dcc-isp-file=/opt/imaging/ox05b1s/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ox05b1s/linear/dcc_2a.bin format-msb=9 ! "
"video/x-raw, format=GRAY8, width=2592, height=1944 ! tee name=t "
"t. ! queue ! tiovxmultiscaler src_0::pool-size=2 target=1 ! video/x-raw, format=GRAY8, width=1920, height=1080 ! "
"queue ! textoverlay name=overlay ! videoconvert name=videoconverter ! v4l2h264enc name=videoencoder ! rtph264pay ! udpsink host=192.168.1.10 port=5000 "
"t. ! appsink name=ir_sink drop=true";

for RGB RTP stream (while simultaneously receiving IR and RGB images in full res):

pipeline_desc =
"v4l2src device=/dev/video4 io-mode=dmabuf-import ! "
"video/x-bayer, width=2592, height=1944, framerate=30/1, format=bggi10 ! "
"queue ! tiovxisp sink_0::pool-size=2 sink_0::device=/dev/v4l-subdev2 sensor-name=SENSOR_OX05B1S "
"dcc-isp-file=/opt/imaging/ox05b1s/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ox05b1s/linear/dcc_2a.bin format-msb=9 ! "
"queue ! tiovxldc dcc-file=/opt/imaging/ox05b1s/linear/dcc_ldc.bin sensor-name=SENSOR_OX05B1S ! "
"video/x-raw, format=NV12, width=2592, height=1944 ! tee name=t "
"t. ! queue ! tiovxmultiscaler src_0::pool-size=2 target=1 ! video/x-raw, format=NV12, width=1920, height=1080 ! "
"queue ! textoverlay name=overlay ! videoconvert name=videoconverter ! "
"v4l2h264enc name=videoencoder ! rtph264pay ! udpsink host=192.168.1.10 port=5000 "
"t. ! appsink name=bgr_sink drop=true "
"v4l2src device=/dev/video3 io-mode=dmabuf-import ! "
"video/x-bayer, width=2592, height=1944, framerate=30/1, format=bggi10 ! "
"queue ! tiovxisp sink_0::pool-size=2 sensor-name=SENSOR_OX05B1S "
"dcc-isp-file=/opt/imaging/ox05b1s/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ox05b1s/linear/dcc_2a.bin format-msb=9 ! "
"video/x-raw, format=GRAY8, width=2592, height=1944 ! appsink name=ir_sink drop=true";

We would like to dynamically be able to switch between both without killing the old pipeline and restarting the other.

Regards,

Christoph

Hello Christoph,

Thanks for sharing your pipeline. I'm just curious, but why did you run both RGB and IR streams even when you only encode and send one stream?

Regards,

Jianzhong

Hi Jianzhong,

we are only running one of the pipelines at the same time.

Our requirements are to send video (IR or RGB) via RTP while receiving IR and RGB image frames in full resolution to send them to our neural networks at the same time.

Currently our approach to switch between IR and RGB is the following:

When IR-RTP stream is requested, we start the first pipeline (IR RTP stream (while simultaneously receiving IR and RGB images in full res)).

When then at another point in time RGB-RTP streamis requested, we stop the currently running IR RTP pipeline, send EOS to it, put the pipeline to GST_STATE_NULL, unref all the elements and then unref the pipeline itself.

After this is done we start the other pipeline (RGB RTP stream (while simultaneously receiving IR and RGB images in full res)).

Then if IR is requested again, we kill the current RGB pipeline and then start the IR pipeline again.

Unfortunately this approach eats up the CMA memory after a while as discussed above.

So we are wondering if there is another way to switch dynamically between IR and RGB which does not need to kill the current pipeline and rebuild a new pipeline.

Regards,

Christoph

Hi Christoph,

You could try dynamically modifying the pipeline (without actually pausing it) by adding a pad probe to a queue element in your pipeline, and modifying the downstream elements in the pad probe callback. This keeps your pipeline up and running, and you can replace the downstream elements to a fakesink if that path isn't used.

A test program to demonstrate the mechanism (videotestsrc -> tee -> 3 paths, turn on processing/displaying on different paths with a command on the console)

#include <gst/gst.h>
#include <stdbool.h>
#include <glib.h>
#include <glib/gprintf.h>
#include <gio/gio.h>
#include <stdbool.h>

static GstPad *blockpad1, *blockpad2, *blockpad3;
static GstElement *pipeline;
static GstElement *q_stream1, *q_stream2, *q_stream3;
GQueue sink[3] = { G_QUEUE_INIT, G_QUEUE_INIT, G_QUEUE_INIT };

static GstPadProbeReturn pad_probe_cb(GstPad *pad, GstPadProbeInfo *info, gpointer user_data)
{
        GstPad *srcpad, *sinkpad;
        GstElement * parent = gst_pad_get_parent_element(pad), *element;
        char name[11];
        int stream = -1;
        gboolean fake = false;

        GST_DEBUG_OBJECT(pad, "pad is blocked now");

        if (g_ascii_strncasecmp(gst_object_get_name(GST_OBJECT(parent)), "queue-stream-1", 14) == 0) {
                stream = 0;
        }
        if (g_ascii_strncasecmp(gst_object_get_name(GST_OBJECT(parent)), "queue-stream-2", 14) == 0) {
                stream = 1;
        }
        if (g_ascii_strncasecmp(gst_object_get_name(GST_OBJECT(parent)), "queue-stream-3", 14) == 0) {
                stream = 2;
        }

        if (g_ascii_strncasecmp(gst_object_get_name(GST_OBJECT(g_queue_peek_head(&sink[stream]))), "fakesink", 8) == 0) {
                fake = true;
        }


        /* empty the queue of elements currently connected */
        while (!g_queue_is_empty(&sink[stream])) {
                element = g_queue_pop_head(&sink[stream]);
                gst_element_set_state(element, GST_STATE_NULL);

                GST_DEBUG_OBJECT(pipeline, "removing %" GST_PTR_FORMAT, element);
                gst_bin_remove(GST_BIN(pipeline), element);
        }

        if (fake) {
                GstElement *overlay, *new_sink;
                char text[9];
                sprintf(name, "overlay%d", stream + 1);
                overlay = gst_element_factory_make("textoverlay", name);
                g_object_set(overlay, "halignment", 0, NULL);
                sprintf(text, "Camera %d", stream + 1);
                g_object_set(overlay, "text", text, NULL);

                sprintf(name, "videosink%d", stream + 1);
                new_sink = gst_element_factory_make("xvimagesink", name);

                GST_DEBUG_OBJECT(pipeline, "adding %" GST_PTR_FORMAT "%" GST_PTR_FORMAT, overlay, new_sink);
                gst_bin_add_many(GST_BIN(pipeline), overlay, new_sink, NULL);

                GST_DEBUG_OBJECT(pipeline, "linking...");
                gst_element_link_many(parent, overlay, new_sink, NULL);

                gst_element_set_state(overlay, GST_STATE_PLAYING);
                gst_element_set_state(new_sink, GST_STATE_PLAYING);
                g_queue_push_tail(&sink[stream], overlay);
                g_queue_push_tail(&sink[stream], new_sink);
        } else {
                GstElement *new_sink;
                sprintf(name, "fakesink%d", stream + 1);
                new_sink = gst_element_factory_make("fakesink", name);

                GST_DEBUG_OBJECT(pipeline, "adding %" GST_PTR_FORMAT, new_sink);
                gst_bin_add(GST_BIN(pipeline), new_sink);

                GST_DEBUG_OBJECT(pipeline, "linking...");
                gst_element_link_many(parent, new_sink, NULL);

                gst_element_set_state(new_sink, GST_STATE_PLAYING);
                g_queue_push_tail(&sink[stream], new_sink);
        }

        GST_DEBUG_OBJECT(pipeline, "done.");
        gst_object_unref(parent);

        return GST_PAD_PROBE_REMOVE;
}

static gboolean cli_cb(GIOChannel *iochannel, GIOCondition cond, gpointer data)
{
        GMainLoop __attribute__((unused)) *loop = data;
        GError *error = NULL;
        gsize length;
        gsize terminator_pos;
        gchar *str_return;

        if (g_io_channel_read_line(iochannel, &str_return, &length, &terminator_pos, &error) == G_IO_STATUS_ERROR) {
                g_warning("Unable to read from stdin!\n");
        }

        if (NULL != error) {
                g_error("%s\n", error->message);
        }

        if (g_ascii_strncasecmp(str_return, "quit", 4) == 0) {
                g_main_loop_quit(loop);
        }
        if (g_ascii_strncasecmp(str_return, "1", 1) == 0) {
                gst_pad_add_probe(blockpad1, GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM, pad_probe_cb, data, NULL);
        }
        if (g_ascii_strncasecmp(str_return, "2", 1) == 0) {
                gst_pad_add_probe(blockpad2, GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM, pad_probe_cb, data, NULL);
        }
        if (g_ascii_strncasecmp(str_return, "3", 1) == 0) {
                gst_pad_add_probe(blockpad3, GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM, pad_probe_cb, data, NULL);
        }

        g_printf("> ");
        return TRUE;
}

static gboolean bus_cb(GstBus *bus, GstMessage *msg, gpointer user_data)
{
        GMainLoop *loop = user_data;

        switch (GST_MESSAGE_TYPE(msg)) {
        case GST_MESSAGE_ERROR:
                {
                GError *err = NULL;
                gchar *dbg;

                gst_message_parse_error(msg, &err, &dbg);
                gst_object_default_error(msg->src, err, dbg);
                g_clear_error(&err);
                g_free(dbg);
                g_main_loop_quit(loop);
                break;
                }
        default:
                break;
        }
}

int main(int argc, char **argv)
{
        GMainLoop *loop;
        GstElement *src, *filter, *q1, *tee, *sink1, *sink2, *sink3;
        GstBus *bus;
        GIOChannel *iochannel;
        GError *error = NULL;
        guint source;

        gst_init(&argc, &argv);

        pipeline = gst_pipeline_new("pipeline");

        src = gst_element_factory_make("videotestsrc", NULL);
        g_object_set(src, "is-live", TRUE, NULL);

        filter = gst_element_factory_make("capsfilter", NULL);
        gst_util_set_object_arg(G_OBJECT(filter), "caps",
                        "video/x-raw, width=320, height=240, framerate=30/1, format=NV12");

        q1 = gst_element_factory_make("queue", NULL);

        tee = gst_element_factory_make("tee", "tee");

        q_stream1 = gst_element_factory_make("queue", "queue-stream-1");
        blockpad1 = gst_element_get_static_pad(q_stream1, "src");
        q_stream2 = gst_element_factory_make("queue", "queue-stream-2");
        blockpad2 = gst_element_get_static_pad(q_stream2, "src");
        q_stream3 = gst_element_factory_make("queue", "queue-stream-3");
        blockpad3 = gst_element_get_static_pad(q_stream3, "src");

        sink1 = gst_element_factory_make("fakesink", "fakesink1");
        if (sink1)
                g_queue_push_tail(&sink[0], sink1);
        sink2 = gst_element_factory_make("fakesink", "fakesink2");
        if (sink2)
                g_queue_push_tail(&sink[1], sink2);
        sink3 = gst_element_factory_make("fakesink", "fakesink3");
        if (sink3)
                g_queue_push_tail(&sink[2], sink3);

        gst_bin_add_many(GST_BIN(pipeline), src, filter, q1, tee, q_stream1, sink1, q_stream2, sink2, q_stream3, sink3, NULL);

        if (!gst_element_link_many(src, filter, tee, NULL) ||
            !gst_element_link_many(tee, q_stream1, sink1, NULL) ||
            !gst_element_link_many(tee, q_stream2, sink2, NULL) ||
            !gst_element_link_many(tee, q_stream3, sink3, NULL)) {
                g_error("Failed to link elements");
                return -2;
        }

        if (gst_element_set_state(pipeline, GST_STATE_PLAYING) == GST_STATE_CHANGE_FAILURE) {
                g_error("Error starting pipeline");
                return 1;
        }

        loop = g_main_loop_new(NULL, FALSE);
        gst_bus_add_watch(GST_ELEMENT_BUS(pipeline), bus_cb, loop);

        iochannel = g_io_channel_unix_new(STDIN_FILENO);
        g_io_channel_set_encoding(iochannel, NULL, &error);
        source = g_io_add_watch(iochannel, G_IO_IN, cli_cb, loop);
        g_print("> ");

        g_main_loop_run(loop);
        gst_element_set_state(pipeline, GST_STATE_NULL);
        g_source_remove(source);
        g_io_channel_unref(iochannel);
        gst_object_unref(blockpad1);
        gst_object_unref(blockpad2);
        gst_object_unref(blockpad3);
        gst_bus_remove_watch(GST_ELEMENT_BUS(pipeline));
        gst_object_unref(pipeline);
        g_main_loop_unref(loop);

        return 0;
}

I hope this helps.

Regards,

Bas Vermeulen