SK-AM68: port imx477 to SK-AM68

csscyt

Prodigy 210 points

Part Number: SK-AM68

make all log:

make_all.log

Tool/software:

Hi,

I need to port imx477 to SK-AM68 platform,
I follow this
https://dev.ti.com/tirex/explore/content/am62ax_academy_9_02_00_00_v1/_build_am62ax_academy_9_02_00_00_v1/source/linux/ch-develop/dev-use-camera.html#enable-a-new-csi-2-sensor

Enable A New CSI-2 Sensor

The Processor SDK supports a few sensors out-of-box. Both the driver modules and the device tree overlays for the supported sensors are pre-built and included in the SDK. To enable a new CSI-2 sensor, the following listed steps are needed. Some steps can be skipped if the driver is already included in the SDK.

Add the sensor driver source code to drivers/media/i2c/<sensor_name>.c (path is relative to <sdk root>/board-support/ti-linux-kernel).

Add the sensor configuration in drivers/media/i2c/Kconfig. For example:

config VIDEO_<SENSOR_NAME>
        tristate "<sensor_name> support"
        depends on VIDEO_DEV && I2C
        depends on ACPI || COMPILE_TEST
        select MEDIA_CONTROLLER
        select VIDEO_V4L2_SUBDEV_API
        select V4L2_FWNODE
        help
          This is a Video4Linux2 sensor driver for the <sensor_name>

Enable the compilation of the sensor driver in drivers/media/i2c/Makefile:
obj-$(CONFIG_VIDEO_<SENSOR_NAME>) += <sensor_name>.o
Enable the kernel module for the sensor in arch/arm64/configs/defconfig:
CONFIG_VIDEO_<SENSOR_NAME>=m
Create a device tree overlay for the sensor and add it to arch/arm64/boot/dts/ti:
arch/arm64/boot/dts/ti/<device_tree_overlay_name>.dtso
Add the device tree overlay to arch/arm64/boot/dts/ti/Makefile:
dtb-$(CONFIG_ARCH_K3) += <device_tree_overlay_name>.dtso
Rebuild and install Linux kernel according to the instructions given in section Build Kernel.
Boot the EVM with the updated kernel. Apply the device tree overlay according to instructions given in the Camera Evaluation section.
Verify the sensor is probed according to Camera Evaluation.

the kernel module(imx477.ko) has been updated to the target board, and enable it by dtb overlay file in /run/media/BOOT-mmcblk1p1/uEnv.txt as below,

name_overlays=k3-j721s2-edgeai-apps.dtbo k3-am68-sk-bb-rpi-cam-imx477.dtbo

but i cann't see the imx477 log during the booting the EVM, did I miss any steps? thanks

over 1 year ago

0 Fabiana Jaimes over 1 year ago

TI__Mastermind 19680 points

Hi,

What was the result of the camera verification? Could you share the output of dmesg | grep -i imx and dmesg | grep -i csi ? If you haven't already, I recommend adding imx477 to the setup_cameras.sh script located in the /opt/edgeai-gst-apps/scripts directory.

Thank you,

Fabiana

0 csscyt over 1 year ago in reply to Fabiana Jaimes

Prodigy 210 points

the output log：

root@am68a-sk:/opt/edgeai-gst-apps# dmesg | grep imx

[ 4.811001] imx477 4-0010: supply dovdd not found, using dummy regulator
[ 4.822631] imx477 4-0010: supply avdd not found, using dummy regulator
[ 4.830100] imx477 4-0010: supply dvdd not found, using dummy regulator
[ 4.863887] imx477 4-0010: failed to find sensor: -5
[ 4.880402] imx477: probe of 4-0010 failed with error -5

[ 4.971633] imx477 5-0010: supply dovdd not found, using dummy regulator
[ 5.069309] imx477 5-0010: supply avdd not found, using dummy regulator
[ 5.133913] imx477 5-0010: supply dvdd not found, using dummy regulator

[ 5.171249] imx477 5-0010: before read reg
[ 5.183112] imx477 5-0010: client->addr is 16
[ 5.192762] imx477 5-0010: failed to find sensor: -5
[ 5.273839] imx477: probe of 5-0010 failed with error -5

and in imx477.c

/* Chip ID */

#define IMX477_REG_ID 0x0016

#define IMX477_ID 0x477

the k3-am68-sk-bb-rpi-cam-imx477.dtso

/dts-v1/;
/plugin/;

#include <dt-bindings/gpio/gpio.h>
#include "k3-pinctrl.h"

&{/} {
	clk_imx477_fixed: imx477-xclk {
		compatible = "fixed-clock";
		#clock-cells = <0>;
		clock-frequency = <24000000>;
	};
};

&exp3 {
	p01-hog {
		/* CSI_MUX_SEL_2 */
		gpio-hog;
		gpios = <1 GPIO_ACTIVE_HIGH>;
		output-high;
		line-name = "CSI_MUX_SEL_2";
	};
};

&main_i2c1 {
	status = "okay";
	#address-cells = <1>;
	#size-cells = <0>;

	i2c-switch@70 {
		compatible = "nxp,pca9543";
		#address-cells = <1>;
		#size-cells = <0>;
		reg = <0x70>;

		i2c-alias-pool = /bits/ 16 <0x10 0x11>;

		/* CAM0 I2C */
		cam0_i2c: i2c@0 {
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <0>;

			imx477_0: imx477_0@10 {
				compatible = "sony,imx477";
				reg = <0x10>;

				clocks = <&clk_imx477_fixed>;
				clock-names = "xclk";

				port {
					csi2_cam0: endpoint {
						remote-endpoint = <&csi2rx0_in_sensor>;
						link-frequencies = /bits/ 64 <456000000>;
						clock-lanes = <0>;
						data-lanes = <1 2>;
					};
				};
			};
		};

		/* CAM1 I2C */
		cam1_i2c: i2c@1 {
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <1>;

			imx477_1: imx477_1@10 {
				compatible = "sony,imx477";
				reg = <0x10>;

				clocks = <&clk_imx477_fixed>;
				clock-names = "xclk";

				port {
					csi2_cam1: endpoint {
						remote-endpoint = <&csi2rx1_in_sensor>;
						link-frequencies = /bits/ 64 <456000000>;
						clock-lanes = <0>;
						data-lanes = <1 2>;
					};
				};
			};
		};
	};
};

&csi0_port0 {
	status = "okay";
	csi2rx0_in_sensor: endpoint {
		remote-endpoint = <&csi2_cam0>;
		bus-type = <4>; /* CSI2 DPHY. */
		clock-lanes = <0>;
		data-lanes = <1 2>;
	};
};

&csi1_port0 {
	status = "okay";
	csi2rx1_in_sensor: endpoint {
		remote-endpoint = <&csi2_cam1>;
		bus-type = <4>; /* CSI2 DPHY. */
		clock-lanes = <0>;
		data-lanes = <1 2>;
	};
};

Could you help me confirm which step is wrong or what is missing? thanks

in k3-am68-sk-bb-rpi-cam-imx477.dtso the device reg addr is 0x10 or 0x1A?

0 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Hi,

Thank you for the data. Due to a US holiday, there will a delay in my response. Thanks for understanding.

-Fabiana

0 csscyt over 1 year ago in reply to Fabiana Jaimes

Prodigy 210 points

root@am68a-sk:/opt/edgeai-gst-apps# dmesg | grep -i csi
[ 0.428166] SCSI subsystem initialized
[ 0.712024] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[ 1.137471] i2c 4-001a: Fixed dependency cycle(s) with /bus@100000/ticsi2rx@4500000/csi-bridge@4504000
[ 1.152089] i2c 5-001a: Fixed dependency cycle(s) with /bus@100000/ticsi2rx@4510000/csi-bridge@4514000
[ 1.185712] gpio-481 (CSI_MUX_SEL_2): hogged as output/high
[ 4.931977] imx477 4-001a: gpio info: CSI0_B_GPIO1
[ 6.379594] imx477 5-001a: gpio info: CSI0_B_GPIO1
[ 7.021885] cdns-csi2rx 4504000.csi-bridge: Probed CSI2RX with 2/4 lanes, 4 streams, external D-PHY
[ 7.090799] cdns-csi2rx 4514000.csi-bridge: Probed CSI2RX with 2/4 lanes, 4 streams, external D-PHY

0 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Hi,

Please expect a delay in response as I am out of office.

Thank you,

Fabiana

0 csscyt over 1 year ago in reply to Fabiana Jaimes

Prodigy 210 points

i checked the hardware configutation and now imx477 works.

 
 root@am68a-sk:/opt/edgeai-gst-apps# 
  CSI Camera 0 detected
    device = /dev/video-rpi-cam0
    name = imx477
    format = [fmt:SRGGB12_1X12/2028x1080]
    subdev_id = /dev/v4l-rpi-subdev0
    isp_required = yes
[   15.576636] sh (986): drop_caches: 1
root@am68a-sk:/opt/edgeai-gst-apps#

and check imx477 camera:

root@am68a-sk:/opt/edgeai-gst-apps#
root@am68a-sk:/opt/edgeai-gst-apps# v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-mbus-codes pad=0
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=0,stream=0)
   0x3012: MEDIA_BUS_FMT_SRGGB12_1X12
   0x300f: MEDIA_BUS_FMT_SRGGB10_1X10
root@am68a-sk:/opt/edgeai-gst-apps# v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-framesizes pad=0,code=0x300f
ioctl: VIDIOC_SUBDEV_ENUM_FRAME_SIZE (pad=0,stream=0)
   Size Range: 1332x990 - 1332x990
root@am68a-sk:/opt/edgeai-gst-apps#

   root@am68a-sk:/opt/edgeai-gst-apps# v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-framesizes pad=0,code=0x3012
ioctl: VIDIOC_SUBDEV_ENUM_FRAME_SIZE (pad=0,stream=0)
   Size Range: 4056x3040 - 4056x3040
   Size Range: 2028x1520 - 2028x1520
   Size Range: 2028x1080 - 2028x1080

Modify the /opt/edgeai-gst-apps/scripts/setup_cameras.sh

setup_imx477(){
    IMX477_CAM_FMT='[fmt:SRGGB12_1X12/2028x1080]' 


 if runs the edgeai-gst-app, select camera it will crash



the crash log:
[   32.371914] ------------[ cut here ]------------
[   32.376529] WARNING: CPU: 1 PID: 1235 at drivers/media/common/videobuf2/videobuf2-core.c:1647 vb2_start_streaming+0xd8/0x15c [videobuf2_common]
[   32.389388] Modules linked in: xt_conntrack xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_addrtype iptable_filter ip_tables x_tables br_netfilter bridge stp )
[   32.389507]  fuse drm drm_panel_orientation_quirks ipv6
[   32.481655] CPU: 1 PID: 1235 Comm: v4l2src0:src Tainted: G           O       6.1.46 #1
[   32.489552] Hardware name: Texas Instruments AM68 SK (DT)
[   32.494933] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   32.501876] pc : vb2_start_streaming+0xd8/0x15c [videobuf2_common]
[   32.508047] lr : vb2_start_streaming+0x68/0x15c [videobuf2_common]
[   32.514214] sp : ffff80000b4cbb60
[   32.517514] x29: ffff80000b4cbb60 x28: ffff000832059400 x27: ffff000830ea14b8
[   32.524632] x26: 0000000000000000 x25: 0000000040045612 x24: 0000000000000000
[   32.531750] x23: ffff80000b4cbcc8 x22: ffff00082f479700 x21: ffff000830ea1438
[   32.538868] x20: ffff000830ea1450 x19: 00000000ffffffa1 x18: 00000000002feac5
[   32.545986] x17: 0000000000000020 x16: fffffc002405f808 x15: 0000000000000000
[   32.553104] x14: 000000000000025d x13: 0000000000000000 x12: 0000000000000000
[   32.560222] x11: 0000000000000000 x10: 00000000000009b0 x9 : ffff80000b4cba30
[   32.567339] x8 : ffff00084204df90 x7 : ffff000b7e1b3340 x6 : 0000000000000000
[   32.574457] x5 : 00000000410fd080 x4 : 0000000000c0000e x3 : ffff00082eddaa48
[   32.581574] x2 : 0000000000000000 x1 : ffff800000e6d000 x0 : ffff00084213c9f0
[   32.588693] Call trace:
[   32.591127]  vb2_start_streaming+0xd8/0x15c [videobuf2_common]
[   32.596951]  vb2_core_streamon+0x90/0x190 [videobuf2_common]
[   32.602598]  vb2_ioctl_streamon+0x5c/0xb0 [videobuf2_v4l2]
[   32.608078]  v4l_streamon+0x24/0x30 [videodev]
[   32.612534]  __video_do_ioctl+0x18c/0x3dc [videodev]
[   32.617504]  video_usercopy+0x21c/0x6d0 [videodev]
[   32.622307]  video_ioctl2+0x18/0x30 [videodev]
[   32.626758]  v4l2_ioctl+0x40/0x60 [videodev]
[   32.631034]  __arm64_sys_ioctl+0xa8/0xf0
[   32.634948]  invoke_syscall+0x48/0x114
[   32.638687]  el0_svc_common.constprop.0+0xd4/0xfc
[   32.643377]  do_el0_svc+0x30/0xd0
[   32.646680]  el0_svc+0x2c/0x84
[   32.649725]  el0t_64_sync_handler+0xbc/0x140
[   32.653981]  el0t_64_sync+0x18c/0x190
[   32.657631] ---[ end trace 0000000000000000 ]---
[   34.119055] cdns-csi2rx 4504000.csi-bridge: Failed to start streams 0x1 on subdev
[   34.136691] cdns-csi2rx 4504000.csi-bridge: Failed to stop stream0
[   34.152889] cdns-csi2rx 4504000.csi-bridge: Failed to stop stream1
[   34.169118] cdns-csi2rx 4504000.csi-bridge: Failed to stop stream2
[   34.185300] cdns-csi2rx 4504000.csi-bridge: Failed to stop stream3
[   34.196974] ------------[ cut here ]------------
[   34.201593] WARNING: CPU: 1 PID: 1248 at drivers/media/common/videobuf2/videobuf2-core.c:1647 vb2_start_streaming+0xd8/0x15c [videobuf2_common]
[   34.214455] Modules linked in: xt_conntrack xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_addrtype iptable_filter ip_tables x_tables br_netfilter bridge stp )
[   34.214582]  fuse drm drm_panel_orientation_quirks ipv6
[   34.306733] CPU: 1 PID: 1248 Comm: v4l2src0:src Tainted: G        W  O       6.1.46 #1
[   34.314631] Hardware name: Texas Instruments AM68 SK (DT)
[   34.320011] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   34.326954] pc : vb2_start_streaming+0xd8/0x15c [videobuf2_common]
[   34.333125] lr : vb2_start_streaming+0x68/0x15c [videobuf2_common]
[   34.339294] sp : ffff80000b7a3b60
[   34.342594] x29: ffff80000b7a3b60 x28: ffff000830ca4900 x27: ffff000830ea14b8
[   34.349713] x26: 0000000000000000 x25: 0000000040045612 x24: 0000000000000000
[   34.356831] x23: ffff80000b7a3cc8 x22: ffff000833959f00 x21: ffff000830ea1438
[   34.363949] x20: ffff000830ea1450 x19: 00000000ffffffa1 x18: 00000000002fb867
[   34.371067] x17: 0000000000000020 x16: fffffc002405f808 x15: 0000003deadedbfe
[   34.378185] x14: 0000000000000322 x13: 0000000000000000 x12: 0000000000000000
[   34.385302] x11: 0000000000000000 x10: 00000000000009b0 x9 : ffff80000b7a3a30
[   34.392420] x8 : ffff00082d78a690 x7 : ffff000b7e1b3340 x6 : 0000000000000000
[   34.399537] x5 : 00000000410fd080 x4 : 0000000000c0000e x3 : ffff00082edda348
[   34.406655] x2 : 0000000000000000 x1 : ffff800000e6d000 x0 : ffff00084214b9f0
[   34.413773] Call trace:
[   34.416207]  vb2_start_streaming+0xd8/0x15c [videobuf2_common]
[   34.422030]  vb2_core_streamon+0x90/0x190 [videobuf2_common]
[   34.427677]  vb2_ioctl_streamon+0x5c/0xb0 [videobuf2_v4l2]
[   34.433155]  v4l_streamon+0x24/0x30 [videodev]
[   34.437612]  __video_do_ioctl+0x18c/0x3dc [videodev]
[   34.442584]  video_usercopy+0x21c/0x6d0 [videodev]
[   34.447380]  video_ioctl2+0x18/0x30 [videodev]
[   34.451830]  v4l2_ioctl+0x40/0x60 [videodev]
[   34.456106]  __arm64_sys_ioctl+0xa8/0xf0
[   34.460020]  invoke_syscall+0x48/0x114
[   34.463759]  el0_svc_common.constprop.0+0xd4/0xfc
[   34.468450]  do_el0_svc+0x30/0xd0
[   34.471754]  el0_svc+0x2c/0x84
[   34.474799]  el0t_64_sync_handler+0xbc/0x140
[   34.479057]  el0t_64_sync+0x18c/0x190
[   34.482707] ---[ end trace 0000000000000000 ]---

Could you help me confirm which step is wrong or what steps are still missing? thanks

0 Praveen Rao over 1 year ago in reply to csscyt

TI__Mastermind 48483 points

Our expert handling this topic is currently out of the office; please expect a 1-2 day delay in response.

Thanks.

0 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Hello,

Try running the below command to grab a frame and share the output:

v4l2-ctl --verbose --device /dev/video-rpi-cam0 --set-fmt-video=width=2028,height=1080 --stream-mmap --stream-to=test-frame.raw --stream-count=1

Thanks,

Fabiana

0 csscyt over 1 year ago in reply to Fabiana Jaimes

Prodigy 210 points

I used SDK version: ti-processor-sdk-linux-edgeai-j721s2-evm-09_02_00_05

port imx477.c from raspberrypi
and connected two cameras(imx219 and imx477),

The driver is probed and I2C communication works,
see the log:

root@am68a-sk:/opt/edgeai-gst-apps# dmesg | grep -i imx
[    4.575593] imx219 5-0010: supply VANA not found, using dummy regulator
[    4.592375] imx219 5-0010: supply VDIG not found, using dummy regulator
[    4.592410] imx477 4-001a: supply dovdd not found, using dummy regulator
[    4.607551] imx219 5-0010: supply VDDL not found, using dummy regulator
[    4.614484] imx477 4-001a: supply avdd not found, using dummy regulator
[    4.621566] imx477 4-001a: supply dvdd not found, using dummy regulator

[    7.145908] imx477 4-001a: Consider updating driver imx477 to match on endpoints
[    7.506788] imx219 5-0010: Consider updating driver imx219 to match on endpoints



IMX219 Camera 1 detected
    device = /dev/video-imx219-cam0
    name = imx219
    format = [fmt:SRGGB8_1X8/1920x1080]
    subdev_id = /dev/v4l-imx219-subdev0
    isp_required = yes
IMX477 Camera 0 detected
    device = /dev/video-imx477-cam0
    name = imx477
    format = [fmt:SRGGB10_1X10/1332x990]
    subdev_id = /dev/v4l-imx477-subdev0
    isp_required = yes

imx219 camera can work，

but when i select IMX477 camera，there is no display and no incorrect log output

if I use v4l2-ctl command to grab a frame，they will stuck and can not return, Also including imx219

imx477:
root@am68a-sk:/opt/edgeai-gst-apps# v4l2-ctl --verbose --device /dev/video-imx477-cam0  --set-fmt-video=width=1332,height=990 --stream-mmap --stream-to=test-frame.raw --stream-count=1                     
VIDIOC_QUERYCAP: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
   Width/Height      : 1328/990
   Pixel Format      : 'UYVY' (UYVY 4:2:2)
   Field             : None
   Bytes per Line    : 2656
   Size Image        : 2629440
   Colorspace        : sRGB
   Transfer Function : Default (maps to sRGB)
   YCbCr/HSV Encoding: Default (maps to ITU-R 601)
   Quantization      : Default (maps to Limited Range)
   Flags             : 
        VIDIOC_REQBUFS returned 0 (Success)
        VIDIOC_QUERYBUF returned 0 (Success)
        VIDIOC_QUERYBUF returned 0 (Success)
        VIDIOC_QUERYBUF returned 0 (Success)
        VIDIOC_QUERYBUF returned 0 (Success)
        VIDIOC_QBUF returned 0 (Success)
        VIDIOC_QBUF returned 0 (Success)
        VIDIOC_QBUF returned 0 (Success)
        VIDIOC_QBUF returned 0 (Success)
        VIDIOC_STREAMON returned 0 (Success)
        
    Stuck here!!!!
  
  
  
    
    imx219:
    
    root@am68a-sk:/opt/edgeai-gst-apps# v4l2-ctl --verbose --device /dev/video-imx219-cam0  --set-fmt-video=width=2028,height=1080 --stream-mmap --stream-to=test-frame.raw --stream-count=1                    
VIDIOC_QUERYCAP: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
   Width/Height      : 2024/1080
   Pixel Format      : 'UYVY' (UYVY 4:2:2)
   Field             : None
   Bytes per Line    : 4048
   Size[  244.125984] imx219 5-0010: imx219_start_streaming
 Image        : 4371840
   Colorspace        : sRGB
   Transfer Function : Default (maps to sRGB)
   YCbCr/HSV Encoding: Default (maps to ITU-R 601)
   Quantization      : Default (maps to Limited Range)
   Flags             : 
        VIDIOC_REQBUFS returned 0 (Success)
        VIDIOC_QUERYBUF returned 0 (Success)
        VIDIOC_QUERYBUF returned 0 (Success)
        VIDIOC_QUERYBUF returned 0 (Success)
        VIDIOC_QUERYBUF returned 0 (Success)
        VIDIOC_QBUF returned 0 (Success)
        VIDIOC_QBUF returned 0 (Success)
        VIDIOC_QBUF returned 0 (Success)
        VIDIOC_QBUF returned 0 (Success)
        VIDIOC_STREAMON returned 0 (Success)
        
    stuck here

Do you know the reason why imx477 doesn't work and how to debug this issue

and does TI have plan to support IMX477 in the subsequent versions?

0 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Hello,

If the camera only produces raw RGB or RGBD images, then further ISP processing is required which can be handled by the Vision Imaging Sub System (VISS) hardware accelerator. For more on ISP tuning, please see the following application note: https://www.ti.com/lit/an/sprad86a/sprad86a.pdf

We are still discussing internally about looking into IMX477 or another similar sensor and if it is something we can support soon.

Thank you,

Fabiana

0 csscyt over 1 year ago in reply to Fabiana Jaimes

Prodigy 210 points

Thank you for your reply.

followed the ISP tuning guide,
I have copied the generated DCC binary files to the target board,

/opt/imaging/imx477/linear/dcc_2a.bin

/opt/imaging/imx477/linear/dcc_ldc.bin

/opt/imaging/imx477/linear/dcc_viss.bin

if select the IMX477 camera via the below edgeAI GUI,

Do I need to set the path of the DCC file, that is,how can the GUI find the newly added DCC file of imx477

If it is needed, where to config it?

+1 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Hello,

The way the custom gui application works is by generating a yaml/configuration file to generate and run a GStreamer pipeline using the provided inputs. Because IMX477 is not supported out of box, the sensor id for this sensor is missing so the format cannot be set to rggb. When enabling a new sensor, you should test functionality by running GStreamer pipelines in the command line. Here is an example for IMX219:

gst-launch-1.0 v4l2src device=/dev/video-imx219-cam0 io-mode=5 ! queue leaky=2 ! \
video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb ! \
tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_SONY_IMX219_RPI" \
dcc-isp-file=/opt/imaging/imx219/dcc_viss.bin \
sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a.bin format-msb=7 ! \
tiovxmultiscaler ! \
video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! \
kmssink driver-name=tidss sync=false

Thank you,

Fabiana

0 csscyt over 1 year ago in reply to Fabiana Jaimes

Prodigy 210 points

These are my two camera devices(media0 is IMX477, media1 is IMX219)

root@am68a-sk:/opt/edgeai-gst-apps# v4l2-ctl --list-devices
j721e-csi2rx (platform:4500000.ticsi2rx):
        /dev/video2
        /dev/video3
        /dev/video4
        /dev/video5
        /dev/video6
        /dev/video7
        /dev/video8
        /dev/video9
        /dev/media0

j721e-csi2rx (platform:4510000.ticsi2rx):
        /dev/video10
        /dev/video11
        /dev/video12
        /dev/video13
        /dev/video14
        /dev/video15
        /dev/video16
        /dev/video17
        /dev/media1

wave5-dec (platform:wave5-dec):
        /dev/video0

wave5-enc (platform:wave5-enc):
        /dev/video1

check the media0 device topology,

root@am68a-sk:/opt/edgeai-gst-apps# media-ctl -p -d /dev/media0
Media controller API version 6.1.80

Media device information
------------------------
driver          j721e-csi2rx
model           TI-CSI2RX
serial          
bus info        platform:4500000.ticsi2rx
hw revision     0x1
driver version  6.1.80

Device topology
- entity 1: 4500000.ticsi2rx (9 pads, 9 links, 1 route)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        routes:
                0/0 -> 1/0 [ACTIVE]
        pad0: Sink
                [stream:0 fmt:UYVY8_1X16/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                <- "cdns_csi2rx.4504000.csi-bridge":1 [ENABLED,IMMUTABLE]
        pad1: Source
                [stream:0 fmt:UYVY8_1X16/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                -> "4500000.ticsi2rx context 0":0 [ENABLED,IMMUTABLE]
        pad2: Source
                -> "4500000.ticsi2rx context 1":0 [ENABLED,IMMUTABLE]
        pad3: Source
                -> "4500000.ticsi2rx context 2":0 [ENABLED,IMMUTABLE]
        pad4: Source
                -> "4500000.ticsi2rx context 3":0 [ENABLED,IMMUTABLE]
        pad5: Source
                -> "4500000.ticsi2rx context 4":0 [ENABLED,IMMUTABLE]
        pad6: Source
                -> "4500000.ticsi2rx context 5":0 [ENABLED,IMMUTABLE]
        pad7: Source
                -> "4500000.ticsi2rx context 6":0 [ENABLED,IMMUTABLE]
        pad8: Source
                -> "4500000.ticsi2rx context 7":0 [ENABLED,IMMUTABLE]

- entity 11: cdns_csi2rx.4504000.csi-bridge (5 pads, 2 links, 1 route)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev1
        routes:
                0/0 -> 1/0 [ACTIVE]
        pad0: Sink
                [stream:0 fmt:SRGGB10_1X10/4056x3040 field:none colorspace:raw xfer:none]
                <- "imx477 4-001a":0 [ENABLED,IMMUTABLE]
        pad1: Source
                [stream:0 fmt:SRGGB10_1X10/4056x3040 field:none colorspace:raw xfer:none]
                -> "4500000.ticsi2rx":0 [ENABLED,IMMUTABLE]
        pad2: Source
        pad3: Source
        pad4: Source

- entity 17: imx477 4-001a (1 pad, 1 link, 0 route)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev2
        pad0: Source
                [stream:0 fmt:SRGGB10_1X10/4056x3040 field:none colorspace:raw xfer:none]
                -> "cdns_csi2rx.4504000.csi-bridge":0 [ENABLED,IMMUTABLE]

- entity 23: 4500000.ticsi2rx context 0 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video2
        pad0: Sink
                <- "4500000.ticsi2rx":1 [ENABLED,IMMUTABLE]

- entity 29: 4500000.ticsi2rx context 1 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video3
        pad0: Sink
                <- "4500000.ticsi2rx":2 [ENABLED,IMMUTABLE]

- entity 35: 4500000.ticsi2rx context 2 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video4
        pad0: Sink
                <- "4500000.ticsi2rx":3 [ENABLED,IMMUTABLE]

- entity 41: 4500000.ticsi2rx context 3 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video5
        pad0: Sink
                <- "4500000.ticsi2rx":4 [ENABLED,IMMUTABLE]

- entity 47: 4500000.ticsi2rx context 4 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video6
        pad0: Sink
                <- "4500000.ticsi2rx":5 [ENABLED,IMMUTABLE]

- entity 53: 4500000.ticsi2rx context 5 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video7
        pad0: Sink
                <- "4500000.ticsi2rx":6 [ENABLED,IMMUTABLE]

- entity 59: 4500000.ticsi2rx context 6 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video8
        pad0: Sink
                <- "4500000.ticsi2rx":7 [ENABLED,IMMUTABLE]

- entity 65: 4500000.ticsi2rx context 7 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video9
        pad0: Sink
                <- "4500000.ticsi2rx":8 [ENABLED,IMMUTABLE]

check the media1 device topology,

root@am68a-sk:/opt/edgeai-gst-apps# media-ctl -p -d /dev/media1
Media controller API version 6.1.80

Media device information
------------------------
driver          j721e-csi2rx
model           TI-CSI2RX
serial          
bus info        platform:4510000.ticsi2rx
hw revision     0x1
driver version  6.1.80

Device topology
- entity 1: 4510000.ticsi2rx (9 pads, 9 links, 1 route)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev3
        routes:
                0/0 -> 1/0 [ACTIVE]
        pad0: Sink
                [stream:0 fmt:UYVY8_1X16/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                <- "cdns_csi2rx.4514000.csi-bridge":1 [ENABLED,IMMUTABLE]
        pad1: Source
                [stream:0 fmt:UYVY8_1X16/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                -> "4510000.ticsi2rx context 0":0 [ENABLED,IMMUTABLE]
        pad2: Source
                -> "4510000.ticsi2rx context 1":0 [ENABLED,IMMUTABLE]
        pad3: Source
                -> "4510000.ticsi2rx context 2":0 [ENABLED,IMMUTABLE]
        pad4: Source
                -> "4510000.ticsi2rx context 3":0 [ENABLED,IMMUTABLE]
        pad5: Source
                -> "4510000.ticsi2rx context 4":0 [ENABLED,IMMUTABLE]
        pad6: Source
                -> "4510000.ticsi2rx context 5":0 [ENABLED,IMMUTABLE]
        pad7: Source
                -> "4510000.ticsi2rx context 6":0 [ENABLED,IMMUTABLE]
        pad8: Source
                -> "4510000.ticsi2rx context 7":0 [ENABLED,IMMUTABLE]

- entity 11: cdns_csi2rx.4514000.csi-bridge (5 pads, 2 links, 1 route)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev4
        routes:
                0/0 -> 1/0 [ACTIVE]
        pad0: Sink
                [stream:0 fmt:SRGGB8_1X8/1920x1080 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range]
                <- "imx219 5-0010":0 [ENABLED,IMMUTABLE]
        pad1: Source
                [stream:0 fmt:SRGGB8_1X8/1920x1080 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range]
                -> "4510000.ticsi2rx":0 [ENABLED,IMMUTABLE]
        pad2: Source
        pad3: Source
        pad4: Source

- entity 17: imx219 5-0010 (1 pad, 1 link, 0 route)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev5
        pad0: Source
                [stream:0 fmt:SRGGB8_1X8/1920x1080 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
                 crop.bounds:(8,8)/3280x2464
                 crop:(688,700)/1920x1080]
                -> "cdns_csi2rx.4514000.csi-bridge":0 [ENABLED,IMMUTABLE]

- entity 23: 4510000.ticsi2rx context 0 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video10
        pad0: Sink
                <- "4510000.ticsi2rx":1 [ENABLED,IMMUTABLE]

- entity 29: 4510000.ticsi2rx context 1 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video11
        pad0: Sink
                <- "4510000.ticsi2rx":2 [ENABLED,IMMUTABLE]

- entity 35: 4510000.ticsi2rx context 2 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video12
        pad0: Sink
                <- "4510000.ticsi2rx":3 [ENABLED,IMMUTABLE]

- entity 41: 4510000.ticsi2rx context 3 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video13
        pad0: Sink
                <- "4510000.ticsi2rx":4 [ENABLED,IMMUTABLE]

- entity 47: 4510000.ticsi2rx context 4 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video14
        pad0: Sink
                <- "4510000.ticsi2rx":5 [ENABLED,IMMUTABLE]

- entity 53: 4510000.ticsi2rx context 5 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video15
        pad0: Sink
                <- "4510000.ticsi2rx":6 [ENABLED,IMMUTABLE]

- entity 59: 4510000.ticsi2rx context 6 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video16
        pad0: Sink
                <- "4510000.ticsi2rx":7 [ENABLED,IMMUTABLE]

- entity 65: 4510000.ticsi2rx context 7 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video17
        pad0: Sink
                <- "4510000.ticsi2rx":8 [ENABLED,IMMUTABLE]

for IMX219, the following command works,

gst-launch-1.0  v4l2src device=/dev/video-imx219-cam0 io-mode=5 ! queue leaky=2 ! \
video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb ! \
tiovxisp sink_0::device=/dev/v4l-subdev4 sensor-name="SENSOR_SONY_IMX219_RPI" \
dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss.bin \
sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a.bin format-msb=7 ! \
tiovxmultiscaler ! \
video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! \
kmssink driver-name=tidss sync=false

for IMX477, I made the software changes followed the ISP tuning guide, and use the following command, it does't work,

could you help check this command, "tiovxisp sink_0::device=/dev/v4l-subdev1" it is correct accoring to

the above device tolopoly?

gst-launch-1.0  v4l2src device=/dev/video-imx477-cam0 io-mode=5 ! queue leaky=2 ! \
video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb ! \
tiovxisp sink_0::device=/dev/v4l-subdev1 sensor-name="SENSOR_SONY_IMX477_RPI" \
dcc-isp-file=/opt/imaging/imx477/linear/dcc_viss.bin \
sink_0::dcc-2a-file=/opt/imaging/imx477/linear/dcc_2a.bin format-msb=7 ! \
tiovxmultiscaler ! \
video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! \
kmssink driver-name=tidss sync=false

I enabled the gst-debug log and added some logs in gsttivoxisp.c and gsttiovxmiso.c,

compared the logs between IMX219 and IMX477,

for IMX477, the initialization function(gst_tiovx_isp_init_module) was not called,and it will stuck there

Could you help me check whether there are still any missing or wrong configurations? thanks.

0 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Hi,

Could you try running the following configuration and share the output?

gst-launch-1.0  v4l2src device=/dev/video-imx477-cam0 io-mode=5 ! queue leaky=2 ! \
video/x-bayer, width=1332, height=990, framerate=30/1, format=rggb10 ! \
tiovxisp sink_0::device=/dev/v4l-imx477-subdev0 sensor-name="SENSOR_SONY_IMX477_RPI" \
dcc-isp-file=/opt/imaging/imx477/linear/dcc_viss.bin \
sink_0::dcc-2a-file=/opt/imaging/imx477/linear/dcc_2a.bin format-msb=7 ! \
tiovxmultiscaler ! \
video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! \
kmssink driver-name=tidss sync=false

Thank you,

Fabiana

0 csscyt over 1 year ago in reply to Fabiana Jaimes

Prodigy 210 points

I run the above command, now it will not stuck, and the screen displays as follows,

I use this configuration for ISP tuning,

SENSOR_ID 477 
PRJ_DIR ../imx477_output
SENSOR_NAME imx477
SENSOR_DCC_NAME IMX477-UB953_D3

SENSOR_WIDTH 1920
SENSOR_HEIGHT 1080

# 0=RGGB; 1=GRBG; 2=GBRG; 3=BGGR, 4=MONO
COLOR_PATTERN 0

# sensor mode: 0 for linear (no decompanding), 1 for WDR (decompanding)
WDR_MODE 0

# raw sensor image BIT_DEPTH: it may be 8, 10, or 12 for linear sensors; typically 12 for WDR mode because of companding
BIT_DEPTH 10

# WDR BIT_DEPTH: WDR raw sensor image bitdepth after decompanding, typically 20 or 24
WDR_BIT_DEPTH 20

# WDR decompanding knee points (comma separated without spaces in between)
WDR_KNEE_X 0,512,1408,2176,4095,65535
WDR_KNEE_Y 0,2048,16384,65536,1048063,1048063

# Sensor black level to subtract before decompanding (for linear sensors only and some Sony WDR sensors)
BLACK_PRE  0

# Sensor black level to subtract after decompanding (for most WDR sensors and all linear sensors)
BLACK_POST  0

# GAMMA value for compressing 20/24-bit WDR raw to 16-bit ISP internal
# typically around 50 (0.5) for 24-bit WDR sensors and 70 (0.7) for 20-bit sensors
GAMMA_PRE 70

# LSB location for H3A input bit range (from bit-H3A_INPUT_LSB to bit-H3A_INPUT_LSB+9) 
H3A_INPUT_LSB 2

and same with IMX219 for the exposure setting in gsttiovxisp.c,

check the imx477 camera configuration as this,

root@am68a-sk:/opt/edgeai-gst-apps# v4l2-ctl -d /dev/v4l-imx477-subdev0 --list-ctrls

User Controls

                       exposure 0x00980911 (int)    : min=4 max=1028 step=1 default=1028 value=1028
                horizontal_flip 0x00980914 (bool)   : default=0 value=0 flags=modify-layout
                  vertical_flip 0x00980915 (bool)   : default=0 value=0 flags=modify-layout

Image Source Controls

              vertical_blanking 0x009e0901 (int)    : min=60 max=8383010 step=1 default=60 value=60
            horizontal_blanking 0x009e0902 (int)    : min=5332 max=65520 step=1 default=5332 value=5332
                  analogue_gain 0x009e0903 (int)    : min=0 max=978 step=1 default=0 value=224
                red_pixel_value 0x009e0904 (int)    : min=0 max=4095 step=1 default=4095 value=4095
          green_red_pixel_value 0x009e0905 (int)    : min=0 max=4095 step=1 default=4095 value=4095
               blue_pixel_value 0x009e0906 (int)    : min=0 max=4095 step=1 default=4095 value=4095
         green_blue_pixel_value 0x009e0907 (int)    : min=0 max=4095 step=1 default=4095 value=4095

Image Processing Controls

                 link_frequency 0x009f0901 (intmenu): min=0 max=0 default=0 value=0 (456000000 0x1b2e0200) flags=read-only
                     pixel_rate 0x009f0902 (int64)  : min=840000000 max=840000000 step=1 default=840000000 value=840000000 flags=read-only
                   test_pattern 0x009f0903 (menu)   : min=0 max=4 default=0 value=0 (Disabled)
                   digital_gain 0x009f0905 (int)    : min=256 max=65535 step=1 default=256 value=256

Could you help me check which configurations are incorrect?

0 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Hi,

I have looped in our ISP tuning expert. Please expect a response from them soon.

Thanks,

Fabiana

0 Gang Hua over 1 year ago in reply to Fabiana Jaimes

TI__Guru 61180 points

Hi csscyt,

The configuration file looks good to me.
Before ISP processing, we need to verify the raw image from sensor.

Are you able to capture some valid raw image frames from the sensor?

0 csscyt over 1 year ago in reply to Gang Hua

Prodigy 210 points

I used these commands,

media-ctl -V '"imx477 4-001a":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'

gst-launch-1.0 -v v4l2src num-buffers=5 device=/dev/video-imx477-cam0 io-mode=dmabuf ! \

video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb10 ! \

multifilesink location="imx477-image-%d.raw"

imx477_raw.tar.gz

0 Gang Hua over 1 year ago in reply to csscyt

TI__Guru 61180 points

Hi csscyt,

Thanks for sharing!

These raw images do not look correct.
The pixels on top ~686 lines are around 0x40 and the rest are all 0s.
That is the same as what you see on the monitor.

There might be something wrong in the sensor driver.

0 csscyt over 1 year ago in reply to Gang Hua

Prodigy 210 points

Hi,

I have verified that the IMX477 camera hardware is normal on other platforms,

I tried different modes（10/12bit with different resolutions） to capture the raw image frames from the sensor,
The raw image data is always similar to that in the previous post,

Do you know which configurations in the sensor driver might lead to this phenomenon, or what other configurations there could be?

0 Gang Hua over 1 year ago in reply to csscyt

TI__Guru 61180 points

Hi csscyt,

I am not familiar with IMX477.
Is there a test pattern you may turn on in IMX477?

Fabiana may have some past experience with that sensor.

0 csscyt over 1 year ago in reply to Gang Hua

Prodigy 210 points

@Fabiana
Do you have any suggestions? thanks

0 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Hi csscyt,

IMX477 is not currently supported and I have not worked with it before, but I could review the sensor driver. Could you share it so I could take a look?

Thank you,

Fabiana

0 csscyt over 1 year ago in reply to Fabiana Jaimes

Prodigy 210 points

Hi,

I used this code (https://github.com/raspberrypi/linux/blob/rpi-6.6.y/drivers/media/i2c/imx477.c)
and tried the following configurations with different modes（10/12bit + different resolutions）

#define IMX477_DEFAULT_LINK_FREQ 450000000
#define IMX477_PIXEL_RATE 840000000

#define IMX477_DEFAULT_LINK_FREQ 900000000
#define IMX477_PIXEL_RATE 840000000

#define IMX477_DEFAULT_LINK_FREQ 900000000
#define IMX477_PIXEL_RATE 420000000

0 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Hi,

Please ensure that the link and clock frequencies match between the driver and dtso.

Thank you,

Fabiana

0 csscyt over 1 year ago in reply to Fabiana Jaimes

Prodigy 210 points

Yes, they are matched.since there would be a hardware check error if they weren't.

0 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Is the dtso you previously shared in this thread the one you are currently using?

Thank you,

Fabiana

0 csscyt over 1 year ago in reply to Fabiana Jaimes

Prodigy 210 points

// SPDX-License-Identifier: GPL-2.0
/**
 * DT Overlay for RPi Camera V2.1 (Sony imx477) interfaced with CSI2 on AM68-SK board.
 * https://datasheets.raspberrypi.org/camera/camera-v2-schematic.pdf
 *
 * Copyright (C) 2023 Texas Instruments Incorporated - https://www.ti.com/
 */

/dts-v1/;
/plugin/;

#include <dt-bindings/gpio/gpio.h>
#include "k3-pinctrl.h"

&{/} {
	clk_imx477_fixed: imx477-xclk {
		compatible = "fixed-clock";
		#clock-cells = <0>;
		clock-frequency = <24000000>;
	};

	clk_imx219_fixed: imx219-xclk {
		compatible = "fixed-clock";
		#clock-cells = <0>;
		clock-frequency = <24000000>;
	};


};

&exp3 {
	p01-hog {
		/* CSI_MUX_SEL_2 */
		gpio-hog;
		gpios = <1 GPIO_ACTIVE_HIGH>;
		output-high;
		line-name = "CSI_MUX_SEL_2";
	};
	p00-hog {
		 /* I2C voltage - set to 3.3V */
		 gpio-hog;
		 gpios = <0 GPIO_ACTIVE_HIGH>;
		 output-high;
		 line-name = "CSI_VIO_SELCSI_VIO_SEL";
    };
};

&main_i2c1 {
	status = "okay";
	#address-cells = <1>;
	#size-cells = <0>;

	i2c-switch@70 {
		compatible = "nxp,pca9543";
		#address-cells = <1>;
		#size-cells = <0>;
		reg = <0x70>;

		i3c-alias-pool = /bits/ 17 <0x10 0x11>;

		/* CAM0 I2C */
		cam0_i2c: i2c@0 {
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <0>;

			imx477_0: imx477_0@10 {
				compatible = "sony,imx477";
				reg = <0x1a>;

				clocks = <&clk_imx477_fixed>;
				clock-names = "xclk";
				reset-gpios = <&exp3 3 GPIO_ACTIVE_HIGH>;

				port {
					csi2_cam0: endpoint {
						remote-endpoint = <&csi2rx0_in_sensor>;
						link-frequencies = /bits/ 64 <450000000>;
						clock-lanes = <0>;
						data-lanes = <1 2>;
					};
				};
			};
		};

		/* CAM1 I2C */
		cam1_i2c: i2c@1 {
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <1>;

			imx219_1: imx219_1@10 {
				compatible = "sony,imx219";
				reg = <0x10>;

				clocks = <&clk_imx219_fixed>;
				clock-names = "xclk";

				reset-gpios = <&exp3 4 GPIO_ACTIVE_HIGH>;

				port {
					csi2_cam1: endpoint {
						remote-endpoint = <&csi2rx1_in_sensor>;
						link-frequencies = /bits/ 64 <456000000>;
						clock-lanes = <0>;
						data-lanes = <1 2>;
					};
				};
			};
		};
	};
};

&csi0_port0 {
	status = "okay";
	csi2rx0_in_sensor: endpoint {
		remote-endpoint = <&csi2_cam0>;
		bus-type = <4>; /* CSI2 DPHY. */
		clock-lanes = <0>;
		data-lanes = <1 2>;
	};
};

&csi1_port0 {
	status = "okay";
	csi2rx1_in_sensor: endpoint {
		remote-endpoint = <&csi2_cam1>;
		bus-type = <4>; /* CSI2 DPHY. */
		clock-lanes = <0>;
		data-lanes = <1 2>;
	};
};

0 csscyt over 1 year ago in reply to csscyt

Prodigy 210 points

Hi,

It supports RAW10/12 formats for imx477 sensor driver,

now i use this format ( fmt:SRGGB12_1X12/2028x1520)
check the configuration by media-ctl

root@am68a-sk:/opt/edgeai-gst-apps# media-ctl -d /dev/media0 -p
Media controller API version 6.1.80[   48.543307] 


Media device information
------------------------
driver          j721e-csi2rx
model           TI-CSI2RX
serial          
bus info        platform:4500000.ticsi2rx

hw revision     0x1
driver version  6.1.80

Device topology
- entity 1: 4500000.ticsi2rx (9 pads, 9 links, 1 route)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
	routes:
		0/0 -> 1/0 [ACTIVE]
	pad0: Sink
		[stream:0 fmt:SRGGB12_1X12/2280x1520 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
		<- "cdns_csi2rx.4504000.csi-bridge":1 [ENABLED,IMMUTABLE]
	pad1: Source
		[stream:0 fmt:SRGGB12_1X12/2280x1520 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
		-> "4500000.ticsi2rx context 0":0 [ENABLED,IMMUTABLE]
	pad2: Source
		-> "4500000.ticsi2rx context 1":0 [ENABLED,IMMUTABLE]
	pad3: Source
		-> "4500000.ticsi2rx context 2":0 [ENABLED,IMMUTABLE]
	pad4: Source
		-> "4500000.ticsi2rx context 3":0 [ENABLED,IMMUTABLE]
	pad5: Source
		-> "4500000.ticsi2rx context 4":0 [ENABLED,IMMUTABLE]
	pad6: Source
		-> "4500000.ticsi2rx context 5":0 [ENABLED,IMMUTABLE]
	pad7: Source
		-> "4500000.ticsi2rx context 6":0 [ENABLED,IMMUTABLE]
	pad8: Source
		-> "4500000.ticsi2rx context 7":0 [ENABLED,IMMUTABLE]

- entity 11: cdns_csi2rx.4504000.csi-bridge (5 pads, 2 links, 1 route)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev1
	routes:
		0/0 -> 1/0 [ACTIVE]
	pad0: Sink
		[stream:0 fmt:SRGGB12_1X12/2028x1520 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range]
		<- "imx477 4-001a":0 [ENABLED,IMMUTABLE]
	pad1: Source
		[stream:0 fmt:SRGGB12_1X12/2028x1520 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range]
		-> "4500000.ticsi2rx":0 [ENABLED,IMMUTABLE]
	pad2: Source
	pad3: Source
	pad4: Source

- entity 17: imx477 4-001a (1 pad, 1 link, 0 route)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev2
	pad0: Source
		[stream:0 fmt:SRGGB12_1X12/2028x1520 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range
		 crop.bounds:(8,16)/4056x3040
		 crop:(8,16)/4056x3040]
		-> "cdns_csi2rx.4504000.csi-bridge":0 [ENABLED,IMMUTABLE]

- entity 23: 4500000.ticsi2rx context 0 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video2
	pad0: Sink
		<- "4500000.ticsi2rx":1 [ENABLED,IMMUTABLE]

- entity 29: 4500000.ticsi2rx context 1 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video3
	pad0: Sink
		<- "4500000.ticsi2rx":2 [ENABLED,IMMUTABLE]

- entity 35: 4500000.ticsi2rx context 2 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video4
	pad0: Sink
		<- "4500000.ticsi2rx":3 [ENABLED,IMMUTABLE]

- entity 41: 4500000.ticsi2rx context 3 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video5
	pad0: Sink
		<- "4500000.ticsi2rx":4 [ENABLED,IMMUTABLE]

- entity 47: 4500000.ticsi2rx context 4 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video6
	pad0: Sink
		<- "4500000.ticsi2rx":5 [ENABLED,IMMUTABLE]

- entity 53: 4500000.ticsi2rx context 5 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video7
	pad0: Sink
		<- "4500000.ticsi2rx":6 [ENABLED,IMMUTABLE]

- entity 59: 4500000.ticsi2rx context 6 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video8
	pad0: Sink
		<- "4500000.ticsi2rx":7 [ENABLED,IMMUTABLE]

- entity 65: 4500000.ticsi2rx context 7 (1 pad, 1 link, 0 route)
             type Node subtype V4L flags 0
             device node name /dev/video9
	pad0: Sink
		<- "4500000.ticsi2rx":8 [ENABLED,IMMUTABLE]

but Only 8 and 16 bit bpp in the format array of j721e-csi2rx.c,

static const struct ti_csi2rx_fmt formats[] = {
	{
		.fourcc			= V4L2_PIX_FMT_YUYV,
		.code			= MEDIA_BUS_FMT_YUYV8_1X16,
		.csi_dt			= MIPI_CSI2_DT_YUV422_8B,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_8,
	}, {
		.fourcc			= V4L2_PIX_FMT_UYVY,
		.code			= MEDIA_BUS_FMT_UYVY8_1X16,
		.csi_dt			= MIPI_CSI2_DT_YUV422_8B,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_8,
	}, {
		.fourcc			= V4L2_PIX_FMT_YVYU,
		.code			= MEDIA_BUS_FMT_YVYU8_1X16,
		.csi_dt			= MIPI_CSI2_DT_YUV422_8B,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_8,
	}, {
		.fourcc			= V4L2_PIX_FMT_VYUY,
		.code			= MEDIA_BUS_FMT_VYUY8_1X16,
		.csi_dt			= MIPI_CSI2_DT_YUV422_8B,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_8,
	}, {
		.fourcc			= V4L2_PIX_FMT_SBGGR8,
		.code			= MEDIA_BUS_FMT_SBGGR8_1X8,
		.csi_dt			= MIPI_CSI2_DT_RAW8,
		.bpp			= 8,
		.size			= SHIM_DMACNTX_SIZE_8,
	}, {
		.fourcc			= V4L2_PIX_FMT_SGBRG8,
		.code			= MEDIA_BUS_FMT_SGBRG8_1X8,
		.csi_dt			= MIPI_CSI2_DT_RAW8,
		.bpp			= 8,
		.size			= SHIM_DMACNTX_SIZE_8,
	}, {
		.fourcc			= V4L2_PIX_FMT_SGRBG8,
		.code			= MEDIA_BUS_FMT_SGRBG8_1X8,
		.csi_dt			= MIPI_CSI2_DT_RAW8,
		.bpp			= 8,
		.size			= SHIM_DMACNTX_SIZE_8,
	}, {
		.fourcc			= V4L2_PIX_FMT_SRGGB8,
		.code			= MEDIA_BUS_FMT_SRGGB8_1X8,
		.csi_dt			= MIPI_CSI2_DT_RAW8,
		.bpp			= 8,
		.size			= SHIM_DMACNTX_SIZE_8,
	}, {
		.fourcc			= V4L2_PIX_FMT_SBGGR10,
		.code			= MEDIA_BUS_FMT_SBGGR10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SGBRG10,
		.code			= MEDIA_BUS_FMT_SGBRG10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SGRBG10,
		.code			= MEDIA_BUS_FMT_SGRBG10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SRGGB10,
		.code			= MEDIA_BUS_FMT_SRGGB10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SBGGR12,
		.code			= MEDIA_BUS_FMT_SBGGR12_1X12,
		.csi_dt			= MIPI_CSI2_DT_RAW12,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SGBRG12,
		.code			= MEDIA_BUS_FMT_SGBRG12_1X12,
		.csi_dt			= MIPI_CSI2_DT_RAW12,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SGRBG12,
		.code			= MEDIA_BUS_FMT_SGRBG12_1X12,
		.csi_dt			= MIPI_CSI2_DT_RAW12,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,

	}, {
		.fourcc			= V4L2_PIX_FMT_SRGGB12,
		.code			= MEDIA_BUS_FMT_SRGGB12_1X12,
		.csi_dt			= MIPI_CSI2_DT_RAW12,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {

		.fourcc			= V4L2_PIX_FMT_SRGGI10,
		.code			= MEDIA_BUS_FMT_SRGGI10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SGRIG10,
		.code			= MEDIA_BUS_FMT_SGRIG10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SBGGI10,
		.code			= MEDIA_BUS_FMT_SBGGI10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SGBIG10,
		.code			= MEDIA_BUS_FMT_SGBIG10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SGIRG10,
		.code			= MEDIA_BUS_FMT_SGIRG10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SIGGR10,
		.code			= MEDIA_BUS_FMT_SIGGR10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SGIBG10,
		.code			= MEDIA_BUS_FMT_SGIBG10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	}, {
		.fourcc			= V4L2_PIX_FMT_SIGGB10,
		.code			= MEDIA_BUS_FMT_SIGGB10_1X10,
		.csi_dt			= MIPI_CSI2_DT_RAW10,
		.bpp			= 16,
		.size			= SHIM_DMACNTX_SIZE_16,
	},

	/* More formats can be supported but they are not listed for now. */
};

Does the bpp in the CSI2RX driver need to match the raw12 of the sensor driver?

If need to modify it, how can we do? Simply changing it to 12 doesn't seem to work.
from this guide https://dev.ti.com/tirex/explore/content/am62ax_academy_9_02_00_00_v1/_build_am62ax_academy_9_02_00_00_v1/source/linux/ch-develop/dev-use-camera.html#enable-a-new-csi-2-sensor，if enable a new CSI-2 sensor,there is no indication that the CSI2RX driver needs to be modified,
could you help to confirm if need to modify csi2rx and v4l2 driver and how to do it? thanks.

0 Fabiana Jaimes over 1 year ago in reply to csscyt

TI__Mastermind 19680 points

Hi,

Could you configure your sensor format to fmt:SRGGB10_1X10/1920x1080 and run the below pipeline?

gst-launch-1.0  v4l2src device=/dev/video-imx477-cam0 io-mode=5 ! queue leaky=2 ! \
video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb10 ! \
tiovxisp sink_0::device=/dev/v4l-imx477-subdev0 sensor-name="SENSOR_SONY_IMX477_RPI" \
dcc-isp-file=/opt/imaging/imx477/linear/dcc_viss.bin \
sink_0::dcc-2a-file=/opt/imaging/imx477/linear/dcc_2a.bin format-msb=7 ! \
tiovxmultiscaler ! \
video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! \
kmssink driver-name=tidss sync=false

Thank you,

Fabiana

Processors

Processors forum

SK-AM68: port imx477 to SK-AM68

Enable A New CSI-2 Sensor