AM5718: CSI2 5Mpixel CMOS sensor

The factory team have been notified. They will respond here.

Upon closer examination it seems that the CAMSS is capable of parsing incoming serial stream into bytes and then internally generating video data in the VIP format and sending it over to the VIP for subsequent processing and moving to SDRAM. Provided CAL supports 24-bit RGB along this data path the RGB->YUV conversion can take place inside the VIP, so there should be no need to use the DSP for it.
The questions I have to the video experts are:

1. Has this data path been validated for 24 bit RGB data?
2. Does Linux capture driver support this?

thank you
Michael

Team,

Any comments on the above proposal? Is it feasible? Does driver support it?

thanks

Michael

Michael,
Due some apps experts on this subject are traveling, you can expect an answer next week. Sorry for the delay.

Team,

Any updates here?

thanks

Michael

Sorry Michael...I am double checking this one...

Have you looked at this wiki page -

It covers the information on feature supported/not supported by the driver.

Capture forwarding through VIP port is not a supported feature.

CSI2-interface on AM5718 can handle 600 MBytes/sec. RGB24 5MP@30 fps capture is doable using the CSI-2 interface.

What's the exact YUV format needed for conversion? There is hardware accelerator named VPE in AM5718 that also handle color conversion but it has internal line buffer width limitation and hence cannot handle color conversion of more than 2K in width. If width is larger than 2K, then VPE can be used in 2 pass mode to do the color conversion (2 pass feature not supported by driver). Note that this can result in some artifacts at the stitching edges. Who is the consumer of this  YUV buffer?

 Here's the link to the VPE driver wiki page - 

Thank you Manisha, I have couple of questions. First let me introduce you our application requirements as following:
background: our application will have AR0521 (5 Mpix sensor) connected thru CSI2 to 5718 and streaming Bayer RGB (RAW) format in 30 fps (preferred 60 fps). Then pipe into three pipes: 1) scale down to 640x480 and color converted to RGB format for display; 2) scale down to 160x120 and color converted to RGB format for internal application consumption in real time; 3) capture 5 consecutive frame at trigger (application defined trigger) at full resolution (2560x1944) then converted to RGB .jpg format, these 5 frames can be captured in raw Bayer format in real time but processed off-line.

So the question is, can 5718 support the above architecture? if we cannot use existing VIP and VPE, could we route the framebuffer to M4 or DSP or CPU in real time for color conversion? Thanks.

AM5718 doesn't have hardware accelerated image pipeline for RAW Bayer to RGB format conversion. That conversion needs to be handled in software on AM5718 device. The entire scaling and conversion for three pipelines for your scenario needs to be handled in software as both VIP and VPE cannot be used here. I will get back if AM57128 A15/DSP can handle the processing need for your requirement.

There's a Catalog Processor device coming up that has ISP (hardware accelerator for RAW BAYER to RGB conversion) on it. Check with your local sales rep on that device.

Allen was able to register CAL with CSI2 interfaced with AR0521 as /dev/video0
when we issue the following command:
yavta -c10 -fSGRBG8 -n 4 -Fframe-001.bin -s2592x1944 /dev/video0

we got the following message and then hang indefinitely and never save the file frame-001.bin, could you help to take a look what's wrong? Thanks a lot for your help.

Device `cal' on `platform:cal-000' is a video output (without [ 5462.931166] cal-000: CAL: cal_try_fmt_vid_cap
mplanes) device.
[ 5462.941036] cal-000: CAL: find_format_by_pix
[ 5462.949294] cal-000: CAL: cal_calc_format_size
[ 5462.954614] cal-000: cal_calc_format_size: fourcc: GRBG size: 2592x1944 bpl:2592 img_size:5038848
[ 5462.964287] cal-000: CAL: find_format_by_pix
[ 5462.968577] cal-000: CAL: __subdev_set_format
[ 5462.974354] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_set_fmt
[ 5462.982842] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_set_params
[ 5462.990362] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_set_power
[ 5463.000045] ar0521 4-0036: ar0521_set_power: on: 1
[ 5463.006058] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_set_power - No power state change needed
[ 5463.017351] cal-000: __subdev_set_format 2592x1944 code:3002
[ 5463.023796] cal-000: CAL: cal_calc_format_size
[ 5463.028264] cal-000: cal_calc_format_size: fourcc: GRBG size: 2592x1944 bpl:2592 img_size:5038848
Video format set: SGRBG8 (47425247) 2592x1944 (stride 2592) fiel[ 5463.038616] cal-000: CAL: cal_g_fmt_vid_cap
d none buffer size 5038848
[ 5463.049188] cal-000: CAL: cal_queue_setup
Video format: SGRBG8 (47425247) 2592x1944 (stride 2592) field no[ 5463.055617] cal-000: nbuffers=4, size=5038848
ne buffer size 5038848
4 buffers requested.[ 5463.078478] cal-000: CAL: cal_buffer_prepare

length: 5038848 offset: 0 timestamp type/source: mono/EoF
Buf[ 5463.084117] cal-000: CAL: cal_buffer_prepare
fer 0/0 mapped at address 0xb69a6000.
length: 5038848 offset: 5[ 5463.095515] cal-000: CAL: cal_buffer_prepare
042176 timestamp type/source: mono/EoF
Buffer 1/0 mapped at add[ 5463.104616] cal-000: CAL: cal_buffer_prepare
ress 0xb64d7000.
length: 5038848 offset: 10084352 timestamp typ[ 5463.114716] cal-000: CAL: cal_buffer_queue
e/source: mono/EoF
Buffer 2/0 mapped at address 0xb6008000.
le[ 5463.124363] cal-000: CAL: cal_buffer_queue
ngth: 5038848 offset: 15126528 timestamp type/source: mono/EoF
[ 5463.133971] cal-000: CAL: cal_buffer_queue
Buffer 3/0 mapped at address 0xb5b39000.
[ 5463.144136] cal-000: CAL: cal_buffer_queue
[ 5463.150688] cal-000: CAL: cal_start_streaming - Start
[ 5463.157127] cal-000: CAL: cal_start_streaming - Point A
[ 5463.163120] cal-000: CAL: cal_get_external_info
[ 5463.167674] cal-000: sensor Pixel Rate: 27000000
[ 5463.174327] cal: CAL: cal_runtime_get
[ 5463.178030] cal-000: CAL: csi2_ctx_config
[ 5463.183415] cal-000: CAL_CSI2_CTX0(1) = 0x00000101
[ 5463.188226] cal-000: CAL: pix_proc_config
[ 5463.193723] cal-000: CAL_PIX_PROC(1) = 0x00080005
[ 5463.198510] cal-000: CAL: cal_wr_dma_config
[ 5463.204240] cal-000: CAL_WR_DMA_CTRL(1) = 0x1e604304
[ 5463.209259] cal-000: CAL_WR_DMA_OFST(1) = 0x00000a20
[ 5463.215792] cal-000: CAL_WR_DMA_XSIZE(1) = 0x0a200000
[ 5463.220896] cal-000: CAL_CTRL = 0xff1fe07e
[ 5463.226617] cal-000: CAL: csi2_lane_config
[ 5463.230732] cal-000: CAL_CSI2_COMPLEXIO_CFG(1) = 0x000dcba9
[ 5463.237512] cal-000: CAL: cal_start_streaming - Enable IRQs
[ 5463.243736] cal-000: CAL: enable_irqs
[ 5463.247413] cal-000: CAL: cal_start_streaming - Init sub device
[ 5463.254512] cal-000: CAL: csi2_phy_init
[ 5463.258364] cal-000: CAL: camerarx_phy_enable
[ 5463.263886] cal-000: CAL_CSI2_COMPLEXIO_CFG(1) = 0x400dcba9 De-assert Complex IO Reset
[ 5463.272470] cal-000: CAL: csi2_phy_config
[ 5463.276496] cal-000: csi2_ddrclk_khz: 27000
[ 5463.280694] cal-000: ths_term: 0 (0x00)
[ 5463.286205] cal-000: ths_settle: 6 (0x06)
[ 5463.290232] cal-000: CSI2_0_REG0 = 0x01000006
[ 5463.295740] cal-000: CSI2_0_REG1 = 0xc002e10e
[ 5463.300115] cal-000: CAL_CSI2_TIMING(1) = 0x0000c197 Stop States
[ 5463.307336] cal-000: CAL_CSI2_TIMING(1) = 0x0000c197 Force RXMODE
[ 5463.315196] cal-000: CAL_CSI2_COMPLEXIO_CFG(1) = 0x4a0dcba9 Powered UP
[ 5463.322459] cal-000: CAL: cal_start_streaming - Sub Device Call
[ 5463.328408] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_s_stream
[ 5463.337352] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_init_gpios
[ 5463.345864] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_s_stream - Enable Stream
[ 5463.355612] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_reg_write = 00
[ 5463.377816] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_reg_write = 01
[ 5463.385907] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_reg_write = 02
[ 5463.395815] ar0521 4-0036: ALLEN: ***************** AR0521 INFO ar0521_reg_write = 1c
[ 5463.404866] cal-000: CAL: cal_start_streaming - Wait for sub device
[ 5463.411793] cal-000: CAL: csi2_wait_for_phy
[ 5463.701622] cal-000: CAL_CSI2_COMPLEXIO_CFG(1) = 0x4a0dcba9 Complex IO Reset Done (250) (timeout)
[ 5463.721793] cal-000: CAL_CSI2_TIMING(1) = 0x0000c197 Stop State Reached (timeout)
[ 5463.729312] cal-000: CSI2_0_REG1 = 0xc002e10e (Bit(31,28) should be set!)
[ 5463.737318] cal-000: CAL: cal_start_streaming - Write DMA
[ 5463.743373] cal-000: CAL: cal_start_streaming - Enable PPI
[ 5463.748880] cal-000: CAL: csi2_ppi_enable
[ 5463.754086] cal-000: CAL: csi2_ppi_enable DONE

Would it be possible to use the Yavta tool (with strace -f prepended) using a MIPI camera? It is hard to compare the non MIPI output (VIP) to CAL output.

The command below was used originally.

strace -f yavta -c60 -fYUYV -Fvout_1280x800_yuyv.yuv -s1280x800 /dev/video1

When using the non MIPI camera, VIP shows up as a device (video0) as well as the actual camera (video1). Using the MIPI camera, video0 is present but there is no video1. My assumption is that the actual camera will not be displayed as its own device since it is a sub-device of CAL. Can you confirm?

Use below command to know the v4l2 devices and what they represent. /dev/video0 is for vpe and not vip. Since you do not see video1, MIPI camera is not initialized well.

#v4l2-ctl --list-devices

Do you have a MIPI image sensor board that you could connect to the 57x EVM? I want to make 100% sure that the image sensor should be listed as its own V4L2 device. I know that it does for VIP image sensors. Also, the documentation suggests that it should. The trouble is that the source code for the MIPI sensors I have seen do not register themselves as a top level device (/dev/video1). All of the drivers I have seen register the image sensor as a sub-device of CAL. Can we confirm this with a real test?

My sensor is initializing all of the registers and being picked up by CAL. Eventually CAL registers itself as /dev/video0. I inserted debug code into the driver and CAL to confirm that all commands issued to /dev/video0 are being routed to the sub device (my sensor driver).

It is completely possible that the device should be listed as its own device. I just don't want to chase that problem down if it isn't a real issue.

If we could confirm this with a MIPI sensor board on the 57x EVM, that would greatly help! Thanks

[   11.827123] ar0521 4-0036: AR0521 INFO : ar0521_set_regs : set 31c2 -> ffff

[   11.828732] ar0521 4-0036: AR0521 INFO ar0521_reg_read

[   11.828964] ar0521 4-0036: ar0521 Product ID 0 Manufacturer ID 457

[   11.828970] ar0521 4-0036: AR0521 INFO ar0521_s_ctrl

[   11.828973] ar0521 4-0036: AR0521 ar0521_s_ctrl V4L2_CID_VFLIP

[   11.828977] ar0521 4-0036: AR0521 INFO ar0521_reg_rmw

[   11.828980] ar0521 4-0036: AR0521 INFO ar0521_reg_read

[   11.832396] ar0521 4-0036: AR0521 INFO ar0521_s_ctrl

[   11.832400] ar0521 4-0036: AR0521 ar0521_s_ctrl V4L2_CID_HFLIP

[   11.832403] ar0521 4-0036: AR0521 INFO ar0521_reg_rmw

[   11.832406] ar0521 4-0036: AR0521 INFO ar0521_reg_read

[   11.832966] ar0521 4-0036: AR0521 INFO ar0521_s_ctrl

[   11.832969] ar0521 4-0036: AR0521 ar0521_s_ctrl V4L2_CID_TEST_PATTERN

[   11.832973] ar0521 4-0036: AR0521 INFO ar0521_set_regs

[   11.832977] cal-000: CAL: cal_async_bound

[   11.832979] cal-000: Using sensor ar0521 4-0036 for capture

[   11.832982] cal-000: subdev ar0521 4-0036: code: 3001 idx: 0

[   11.832985] cal-000: matched fourcc: BA81: code: 3001 idx: 0

[   11.832988] cal-000: subdev ar0521 4-0036: code: 3013 idx: 1

[   11.832990] cal-000: matched fourcc: GBRG: code: 3013 idx: 1

[   11.832993] cal-000: subdev ar0521 4-0036: code: 3002 idx: 2

[   11.832995] cal-000: matched fourcc: GRBG: code: 3002 idx: 2

[   11.832997] cal-000: subdev ar0521 4-0036: code: 3014 idx: 3

[   11.832999] cal-000: matched fourcc: RGGB: code: 3014 idx: 3

[   11.833001] cal-000: subdev ar0521 4-0036: code: 3007 idx: 4

[   11.833003] cal-000: matched fourcc: BG10: code: 3007 idx: 4

[   11.833005] cal-000: subdev ar0521 4-0036: code: 300e idx: 5

[   11.833007] cal-000: matched fourcc: GB10: code: 300e idx: 5

[   11.833009] cal-000: subdev ar0521 4-0036: code: 300a idx: 6

[   11.833011] cal-000: matched fourcc: BA10: code: 300a idx: 6

[   11.833013] cal-000: subdev ar0521 4-0036: code: 300f idx: 7

[   11.833015] cal-000: matched fourcc: RG10: code: 300f idx: 7

[   11.833017] cal-000: subdev ar0521 4-0036: code: 3008 idx: 8

[   11.833019] cal-000: matched fourcc: BG12: code: 3008 idx: 8

[   11.833021] cal-000: subdev ar0521 4-0036: code: 3010 idx: 9

[   11.833141] cal-000: matched fourcc: GB12: code: 3010 idx: 9

[   11.833144] cal-000: subdev ar0521 4-0036: code: 3011 idx: 10

[   11.833147] cal-000: matched fourcc: BA12: code: 3011 idx: 10

[   11.833149] cal-000: subdev ar0521 4-0036: code: 3012 idx: 11

[   11.833151] cal-000: matched fourcc: RG12: code: 3012 idx: 11

[   11.833153] cal-000: CAL: cal_complete_ctx

[   11.833155] cal-000: CAL: cal_complete_ctx - initialize queue

[   11.833158] cal-000: CAL: cal_complete_ctx - init video dma queues

[   11.833328] cal-000: V4L2 device registered as video0

[   11.833343] cal-000: CAL: cal_async_complete

[   11.833345] cal-000: CAL: __subdev_get_format

[   11.833350] ar0521 4-0036: AR0521 INFO ar0521_get_fmt

[   11.833353] cal-000: __subdev_get_format 648x488 code:3001

[   11.833355] cal-000: CAL: find_format_by_code

[   11.833357] cal-000: CAL: cal_calc_format_size

[   11.833361] cal-000: cal_calc_format_size: fourcc: BA81 size: 648x488 bpl:656 img_size:320128

[   11.833365] ar0521 4-0036: ar0521 4-0036 sensor driver registered !!

 

 

I do not have a board setup at my end to share the working logs . The image sensor should NOT be listed as it's own V4L2 device.

Looking at your prior log -

CAL: cal_start_streaming - Start [ 5463.157127] cal-000: CAL:

cal_start_streaming - Point A [ 5463.163120] cal-000: CAL:

cal_get_external_info [ 5463.167674] cal-000: sensor Pixel Rate:

27000000

The reported Pixel Rate here is way to low. It looks more like the value of the source oscillator.

For instance with our the OV490 sensor we use the Pixel Rate is calculated to be about 201000000 pixels/second.

This value along with the selected pixel size and the number of data lanes is used to compute the PHY timing parameters, which in this case are probably out of range. Which would explain why the capture does not start because the PHY is not able to properly handshake with the sensor.

Normally sensor datasheet provides a way to calculate/derive the pixel rate.

It is usually something like this:

/*

* = fvco / pixel_width * num_lanes

* = 804,000,000 / 16 bits * 4 lanes

*/

#define OV490_PIXEL_RATE                201000000

Hope this helps to fix the problem.