AM67: How to link media-ctl pad with mismatched MEDIA_BUS_FMT

Evan Williams

Part Number: AM67

Tool/software:

See the attached log below.

We are using isl7998x which only supports UYVY8_2X8, however it is automatically linked to cdns_csi2rx which only supports UYVY8_1X16.

I have modified the isl7998x driver to report the incorrect format so that I can open the stream, but of course this results in a garbled image.

What is the proper way to link these two mismatched pad formats?

media-ctl -d /dev/media1 -p
Media controller API version 6.6.44

Media device information
------------------------
driver          j721e-csi2rx
model           TI-CSI2RX
serial
bus info        platform:30122000.ticsi2rx
hw revision     0x1
driver version  6.6.44

Device topology
- entity 1: 30122000.ticsi2rx (5 pads, 5 links, 1 route)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        routes:
                0/0 -> 1/0 [ACTIVE]
        pad0: Sink
                [stream:0 fmt:UYVY8_1X16/720x480 field:none]
                <- "cdns_csi2rx.30121000.csi-bridge":1 [ENABLED,IMMUTABLE]
        pad1: Source
                [stream:0 fmt:UYVY8_1X16/720x480 field:none]
                -> "30122000.ticsi2rx context 0":0 [ENABLED,IMMUTABLE]
        pad2: Source
                -> "30122000.ticsi2rx context 1":0 [ENABLED,IMMUTABLE]
        pad3: Source
                -> "30122000.ticsi2rx context 2":0 [ENABLED,IMMUTABLE]
        pad4: Source
                -> "30122000.ticsi2rx context 3":0 [ENABLED,IMMUTABLE]

- entity 7: cdns_csi2rx.30121000.csi-bridge (5 pads, 2 links, 1 route)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        routes:
                0/0 -> 1/0 [ACTIVE]
        pad0: Sink
                [stream:0 fmt:UYVY8_1X16/720x480 field:none]
                <- "isl7998x 3-0044":0 [ENABLED,IMMUTABLE]
        pad1: Source
                [stream:0 fmt:UYVY8_1X16/720x480 field:none]
                -> "30122000.ticsi2rx":0 [ENABLED,IMMUTABLE]
        pad2: Source
        pad3: Source
        pad4: Source

- entity 13: isl7998x 3-0044 (5 pads, 1 link, 0 routes)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev2
        pad0: Source
                [stream:0 fmt:UYVY8_2X8/720x480 field:seq-bt]
                -> "cdns_csi2rx.30121000.csi-bridge":0 [ENABLED,IMMUTABLE]
        pad1: Sink
                [stream:0 fmt:UYVY8_2X8/720x480 field:seq-bt]
        pad2: Sink
                [stream:0 fmt:UYVY8_2X8/720x480 field:seq-bt]
        pad3: Sink
                [stream:0 fmt:UYVY8_2X8/720x480 field:seq-bt]
        pad4: Sink
                [stream:0 fmt:UYVY8_2X8/720x480 field:seq-bt]

- entity 23: 30122000.ticsi2rx context 0 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video2
        pad0: Sink
                <- "30122000.ticsi2rx":1 [ENABLED,IMMUTABLE]

- entity 29: 30122000.ticsi2rx context 1 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video3
        pad0: Sink
                <- "30122000.ticsi2rx":2 [ENABLED,IMMUTABLE]

- entity 35: 30122000.ticsi2rx context 2 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video4
        pad0: Sink
                <- "30122000.ticsi2rx":3 [ENABLED,IMMUTABLE]

- entity 41: 30122000.ticsi2rx context 3 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video5
        pad0: Sink
                <- "30122000.ticsi2rx":4 [ENABLED,IMMUTABLE]

4 months ago

0 Jared McArthur 4 months ago

TI__Mastermind 18005 points

Hi Evan,

Unfortunately we don't support UYVY8_2X8. Unfortunately we don't support UYVY8_2X8. CSI is a serial bus, but UYVY8_2X8 is a format for parallel bus.

You would need to either change sensors, or edit the sensor driver to support UYVY8_1X16.

Best,
Jared

0 Evan Williams 4 months ago in reply to Jared McArthur

Intellectual 565 points

Hi Jared,

This isl part has CSI_CLK, CSI_D0, CSI_D1.
Should we expect to get 1 camera stream on each D0 and 1 camera stream on D1?
The isl driver configures 2 streams on D0 and 2 streams on D1 for 4 streams simultaneously so it's surprising to me that it's not supported in the TI driver.
It's a huge bummer because we were hoping to access all 4 streams at a time.

I was able to get all 4 channels to stream at the same time interlaced in the same stream by changing each lane's virtual channel assignment to 0 inside the isl chip. My understanding is that the virtual assignment is a label in the CSI stream's "Virtual-Channel field", so it appears that the TI+Cadence driver is somewhat able to selectively ignore or include data via this virtual assignment and is supposed to support multiple streams on a single set of data pins.

0 Jared McArthur 4 months ago in reply to Evan Williams

TI__Mastermind 18005 points

Hi Evan,

Evan Williams said:
It's a huge bummer because we were hoping to access all 4 streams at a time.

The issue with the format is a separate issue. The reasoning can be found in this post: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1388645/am625-camera-porting-to-ti-am625-09_02_01_10-sdk/5316476#5316476

CSI drivers shouldn't support UYVY8_2X8, because CSI is a serial bus, not a parallel bus.

Evan Williams said:
I was able to get all 4 channels to stream at the same time interlaced in the same stream by changing each lane's virtual channel assignment to 0 inside the isl chip. My understanding is that the virtual assignment is a label in the CSI stream's "Virtual-Channel field", so it appears that the TI+Cadence driver is somewhat able to selectively ignore or include data via this virtual assignment and is supposed to support multiple streams on a single set of data pins.

The CSI2RX module can handle up to 16 virtual channels (partial MIPI CSI-2 v2.0 feature), so it can handle 4 streams interleaved on 2 data lines.

Best,
Jared

0 Evan Williams 4 months ago in reply to Jared McArthur

Intellectual 565 points

Hi Jared,

Thanks for this link. Perhaps I should modify the isl driver to support 1X16?

If the data is actually formatted 2X8 though, do I need to modify register settings on the chip to change it?

Or is there a setting I could change in my device tree?

Just FYI, I tried modifying the CSI drivers to add support back for 2X8, and the image appears the same as when I set the isl driver to 1X16.
Out of 480 lines, it's only drawing about half. I Have to enable "Pseudo Frame Mode" to get the image to start in the same location each frame too.
Could this be an interlacing problem? Missing half the lines?

0 Jared McArthur 4 months ago in reply to Evan Williams

TI__Mastermind 18005 points

Hi Evan,

Evan Williams said:
Perhaps I should modify the isl driver to support 1X16?

If the data is actually formatted 2X8 though, do I need to modify register settings on the chip to change it?

There may be no difference between UYVY8_2X8 and UYVY8_1X16 from the isl7998x since CSI is inherently serial. If there is, then there should be a register within the isl7998x that can control the format.

You can look at https://docs.kernel.org/userspace-api/media/v4l/subdev-formats.html for the formats' data organization schemes.

Evan Williams said:
Out of 480 lines, it's only drawing about half. I Have to enable "Pseudo Frame Mode" to get the image to start in the same location each frame too.
Could this be an interlacing problem? Missing half the lines?

What is pseudo frame mode? It could be an issue with the image format, but if so, it would likely drop half the width, not height. Can you send an image of the incorrect image?

Best,
Jared

0 Evan Williams 4 months ago in reply to Jared McArthur

Intellectual 565 points

Hi Jared,

Please see the image attached.

I've tried setting V4L2_FIELD_INTERLACED and V4L2_FIELD_INTERLACED_BT from within the driver, but they do not seem to make a difference. Perhaps because the CSI drivers aren't using them or forwarding them? I see de-interlacing could possibly be done in "m2m-deinterlace", but I don't have that in my media-ctrl pipline, so I don't think it's happening?

+1 Jared McArthur 4 months ago in reply to Evan Williams

TI__Mastermind 18005 points

Hi Evan,

Evan Williams said:
I've tried setting V4L2_FIELD_INTERLACED and V4L2_FIELD_INTERLACED_BT from within the driver, but they do not seem to make a difference.

Why would you need to set V4L2_FIELD_INTERLACED or V4L2_FIELD_INTERLACED_BT? Is the video interlaced?

The data should come in as 4 virtual channels on 2 data lines. The CSI2RX IP will give each virtual channel a separate DMA section, the isl7998x driver needs to create 4 pads for each virtual channel.

Evan Williams said:
I see de-interlacing could possibly be done in "m2m-deinterlace", but I don't have that in my media-ctrl pipline, so I don't think it's happening?

Why do you need memory to memory deinterlacing?

I think you are confusing virtual channels, serial vs parallel buses, and V4L2 interlacing.

Virtual channels: multiple camera streams within the same CSI port. Handled and separated by the CSI2RX IP.
Serial vs parallel buses: bus type for the protocol. CSI is a serial bus protocol.
V4L2 interlacing: whether the video is progressive or interlaced: https://docs.kernel.org/userspace-api/media/v4l/field-order.html

Can you first try to debug one videostream, and then gradually increase the number of cameras that you are pulling data from?

Additionally, this thread has drifted off topic. The original question/answer is no, the CSI2RX driver will not and should not support UYVY8_2X8. I will close the thread as it's gotten off topic.

To continue debugging, can you open a new thread?

Best,
Jared

0 Evan Williams 4 months ago in reply to Jared McArthur

Intellectual 565 points

Hi Jared,

Sure, I'll open another ticket to try to resolve the underlying issue.

The reason I suspected interlacing was because the NXP driver has the interlacing flag set for this chip and had to add special handling to their CSI driver.

I'd prefer to use the natively supported driver if possible.
Also, after looking at the link you sent, it still looks like some kind of interlacing problem to me, being more pedantic, a "Field order" problem.

Also notice the function ti_csi2rx_try_fmt_vid_cap says interlaced formats are not supported.

Thanks!

0 Jared McArthur 4 months ago in reply to Evan Williams

TI__Mastermind 18005 points

Hi Evan,

Thank you.

Best,
Jared

0 Evan Williams 4 months ago in reply to Jared McArthur

Intellectual 565 points

Hi Jared,

Just another comment. I did like you suggested and set up streaming with only 1 channel (D0) and 1 stream (Camera input1)
I get 2 copies of the same image like in the new ticket, sort of like in field order Top/Bottom.
It looks like it's not being combined into a single image.

Thanks

0 Jared McArthur 4 months ago in reply to Evan Williams

TI__Mastermind 18005 points

Hi Evan,

Thank you, I will continue this in the new thread.

Best,
Jared

Processors

Processors forum

AM67: How to link media-ctl pad with mismatched MEDIA_BUS_FMT