This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-AM62P-LP: Connecting FPGA CSI-2 Tx to AM62P CSI-2 Rx for 2560x1024 @ 50fps (24bpp) - Device Tree & Kernel Config Help

Part Number: SK-AM62P-LP
Other Parts Discussed in Thread: AM62P

Tool/software:

Hi everyone,

I'm working with the SK-AM62P-LP development kit and have successfully tested the IMX219 camera using the provided "k3-am62x-sk-csi2-imx219.dtso" overlay.

Now, I'm trying to integrate an FPGA acting as a CSI-2 Transmitter (Tx) core with the AM62P's CSI-2 Receiver (Rx) interface. My goal is to receive video data from the FPGA at a resolution of 2560x1024, 24 bits per pixel (bpp), and 50 frames per second (fps).

I understand that this requires specific device tree modifications and potentially kernel configurations. I've been looking at existing camera overlays for guidance, but adapting them for a custom FPGA CSI-2 Tx (which might not have an I2C control interface like a standard sensor) is proving challenging.

Could anyone provide detailed guidance on the necessary device tree modifications and/or kernel configurations for this setup?

Specifically, I'm looking for:

  1. Device Tree Overlay (.dtso) example: A template or detailed explanation for a device tree node that describes a "virtual" sensor representing the FPGA's CSI-2 Tx, connecting it to the AM62P's CSI-2 Rx. This should include properties for resolution, pixel format (24bpp, likely RGB888 or similar), and ideally, information about using all 4 CSI-2 data lanes.
  2. Relevant Kernel Configurations: Any specific kernel options (e.g., CONFIG_VIDEO_CADENCE_CSI2RX) that need to be enabled or verified for the CSI-2 receiver to function correctly with a custom source.
  3. Data Rate Considerations: Confirmation that 2560x1024 @ 50fps at 24bpp (approx. 3.145 Gbps) can be handled by the AM62P's 4-lane CSI-2 Rx, and if there are any specific settings or limitations to be aware of.

Additionally, if my FPGA CSI-2 Tx IP core has limitations that prevent it from supporting the full 2560x1024 resolution in a single stream, could you suggest an alternative method to receive this as two camera data streams (e.g., two 1280x1024 streams)? I'm particularly interested in how this would be handled on the AM62P side, considering it typically has a single CSI-2 Rx controller. Would this involve MIPI CSI-2 Virtual Channels from the FPGA, and how would that be reflected in the device tree and V4L2 framework?

Any insights, code snippets, or pointers to relevant documentation would be greatly appreciated!

Thank you in advance!

  • Hello,

    The first question really depends on your FPGA CSI-2 Tx and its driver. The resolution and pixel format should be specified in the driver, not in the device tree. Please take a look at the IMX219 driver and the device tree overlay for reference.

    For the second question, as long as your FPGA sends data in MIPI CSI-2 format, you don't need to make any changes to kernel config.

    Yes, 2560x1024 @ 50fps at 24bpp (approx. 3.145 Gbps)  can be handled by AM62P which supports up to 2.5Gbps/lane with 4 lanes.

    could you suggest an alternative method to receive this as two camera data streams (e.g., two 1280x1024 streams)?

    If your FPGA IP supports virtual channels, you can 2 virtual channels to send two streams.

    Regards,

    Jianzhong

  • Hello Jianzhong,

    Thank you for your prompt response and the valuable insights.

    I appreciate the confirmation that the AM62P's 4-lane CSI-2 Rx can handle the data rate of 2560x1024 @ 50fps at 24bpp (approx. 3.145 Gbps), which is great news for our planned setup. Also, it's good to know that general kernel configurations shouldn't require changes as long as the FPGA outputs standard MIPI CSI-2.

    Regarding my first question about the device tree and driver, I understand that the resolution and pixel format are typically defined within the sensor's driver. My challenge stems from the fact that this is a custom FPGA CSI-2 Tx, not a standard camera sensor with an existing Linux driver and an I2C control interface. We're essentially trying to define a "virtual" sensor in the device tree to represent the FPGA's output, which will then be "received" by the AM62P's CSI-2 Rx.

    While the IMX219 driver and overlay are helpful for understanding existing camera integrations, they rely on an I2C interface to configure the sensor. For an FPGA as a CSI-2 Tx, there won't be an I2C control from the AM62P side in the same way.

    Therefore, I was hoping for more specific guidance on how to structure the device tree node for such a "headless" or "virtual" CSI-2 source. This would involve:

    • How to define the fixed parameters (like resolution, pixel format, number of lanes) within the device tree itself or a minimal "dummy" driver, given there's no I2C control loop.
    • What properties are essential in the device tree to properly connect this virtual sensor to the Cadence CSI-2 Rx block on the AM62P.
    • Any specific quirks or best practices for the AM62P's CSI-2 receiver when dealing with a constantly streaming, unconfigurable source like an FPGA.

    Furthermore, regarding the use of virtual channels, thank you for confirming that the AM62P supports receiving two streams via MIPI CSI-2 Virtual Channels if the FPGA supports it. Could you elaborate on how this would be reflected in the AM62P's device tree and within the V4L2 framework? Specifically, would we see two separate V4L2 video nodes (e.g., /dev/video0 and /dev/video1) or would it be handled differently? Any device tree snippets or pointers on configuring the AM62P's CSI-2 Rx to recognize and process multiple virtual channels would be extremely beneficial.

    We are seeking more detailed examples or conceptual guidance on setting up the AM62P's receiver side for custom, non-standard CSI-2 sources like an FPGA, especially concerning device tree configurations for such scenarios and the handling of virtual channels.

    Thank you again for your continued support.

    Best regards,

    Narendra Kumar Reddy

  • My challenge stems from the fact that this is a custom FPGA CSI-2 Tx, not a standard camera sensor with an existing Linux driver and an I2C control interface. We're essentially trying to define a "virtual" sensor in the device tree to represent the FPGA's output, which will then be "received" by the AM62P's CSI-2 Rx.

    What you can do is to implement a dummy sensor driver for your FPGA CSI-2 Tx device. This driver can specify the sensor parameters like data format, resolution, virtual channel id, etc. You can use IMX219 driver as a template, removing all I2C read/write function calls. This driver will register your virtual sensor to the V4L2 framework the same way as a normal sensor (like IMX219) does. You can then build your application using the v4l2 framework.

    The only thing is that the dummy driver can't communicate to your FPGA device when streaming starts on the CSI Rx side. I assume you can control the FPGA device and start sending data through some mechanism. If that's the case, you can start sending data on the FPGA, and then start streaming operation the CSI Rx side. Below is a sequence of operation for this scheme:

    1. virtual sensor probes via the dummy driver, the same way as IMX219 probes, even though there is no communication with the FPGA device

    2. control the FPGA and start sending data

    3. start receiving on the CSI Rx side, for example, using v4l2-ctl or yavta to capture data.

    Regards,

    Jianzhong

  • Hello Jianzhong Xu,

    Following up on our previous discussions, we've made good progress! We're now successfully receiving CSI data from our FPGA CSI IP Core by registering a dummy sensor node (mimicking imx219).

    We've been conducting tests with a dual monitor setup (1280x1024 @60Hz each) using a color bar test pattern. Below are our observations with different GStreamer commands and configurations:

    Test Observations & Commands:

    We used the following GStreamer commands:

    • gst-launch-1.0 v4l2src="/dev/video2" ! video/x-raw, format=BGRx, width=2560, height=1024 ! fpsdisplaysink video-sink=waylandsink sync=true

    • gst-launch-1.0 v4l2src="/dev/video2" ! video/x-raw, format=BGRx, width=2560, height=1024 ! fpsdisplaysink video-sink=waylandsink sync=false

    Here's a summary of our results:

    Resolution    FORMAT    LANES    LaneRate       Expected    Observed    DisplaySink        sync    droprate
    --------------------------------------------------------------------------------------------------------------------------------------------------

    2560x1024    RGB        2        800 Mbps/Lane       25.2 FPS     22 FPS        waylandsink        true     2 FPS
    2560x1024    RGB        4        800 Mbps/Lane       36 FPS         32 FPS        waylandsink        true    2 FPS
    2560x1024    RGB        2        1.5 Gbps/Lane         46 FPS        2 FPS        waylandsink          true     Random
    2560x1024    RGB        4        1.5 Gbps/Lane        92 FPS        2 FPS        waylandsink           true     Random

    2560x1024    RGB        2        800 Mbps/Lane    25.2 FPS      25 FPS        waylandsink        false    -
    2560x1024    RGB        4        800 Mbps/Lane    36 FPS        35 FPS        waylandsink        false    -
    2560x1024    RGB        2        1.5 Gbps/Lane    46 FPS         41 FPS        waylandsink        false    -
    2560x1024    RGB        4        1.5 Gbps/Lane    92 FPS         41 FPS        waylandsink        false    -

    As per our previous discussions, the AM62P's CSI-2 Rx is expected to handle up to 2.5 Gbps/lane, which should accommodate our target of 2560x1024 @ 50fps at 24bpp (approx. 3.145 Gbps total). We've also verified our FPGA CSI-2 Tx IP core through an external loopback to an FPGA CSI-2 Rx IP core, confirming that it transmits data correctly and achieves the expected FPS.

    Our Current Questions:

    1. Given the AM62P's stated capabilities, why are we unable to achieve 50 FPS for 2560x1024 RGB on the SK-AM62P-LP, especially with sync=true?

    2. Are there any specific GStreamer command alterations or other configurations we should explore to reach our target FPS?

    3. Is it generally advisable or acceptable to proceed with sync=false mode to achieve higher observed FPS, or could this lead to other issues in a production environment?

    Any further insights or suggestions would be greatly appreciated!

    Best Regards,

    Narendra Kumar Reddy

  • Hello Narendra Kumar Reddy,

    Sorry for my late response. I was out in the past two weeks.

    Can you try a couple of things to narrow down what causes frame drops?

    1. use a fakesink instead of waylandsink and see if you can get the desired fps

    2. use yavta or v4l2-ctl to capture and see if you can get the desired fps

    Thank you.

    Jianzhong

  • Hello Jianzhong,

    Thanks for getting back to me, and no worries about the late response!

    I've performed the tests you suggested using fakesink and yavta/v4l2-ctl to narrow down the cause of frame drops. The results for fakesink are detailed below, and the yavta logs reflect similar findings (attached separately).


    Fakesink Test Results

    Test Commands:

    • gst-launch-1.0 v4l2src="/dev/video2" ! video/x-raw, format=BGRx, width=2560, height=1024 ! fpsdisplaysink video-sink=fakesink sync=true

    • gst-launch-1.0 v4l2src="/dev/video2" ! video/x-raw, format=BGRx, width=2560, height=1024 ! fpsdisplaysink video-sink=fakesink sync=false

    Resolution FORMAT LANES LaneRate Expected FPS Observed FPS DisplaySink sync droprate
    2560x1024 RGB 2 800 Mbps/Lane 25.2 FPS 25.14 FPS fakesink true -
    2560x1024 RGB 4 800 Mbps/Lane 36 FPS Undetermined fakesink true -
    2560x1024 RGB 2 1.5 Gbps/Lane 46 FPS 38.67 FPS fakesink true -
    2560x1024 RGB 4 1.5 Gbps/Lane 92 FPS Undetermined fakesink true -
    2560x1024 RGB 2 800 Mbps/Lane 25.2 FPS 25.05 FPS fakesink false -
    2560x1024 RGB 4 800 Mbps/Lane 36 FPS Undetermined fakesink false -
    2560x1024 RGB 2 1.5 Gbps/Lane 46 FPS 41.42 FPS fakesink false -
    2560x1024 RGB 4 1.5 Gbps/Lane 92 FPS Undetermined fakesink false -
  • Ok. So you have 4 combinations of # of lanes and data rate: (2, 800), (4, 800), (2, 15000), (4, 15000).

    How did you configure these in the FPGA CSI Tx and how did you configure at the AM62P receive side?

  • Hello ,

    A few things to check:

    1. You'll need to configure the link frequency (MHz) parameter in the sensor device tree as in the IMX219 overlay: git.ti.com/.../k3-am62x-sk-csi2-imx219.dtso

    This parameter is passed to the CSI Rx driver to configure the D-PHY lane speed (Mbps): git.ti.com/.../cdns-csi2rx.c

    Then the DPHY Rx driver gets the frequency and multiplies by 2 to change to Mbps - https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/phy/cadence/cdns-dphy-rx.c?h=ti-linux-6.6.y#n108.

    Finally, DPHY Rx driver configures the band control register (for lane speed) - https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/phy/cadence/cdns-dphy-rx.c?h=ti-linux-6.6.y#n190.

    So to use 1.5Gbps, you'll need to set the link frequency to 750MHz. You can check register 0x30110B00 to verify that the lane speed is configured properly:

    The value of bits 4:0 should correspond to the index of this table: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/phy/cadence/cdns-dphy-rx.c?h=ti-linux-6.6.y#n72

    2. When you send CSI data at 1.5Gbps, you'll need to configure the transmitter to send a deskew sequence so that the receiver will calibrate itself to be ready to receive the data.

    Regards,

    Jianzhong