TDA4VM: OpenCV not able to open IMX219 camera

Hi Tapan,

My suspicion for the error is that the final format from GStreamer is not compatible with OpenCV. This is a limitation that comes from OpenCV, but it only takes in BGR image format instead of RGB. You may do something like the final pipeline in this thread where a videoconvert plugin converts RGB into BGR: https://e2e.ti.com/support/microcontrollers/arm-based-microcontrollers-group/arm-based-microcontrollers/f/arm-based-microcontrollers-forum/1270814/reading-video-or-feed-from-imx-219-using-gstreamer-and-opencv-in-a-python-code

As for some other things we had discussed outside the forum...

For the low FPS issue, please try using GStreamer's tracer feature to see if there is an element that is taking up an unusually large amount of latency. We have a parser in the EdgeAI image that can be used to parse the tracer output to a more readable format: https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-sk-tda4vm/10_00_00/exports/edgeai-docs/common/measure_perf.html#parse-gst-tracers

My suspicion is either user application or camera driver. If it is user application, then creating a GStreamer pipeline that does not pipe into the user application and instead directly to display should show performance increase - this will be a good experiment to try. Otherwise, I do see for IMX219 driver there is a mode struct that seems to only have definitions up until 30 fps: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/media/i2c/imx219.c?h=ti-linux-6.1.y#n377. There may be the need to modify the camera driver to support higher fps.

And lastly, for the barcode reading examples, we at one point had ran some April Tag demos from community on the SK-TDA4VM board: https://software-dl.ti.com/jacinto7/esd/robotics-sdk/08_06_01/TDA4VM/docs/source/docs/april_tag.html. But looking in our latest 10.0 SDK, it seems mentions of this demo has been removed from our documentation. But technically, this is something that is possible. However, QR code will require a higher resolution than April Tags, so compute may take a bit if comparing between QR code and April Tag usecase.

Regards,

Takuma

Thank you Takuma. 

1. The OpenCV is working now.

2. We used a bin file provided long back by you to Yogesh Rao, solved the ISP issue for 640x480 resolution in ISP

3. We are now building a IMX219 Driver to support the frame rate for IMX219.  Is there any example for steps to modify the TI SDK and rebuild

Hi Tapan,

The Linux source code can be downloaded through the installer here: https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-J721E

After installing the SDK, the Linux source code is located under board-support/ti-linux-kernel-* folder. 

Lastly, if it is just the kernel drivers to update, the "Top-Level Makefile" may be used to build the kernel: https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-jacinto7/10_00_00_08/exports/docs/linux/Overview_Top_Level_Makefile.html. This will take much less time than Yocto to build.

Regards,

Takuma

Dear Takuma,

My engineer did the following.  But it seems the frame rate is still coming 30fps.

Would you please help here.

1. To make the frequency to reach 100 FPS, I changed
   #define IMX219_VTS_30FPS_640x480    0x06e3
   to
   #define IMX219_VTS_30FPS_640x480    0x0211
   I'm wondering if this is correct and if there is anything else to modify.
2. As I have no experience in firmware, after changing this file, can you please tell me how to build it and place it to the SD card. I have tried both ways mentioned in the document but both not working:
   1. Build using Yocto: After changing the imx219.c inside board-support/ti-linux-kernel-* folder, I found the yocto-build folder seems not use this kernel source code for building, where can I locate the source code that yocto use and modify it?
   2. Build using Top-Level Makefile: After changing the imx219.c and building, I ran all the commands in 1.3.2. Usage Examples (https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-jacinto7/latest/exports/docs/linux/Overview_Top_Level_Makefile.html) in the document. The SD card already has SDK 9.02, and after doing all steps including 'install to SD card', the camera still runs at 30 FPS. Is there any other extra steps needed for this method?

Hi Tapan,

Expertise for the Sony camera is limited within Texas Instruments, so I cannot comment on whether the changes to their drivers need additional modifications. However, I can comment on components that come from TI.

• As you might found in one of the posts you referenced, there is a setup_cameras.sh script that gets ran when logging in: https://github.com/TexasInstruments/edgeai-gst-apps/blob/main/scripts/setup_cameras.sh
  • This constructs a command to set the format and fps, so might need to modify this script more.
  • This document would be a good reference, since this is essentially what the script does: https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-jacinto7/10_00_00_08/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/Camera/CSI2RX.html
• If GStreamer is used, then GStreamer pipeline also should have a capsfilter for higher fps like in this thread: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1112142/sk-tda4vm-issue-with-framerate/4137377?tisearch=e2e-sitesearch&keymatch=%25252520user%2525253A524719#4137377

As for building the firmware, using the Top-Level Makefile is recommended since it is faster. For Linux drivers in general, you could put a printk statement (something like printk("testing if this is printed");) in the probe function of the imx219 to see if you're changes are being flashed. Then, running something like (dmesg | grep testing) should search the boot logs to see if the printk statement was printed out.

Regards,

Takuma