AM5726: GPIO error

Hi Gert,

Can you describe how you've ported you DT to the newer kernels in the newer SDKs? DT files need to be rebuilt against the new kernel and this will use new include files that might have changes.

I would start looking at the interfaces that you are using. For example, if you are running off of SD card, I would take a look at those nodes first. It is obviously something that the system is using repeatedly.

Thanks.

Good Day RonB

We have used SDK3 successfully for the past year or two.. But as our design requires the DSP functionality the SDK3 did not allow for DSP access..SDK5 was recommended as that was the "latest" release when we started to port our design over.. We have not ported any of our "old" code from the SDK3 ... We used the SDK5 as our base..

Modified the Menuconfig to be inline our requirement, Updated our DTS and Pinmux to our custom board and this has been running for quite a while with no issues (while getting the rest of the SDK5 up and running).

Then we progressed with the Wifi Drivers and other hardware hardware specific drivers... Each progression of additional drivers was then tested with no adverse effects.

We also then started porting our services and python scripts to the newer base as well as using the PRU.

While running this we started to notice we have these gpio-request issues and not 100% sure why this will happen..But when this happens the Wi-Fi drops out first as the interrupt on the GPIO is no longer recognized and then on closer investigation we can see a whole list of gpio-request errors..

Now to me this seems to be that the gpio driver has been "Modified" or changed as the only way to correct this is to reinstall the whole OS.. once this is done all is good for a day or two..then the problem returns...

Kind Regards

Gert

Gert,

If I'm following you, you have to reboot and reprogram? A reboot alone doesn't bring things back? This would imply your stored image in flash is getting corrupted, right? Have you tried just reflashing specific pieces to see what specifically is getting trashed?

What kind of flash are you using?

Thanks.

Hi Ron,

I work with Gert, and we are working on this problem together.

Our SDK-5 build appears to run fine for several days, then all of a sudden starts having IOCTL errors (at runtime) and GPIO errors (during boot).

Rebooting does not fix the problem, so once the problem occurs it remains persistent between reboots. The board is pretty unstable at this point. Only re-imaging the eMMC or running from the SDCard with a clean image will resolve the problem. 

Our eMMC is partitioned to an A and B partition. Once the A partition becomes corrupt, booting into a clean B partition is ok. So this implies the u-boot bootloader is ok. I have also attached both dmesg logs from the A partition with GPIO errors and the B partition, without GPIO errors.

It appears the rootfs filesystem becomes corrupt like you mentioned. I suspected the device tree definition file(s), however after decompiling the SDK-5 dtb file and comparing them to an SDK-3 dtb for the same board, nothing stood out from the diff. I have attached both DTS files.

I suspected the DTS files were misconfigured, however could not find any differences between the SDK-3 and SDK-5 DTS files to support this.

Have tried running from a clean OS build, with no application software installed. Once the problem starting happening, I have tried the following to resolve the issue: blacklisting the custom WiFI driver and custom RTC driver kernel modules. This made no difference. It's possible the rootfs corruption had already occurred and disabling the custom drivers at this point will have no affect. I will try disabling them with a clean build and see if the problem persists.

To just clarify, the SDK-5 DTB files were decompiled from the eMMC A rootfs partition after the GPIO and IOCTL errors started being logged. Running sha1sums on the contents of the /boot folder showed these files (DTB, zImage, etc.) had not been modified from the clean build.

Mark,

That was a good test and it confirms the DT file binary is not getting corrupted. The DT file is "flattened" into Kernel memory and represented by /proc/devicetree. It is my guess that this space is what is getting corrupted. Since this is readble/writeable, you can confirm this by checking it once you get into a bad spot. You might even be able to "reload it", but I'm not sure. You could definitely write it to fix a specific problem to confirm that is the problem.

I think this might have some helpful pointers:

https://elinux.org/Device_Tree_Reference#Debugging

Of course, the bigger issue is what is causing the corruption. Disabling/enabling parts of the system will be one of the better ways to help isolate this.

I hope this is helpful to you.

Hi Ron,

Thanks for responding.

I have parked this issue for now. I will open another query once we get around to debugging it further.

Thanks,

Mark