AM6442: mmapping the GPMC interface under Linux with SDK 11.00.09.04

Hi Mario,

Mario Trams said:

That time. the trick has been to add the following to the device tree file:

&gpmc0 {
        status = "okay";
};

In the previous e2e thread, you already set status to "okay" in &gpmc0 node but wasn't able to access GPMC registers, but now you said you could, what did I miss?

Hi Bin,

the issue was/is two-fold. That time I got rid of the bus error because the GPMC has been completely shut down by default. 

After activating the GPMC I was able to access the GPMC CSRs. However, the secondary issue then was that accessing some CSRs appeared not working. I.e. I was able to access them, but the system did not behave as expected - presumably because of the privilege level as you pointed out that time. Now I wanted to pick up the work from there and dig into the kernel driver matter.

However, with the newer SDK I'm faced with the fact that I'm seeing those bus errors despite setting the status to "okay" within the device tree file.

Hi Mario,

Thanks for the clarification. I understand the issue now.

Mario Trams said:

However, with the newer SDK I'm faced with the fact that I'm seeing those bus errors despite setting the status to "okay" within the device tree file.

I don't expect this to happen, but do you see any gpmc related message in kernel boot log which would indicate any error? The only problem that I can think of would be that you only enabled gpmc0 node in devicetree but didn't provide any gpmc config parameters, which might cause gpmc init failure then not powered on.

The SDK11.0 kernel also provides an gpmc overlay dtbo file. If you apply this dtbo for kernel, does the bus error still happen?

Hello Bin,

I don't see any gpmc-related errors (or even regular messages) in dmesg (i.e. dmesg | grep gpmc). That's not surprising for me, since there is no gpmc-related driver as such loaded by default. 

Actually, I took over all the settings into the device tree file that I did use in the 09 SDK as I did describe in that other thread

(https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1403803/am6442-attempts-for-manually-toggling-gpmc0_wpn-under-linux-is-failing-gpmc_config-register-reads-formally-illegal-data)

However, as far as I remember, especially those various gpmc0 pins settings were not really associated regarding the bus error issue that time.

One point I noticed was that in SDK 11 there is a new entry regarding gpmc0 in k3-am642-evm.dts:

&gpmc0 {
        ranges = <0 0 0x00 0x51000000 0x01000000>; /* CS0 space. Min partition = 16MB */
};

I did also remove that one for a test, but this does not change anything.

You mention this gpmc overlay dtbo file. How do I have to apply this to the kernel?

Mario Trams said:

You mention this gpmc overlay dtbo file. How do I have to apply this to the kernel?

If you boot from a SD card, you can add the following line in file uEnv.txt on the SD card boot partition (where all the U-Boot binaries are located), then U-Boot script will load it for kernel.

name_overlays=k3-am642-evm-nand.dtbo

Hi Bin,

as for this uEnv.txt extension: I tried this out. There were additional obstacles. I also had to copy /boot/dtb/ti/k3-am642-evm-nand.dtbo (from the rootfs, not bootfs!) into /boot/dtb/ (i.e. k3-am642-evm-nand.dtbo one level up). Else u-boot won't find it (it does not search in the ti-subdirectory).

When I'm booting that way directly from the SD card (i.e. not TFTPBoot and no NFS root) and then execute 

cat /sys/firmware/devicetree/base/bus@f4000/memory-controller@3b000000/status

I'm also getting "okay". Without the overlay I'm receiving "disabled", btw.

Though, dmesg is not showing me these gpmc-related messages that you cited lastly.

However, when I'm trying to read out the memory location at 0x3b000000 using devmem2, for instance, I'm receiving a bus error:

root@am64xx-evm:~#  devmem2 0x03B000000
/dev/mem opened.
Memory mapped at address 0xffff85f2f000.
Bus error (core dumped)
root@am64xx-evm:~#

Bin Liu said:

I just added status = "okay" in kernel k3-am642-evm.dts, &gpmc0 node, and I can see gpmc driver is loaded during kernel boot:

That's strange. Could you please cite the detailed change of k3-am642-evm.dts? And what is your procedure here after the change? I worked with tftpboot and rootfs here, and what I'm doing after the modification of the device tree file is:

make linux

make linux_stage

make linux_install

sudo cp board-support/built-images/Image /tftpboot/Image-am64xx-evm.bin

sudo cp board-support/built-images/dtb/ti/k3-am642-evm.dtb /tftpboot/ti/k3-am642-evm.dtb

I just tried cat /sys/firmware/devicetree/base/bus@f4000/memory-controller@3b000000/status 

as well here, and it returns me "okay" too. But accessing the CSR results in a bus error.

One thing that's unclear for me is why you see these gpmc-related messages in dmesg and I don't see them. 

What is doing devmem2 0x03B000000 on your side? 

I'm not sure, but for me this is looking like in the recent Linux kernel version they completely forbid this mmapping of such CSRs. It's kind of odd that a bus error is thrown then, but who knows....

Hi Bin,

as for the plain experiment using the overlay, I did re-attempt this in a very defined way from the scratch as follows:

• I wrote the default image (tisdk-default-image-am64xx-evm-11.00.09.04.rootfs.wic) onto an microSD card.
• I mounted the microSD card resp. it's boot partition.
• In the boot partition I renamed tiboot3-am64x-gp-evm.bin into tiboot3.bin (for the processor variant on the eval kit I need to do this).
• I added name_overlays=ti/k3-am642-evm-nand.dtbo into uEnv.txt on the boot partition.

I did boot the system from the prepared microSD card (no tftpboot). Afterwards

• dmesg | grep gpmc is showing me nothing
• cat /sys/firmware/devicetree/base/bus@f4000/memory-controller@3b000000/status is returning me "okay"
• devmem2 0x03B000000 is giving me a bus-error.

So what could I make wrong here?

As for your hint regarding my missing modules rebuild/update: I did check that. I re-installed the recent SDK 11 from scratch and set everything up for tftpboot and NFS root. When booting up, cat /sys/firmware/devicetree/base/bus@f4000/memory-controller@3b000000/status is returning me "disabled", which is as expected at that moment.

Next I did adjust board-support/ti-linux-kernel-6.12.17+git-ti/arch/arm64/boot/dts/ti/k3-am642-evm.dts in a way so that the relevant entry is looking as follows:

&gpmc0 {
        ranges = <0 0 0x00 0x51000000 0x01000000>; /* CS0 space. Min partition = 16MB */
        status = "okay";
};

So I just added status = "okay";

Then I re-build the kernel as previously described:

make linux

make linux_stage

make linux_install

Btw., as it seems this process does actually build and install all kernel modules as well as far as I can oversee the situation. 

Then I did just update the .dtb file and left the kernel image untouched:

sudo cp board-support/built-images/dtb/ti/k3-am642-evm.dtb /tftpboot/ti/k3-am642-evm.dtb

Sidenote here: Apparently there is still a bug in the SDK 11 so that the k3-am642-evm.dtb file is required to reside in /tftpboot/ti/ and not in /tftpboot/ where it resides by default. 

The result is the same as described above for the test with the default image and this NAND overlay. To repeat that:

• dmesg | grep gpmc is showing me nothing
• cat /sys/firmware/devicetree/base/bus@f4000/memory-controller@3b000000/status is returning me "okay"
• devmem2 0x03B000000 is giving me a bus-error.

Copying also the build kernel image into /tftpboot/ does not change anything here.

So to conclude, the process is quite straight-forward, actually. And I'm not seeing where I'm doing a major mistake here.

Hi Mario,

I am in the middle of another debug work and didn't review your latest details. Can you please following the steps below to just update the dtb on the sdcard to see if you still get bus error, then we will figure out the rest.

- flash the SDK default wic image to the sdcard;

- add status = "okay" to the &gpmc0 node in k3-am642-evm.dts to enable gpmc,

- do "make linux-dtbs" to compile kernel devicetree,

- then copy "board-support/ti-linux-kernel-.../arch/arm64/boot/dts/ti/k3-am642-evm.dtb" to sdcard root partition /boot/dtb/ti/ directory where should have the original k3-am642-evm.dtb file.

- boot the board with the sdcard, the /sys/firmware/devicetree/base/bus@f4000/memory-controller@3b000000/status should be "okay",

- and kernel boot log should have a gpmc init message. If not, you would get bus error when reading gpmc registers.

- then check if devmem2 0x03B000000 still has bus error. 

Hi Bin,

Ok, I did try to do this in a very defined manner as follows:

• I prepared a microSD card with the default image (tisdk-default-image-am64xx-evm-11.00.09.04.rootfs.wic).
• I re-installed the SDK 11.00.09.04 (I deleted the existing installation completely).
• I fixed a bug in bin/setup-targetfs-nfs.sh of the SDK (I don't think that this is related here, but it's worth mentioning as this bug is still present in the recent SDK 11.00.09.04. I reported that one here: PROCESSOR-SDK-AM64X: Bug in setup script for version 09.02.01.10 (solved) )
• I did execute setup.sh. Actually straight-forward including configuration of tftpboot, NFS, etc. (although not necessarily needed for the particular test as we are not going to use either one here).

• I did adjust board-support/ti-linux-kernel-6.12.17+git-ti/arch/arm64/boot/dts/ti/k3-am642-evm.dts in a way so that the relevant entry is looking as follows:

  &gpmc0 {
          ranges = <0 0 0x00 0x51000000 0x01000000>; /* CS0 space. Min partition = 16MB */
          status = "okay";
  };

• I did run make linux-dtbs  
• I copied board-support/ti-linux-kernel-6.12.17+git-ti/arch/arm64/boot/dts/ti/k3-am642-evm.dtb into /boot/dtb/ti/ of the root-Partition (not the boot-Partition, to make that clear) of the prepared microSD card.
• In the boot partition of the microSD card I renamed tiboot3-am64x-gp-evm.bin into tiboot3.bin (for the processor variant on the eval kit I need to do this).
• Then I did boot the system straight from the microSD card.

Now guess what: The issue remains the same. I.e. no reference to gpmc in dmesg, status is showing "okay", and devmem2 is resulting in a bus error.

I have got another suspicion in the mean time. Plain and simple: The standard kernel module for the GPMC is not being built by default in the SDK 11.00.09.04. In the old SDK 09 I'm finding  a file board-support/ti-linux-kernel-6.1.83+gitAUTOINC+c1c2f1971f-ti/drivers/memory/omap-gpmc.o - i.e. a compiled source file. In the SDK 11.00.09.04 I'm just finding board-support/ti-linux-kernel-6.12.17+git-ti/drivers/memory/omap-gpmc.c, which presumably is the source file to create omap-gpmc.o from. Nonetheless it is strange that things appear to work on your site.

Hi Mario,

I will try this exact process once I am in office to see what is missing. I did see 2 gpmc related messages in kernel boot log in my last test and devmem2 can read gpmc registers.

Mario Trams said:

In the SDK 11.00.09.04 I'm just finding board-support/ti-linux-kernel-6.12.17+git-ti/drivers/memory/omap-gpmc.c, which presumably is the source file to create omap-gpmc.o from.

If you only did 'make linux-dtbs', omap-gpmc.o won't be generated, this command only compile the kernel arch/arm64/dts/ directories.

Please try 'make linux' to see if omap-gpmc.o will be generated.

Hi Bin,

no, it's not just because of make linux-dtbs only. I already did a complete build before posting this above. And this module is also missing in the default image then, consequently. I suspect you are testing with another Linux kernel configuration. I'm already about to search where I can enable the compilation of this default gpmc module.

Yes, I typically directly work on the kernel repo (for easily switching kernel versions and upstream remotes, etc), not the one in the SDK. But I will check the kernel in the SDK11.0 to see why omap-gpmc is not enabled/compiled.

Hi Bin,

ok, so that's kind of clear to me now....

CONFIG_OMAP_GPMC is definitely not set in board-support/ti-linux-kernel-6.12.17+git-ti/.config (it is set in the SDK 09 at the according place). However .config is automatically generated and changing it is useless.

In the mean time it seems that I'm unable to locate where I have to enable the default GPMC driver.

I found indications that in make menuconfig it is to be found under Device Drivers -> Memory Technology Device -> NAND. But I'm not finding anything related there.  I did also scan through many other menus - nothing...

Hi Bin,

ok, that did the trick! Thanks a lot! I did a complete SDK reinstall again and applied that patch prior to make linux etc. 

Glad the issue is resolved. I wasn't expect the gpmc driver isn't enabled in SDK11.0 kernel defconfig, though I don't touch gpmc very often.

Seems I'm one out of three persons on the planet dealing with the GPMC. But I believe that this is a very powerful thing when it comes to the attachement of proprietary hardware to the processor - taking aside a PCIe option. And it seems that this is a unique feature among the competitors. It's a bit sad that the larger brother of the AM64x (AM65x) is just supporting a 16 bit wide GPMC

Well, I meant to say I didn't touch gpmc often in SDK11.0 since it was new. Most of my customers using gpmc are on SDK9.x and 10.x. gpmc is very popular in attaching fpga usecases.

I'm also aiming at FPGA interfacing. Yes, it makes certainly not necessarily sense to run after the newest SDK. I was kind of joking with those "3 persons", but I guess that not many make use of the GPMC - though I think that it is very important.

We do recommend to use the latest SDK for any new project, not only because of bug fixes, but also for longer support time.

Bin Liu said:

Most of my customers using gpmc are on SDK9.x and 10.x.

Only because those customers have started the projects a couple years ago.