SK-AM62: Cannot Enable DMA for MCSPI0 on Linux

Hi Claudio,

doing a quick check on your changes I'd say the thread IDs look good, so we'll need to look a bit closer here as to why it's not doing the trick. I've re-assigned the query to one of my team members that will coordinate further.

Regards, Andreas

Hello Claudio,

Questions

Which version of the Linux SDK are you using?

Are you using the standard boot flow of the Linux SDK (SPL), or something else?

Initial discussion

I am not a DMA or SPI expert, so I will be learning with you.

When I check the default resource management configuration in AM62x Linux SDK 8.6 under /board-support/k3-image-gen-2022.01/soc/am62x/evm/rm-cfg.c, I do not see any mention of PSI or spi. Same for when I check AM62x Linux SDK 9.0 in the new resource management location of board-support/ti-u-boot/board/ti/am62x/rm-cfg.yaml

So assuming you are using one of those two SDK releases, and using the standard Linux SPL boot with the standard resource management settings, I would not expect it to be an issue of "SPI2 is getting some DMA resources allocated in the default RM configs that SPI0 is not getting".

We will ask around with some other team members, and plan to reply back to you next week.

Regards,

Nick

Hi Andreas, Hi Nick,

First thanks for your time troubleshooting with me.

Since we are here using some customized sources at the time I just did the following test to be sure the problem was not caused by our changes:

I installed the latest processor SDK for AM62x (v09.00.00.03) from here: https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-AM62X/09.00.00.03. I copied the shipped pre-built image from SDK to SD card using the <sdk-path>/bin/create-sdcard.sh script. I selected tisdk-default-image-am62xx-evm.tar.xz tarball. I made the image boot successfully on my AM62x-SK EVM (revision E3A). Then I took the original device tree file from <sdk-path>/board-support/ti-linux-kernel/arch/arm64/boot/dts/ti/k3-am625-sk.dts and only added the following:

diff --git a/arch/arm64/boot/dts/ti/k3-am625-sk.dts b/arch/arm64/boot/dts/ti/k3-am625-sk.dts
index b17379781..d8ae5a256 100644
--- a/arch/arm64/boot/dts/ti/k3-am625-sk.dts
+++ b/arch/arm64/boot/dts/ti/k3-am625-sk.dts
@@ -191,6 +191,16 @@ main_wlirq_pins_default: main-wlirq-pins-default {
                        AM62X_IOPAD(0x128, PIN_INPUT, 7) /* (B23) MMC2_SDWP.GPIO0_72 */
                >;
        };
+
+       uec_spi0_pins_default: uec-spi0-pins-default {
+               pinctrl-single,pins = <
+                       AM62X_IOPAD(0x01bc, PIN_INPUT, 0) /* (A14) SPI0_CLK */
+                       AM62X_IOPAD(0x01b4, PIN_INPUT, 0) /* (A13) SPI0_CS0 */
+                       AM62X_IOPAD(0x01b8, PIN_INPUT, 0) /* (C13) SPI0_CS1 */
+                       AM62X_IOPAD(0x01c0, PIN_INPUT, 0) /* (B13) SPI0_D0 */
+                       AM62X_IOPAD(0x01c4, PIN_INPUT, 0) /* (B14) SPI0_D1 */
+               >;
+       };
 };
 
 &main_i2c1 {
@@ -222,6 +232,22 @@ exp1: gpio@22 {
        };
 };
 
+&main_spi0 {
+       status = "ok";
+       pinctrl-names = "default";
+       pinctrl-0 = <&uec_spi0_pins_default>;
+       ti,pindir-d0-out-d1-in = <1>;
+
+       dmas = <&main_pktdma 0xc300 0>, <&main_pktdma 0x4300 0>;
+       dma-names = "tx0", "rx0";
+
+       spidev@0 {
+               compatible = "micron,spi-authenta";
+               spi-max-frequency = <50000000>;
+               reg = <0>;
+       };
+};
+
 &sdhci1 {
        vmmc-supply = <&vdd_mmc1>;
        vqmmc-supply = <&vdd_sd_dv>;

The I booted again on same SK EVM and checked the boot messages:

root@am62xx-evm:~# dmesg | grep dma
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: initialized node ipc-memories@9c800000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: initialized node m4f-dma-memory@9cb00000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: initialized node m4f-memory@9cc00000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@9da00000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: initialized node r5f-memory@9db00000, compatible id shared-dma-pool
[    1.156999] ti-udma 485c0100.dma-controller: Number of rings: 82
[    1.165196] ti-udma 485c0100.dma-controller: Channels: 48 (bchan: 18, tchan: 12, rchan: 18)
[    1.176355] ti-udma 485c0000.dma-controller: Number of rings: 150
[    1.186407] ti-udma 485c0000.dma-controller: Channels: 35 (tchan: 20, rchan: 15)
[    5.699787] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.763267] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.774411] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.782827] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.790823] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.800911] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.813415] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.821406] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.830086] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.843332] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.853303] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.864095] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.877750] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.886445] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.895617] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.903502] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.903527] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.923664] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.931547] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.939441] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.952541] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.960419] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.968814] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.977048] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.985090] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    5.993039] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    6.002824] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    6.014691] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    6.029488] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    6.037492] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    6.050957] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    6.058844] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    6.066748] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    6.079964] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    6.091208] ti-udma 485c0000.dma-controller: No configuration for psi-l thread 0x4300
[    6.102694] ti-udma 485c0000.dma-controller: get channel fail in udma_of_xlate.
[    6.167567] k3-m4-rproc 5000000.m4fss: assigned reserved memory node m4f-dma-memory@9cb00000
[    6.218537] rproc-virtio rproc-virtio.0.auto: assigned reserved memory node m4f-dma-memory@9cb00000
[    6.470782] platform 78000000.r5f: assigned reserved memory node r5f-dma-memory@9da00000
[    6.506600] rproc-virtio rproc-virtio.3.auto: assigned reserved memory node r5f-dma-memory@9da00000

So I get the same error also on latest default image.

Here is the output of directory /sys/devices/platform/bus@f0000/20100000.spi/:

root@am62xx-evm:~# ls -l /sys/devices/platform/bus@f0000/20100000.spi/
total 0
lrwxrwxrwx 1 root root    0 Jan  1 00:00 driver -> ../../../../bus/platform/drivers/omap2_mcspi
-rw-r--r-- 1 root root 4096 Jan  1 00:00 driver_override
-r--r--r-- 1 root root 4096 Jan  1 00:00 modalias
lrwxrwxrwx 1 root root    0 Jan  1 00:00 of_node -> ../../../../firmware/devicetree/base/bus@f0000/spi@20100000
drwxr-xr-x 2 root root    0 Jan  1 00:00 power
drwxr-xr-x 3 root root    0 Jan  1 00:00 spi_master
lrwxrwxrwx 1 root root    0 Apr 28  2022 subsystem -> ../../../../bus/platform
lrwxrwxrwx 1 root root    0 Jan  1 00:00 supplier:platform:44043000.system-controller:clock-controller -> ../../../virtual/devlink/platform:44043000.system-controller:clock-controller--platform:20100000.spi
lrwxrwxrwx 1 root root    0 Jan  1 00:00 supplier:platform:44043000.system-controller:power-controller -> ../../../virtual/devlink/platform:44043000.system-controller:power-controller--platform:20100000.spi
lrwxrwxrwx 1 root root    0 Jan  1 00:00 supplier:platform:f4000.pinctrl -> ../../../virtual/devlink/platform:f4000.pinctrl--platform:20100000.spi
-rw-r--r-- 1 root root 4096 Apr 28  2022 uevent
root@am62xx-evm:~#

there are no dma:rx0, dma:tx0 symlinks inside which proofs that DMA was not enabled for MCSPI0.

Regards,

Claudio

Hi Claudio

Please expect additional delays on us getting back on this, as Andreas is on a customer visit last week and upcoming week. 

We regret the delay.

Regards

Mukul 

Hi Claudio,

I'm finally back in the office and I'm prioritizing your thread here from now on. Currently setting things up so I can see if I can re-create what you see and then debug. Will get back soon.

Regards, Andreas

Hi Claudio,

can you please confirm whether the above worked for you in the context of SDK v9.1, and it make some impact on your SPI transfer?

Your recent/related post at https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1310321/sk-am62-improve-poor-bandwidth-utilization-on-mcspi-bus seems to confirm the DMA is being used now, but I wanted to make sure before closing this thread here (of course we still need to look at the new concern you reported).

Regards, Andreas

Hi Andreas, 

sorry for late reply, there was vacation in between. 

I did not find time to test your solution by now, I will try it by end of this week and reply again. 

But don't get me wrong here: I'm still not able to use DMA on MCSPI0 (specifically port #0). In the other post you mentioned we were using MCPSI #2, there we never had issues (like mentioned above). But now I looked over my old post and saw that I have not mentioned using MCSPI2 explicitly, I will add this.

So to wrap up:

• We still cannot use DMA on MCSPI0 (but I have not yet tested your solution above).
• On MCSPI2 we were always able to use DMA but we still have too long word delays (that's why I posted the other thread).

Regards, Claudio 

Ah ok, thanks for the clarification. Will be waiting for your feedback on MCSPI0 then.

Thanks, Andreas

Hi Andreas,

sorry for the late reply but finally I could find some time testing your solution. I can confirm that your solution also works on my side. At least I can no longer see those "No configuration for psi-l thread" errors and I can also see that dma rx and tx values are set now under /sys/devices/platform/bus@f0000/20100000.spi/.  But there are still two questions which I would like to have answered:

1) Given your setup (using the piggy-backed  k3-am625-sk-mcspi-loopback.dtso file) I was not able to see any signal on the SPI0 lines at SKEVM User Expansion Header. For example from the pin muxing in k3-am625-sk-mcspi-loopback.dtso I was expecting the SPI0 CLK signal on pin  23 of SKEVM User Expansion Header, but I couldn't see anything there, same goes for D0, D1, etc. Here is the simple code I used for testing from user space: 

#include <fcntl.h>
#include <linux/spi/spidev.h>
#include <string>
#include <string.h>
#include <sys/ioctl.h>

#define BUFF_SIZE 4096

struct spiDev
{
    int fd;
    int mode = 3;
    uint8_t bits_per_word = 8;
    uint16_t delay_usecs = 0;
    uint8_t word_delay_usecs = 0;
    uint32_t speed_hz = 31200000;
    uint8_t txBuf [BUFF_SIZE] = {0};
    uint8_t rxBuf [BUFF_SIZE] = {0};
};

int SPIDataRW(spiDev &spidev, int len)
{
    struct spi_ioc_transfer spiTransfer;
    memset(&spiTransfer, 0, sizeof(spiTransfer));
    spiTransfer.tx_buf = (uint64_t) spidev.txBuf;
    spiTransfer.rx_buf = (uint64_t) spidev.rxBuf;
    spiTransfer.len = len;
    spiTransfer.speed_hz = spidev.speed_hz;
    spiTransfer.bits_per_word = spidev.bits_per_word;
    spiTransfer.word_delay_usecs = spidev.word_delay_usecs;
    spiTransfer.delay_usecs = spidev.delay_usecs;
    // spiTransfer.cs_change = static_cast<bool>(false);
    
    return ioctl(spidev.fd, SPI_IOC_MESSAGE(1), &spiTransfer);
}

int main(int argc, char **argv)
{
  char *spi_devname = "/dev/spidev1.0";

  if (argc == 2) spi_devname = argv[1];

  printf("using %s\n", spi_devname);

    spiDev spidev {};
    spidev.fd = open(spi_devname, O_RDWR);
    
    ioctl(spidev.fd, SPI_IOC_WR_BITS_PER_WORD, &spidev.bits_per_word);
    ioctl(spidev.fd, SPI_IOC_RD_BITS_PER_WORD, &spidev.bits_per_word);
    ioctl(spidev.fd, SPI_IOC_WR_MAX_SPEED_HZ, &spidev.speed_hz);
    ioctl(spidev.fd, SPI_IOC_RD_MAX_SPEED_HZ, &spidev.speed_hz);
    
    printf("start sending one byte per tx (BUFF_SIZE = %i)\n", BUFF_SIZE);
    while(true)
      spidev.txBuf[0] = 0xAA; // write dummy byte
      SPIDataRW(spidev, BUFF_SIZE);
  

    return 0;
}

Perhaps there are some DT settings overwriting the SPI settings from the overlay. Could you give me a hint what else I have to change in order to see the signals? It would help me to confirm that the solution is really using DMA in the end (I usually can see this from the signals).

2) The key difference of your posted solution is that you used a newer version of the SDK (09.01.00.08). Since on the old SDK version 09.00.00.03 it was not working I'm now wondering what change to the SDK sources was fixing our DMA issue. This would be very beneficial for us since for our final application we are using a custom board with custom Linux Kernel and we would now have to apply the same fix(es) to make it run on our side. Any hint s there would be nice!

Thanks Andreas for your time.

Regards, Claudio

Claudio Raisch said:

Given your setup (using the piggy-backed  k3-am625-sk-mcspi-loopback.dtso file) I was not able to see any signal on the SPI0 lines at SKEVM User Expansion Header.

I don't have the HW setup right now to look at the signals, but here's an E2E thread that has a working solution for AM62x SPI0. Can you please check it and see if you can make it work. Perhaps something with the pinmux or the address-cells / size-cells device properties is wrong. https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1263441/sk-am62-no-spi-signals-on-expansion-headers/4783737

Claudio Raisch said:

2) The key difference of your posted solution is that you used a newer version of the SDK (09.01.00.08). Since on the old SDK version 09.00.00.03 it was not working I'm now wondering what change to the SDK sources was fixing our DMA issue. 

If I was to take an educated guess I'd start with looking for differences caused by TIFS firmware (which is responsible for the DMA "resource management" in the device), or the associated board configuration data (contained in the U-Boot source tree starting SDK v9.0). Since both TIFS firmware and board config data get loaded by the ROM bootloader they are both part of the initial boot binary known as tiboot3.bin. So for a quick experiment you could just plug the tiboot3.bin from SDKv9.1 into an SDK card boot image from SDK v9.0, and only transplanting this file. And see if this works. Then we can narrow down further.

Regards, Andreas

Hi Andreas,

Andreas Dannenberg said:

I don't have the HW setup right now to look at the signals, but here's an E2E thread that has a working solution for AM62x SPI0. Can you please check it and see if you can make it work. Perhaps something with the pinmux or the address-cells / size-cells device properties is wrong. https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1263441/sk-am62-no-spi-signals-on-expansion-headers/4783737

Thanks to the steps explained in the posted thread I'm now able to see the signals. I can also confirm now that DMA indeed is indeed enabled.

Andreas Dannenberg said:

If I was to take an educated guess I'd start with looking for differences caused by TIFS firmware (which is responsible for the DMA "resource management" in the device), or the associated board configuration data (contained in the U-Boot source tree starting SDK v9.0). Since both TIFS firmware and board config data get loaded by the ROM bootloader they are both part of the initial boot binary known as tiboot3.bin. So for a quick experiment you could just plug the tiboot3.bin from SDKv9.1 into an SDK card boot image from SDK v9.0, and only transplanting this file. And see if this works. Then we can narrow down further.

Because I only had an image of SDK v9.1 at hand I tested it the other way round. I was taking the tiboot3.bin of SDK image v9.0 and copied it to the image v9.1. Turning around the logic I would now expect the DMA to not work now on SPI0. However the experiment showed that DMA was configured successful: I could see the dma:rx/tx entries at /sys/devices/platform/bus@f0000/20100000.spi/.

Given that result we can say that tiboot3.bin is not the cause of the errors. I guess then we should go on looking at changes made to u-boot or kernel, or do you have a more precise idea where to look? 

Regards, Claudio

Claudio Raisch said:

Thanks to the steps explained in the posted thread I'm now able to see the signals. I can also confirm now that DMA indeed is indeed enabled.

Glad to hear!

Claudio Raisch said:

Given that result we can say that tiboot3.bin is not the cause of the errors. I guess then we should go on looking at changes made to u-boot or kernel, or do you have a more precise idea where to look? 

Ok. So since your concern is around Kernel I'll say we cross U-Boot off the list right way. Then, as for the Kernel, generally speaking I'd expect changes to be more of "evolutionary" nature between the TI SDKs, but somewhere in there should be the trigger. What I'd do next is A-B test combinations of kernel and device tree files to see if the behavior is cause by a "bad" Kernel or "bad" device tree files.

Regards, Andreas

We now started searching through the ti-linux repo for patches applied and related to MCSPI subsystem. We found already one very promising commit: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/drivers/dma/ti/k3-psil-am62.c?h=ti-linux-6.1.y-cicd&id=24372b26924880b910204c0b481766c6349efd31. There the missing PSI-L thread numbers are added which were previously causing the errors.

We applied that patch our custom image to see if it already resolves our problem. But even though the error messages (mentioned above) are now gone, DMA still seems not to be enabled. I can see this from looking at the SPI CLK signal which still has very long inter-word gaps (~1us). Usually when DMA is working we see that these gaps reduce down to ~200ns and therefore the overall transfer time per byte reduces drastically. But we don't see this after applying this patch.

We will go on testing out patches which sound helpful. I just wanted to mention this here.

Thanks for the update. As mentioned in the other thread I'm planning on spending some hands-on time with this next week.

DMA was enabled successfully, so closing this older thread. There's still some ongoing discussion regarding SPI inter-byte gap optimization on a related thread here https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1310321/sk-am62-improve-poor-bandwidth-utilization-on-mcspi-bus/5209352

Regards, Andreas