AM6442: R5 SPI DMA Troubleshooting

Hello Lawrence,

I am working on your thread.

Please give me a day or two to get back to you.

Regards,

S.Anil

Hello Lawrence,

Is it ok to share your project for further debugging?

Regards,

S.Anil.

Sure that is fine

Hello Lawrence,

Please share your project for further debugging.

Regards,

S.Anil.

Hi Swargam,

Sorry for the delay. https://txn.box.com/s/et5qlg3sw9h69guijbdtozxmwnohqqzx 

Let me know if you have any trouble downloading the project.

Regards,

Lawrence

Hello Lawrence ,

I downloaded your software .

Please allow me some time to get back to you .

Regards,

S.Anil.

Hello Lawrence ,

As per your comments customer is using two MCSPI channels with DMA mode but I see single MCSPI channel +DMA in sysconfif file .

Can you please share the actual sysconf file that uses 2MCSPI channels + DMA.

I need to check how they have allocated DMA channels for 2MCSPI channels as they are facing problem in DMA and not in interrupt mode.

Regards,

S.Anil. 

Hi Swargam

Attached the config file and shared the project with 2 MCSPI channels with DMA mode.

Customer have created 2MCSPI interfaces+DMA mode, on top of ipc rpmsg echo linux example project. Also enabled the UART logging and replaced the SRAM to DDR1 in linker.cmd file. With these changes getting the error while creating end point. If they change the DMA to interrupt mode on both 2 McSPI configuration then it is working.

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/1045/8321.example.syscfg

Project file for 2x McSPI

https://txn.box.com/s/iid9303qcbkcdty4ewvpc9yaaex6syt1

Regards,

Lawrence

Hello Lawrence ,

"I need your help to isolate the issue with either DMA channel configuration to MCSPI or resource allocation of RTOS vs Linux.

Your customer needs to do one test. Please do a test to see if MCSPI is communicating to sensors without integrating the Linux RPMSG echo Linux example in the project and share the results.

This way, we can easily find the problem with either DMA channel allocation for MCSPI or resource allocation of RTOS vs Linux.

Meanwhile, I can look at your shared configuration file and let you know if you've done anything wrong."

Regards,

S.Anil.

Hello Lawrence,

I looked at your sysconfig file and software changes.

I found that you missed the initialization of the UDAM for the MCSPI channel in the software.

Regards,

S.Anil.

Hi Swargam,

Feedback from customer in trying to isolate the DMA (waiting for your testing on UDAM initialization) :

1. My team Hari and Prashant worked on MCSPI on our BD project which I don’t have the results of it.

So to isolate the issue I have taken RP Msg hello world example and enabled the MCSPI DMA mode to reproduce the issue. As soon as I changed from DMA to Interrupt of MCSPI then “virtio_rpmsg_bus virtio1: creating channelchannels” were created in R5 and from A53, I can able to communicate.

With the DMA I don’t see these prints in R5F

2. Another issue which is not related to above:

When I am trying to transfer large data around 724 bytes from A53 to R5 using RPMsg, I am getting an error “Message is too long”

As per the below documentation I have created pointer to the data and send it across but still getting the same error.

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1130560/am3358-remoteproc-rpmsg-with-large-payload-meta-ti/4402841?tisearch=e2e-sitesearch&keymatch=Can%252527t%252520write%252520to%252520rpmsg%252520endpt%252520device#4402841

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1169066/sk-am64-follow-up-question-how-to-read-write-data-with-the-pru_icssg-xfr2vbus-hardware-accelerator/4402839#4402839

From the above thread looks like RPMsg can be used as a notification service between cores alongside shared memory and suggesting to use the “rpmsg_char_zerocopy” sharedmemory example to transfer the large data. Can we use the “rpmsg_char_zerocopy” or please suggest the right example which we can use to copy the large data between the cores.

 3. When I run the DDS(runs every 1ms cycle)  + RpMsg Linux application on A53 getting the some warning on the serial console and after sometime board doesn’t respond. I have to poweroff/on the board. Attached the log for reference.

Regards,

Lawrence

Hello Lawrence,

I will be commenting from the Linux RPMsg side.

RPMsg & DMA

I have not tried running RPMsg with different DMA settings, so there could be something that I am missing. However, RPMsg does not use DMA. VIRTIO buffers are defined somewhere in memory, and then each core doing RPMsg reads or writes data to those buffers.

RPMsg packet size

Please note this "important usage point" in the screenshot above:
"When Linux is one end of the IPC message exchange, the max RPMessage packet or buffer size is 512 bytes"

Linux RPMsg actually ONLY uses buffers of size 512 bytes. So whenever you send or receive a Linux RPMsg message, no matter how many characters or bits you are trying to send you are sending or receiving exactly 512 bytes of information. This is different from RPMsg between two MCU+ cores.

16 bytes of that is actually a header, so you can only send 496 bytes of data per RPMsg packet.

If your usecase needs to send a lot more data between Linux and the remote core, the rpmsg_char_zerocopy example is a good example to look at. This defines a shared memory region, one core writes to the shared memory region, and then that core sends an RPMsg message to notify the other core once the shared memory region is ready to be read.

Regards,

Nick

Nick, Swargam,

Some clarification on the DMA usage.  

Please find the Ipc RPMSG Echo Linux example with the MCSPI DMA configuration below:

1. Without any MCSPI, rpmsg is working.
2. With one MCSPI+DMA, rpmsg is working on every boot cycle, but when we restart the R5F core(using stop/start command) rpmsg is not working.

3. With two MCSPI+DMA, rpmsg is not working on every boot cycle, also when we restart the R5F core(using stop/start command) rpmsg is not working.

4. With one MCSPI+Interrupt, rpmsg is working always on every boot cycle and restart of the R5F core.
5. With two MCSPI+Interrupt, rpmsg is working always on every boot cycle and restart of the R5F core.

Project file: 

https://txn.box.com/s/is3l05ss8wqtpm1gxkm64gkk71132n0r 

Hello Lawrence,

Double-checking on the customer's observations:

I want to verify something here. When you say "rpmsg is working" or "rpmsg is not working", are you JUST talking about RPMsg? Or are you talking about the entire example?

e.g., for 3., are you saying this:

3. With two MCSPI+DMA, rpmsg is not working on every boot cycle, but the MCSPI code is working as expected. Also when we restart the R5F core(using stop/start command) rpmsg is not working, but the McSPI code is working as expected.

or are you saying this:

3. With two MCSPI+DMA, the entire example (including rpmsg) is not working on every boot cycle. Also when we restart the R5F core(using stop/start command) the entire example (including rpmsg) is not working.

If the SPI and DMA operations are working fine after each boot cycle, but RPMsg is the only remote core feature not working properly, that tells us something different about what is going on.

Alternative methods of debug: Using CCS

If RPMsg was the debug method the customer was using to gain visibility into the remote core, please reference this FAQ where we discuss using CCS to observe code that is currently running on a remote core: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1188177/faq-am62x-am64x-how-to-use-ccs-to-debug-a-running-m4f-core-that-was-started-by-linux?tisearch=e2e-sitesearch&keymatch=faq%3Atrue

Regards,

Nick

FYI: graceful shutdown as of SDK 8.5

Another thing to keep in mind: As of SDK 8.5, we do not support graceful shutdown of remote cores. That means if you use the stop command to bring a remote core down, the core is brought down in an unknown state. So sometimes the core can be restarted, but sometimes trying to start the remote core will just crash the processor.

In general, for now it is best to reboot the entire processor when loading new firmware to a remote core (or when reloading the same firmware).

We are planning to add graceful shutdown for remote cores sometime in 2023. I am double-checking on the timeline.

-Nick

Also, please attach the customer's Linux devicetree files 

After skimming through ipc_rpmsg_echo_linux_am64x-evm_r5fss0-0_freertos_ti-arm-clang-ipcDMATest, the Linux RPMsg settings look the way I would expect. However, I did not see any McSPI code in the .c files. Am I missing something? Or have the customers only enabled McSPI in the SysConfig settings without adding any code at this point in time?

It would be helpful to see the Linux devicetree files that the customers are using to make sure there are not any conflicting settings between R5F and Linux.

Regards,

Nick

Hi Nick,

I am getting more information from customer.  For now this is what they have shared:

================

Can you confirm if the team is able reproduce the problem as per the steps ? We are just initializing the UDMA driver and not using the MCSPI APIs yet (only sysconfig change). To me it looks like System Initialization issue, but we do not see the debug logs on serial to confirm if UDMA init failed.

A few observations:

• When we use interrupt mode the UDMA driver is not initialized. – RPMsg char device creation succeeds

• When we introduce MCSPI with DMA mode during system init the UDMA driver is initialized.  – RPmsg char device creation fails

• UDMA initialization failure shall not prevent rpmsg_chrdev channel creation
• Is this linked to the graceful shutdown fo r5 for which fix is expected in near future ?

>> Another thing to keep in mind: As of SDK 8.5, we do not support graceful shutdown of remote cores. That means if you use the stop command to bring a remote core down, the core is brought down in an unknown state. So sometimes the core can be restarted, but sometimes trying to start the remote core will just crash the processor.

I don’t think above msg in failed case suggest processor crashed. Will add more to this. Also please let us know if we need to capture other states of the processor to confirm. FYI the example says evm but we are using it on am64x-sk does that make any difference for the RPMsg example on SK?

=================

Additional feedback from customer to your questions:

I want to verify something here. When you say "rpmsg is working" or "rpmsg is not working", are you JUST talking about RPMsg? Or are you talking about the entire example?

[Customer] rpmsg is working means “RPmsg char device creation working” and rpmsg_char_simple -r 2 -n 10 command is working(on Linux A53 core)

With two MCSPI+DMA, rpmsg is not working on every boot cycle,but the MCSPI code is working as expected. Also when we restart the R5F core(using stop/start command) rpmsg is not working, but the McSPI code is working as expected.

[Customer] No. We selected MCSPI+DMA and generated driver code but we didn’t  used/tested  MCSPI in our application.

With two MCSPI+DMA, the entire example (including rpmsg) is not workingon every boot cycle. Also when we restart the R5F core(using stop/start command) the entire example (including rpmsg) is not working.

[Customer] Yes, We checked only rpmsg, We selected MCSPI+DMA and generated driver code but we didn’t used/tested  MCSPI in our application

After quick review through ipc_rpmsg_echo_linux_am64x-evm_r5fss0-0_freertos_ti-arm-clang-ipcDMATest, the Linux RPMsg settings look the way I would expect. However, I did not see any McSPI code in the .c files. Am I missing something? Or have you only enabled McSPI in the SysConfig settings without adding any code at this point in time?

[Customer] We enabled MCSPI in the SysConfig settings without adding any application code for the UDMA.

Hello Lawrence,

1) Can we get the customer to connect to the M4F core in CCS to see where the program is? I am curious to see if the code is hanging somewhere, in an error state, etc.

More information about connecting to a running M4F core is here: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1188177/faq-am62x-am64x-how-to-use-ccs-to-debug-a-running-m4f-core-that-was-started-by-linux

2) Can we get the customer to share their Linux devicetree?

We will check with the MCU+ developers to verify that SPI + DMA is tested and supported, and to check whether they have run any tests while Linux was up.

Regards,

Nick

Hello Kiran, Venkat, Prashant,

Following up from our discussion today. If you can run this debug on your side on Tuesday, it will help my discussions later this week.

STEP 1: Make sure Linux devicetree was not requesting the same resources by using a different devicetree file 

When I check the AM64x SK Linux devicetree files, mcu_spiX instances are disabled in the Linux devicetree file k3-am642-sk.dts. This is good - if a remote core is using a peripheral, you do NOT want Linux to try to enable the peripheral and use it.

HOWEVER, based on your screenshots above it looks like you are using the main domain SPI instead of the MCU domain SPI. If that is true, I would expect you to have problems with an unmodified Linux devicetree. We can see main_spiX instances defined in k3-am64-main.dtsi. main_spi0 even requests DMAs. The main_spiX nodes are not referenced again in the board-level devicetree file k3-am642-sk.dts.

This is the problem: if Linux defines a resource table node, even if it is in a .dtsi file, then that node is enabled by default. That means Linux may try to claim the SPI instances. To prevent that, we need to explicitly disable the main domain SPI instances in the Linux devicetree.

I verified that yall are using RT Linux SDK 8.5. The code I am about to attach was created with that version of the SDK.

Please try using this Linux devicetree where I disable all main domain SPI peripherals (OSPI is currently still enabled):

Output file: https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/k3_2D00_am642_2D00_sk.dtb

patch to modify the source file yourself: https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/0001_2D00_disable_2D00_all_2D00_main_2D00_domain_2D00_SPI_2D00_instances.patch

steps to build Linux devicetree file: (does NOT require a kernel build, but the compiler must be added to the path. Can use the default kernel configuration): https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/08_05_00_21/exports/docs/linux/Foundational_Components_Kernel_Users_Guide.html 

STEP 2: Isolate whether the issue is related to RPMsg or not 

If the updated Linux devicetree file does not fix the issue, then I would like us to verify whether RPMsg has anything to do with the issue or not.

Please follow the steps at this FAQ to add an empty resource table to the MCU+ project, WITHOUT enabling Linux IPC. Do you still see issues? https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1193620/faq-am62x-am64x-how-to-allow-linux-to-load-m4f-r5f-firmware-by-adding-a-resource-table

Apologies if the FAQ is a bit rough, I pushed out this rough draft to enable you today. If you are not familiar with git repos, in a Linux terminal you can cd to the base directly of the MCU+ SDK and add in the projects I created with
git init (this creates the git repo)
git apply [patch]

Regards,

Nick

Request for feedback:

After going through the resource table FAQ:

1) My assumption with the code is that all customers would have some kind of git version control for their code, and thus all customers would be able to apply a git patch. Does that seem reasonable?

2) In the first draft, I provided the code as entirely new projects. However, I could also provide the code just as a modification to the original hello_world project - in that case, the patch file would more clearly show the differences between the base hello_world and the modified hello_world. Any thoughts about which is more effective?

3) any other feedback?

Regards,

Nick

Hi Nick.

After modifying "Output file: k3-am642-sk.dtb" file still which has disabling  all main domain SPI peripherals on  Linux devicetree, we still see the issue as RPMsg channel is not initialized.

Accidentally deleted Venkata's reply. re-posting it here:

Hi,

I have took the helloworld cpp project and added empty resource table as mentioned in "examples/drivers/ipc/ipc_rpmsg_echo_linux/am62x-sk/m4fss0-0_freertos/ti-arm-clang/linker.cmd" and enabled the Rpc IPC setting and loaded. With the change I am getting the Hello_World print.

After that I have added MCSPI with DMA and compiled and tested and this time I couldn't see Hello world Print.

Then I loaded the helloword*.out file through CCS and it was asserting in udma init.

Hello Venkata,

Summarizing what we learned from your responses

TEST 1: does the Linux devicetree disabling SPI peripherals make a difference? NO

TEST 2: Is RPMsg causing the issue, or is it something else? not completed.

I was looking for you to use Linux to load the remote core WITHOUT Linux IPC / RPMsg enabled to see if the McSPI would work. My assumption is no, RPMsg will not make a difference. But if you want to make sure, you would want to follow approach 2 version A or B, not approach 1. https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1193620/faq-am62x-am64x-how-to-allow-linux-to-load-m4f-r5f-firmware-by-adding-a-resource-table 

Next steps 

It is possible that the resource manager configs (rm configs) that are getting used in your CCS loading method are different than the rm configs that are used when Linux boots. That could cause issues if the R5 core is not given access to the DMA. We are comparing the MCU+ SDK rm configs against the Linux SDK rm configs for SDK 8.5, and will get back to you.

Regards,

Nick

Regarding TEST 2: Is RPMsg causing the issue, or is it something else?

In the same helloworld project, I made Linux to load the remote core WITHOUT Linux IPC / RPMsg enabled then core it self is not loading. Event I tried rebooting still the result is same.

Core Stop/start:

With Reboot:

When I tried loading from the ccs getting an error to load:

After adding the Linux A53 IPC RP Message, I can see core is loading.

We tried Helloworld with "Linux A53 IPC RP Message" enabled and added two MCSPI line with PKT DMA configuration and loaded from linux, then we don't see any "Hello World!" print.

When change from DMA to Interrupt in MCSPI then we are getting the debug logs, here DMA is causing the issue:

Did you notice that line in the terminal output "header-less resource table"? That means the resource table either does not exist in the project, or is not formatted properly.

Please try to follow the steps in the FAQ again. I have posted a reply there to help.
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1193620/faq-am62x-am64x-how-to-allow-linux-to-load-m4f-r5f-firmware-by-adding-a-resource-table/4503953#4503953 

Please try either approach 2A, or 2B. If you are unable to add a resource table as demonstrated, please attach your project in the e2e ticket for us to review.

Regards,

Nick

Hello All, 

I examined the rm_c files of both Linux and RTOS of PKTDMA and IPC source allocation.

I discovered that we allocated the same resources for both.

I need to check with the Design Team about this resource allocation, as well as discuss it with a colleague of mine.

Today, he is on leave, and I will update the final status tomorrow after discussing the issue with him regarding the resource allocation.

Regards,

S.Anil.

Hello,

Partial update: we are still double-checking on the resource allocation. We will have more information tomorrow.

Regards,

Nick

Hello venkata swamy,

I have completed the SOC initialization using Linux SDK binaries, which means that I am currently using the SD boot mode.

In addition, I have taken the hello world example from MCU+SDK and added one MCSPI along with UDMA.

Afterwards, I ran the hello world example from CCS and noticed that the UART console showed a debug error.

Have you faced the same issue? I apologize for not being a Linux expert, and loading an application on Linux may take more time for me .

Therefore, I decided to load an example from CCS.

Regards,

S.Anil.

HI Anil,

When I tried to load the helloworld project with MCSPI_DMA using CCS, I also ended up asserting in the same location.

Our use case is to load the r5 cores from the linux boot not from the CCS.

Hello Venkata,

SUMMARY

Ok, so I have some good news: the SPI DMA can work just fine on AM64x while Linux is running.

Here is the complication: I can get it working just fine on AM64x EVM that was made with the silicon from early 2022 (so GP silicon, before HS-FS). However, when I make the same devicetree changes on AM64x SK, I see the same UDMA issues you see. I am still trying to debug exactly what is going on that is preventing my device tree settings from getting applied.

DETAILS 

A developer was running the SPI example just fine on an AM64x EVM, so I tried to copy him.

Test 1: use modified mcspi_loopback_dma project with unmodified AM64x EVM devicetree file. The same UDMA initialization errors happened as in your posts above. This makes sense, because AM64x EVM specifically uses main domain SPI0 in the board devicetree file.

Test 2: set JUST SPI0 to "disabled", and delete the associated pinmux settings, in the AM64x EVM devicetree file. use modified mcspi_loopback_dma project. Everything works as expected.

NEXT STEPS 

I need to sign off of work for now. I'll try to log back on later tonight to keep debugging what could be going on. When I log back on, I will provide the modified MCU+ project and the modified devicetree files.

(summary of MCU+ project modifications: enable Linux IPC, add memory logs, move debug UART from instance 0 to instance 1)

Regards,

Nick

Hello Venkata,

Ok, I got some information from other team members. I have not had time to test on my end, but I assume this is correct:

HS-FS devices stores the kernel and device tree binaries (DTBs) in a kernel FIT image (I assume the fitImage in the filesystem /boot/ directory). There is a uboot environment variable called "boot_fit" that controls whether uboot boots using the FIT image, or the individual kernel and devicetree files as documented here in the SDK docs: https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/08_05_00_21/exports/docs/linux/Foundational_Components_Kernel_Users_Guide.html#installing-the-kernel-image-and-device-tree-binaries

boot_fit = 1 uses the FIT image (this is default)
boot_fit = 0 loads the dtb from the /boot/ directory

Files will be attached in this post. I'll discuss test procedure in the next post.

McSPI sample project 

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/0001_2D00_Initial_2D00_McSPI_2D00_Debug_2D00_project.patch

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/mcspi_5F00_loopback_5F00_dma.debug.out

Note: I expect this binary should load fine from Linux and work. However, because of the time crunch to figure out the device tree, I mostly tested it by connecting CCS after the board had booted, then manually loading the firmware into the R5F0 core and running.

Linux devicetree files 

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/0001_2D00_k3_2D00_am642_2D00_sk.dts_2D00_disable_2D00_all_2D00_main_2D00_domain_2D00_SPIs.patch

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/1007.k3_2D00_am642_2D00_sk.dtb

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/0001_2D00_k3_2D00_am642_2D00_evm.dts_2D00_disable_2D00_main_2D00_domain_2D00_spi0.patch

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/k3_2D00_am642_2D00_evm.dtb

Hi Nick,

Thanks for debugging the issue.

When you mention the test procedure, Please give us in details on which mcu sdk needs to apply these patches and where to place these dtb files and any other files need to changes along with these etc.

Test procedure 

1) Build Linux devicetree, copy devicetree to SD card. 

$ cd /path/to/ti-processor-sdk-linux-am64xx-evm-08.05.00.21/board-support/linux-5.10.153+gitAUTOINC+3eee621d16-g3eee621d16
// export the path to the build tools
$ export PATH=/path/to/ti-processor-sdk-linux-am64xx-evm-08.05.00.21/linux-devkit/sysroots/x86_64-arago-linux/usr/bin:$PATH
// use the default kernel configs
$ make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- tisdk_am64xx-evm_defconfig
// build the modified devicetree file
$ make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- ti/k3-am642-sk.dtb
// copy the devicetree file to the AM64x-SK filesystem. In case of SD card:
// attach to SD card. I placed my SD card partition at /mnt/sdb1
// rename the original devicetree file
$ ls /mnt/sdb1/boot/
fitImage                       Image                       k3-am642-evm.dtb                   k3-am642-evm-icssg1-dualemac-mii.dtbo  k3-am642-sk.dtb  vmlinux.gz-5.10.153-g3eee621d16
fitImage-5.10.153-g3eee621d16  Image-5.10.153-g3eee621d16  k3-am642-evm-icssg1-dualemac.dtbo  k3-am642-evm-nand.dtbo                 vmlinux.gz
$ sudo mv /mnt/sdb1/boot/k3-am642-sk.dtb /mnt/sdb1/boot/k3-am642-sk.dtb.orig
// copy the devicetree binary to the SD card
$ sudo cp arch/arm64/boot/dts/ti/k3-am642-sk.dtb /mnt/sdb1/boot/
$ ls /mnt/sdb1/boot/
fitImage                       Image                       k3-am642-evm.dtb                   k3-am642-evm-icssg1-dualemac-mii.dtbo  k3-am642-sk.dtb       vmlinux.gz
fitImage-5.10.153-g3eee621d16  Image-5.10.153-g3eee621d16  k3-am642-evm-icssg1-dualemac.dtbo  k3-am642-evm-nand.dtbo                 k3-am642-sk.dtb.orig  vmlinux.gz-5.10.153-g3eee621d16
$ sudo umount /mnt/sdb1

2) Build the MCU+ project in CCS or through the terminal.

The R5F binary can be copied to the SD card /lib/firmware/pdk-ipc with the same method as the Linux devicetree.

3) Update Linux to point to the desired binary:

am64xx-evm login: root
root@am64xx-evm:~# cd /lib/firmware/
root@am64xx-evm:/lib/firmware# ls pdk-ipc/
hello_world_no_ipc.out                              ipc_echo_baremetal_test_mcu1_1_release_strip.xer5f  ipc_echo_baremetal_test_mcu2_1_release_strip.xer5f  mcspi_loopback_dma_ipc_logs.out
ipc_echo_baremetal_test_mcu1_0_release_strip.xer5f  ipc_echo_baremetal_test_mcu2_0_release_strip.xer5f  ipc_echo_baremetal_test_mcu3_0_release_strip.xer5f
root@am64xx-evm:/lib/firmware# ls
LICENCE.ibt_firmware      am64x-pru0_0-fw  am64x-rtu1_1-fw        iwlwifi-8000C-22.ucode  iwlwifi-8265-31.ucode              iwlwifi-9260-th-b0-jf-b0-43.ucode  ti-connectivity
LICENCE.iwlwifi_firmware  am64x-pru0_1-fw  am64x-txpru0_0-fw      iwlwifi-8000C-27.ucode  iwlwifi-8265-34.ucode              iwlwifi-9260-th-b0-jf-b0-46.ucode  ti-pruss
am64-main-r5f0_0-fw       am64x-pru1_0-fw  am64x-txpru0_1-fw      iwlwifi-8000C-31.ucode  iwlwifi-8265-36.ucode              mcusdk-benchmark_demo
am64-main-r5f0_1-fw       am64x-pru1_1-fw  am64x-txpru1_0-fw      iwlwifi-8000C-34.ucode  iwlwifi-9260-th-b0-jf-b0-33.ucode  pdk-ipc
am64-main-r5f1_0-fw       am64x-rtu0_0-fw  am64x-txpru1_1-fw      iwlwifi-8000C-36.ucode  iwlwifi-9260-th-b0-jf-b0-34.ucode  pru
am64-main-r5f1_1-fw       am64x-rtu0_1-fw  intel                  iwlwifi-8265-22.ucode   iwlwifi-9260-th-b0-jf-b0-38.ucode  regulatory.db
am64-mcu-m4f0_0-fw        am64x-rtu1_0-fw  iwlwifi-3160-17.ucode  iwlwifi-8265-27.ucode   iwlwifi-9260-th-b0-jf-b0-41.ucode  regulatory.db.p7s
root@am64xx-evm:/lib/firmware# ln -sf /lib/firmware/pdk-ipc/mcspi_loopback_dma_ipc_logs.out am64-main-r5f0_0-fw
// you can check where am64-main-r5f0_0-fw is pointing by using $ls -al

4) Reboot to make sure that the Linux devicetree you added is actually getting used (instead of the FitImage), and to load the new binary into R5

# reboot -f

// press a button on the keyboard during uboot to pause boot and enter the uboot terminal
//show the current uboot environment variables
=> printenv
// set boot_fit to use the .dtb and kernel image in /boot directory
// note this is /boot directory in the root partition, NOT the boot partition
// the boot partition is the smaller partition with only a couple of files
=> setenv boot_fit 0
=> saveenv
//double check settings
=> printenv
// boot
=> boot

5) check from the Linux side if the core booted

am64xx-evm login: root
// note that the remoteproc number assigned to the core can change between boots
// so it is best practice to check to make sure you are looking at the right core
root@am64xx-evm:~# head /sys/class/remoteproc/remoteproc*/name
...
==> /sys/class/remoteproc/remoteproc0/name <==
78000000.r5f
...
root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc0/state
running

root@am64xx-evm:~# cat /sys/kernel/debug/remoteproc/remoteproc0/trace0
// the last time I ran this from Linux I had not yet updated UART output to UART1,
// so the firmware never finished initializing even though Linux thought it was
// running. I would expect you to see the SPI test output in this trace with
// the new attached binary

I used Linux SDK 8.5 and MCU+ SDK 8.5.

I'm going to start getting ready for bed. I'll check back in 20-30 minutes to see if you have any last-minute questions.

-Nick

If the Linux trace0 doesn't show anything, that's fine. Ignore that part of the test for now, and just load the binary into the R5 using CCS while Linux is running. If Linux did not try to initialize McSPI0, then the CCS loaded binary should work. If Linux did initialize McSPI0, then CCS will show the same error you have been seeing.

I have tried to download the sdk mentioned in the test step.

But due to very bad internet in office I was not able to download the SDK.

I have tried the RT-Linux SDK which was present already in my machine. With that I am getting errors while building it

Another partial update:

That boot_fit option gets overwritten by something every time the board boots. So for now, if we want to use separate Linux and device tree files, you need to manually do setenv boot_fit 0 every boot cycle (or modify the uEnv.txt file, which I have not had the chance to look into yet).

This is frustrating. Thank you for your patience. I am working with the team in the background to better document what the HS-FS processor default behavior is, and driving a discussion about whether this is the right default behavior, or if we want to change it in future releases.

the RT linux configs are named slightly different. 

try $ ls arch/arm64/configs/

In my RT Linux SDK, I see tisdk_am64xx-evm-rt_defconfig.

Thanks Nick for the details.

I have downloaded the SDK ti-processor-sdk-linux-am64xx-evm-08.05.00.21 and created the device tree file "k3-am642-sk.dtb" and copied in /boot folder in sd card as mentioned in Test procedure above "Build Linux devicetree, copy devicetree to SD card."

Set the boot_fit = 0 in boot prompt as well as below command in uEnv.txt

uenvcmd=setenv boot_fit 0; else run do_psdk_setup; mw.b ${loadaddr} 0 1; fatwrite mmc 1:1 ${loadaddr} .psdk_setup 1; reset; fi

Loaded "mcspi_loopback_dma" in ccs and compiled and loaded the
"mcspi_loopback_dma_am64x-sk_r5fss0-0_freertos_ti-arm-clang.out" out on the target.

I have loaded the "mcspi_loopback_dma.debug.out" which is attached above and started and with that also getting the assert.

Can you please provide the final test steps. As you mentioned this thread becomes clumsy and cannot point out the exact working steps.

Can we have call to execute the final test steps on my setup.

test dtb file:

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/8473.k3_2D00_am642_2D00_sk.dtb

=>
U-Boot SPL 2021.01-g3983bffabc (Dec 14 2022 - 11:58:15 +0000)
EEPROM not available at 80, trying to read at 81
Resetting on cold boot to workaround ErrataID:i2331
resetting ...

U-Boot SPL 2021.01-g3983bffabc (Dec 14 2022 - 11:58:15 +0000)
EEPROM not available at 80, trying to read at 81
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.5.3--v08.05.03 (Chill Capybar')
SPL initial stack usage: 13424 bytes
Trying to boot from MMC2
mmc fail to send stop cmd
Error reading cluster
** Unable to read file tispl.bin **
Authentication passed
Authentication passed
Authentication passed
Authentication passed
Starting ATF on ARM64 core...

NOTICE: BL31: v2.7(release):v2.7.0-359-g1309c6c805-dirty
NOTICE: BL31: Built : 11:49:05, Dec 14 2022
I/TC:
I/TC: OP-TEE version: 3.19.0-15-gd6c5d0037 (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10))) #1 Wed Dec 14 11:53:26 UTC 2022 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check optee.readthedocs.io/.../porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.5.3--v08.05.03 (Chill Capybar')
I/TC: HUK Initialized
I/TC: Activated SA2UL device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.01-g3983bffabc (Dec 14 2022 - 11:53:57 +0000)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.5.3--v08.05.03 (Chill Capybar')
Trying to boot from MMC2
Authentication passed
Authentication passed

U-Boot 2021.01-g3983bffabc (Dec 14 2022 - 11:53:57 +0000)

SoC: AM64X SR2.0 HS-FS
Model: Texas Instruments AM642 SK
Board: AM64B-SKEVM rev A
DRAM: 2 GiB
NAND: 0 MiB
MMC: mmc@fa00000: 1
Loading Environment from FAT... OK
In: serial@2800000
Out: serial@2800000
Err: serial@2800000
Net: eth0: ethernet@8000000port@1
Warning: ethernet@8000000port@2 MAC addresses don't match:
Address in ROM is 34:08:e1:80:ad:1d
Address in environment is 70:ff:76:1e:bd:70
, eth1: ethernet@8000000port@2
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc1 is current device
SD/MMC found on device 1
Failed to load 'boot.scr'
1671 bytes read in 1 ms (1.6 MiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
Running uenvcmd ...
1 bytes read in 1 ms (1000 Bytes/s)
Already setup.
mmc fail to send stop cmd
** fs_devread read error - block
Failed to load '/boot/Image'
** No partition table - mmc 1 **
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
No FDT memory address configured. Please configure
the FDT address via "fdt addr <address>" command.
Aborting!
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
=>

Hello Venkata,

Ok, here are the things to test before our call tomorrow. Basically, I am trying to get your setup to boot with the modified Linux dtb file. Once that is working, then you can start trying to load your test remote core firmware.

STEP 1: return uEnv.txt to default settings

The SD card should boot fine out-of-the-box. So let's undo all the changes that might be having an impact.

Please delete uEnv.txt from the SD card's boot partition. Then you can copy the uEnv.txt file in the Linux SDK board-support/prebuilt-images/uEnv.txt.

Now try manually setting boot_fit to zero in the uboot terminal. (remember that you will need to stop the boot and manually do this change each time you boot). Are you able to successfully load into Linux?

setenv boot_fit 0

STEP 2: If step 1 doesn't work, then try creating a fresh SD card image 

If you are using an unmodified AM64x SK, and a fresh SD card image, then there should not be any differences between your setup and my setup (assuming your board is set to boot from SD card instead of some other boot source).

Create a new SD card with the Linux SDK 8.5 image: https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/08_05_00_21/exports/docs/linux/Overview/Processor_SDK_Linux_create_SD_card.html

2a) test that the default boot with the fitImage works

2b) test that booting with the kernel image and devicetree file on the SD card works by manually setting boot_fit to zero in the uboot terminal. Does it still boot?

2c) now copy over your Linux dtb file, and manually set boot_fit to zero again. Does it still boot?

STEP 3: figure out how to build the FitImage. This is a bunch of extra steps, so my team members who are more familiar with U-Boot do NOT suggest doing this unless we cannot get steps 1 and 2 working. 

https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/08_05_00_21/exports/docs/linux/Foundational_Components_Kernel_Users_Guide.html#id14 

Regards,

Nick

STEP 1: return uEnv.txt to default settings

The SD card should boot fine out-of-the-box. So let's undo all the changes that might be having an impact.

Please delete uEnv.txt from the SD card's boot partition. Then you can copy the uEnv.txt file in the Linux SDK board-support/prebuilt-images/uEnv.txt.

Now try manually setting boot_fit to zero in the uboot terminal. (remember that you will need to stop the boot and manually do this change each time you boot). Are you able to successfully load into Linux?

setenv boot_fit 0

[Venkat]
I have deleted the uEnv.txt from the SD card's boot partition. Then copied the uEnv.txt file from the Linux SDK board-support/prebuilt-images/uEnv.txt

I have original k3-am642-sk.dtb file under boot folder.

During the boot prompt set the boot_fit to 0 and saved the env value and the booted with command boot.
I see the same yesterday's error "Could not find a valid device tree".

*************************************************************************************
STEP 2: If step 1 doesn't work, then try creating a fresh SD card image

If you are using an unmodified AM64x SK, and a fresh SD card image, then there should not be any differences between your setup and my setup (assuming your board is set to boot from SD card instead of some other boot source).
[Venkat] My setup is configured for boot from sdcard.

Create a new SD card with the Linux SDK 8.5 image: software-dl.ti.com/.../Processor_SDK_Linux_create_SD_card.html
[Venkat] I have used the SKAM64x Linux SDK image.

2a) test that the default boot with the fitImage works
[Venkat] Yes , with the fresh SD card image, boot from fitImage worked.
2b) test that booting with the kernel image and devicetree file on the SD card works by manually setting boot_fit to zero in the uboot terminal. Does it still boot?
[Venkat] After setting the boot_fit to 0, devicetree didn't loaded. Getting the same error "Could not find a valid device tree".

2c) now copy over your Linux dtb file, and manually set boot_fit to zero again. Does it still boot?
[Venkat] Getting the same error "Could not find a valid device tree".

*************************************************************************************
STEP 3: figure out how to build the FitImage. This is a bunch of extra steps, so my team members who are more familiar with U-Boot do NOT suggest doing this unless we cannot get steps 1 and 2 working.
[Venkat] I am trying to build the FitImage as per steps mentioned in the link. Build is in progress.

Based on our call today, it looks like an SD card issue.

Regards,

Nick

I have modified the uEnt.txt file to boot always from the Devicetree. I have tested around 20 times and around 7 times it is failing with the same error "Could not find a valid device tree". If it is the sdcard problem it should fail always. failing 30 percent of time is not expected. Can we have call to check this issue.

I have tested the Rpmsg echo linux(RPmsg+MCSPI-DMA) with the new devicetree binary and it was loading and logs were coming. But when we load our  project with RPMsg+MCSPI-DMA , rpmsg channel is not created during the boot up. I will check this tomorrow again.

Meanwhile Can you share your "mcspi_loopback_dma.debug" project with us. We want to compare the mcspi configuration with our project.

Hello Venkat,

1) Can you please get some SanDisk Ultra 16GB SD cards to test? I am doing some followup with the team, but it looks like we have had issues with certain SD card brands with a previous version of the AM64-SK board. I am not sure yet if those issues ever got resolved, but it looks like that brand of SD card consistently worked. That is also the brand of SD card I am using in my test setup.

2) I did not add RPMsg code to the .c files of the modified mcspi_loopback_dma project. You will need to add it in manually. The diff for the source code is in this post: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1185872/am6442-r5-spi-dma-troubleshooting/4514574#4514574

Regards,

Nick