AM6422: XDP-Zero Copy on AM64

Hello Alex,

What version of software is being run?

Regards,

Nick

Hi Alex, Customer Team, 

I ran a test this afternoon with the following setup 

1. AM62Px - 4 slaves setup to control 8 axes

2. Moved my EtherCAT RX and TX IRQs to CPU3 (same CPU core as EtherCAT_Task)

3. Elevated my EtherCAT RX and TX IRQs to priority 57 to be higher than EtherCAT_Task (EtherCAT_Task priority was at 56) 

4. NOTE: DID NOT YET try the "WaitForPacket" parameter since I wanted to start simple. I may test this next

5. Captured kernel trace with this setup (trace-customer-090326) and compared with my previous configuration with RX and TX IRQs on CPU2 and at same priority as EtherCAT_Task (trace-090226). Please below zip files for the traces. 

trace-customer-090326.zip

trace-090226.zip

trace-customer-090826.zip

Below is list of my observations and theories to why might be occurring regarding the larger than expected ksoftirqd execution times:

From these results and comparison I think that the real solution to eliminating the irq handler events from ksoftirqd is to have a dedicated core to handle RX and TX IRQ processing of the EtherCAT port.

I don't yet have the solution to EtherCAT_Task being interrupted will need to look into it further (could simply be because IRQs higher in priority than EtherCAT_Task).

-Daolin

Update:

I realized that some cycles from the trace-customer-090326.dat had larger ksoftirq execution time so I have corrected the execution time in the table above. With the correction, the execution time is ~100us closer to what was observed on trace-260625-xdp_zero-copy-rx-delay-1325.dat. 

I've also added to the above table a test I did today with only setting IRQs to same CPU core as EtherCAT_Task but keeping the priority the same as EtherCAT_Task. It looks like its not increasing the IRQ priority that caused ksoftirq execution time but that the IRQs are on the same core as EtherCAT_Task.

I'm working on testing with WaitForPacket parameter next. On initial test, I see that the RxErrorCount statistic from the IDE increment rapidly. Is this something you have also noticed on your setup when testing with the WaitForPacket parameter?

-Daolin

Hello,

In addition to the tests we discussed in the call for customer to try, can customer also try moving the Ethernet TX and RX IRQ to CPU0 instead of CPU1?

From what I found about the "WaitForPacket" parameter, this parameter requires the EtherCAT frame to be received on the same cycle as when it was transmitted as opposed to being received on the next cycle. If there are ksoftirq events like irq_handler_entry events showing up, this will significantly increase ksoftirq execution time therefore also increasing the time EtherCAT_Task is interrupted. The goal is to move this to another CPU core that wouldn't have to share resources with EtherCAT_Task core. Since only CPU0 is available as the other core, you should give that a try as a sanity check. 

I will try to investigate if the TX and RX IRQs can be handled by R5 core if the main application program (EtherCAT_Task is on A53 core), at this moment in time, I'm not sure if it is possible.

-Daolin

We have managed to capture initial traces on the CPSW (using a mix: one interface via CPSW for EtherCAT and one via PRU-ETH for everything else). However, this was done without core isolation, pinning and interrupt prioritization — simply to see if the "ksoftirqd effect" is fundamentally present there as well. Surprisingly, neither ksoftirqd nor the interrupts (!) appeared in the traces in this instance — neither for the CPSW nor the PRU-ETH.
We therefore intend to repeat the tests: This time with core isolation, pinning and interrupt prioritization, to confirm our observations under the same conditions as before. This is currently being prepared.

We will also then check for pinning Ethernet TX and RX IRQs to the other core.

Hi Florian, 

Thanks for the update on your next steps. 

Daolin Qiu said:

I will try to investigate if the TX and RX IRQs can be handled by R5 core if the main application program (EtherCAT_Task is on A53 core), at this moment in time, I'm not sure if it is possible.

One question I have for you regarding this, do you have a specific requirement to only use A53 cores for EtherCAT or are receptive to mapping IRQs to R5 cores if it is possible?

-Daolin

Hi Daolin,

If that is possible, we could consider doing that

To my knowledge up to now we're not using R5 at all. But if we wanted to do this in future, it would be interesting to know, if that then could co-exist with the IRQ handling (if possible at all)?

The user logic executed as part of the EtherCAT cycle, at the moment definitely needs to be executed on A53, since CODESYS has no compiler backend for R5 and in general - to my knowledge - no concept for this kind of asymmetric multicore architecture.

I'll share the traces as soon as we repeated the test.

-Florian

Hi Florian, 

Apologies for the delay in response. 

Florian Miedniak said:

To my knowledge up to now we're not using R5 at all. But if we wanted to do this in future, it would be interesting to know, if that then could co-exist with the IRQ handling (if possible at all)?

Unfortunately after discussing internally about using the R5 core to handle IRQ processing, we don't think using the R5 cores while using A53 for CODESYS will be a viable solution. The reason is because our only current example of tunneling between the A53 and R5 will have too much latency, my understanding is >3ms which is definitely not fast enough to meet EtherCAT cycle times. 

If I may ask, is there a hard requirement to use CODESYS for your application? Since you are trying to implement an EtherCAT use case on AM64x which has 4 R5 but only 2 A53, why not use an EtherCAT stack capable of running on R5 cores?

Additionally, this week a colleague of mine was able to recreate the behavior you observed on an AM64x EVM with ICSSG port (the IDE port being CPSW) with XDP ZC and the WaitForPacket enabled. We also tried moving the TX and RX IRQs from CPU1 to CPU0 and found some potential improvements in EtherCAT_Task execution time compared to if the IRQs remained on CPU1. Unfortunately, there doesn't appear to be a solution to completely remove the EtherCAT_Task from being interrupted but I think that is to be expected with WaitForPacket, based on what I understand how WaitForPacket is supposed to work. 

My colleague will share the results of our capture shortly on this thread.

-Daolin

Hi, 

Here are the results from our experiments with moving the IRQs from CPU1 to CPU0

core_isolation.zip
Preface:

• CPU1 isolated with only EtherCAT_Task and select IRQs
• CPU0 contains the other CODESYS child tasks and OS housekeeping tasks
• XDP_ZeroCopy and WaitForPacket enabled for all tests

Like Daolin said above, with WaitForPacket enabled EtherCAT_Task is practically guaranteed to be interrupted by the IRQs which we believe is due to how WaitForPacket is designed to work. The best we saw to reduce the time interrupted was with the CPU isolation configurations above, but even then we are limited by only having two A53 cores on the AM64x. There is simply not enough processing power available to run the CODESYS child tasks and EtherCAT_Task without some sort of interruptions or delayed. As Daolin said above it may be beneficial to look into an R5 compatible EtherCAT stack so that you can utilize the R5 cores on the AM64x.

-Rohit 

Hi Daolin,

never mind, as we're also a bit behind our schedule to (re-)test with CPSW :-/ First tests both with PRU-ETH and CPSW doesn't seem that bad, we'll expect to have detailed measurements by this week.

We already suspected that the inter-processor communication between A53 and R5 could be the next/worse bottleneck when moving the IRQ processing to the R5's, but anyway: Good to have this confirmed, so we can definitely rule that out.

As to your question, if we already thought about an EtherCAT stack running on R5 cores: Yes, we definitely did. But due to the fact that the user application part of the EtherCAT processing today is user-programmable in CODESYS and they aren't able to generate code for R5, we'd always would need some inter-processor communication between A53 and R5 which is performant enough in terms of latency to satisfy the EtherCAT cycle time requirements. Anyway, if there is anything we might have overseen there, we'd be happy to hear about.

Thank you very much for the continued, detailed support on this!

-Florian