AM625: Linux cyclictest results

Koichiro Tashiro

Part Number: AM625

Hi,

My customer evaluates AM625 performance using TI EVM and SDK.
They used AM437x before and compared these results between AM437x and AM625x.

SDK version:
v08.06.00.42 for AM62x
v03.03.00.04 for AM437x

How to test (commands used for the test):
stress-ng -l80 -c1
cyclictest -p 50 -l100000 -S -h2000

Results:
AM437x
# Min Latencies: 00005
# Avg Latencies: 00021
# Max Latencies: 00366

AM62x
# Min Latencies: 00005 00005 00005 00005
# Avg Latencies: 00009 00012 00008 00009
# Max Latencies: 00100 00665 00153 00104

As you can see, the average latencies are shorter on AM62x, but the max latencies are longer on AM62x.
(366us on AM437x vs. 665us on AM62x)
The customer wants to know the reason and workaround to avoid these long latencies.

Background of the question:
The customer observed TCP communication delay under heavily loaded condition.
From the kernel trace, they noticed a kernel process (irq/224-80000000) and some user process used in communication are sometimes launched in delayed fashion.

Thanks and regards,
Koichiro Tashiro

over 3 years ago

+1 Pekka Varis over 3 years ago

TI__Mastermind 27050 points

The average latency improves significantly (4x better as you show) and worst case latency stays somewhat similar is an expect result. With core isolation, meaning dedicating one or two cores to the real-time tasks one can get further benefits. In your example the worst core is 665us, while the other three are 100-153us, without really doing any core affinity or cpu isolation. Short term (SDK 8.6 and older) we do have an issue we are debugging with stress-ng memory intensive tests running on the same core as the real-time task, but the normal steps real-time with core isolation we do see good worst case latency see for example https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1213907/am6442-memcpy-impact-real-time-performance/ .

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1172055/faq-am625-how-to-measure-interrupt-latency-on-multicore-sitara-devices-using-cyclictest has an example of using this approach. Further tuning example at https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1183526/faq-linux-how-do-i-test-the-real-time-performance-of-an-am3x-am4x-am6x-soc/ ,

Specifically for networking I also suggest:

ps aux | grep ksoftirq

#see the PID of the ksoftirqs, most likely 10 and 21 on AM64, AM62 will have 4 one per core
chrt -f -p 10 10
chrt -f -p 10 21

This moves the interrupt service threads more details in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1163896/tmds64gpevm-1000mbit-s-ethernet-is-not-working/4409900#4409900

Here are a couple good generic pages on steps to take on multicore real-time:

https://www.linutronix.de/blog/A-Checklist-for-Real-Time-Applications-in-Linux

https://content.helpme-codesys.com/en/CODESYS%20Control/_rtsl_performance_optimization.html

Pekka

0 Koichiro Tashiro over 3 years ago in reply to Pekka Varis

TI__Mastermind 22945 points

Hi Pekka,

Thanks for your reply.
The customer is checking your suggestions.
If they have further questions, I will let you know.

Thanks and regards,
Koichiro Tashiro

Processors

Processors forum

AM625: Linux cyclictest results