Koichiro Tashiro said:

TCP delay is observed

Is there any more information on what this is and how this is measured?

Hi Pekka,

Sorry for my late reply.
I got detailed information from the customer.
I will send them to you offline.

Thanks and regards,
Koichiro Tashiro

Short term suggestion is to try out on 9.0 SDK where there is several improvements and updates related to realtime performance.

Networking stack, specifically TCP is a shared interface, there is no prioritization of the intended to be realtime TCP traffic and what the TCP traffic the web browser creates. There have been many changes in Linux kernel and networking stack since 2015, the kernel in SDK 3.0 is I believe something like 3.x kernel, 9.0 is 6.1.

Is there any other traffic than the TCP test program in the Ethernet? Or is the web browser sharing that wire?

Sorry for late response. I got some feedback from the customer. I will put it here soon.

AM62x with background load on 9.0 SDK:

root@am62xx-evm:~# stress-ng -c4 --cpu-method=all &
root@am62xx-evm:~# cyclictest -m -Sp98 -i200 -h800 -l100000000 -q > output


# Min Latencies: 00005 00005 00005 00005
# Avg Latencies: 00007 00006 00007 00007
# Max Latencies: 00083 00102 00095 00103

Hi Pekka,

Please find below feedback from the customer.

Pekka Varis said:

Short term suggestion is to try out on 9.0 SDK where there is several improvements and updates related to realtime performance.

They tried SDK 9.0, but results were the same.

Pekka Varis said:

Is there any other traffic than the TCP test program in the Ethernet?

There is no other traffic than the TCP test program. DHCP and IPv6 are disabled, so no other packets are there.
The customer tried to check what kinds of interrupt are triggered while web browser screen update scenario.
"irqtop" command was used. Below IRQ were generated (sorted by "DELTA" descending order).
 

IRQ    TOTAL        DELTA    NAME
11      70091      367         GICv3 30 Level arch_timer
IPI0    54974      105         Rescheduling interrupts
448     14156       60          GICv3 116 Level tidss
IPI1    8309        33          Function call interrupts
449     2664       13          GICv3 118 Level pvrsrvkm
26       3611       12          GICv3 193 Level 20000000.i2c
458      1142         9          GICv3 220 Level xhci-hcd:usb2

CPU0       CPU1       CPU2       CPU3
11:      46374      55637      86196      19519     GICv3  30 Level     arch_timer
IPI0:     26716      48520      14792      26313       Rescheduling interrupts
IPI1:     19973      13713        400        879       Function call interrupts

echo 7 > /proc/irq/11/smp_affinity
-sh: echo: write error: Input/output error

Tashiro-san

Pekka is out of office till middle of next week, responses will be delayed 

I'd suggest to use the optargs="isolcpus=3" approach as show in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1172055/faq-am625-how-to-measure-interrupt-latency-on-multicore-sitara-devices-using-cyclictest/ to tell the kernel not to have a CPU part of the SMP scheduling.

Tashiro-san,

Did you try optargs="isolcpus=3"? what result did you get? 

I got only 3 CPU result after apply isolcpus on SDK9.

Hi,

If you are using SDK 9 you will see only non-isolated  CPU in results and not all CPUs... Proposal and tests from Pekka link are applicable in case of SDK 8.x

Difference is coming from cyclictest version... Older version of cyclictest in SDK 8.x will trigger threads to all cores (4 threads for 4 cores). In SDK 9 cyclictest is different and will not start thread on CPU that is isolated (optargs="isolcpus=3" -> no thread on CPU3)

This is why you have different result in compare with test proposed by Pekka

We have same issue like you... 

Hi,

The customer will check with SDK 9.x.

Thanks and regards,
Koichiro Tashiro

Just to add additional observation on SDK 9.x

Am64x optargs="isolcpus=1" 

Test 1:

stress-ng --memrate 1 --memrate-rd-mbs 70 --memrate-wr-mbs 140 --taskset 0 &

taskset 1 cyclictest -l100000000 -m -Sp90 -i200 -h400 -q > cpu0.txt &

taskset 2 cyclictest -l100000000 -m -Sp90 -i200 -h400 -q > cpu1.txt &

CPU1: Max Latencies: 00081

Test 2:

stress-ng --memrate 1 --memrate-rd-mbs 70 --memrate-wr-mbs 140 --taskset 0 &

taskset 2 cyclictest -l100000000 -m -Sp90 -i200 -h400 -q > cpu1.txt &

CPU1:  Max Latencies: 00191

NOTE: taskset 2 cyclictest -l100000000 -m -Sp90 -i200 -h400 -q  - This command with cyclictest will start only one thread on CPU1 only

Noting that the above from Milan is with AM64x (2 core A53), while the original poster is running AM62x (4 core A53).

One other note is on core affinity. cyclictest run with taskset N does not always result in what you'd expect. Using -t 1 -a 2 option (and no SMP option  -S) looks to me to be more consistently placing the worker thread of cyclictest on the core you specify with -a N. There is always one housekeeping time slice thread as well. Similarly as for stress-ng I (and Milan above) also use --taskset N option as a parameter instead of trying to pin it from the outside with taskset N stress-ng. Possibly won't change the results but seems to give more guaranteed control over what core the worker threads are running. Stress-ng at root privilege can do things you might not expect.

  Pekka

Hi Pekka,

Could you try on SDK 9.x option -t 1 -a 2 where this CPU is ioslated... On my AM64x boad with two cores this is output

root@am64xx-evm:~# cat /sys/devices/system/cpu/isolated
1

root@am64xx-evm:~# cyclictest -l100000000 -m -a 1 -t 1 -i200 -h400
WARN: Couldn't setaffinity in main thread: Invalid argument
FATAL: No allowable cpus to run on

If CPU is isolated, with SDK 9.x where you have different cyclictest you must use taskset to assign to  CPU1... -t 1 -a 1 is not working on my side

AM62x with four cores or AM64x with two cores, I am pretty sude behavior is same

Milan,

Sorry this took a while. Cyclictest has several fairly recent commits on affinity. There is the housekeeping thread, for which now the affinity can be set separately with --mainaffinity, and the actual measurement thread. Looks like the default is to use the same core. The error you are is from that setting of the housekeeper thread affinity. My point being this seems like a cyclictest issue, although I have not tried comparing --mainaffinity , taskset, and -a options exhaustively.

Maybe more important we have identified some major improvement options to the worst case latency and plan to have these included in the 9.1 release planned for December 15th on ti.com.

  Pekka

Thanks for info... let see in SDK 9.1... 

In pararlell, we will try to investigate  cpusets instead of isolcpus as this is deprecated

Milan Stevanovic said:

cpusets instead of isolcpus as this is deprecated

Good point, this explains why I had some trouble replicating the exact test. Need to move to using CPUSETS, https://docs.kernel.org/admin-guide/cgroup-v1/cpusets.html .

I'll re-close this thread as for the original topic. If there still persists a cyclictest and core affinity issue with 9.1 SDK (6.1 kernel) we can dig into that in a new thread.