# AM62L RT Linux Performance — Generic Optimization Summary **Prepared for:** customer (CodeSys / EtherCAT real-time use case) **Platform:** AM62L3, AM62LX SR1.1 HS-FS, dual Cortex-A53, kernel 6.12.57-ti-rt (PREEMPT_RT), SDK 11.02.08.02 **Scope:** Generic RT Linux tuning — general best practices applicable to any workload, to establish the best-case latency the hardware can deliver before any CodeSys-specific debug. --- ## Bottom line Four levers determine worst-case latency on this platform. In priority order: 1. **Enable DDR QoS** (A53 read priority over write). Largest single improvement; mandatory when both cores drive memory traffic. 2. **Keep core isolation ON** for the production use case. Isolation is what prevents core0 activity from perturbing the core1 RT task once both cores are loaded. 3. **Avoid the OP-TEE hardware TRNG path** (use the software PRNG build). Removes a core0 latency-outlier source. Not a factor if OP-TEE is never invoked. 4. **Filesystem size does not matter** for latency. Ship the smallest image for other reasons, not for jitter. Best measured result with all generic settings applied: **core0 max 48 µs, core1 max 123 µs** under heavy dual-core stress. Generic tuning has reached its floor; further gains require workload-specific outlier analysis. Important measurement caveat, confirmed repeatedly across this campaign: **there is no 1:1 mapping between cyclictest numbers and CodeSys performance.** Use cyclictest to compare configurations, but validate final settings against actual CodeSys behavior. --- ## Test campaign overview - **56 cyclictest runs** (57 histograms), each 2–48 hours, spanning 2026-05-18 to 2026-06-22, plus 4 system-state diagnostic captures. - Two stress/measurement profiles were used, and they respond **differently** to the same settings, so they are reported separately: | Profile | Stress | cyclictest | |---|---|---| | **SDK test** | `stress-ng --cpu-method=all -c 4` (all cores) | `-Sp80 -i200 -M -q` | | **Customer test** (core1 only) | `taskset -c 1 stress-ng --cpu 1 --cpu-load 70 --vm 1 --vm-bytes 80%` | `-a 0-1 -t 2 -p99 -i1000 -q` | | **Customer test 2** (both cores) | above **plus** `taskset -c 0 stress-ng --cpu 1 --cpu-load 50` | same | - **Customer test 2** was added specifically because the real deployment loads both cores: core0 runs CANopen and Ethernet CODESYS tasks (0–50% dynamic load), core1 runs the EtherCAT master. The single-core-stress test understates the worst case the production system will see. - The metric of interest throughout is **core1 maximum latency** (the RT core). --- ## Findings by lever ### 1. DDR QoS — largest impact Applied at two hardware arbitration stages before running the workload: ```bash # Stage 1 — CBASS: raise A53 READ port priority (EPRIORITY 7 -> 6) devmem2 0x45D20500 w 0x00006000 # Stage 2 — DDRSS DEF_PRI_MAP: map VBUSM priority 7 -> DDR AXI priority 1 devmem2 0x0F300030 w 0x00000001 ``` **Measured:** improved both cores' max latency in 8/8 customer-test pairs (mean core0 max −134 µs, core1 max −39 µs; up to 1.7–2.4× lower). No effect on the SDK test, whose stress pattern does not load the DDR path the same way. **Mechanism:** an RT task waking with a cold cache issues cache-fill reads; by default reads and writes arbitrate at equal priority inside the DDR controller, so the RT task stalls behind background write traffic until its data arrives. Prioritizing reads removes that stall. (On AM64x with a different set of stressors, the same change reduced worst-case latency from 800+ µs to ~170 µs.) ### 2. Core isolation — required for the production (dual-core) case The naive single-core-stress result is misleading, and understanding why is the key insight of this campaign. **Observation A — with core0 idle, isolation looked *worse* for core1:** In the single-stressor Customer test (core0 unloaded), non-isolated core1 max was 49–78 µs versus 113–266 µs isolated. **Observation B — the reason is cross-core load migration:** Comparing isolated vs non-isolated single-stressor runs (8 matched pairs), core0 average latency rose when isolation was OFF (mean 7.1 → 11.4 µs, 7/8 pairs), while core1 was simultaneously relieved (avg 23.0 → 12.6 µs; max 156 → 67 µs). *Interpretation:* with no isolation, the Linux scheduler spreads the core1 workload onto the idle core0 — which is only possible because core0 was free. **Observation C — that free core does not exist in production.** When core0 is also loaded (Customer test 2), the spillover path is gone, and the same scheduler freedom that let core1 offload onto core0 would let core0 work migrate toward core1 and add latency to the RT task. **Conclusion:** isolation did not "hurt" — it removed a spillover shortcut that a real dual-loaded system cannot use anyway. Because core0 activity must not perturb core1 execution, **core isolation is the mechanism that enforces that guarantee.** Recommend keeping isolation ON and validating with CodeSys. ### 3. OP-TEE — TRNG vs PRNG **Measured:** rebuilding OP-TEE with the software PRNG (instead of the hardware TRNG) cut core0 max in 8/8 customer pairs (mean −163 µs); little effect on core1. **Mechanism:** each hardware-RNG call forces a Linux→OP-TEE secure-monitor context switch that lengthens interrupt response. The PRNG build lets Linux serve RNG itself and avoids the switch. This is **not** the same as disabling OP-TEE — if the application invokes OP-TEE regularly, the latency returns. **Applicability to the customer:** likely a non-factor, since OP-TEE is not loaded in the product. Relevant only if OP-TEE is later used. **Reproducing TI's published benchmark:** TI's RT Performance Guide numbers were obtained with u-boot rebuilt against a PRNG OP-TEE. Building OP-TEE with `CFG_WITH_SOFTWARE_PRNG=y PLATFORM=k3-am62lx` and rebuilding u-boot is required to match them. ### 4. Filesystem (full Arago vs minimal base image) **Measured:** no latency effect above run-to-run variability. Base-FS median core1 max was ~13 µs lower, within noise, and signs disagreed across conditions. A smaller filesystem is safe (never worse) but is not a latency lever. ### 5. Kernel and userspace The best-case run applied a `disable_kernel_userspace.sh` script before starting cyclictest. What that script actually did: - **Stopped userspace services** (and their `.socket` units): avahi-daemon, containerd, lldpd, netserver, rpcbind, systemd-networkd, systemd-resolved, systemd-timesyncd, systemd-udevd, systemd-userdbd, getty@tty1, systemd-journald. containerd was the heaviest of these (Go runtime, ~1.75 GiB VSZ) and had not been isolated from the earlier tests. - **Disabled KSM at runtime:** `echo 0 > /sys/kernel/mm/ksm/run`. ksmd was confirmed running in the earlier test builds and scans pages under the `--vm-bytes 80%` memory pressure. - **Attempted to disable transparent hugepages:** `echo never > /sys/kernel/mm/transparent_hugepage/enabled`. This command failed in the run — the path did not exist ("No such file or directory") — so THP was not changed by the script. [INFERENCE] The absent sysfs path is consistent with THP already being compiled out of this base-FS kernel, but that was not separately confirmed. These changes were applied together as one configuration; they were not isolated as individual A/B latency measurements, so no per-item latency delta was captured. **Devicetree observation:** with the full EVM devicetree, four of the eight Ethernet TX-queue IRQs (ethernet-tx1/3/5/7) had their affinity set to the isolated core1 despite `irqaffinity=0` — the odd-numbered queues map to CPU1, the even ones to CPU0. This is from the `smp_affinity_list` capture in the 260619 default-DTS dump; interrupt counts were zero during that idle capture, so it is an affinity assignment, not observed traffic on core1. The minimal devicetree does not expose these IRQs at all. Any full-devicetree deployment would need these queues re-pinned to core0 before relying on isolation. ### 6. cyclictest parameters (measurement, not system performance) - `-i200` reads lower than `-i1000` (more frequent wakeups keep caches warm) — a measurement artifact, not a real system improvement. - `-M` is display-only and inert with `-q`; observed differences traced to board-to-board variability. - Fix cyclictest parameters before comparing configurations. ### 7. JIT (enabled vs disabled) No consistent effect; differences were within run-to-run noise. --- ## Best-case result Full generic configuration (base FS, minimal devicetree, DDR QoS, OP-TEE PRNG, KSM off, non-essential services stopped, core isolation on), under Customer-test-2 dual-core stress: - **Core0: min 5 / avg 7 / max 48 µs** - **Core1: min 8 / avg 24 / max 123 µs** These numbers are from a **2-hour** run. Additional 6-hour runs with the exact same configuration (the output was not preserved) showed no statistical difference in the worst-case latency for both core0 & core1 between this fully-optimized configuration and a plain minimal-image build with TI's RT configs. Generic optimization has reached its practical floor on this hardware. --- ## Lessons learned 1. **DDR QoS is the highest-value change** and becomes essential — not optional — once both cores drive memory traffic. Without it, adding a core0 load raised core1 worst-case by 64–145 µs; with it, the same addition cost only 10–15 µs. 2. **Test the real load condition.** The single-core-stress test made a non-isolated system look best, but that advantage came from core1 work spilling onto an idle core0 - a shortcut the dual-loaded production system cannot use. Benchmarks that do not model both cores being busy will point to the wrong configuration. 3. **Core isolation is a determinism guarantee, not a latency knob.** Its value is preventing core0 from stealing core1 time under load — exactly the production requirement — not lowering the number in an artificial idle-core0 test. 4. **cyclictest is a comparison tool, not a target.** No 1:1 relationship to CodeSys performance was established. Validate final settings against CodeSys. 5. **Match the reference setup before tuning.** TI's benchmark numbers required a PRNG OP-TEE rebuild; without it the out-of-box image cannot reproduce the documented ~69 µs. --- ## Recommended configuration for the CodeSys deployment The customer's kernel config is fundamentally sound. A symbol-level comparison against TI's RT config (both 6.12.57, same toolchain) shows it already inherits every big-ticket RT latency setting: PREEMPT_RT on; KPTI, KASLR, kernel-stack randomization, and RODATA_FULL off; ARMv8.1 LSE atomics off; IRQ time accounting off; all cgroup controllers off; CPUIdle, CPUFreq, hotplug, and suspend off; FTRACE and lockdep off. The customer also correctly disabled COMPACTION, MIGRATION, and SWAP (all good for RT) and stripped unused subsystems. Keep all of that. The items below are narrow deltas. Runtime and platform settings (measured in this campaign): 1. Apply the DDR QoS register writes at startup, before CodeSys. 2. Keep core1 isolated (`isolcpus`), core0 for all other tasks; re-pin the Ethernet TX-queue IRQs to core0 if using the full devicetree. 3. Disable KSM (`CONFIG_KSM=n`, or `echo 0 > /sys/kernel/mm/ksm/run` at runtime). Transparent hugepages appear already compiled out on the tested image - the runtime sysfs path was absent - so verify rather than assume it needs disabling on the production kernel. 4. Stop non-essential userspace services. 5. If OP-TEE is used, build with the software PRNG. Kernel-config deltas vs TI's RT config (from a config review, not individually latency-tested in this campaign - validate before committing): 6. Set `CONFIG_CONTEXT_TRACKING_USER_FORCE=n` (highest confidence). It is a test-only option that adds accounting to every syscall, exception, and IRQ entry/exit, and buys nothing when a periodic tick is running. 7. Decide the tick mode deliberately. The customer currently ships `HZ_PERIODIC`; TI's tested kernel used `NO_HZ_FULL` with the matching `nohz_full=` / `rcu_nocbs=` / `isolcpus=` / `irqaffinity=0` boot args. For a single hot RT thread on an isolated core, `NO_HZ_FULL` (with those boot args) generally gives the lower worst case by removing the 1 kHz tick from the RT core; if the RT core runs multiple runnable tasks, `NO_HZ_FULL` cannot drop the tick and `HZ_PERIODIC` is the more robust choice. Measure both under real load. Either way, drop the inert `CONFIG_NO_HZ=y` (no effect without `NO_HZ_COMMON`). 8. Reconsider `CONFIG_CPUSETS=y`. TI's RT config disables it (its notes cite ~40 µs of added worst-case latency from cgroup controllers). If cpusets are used only for boot-time isolation, `isolcpus=` plus `taskset` achieve the same with no runtime cgroup overhead. Keep it only if you need dynamic runtime repartitioning. 9. Validate the final configuration with CodeSys under realistic dual-core load - the cyclictest numbers are a guide, not the acceptance criterion. --- ## Caveats and limitations - No same-condition replicate runs, so differences below ~15 µs on max latency are within the run-to-run noise floor and should be read as "no change." - Comparisons span up to ~2–3 weeks of calendar time and, in some parameter tests, two different boards; small shifts may reflect drift or hardware variability. - The load-migration explanation rests on core0 average latency as a proxy for core0 utilization; a direct per-core utilization capture (`mpstat` / `/proc/stat`) during an isolated-vs-non-isolated run would confirm it. - The isolation-under-load conclusion is supported indirectly (core0-idle migration evidence plus the dual-core DDR results). A **non-isolated Customer test 2** run — which does not exist in the current data set — would test it directly.