Core Isolation Impact on Cyclictest Latencies

Headline (most important finding for the customer goal):

In the customer test (-i1000), enabling core isolation HURTS Core 1 max in every single pair — 8/8 pairs show isolated Core1 max higher than non-isolated, by +44 µs to +188 µs (mean ≈ +89 µs). The customer-test Core1 average also doubles (≈13-14 µs unisolated vs ≈22-23 µs isolated).
In the SDK test the result is the opposite: isolation improves Core1 max in 7/8 pairs, dramatically so when paired with OPTEE-PRNG (max1 collapses to ~37 µs).
If the goal is to minimize customer-test Core 1 max, do NOT enable core isolation. This is the largest single-factor effect observed in this data set on the metric the customer cares about.
Isolation's effect on Core 0 max in the customer test is mixed and small relative to its effect on Core 1.

What is being compared

"isolate CPUs" rows in the test table use a kernel configuration that removes one or more cores from the default scheduler domain.

Note: the SDK test header switches the stress-ng command between the two isolation states (isolated: taskset -c 0,1 stress-ng -c 4; non-isolated: stress-ng -c 4 across all cores). The customer test uses the same stress-ng command in both states. So the SDK isolation pairs change two things at once (kernel isolation AND stress-ng placement); the customer pairs are a cleaner isolation A/B.

SDK test pairs (8)

Aggregate Δmax (B − A) where A=no-iso, B=isolated, so negative = isolation better:
Core0: mean Δmax0 = +7.0 µs | isolation better in 3/8 pairs, worse in 4, tied in 1.
Core1: mean Δmax1 = -13.2 µs | isolation better in 7/8 pairs, worse in 1, tied in 0.

condition (fixed)	no-iso file	max0	max1	iso file	max0	max1	Δmax0	Δmax1	plot
col=SDK test, fs=base, optee=oob, iso=n, ddr=n	260522_2	51	122	260530_1	108	102	+57	-20	histograms ↓
col=SDK test, fs=base, optee=oob, iso=n, ddr=Y	260524_1	72	85	260524_3	93	84	+21	-1	histograms ↓
col=SDK test, fs=base, optee=prng, iso=n, ddr=n	260522_4	54	63	260531_1	62	37	+8	-26	histograms ↓
col=SDK test, fs=base, optee=prng, iso=n, ddr=Y	260523_3	54	83	260527_1	67	37	+13	-46	histograms ↓
col=SDK test, fs=default, optee=oob, iso=n, ddr=n	260518_1	126	66	260530_4	101	111	-25	+45	histograms ↓
col=SDK test, fs=default, optee=oob, iso=n, ddr=Y	260526_4	104	107	260526_2	104	99	0	-8	histograms ↓
col=SDK test, fs=default, optee=prng, iso=n, ddr=n	260519_1	62	55	260530_2	61	36	-1	-19	histograms ↓
col=SDK test, fs=default, optee=prng, iso=n, ddr=Y	260531_4	81	68	260524_2	64	37	-17	-31	histograms ↓

Customer test (-i1000) pairs (8)

Aggregate Δmax (B − A) where A=no-iso, B=isolated, so negative = isolation better:
Core0: mean Δmax0 = -1.1 µs | isolation better in 5/8 pairs, worse in 3, tied in 0.
Core1: mean Δmax1 = +89.5 µs | isolation better in 0/8 pairs, worse in 8, tied in 0.

condition (fixed)	no-iso file	max0	max1	iso file	max0	max1	Δmax0	Δmax1	plot
col=Customer -i1000, fs=base, optee=oob, iso=n, ddr=n	260529_1	460	72	260529_4	250	180	-210	+108	histograms ↓
col=Customer -i1000, fs=base, optee=oob, iso=n, ddr=Y	260528_3	203	69	260525_1	240	113	+37	+44	histograms ↓
col=Customer -i1000, fs=base, optee=prng, iso=n, ddr=n	260531_3	198	73	260530_3	175	149	-23	+76	histograms ↓
col=Customer -i1000, fs=base, optee=prng, iso=n, ddr=Y	260528_2	83	49	260526_1	72	116	-11	+67	histograms ↓
col=Customer -i1000, fs=default, optee=oob, iso=n, ddr=n	260529_2	269	78	260522_3	475	266	+206	+188	histograms ↓
col=Customer -i1000, fs=default, optee=oob, iso=n, ddr=Y	260528_4	251	59	260525_2	226	133	-25	+74	histograms ↓
col=Customer -i1000, fs=default, optee=prng, iso=n, ddr=n	260531_2	252	72	260529_3	251	159	-1	+87	histograms ↓
col=Customer -i1000, fs=default, optee=prng, iso=n, ddr=Y	260528_1	83	63	260523_2	101	135	+18	+72	histograms ↓

Aggregate trends across multiple conditions

Sorting the 16 customer-test Core1 maxima (-i1000 column) by isolation state, without controlling for any other factor:

Non-isolated Core1 max range: 49 to 78 µs (n=8). Median ≈ 67 µs.
Isolated Core1 max range: 113 to 266 µs (n=8). Median ≈ 142 µs.

The two distributions do not overlap. The isolation state alone explains the customer-test Core1 max distribution better than any combination of FS, OPTEE, or DDR within the isolated half.

Conclusions

For the customer test: turn off core isolation. This is the only factor in the data set that, by itself, gates whether customer-test Core1 max stays under 100 µs.
For the SDK test: isolation is beneficial (combined with OPTEE-PRNG it gets Core1 max under 40 µs). The SDK and customer tests have opposite isolation behavior, so a single platform configuration cannot optimize both.

Caveats

SDK isolation pairs change both kernel isolation and stress-ng pinning (because the SDK test header rewrites the stress command based on isolation). The SDK trend therefore mixes two factors and is not a clean isolation A/B.
The customer-test isolation pairs are cleaner (same stress command both sides), so the customer-test conclusion is the more defensible one.
No same-condition replicates exist, so the run-to-run noise floor on max latency cannot be directly measured.

Appendix — per-pair histogram overlays

Reading the plots: filled circles are the baseline (A) run, "x" markers are the changed (B) run. The top panel is Core 0, the bottom panel is Core 1, both on a log-y count axis. A bump in the A curve that vanishes in the B curve is direct visual evidence that the factor under test produced or removed a latency cluster.

SDK test

fs=base, OPTEE=oob, iso=n, DDR=n | no-iso=260522_2 (max0=51, max1=122) vs iso=260530_1 (max0=108, max1=102) ↑ back to top

fs=base, OPTEE=oob, iso=n, DDR=Y | no-iso=260524_1 (max0=72, max1=85) vs iso=260524_3 (max0=93, max1=84) ↑ back to top

fs=base, OPTEE=prng, iso=n, DDR=n | no-iso=260522_4 (max0=54, max1=63) vs iso=260531_1 (max0=62, max1=37) ↑ back to top

fs=base, OPTEE=prng, iso=n, DDR=Y | no-iso=260523_3 (max0=54, max1=83) vs iso=260527_1 (max0=67, max1=37) ↑ back to top

fs=default, OPTEE=oob, iso=n, DDR=n | no-iso=260518_1 (max0=126, max1=66) vs iso=260530_4 (max0=101, max1=111) ↑ back to top

fs=default, OPTEE=oob, iso=n, DDR=Y | no-iso=260526_4 (max0=104, max1=107) vs iso=260526_2 (max0=104, max1=99) ↑ back to top

fs=default, OPTEE=prng, iso=n, DDR=n | no-iso=260519_1 (max0=62, max1=55) vs iso=260530_2 (max0=61, max1=36) ↑ back to top

fs=default, OPTEE=prng, iso=n, DDR=Y | no-iso=260531_4 (max0=81, max1=68) vs iso=260524_2 (max0=64, max1=37) ↑ back to top

Customer test (-i1000)

fs=base, OPTEE=oob, iso=n, DDR=n | no-iso=260529_1 (max0=460, max1=72) vs iso=260529_4 (max0=250, max1=180) ↑ back to top

fs=base, OPTEE=oob, iso=n, DDR=Y | no-iso=260528_3 (max0=203, max1=69) vs iso=260525_1 (max0=240, max1=113) ↑ back to top

fs=base, OPTEE=prng, iso=n, DDR=n | no-iso=260531_3 (max0=198, max1=73) vs iso=260530_3 (max0=175, max1=149) ↑ back to top

fs=base, OPTEE=prng, iso=n, DDR=Y | no-iso=260528_2 (max0=83, max1=49) vs iso=260526_1 (max0=72, max1=116) ↑ back to top

fs=default, OPTEE=oob, iso=n, DDR=n | no-iso=260529_2 (max0=269, max1=78) vs iso=260522_3 (max0=475, max1=266) ↑ back to top

fs=default, OPTEE=oob, iso=n, DDR=Y | no-iso=260528_4 (max0=251, max1=59) vs iso=260525_2 (max0=226, max1=133) ↑ back to top

fs=default, OPTEE=prng, iso=n, DDR=n | no-iso=260531_2 (max0=252, max1=72) vs iso=260529_3 (max0=251, max1=159) ↑ back to top

fs=default, OPTEE=prng, iso=n, DDR=Y | no-iso=260528_1 (max0=83, max1=63) vs iso=260523_2 (max0=101, max1=135) ↑ back to top