AM6442: Unexpected high latencies when executing high priority ISR from TCM

A few more notes:

I've noticed that the FreeRTOS code sets the DLFO bit, even though that shouldn't be necessary. I ran the code without DLFO, but that apparently didn't affect the performance.

I've also changed FreeRTOS / HwiP to disable the VIM vectored interface (VE bit), again without any noticeable effect.

Hi Dominic,

Thanks for bringing this to our notice. Can you please provide "background task" code? Also can you please share the .map and linker cmd file?

I have routed the question to respective organization to address this issue. Some one from that organization would get in touch with you for any further details.

Thanks

Hello Aravind,

I've made some progress, since today the GEL files approach for initializing DDR worked. No idea yet what was wrong yesterday, but I'd like to stay away from GEL files anyway.

• The problem is only visible on our own hardware (which is using LPDDR4)
  • the problem is visible both when using the prebuilt bootloader + GEL script for LPDDR4 init
  • and when using our modified bootloader that initializes LPDDR4 using code from the PDK
• The problem is not visible on the TI EVM (which is using DDR4)
  • when using the prebuilt bootloader + GEL script for DDR4 init

Though I couldn't find any details on this, my current assumption is that code fetches are not abortable, contrary to data accesses, which are abortable when the memory is marked as normal (which it is, in my case). Is there someone at TI who could confirm or refute that assumption?

Since the test code doesn't use any other board specific hardware, I'm guessing that the issue is the difference between DDR4 on the EVM and LPDDR4 on our custom hardware. That leaves me with either a bug in our LPDDR4 configuration or some hardware related issue.

At this point I don't think the actual instructions of the background code matter anymore, but here's the code I'm using to build my 1MB function at 0x80000000:

    ddr_loop_mem[0] = 0xe3a00003;       /* mov r0, #3 */
    ddr_loop_mem[1] = 0xe1a00000;       /* nop        */

    for (i = 2; i < 0x3fffe; i+=2)
    {
        ddr_loop_mem[i + 0] = 0xee090f1c;   /* mcr        p15, #0, r0, c9, c12, #0 */
        ddr_loop_mem[i + 1] = 0xe59fc000;   /* ldr        r12, [pc]                */
    }

    ddr_loop_mem[0x3fffe] = 0xe1a00000;     /* nop               */
    ddr_loop_mem[0x3ffff] = 0xe1a0f00e;     /* mov        pc, lr */
    
    CacheP_wbInvAll(CacheP_TYPE_ALLD);
    CacheP_wbInvAll(CacheP_TYPE_ALLP);

    __builtin_arm_dmb(0xF);
    __builtin_arm_dsb(0xF);

The task then just looks at the time via ClockP_getTimeUsec, outputs logging every 1 and 10 seconds (tried other intervals, with no effect on the frequency of excess latencies), calls the 1 MB function at 0x80000000, and delays for 1 ms to make sure the idle task gets to run now and then.

Every time I've seen excess latencies in the ISR the background task was interrupted somewhere within 0x80000000...0x80100000. There's nothing else in DDR memory (it's not even defined in the linker script). If you want I can send you the complete test project (code is somewhat dirty, due to the many changes in search of the culprit I've made), but I'm not sure if that's useful at this point, since I'm unable to reproduce the issue on the EVM.

Regards,

Dominic

Another update:

I've measured access latencies when reading from DDR memory mapped as strongly ordered. The measurement loop runs with IRQs disabled from TCM, and measures processor cycles required to read a field of N 32-bit values. That measurement is repeated 100,000 times, and min/max and a histogram are generated.

The average latency for a single access is ~160 cycles, but the latencies for accessing the whole field of N values differs by ~21,000 cycles, independent of the size N:

field size 512: min: 83911, max: 104928
field size 1024: min: 167867, max: 188897
field size 2048: min: 335820, max: 357324

The histogram data shows that there are only two distinct values (+-1000):

field size 512, lower boundary 80000, step size 1000: 0;0;0;1949;97988;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;20;43;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0
field size 1024, lower boundary 160000, step size 1000: 0;0;0;0;0;0;0;2137;97738;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;21;104;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0
field size 2048, lower boundary 320000, step size 1000: 0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;12051;87699;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;19;230;1;0;0;0;0;0;0;0;0;0;0;0;0

(i.e. for N=512 the latency is either [83000,84000[ or [103000,104000[, for 1024 it's either ~167000 or ~188000, for 2048 it's either ~335000 or ~357000, but no latencies in between are seen).

This is a good match for the worst case latencies for my timer ISR of ~25us that I've seen.

The EVM doesn't show this behavior:

field size 512: min: 86301, max: 87370
0;0;0;0;0;0;63285;36715;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0
field size 1024: min: 172713, max: 174400
0;0;0;0;0;0;0;0;0;0;0;0;490;96731;2779;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0
field size 2048: min: 346018, max: 348358
0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;6705;92391;904;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

(i.e. the min/max difference is ~1000 cycles)

• I'd appreciate any ideas regarding what might cause these latencies. Our own memory is slightly different from the LPDDR4 default values in the Processor_DDR_Config_0.03.00.0000 tool, but so far I haven't seen any obvious issues. There are only 14 values set in the .syscfg file, the others are default values.

Regards,

Dominic

Hi Dominic

Thanks for all the details and information.  We're reviewing and will get back shortly.