We are seeing a mysterious lockup issue with the AM5K2E02. We have found the processor to hang occasionally in the same section of code. When the processor hangs we are not able to access any core registers using the debugger however we can accesses other peripheral registers and DDR memory. Reviewing the errata we found an ARM specific errata that we had not yet fully dispositioned. The errata is copied below for reference. After implementing the work-around that uses the hazard timeout detection the frequency of lockups was reduced significantly however we continue to see core stalls in the same area of the code. The difference now after enabling the hazard timeout detection is that the core starts executing again after we get an external interrupt that causes the program counter to jump to an interrupt handler. The core by this time however has stalled for too long and so we get cycle overrun problems.
The current timeline we believe we are seeing goes as follows:
1) A function is executed which causes the core to stall.
2) After 1024 clock cycles the hazard detect timeout reissues the instruction to the L2.
3) The core either is released from deadlock and immediately deadlocks again or the reissued instruction does not release the core from deadlock.
4) An external interrupt causes the PC to jump to an interrupt handler however the core is still stalled.
5) The hazard detect timeout triggers again and causes the core now to start executing the code in the interrupt handler. In this state we are able to attach to the core and debug normally.
Prior to enabling the hazard detect timeout the core appeared to hang at step 3 above, we would eventually see the PC change to the interrupt handler address after the external interrupt but the core would remain hung and we would not be able to attach to it using the debugger.
The function that is being executed when the core appears to stall is pretty standard code, nothing out of the ordinary however there are quite a few pointers that are being de-referenced which may be causing L2 congestion at the time. The last instruction that we see in the instruction trace buffer before the jump to the interrupt handler is a "beq". It should also be noted that this function gets executed 100 times per second and its only after about 8 hours of running that the case occurs where the core hangs long enough to overrun the external interrupt. The SOC has two ARM cores however code is only executing on one of the cores, the second core is in a disabled state.
How might determine what is causing the core to deadlock?
798870: A memory read can stall indefinitely in the L2 cache
Category B
Products Affected: Cortex-A15 MP Core -NEON.
Present in: r2p0, r2p1, r2p2, r2p3, r2p4
Description
If back-to-back speculative cache line fills (fill A and fill B) are issued from the L1 data cache of a CPU to the L2 cache, the second request (fill B) is then cancelled, and the second request would have detected a hazard against a recent write or eviction (write B) to the same cache line as fill B then the L2 logic might deadlock.
Note: This erratum matches bug #5418 in the ARM internal JIRA database.
Configurations affected:
This erratum does not affect the processor if the revision and variant reported by the MIDR is r2p0, r2p1, r2p2, r2p3 or r2p4 and REVIDR[2] is set to 1.
To be affected by this erratum, the “L2 arbitration register slice” configuration option must be included
Conditions
1) The L2 memory system is congested, typically due to requests from other cores, or slow memory responses
2) Four data read requests are issued from one CPU (CPUA) to the L2 and added to the Load Request Queue (LRQ)
3) The four read requests are issued from the LRQ to the L2 pipe, but all stall due to the congestion
4) An eviction or write to line B (write B) is issued from CPUA to the L2 and is added to the Write Request Queue (WRQ)
5) Two speculative cache line fills (fill A and fill B) are issued from CPUA in sequential cycles
6) Fill B is cancelled and the actual fill is not issued
7) Fill B is to the same cache line as write B
8) Write B completes
9) A read request (read C) is issued from CPUA
Implications
If the erratum conditions are met, a false hazard is created between the final read request (read C) and write B (which has completed and deallocated). Read C cannot be issued from the LRQ until a deallocation event is seen for write B. However, as write B has already deallocated, this never occurs. Read C therefore deadlocks. Other cores are likely to stall at the next DSB instruction. Interrupts will not break the deadlock.
Workaround
When Bit[7] of the L2 Auxiliary Control Register (L2ACTLR[7]: Enable hazard detect timeout) is set, any memory transaction in the L2 that has been stalled for 1024 cycles is reissued to verify that its hazard condition still exists. In the erratum case, if the final read (read C) reissues, it determines that no hazard exists and completes normally.
However, setting this bit is not a complete workaround because the L2 may go into idle state if there is no activity for 256 cycles. If any core is active and doing any memory transactions (including cacheable loads hitting the L1 data cache), or if there are any ACP requests or ACE coherence requests, the L2 will stay active. If the L2 is idle, however, the reissue of Read C will be blocked.
For a single core A15 without ACE or ACP traffic there is no complete workaround. Setting L2ACTLR[7] will help, but will not completely remove the erratum. However it is not expected that any single core configuration will include the “L2 arbitration register slice” configuration option.
For an A15 with two or more cores active, or with ACE or ACP traffic, setting L2ACTLR[7] to 1’b1 is a good workaround. The L2 may go idle, but will be periodically active and allow the request to reissue. Typical worst case delay would be the next timer interrupt. If all but one of the A15 cores are powered down (such that there will be no further L2 traffic or interrupts to other cores), a watchdog timer outside of A15 should be used to wake up an additional core if the single active core becomes unresponsive.
