L2 cache issue when running from NOR while accessing another GPMC CS

We have applied the following ARM errata....

460075
430973
458693

Did you utilize this ROM service prior to enabling the L2?

Can you post the code that enables the L2 cache for review?

We are currently not using the ROM service.  I will look into getting that enabled.

Below is the code we are using to enable the L2 cache and MMU.

// Invalidate I-Cache
__asm__ volatile ("mcr  p15,0,%0,c7,c5,0" : : "r" (0x0));
__asm__ volatile ("isb");

/*
 * Setup the MMU Table here.  Leaving code out to save space.
 */

/* Drain write buffer */
__asm__ volatile ("mcr  p15,0,%0,c7,c10,4" : : "r" (0));

/* TLB Flush */
__asm__ volatile ("mcr  p15,0,%0,c8,c7,0" : : "r" (0));

/* Load TTBR0 */
__asm__ volatile ("mcr  p15,0,%0,c2,c0,0" : : "r" (TTBR0_BASEADDR((uint32 )trans_table) |
                                                       TTBR0_C                               |
                                                       TTBR0_RGN_WB_WA));

// L2 Cache enable
__asm__ volatile ("mrc p15,0,%0,c1,c0,1" : "=r" (ctrlReg) :);
ctrlReg = ctrlReg | 0x2;
__asm__ volatile ("mcr p15,0,%0,c1,c0,1" : : "r" (ctrlReg));

/* Set default domain access (all manager) */
__asm__ volatile ("mcr  p15,0,%0,c3,c0,0" : : "r" (0xfffffffd));

/* Enable MMU, alignment, instruction cache, branch prediction, data cache */
__asm__ volatile ("mrc  p15,0,%0,c1,c0,0" : "=r" (mmu_ctrl) :);
mmu_ctrl |= (SCTLR_I | SCTLR_Z | SCTLR_C | SCTLR_A | SCTLR_M);
__asm__ volatile ("mcr  p15,0,%0,c1,c0,0" : : "r" (mmu_ctrl));
__asm__ volatile ("isb");

// Invalidate I-Cache
__asm__ volatile ("mcr  p15,0,%0,c7,c5,0" : : "r" (0x0));
__asm__ volatile ("isb");

Looked more into using the ROM service to invalidate the L2 cache.  We are currently using the following code to do achieve the same functionality.  We are disabling the L2 at this point and enabling a little later when we setup the MMU. 

    /* Invalidate L1 */
    mov     r0, #0                 
    mcr     p15, 0, r0, c8, c7, 0   /* Invalidate TLBs */
    isb
    mcr     p15, 0, r0, c7, c5, 0   /* Invalidate I-Cache */
    isb

    /* Invalidate L2 */
    mrc p15, 1, r0, c0, c0, 1
    ands r3, r0, #0x7000000
    mov r3, r3, lsr #23
    beq finished
    mov r10, #0

loop1:
    add r2, r10, r10, lsr #1
    mov r1, r0, lsr r2
    and r1, r1, #7
    cmp r1, #2
    blt skip
    mcr p15, 2, r10, c0, c0, 0
    isb
    mrc p15, 1, r1, c0, c0, 0
    and r2, r1, #0x7
    add r2, r2, #4
    ldr r4, =0x3ff
    ands r4, r4, r1, lsr #3
    clz r5, r4
    ldr r7, =0x00007fff
    ands r7, r7, r1, lsr #13
loop2:
    mov r9, r4
loop3:
    orr r11, r10, r9, lsl r5
    orr r11, r11, r7, lsl r2
    mcr p15, 0, r11, c7, c6, 2
    subs r9, r9, #1
    bge loop3
    subs r7, r7, #1
    bge loop2
skip:
    add r10, r10, #2
    cmp r3, r10
    bgt loop1
finished:
   
    /* Turn off L2 Cache */
     isb
     MRC p15, 0, r0, c1, c0, 1
     and r0, r0, #0xFFFFFFFD
     MCR p15, 0, r0, c1, c0, 1
     isb

Clayton Shotwell said:

Looked more into using the ROM service to invalidate the L2 cache.  We are currently using the following code to do achieve the same functionality.

It is absolutely required to use the ROM service.  It's not possible for you to achieve the same result because you cannot run in secure mode like the ROM code. 

I added the following lines just above the assembly I posted earlier.

/* Use the ROM to invalidate the L2 cache */
moveq r12, #0x1
smc #1

I have been working with the MMU configuration to see if there is a different configuration that might work better with our hardware setup.  I have tried configuring the NOR flash as strongly ordered or device memory and that allowed the bootloader to run but it slowed everything way down.  Next I tried to configure the L1 and L2 cache models separately to see if I could get better results.  I ended up configuring the L1 as non-cacheable and the L2 as write-through, no write allocate (write back with allocate or no-allocate also work) and not shareable (enabling the shareable bit in the MMU table slows everything down considerably).  This allows the bootloader to run without errors.  I found the configuration table I referenced on page 7-5 of the Cortex-A8 TRM revr3p2 for normal memory and another configuration table in the ARMv7 Architecture Reference Manual in section B3.8.2 that details the register values I've set to configure the MMU.  I am not sure why I am having to disable the caching in L1 to avoid the GPMC bad address errors but it seems to help.

To backtrack just a little bit for some more information on the errors I am seeing. The errors occur on the GPMC interface during reads and writes.  I am seeing bad address errors when I check the device before I try a read or a write to the deviceX (not the NOR flash device).  I checked the addresses from many different failures and none of the addresses fall into a valid address range that was configured in the GPMC config7_i register.  What's really odd is the address I am trying to access is not the address in the GPMC error address register (not even close). 

Clayton Shotwell said:

I added the following lines just above the assembly I posted earlier.

/* Use the ROM to invalidate the L2 cache */
moveq r12, #0x1
smc #1

I was hoping that would fix the issue, but in any case it's definitely the right thing to do so keep it there.

Clayton Shotwell said:

I checked the addresses from many different failures and none of the addresses fall into a valid address range that was configured in the GPMC config7_i register.

Will you please provide the address you accessed (virtual and physical) and the address of the failure?  A few register values would be good too, e.g. GPMC_config, any registers pertinent to the error you're seeing, etc.

One other issue comes to mind that I should mention.  When you enable cache for the bootloader you need to be very careful not to have data (i.e. the instructions) "stuck" inside the cache.  If you have configured write-through cache then that shouldn't be an issue as all levels of memory would be updated immediately.  However, in the case of write-back cache I think you would need to manually flush the data in order to "push" the instructions to the physical memory.

Have you done any profiling of the boot sequence, i.e. back when everything worked fine but was slow, what was the biggest offender?  Was it the actual copying of the application from NOR flash to DDR?  Or was it the execution of some piece of initialization code?

In order to narrow down your issue I think we should try to come up with a specific configuration and then work to debug it.  Right now there are a lot of variables changing and it's hard for me to understand what all is happening.  For example, I was thinking something like this:

• DeviceX: Strongly Ordered
• NOR flash: Normal (Cacheable/Bufferable), read allocate
• DDR: Normal (Cacheable/Bufferable), read/write allocate, write-through

Here are a few of the address errors I have seen.  There doesn't seem to be any pattern in the values.  These errors occur on both reads and write with no pattern and they do not always happen at the same point.

Address Accessed in GPMC     Error Address from GPMC register
0x18600006                                  0x15F1ECC0
0x18086020                                  0x02BEDB40
0x18600002                                  0x35C7ED80
0x18600006                                  0x38FD6D90
0x18600006                                  0x17C7ED80
0x18600002                                  0x03726690

Below are the address ranges I have mapped in the GPMC and the MMU mappings that are configured.
Device         GPMC Address    GPMC Size     MMU Address      MMU Size
NOR            0x08000000          64MB              0x08000000         16MB
                                                                              0x09000000         16MB
DeviceX      0x18000000         128MB             0x18000000         16MB
                                                                              0x1C000000        16MB
                                                                              0x1D000000        16MB

GPMC is being configured with smart idle and auto idle enabled.  DeviceX uses its wait pin while the NOR flash does not use its wait pin. 

Brad, Clayton is out this week and I'll be filling in.

To continue with what you guys had already checked out....  We believe all the setup for cache and MMU are correct (very similar to other project we've done and per the TRM for the most part).  Do we want to go through our use case of the GPMC interface and the really long processor stalls we're doing with the Wait lines?  I believe that is our root cause, but I haven't been able to find enough support evidence to figureout how to workaround/fix it.

I have confirmed with my hardware guy that the GPMC CS3 wait line can be held from 10s of uSec to max ~2.5ms. 

If I set the GPMC module to timeout, my transactions fail on CS3 (the one using the waits).  Which tells me the wait line is being held for longer then 6uS  (max GPMC wait that can be configured.  So this is in essence canceling my transactions.  So I have to stay configued at the GPMC level with no timeout.  Next I look at the L3 timeout settings.  When I calculate my max timeout on that interface it's about 1.5ms.  I don't know how to catch that interface's failure case if it's ever exceeded.  So my theory is the GPMC stalls out the L3 beyond it's timeout, but the GPMC transaction still attempts to complete... Which in this state, possibly??? leaves some unknown values locked in for the address, causing the GPMC access err? (Can you confirm the behavior of each module if a wait line is held beyond GPMC and L3 timeouts? And idea why this works with only L1 enabled?)

Section 5.2.3.4.2 "Time-Out" of the OMAP3530 TRM discusses the various registers associated with configuring and detecting a timeout on the L3 interconnect.  Do you see a timeout error being logged in the L3_TA_AGENT_STATUS register for GPMC (0x6800 2428)?  Note that it is a 64-bit register, i.e. get the whole thing.

What is the value of L3_TA_AGENT_CONTROL for the GPMC (0x6800 2420)?  Note that it is a 64-bit register, i.e. get the whole thing.

Hopefully looking at the registers will show that the root cause of the issue is truly an L3 timeout.  If that's the case it seems like you would simply want to turn off the timeout.

The last time we checked, I believe it was showing one of the 5 MPU agent ids as the timeout reason in the status reg.  I'll get a dump of those for you.

Can you completely turn off the timeout?  Would it truely stall the L2 controller or is there a level of inhert timeout in that controller that we're going to run into?

Matt Weber said:

Can you completely turn off the timeout?  Would it truely stall the L2 controller or is there a level of inhert timeout in that controller that we're going to run into?

It looks like there are a number of different time outs.  As far as I can tell both the GPMC and L3 Interconnect timeout can be turned off.  I did some quick searching of the ARM documentation and didn't see anything related to a timeout mechanism.  That said, it seems plausible that there might still be some kind of timeout mechanism in the Cortex A8 that I've not yet discovered.  Maybe I'm using the wrong word in my search or something...  That would be the only thing that would make any sense out of the fact that things work/break depending on how the cache is configured, etc.  We both need to dig some more to try and figure that out!

In the mean time I think the register dumps related to the interconnect are a good place to start.  Once we can get rid of the timeouts at the Interconnect then we can worry about the Cortex A8.

Brad, I'm back from vacation and back to work.

I did a register dump after disabling the L3 timeouts for the MPU, GPMC, and the RT (this one should disable all of the timeouts).  The timeouts are getting disabled right after clocks are being setup.  I get the following register values when I get a GPMC error.  I have two sets below.  One is for a GPMC write error and the other is for a GPMC read error.  From what I have read, this looks like and error with an MPU transaction.  The MPU L3 Error Log register points to an initiator ID of 0x19 which is the MPU subsystem.  I'll start digging through the ARM documentation to see what I can find.

  IO Write Access Post Err @[0x18082042]
  GPMC ERR Addr  [0x0c9e8b80]
  GPMC ERR Type  [0x00000111]
  GPMC Status    [0x00000001]
  GPMC L3 Ctrl   [0x0000000003000000]
  GPMC L3 Status [0x0000000000000000]
  GPMC L3 Error  [0x0000000000000000]
  GPMC L3 ErrAddr[0x0000000000000000]
  MPU L3 Ctrl    [0x000000003e000000]
  MPU L3 Status  [0x0000000010000010]
  MPU L3 Error   [0x0000000082001901]
  MPU L3 ErrAddr [0x0000000047f20ac0]
  SI Flg Sts0    [0x0000000000000004]
  SI Flg Sts1    [0x0000000000000000]

  IO Read Access Post Err @[0x18082028]
  GPMC ERR Addr  [0x3f7ab580]
  GPMC ERR Type  [0x00000111]
  GPMC Status    [0x00000001]
  GPMC L3 Ctrl   [0x0000000003000000]
  GPMC L3 Status [0x0000000000000000]
  GPMC L3 Error  [0x0000000000000000]
  GPMC L3 ErrAddr[0x0000000000000000]
  MPU L3 Ctrl    [0x000000003e000000]
  MPU L3 Status  [0x0000000010000010]
  MPU L3 Error   [0x0000000084001900]
  MPU L3 ErrAddr [0x0000000000000000]
  SI Flg Sts0    [0x0000000000000004]
  SI Flg Sts1    [0x0000000000000000]

Here are some notes I'm taking related to your first register dump.

Looking at the GPMC related registers I see the following:

Clayton Shotwell said:

  GPMC ERR Type  [0x00000111]

I see the following errors:  ILLEGALMCMD and ERRORNOTSUPPADD, i.e. no timeout.

Looking at the L3 interconnect related registers I see the following:

Clayton Shotwell said:

  SI Flg Sts0    [0x0000000000000004]

According to Table 5-29 "L3_SI_FLAG_STATUS_0 for Application Error" this is considered a "Functional Inband Error".

Clayton Shotwell said:

  MPU L3 Status  [0x0000000010000010]

I see INBAND_ERROR_PRIMARY and REQ_ACTIVE are set.

Clayton Shotwell said:

  MPU L3 Error   [0x0000000082001901]

This decodes as:

MULTI=1 (There are other errors in addition to this one.)

SECONDARY=0

CODE=2 (Address hole according to Table 5-26 "CODE Field Definition")

INITID=0x19

CMD=1

Clayton Shotwell said:

  MPU L3 ErrAddr [0x0000000047f20ac0]

This looks like a Reserved/invalid address.  This doesn't even map to the GPMC which I expect is related to why we have MULTI=1 (i.e. there was probably another error that did actually get to the GPMC).

Now looking at the second error (i.e. the read).

Clayton Shotwell said:

  SI Flg Sts0    [0x0000000000000004]

Functional in-band error

Clayton Shotwell said:

  MPU L3 Status  [0x0000000010000010]

IN_BAND_ERROR_PRIMARY=1

REQ_ACTIVE=1

Clayton Shotwell said:

  MPU L3 Error   [0x0000000084001900]

MULTI=1

CODE=4 (in-band error)

INITID=0x19 (MPU SS)

CMD=0 (not applicable for in-band error)

For an in-band error the documentation says to look at the PM_ERROR_LOG registers to see if this is related to a firewall.  I imagine that's not the case since you can access GPMC sometimes.  The final step is to look at the target specific registers, i.e. the GPMC. 

Clayton Shotwell said:

  GPMC ERR Type  [0x00000111]

ILLEGALMCMD = 1

ERRORNOTSUPPADD = 1

ERRORVALID = 1

Clayton Shotwell said:

  GPMC ERR Addr  [0x3f7ab580]

This address falls into the 1GB of space allocated to the GPMC (barely).

Clayton Shotwell said:

  IO Read Access Post Err @[0x18082028]

I don't understand exactly what this means. Where did 0x18082028 come from?

Brad Griffis said:

I don't understand exactly what this means. Where did 0x18082028 come from?

This is the address the GPMC is supposed to be accessing.  The code checks the GPMC Error register for an error before and after every transaction.  Most of the errors are detected after the transaction completes.  I have not been able to find any correlation between the address that we are trying to access and the address listed in the GPMC error address.

Is there any consistency in terms of where/when these errors occur?  For example, does it always occur on the first access or always on the nth access (where n is consistent)?  Or does it seem to happen randomly?  You might try power cycling the board between tests to see if you can improve repeatability of the test, i.e. to see if you can make it fail more consistently in the same place.  I'm starting to wonder if you might have some kind of hardware issue, e.g. power or clocking problem.

There is not a point where the failure always occurs but over several power cycles, there are a few specific places where the failures occur.  We can only detect the error when GPMC deviceX transfers happen.  We are checking both before and after the accesses to determine exactly when the failure occurs.  Almost all of the failures are detected after a transfer takes place which leads me to believe the issue with with the really slow transfer rather than the accesses to NOR flash.  I question a hardware problem because everything works fine, albeit a little slower, with the L2 cache turned off.  I feel like a hardware issue would manifest itself with or without the cache turned off.  

Could there be some kind of a timeout in the L2 cache that is not listed in the TRM or ARM Cortex-A8 A.R.M.?  I have not found any documentation stating what signals the GPMC send back (if any) while it is waiting for a transaction to complete.  Could the lack of response from the GPMC cause some kind of issue in the L2 cache since all traffic is routed through it when it is enabled?

The reason why I was considering a potential hardware issue was mainly due to the fact that you are seeing bizarre addresses show up in the error address registers.  Based on your register dumps it seems that the root of the failure is an access to a non-existent memory address.

I think some more data points would be useful.  In particular:

• Can you intentionally insert a bad access to a non-existent address (perhaps with cache disabled just to avoid any issues caused by cache).  For example, try doing a read from address 0x3F00_0400.  In particular I'm looking to see if you get the same type of address hole error, and also what address is reported by the GPMC.
• Can you use the DMA to perform some accesses on deviceX?  It would be good to know if the interconnect, GPMC, etc. are all performing properly.
• If you get both of the above done, then it's probably easy to try a combo, i.e. use DMA to intentionally perform a bad access to see again whether there's any consistency in terms of GPMC error reporting.

Matt Weber said:

We have applied the following ARM errata....

460075
430973
458693

What about 454179 and 621766?  Here's a snippet of code from u-boot that sets up the Auxiliary Control Register:

GPL CODE: u-boot/arch/arm/cpu/armv7/omap3/board.c said:

static void omap3_setup_aux_cr(void)
{
    /* Workaround for Cortex-A8 errata: #454179 #430973
     *    Set "IBE" bit
     *    Set "Disable Brach Size Mispredicts" bit
     * Workaround for erratum #621766
     *    Enable L1NEON bit
     * ACR |= (IBE | DBSM | L1NEON) => ACR |= 0xE0
     */
    omap3_update_aux_cr_secure(0xE0, 0);
}

Note that the omap3_update_aux_cr_secure API is using one of the secure monitor calls that I mentioned in order to write to that register.  In your code are you setting all 3 of the bits shown above?

I also saw another piece of code that looked interesting:

GPL CODE: u-boot/arch/arm/cpu/armv7/omap3/board.c said:

void v7_outer_cache_enable(void)
{
	/* Set L2EN */
	omap3_update_aux_cr_secure(0x2, 0);

	/*
	 * On some revisions L2EN bit is banked on some revisions it's not
	 * No harm in setting both banked bits(in fact this is required
	 * by an erratum)
	 */
	omap3_update_aux_cr(0x2, 0);
}

I'm not sure which errata that's referring to.  I couldn't find any errata related to L2EN.  ???  In any case, one thought is that perhaps there's an errata somewhere or some kind of hiccup in the initialization procedure that's causing grief.

Applied the other errata and saw no changes.

Attempted the invalid GPMC address access and caused a GPMC invalid address error.  I tried to access 0x3F00_0400 and got a GPMC invalid address error with the correct address in the address error register.  I tried several other values on all sides of the addresses setup in the GPMC and got the same results.  This test did not cause any L3 bus errors. This proves that a valid address is being sent into the GPMC and then it somehow corrupts.

I am still working on getting the DMA controller setup and working so that should be done early next week. 

Clayton Shotwell said:

Attempted the invalid GPMC address access and caused a GPMC invalid address error.  I tried to access 0x3F00_0400 and got a GPMC invalid address error with the correct address in the address error register.  I tried several other values on all sides of the addresses setup in the GPMC and got the same results.  This test did not cause any L3 bus errors. This proves that a valid address is being sent into the GPMC and then it somehow corrupts.

Since the address is somehow being messed up perhaps you should look more closely at your MMU initialization.  Perhaps a page table is not correctly setup or maybe your initialization sequence doesn't invalidate the TLB properly?

The DMA test should be able to solidify or refute the theory.  In other words, if everything works consistently when using DMA that would further point at an MMU issue rather than a timing issue.  I think using the DMA is a good test because it's performance will be optimal (similar to having all the caching enabled) but it will not have any of the MMU logic.  So if things work fine with DMA then I would suspect an MMU issue but if things fail with DMA then we might have some kind of timing issue after all.

I was able to get the SDMA working today between two SDRAM locations but I was unable to either read or write to the GPMC devices using the same DMA setup.  After a lot of debugging, I found no errors being reported by the DMA controller.  The controller acted like the transfer was successful even though it was not.  I did notice that my check of the channel enable bit caused my delay loop to hit (~100ms wait) on the transfers that failed.  I attempted to transfer both single words and long (256 byte) data streams without any success.  Could there be a timeout in the SDMA controller I have overlooked?  Below is a register dump of the SDMA registers.  I have been using channel 0 for all of the transfers.

___address____|________0________4________8________C_
  NSD:48056000| 00000040 ???????? 00000000 00000000
  NSD:48056010| 00000000 00000000 00000000 00000000
  NSD:48056020| 00000000 00000000>00000001 00000000
  NSD:48056030| 00000000 00000000 00000000 00000000
  NSD:48056040| FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
  NSD:48056050| ???????? ???????? ???????? ????????
  NSD:48056060| 00000000 000C0000 ???????? 000001FF
  NSD:48056070| 000000F3 000000FE 00014001 ????????
  NSD:48056080| 01C05000 00000000 00000000 00000000
  NSD:48056090| 0016E1C0 00000002 00000001 18600000
  NSD:480560A0| 8000FF00 00000000 00000000 00000000
  NSD:480560B0| 00000000 18600002 8000FF02 00000002
  NSD:480560C0| 00000000 00000000 ???????? ????????
  NSD:480560D0| ???????? ???????? ???????? ????????

Clayton Shotwell said:

The second issue was the initial steps to flush the L2 cache and enable the chip.  Below is the code that ended up correctly setting up the L2 cache and allowing the system to run without any of the GPMC faults.  Is there a reason the MMU has to be disabled?  I believe the TRM has the MMU disable by default.

Coming out of reset the MMU is disabled in hardware, i.e. I wouldn't expect you to have to disable it because it should already be disabled.  This sounds to me like an issue that can be introduced when you load/run the same software multiple times, i.e. you do not have a consistent, known state from which the software runs each time.  In other words, for the very first run of the software you would have MMU disabled due to the power-on reset.  If you then re-run the same software the MMU is already enabled from the previous run when you go to configure it.  I've seen many variants of this same issue on many different peripherals (e.g. things like Timers, SPI, etc.).  For situations where the same software may run multiple times without a hardware reset I usually recommend that the software get in the habit of locally disabling/resetting the peripheral as part of the initialization process such that the software can "force" the peripheral into a known state at the start of each initialization.  That removes any dependencies on previous setups.