AM2434: PSRAM Temperature Boot Problem

Christian, 

A copule of questions to help narrow down and determine where progress can potentially be made. 

• Did anything change between the older and newer boards such as layout, PSRAM device selection, etc?
• Is this happening on all new boards 100% of the time, or only a percentage?
• Are you booting at room temp and increasing, or simply trying to start up at this elevated temperature starting from a cold boot?
• How is PSRAM used in the startup flow?

My assumption is that some non-volatile memory stores contents that are loaded into the PSRAM during startup. Is it possible on a good startup to do some read/write testing as the memory contents being corrupted on inspection may indicate something wrong with the write side. 

If you are also able to share what mode and frequency of operation you are using for the PSRAM, a memory type, and/or a partial schematic of the memory interface it could help further review. 

Thanks,

Chris 

• Did anything change between the older and newer boards such as layout, PSRAM device selection, etc?
  No, layout etc is identical. PSRAM seems to be the same.
• Is this happening on all new boards 100% of the time, or only a percentage?
  It seems to happen only to a few boards, some are more affected (lower temperature) than others. I don't know how many are affected.
• Are you booting at room temp and increasing, or simply trying to start up at this elevated temperature starting from a cold boot?
  I'm booting a elevated temperature, all boards seem to boot normal at low temperature
• How is PSRAM used in the startup flow?
  Not sure how to answer this, some code is loaded to the PSRAM, also some data is located there. Heap as well.
  
  
  I did some very limited memory testing during my tests to find out what is wrong. On good startups all data was there, no corruption. On bad startups, no data from this test appeared in the debugger mem view, even though the code must have run at least partially past this point.
  
  I shared some of this with Manuel, I can't put this here. 
    
  Thanks,
  Christian

Hi Christian, 

Below are a few more questions to help understand the issue further:

• Are you able to lower the clock frequency on a board that fails at high temperature and see if the behavior persists? 
• Are there any other connections on the signal traces besides the typical passives? Are the pull resistors connected in line with the signal trace? 
• Can you provide a scopeshot of the clock and one of the data signals for a board that is failing at high temp as well as a capture of the signals when startup is as expected with room temperature? 
• Are you connecting the DQS between the PSRAM and AM243x or are you using an internal/external loopback of the clock for the DQS input?
• Can you boot at room temperature and increase temperature to the AM243x with constant read/writes to the PSRAM and observe when the failure occurs?

Regards,

Erik

Hi Christian,

I look forward to the results of the testing and the feedback from the hardware team.

Christian Dahlke said:

For the last point, I don't think this is possible. The problem does not occur deterministically, and it is not easy to reproduce. The boot time is short, and I can't heat the board in a controlled environment that quickly. Once the firmware is running, it seems that the problem does not occur anymore, regardless of the temperature.

Can you confirm this point? Normal read/writes can take place without issue after successfully booting, even at high temperature?

Regards,

Erik

Hi Erik,

I can confirm that normal read/write operations work at high temperature after booting. Once booted, the firmware runs without problems, even when heating the board to the same temperatures at which the boot problem occurs. I will try to read/write to a specific location where I saw the problem in the debugger, but this may take some time.

Some additional information regarding the PSRAM:
It is an asynchronous PSRAM, type: IS67WVE4M16TBLL-70BLA2. No clock and no DQS.

Regards,

Christian

Hi Christian,

Are you re-configuring GPMC after ROM boot, when you have no problems at high temperature? If so can you share your GPMC_CONFIG[7:0]_0 registers?

If your config is different, can you try a write/read/verify test to use the GPMC registers set by the ROM? You can dump the registers right after GPMC boot. They should match below:

//ROM GPMC NOR boot GPMC_CONFIG[7:0]_0 registers
    CSL_REG32_WR(0x3B000060, 0x00001010); //GPMC0_CONFIG1_0 used by ROM GPMC NOR boot
    CSL_REG32_WR(0x3B000064, 0x00101c01); //GPMC0_CONFIG2_0 used by ROM GPMC NOR boot
    CSL_REG32_WR(0x3B000068, 0x23060917); //GPMC0_CONFIG3_0 used by ROM GPMC NOR boot
    CSL_REG32_WR(0x3B00006C, 0x1005bc1a); //GPMC0_CONFIG4_0 used by ROM GPMC NOR boot
    CSL_REG32_WR(0x3B000070, 0x011b111e); //GPMC0_CONFIG5_0 used by ROM GPMC NOR boot
    CSL_REG32_WR(0x3B000074, 0x8f070000); //GPMC0_CONFIG6_0 used by ROM GPMC NOR boot
    CSL_REG32_WR(0x3B000078, 0x00000c50); //GPMC0_CONFIG7_0 used by ROM GPMC NOR boot

Can you also confirm which silicon revision you have? You could send a photo of the AM243 device.

Quickly studying the timings of the GPMC NOR boot mode against the ISSI pSRAM appears to show good margins. A more in-depth timing analysis may reveal some timing marginality. Did you do this activity? Errors that occur only when raising the temperature might indicate a timing margin issue.

The ROM config translates to the below waveforms for reads.

Finally, the ROM was intended for NOR flash - not pSRAM. The timings appear compatible at first glance.
Is the pSRAM a suitable non-volatile memory for booting from? My understanding is that it retains data as long as power is supplied. Are you cycling power? Is voltage stable between boots?

Looking forward to solving this issue together.

-Mark

Hi Mark,

I will try to read the registers, but it may take some time. I will also check if it is possible to measure the timings.

This is the AM243 on a newer board, where the problem occurs at about 65°C:

This is the AM243 on an older Board where the problem occurs above 110°C:

There is no power cycle when the PSRAM is in use. The boot sequence looks like this as I understand it:

1. Power up
2. ROM Bootloader
3. The image (Core 1) is loaded from flash (into INTRAM and PSRAM)
4. Second Stage Bootloader
5. Core 1 startup
6. Core 3 image loaded from flash (into INTRAM and PSRAM) [executed from Core 1, Core 1 waits for Core 3 after this step]
7. Core 3 startup
8. Core 1 + Core 3 syncing + startup sequence => the problem occurs somewhere in this step, but it is not deterministic
9. Core 1 + Core 3 running

Kind Regards,

Christian