AM3358: cold start problem

Bartosz,

Please describe the hardware environment in detail. If possible, post the schematic.

Another question: Is this failure seen on a single board, or across multiple boards?

Hi Biser,

I've sent you the description of the setup and schematic offline (confidential).

So far the issue has been observed with one board.

Kind regards,
Bartosz

Is this only 1 board out of many? If so, I would suggest you check the board for manufacturing defects, esp. bad soldering.

By the way, the files you have sent are of no use. I was asking about the schematic in PDF format.

Biser,

We don't see manufacturing defects on the board.

The schematic in pdf has been sent to you now via e-mail.

Kind regards
Bartosz

Is this only 1 board out of many, or you see the issue on multiple boards?

Bartosz, as Biser alludes to, know how many boards has this issue is important to determine what kind of problem it is.

Based on your screenshot, it looks like you are getting a data abort in MLO. A lot of times, this is a result of a mis-configured DDR. Since it is intermittent or maybe temp related, the configuration may be marginal.

A couple of things you can try:
-On a failing board, try opening up a memory window in the DDR memory to see if the values are still stable (hit the continuous refresh button)
-Since you fail in MLO, you can put breadcrumbs in your code to determine how far you execute. Find a memory location in internal SRAM and just increment the value in different places in code. My suspicion is that the problem is happening during the loading of u-boot, when a lot of data is being transferred to DDR.
-In your screenshot, i notice a couple of ARM core registers with addresses such as 0x81FFFEE8 and 0x81FFFF20. These look to be stack located in DDR. Immediately after a fail, open up a memory window to see if these values are stable.
-In your screenshot, your SP is 0x00000000, which doesn't seem correct. May want to debug to try to understand why the stack pointer is being set to 0x0

Regards,
James

Hi James,

thanks for your feedback. Please find the comments below, looking forward for your help.

Kind regards,
Bartosz

Customer: I attach the trace vector, seen with debugger. Booting the card from the Ethernet source (SYSBOOT[15:0]=0x8050), the trace vector lowest value is 0xD03E, if the card boots from eMMC SYSBOOT[15:0]=0x8057 the trace vector value is 0x10009E.

Note: If the warm- or cold reset or the PWR_EN of the PMIC signals are pulled to low, the problem is still remaining. I also checked the Core and the MPU voltages for the OPP100 boot mode.

JJD said:

Based on your screenshot, it looks like you are getting a data abort in MLO. A lot of times, this is a result of a mis-configured DDR. Since it is intermittent or maybe temp related, the configuration may be marginal.

Could we configure the DDR in the MLO?

JJD said:

A couple of things you can try:
-On a failing board, try opening up a memory window in the DDR memory to see if the values are still stable (hit the continuous refresh button) 

It looks stable (picture attached in the zip). I cooled the card with cold spray.

JJD said:

-Since you fail in MLO, you can put breadcrumbs in your code to determine how far you execute. Find a memory location in internal SRAM and just increment the value in different places in code. My suspicion is that the problem is happening during the loading of u-boot, when a lot of data is being transferred to DDR.  

It is hard to reproduce the problem. The "data abort exception" is what we have only found so far.

JJD said:

-In your screenshot, i notice a couple of ARM core registers with addresses such as 0x81FFFEE8 and 0x81FFFF20. These look to be stack located in DDR. Immediately after a fail, open up a memory window to see if these values are stable.  

It also looks stabe (picture attached).

JJD said:

-In your screenshot, your SP is 0x00000000, which doesn't seem correct. May want to debug to try to understand why the stack pointer is being set to 0x0 

If I click on the run it will reload the right value of the SP (picture attached).

traces.zip

Hi James,

would you share your thoughts?

Thank you,
Bartosz

Hi Bartosz, one important point to make is that when you try to boot using MLO and then inspect the results using CCS, ensure that you have all of the GEL files disabled or removed from your target configuration. In several of your screenshots, it looks like a GEL script has run, which will render any results you see in CCS useless.

Can you retry some of the experiments above without the GELs? See if the memory window in DDR is stable after a crash. Can you also dump the EMIF registers after a boot from Ethernet and a boot from serial port, and compare the two.

Regards,
James

Bartosz, for further DDR analysis, can you use the DSS scripts described here: processors.wiki.ti.com/.../How_to_use_the_AM335x_IBIS_Models

This will give us more information beyond just a register dump.

Regards,
James

Hi James,

please see feedback from the engineer:

1. I created a new target configuration as described in the link. Then I waited for an hour to fail in the MLO, the card tried boot from MMC. I attached the results of the DDR analysis and the Boot script (link), the files begin with 1.

2. Then I connected to the card with the debugger for checking the DDR stable (attached the screenshot, begin with 2.) and exported the value of the EMIF (txt begin with 2.).

3. I set the SYSBOOT pins to Eth boot configuration and waited an hour again to fail in the MLO, and I attached the results of the DDR analysis and the Boot script (the files begin with 3.).

4. Then I connected to the card with the debugger again for checking the DDR stable (attached the screenshot, begin with 4.) and exported the value of the EMIF (txt begin with 4.).

5. I attached the results of the DDR analysis and the Boot script (the files begin with 5.) after a normal boot from MMC

6. I attached the results of the DDR analysis and the Boot script (the files begin with 6.) after a normal boot from Ethernet.

Looking forward for your feedback.

Thank you,

Kind regards,
TI Customer 

3704.files.zip