DRA821U: DRA821 POST/BIST test without DDR

Hi Luke,

The assigned engineer is on vacation on Wed, so please expect a delay in response on this thread.

regards

Suman

Luke,

Luke said:

ycle testing (between -60C to 90C) of their board using DRA821U. We need your support to figure out the cause of such failures. 

DRA821 is known to work from -40C. -60C is not even recommended.

What software is being used? Any logs when the freeze happens? Any crash logs?

The eMMC/DDR can also cause issues at that temperature.

- Keerthy

We are running our POST test on a loop in U-Boot in the DRA821. The processor is able to operate at -60 and 90 without the rapid cycling. When we cycle the temperature from lower limit to upper limit it will get stuck in the middle temperature ranges like -20-20C which is not close to the cutoff. Before it gets stuck the voltages are all normal but we get some BIST issues that may be linked to the DRA or the DDR memory. Sometimes it will reset itself but eventually it always just gets stuck.

The team has the following questions:

1; Is there any way to determine if DRA821 is hung up internally or external bus?

2: Is there any PINs on the DRA821 which could provide information on above?

I put all the information below let me know if you need anything else.

Any information that you have put has not come through? Did you share a document?

- Keerthy

We are running our POST test on a loop in U-Boot in the DRA821. The processor is able to operate at -60 and 90 without the rapid cycling. When we cycle the temperature from lower limit to upper limit it will get stuck in the middle temperature ranges like -20-20C which is not close to the cutoff. Before it gets stuck the voltages are all normal but we get some BIST issues that may be linked to the DRA or the DDR memory. Sometimes it will reset itself but eventually it always just gets stuck.

Here are the post errors that our custom image U-boot returns before the processor gets stuck.

"POST Error: BIST startup tests, Error detected while running BIST on startup"

"POST Error: Memory marching 1s address, Marching 1 address data compare error"

The team has the following questions:

1; Is there any way to determine if DRA821 is hung up internally or external bus?

2: Is there any PINs on the DRA821 which could provide information on above?

Nicholas,

I am looping in our hardware expert for the above queries.

- Keerthy

Hello,

When a hang occurs, generally, the practice is to try and connect via JTAG to the core running the test and inspect its state.  You first try and halt the running core and inspect its state, and if there is issue with that you attach to the DAP port and use its master to read endpoints.  If you are unable to halt the core (assuming R5), usually it is indicative of a CPU to end point failure.It seems likely in this case your DDR may have collapsed and that caused the failure.

With the code composer studio do you know which registers specifically I should be probing in R5 to find these errors and also find the meaning of the values in the registers?

You will need to know information about the expected code execution to judge what you see upon connection.  Without this information it will be hard to make sense of what is seen.  You should use the unstripped ELF file to see symbols or at least of the build's symbol map file. I would suggest look at the PC and see what function it is using the map file or symbols.  Maybe its in some function which is polling for a status which never comes, maybe its stuck in an abort loop, ...   For the given PC you should look at the instructions just before and just after the PC.  Look at the registers associated with those instructions and see if they correspond with device addresses if so which ones. Also if the registers have 'junk' illegal values relative to the context of the function that can be a clue about what happened.

If the code is unknown to you and you are mainly trying to check out the hardware.  Often its best to use your own bare metal or simple code.  Something like a memtester running on internal memory only, then ddr only is recommended. A failures of such very structured and controlled code allow for understanding if things like memories have issues.  For example if the DDR timings are marginal for some lane, you tend to see errors during tests which correspond to the weak bit.

If you are running black box tests without pre-work you might be limited to pass/fail.  Getting into the hardware and software will be needed.  Monitoring power supplies and clocks to see if they are stable through the temperature shift is a way to understand if some external component might be giving out.  I recall in the past seeing some external SMPS with a wrongly installed capacitor resulting in regulation issues which caused similar crashes.   There will be many possible issue points in HW and SW.  For a new design if often makes sense to test on multiple boards as things like soldering issues happen.

Okay great thank you is there TI documentation on how to setup some of these suggestions in code composer studio? I am new to the tool.

Hello,

There is a CCS specific forum for startup and usage questions.  There is also some academy video content which may be helpful in ramping on the tool.

Code Composer Studio forum - Code Composer Studio︎ - TI E2E support forums

Code Composer Studio Academy

Here is data of two runs where the board gets stuck during our rapid cycle thermal testing. It will get stuck at temperatures that are within range and have memory and BIST errors prior to getting stuck in the middle of the POST test we have cycling. We are wondering how to interpret the program counter register data since it does not look like RAM addresses that we are familiar with. What would be a good thing to explore next? Any insights you have on what the DRA issue could be would be helpful for our team to create some improvement opportunities. 

When I first scanned this question, as you were mentioning BIST, I anticipated you were running from an R5 core.  R5 can use an SPL which is u-boot based or an SBL which is ~RTOS related.   This is where I have seen more of the HW BIST engines running.  What you show above is it seems your test is running at the A72 u-boot level.   What type of BIST tests are being run at this level?  SW driven ones?

The 0xBxxxxxxx address is in DDR and u-boot does run in DDR, I would assume this is u-boot, or your code if a72 is the core BIST is running on.  The assembly looks to be well enough structured. I would guess checking your generated u-boot.map file to see what code is actually running here.  U-boot can relocate itself, and become out of sync with the compiled symbols.  The point in time in u-boot where you launch your code test would determine which addresses it was running at.   I am not familiar with the current SDK u-boots so further inputs would have to be looked at by area experts.