DRA74P: DRA7xxP HW leveling sometimes timeout only on EMIF1 /2Gb

Hi All,

we made additional tests ONLY on EMIF1 trying to isolate the problem: 

• emif_ddr_phy_ctlr_1_init = 0x0624400B - skip Gate training and skip write leveling training : 100% board start whit leveling good

• emif_ddr_phy_ctlr_1_init = 0x0424400B - skip gate training ONLY: we still saw leveling FAIL occurence

• emif_ddr_phy_ctlr_1_init = 0x0224400B - skip write leveling training ONLY: we still saw leveling FAIL occurence

How we can interpret these results?

can we assume that the problem is isolated in the DDR which seems does't send back the leveling info to the EMIF PHY  ( EMIF signal a timeout in tha leveling operation )?

thanks in advance

Regards

Michele and Nello

Hi Michele and Nello,

In the log file, it shows the following registers having a value of 0x1500150, but in the XLS, it shows them as 0x0. How are these configured in your software code?

Reg 0x4c0002c8: 0x1500150
Reg 0x4c0002cc: 0x1500150
Reg 0x4c0002d0: 0x1500150
Reg 0x4c0002d4: 0x1500150
Reg 0x4c0002d8: 0x1500150
Reg 0x4c0002dc: 0x1500150
Reg 0x4c0002e0: 0x1500150
Reg 0x4c0002e4: 0x1500150
Reg 0x4c0002e8: 0x1500150
Reg 0x4c0002ec: 0x1500150

Hi Kevin,

in our code, wiht HW_Leveling enabled, these register are configured all 0x0 ( you can see the value in the xls file which contains all reg values read back from EMIF after its configuration);

Log file, is related to SW_Leveling, these registers are not configured by our code ( we configure regs till EMIF_EXT_PHY_CONTROL_25 ), log files contains all reg value read back from EMIF after its configuration ;

Regards

Michele

Hi Michele,

Ok, thanks. It looks like training is failing to a specific memory (lower two bytes). It also looks like all training types (for those byte lanes) are yielding poor results. This is generally true if the first training type fails; however it seems like both write leveling and gate training fail per your results.

Is it a single board failing or multiple? If multiple failing boards, do all boards show same behavior (failing on same EMIF / byte lanes)?

Additionally, is this design relatively new, or has it been in production for some time?

Regards,
Kevin

Hi Kevin,

The fail is only on this single board (which is one of more 10k boards production batch)
We have to provide to our customer the 8D_report with the root cause in short time.
If we understood well from your reply, the problem seems to be located in memory with lower two Data_bytes.

We have to find root cause. what do you suggest?
Please could you provide test to isolate the problem?

Regards Nello Michele

Hi Nello,

Ok thanks for that additional information.

Would you be able to perform an A-B-A swap of the DDR3 memory connected to the two lower bytes to see if the failure follows the DDR3 device currently populated on the PCB?

Regards,
Kevin

No ,we can only replace DDR chip with new one (Because we do not have reboll machine, so we can not swap with 100% sucess)
Moreover we need to be sure to have done all the checks in current status to find the root cause .

We could suppose 3 possible results:
1 problem follows ddr chip (we ask Micron chip analisis), confirm?
2 problem disappear . (in this case production problem?) please consider that when leveling is ok, the board has been stressed -20/+60 celsius for 10 h without problems.
3 problem remain as it is. (in this case DDR Controller?) other test possible to continue in this direction before reworking ?

regards Nello

Hi Kevin,

  Sorry to pres you : i'm the MTA HW design manager. Customer is escalating to have the root cause. 

I ask to you :

if you can focus on case 3 and say to us if we could run further tests  before starting with the rework  , that as Nello said, is presenting risk to damage ddr and to close analysis unsuccessfully.

Many thanks

Br

Paolo

Hi Paolo,

Because the failure is specific to hardware training, there is not much from a software perspective that can be performed to isolate the issue further. A-B-A swaps and/or physical measurements would be needed.

A few thoughts:

• Check voltage rails (vdd_core / DDR IO supply)
  • Measure with scope / probe, ensuring value matches expected value
  • Try increasing vdd_core and the DDR IO supply by 50 mV to see if there is any impact on the failure
• Since re-initializing / re-training resolves the issue, verify that the power-up / reset procedure matches expectations. At minimal, 
  • Ensure power supplies ramped and stable at least 200 us before DDR reset released (rising edge)
  • Check that there is a 500 us delay between DDR reset (rising edge) and DDR cke (rising edge)
  • Check that clocks are locked and stable prior to DDR cke active (rising edge)
• Since the device fails write leveling,
  
  • Turn off all trainings except write leveling,
  • Try to probe DQS and DQ during the write leveling procedure to ensure activity. Perform this on a good byte lane (byte lane 2 or 3) as well as the bad byte lanes (byte lane 0 or 1) for comparison. 
  • If the TI processor is sending activity on the DQS but the DRAM is not returning anything on the DQ, then it seems like the issue may be on the DRAM side.

In the case of #3 (problem still exists with new DDR memory), you might be able to return the TI device for further analysis as outlined here: https://www.ti.com/support-quality/additional-information/customer-returns.html . 

Regards,
Kevin

Hi Kevin,

in case "HW Leveling" fails, do yuo have a retry procedure to redo HW Leveling ?

What you suggest to do in case something goes wrong during HW Leveling procedure ( done by bootloader software )?

As we already said ( init of this thread ) "We did a preliminary workaround introducing 100 ms delay between a failure event and a retry of EMIF init configuration and leveling (no action on DDR just action on EMIF) with 100% positive result:  EMIF LEVELING ok .", we wonder if you have a retry procedure to be followed.

Thanks in advance

Regards

Michele

Hi Kevin,

we inform you that we changed one DDR3 CI , the one connected to two lower bytes of EMIF1 of J6+.

After more than thousands of restarts we don't see again the issue.

Thanks for your support

Regards

Michele and Nello