Linux/AM3354: DDR3 Logical Address Mapping to SDRAM address

Bob Duke said:

In our case, IBANK _POS=3 and EBANK_POS=0, so that would be table 7-210.

So, [9:0] would be column address, [10:24] row address, and [25:27] bank address.

Correct.

Bob Duke said:

So, for example,
32-bit memory location 0x9000 0000, would translate to bank 4, row 0, col 0 for lower 16-bits and bank 4, row 0, col 1 for upper 16-bits
32-bit memory location 0x9000 0004, would translate to bank 4, row 0, col 2 for lower 16-bits and bank 4, row 0, col 3 for upper 16-bits

Does this make sense?

Not quite.  You have bit 28 set which is undefined, and I don't know what you mean by "upper and lower 16-bits".  AM335x has one 16-bit DDR3 interface.

Brad Griffis said:

Not quite.  You have bit 28 set which is undefined, and I don't know what you mean by "upper and lower 16-bits".  AM335x has one 16-bit DDR3 interface.

Bit 28 should be the start of 256MB,  its a 512MB memory.

The "upper and lower 16-bits" references what we expect to happen when doing a 32-bit memory write/read.

You're right.  That's a valid address within the context of 512 MB of DDR3 RAM.  I was looking at this statement from Bob which I originally agreed with:

[Bob] So, [9:0] would be column address, [10:24] row address, and [25:27] bank address.

That's why I thought we shouldn't be using bit 28.  I think I see where we took a wrong turn.  Since the bus itself is 16 bits wide, you wouldn't provide different addresses to the DDR3 based on byte 0 or byte 1.  That's controlled by the DM signals.  So in short, we should not be considering address bit 0 as part of this address mapping.

That said, I think it should look like this:

• Bits 28:26 = bank address
• Bits 25:11 = row address
• Bits 10:1 = column address

Jonathan Cormier said:

The "upper and lower 16-bits" references what we expect to happen when doing a 32-bit memory write/read.

In normal operation the size of your access won't matter as the L2 cache controller will grab an entire cache line worth of data regardless. 

All of the physical accesses generated by the EMIF controller will be bursts of 8 sequential words.  If for example you were doing a 32-bit noncacheable write you would still see an 8 word burst, but the DM signals would mask off all the unwritten words.  Or similarly on a noncacheable read the EMIF would throw away the unwanted data.

Brad Griffis said:

Sounds like you're seeing some kind of issue. Is there consistency in the location (e.g. specific bank, etc)? I suggest following the steps from here:

processors.wiki.ti.com/.../How_to_use_the_AM335x_IBIS_Models

The output will be a txt file that you can attach to the forum. It would be a good sanity check that nothing is configured unexpectedly. Also, if you can share some specifics on some of the errors you're seeing, I might have further comment (e.g. expected data vs actual data, failure rates, etc.).

Memory failures have increased in our recent builds (several percent fallout). However they tend to be intermittent.  So we've been looking into memory timings and also trying to determine if memory test failures might indicate a specific ball in the memory module.

Reflowing some of the faulty boards has fixed them so we believe at least some of the failures are physical in nature.

Attached ddr analysis file.

EMIF: SDRAM_CONFIG = 0x61c05332
  * Bits 26:24 (reg_ddr_term) set for RZQ/4 (001b)
  * Bits 19:18 (reg_sdram_drive) set for RZQ/6 (00b)
EMIF: PWR_MGMT_CTRL = 0x00000000
 * ERROR: Bits 7:4 (reg_sr_tim) are in violation of Maximum Self-Refresh Command Limit
 * Please see the silicon errata for more details.
DDR PHY: DDR_PHY_CTRL_1 = 0x00100007
  * Bits 9:8 (reg_phy_rd_local_odt) configured as no termination (00b)

*********************
*** Register Dump ***
*********************

*(0x4c000000) = 0x40443403
*(0x4c000004) = 0x40000000
*(0x4c000008) = 0x61c05332
*(0x4c00000c) = 0x00000000
*(0x4c000010) = 0x00000c30
*(0x4c000014) = 0x00000c30
*(0x4c000018) = 0x0aaad4db
*(0x4c00001c) = 0x0aaad4db
*(0x4c000020) = 0x26437fda
*(0x4c000024) = 0x26437fda
*(0x4c000028) = 0x501f83ff
*(0x4c00002c) = 0x501f83ff
*(0x4c000038) = 0x00000000
*(0x4c00003c) = 0x00000000
*(0x4c000054) = 0x00ffffff
*(0x4c000058) = 0x8000140a
*(0x4c00005c) = 0x00021616
*(0x4c000080) = 0x04867045
*(0x4c000084) = 0x00db03ef
*(0x4c000088) = 0x00010000
*(0x4c00008c) = 0x00000000
*(0x4c000090) = 0x11766b35
*(0x4c000098) = 0x00050000
*(0x4c00009c) = 0x00050000
*(0x4c0000a4) = 0x00000000
*(0x4c0000ac) = 0x00000000
*(0x4c0000b4) = 0x00000000
*(0x4c0000bc) = 0x00000000
*(0x4c0000c8) = 0x50074be4
*(0x4c0000d4) = 0x00000000
*(0x4c0000d8) = 0x00000000
*(0x4c0000dc) = 0x00000000
*(0x4c0000e4) = 0x00100007
*(0x4c0000e8) = 0x00100007
*(0x4c000100) = 0x00000000
*(0x4c000104) = 0x00000000
*(0x4c000108) = 0x00000000
*(0x4c000120) = 0x00000305

************************
*** IOCTRL Registers ***
************************

CONTROL: DDR_CMD0_IOCTRL = 0x0000018b
  * ddr_ba2 Pullup/Pulldown disabled
  * ddr_wen Pullup/Pulldown disabled
  * ddr_ba0 Pullup/Pulldown disabled
  * ddr_a5 Pullup/Pulldown disabled
  * ddr_ck Pullup/Pulldown disabled
  * ddr_ckn Pullup/Pulldown disabled
  * ddr_a3 Pullup/Pulldown disabled
  * ddr_a4 Pullup/Pulldown disabled
  * ddr_a8 Pullup/Pulldown disabled
  * ddr_a9 Pullup/Pulldown disabled
  * ddr_a6 Pullup/Pulldown disabled
  * Bits 9:5 control ddr_ck and ddr_ckn
    - Slew slow
    - Drive Strength 9 mA
  * Bits 4:0 control ddr_ba0, ddr_ba2, ddr_wen, ddr_a[9:8], ddr_a[6:3]
    - Slew slow
    - Drive Strength 8 mA
CONTROL: DDR_CMD1_IOCTRL = 0x0000018b
  * ddr_a15 Pullup/Pulldown disabled
  * ddr_a2 Pullup/Pulldown disabled
  * ddr_a12 Pullup/Pulldown disabled
  * ddr_a7 Pullup/Pulldown disabled
  * ddr_ba1 Pullup/Pulldown disabled
  * ddr_a10 Pullup/Pulldown disabled
  * ddr_a0 Pullup/Pulldown disabled
  * ddr_a11 Pullup/Pulldown disabled
  * ddr_casn Pullup/Pulldown disabled
  * ddr_rasn Pullup/Pulldown disabled
  * Bits 4:0 control ddr_15, ddr_a[12:10], ddr_a7, ddr_a2, ddr_a0, ddr_ba1, ddr_casn, ddr_rasn
    - Slew slow
    - Drive Strength 8 mA
CONTROL: DDR_CMD2_IOCTRL = 0x0000018b
  * ddr_cke Pullup/Pulldown disabled
  * ddr_resetn Pullup/Pulldown disabled
  * ddr_odt Pullup/Pulldown disabled
  * ddr_a14 Pullup/Pulldown disabled
  * ddr_a13 Pullup/Pulldown disabled
  * ddr_csn0 Pullup/Pulldown disabled
  * ddr_a1 Pullup/Pulldown disabled
  * Bits 4:0 control ddr_cke, ddr_resetn, ddr_odt, ddr_csn0, ddr_[a14:13], ddr_a1
    - Slew slow
    - Drive Strength 8 mA
CONTROL: DDR_DATA0_IOCTRL = 0x0000018b
  * ddr_d8 Pullup/Pulldown disabled
  * ddr_d9 Pullup/Pulldown disabled
  * ddr_d10 Pullup/Pulldown disabled
  * ddr_d11 Pullup/Pulldown disabled
  * ddr_d12 Pullup/Pulldown disabled
  * ddr_d13 Pullup/Pulldown disabled
  * ddr_d14 Pullup/Pulldown disabled
  * ddr_d15 Pullup/Pulldown disabled
  * ddr_dqm1 Pullup/Pulldown disabled
  * ddr_dqs1 and ddr_dqsn1 Pullup/Pulldown disabled
  * Bits 9:5 control ddr_dqs1, ddr_dqsn1
    - Slew slow
    - Drive Strength 9 mA
  * Bits 4:0 control ddr_d[15:8], ddr_dqm1
    - Slew slow
    - Drive Strength 8 mA
CONTROL: DDR_DATA1_IOCTRL = 0x0000018b
  * ddr_d0 Pullup/Pulldown disabled
  * ddr_d1 Pullup/Pulldown disabled
  * ddr_d2 Pullup/Pulldown disabled
  * ddr_d3 Pullup/Pulldown disabled
  * ddr_d4 Pullup/Pulldown disabled
  * ddr_d5 Pullup/Pulldown disabled
  * ddr_d6 Pullup/Pulldown disabled
  * ddr_d7 Pullup/Pulldown disabled
  * ddr_dqm0 Pullup/Pulldown disabled
  * ddr_dqs0 and ddr_dqsn0 Pullup/Pulldown disabled
  * Bits 9:5 control ddr_dqs0, ddr_dqsn0
    - Slew slow
    - Drive Strength 9 mA
  * Bits 4:0 control ddr_d[7:0], dqm0
    - Slew slow
    - Drive Strength 8 mA
CONTROL: DDR_IO_CTRL = 0x00000000
  * Bit 31: DDR_RESETn controlled by EMIF.
  * Bit 28 (mddr_sel) configured for SSTL, i.e. DDR2/DDR3/DDR3L operation.
CONTROL: VTP_CTRL = 0x00010167
  * VTP not disabled (expected in normal operation, but not DS0).
CONTROL: VREF_CTRL = 0x00000000
  * VREF supplied externally (typical).
CONTROL: DDR_CKE_CTRL = 0x00000001
  * CKE controlled by EMIF (normal/ungated operation).

Jonathan,

Here's an excerpt from your report:

EMIF: PWR_MGMT_CTRL = 0x00000000

* ERROR: Bits 7:4 (reg_sr_tim) are in violation of Maximum Self-Refresh Command Limit

* Please see the silicon errata for more details.

DDR PHY: DDR_PHY_CTRL_1 = 0x00100007

 * Bits 9:8 (reg_phy_rd_local_odt) configured as no termination (00b)

The PWR_MGMT_CTRL register should be written with a value of 0xA0 if you want to support DeepSleep0.  In the current u-boot I see this performed in arch/arm/mach-omap2/am33xx/ddr.c :

/*
* Configure EMIF4D5 registers and MR registers For details about these magic
* values please see the EMIF registers section of the TRM.
*/
void config_sdram_emif4d5(const struct emif_regs *regs, int nr)
{
writel(0xA0, &emif_reg[nr]->emif_pwr_mgmt_ctrl);
writel(0xA0, &emif_reg[nr]->emif_pwr_mgmt_ctrl_shdw);

I'm pretty sure that has nothing to do with your current issue, but I recommend fixing it.

The other message pertains to ODT.  Currently you have it disabled on the AM335x.  That's going to impact reads specifically.  Can you tell if your memory errors relate to reads or writes?  If you are getting different values returned when repeatedly reading the same location, that indicates that (at a minimum) you have a read issue.

You mentioned earlier that some of the boards were fixed through reflow.  Perhaps all the issues are manufacturing-related, or perhaps there are actually multiple issues.  It's hard to know for sure.  If you have boards that were NOT fixed by the reflow, I think it would be very interesting to see if enabling ODT resolves their issue.

Was it a conscious decision to disable ODT (i.e. for reduced power consumption) or was that just due to copying values from the AM335x EVM?  If you want to disable ODT then you likely need to do some IBIS modeling of the interface.  That will enable you to tune the ODT, drive strength, etc. for both the AM335x and the DDR.  More importantly it gives you insight into what the signal integrity could look like at the corners of process, voltage and temperature.

Brad Griffis said:

Jonathan,

Here's an excerpt from your report:

EMIF: PWR_MGMT_CTRL = 0x00000000

* ERROR: Bits 7:4 (reg_sr_tim) are in violation of Maximum Self-Refresh Command Limit

The PWR_MGMT_CTRL register should be written with a value of 0xA0 if you want to support DeepSleep0.  In the current u-boot I see this performed in arch/arm/mach-omap2/am33xx/ddr.c :

/*
* Configure EMIF4D5 registers and MR registers For details about these magic
* values please see the EMIF registers section of the TRM.
*/
void config_sdram_emif4d5(const struct emif_regs *regs, int nr)
{
writel(0xA0, &emif_reg[nr]->emif_pwr_mgmt_ctrl);
writel(0xA0, &emif_reg[nr]->emif_pwr_mgmt_ctrl_shdw);

I'm pretty sure that has nothing to do with your current issue, but I recommend fixing it

We are running u-boot 2013.10.  I have deep sleep working for a different project using the 3.2 kernel so perhaps the 3.2 kernel is setting these registers.

If I look at u-boot 2017.01, looks like I can grab the following commit to enable that.

author Dave Gerlach <d-gerlach@ti.com> Tue Feb 18 07:31:59 2014 -0500
committer Tom Rini <trini@ti.com> Tue Mar 4 09:42:07 2014 -0500

ARM: AM43xx: EMIF: configure self-refresh entry delay

Per a suggestion from the hardware team, program the emif_pwr_mgmt_ctrl
and emif_pwr_mgmt_ctrl_shdw registers within the EMIF to hold the
desired delay in cycles that the EMIF waits without an access to enter                               
self-refresh, in this case 8192 cycles. With this, code desiring to                                  
enter self refresh only has to toggle one bit to enable it.

Signed-off-by: Dave Gerlach <d-gerlach@ti.com>

diff --git a/arch/arm/cpu/armv7/am33xx/ddr.c b/arch/arm/cpu/armv7/am33xx/ddr.c
index d05e666a742f..4173a10b143b 100644
--- a/arch/arm/cpu/armv7/am33xx/ddr.c
+++ b/arch/arm/cpu/armv7/am33xx/ddr.c
@@ -80,8 +80,8 @@ static void configure_mr(int nr, u32 cs)
  */
 void config_sdram_emif4d5(const struct emif_regs *regs, int nr)
 {
-   writel(0x0, &emif_reg[nr]->emif_pwr_mgmt_ctrl);
-   writel(0x0, &emif_reg[nr]->emif_pwr_mgmt_ctrl_shdw);
+   writel(0xA0, &emif_reg[nr]->emif_pwr_mgmt_ctrl);
+   writel(0xA0, &emif_reg[nr]->emif_pwr_mgmt_ctrl_shdw);
    writel(0x1, &emif_reg[nr]->emif_iodft_tlgc);
    writel(regs->zq_config, &emif_reg[nr]->emif_zq_config);

Brad Griffis said:

* Please see the silicon errata for more details.

DDR PHY: DDR_PHY_CTRL_1 = 0x00100007

 * Bits 9:8 (reg_phy_rd_local_odt) configured as no termination (00b)

The other message pertains to ODT.  Currently you have it disabled on the AM335x.  That's going to impact reads specifically.  Can you tell if your memory errors relate to reads or writes?  If you are getting different values returned when repeatedly reading the same location, that indicates that (at a minimum) you have a read issue.

You mentioned earlier that some of the boards were fixed through reflow.  Perhaps all the issues are manufacturing-related, or perhaps there are actually multiple issues.  It's hard to know for sure.  If you have boards that were NOT fixed by the reflow, I think it would be very interesting to see if enabling ODT resolves their issue.

Was it a conscious decision to disable ODT (i.e. for reduced power consumption) or was that just due to copying values from the AM335x EVM?  If you want to disable ODT then you likely need to do some IBIS modeling of the interface.  That will enable you to tune the ODT, drive strength, etc. for both the AM335x and the DDR.  More importantly it gives you insight into what the signal integrity could look like at the corners of process, voltage and temperature.

This setting was likely copied either from the beaglebone black or the EVM. Looking through all the RAM #defines in ddr_def.h, none of them have Bits 9:8 set for the #define *_READ_LATENCY which is used to configure DDR_PHY_CTRL_1

I will see if I can track down a board that wasn't fixed by reflow.

How would we go about doing the IBIS modeling?  Where would these tuneable settings be set in u-boot?  Any examples?

Jonathan Cormier said:

We are running u-boot 2013.10. 

I strongly urge you to update to something newer.  We've fixed more issues than I can keep track of.  One really big issue that comes to mind was resolved with this patch:

https://lists.denx.de/pipermail/u-boot/2013-November/166451.html

At a bare minimum, please be sure you've integrated that patch and are not touching the DLL_LOCK_DIFF registers.  Only the default value should be used.

Brad Griffis said:

Jonathan Cormier

We are running u-boot 2013.10. 

I strongly urge you to update to something newer.  We've fixed more issues than I can keep track of.  One really big issue that comes to mind was resolved with this patch:

https://lists.denx.de/pipermail/u-boot/2013-November/166451.html

At a bare minimum, please be sure you've integrated that patch and are not touching the DLL_LOCK_DIFF registers.  Only the default value should be used.

I've cherry picked this commit.

Jonathan Cormier said:

I've noticed our ODT register values are different, do these make sense?

SDRAM_CONFIG.REG_DDR_TERM set to RZQ/4
SDRAM_CONFIG.REG_DYN_ODT set to RZQ/2
SDRAM_CONFIG.REG_SDRAM_DRIVE set to RZQ/6

Jonathan Cormier said:

How would we go about doing the IBIS modeling?  Where would these tuneable settings be set in u-boot?  Any examples?

This is something you would do in simulation software on your computer.  I can't tell you which specific software to use or how to use the software.  That's something that perhaps someone on your hardware team would be familiar with.

The general flow is that you download an IBIS model from both TI and your memory vendor, and then you use the simulation software to get insight into what signal integrity on the bus will look like.  The real benefit is that you can model the corners of design where problems generally occur.  You want to drive the signals hard enough to avoid non-monotonic edges, but on the other hand you don't want to drive TOO hard where you induce excessive over/undershoot and ringing on the lines.

The AM335x IBIS models can be downloaded on the product folder page:

http://www.ti.com/product/AM3358/toolssoftware#simulationmodels

In general I found there was a lot of confusion about how the various options in the IBIS model itself relate back to the software.  I wrote this wiki page to help bridge the gap:

http://processors.wiki.ti.com/index.php/How_to_use_the_AM335x_IBIS_Models

The "register mapping" section tries to bridge that divide.  The registers mentioned there are structures in the board.c file:

const struct ctrl_ioregs ioregs_bonelt = {
.cm0ioctl = MT41K256M16HA125E_IOCTRL_VALUE,
.cm1ioctl = MT41K256M16HA125E_IOCTRL_VALUE,
.cm2ioctl = MT41K256M16HA125E_IOCTRL_VALUE,
.dt0ioctl = MT41K256M16HA125E_IOCTRL_VALUE,
.dt1ioctl = MT41K256M16HA125E_IOCTRL_VALUE,
};

Currently they're all being programmed with the same value (0x18B), but those values would potentially change based on the results of your IBIS modeling.  And then the final piece of the puzzle is that DSS script you ran earlier reads these registers and decodes them.  The intent was to be able to match up those outputs with what the hardware team was recommended based on the analysis.  I saw a few cases where the hardware team spent a lot of time doing the analysis, but due to miscommunications or misunderstandings the values were never properly implemented!  The script was meant to help close the loop between hardware and software.

Best regards,
Brad

Brad Griffis said:

Jonathan Cormier

I've noticed our ODT register values are different, do these make sense?

SDRAM_CONFIG.REG_DDR_TERM set to RZQ/4
SDRAM_CONFIG.REG_DYN_ODT set to RZQ/2
SDRAM_CONFIG.REG_SDRAM_DRIVE set to RZQ/6

Jonathan,

Sorry for the delay. I suggest changing DYN_ODT to RZQ/4 to match DDR_TERM.   That's sort of a blanket recommendation if you haven't done IBIS modeling to tune the parameters.

Wouldn't changing to RZQ/4 be reducing the amount of dynamic termination resistance? Why is that the blanket recommendation?