DDR3 SODIMM

I have the invert clock enabled in the excel spreadsheet, maybe the solution is to disable it since I have the long SODIMM fly-by route delays.

Colin Hankins

I have the invert clock enabled in the excel spreadsheet, maybe the solution is to disable it since I have the long SODIMM fly-by route delays.

I switched to non-inverted clock and all but one of the WRLVL_INIT_RATIO values became 0x00, is that correct? 

Either way, this did not work.

Colin Hankins

I switched to non-inverted clock and all but one of the WRLVL_INIT_RATIO values became 0x00, is that correct? 

Either way, this did not work.

This method so far produces the best results:

My test app, after initializing the hardware, begins by writing the following pattern into the DDR3

Addr 0x0 : 12345678
Addr 0x1 : 87654321
Addr 0x2 : A5A5A5A5
Addr 0x3 : 11111111
Addr 0x4 : C0FFEE00
Addr 0x5 : CAB1CAB2
Addr 0x6 : DEADBEEF
Addr 0x7 : FFFFFFFF

The program then sits in a loop and continuously reads those DDR3 locations and writes the values into L2 SRAM. Every now and then I read back the correct data pattern. Which must mean the initial write worked. However, more often than not, I read back data with a few of the byte lanes having bad data:

Addr 0x0 : 123456FF
Addr 0x1 : CF654321
Addr 0x2 : A5A5A500
Addr 0x3 : 10111111
Addr 0x4 : C0FFEEEF
Addr 0x5 : EFB1CAB2
Addr 0x6 : DEADBE00
Addr 0x7 : 28FFFFFF

In the pattern above byte lanes 0 and 7 were consistently wrong. But sometimes I read back the data and it is all wrong:

Addr 0x0 : ff7fafff
Addr 0x1 : cffdefff
Addr 0x2 : 1800
Addr 0x3 : 10000010
Addr 0x4 : ffffdfef
Addr 0x5 : efffebff
Addr 0x6 : 0
Addr 0x7 : 28008700

I have full automatic leveling enabled, incremental read DQS gate training enabled and incremental read data eye training enabled. I feel like the issue is sometimes the read leveling achieves a successful lock, and then sometimes not at all or misses on a few byte lanes? Any assistance on this would be much appreciated.

I have worked out some bugs and I now have the DDR3 interface using a non-inverted clock fairly stable. One remaining issue with the reads is a few of the byte lanes still glitch every once in a while. For example, I write this data pattern to the DDR3 and then I sit in a loop and read it out. More often than not it is correct but too frequently  the same byte lanes glitch. The first set of data is the proper pattern. The second set is the glitch and it pretty much always glitches in the same way.

Good data pattern

Addr 0x0 : 12345678
Addr 0x1 : 87654321
Addr 0x2 : a5a5a5a5
Addr 0x3 : cab1cab2
Addr 0x4 : ffffffff
Addr 0x5 : ffffffff
Addr 0x6 : 0
Addr 0x7 : 0

Bad data pattern
Addr 0x0 : 12345678
Addr 0x1 : 87654321
Addr 0x2 : a5a5a5a5
Addr 0x3 : cab111b2
Addr 0x4 : c0ffffff
Addr 0x5 : ffffcaff
Addr 0x6 : de000000
Addr 0x7 : ff00

It is just byte lanes 3, 5, and 7. 

I see the same results with either partial automatic leveling or full automatic leveling with incremental read gate and read eye training modes enabled.

Any ideas?

All of the WRLVL_INIT_RATIO values are already 0x00 after moving to the non-inverted clock. I went ahead and zeroed out the GTLVL_INIT_RATIO values to see if it would have an impact... Nothing changed. The interface works as well as before, minor glitch and all. Does this mean automatic leveling is not being enabled? I used the emif4 mcsdk code and followed the DDR3 init guidelines. Also, my DDR STATUS register reads 0x40000004.

here is the relevant code:

/**************** 3.3 Leveling register configuration ********************/
DDR3_CONFIG_REG_0 &= ~(0x007FE000); // clear ctrl_slave_ratio field
DDR3_CONFIG_REG_0 |= 0x00100000; // set ctrl_slave_ratio to 0x80
DDR3_CONFIG_REG_12 &= ~(0x08000000); // Set invert_clkout = 0
//DDR3_CONFIG_REG_0 |= 0x00200000; // set ctrl_slave_ratio to 0x100
//DDR3_CONFIG_REG_12 |= 0x08000000; // Set invert_clkout = 1

DDR3_CONFIG_REG_0 |= 0xF; // set dll_lock_diff to 15

DDR3_CONFIG_REG_23 |= 0x00000200; // See section 4.2.1, set for partial automatic levelling

/**************** 3.3 Partial Automatic Leveling ********************/
DATA0_WRLVL_INIT_RATIO = 0x00;
DATA1_WRLVL_INIT_RATIO = 0x00;
DATA2_WRLVL_INIT_RATIO = 0x00;
DATA3_WRLVL_INIT_RATIO = 0x00;
DATA4_WRLVL_INIT_RATIO = 0x00;
DATA5_WRLVL_INIT_RATIO = 0x00;
DATA6_WRLVL_INIT_RATIO = 0x00;
DATA7_WRLVL_INIT_RATIO = 0x00;
DATA8_WRLVL_INIT_RATIO = 0x00;

DATA0_GTLVL_INIT_RATIO = 0x00;
DATA1_GTLVL_INIT_RATIO = 0x00;
DATA2_GTLVL_INIT_RATIO = 0x00;
DATA3_GTLVL_INIT_RATIO = 0x00;
DATA4_GTLVL_INIT_RATIO = 0x00;
DATA5_GTLVL_INIT_RATIO = 0x00;
DATA6_GTLVL_INIT_RATIO = 0x00;
DATA7_GTLVL_INIT_RATIO = 0x00;
DATA8_GTLVL_INIT_RATIO = 0x85;

//Do a PHY reset. Toggle DDR_PHY_CTRL_1 bit 15 0->1->0
DDR_DDRPHYC &= ~(0x00008000);
DDR_DDRPHYC |= (0x00008000);
DDR_DDRPHYC &= ~(0x00008000);

/***************** 3.4 Basic Controller and DRAM configuration ************/

//DDR_SDRFC = 0x00004F59; // enable configuration
DDR_SDRFC = 0x00005162; // enable configuration

/* DDR_SDTIM1 = 0x1113783C; */
TEMP = 0;
TEMP |= 0x8 << 25; // T_RP bit field 28:25
TEMP |= 0x8 << 21; // T_RCD bit field 24:21
TEMP |= 0x9 << 17; // T_WR bit field 20:17
TEMP |= 0x17 << 12; // T_RAS bit field 16:12
TEMP |= 0x20 << 6; // T_RC bit field 11:6
TEMP |= 0x4 << 3; // T_RRD bit field 5:3
TEMP |= 0x4; // T_WTR bit field 2:0
DDR_SDTIM1 = TEMP;

/* DDR_SDTIM2 = 0x30717FE3; */
TEMP = 0;
TEMP |= 0x3 << 28; // T_XP bit field 30:28
TEMP |= 0x6E< 16; // T_XSNR bit field 24:16
TEMP |= 0x1ff << 6; // T_XSRD bit field 15:6
TEMP |= 0x4 << 3; // T_RTP bit field 5:3
TEMP |= 0x3; // T_CKE bit field 2:0
DDR_SDTIM2 = TEMP;

/* DDR_SDTIM3 = 0x559F86AF; */
TEMP = 0;
TEMP |= 0x5 << 28; // T_PDLL_UL bit field 31:28 (fixed value)
TEMP |= 0x5 << 24; // T_CSTA bit field 27:24 (fixed value)
TEMP |= 0x4 << 21; // T_CKESR bit field 23:21
TEMP |= 0x3F << 15; // T_ZQCS bit field 20:15
TEMP |= 0x67 << 4; // T_RFC bit field 12:4
TEMP |= 0xF; // T_RAS_MAX bit field 3:0 (fixed value)
DDR_SDTIM3 = TEMP;

DDR_DDRPHYC = 0x0010000F;

DDR_ZQCFG = 0x70073214;

DDR_PMCTL = 0x0;

//DDR_SDRFC = 0x00004F59; // enable configuration
DDR_SDRFC = 0x00005162; // enable configuration

/* DDR_SDCFG = 0x63062A32; */
/* New value with DYN_ODT disabled and SDRAM_DRIVE = RZQ/7 //0x63222A32; // last config write DRAM init occurs */
TEMP = 0;
TEMP |= 0x3 << 29; // SDRAM_TYPE bit field 31:29 (fixed value)
TEMP |= 0x0 << 27; // IBANK_POS bit field 28:27
TEMP |= 0x3 << 24; // DDR_TERM bit field 26:24
TEMP |= 0x1 << 21; // DYN_ODT bit field 22:21
TEMP |= 0x1 << 18; // SDRAM_DRIVE bit field 19:18
TEMP |= 0x2 << 16; // CWL bit field 17:16
TEMP |= 0x0 << 14; // NM bit field 15:14
TEMP |= 0xA << 10; // CL bit field 13:10
TEMP |= 0x6 << 7; // ROWSIZE bit field 9:7
TEMP |= 0x3 << 4; // IBANK bit field 6:4
TEMP |= 0x0 << 3; // EBANK bit field 3:3
TEMP |= 0x2; // PAGESIZE bit field 2:0
DDR_SDCFG = TEMP;

/* assuming max device speed, 1.4GHz, 1 cycle = 0.714 ns *
* so, 100 us = 100000 ns = 140056 cycles
thereby, 600us=840336 */
ddr3_wait(840336); //Wait 600us for HW init to complete

DDR_SDRFC = 0x00013C0; //Refresh rate = (7.8*666MHz]

/***************** 4.2.1 Partial automatic leveling ************/
DDR_RDWR_LVL_RMP_CTRL = 0x80000000; //enable automatic leveling

/*Trigger automatic leveling - This ignores read DQS leveling result and uses ratio forced value
Wait for min 1048576 DDR clock cycles for leveling to complete = 1048576 * 1.5ns = 1572864ns = 1.57ms.
Actual time = ~10-15 ms */
DDR_RDWR_LVL_CTRL = 0x80000000;

/* assuming max device speed, 1.4GHz, 1 cycle = 0.714 ns *
* so, 100 us = 100000 ns = 140056 cycles
thereby, 3ms=3000us=4201680 */
ddr3_wait(0x7270E0); //Wait 3ms for leveling to complete

Hello--can someone from TI please help here!!

I was able to resolve this problem. 

However, I would still like further documentation or assistance on the time delta for the various registers. I see in the DDR3 Init Guide there is some discussion about the WRLVL_INIT_RATIO and GATELVL_INIT_RATIO:

The result values are the initial values in units of DLL taps of which
there are 256 per clock period. Additionally, since the initial leveling algorithm only
adapts in the positive direction, the initial values are offset 128 DLL steps in the
negative direction.

So the time delta on those values is CLK_PER/256. What about the other DDR3 Phy configuration registers?

Hi Colin,

How did you end up resolving your issue?

I'll see if I can dig up what the values in the other PHY registers correspond to in terms of time delays.

Keegan@TI

Hi Colin,

How did you end up resolving your issue?

Well... I don't think what I have done, via software config, is necessarily what TI would recommend. But I could not find anything else that TI recommended that would work! 

I use a modified hwEmif4p0Enable(). Modified in that it uses our SDTIM and other configuration values as well as a non-inverted clock setup. I left partial automatic leveling as the configuration so it also works with our Rev 1 silicon and I do not have incremental leveling enabled. If I run this function once, the reads are always glitchy, as described in my posts above. The writes are solid, I wrote a function to read the DDR3 locations many millions  of times and the number of non-glitchy reads far exceeded the amount of glitchy reads. Indicating this is a deterministic process and the write was successful. So, write leveling is fine, the issue is with the read leveling on the C6678. 

The only way I can get it to work is to do run the hwEmif4p0Enable() function and then perform a number of writes and reads to the DDR3, and then run hwEmif4p0Enable() a second time. After which the interface works perfectly. I have left it alone for hours (doing nothing but refreshes) and returned to a still functioning DDR3 interface. Also, after the first call to hwEmif4p0Enable() if I do not do the writes and reads to the DDR3 and then make the second call to hwEmif4p0Enable(), the reads are still glitchy. I have also tried inserting delays between the to successive calls to hwEmif4p0Enable() but that also leaves the reads in a glitchy state. I do not know what is so magical about doing writes and reads to the DDR3 that makes the second call to hwEmif4p0Enable() correct the read leveling.

It appears I am either doing something incorrectly to configure the read leveling interface, or something else I am not aware of. I don't know if my description gives you any clues but any help is appreciated. Because it looks like there is no issue with board or with the writes, but something is off on the reads. I would like to know what because I hate to have to configure the interface in this manner as it seems a little 'voodoo'.

Maybe declaring a memory address as volatile is not enough to overcome caching? And the cache mechanism only decides to cache a region and value when the DDR3 memory is reinitialized?

Colin Hankins

Maybe declaring a memory address as volatile is not enough to overcome caching? And the cache mechanism only decides to cache a region and value when the DDR3 memory is reinitialized?

I disabled the L1D cache and set the DDR3 MAR, corresponding the address ranges I am using, to 0x0.

I still see the same behavior as I described above. However, if I write to these addresses with values other than what I wrote in between the hwEmif4p0Enable() calls, the read back is glitchy! If I write back to these addresses with the values I used in the writes in between the  hwEmif4p0Enable() calls, the read back works! 

Colin,

Can you confirm that you designed the board to meet all of the rules in our design guidelines document, SPRABI1, DDR3 Design Requirements for KeyStone Devices?

If you did conform to those guidelines, then it should work. The other things to look at are proper voltages, termination resistors, board noise, etc.

Regards,
RandyP

It follows those guidelines except half of the byte lanes are referenced to the 1.5V power plane. However there is no correlation between those lanes and the read glitching. 

The DSP and SODIMM byte lanes are reversed:

DSP    |    SODIMM

B0       ->      B7

B1       ->      B6

 ...

B7       ->      B0

So that means in the fly-by-topology, the DSP DDR3 controller B0 is the longest delay and B7 the shortest. Is that an issue?

Reading through the requirements guide again, I noticed this note:

Note—IMPORTANT: The TI DSP DDR3 controller cannot be held in reset for more
than 1 hour during the initial power-up. Also, the TI DSP DDR3 controller cannot be
held in reset for more than 5% of its total power-on hours.

What is the impact of having the DDR3 controller in reset for more than 5% of its total power-on hours in the early period of its life?

Also, what is the termination value used on the DSP controller during reads when the DDR_PHY_CTRL1 register bit 8 is set?  

Please provide the following information:

• Completed PHY CALC spreadsheet
• Completed REG CALC spreadsheet
• Manufacturer's part number for the DDR3 DRAM devices
• DDR reference clock rate and DDR PLL settings

HI Tom,

Thanks for your response. We are working with the customer to take this conversation onto the internal forum so they can talk more candidly without divulging their IP.

Take care,

Amy