66AK2H12 DDR3 problems

The PGSR0 errors we're getting have to do with Write Leveling errors.    That is why we tried using the inverted CK.  

However, today we noticed that when we enabled the inverted CK (per above post), we cannot access the DDR3 memory space at all -- whereas we could access it when we used the non-inverted CK.

Please advice on how to properly enable the inverted CK.    Besides the PGCR0 (bits 31:26), are there any other register modifications needed to enable the inverted CK?     And, when in the DDR3 initialization procedure should this be done?   

thanks,   Scott 

Hi,

We are also having a custom board based on 66AK2H12.  During U-boot bring-up, we were stuck with DDR write leveling errors while loading with CCS.  I disabled ddr_reset_workaround() temporarily in U-Boot.  I was able to start u-boot and startup linux.  I have not enabled ddr_reset_workaround yet.  

What are all other possible reasons for PGSR0 errors?  How can we rule out  any possible issues with DDR3 interface?

Regards

Rams

Hi Scott,

Sorry that you're having trouble. Can you provide more information about your layout and setup for the memory device. I can't find the part number you mentioned on the Micron website so I will need access to the data sheet. I also need the details on your layout. Can you provide the length matching data for all the traces associated with the DDR. It would also be helpful to have the stackup for your board and layer used for each DDR signal. Are you configuring the DDR using software or using a gel? Can you provide the settings that you are using?

Regards, Bill

Bill,

I'm looking into getting the DDR3 layout level info to you.   Is there another way to send it to you, besides attaching it to this forum?

We're configuring the DDR using a Uboot routine.   For the settings, we're using the 'ddr3phy_1333_32' and 'ddr3_1333_32' structures from the Uboot repository for EVM board.    As mentioned, we set CK to inverted using the PCGR0 register, but that made things worse.

Scott

Hi Bill,

I've attached the trace length info for DDR3A interface.   The DDR3B interface is similar.    We have a 26 layer board, and are using 7 layers for the DDR3 signals (4, 6, 8, 17, 19, 21, 23).    Our GND layers are (3,5,7,9,18,20,23,24), and +1.5V DDR3 voltage is on layer 16.     Therefore we either have GND-signal-GND or +1.5V-signal-GND arrangement for the DDR3 signals.   We using 40 and 80 ohm trace impedance for the single-ended and diff-pairs, respectively.    We followed the EVM board design for Addr, Cntr, Cmd, and CK DDR3 signals termination.

I'll provide you the datasheet in another reply (not sure how to attach more than one file to a post).

Let me know if you need anything else.

Scott

Bill,    Here is the datasheet for the Micron DDR3 device we are using.    It doesn't list all of the detailed timing parameters, but refers to 4Gb device datasheet.    Note: we're using it with +1.5V Vcc, so the timing parameters can be found at the following link.

http://www.micron.com/parts/dram/ddr3-sdram/mt41j256m16ha-125?pc={338582DE-C8D6-4594-91B4-B3E86A5D5374}

Scott,

You mentioned that you used a default configuration found in UBOOT. These configurations are specific to the devices found on the EVM. Have you recalculated the register values based on the timing in the data sheet that you sent?

Regards, Bill

Bill,

Yes, we calculated the register values for our Micron device, and initialized with those instead of Uboot EVM settings.    We saw the same result -- PGSR0 errors on write leveling during init, however the later on the DDR3B memory passed our memory test.

Can you shed some light on the section 4.3.2.9 of the SPRABI1A?   This talks about the write leveling skew min/max routing requirements.    As I interpret it, we're not meeting the min skew.     Is this right, and what are the workarounds?

thanks,

Scott

To be clearer on the configuration. Here are the values we are using for the 9-9-9 setting:

static struct ddr3_phy_config ddr3phy_1333_32_999 = {
        .pllcr   = 0x0005C000ul,
        .pgcr1_mask     = (IODDRM_MASK | ZCKSEL_MASK),
        .pgcr1_val      = ((1 << 2) | (1 << 7) | (1 << 23)),
        .ptr0    = 0x42C21590ul,
        .ptr1    = 0xD05612C0ul,
        .ptr2    = 0, /* not set in gel */
        .ptr3    = 0x0B4515C2ul,
        .ptr4    = 0x0A6E08B4ul,
        .dcr_mask       = (PDQ_MASK | MPRDQ_MASK | BYTEMASK_MASK),
        .dcr_val        = ((1 << 10)),
        .dtpr0   = 0x85589955ul,  
        .dtpr1   = (9<<26)|0x028573C0ul, 
        .dtpr2   = 0x5002C200ul,
        .mr0     = 0x00001A50ul,  
        .mr1     = 0x00000006ul,
        .mr2     = 0x00000010ul,
        .dtcr    = 0x710035C7ul,
        .pgcr2   = 0x00F0B6D0ul,         
        .zq0cr1  = 0x0000005Dul,
        .zq1cr1  = 0x0000005Bul,
        .zq2cr1  = 0x0000005Bul,
        .pir_v1  = 0x00000033ul,
        .pir_v2  = 0x0000FF81ul,
        .pgcr0_mask     = 0x00000000ul, //0xFC000000ul,
        .pgcr0_val      = 0x00000000ul, //0x54000000ul, //set bits 31:26 to 010101b for CK inverted
};

static struct ddr3_emif_config ddr3_1333_32_999 = {
        .sdcfg   = 0x62009A62ul,  
        .sdtim1  = 0x125C8044ul,
        .sdtim2  = 0x00001D08ul,  
        .sdtim3  = 0x32CDFF43ul,
        .sdtim4  = 0x543F0ADFul,
        .zqcfg   = 0x70073200ul,
        .sdrfc   = 0x00001457ul,
};

With these settings UBoot can setup and run in DDR3 mostly stable and the "ddr test" command PASSes. But the Write Level error is always present, and it seems like QM DMA doesn't function properly because the Networking configurations just don't work. MAC and SGMII loopback even fail. This is only when UBoot is in DDR.

Thanks.

Scott,

What does PGSR0 look like with invert clock enabled? You will probably need a more robust stress test that pounds the DDR3 with a peripheral like EDMA to better understand the failure rate/pattern.

Hi Aditya, I'm working with Scott on this project. With the inverted clock enabled, PGSR0 reads the same value but the behavior is actually worse.

Without the inverted clock, I am able to read and write from the DDR3 using "md", "mm", or even "ddr test" commands and it is successful. When the clock is inverted any access to the DDR3 actually locks up the processor.

It's weird because I can actually get UBoot to finish booting in DDR3 90% of the time, despite the constant Write Level Error, but then things like using sspi don't work as expected. I was reading back values just fine from a slave using sspi, but writes were failing. Failing meaning that the SPI was being clocked as expected, but the data read back was always 0. Once I moved the same code to be based out of SRAM it worked just fine.

Have you seen anything like this before? Do you have any suggestions such as a DDR3 register field to play with that might address this behavior?

The way I am able to initialize DDR3 using UBoot and perform these tests on the DDR3 is to force Uboot to exist in SRAM by making the following modifications in the config file:

#define CONFIG_NR_DRAM_BANKS            0
#define CONFIG_SYS_SDRAM_BASE           0x0c200000
#define CONFIG_SYS_MALLOC_LEN           (1024 << 10)    /* 1 MiB */

This way I can have a stable UBoot and perform tests on the DDR3 memory.

Justin,

Thanks for the information. It looks like you are dealing with some marginality problem with the invert clock disabled. I will discuss internally with our experts and get back to you with suggestions. 

What do "md", "mm" and "ddr test" do?

Also, what does PGSR0 read after initialization has completed in either invert clock case?

Thanks Aditya. Those are UBoot commands that read and write memory using simple C pointers (no dma).

PGSR0 for DDR3A and DDR3B always reads back leveling errors, typically reading back 0x80C00FFF or 0x80E00FFF.

To eliminate a detail, what resistor value do you have connected to the DDR3ARZQ0-2 pins?

Bill

Bill,   We have 240 ohm, 1% tolerance resistors on each of those pins to GND.

Scott

Scott,

That's the correct value. We have seen designs with an incorrect value causing problems. One more thing to check off the list.

Bill

Just to be sure, we measured those on our board to verify that the correct resistor values were used.   They are indeed 240 ohm.

Scott

Bill,    Please let us know if there are other details you would like.   I keep fixating on that we're not meeting the min. write leveling skew requirement.   Is there a workaround for this using the inverted CK?

thanks,

Scott

Scott,

I will do a sanity check on your register configuration values and get back today. We always do that for all customers reporting initialization problems.

In the meantime, have you tried using a GEL file to perform DDR3 initialization rather than through Uboot? It is much simpler, allows direct register access for debug through CCS and will eliminate a variable in your testing.

You can modify the standard EVM GEL file that ships with the MCSDK package. Do let us know how it goes or if you have any questions.

We discovered that we needed to set the DXEN bits to '0' for all of the unused byte lanes in the DX4GCR, DX5GCR, DX6GCR, DX7GCR, DX8GCR registers.     After we doing this, we aren't getting any errors in the PGSR0 register (with our settings).    However, we're still getting strange behavior when executing out of DDR3.   E.g. when running an Ethernet task that's using DMA, it fails.    However, when running the same code when Uboot is loaded into shared memory, it works fine.  

We also tried using Aditya's settings, and we got an error in PGSR0 (0x80400FFF), and the processor immediately locked up.    Can you tell us how were these settings computed?  (i.e. can you share your spreadsheet?).    There are significant differences between them and our settings, and I couldn't reconcile the differences.

Scott

Some more questions on the DDR3 settings:

- DTPR0, DTPR1, DTPR2 registers: the user guide says to 'Configure these parameters according to the highest speed grade supported by your device data sheet'.  Our Micron devices are capable of DDR3-1600.  To clarify, even if we running at DDR3-1333 speeds or slower, do we still set these registers with the DDR3-1600 timing parameters?

Besides the DXnGCR registers, are there any other register settings where we need to disable unused data byte lanes?   

thanks,
Scott

In addition to Scott's questions above. I have some details on some of the information collected recently.

I created a gel file that matches the 1333_32 9-9-9 DDR3 initialization that I posted before, so that we can do some DSP core testing. Using this gel, I demonstrated that DDR3->SPI TX/RX DMA worked when running from the DSP corepac. Next I demonstrated that DDR3->DDR3 DMA (multiple 32 bit words) worked when running from the DSP corepac.

Given that DDR3 DMA was shown to work when run from the DSP corepac, I decided to try to setup DMA from DDR3 as a UBoot command. I modified the ddr test command to use DMA instead of pointers and proved that it worked as expected on the EVM first before moving the same code to the custom board. Once I moved to the custom board, I discovered an issue.

There was some limited capability on the custom board. I could only DMA within the same 32 bit memory address, but not outside of it. Part of the function did byte size transfers from 0x80000000 as the source pointer and incremented the destination pointer by bytes until some chosen ending address. I could DMA from 0x80000000 to 0x80000003 (transfer size being in bytes) but not to 0x80000004. Byte length DMA did not work from any source to destination. I could not DMA from 0x80000000 to 0x80000001 or anywhere else. When I did 32 bit transfers I could likewise not transfer from 0x80000000 to 0x80000004. This occurred given any base address range (not just 0x8000000X). As mentioned previously, this same code worked on the eval board.

Thanks for your attention.

We changed our settings according to the TI calculated settings spreadsheet, and made the 'DDR Output Clock Frequency' equal to 666 2/3 MHz (such that 'DDR Output Clock Period' is exactly 1.5ns).   When we ran with those settings, PGSR0 returns with 0x80C00FFF indicating a 'Write Leveling Adjustment Error' and 'DQS Gate Training Error'.

We then changed back to our original PGCR1 register setting of 0x0280C485 (as opposed to TI's setting of 0x0080C487), and with that we're no longer getting those errors in the PGSR0 register.

However, in both cases we're still getting the DMA memory errors Justin discussed in the above posting.

Please advise on any tests or setting changes you would like us to perform.

Thanks for your help,

Scott

I am attaching my ARM (uboot) DMA code and DDR3 initialization code. I misspoke in my last post when I said byte length DMA in the DDR3 memory worked in a limited way. It does not work at all in fact. As I said before, the ddr3 dma test code I am attaching "passes" when run on the evm.

Justin

Sorry for the delay in getting back due to other pressing issues. I will review your feedback/observations over the past week and get back today.

Thanks for your patience. We will be reviewing our register calc spreadsheet in case we are missing something, but that may not be until middle of next week until the spreadsheet owner is back in office.

Thank you for bringing the statement regarding DTPRx to our attention. DTPRx registers should be configured according to the intended interface frequency and not the max speed grade supported by the memory part. This update will make it into the next user guide revision.

Based on your observations using GEL v/s Uboot and EVM v/s custom board, I would like to separate the testing of robustness of their DDR3 interface from any other potential problems (Uboot or otherwise). Would it be possible to extend your DMA test to run for a few hours or overnight for verifying the robustness of the interface? Please use your own configuration settings. If we prove that it is robust, then we can focus on what is going wrong when you try to run on Uboot on the custom board.

Just to re-iterate, we will be reviewing the spreadsheet to ensure it is kosher but the above points of testing will also be useful.

Aditya, I ran the DDR3 dma test using our DDR3 settings for a couple of hours without issue (on the dsp corepac).

On another, I think very important, note - we have the DDR3A_REMAP_EN pin floating (so IPD). I had the thought to configure the EVM the same way and see what happens. The behavior matches that of our custom board. The uboot dma code I posted earlier fails, and the kernel boot process errors out when the MMU is enabled (error vector 0xFFFF000C).

I think this new information brings us a lot closer to understanding what exactly is going on here.

Justin

Justin,

Thanks for checking, so it looks like the interface is fairly robust.

Great observation on the DDR3A_REMAP_EN pin. The status of this pin is latched at boot time and needs to be either 0 or 1. I do not think it can be left floating, since it controls how the 2GB DDR address space "0x00_8000_0000 to 0x00_FFFF_FFFF" is mapped. If the pin is ‘1’, this region is mapped to the first 2GB of DDR3A which is aliased to 0x08_0000_0000 to 0x08_7FFF_FFFF. If the pin is ‘0’, this region is mapped as 2GB of DDR3B (please see Table 6-1 in the K2H data manual). If it gets remapped as DDR3B space with your test assuming DDR3A, it will probably fail.

Have you tried re-running your qual test on the custom board after configuring the pin the same way as the EVM?

DDR3A_REMAP_EN has an internal pull down which we were relying on in our design. Are you saying that we cannot rely on the IPD for this pin? Modifying the custom board to have an external PU/PD is not an option for us right now, so I cannot do the test you ask about in your last sentence.

To reiterate the test, I run a ddr3 dma program on the DSP corepac which tests from 0x60000000 to 0xC0000000 which covers both DDR3A and DDR3B due to the DSP corepac's point of view regardless of the remap pin. And trying something similar on the ARM core (uboot) is where the test immediately fails.

Ah ok. Bill is probably more knowledgable on this. I will ask him to respond.

The internal pulling resistor should be sufficient for holding the pin in a steady state if nothing else is attached to the pad. If any other connection is present than an external pulling resistor should be used. An external connection includes any short traces, test pads or vias. For I/Os that are used for configuration, we always recommend an external resistor to ensure that level of the pins is held steady. 

Is there anything else connected to the pad for GPIO16? Are you observing behavior that would be consistent with this pin latched high?

Regards, Bill

Bill,

That pin (A36) is soldered to a pad on our board.   Nothing is connected to the pad (no vias, traces, etc).

Scott

We've been able to do a memory dump from the ARM that shows the contents are the same at both 0x60000000 and 0x80000000, after a pattern has been written, which is consistent with the internal pull down functioning.

The strangeness is in the dma behavior if you read my previous posts. Could there be some mix-up in the support package when you don't remap ddr3A?

Justin

Yes, that behavior is consistent with the internal pull down functioning. Do note however, that the DMA i.e. the "SoC view" and ARM view may not be the same per Table 6-1.

If DDR3A_REMAP_EN = 0, the address space 0x00_8000_0000 to 0x00_FFFF_FFFF is viewed by ARM as 2GB of DDR3B space, but the SoC (DMA) views it as DDR3A data space.

Is this relevant to your DMA test?

Aditya, I'm not clear on what exactly SOC View really is? In the test that works on the DSP it's view is DDR3B which according to the table would mean the SOC view is DDR3A. But this doesn't seem to cause an issue for the DSP-based test. Could you elaborate on what exactly the SOC view is and how it would affect various memory accesses in this case? 

The SoC view is the address space viewed by non-DSP and non-ARM masters like EDMA.

I am not sure how you initialize/verify your data buffers before and after the DMA transfer, but an example would be if ARM initializes source/dest buffers placed starting 0x00:8000_0000 (which will be DDR3B with remap_en=0) and the DMA is programmed with those same src/dest buffer addresses which it views as DDR3A. So the ARM initializes buffers in DDR3B with different data patterns, the DMA is triggered and will transfer data to/from buffers which lie in DDR3A. ARM will then find src !=dest in DDR3B and return a fail during the data integrity check.

Aditya, can you explain how this DSP code is functional then, given the differences in memory points of view between the DSP and SoC:

CSL_Edma3Handle hModule;
CSL_Edma3ParamHandle paramHandle0;
CSL_Edma3ChannelAttr  chParam;
CSL_Edma3ChannelObj  ChObj0;
CSL_Edma3ChannelHandle  hChannel;
CSL_Edma3HwDmaChannelSetup chSetup;
CSL_Edma3ParamSetup paramSetup;
CSL_Edma3Obj  moduleObj;
CSL_Edma3CmdIntr regionIpr;
CSL_Status edma_status;

#define CSL_EDMA3_CHA_2  2
#define CSL_EDMA3_CHA_3  3
#define CSL_EDMA3_CHA_6  6
#define CSL_EDMA3_CHA_10  10
#define CSL_EDMA3_CHA_11  11

#define NWORDS 1
#define EDMA_WORD_SIZE (32*NWORDS)
#define EDMA_DMA_ACNT (EDMA_WORD_SIZE/0x8)

#define EDMA_TX_BUF				0x80000000
#define EDMA_RX_BUF				0x80000004

#define MEM_TOP					0xC0000000

#define TEST_SIZE	((MEM_TOP-EDMA_RX_BUF)>>2)

#define TEST_WORD	0x65B69152

volatile Uint32 * srcBuf = (volatile Uint32 *)EDMA_TX_BUF;
volatile Uint32 * dstBuf = (volatile Uint32 *)EDMA_RX_BUF;

void sleep(Int32 cycles)
{
	int i=0;
	for(i=0; i<cycles; i++) {
		asm("NOP");
	}
}

void Enable_Edma_Channels(void)
{
	// Enable channel
	CSL_edma3HwChannelControl(hChannel,CSL_EDMA3_CMD_CHANNEL_SET,NULL);
}

int GetEdmaChannelNumber(unsigned int index, Int *channel)
{
	if(index == 0) {
		*channel = CSL_EDMA3_CHA_6;
	}
	else {
		return -1;
	}

	return 0;
}

int SetupEdmaInit (unsigned int index, Int channel_num)
{
	CSL_InstNum edmaInstNum;

	if(index == 0) {
		edmaInstNum = CSL_EDMACC_1;
	}
	else if(index == 1) {
		edmaInstNum = CSL_EDMACC_2;
	}
	else if(index == 2) {
		edmaInstNum = CSL_EDMACC_3;
	}
	else {
		return -1;
	}

	// EDMA Module Initialization
	CSL_edma3Init(NULL);

	// EDMA Module Open
	hModule = CSL_edma3Open(&moduleObj,edmaInstNum,NULL,&edma_status);

	// Channel Open
	chParam.regionNum  = CSL_EDMA3_REGION_GLOBAL;
	chSetup.que        = CSL_EDMA3_QUE_3;
	chParam.chaNum     = channel_num;

	hChannel = CSL_edma3ChannelOpen(&ChObj0, edmaInstNum, &chParam, &edma_status);
	chSetup.paramNum   = chParam.chaNum;
	CSL_edma3HwChannelSetupParam(hChannel,chSetup.paramNum);

	//Enable_Edma_Channels();

	return 0;
}


void Setup_Edma_Params (Uint32 srcBuf,Uint32 dstBuf, Int channel_num, Uint32 b_len)
{
	// Parameter Handle Open
	// Open all the handles and keep them ready
	paramHandle0            = CSL_edma3GetParamHandle(hChannel,channel_num,&edma_status);
	//paramHandle1            = CSL_edma3GetParamHandle(hChannel_rx,channel_revt,&edma_status);

	paramSetup.aCntbCnt     = CSL_EDMA3_CNT_MAKE(EDMA_DMA_ACNT,(b_len));
	paramSetup.srcDstBidx   = CSL_EDMA3_BIDX_MAKE(EDMA_DMA_ACNT,0 );
	paramSetup.srcDstCidx   = CSL_EDMA3_CIDX_MAKE(0,0);
	paramSetup.cCnt         = 1;
	paramSetup.option       = CSL_EDMA3_OPT_MAKE(FALSE,FALSE,FALSE,TRUE,channel_num,CSL_EDMA3_TCC_NORMAL, \
			CSL_EDMA3_FIFOWIDTH_NONE,CSL_EDMA3_STATIC_EN,CSL_EDMA3_SYNC_A,CSL_EDMA3_ADDRMODE_INCR,CSL_EDMA3_ADDRMODE_INCR);

	paramSetup.srcAddr      = (Uint32)(srcBuf);
	paramSetup.dstAddr      = (Uint32)(dstBuf);

	paramSetup.linkBcntrld  = 0xFFFF;

	CSL_edma3ParamSetup(paramHandle0,&paramSetup);

	Enable_Edma_Channels();
}

void DMA_Request(Int channel_num)
{
	unsigned int mask_tx = (1<<channel_num);

	/* Interrupt enable (Bits 0-1)  for the global region interrupts */
	regionIpr.region = CSL_EDMA3_REGION_GLOBAL;
	regionIpr.intr   = mask_tx;
	regionIpr.intrh  = 0x0000;
	CSL_edma3HwControl(hModule,CSL_EDMA3_CMD_INTR_ENABLE,&regionIpr);
}

void Test_Edma(Int channel_num)
{
	unsigned int mask_tx = (1<<channel_num);

	// Wait for interrupt
	regionIpr.region  = CSL_EDMA3_REGION_GLOBAL;
	regionIpr.intr    = 0x0;
	regionIpr.intrh   = 0;
	do{
		CSL_edma3GetHwStatus(hModule,CSL_EDMA3_QUERY_INTRPEND,&regionIpr);
	}while ((regionIpr.intr & mask_tx) != mask_tx);

	/* Clear pending interrupt */
	CSL_edma3HwControl(hModule,CSL_EDMA3_CMD_INTRPEND_CLEAR, &regionIpr);

	CSL_edma3GetHwStatus(hModule,CSL_EDMA3_QUERY_INTRPEND,&regionIpr);
}

void Close_Edma()
{
	CSL_edma3HwChannelControl(hChannel, CSL_EDMA3_CMD_CHANNEL_CLEAR, NULL);
	CSL_edma3HwChannelControl(hChannel, CSL_EDMA3_CMD_CHANNEL_DISABLE, NULL);

	// clear Event Missed Register
	hModule->regs->TPCC_EMCR  = 0xFFFF;
	hModule->regs->TPCC_EMCRH = 0xFFFF;

	// clear Secondary Event Register
	hModule->regs->TPCC_SECR  = 0xFFFF;
	hModule->regs->TPCC_SECRH = 0xFFFF;

	CSL_edma3ChannelClose(hChannel);

	CSL_edma3GetHwStatus(hModule,CSL_EDMA3_QUERY_INTRPEND,&regionIpr);

	CSL_edma3Close(hModule);

	hModule = 0;
	hChannel = 0;
	paramHandle0 = 0;
}

int main (void)
{
	Uint32 length;
	Uint32 index = 0;
	Int channel; // channel number
	Uint32 test=1;

	unsigned long long number_test = 0;

	length = 1;
	index = 0;

	while(test) {
		number_test += 1;
		printf("Test %llu START...\n",number_test);

		srcBuf = (volatile Uint32 *)EDMA_TX_BUF;
		dstBuf = (volatile Uint32 *)EDMA_RX_BUF;

		*(volatile Uint32*)srcBuf = (TEST_WORD>>1);

		while((Uint32)dstBuf < MEM_TOP) {

			printf("0x%08X --> 0x%08X\n",srcBuf,dstBuf);

			GetEdmaChannelNumber(index, &channel);

			//Initialize EDMA for transfer
			SetupEdmaInit(index, channel);

			//Enable EDMA for transfer
			Setup_Edma_Params((Uint32)(srcBuf),(Uint32)(dstBuf), channel, length);

			// Enable DMA Request
			DMA_Request(channel);

			//Check EDMA transfer completion status
			Test_Edma(channel);

			//Close EDMA channels/module
			Close_Edma();

			if(*(volatile Uint32*)(srcBuf) != *(volatile Uint32*)(dstBuf)) {
				printf("Test failed at 0x%08X\n",dstBuf);
				test = 0;
				break;
			}

			dstBuf = (volatile Uint32 *)((Uint32)dstBuf+0x0014BCD0);
		}

		if(!test) {
			break;
		}

		sleep(500000);

		srcBuf = (volatile Uint32 *)EDMA_TX_BUF;
		dstBuf = (volatile Uint32 *)EDMA_RX_BUF;

		*(volatile Uint32*)srcBuf = TEST_WORD;

		while((Uint32)dstBuf < MEM_TOP) {

			printf("0x%08X --> 0x%08X\n",srcBuf,dstBuf);

			GetEdmaChannelNumber(index, &channel);

			//Initialize EDMA for transfer
			SetupEdmaInit(index, channel);

			//Enable EDMA for transfer
			Setup_Edma_Params((Uint32)(srcBuf),(Uint32)(dstBuf), channel, length);

			// Enable DMA Request
			DMA_Request(channel);

			//Check EDMA transfer completion status
			Test_Edma(channel);

			//Close EDMA channels/module
			Close_Edma();

			if(*(volatile Uint32*)(srcBuf) != *(volatile Uint32*)(dstBuf)) {
				printf("Test failed at 0x%08X\n",dstBuf);
				test = 0;
				break;
			}

			dstBuf = (volatile Uint32 *)((Uint32)dstBuf+0x0014BCD0);
		}

		if(test) {
			printf("Test %llu passed, restarting...\n",number_test);
			sleep(500000);
		}
	}

	return 0;
}

My ARM DDR dma test code had previously been using QDMA to perform the test. This test passes when src and dst were from 0x60000000 to 0x7FFFFFFC but failed when up in 0x80000000 to 0xBFFFFFFC. Remember ddr3A is not remapped so to the ARM these were both DDR3B but to the SOC they are different.

I modified the code to match what the DSP test did by using EDMA and now both sections work. There must be some underlying difference in how QDMA and EDMA interpret which DDR bank to use/which memory point of view to use.

This is where we are at now - I cannot boot Linux. As soon as it tries to enable MMU it aborts (0xFFFF000C). Any knowledge on what would cause this problem?

Besides the triggering mechanism, there should not be any difference in the way QDMA v/s EDMA access DDR3 address space as long as you use the same src/dst addresses. Both sections should work. When you say "failed when up in 0x80000000 to 0xBFFFFFFC" - what is the failure signature? No transfer? Partial transfer?

Anyway, looks like both DDR3A and DDR3B are working robustly for you now (using DSP and EDMA). So your issue is now isolated to Linux boot failure. I will get an ARM/Linux expert to address this.

Justin,

What is your Hawking silicon revision (1.0/1.1 ?)

Thanks Aditya, I created a new post here: http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/344869.aspx

Please redirect the conversation to the new thread.

Justin,

Would you mind marking this thread as verified? Feel free to post an update if you have any questions pertaining to this thread.

So the issue I was seeing with the DDR3 DMA test code was rectified by doing a hardware modification to the board and pulling up DDR3A_REMAP_EN.  In fact, it seems like a lot of the issues I was having bringing up my custom board stem from the issue of having DDR3A_REMAP_EN floating (pulled down internally).

You can reproduce this on the EVM in house, and I request that you do so. Create a new bootmode in the BMC by issuing the command "bootmode 8 0 012005 ARM_B_SPI". Boot the stock u-boot. Try to use the ethernet ports and let me know if they work or not. Also try using a QDMA from DDR3 in uboot (I posted the code a while back). Both of those tests fails for me on my v1.1 EVM and on my silicon 1.1 custom board.

I think this needs to be really highlighted.. could avoid a lot of headaches. Please update me on what your evm test reveals.

Justin

Quick question on a couple of register settings (PGCR1 and DCR):

What is the end value that is programmed into the registers: is it just the "_val" or a masked version of the _val?

.pgcr1_mask     = (IODDRM_MASK | ZCKSEL_MASK),
.pgcr1_val      = ((1 << 2) | (1 << 7) | (1 << 23)),

.dcr_mask       = (PDQ_MASK | MPRDQ_MASK | BYTEMASK_MASK),
.dcr_val        = ((1 << 10)),

Here's a snippet of the code:

tmp =  __raw_readl(base + KS2_DDRPHY_DCR_OFFSET);
tmp &= ~(phy_cfg->dcr_mask);
tmp |= phy_cfg->dcr_val;
__raw_writel(tmp, base + KS2_DDRPHY_DCR_OFFSET);

Can be found here: https://git.ti.com/keystone-linux/u-boot/blobs/master/arch/arm/cpu/armv7/keystone/ddr3.c

Aditya,

Your spreadsheet is out of date with our latest settings.  I attached our latest register settings (column B), along with some updates to the 'TI calc values' (column C), from the TI 'K2 DDR3 Register Calc v1p40.xlxs' spreadsheet.    One difference in the 'TI Calc values', is that we used a 'DDR3 Output Clock Frequency' of 666 2/3 MHz, instead of 666.67MHz.   This changed quite a few timing register values in the TI spreadsheet, due to 'ceiling' or 'round' function used to computed the register values.

Now there are only two registers where we don't have agreement:   PGCR0 and PGCR1.    We're using the default value for PGCR0.   For PGCR1, we tried using the TI Calc value (0x0080C487), however, the PGSR0 returns with 0x80C00FFF indicating a 'Write Leveling Adjustment Error' and 'DQS Gate Training Error'.   When we used our value (0x0280C485), the PGSR0 returns with no errors.

Please let us know the correct values to use for PGCR0 and PGCR1.

thanks,

Scott