Values stored to DDR3 unreliable

When I launch the target configuration I get,

C66xx_0: GEL Output: StartUp: ...
C66xx_0: GEL Output: Setup_Memory_Map...
C66xx_0: GEL Output: Setup_Memory_Map... Done.
C66xx_0: GEL Output: StartUp: ... Done

and when I connect to the DSP I get

C66xx_0: GEL Output: OnTargetConnect: ...
C66xx_0: GEL Output: OnReset: ...
C66xx_0: GEL Output: PllcInit: ...
C66xx_0: GEL Output: PLL1 Setup...
C66xx_0: GEL Output: PLL1 Setup for DSP @ 1200.0 MHz.
C66xx_0: GEL Output: SYSCLK2 = 400.0 MHz, SYSCLK5 = 240.0 MHz.
C66xx_0: GEL Output: SYSCLK8 = 18.75 MHz.
C66xx_0: GEL Output: PLL1 Setup... Done.
C66xx_0: GEL Output: PllcInit: Done
C66xx_0: GEL Output: Ddr3Init: ...
C66xx_0: GEL Output: DSP 1
C66xx_0: GEL Output: 800MHz Settings
C66xx_0: GEL Output: TMS320C6670 or TMS320C6678
C66xx_0: GEL Output: Ddr3 Setup for 64 bits DDR @ 1600.0 MHz...
C66xx_0: GEL Output: DDR3 refresh rate set to 31.25us...
C66xx_0: GEL Output: SDRAM configured...
C66xx_0: GEL Output: Leveling complete...
C66xx_0: GEL Output: DDR3 refresh rate set to 7.8us...
C66xx_0: GEL Output: Leveling completed successfully...
C66xx_0: GEL Output: DDR3_CONFIG_REG_0 0x7FA0047F...
C66xx_0: GEL Output: DDR_DMCSTAT 0x40000004...
C66xx_0: GEL Output: Ddr3 Setup... Done.
C66xx_0: GEL Output: Ddr3Init: ... Done
C66xx_0: GEL Output: OnReset: ... Done
C66xx_0: GEL Output: OnTargetConnect: ... Done

Hi,

Could you elaborate, are you using some SW release (which one?) or is this just a GEL file initialization of the DDR?

Best Regards,
Yordan

Thanks for your reply. I am afraid I don't quite understand your question.

Our HW came with a .gel file written by the manufacturers, CommAgility. The contents are VERY large, here is a section

"/ DDR3 tuning registers
#define DATA0_GTLVL_INIT_RATIO (*(unsigned int*)(CHIP_LEVEL_REG + 0x043C))
#define DATA1_GTLVL_INIT_RATIO (*(unsigned int*)(CHIP_LEVEL_REG + 0x0440))
#define DATA2_GTLVL_INIT_RATIO (*(unsigned int*)(CHIP_LEVEL_REG + 0x0444))
#define DATA3_GTLVL_INIT_RATIO (*(unsigned int*)(CHIP_LEVEL_REG + 0x0448))
#define DATA4_GTLVL_INIT_RATIO (*(unsigned int*)(CHIP_LEVEL_REG + 0x044C))
#define DATA5_GTLVL_INIT_RATIO (*(unsigned int*)(CHIP_LEVEL_REG + 0x0450))
#define DATA6_GTLVL_INIT_RATIO (*(unsigned int*)(CHIP_LEVEL_REG + 0x0454))
#define DATA7_GTLVL_INIT_RATIO (*(unsigned int*)(CHIP_LEVEL_REG + 0x0458))
#define DATA8_GTLVL_INIT_RATIO (*(unsigned int*)(CHIP_LEVEL_REG + 0x045C))

#define RDWR_INIT_RATIO_0 (*(unsigned int*)(CHIP_LEVEL_REG + 0x040C))
#define RDWR_INIT_RATIO_1 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0410))
#define RDWR_INIT_RATIO_2 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0414))
#define RDWR_INIT_RATIO_3 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0418))
#define RDWR_INIT_RATIO_4 (*(unsigned int*)(CHIP_LEVEL_REG + 0x041C))
#define RDWR_INIT_RATIO_5 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0420))
#define RDWR_INIT_RATIO_6 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0424))
#define RDWR_INIT_RATIO_7 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0428))
#define RDWR_INIT_RATIO_8 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0430))

#define DDR3_CONFIG_REG_0 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0404))
#define DDR3_CONFIG_REG_12 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0434))
#define DDR3_CONFIG_REG_23 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0460))
#define DDR3_CONFIG_REG_24 (*(unsigned int*)(CHIP_LEVEL_REG + 0x0464))"

I thought from the debug output in my previous post that it might be possible to gauge enough information.

Could you help me more generally, though too - is it likely that the problem I am seeing is to do with DDR3 initialization? Should I tread values in DDR3 differently, e.g. do I need to use memcpy, rather than writing to elements using an iterator?

Thanks,

David

p.s. for example the debug output from the .gel says

"C66xx_0: GEL Output: Leveling completed successfully..."

so I wondered if the leveling is OK, and the issue I am seeing is something different.

I think that the DDR is configured correctly if you use TI standard gel file and you run on a standard EVM.

I have some suggestion how to debug the issue:
1. First disable the caches (If you do not know how to do it search on cache in e2e and you will find many posting about cache disable and enable) and make sure that your problem is not cache issue
2. Next I would change the array dimension. Currently you use 2D array where the first size is 1. I understand you do it because in the future you may use a larger size, but for now, for debug purposes just make a vector instead of array. It really does not matter but it makes the map file clearer.
3. Next look at the map file that is generated and write down the base address of the two arrays (vectors really) that are used, and then, after selecting a core, use CCS memory browser to modify these values. See if you can read and write to the DDR location. If you can, the problem is not DDR problem.

Please report your finding so we can continue from there

Ran

Thanks, I will try these.

One question - despite the names, these values aren't shared between cores. Is caching still a potential issue?

I do not see your complete code so I do not know exactly how does the code insert values to the memory. It is a good practice every time you have a memory issue to disable the cache and see if what is the result. The default behavior of L1D cache is different than the behavior of L2 Cache in terms of write through (when the code write to address that is not on the cache, does the value is changed in the cache or in the memory or both.

Let's try and see

Ran

Thanks, I am searching around the forum looking at Caching. Is it sufficient to disable Caching module in .cfg?

I usually set the size to 0 and or use the MAR registers to disable cache, but the cfg or the platform definition is good as well


Ran

Hi Ran,

I simplified x_ll_shared so it is a 1D vector (and I made x_ll a normal global, as it is much smaller, so I don't need to worry about this).

In the .map file I have

80258000 80258000 0014dffc 00000000 rw-
80258000 80258000 0010e000 00000000 rw- .x_ll_shared
80366000 80366000 0003fffc 00000000 rw- .UserLogCanc
803a6000 803a6000 0003fffc 00000000 rw-
803a6000 803a6000 0003fffc 00000000 rw- .UserLogCanc2

Where UserLogCanc(2) are where I store some output debug. These arrays don't seem to have been problematic.

I used the Memory Browser to go to "80258000" or "x_ll_shared_data", and there are values here. "L1D Cache" is ticked, as is "L2 Cache". If I untick either of them, the values all turn to "??". This seems to me like Caching is still enabled.

With both cache tick boxes ticked, If I highlight and change the values manually, to say "1234...." and press "Enter" the values return to what they were before.

Thanks!

I can see in the Platform.xdc

l2Mode:"0k",
l1PMode:"32k",
l1DMode:"32k",

but I am not clear how to change this, Tools>RTSC Tools>Platform seemed the way to go, but I can't work out how to get it to open our platform to edit.

I added these lines

"var ti_sysbios_family_c66_Cache = xdc.useModule('ti.sysbios.family.c66.Cache');
ti_sysbios_family_c66_Cache.initSize.l1dSize = ti_sysbios_family_c66_Cache.L1Size_0K;
ti_sysbios_family_c66_Cache.initSize.l1pSize = ti_sysbios_family_c66_Cache.L1Size_0K;
ti_sysbios_family_c66_Cache.initSize.l2Size = ti_sysbios_family_c66_Cache.L2Size_0K;"

to the .cfg file. Is that sufficient? It gives a warning

"Description Resource Path Location Type
Cache settings were changed in user configuration. User configuration options will override platform settings. Check your memory map to make sure that Cache does not conflict with your L1/L2 memory placement. To avoid conflicts between L1/L2 memory and cache, we recommended specifying cache sizes along with memory sizes in a platform package. UserProcessProject.cfg /ICSCPRI ti.sysbios.family.c66.Cache XDCTools Configuration Marker"

Thanks!

If I paused core, then I can adjust the values in the memory browser of x_ll_shared. Is that conclusive?

Sometimes for the first ~700 values in x_ll_shared, some of the values are OK, and some are hugely negative, after that the values are 0s. Sometimes the values for the first ~1000 samples are fine, and after that they are sometimes OK and sometimes hugely negative.

I should also comment that if I put the values in MSMCSRAM, then there is no issue at all, but unfortunately it is not a big enough memory going forwards and causes other problems!

Thanks, I am afraid it is a very complicated project.

CORE_0 CPRI, read and write via HWI
CORE_1 receives SWI from CORE_0 and does processing, this passes messages to CORE_2
CORE_2 Hyperlink connection to DSP2
...

Could the data be overwritten by another core that is using DDR3, as the cores *are* compiled independently?

I've looked at the memory browser for UserLogCanc2 with continuous refresh on and it is being continuously updated with random values, and regular 0s, although I am not using it. Is this likely to do with another core, or could it be a nuance of using DDR3 (caching, levelling...)?

The reason I suspected it was something to do with writing to DDR3 is the way that every few values are fine, with strange ones in between.

Thanks for all your help.

Thanks Ran, I suspect you are right and CORE_0 is overwriting it. I hadn't thought about the maps being generated independently for different cores, and the CORE_0 code was rather inherited.

CORE_0

80000000 80000000 0043b880 00000000 rw-
80000000 80000000 0012c000 00000000 rw- .CpriRxBuffer
8012c000 8012c000 0012c000 00000000 rw- .CpriTxBuffer
80258000 80258000 001e1000 00000000 rw- .extData
80439000 80439000 00002400 00000000 rw- .qmss
8043b400 8043b400 00000480 00000000 rw- .cppi

CORE_1

80000000 80000000 00096000 00000000 rw-
80000000 80000000 00096000 00000000 rw- .CpriRxBuffer
8012c000 8012c000 00096000 00000000 rw-
8012c000 8012c000 00096000 00000000 rw- .CpriTxBuffer
80258000 80258000 00168000 00000000 rw-
80258000 80258000 00168000 00000000 rw- .x_ll_shared

so x_ll_shared conflicts with extData.

What is best practice to avoid this? Presumably I can adjust the Platform in order to predefine areas for extData, qmss etc. Any other/better ways?

Is that the way forwards? If so could you please suggest some reference for how to adjust the platform (as I said Tools/RTSC Tools.... Edit, I couldn't find the appropriate file/extension to open the wizard).

David

Thanks Ran, I seem to be making some headway with this.

I am just wondering about initilializing when I use the DDR3 memory? In the .map file, the section I have generated in DDR3 is "RWIX" where I is Initializable? Also it says

".x_ll_shared 

* 0 a0000000 0010e000 UNINITIALIZED
a0000000 0010e000 user_process_ICS+RF+.obj (.x_ll_shared)"

Do I need to do some initialization, and if so how? Something in the performance of the code is leading me to believe that writing to this memory is slow for the first few loops.

Thanks!