AM572x DDR Memory Configuration

Hi,

I will ask the AM57X team to comment.

Hello Gerard,

Your use case described in your post is applicable for AM5728.
The EMIF manages data bus read/write accesses between external memories.

The most important step in your memory configuration is the right choice of the values in LISA_MAP register. The DMM consists of a local interconnect and synchronization agent (LISA) to synchronize all DMM subsystems and provide access to their configuration registers. An important role of the LISA is to configure the interleaving scheme, which optimizes the bandwidth of the two EMIFs. The DMM_LISA_MAP_i register values (where i = 0 to 3) configure the interleaving of each of the four sections of the DMM. Each section is defined based on its system address, size,
physical address, address space, target memory controller, and interleaving scheme.

You must apply your memory configurations in sdram.c file - ti-processor-sdk-linux-am57xx-evm-02.00.00.00/board-support/u-boot-2015.07+gitAUTOINC+d49aa5effa/arch/arm/cpu/armv7/omap5/ sdram.c

* For any new board with different memory devices over-ride one or more
* of the following functions as per the CONFIG flags you intend to enable:
* - emif_get_reg_dump()
* - emif_get_dmm_regs()
* - emif_get_device_details()
* - emif_get_device_timings()

Please use the following configuration as an example:
/*
* DRA752 EVM board has 1.5 GB of memory
* EMIF1 --> 2Gb * 2 = 512MB
* EMIF2 --> 2Gb * 4 = 1GB
* so mapping 1GB interleaved and 512MB non-interleaved
*/
const struct dmm_lisa_map_regs lisa_map_2G_x_2_x_2_2G_x_1_x_2 = {
.dmm_lisa_map_0 = 0x0,
.dmm_lisa_map_1 = 0x80640300,
.dmm_lisa_map_2 = 0xC0500220,
.dmm_lisa_map_3 = 0xFF020100,
.is_ma_present = 0x1
};

/*
* DRA752 EVM EMIF1 ONLY CONFIGURATION
*/
const struct dmm_lisa_map_regs lisa_map_2G_x_1_x_2 = {
.dmm_lisa_map_0 = 0x0,
.dmm_lisa_map_1 = 0x0,
.dmm_lisa_map_2 = 0x80500100,
.dmm_lisa_map_3 = 0xFF020100,
.is_ma_present = 0x1
};

/*
* DRA752 EVM EMIF2 ONLY CONFIGURATION
*/
const struct dmm_lisa_map_regs lisa_map_2G_x_2_x_2 = {
.dmm_lisa_map_0 = 0x0,
.dmm_lisa_map_1 = 0x0,
.dmm_lisa_map_2 = 0x80600200,
.dmm_lisa_map_3 = 0xFF020100,
.is_ma_present = 0x1
};

/*
* DRA722 EVM EMIF1 CONFIGURATION
*/
const struct dmm_lisa_map_regs lisa_map_2G_x_2 = {
.dmm_lisa_map_0 = 0x0,
.dmm_lisa_map_1 = 0x0,
.dmm_lisa_map_2 = 0x80600100,
.dmm_lisa_map_3 = 0xFF020100,
.is_ma_present = 0x1
};

The u-boot does not change the values in the EMIF registers. Take a look on SDRAM initialization in arch/arm/cpu/armv7/omap-common/emif-common.c:
/*
* SDRAM initialization:
* SDRAM initialization has two parts:
* 1. Configuring the SDRAM device
* 2. Update the AC timings related parameters in the EMIF module
* (1) should be done only once and should not be done while we are
* running from SDRAM.
* (2) can and should be done more than once if OPP changes.
* Particularly, this may be needed when we boot without SPL and
* and using Configuration Header(CH). ROM code supports only at 50% OPP
* at boot (low power boot). So u-boot has to switch to OPP100 and update
* the frequency. So,
* Doing (1) and (2) makes sense - first time initialization
* Doing (2) and not (1) makes sense - OPP change (when using CH)
* Doing (1) and not (2) doen't make sense
* See do_sdram_init() for the details
*/
void sdram_init(void)

I hope that the information above will help you.

Best regards,
Yanko

Gerard said:

DSP1 and DSP2 have an identical virtual memory map (mapping to different physical addresses)

This will require the MMUs within each DSP subsystem to be programmed.  Within Linux this is programmed using the "IOMMU" driver.  If you search the Linux code as well as e2e (or just google) for "IOMMU" you will hopefully be able to get that running.

Thanks for the response. The examples are straight forward for the 2GB total memory (or less) use cases but don't really dive into the trickier 4GB total meory use case I'm dealing with. I realize I have to do the following:

• Enable ARM LPAE support in the kernel
• Configure the MPU Memory Adapter to enable high memory interleaving
• Configure the MPU Memory Adapter/DMM LISA map sections
• Configure the DSP1/DSP2 MMU (to achieve the goal of an identical virtual address)

Focusing now on the MPU Memory Adapter/DMM LISA mapping concerns, could you provide more specific configuration information related to:

1. How does the MPU_MA_LSM configuration relate to the DMM_LISA_MAP_i configuration? I'm assuming both must be configured to satisfy MPU and DSP access needs.
   1. Does only the high memory mapping go in the MPU_MA_LSM configuration registers and the lower memory mapping goes in the DMM_LISA_MAP_i registers?
2. Since I'm using LPAE, my overall EMIF address range is 0x02_0000_0000 to 0x02_FFFF_FFFF. What does this translate to when I'm configuring the mapping registers? Specifically, what would I use for bits 31:24, the SYS_ADDR field? 0x00? 0x20?
3. The MPU high memory interleaving is fixed at 256-byte interleaving. Does the lower memory interleaving (which is configurable) have to match this for performance reasons?

Thanks!

Hello Gerard,

1. How does the MPU_MA_LSM configuration relate to the DMM_LISA_MAP_i configuration? I'm assuming both must be configured to satisfy MPU and DSP access needs. - 

   MA_LISA_LOCK corresponds to the DMM_LISA_LOCK register description, and MA_LISA_MAP_i corresponds to the DMM_LISA_MAP_i register description. 

   Memory Adapter [MA] registers which mirror the LISA settings. The purpose of the MA is to improve the missed latency of the L2 cache between the
   ARM Cortex-A15 processor and external memory.

   LSM supports programmable interleaving for 2 GiB of shared memory:
   Programmable multizone DRAM mapping and interleaving configuration
   Supports four prioritized sections for defining configuration regions within the external memory
   Supports interleaving at 128-, 256-, and 512-byte boundaries

   1. Does only the high memory mapping go in the MPU_MA_LSM configuration registers and the lower memory mapping goes in the DMM_LISA_MAP_i registers?

      See the following explanation:

      The MA_LSM is associated with the lower 2GiB shared memory space, that is, emif(a) and emif(b). In this case there is programmable interleaving (128B, 256B and 512B) and programmable MPU-to-EMIF address mapping configured trough the MA_LISA_MAP_i registers.
      
      

      

2. Since I'm using LPAE, my overall EMIF address range is 0x02_0000_0000 to 0x02_FFFF_FFFF. What does this translate to when I'm configuring the mapping registers? Specifically, what would I use for bits 31:24, the SYS_ADDR field? 0x00? 0x20?

   - I think that you must take in mind - DMM_LISA_MAP_3[22:20], which is the SYS_SIZE, should be configured to match the LPDDR2 size. SYS_SIZE = 0x5 indicates a 512MB section, 0x6 a 1GB section, and 0x7 a 2GB section.

   DMM_LISA_MAP_i[31:24] SYS_ADDR - DMM system section address MSB for view mapping i

   Take a look on section 15.2.4.2.1 Case 1: Use of One Memory Controller - In this example, assume there is 1 GiB of external memory evenly spread onto two address spaces. The address range for address space 0 must start at offset 0x2000_0000. See the SDRAM configuration in u-boot: 

   /*
   * DRA752 EVM board has 1.5 GB of memory
   * EMIF1 --> 2Gb * 2 = 512MB
   * EMIF2 --> 2Gb * 4 = 1GB
   * so mapping 1GB interleaved and 512MB non-interleaved
   */
   const struct dmm_lisa_map_regs lisa_map_2G_x_2_x_2_2G_x_1_x_2 = {
   .dmm_lisa_map_0 = 0x0,
   .dmm_lisa_map_1 = 0x80640300,
   .dmm_lisa_map_2 = 0xC0500220,
   .dmm_lisa_map_3 = 0xFF020100,
   .is_ma_present = 0x1
   };

   /*
   * DRA752 EVM EMIF1 ONLY CONFIGURATION
   */
   const struct dmm_lisa_map_regs lisa_map_2G_x_1_x_2 = {
   .dmm_lisa_map_0 = 0x0,
   .dmm_lisa_map_1 = 0x0,
   .dmm_lisa_map_2 = 0x80500100,
   .dmm_lisa_map_3 = 0xFF020100,
   .is_ma_present = 0x1
   };

   /*
   * DRA752 EVM EMIF2 ONLY CONFIGURATION
   */
   const struct dmm_lisa_map_regs lisa_map_2G_x_2_x_2 = {
   .dmm_lisa_map_0 = 0x0,
   .dmm_lisa_map_1 = 0x0,
   .dmm_lisa_map_2 = 0x80600200,
   .dmm_lisa_map_3 = 0xFF020100,
   .is_ma_present = 0x1
   };

   /*
   * DRA722 EVM EMIF1 CONFIGURATION
   */
   const struct dmm_lisa_map_regs lisa_map_2G_x_2 = {
   .dmm_lisa_map_0 = 0x0,
   .dmm_lisa_map_1 = 0x0,
   .dmm_lisa_map_2 = 0x80600100,
   .dmm_lisa_map_3 = 0xFF020100,
   .is_ma_present = 0x1
   };

   To check whether an address hits a section, use the 8 upper address bits of the address and mask them with the hit mask: 28–2^SYS_SIZE. If the result is equal to SYS_ADDR, the section is hit.

   LISA MAP configuration on VAYU EVM:

   root@dra7xx-evm:~# omapconf read 0x4E000044    80640300    - DMM_LISA_MAP_1

   root@dra7xx-evm:~# omapconf read 0x4E000048    C0500220   -  DMM_LISA_MAP_2

   root@dra7xx-evm:~# omapconf read 0x4E00004C    FF020100   -  DMM_LISA_MAP_3

    Specifically, what would I use for bits 31:24, the SYS_ADDR field? 0x00? 0x20? 

   - Therefore for your SYS_ADDR you must use 0x80.

3. The MPU high memory interleaving is fixed at 256-byte interleaving. Does the lower memory interleaving (which is configurable) have to match this for performance reasons?

The high-order memory space, which is accessible only from the MPU subsystem, uses a simple fixed interleaving scheme, which can be disabled. Heavy use in high memory space under noninterleaved configuration affects the balancing of the system access in lower-order memory.

The MA_LSM is NOT associated with the 6GiB of MPU-only memory, that is, Emif(c) through Emif(h). In this case there is fixed interleaving (256B only) and fixed MPU-to-EMIF address mapping which cannot be programmed using the MA_LISA_MAP_i registers because MA_LSM is not used with address ranges greater than 2GiB.

- See the options in register DMM_LISA_MAP_i[19:18] SDRC_INTL - SDRAM controller interleaving mode 

0x0: No interleaving
0x1: 128-byte interleaving
0x2: 256-byte interleaving
0x3: 512-byte interleaving
The 128-/256-/512-byte interleaving applies only to
nontiled regions. If accesses are made to tiled regions,
interleaving is forced to 1kiB. SDRC_INTL is don't care if
SDRC_MAP is not 0x3 (no interleaving).

In case of 256-byte interleaving, the first chunk of 256 bytes is mapped to the first controller, the second chunk to the second controller, and so on. This results in system address bit 8 to be decoded as the controller indicator, 0 for the first controller, and 1 for the second controller (system address bit 7 is used for interleaving at the 128-byte boundary, and bit 9 for 512 bytes). This bit is not included in the computed physical address, meaning the upper system address bits [31:9] are shifted to bits [30:8] when generating the physical address.

Best regards,

Yanko

Yanko Todorov-XID said:

Hello Gerard,

1. How does the MPU_MA_LSM configuration relate to the DMM_LISA_MAP_i configuration? I'm assuming both must be configured to satisfy MPU and DSP access needs. - 

   MA_LISA_LOCK corresponds to the DMM_LISA_LOCK register description, and MA_LISA_MAP_i corresponds to the DMM_LISA_MAP_i register description. 

   Memory Adapter [MA] registers which mirror the LISA settings. The purpose of the MA is to improve the missed latency of the L2 cache between the
   ARM Cortex-A15 processor and external memory.

   LSM supports programmable interleaving for 2 GiB of shared memory:
   Programmable multizone DRAM mapping and interleaving configuration
   Supports four prioritized sections for defining configuration regions within the external memory
   Supports interleaving at 128-, 256-, and 512-byte boundaries

   1. Does only the high memory mapping go in the MPU_MA_LSM configuration registers and the lower memory mapping goes in the DMM_LISA_MAP_i registers?

      See the following explanation:

      The MA_LSM is associated with the lower 2GiB shared memory space, that is, emif(a) and emif(b). In this case there is programmable interleaving (128B, 256B and 512B) and programmable MPU-to-EMIF address mapping configured trough the MA_LISA_MAP_i registers.
      
      

      

What wasn't clear to me was when you use MPU_MA LISA map registers vs. DMM LISA map registers. In section 4.3.4.1.2.4 of the AM572x TRM it states the following:

The configuration of the MPU_MA must be programmed by the Cortex-A15 MPU to match the configuration of the DMM during boot.

This makes it sound like the LISA map in the DMM for the lower 2GB needs to match the LISA map in the MPU MA registers for the lower 2 GB. Could you please clarify this? I'm not initially seeing references in the U-Boot code that does this.

Thank you!

Brad Griffis said:

Gerard

DSP1 and DSP2 have an identical virtual memory map (mapping to different physical addresses)

This will require the MMUs within each DSP subsystem to be programmed.  Within Linux this is programmed using the "IOMMU" driver.  If you search the Linux code as well as e2e (or just google) for "IOMMU" you will hopefully be able to get that running.

Thank you for the previous response, it did help clarify the LISA map portion of the question.

Going back now to the DSP MMU portion of the overall memory configuration goal:

1. Since we are allocating 256MB for DSP1 and 256MB for DSP2 it would appear that our strategy should be to write 16 individual 16MB-page TLB entries into each DSP MMU. Is that correct? The translation table alternative (to achieve smaller page sizes) doesn't seem worthwhile for our use case.
2. Are there any working Linux code examples available showing TLB entries being written? I have not come across any in my searching.

Thanks!

Please start a new thread. People with deep knowledge of the IOMMU code are likely not the same people with deep knowledge of the DDR configuration (i.e. the subject of this thread).

Done.

Background:

We’re using the DRA744 and are deviating from the DRA7x EVM in that we have 2x the DDR memory as in the previous post

System Memory:

EMIF1:   2 GB DDR3

EMIF2:   2 GB DDR3

Starting development point: TI BSP for android 6AJ1.3

omapedia.org/.../6AJ.1.3_Release_Notes

We have followed the points explained above:

• Enable ARM LPAE support in the kernel
• Configure the MPU Memory Adapter to enable high memory interleaving
• Configure the MPU Memory Adapter/DMM LISA map sections

- we enabled "LPAE" support in kernel kconfig (it has to be disabled the TI_OMAP2 support)

- in u-boot (based on TI BSP for Android), we configured the MPU Memory Adapter to enable high memory interleaving

- we configured the LISA MAP :

const struct dmm_lisa_map_regs lisa_map_4G_x_2_x_2 = {

.dmm_lisa_map_0 = 0x0,
.dmm_lisa_map_1 = 0x0,
.dmm_lisa_map_2 = 0x80740300,
.dmm_lisa_map_3 = 0xFF020100,
.is_ma_present = 0x1

};

but we are facing a kernel block during boot with the following prints (visible only with early-printk enabled)

..
..
6cma: CMA: reserved 32 MiB at 9d000000
6cma: CMA: reserved 112 MiB at a8800000
Memory policy: ECC disabled, Data cache writealloc

It seems that the LPAE support in the kernel referenced by the 6AJ.1.3 needs something to patch to be really usable.

If we disabled the LPAE system starts with 2 GB RAM visible.

Is there a guide or a more specific setup to follow to test all the 4 GB RAM access starting from TI android BSP?

Thanks in advance

Paolo, Dario, Michele

Hi Brad Griffis,

I have a custom board which is designed based on beaglebone-x15. Could you please give me some advice with below questions?

1/ I designed DDR3 with size 2 GiB (16 bit data bus) and boot OK. I want to extend DDRAM to 4GiB (16 bit data bus) --> Does AM572x support?

I modified the u-boot to test with lower memory size (1GiB) and it boot OK.   You can see below my configure:

Default configure (2GiB):

static const struct dmm_lisa_map_regs beagle_x15_lisa_regs = {
    .dmm_lisa_map_3 = 0x80740300,
    .is_ma_present  = 0x1
};

and my modify (1GiB) and my board boot OK:

static const struct dmm_lisa_map_regs beagle_x15_lisa_regs = {
    .dmm_lisa_map_3 = 0x80640300,
    .is_ma_present  = 0x1
};

Now I want to configure size DDRAM 4GiB, what modules or registers i must configure ? How does beagle_x15_lisa_regs configure ?

I look forward to hearing from you.
Thanks and best regards,