issues in GPMC interfacing of FPGA DDR3

Hi,

We are using DM814x processor and FPGA in our custom board. A 2Gb DDR3 is  connected to FPGA. We should interface with DDR3 using GPMC connected to FPGA. I've written FPGA driver for I2C and SPI communication between processor and FPGA (for FPGA control register access only) and also to access that FPGA DDR3 using GPMC.  Here is the piece of code that I've written.

err = i2c_add_driver (&i2c_fpga_driver);
if (err != 0) {
printk(KERN_ERR "i2c_add_driver failed\n");
return err;
}

err = spi_register_driver(&spi_fpga_driver);
if (err != 0) {
printk(KERN_ERR "spi_register_driver failed\n");
return err;
}

printk(KERN_DEBUG "Getting Chip Select\n");

if (gpmc_cs_request(GPMC_CHIP_SEL_NUM, SZ_256M, &fpga_i2c_di->control_mem_base) < 0) {
printk(KERN_DEBUG "%d: %s in %s\n", __LINE__, __func__, __FILE__);
printk(KERN_ERR "gpmc_cs_request: fail\n");
goto free;
}
printk(KERN_DEBUG "Got CS1, address = %lx\n",  fpga_i2c_di->control_mem_base);

if (!request_mem_region(fpga_i2c_di->control_mem_base, SZ_256M, "fpga_cs")) {
printk(KERN_DEBUG "%d: %s in %s\n", __LINE__, __func__, __FILE__);
printk(KERN_ERR "Request_mem_region failed.\n");
gpmc_cs_free(GPMC_CHIP_SEL_NUM);
goto free;
}

fpga_i2c_di->ctl_addr = ioremap_nocache(fpga_i2c_di->control_mem_base, SZ_256M);

printk(KERN_DEBUG "After Chipselect & remap.\n");
printk(KERN_DEBUG "Config1: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG1));
printk(KERN_DEBUG "Config2: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG2));
printk(KERN_DEBUG "Config3: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG3));
printk(KERN_DEBUG "Config4: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG4));
printk(KERN_DEBUG "Config5: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG5));
printk(KERN_DEBUG "Config6: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG6));
printk(KERN_DEBUG "Config7: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG7));
printk(KERN_DEBUG "\nRemap, address = 0x%x\n",  (unsigned int) fpga_i2c_di->ctl_addr);

When I execute the above driver code, 

physical addr returned by gpmc_cs_request (fpga_i2c_di->control_mem_base) = 0x10000000

virtual addr returned by ioremap_nocache function(fpga_i2c_di->ctl_addr) = 0xdc000000

We are using address and data multiplexed interfacing of NOR protocol. So, 27 lines are connected to DDR3 to access 256MB (each memory location is of 2 bytes). The problem is, while making read and write transactions using the virtual address returned from ioremap, physical address received on bus 0x4000000.  But I expect the address 0x0000000  on the bus. What is the mistake in this code?? how is this possible?? Can anyone please help in this issue?? 

Also we use only two chip selects in our board. NAND uses only 16MB which is not colliding with this ddr3 address.

  • Hi,

    Could you give us your config infor from prink?

    printk(KERN_DEBUG "Config1: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG1));
    printk(KERN_DEBUG "Config2: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG2));
    printk(KERN_DEBUG "Config3: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG3));
    printk(KERN_DEBUG "Config4: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG4));
    printk(KERN_DEBUG "Config5: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG5));
    printk(KERN_DEBUG "Config6: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG6));
    printk(KERN_DEBUG "Config7: 0x%x\n", gpmc_cs_read_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG7));

    BR,

    Viet

  • In reply to Viet Dinh:

    Hi Viet,

    Sorry. I missed out that info in my post. The config register values are,

    gpmc_cs_write_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG1, 0x28601201);
    gpmc_cs_write_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG2, 0x00141403);
    gpmc_cs_write_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG3, 0x00070705);
    gpmc_cs_write_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG4, 0x14071407);
    gpmc_cs_write_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG5, 0x020f1414);
    gpmc_cs_write_reg(GPMC_CHIP_SEL_NUM, GPMC_CS_CONFIG6, 0x0A070F04);

    printk(KERN_DEBUG "Getting Chip Select\n");

    if (gpmc_cs_request(GPMC_CHIP_SEL_NUM, SZ_256M, &fpga_i2c_di->control_mem_base) < 0) {

    printk(KERN_DEBUG "%d: %s in %s\n", __LINE__, __func__, __FILE__);
    printk(KERN_ERR "gpmc_cs_request: fail\n");
    goto free;
    }

    This is the piece of code which is used for configuring GPMC.  Also when I read back, I get the same values what I've written. 

    CONFIG7 register value: 0x50

    DDR read write transactions happens as expected. These config register values are calculated by consulting our FPGA team too. The only problem is,  address should start from 0x0000000 (since gpmc_cs_request returns 0x10000000) but we receive 0x4000000 as starting address in the bus which restricts our access to 128MB instead of 256MB. 

    Regards

    Vaishnavi

  • In reply to Vaishnavi Raman:

    Hi Vaishnavi,

    The GPMC address cannot be start from 0 since the first 16 MB is reserved for boot from Table 2-3. L3 Memory Map in Data Manual.  If your

    CONFIG7 register value: 0x50

    then the address is start at 0x1000_0000.  Could you please double check?

    BR,

    Viet

  • In reply to Viet Dinh:

    Hi Viet,

    Yes. You are right. Even I've mentioned the same in my first post. I've quoted those lines from my post below.

    "When I execute the above driver code, 

    physical addr returned by gpmc_cs_request (fpga_i2c_di->control_mem_base) = 0x10000000"

    Which implies that, the controller has given me access for 256MB. i.e 0x1000_0000 is a 32 bit address. To access 256MB (address location of 2 bytes), 27 lines are connected in our hardware.  So, from the 32 bit start address (0x1000_0000) received,  I should get the lower 27 bits in the address bus which is 000_0000 while performing transactions. But I am getting, 400_0000 in the bus. How is this possible?? I am checking this by writing from driver code(kernel space)  to the ioremapped location.

    For 27 lines, if

    start address : 0x000_0000

    end address :  0x7FF_FFFF,  we can access 256MB. That is how we calculated. 

    But In the current scenario, we receive the  start_address : 0x400_0000, which restricts our access to 128MB. 

    Regards

    vaishnavi

  • In reply to Vaishnavi Raman:

    Adding to the above information, I did some experiments and the results are listed below. I tried changing the size of requested memory in the gpmc_cs_request function as 16MB, 32MB, 64MB etc in the below piece of code.

    "gpmc_cs_request(GPMC_CHIP_SEL_NUM, SZ_256M, &fpga_i2c_di->control_mem_base"

    request size       physical address returned by gpmc_cs_request       actual address received in the gpmc bus while writing to ioremapped memory                    

    16MB                      0x100_0000                                                                           0x480_0000

    32MB                      0x200_0000                                                                           0x500_0000

    64MB                      0x400_0000                                                                           0x600_0000

    128MB                    0x1000_0000                                                                         0x400_0000

    256MB                    0x1000_0000                                                                         0x400_0000

    Also, the information given above is consistent always. The code used is same as given in the first post. Also, we have used "mem=364M mem=320M@0x9FC00000 vmalloc=500M" in our boot args. 

    What is happening in the gpmc address bus??? Is there any code error. please comment.

  • In reply to Vaishnavi Raman:

    Hi Vaishnavi,

    How do you read the actual address receive in the gpmc bus?  I see you have configure the device as multiplexed address data 16-bit.  Have you look at the Table 11-2 GPMC Pin Multiplexing Options to make sure you got the right data?

    BR,

    Viet

  • In reply to Viet Dinh:

    Hi Viet,

    We read the actual address in the gpmc bus using xilinx FPGA utility chipscope. For multiplexed address data 16-bit device, we should use A1-A11 as address bus and D0-D16 as bidirectional bus. So, totally we use 27 address lines to access 256MB (each location of 2 bytes). 

    When I add an offset (eg. 0x1, 0x2 etc..) to the ioremapped address and perform transaction, I receive the address without any shift. ie..

    sent address from driver                                      received address on bus

    ioremapped address + 0x1                                      0x400_0001

    ioremapped address + 0x2                                      0x400_0002 etc..

    So, we are not sure about shift in address.

    One more information is, When we do the same GPMC configuration from CCS and then a write/read transaction,  I am able to get the expected start address 000_0000 in the gpmc bus. So, from CCS, we are able to prove the interface. Only from linux, we are facing the above issues.

    Regards

    Vaishnavi

  • In reply to Vaishnavi Raman:

    Hi Vaishnavi,

    I had contact the Linux team for this issue.  I will get back to you soon.

    BR,

    Viet

  • In reply to Viet Dinh:

    Hi Viet,

    In my previous post, the statement  that  "from CCS, we are able to prove the interface"  has been proved as false now. Due to the wrong pin muxing of GPMC_A[11] pin in CCS, it has worked which I found yesterday. So, even from CCS, the behaviour is same as from UBoot and kernel. ie.. the Address issue still remains in CCS too.. 

    Thanks and Regards

    Vaishnavi

  • In reply to Vaishnavi Raman:

    Hi Viet,

    While debugging further, we found the observations below. In previous posts, I've mentioned that 

    "sent address from driver                                      received address on bus

    ioremapped address + 0x1                                      0x400_0001

    ioremapped address + 0x2                                      0x400_0002 etc.."

    ie. Address line A11 is always high. But that is not the actual case. Since we are using FPGA, every IO pin will have a pin property (PULLUP or PULLDOWN etc). So, it is the pin property of FPGA which has reflected on that line. What is happening is, I am unable to drive that A11  address line.  The other address lines are becoming low, because I'm able to drive those lines. Since, I am unable to drive A11, the default pin property has reflected.

    I've already given my gpmc config register values and also the code. Is there any mistake in that?? Why am I not able to drive A11?? Any help would be greatly appreciated.

    Regards

    Vaishnavi