This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

HPI memory access issue

Other Parts Discussed in Thread: AM3358

Hi,

In our current project we have two CPUs, AM3358 and C6746, ARM runs Linux, DSP runs SYS/BIOS and they are communicating with each other over HPI. ARM’s GPMC drives DSP’s HPI port. ARM sets executable to DSP and boot it. After that ARM and DSP starts communicating periodically. Transferred data size between CPUs is around 1Mbyte/second.

The issue is ARM cannot write data sometimes/randomly to DDR of C6746. We have been investigating that DDR usage ratio of DSP core can corrupt HPI communication. When I check technical manual I see that HPI has the highest priority and can be read/write in any case.

Is my interpretation correct? Could you please direct me, what points should I check to solve the issue?

 

CCS v6.0.0

BIOS v6.40.01.15

XDC v3.30.01.25

 

Thanks & Regards,

  • Hi

    The issue is ARM cannot write data sometimes/randomly to DDR of C6746. We have been investigating that DDR usage ratio of DSP core can corrupt HPI communication. When I check technical manual I see that HPI has the highest priority and can be read/write in any case.

    Can you elaborate further on "sometimes/randomly" and the nature of failure (missing data, corrupted data)? How are you isolating whether it is a board stability, software stability or purely throughput / contention issue?

    Please also note that HPI on c6746 by default is at priority 6. Refer to the TRM, SYSCFG chapter MSTPRIx control registers.

    Recommend

    1) Checking if the interface speed makes a difference between pass/fail. If it is feasible to increase / decrease speed of the interface to see if the problem becomes more or less prominent

    2) Recommend ensuring that your DDR design is robust, by running some stress test on DDR

    3) Recommend ensuring your GPMC/HPI link is robust and no software issues on data communication.

    4) If you think it is a priority/DDR contention issue, can you see if changing the ARM access over HPI from DDR to say Shared RAM or DSP internal memory makes it more stable?

    Hope this helps some.

    Regards

    Mukul

    4) If you think it is prioritization/contention issue

  • Hi Mukul,

    We suspect for DDR contention issue based on the result we got from the following experiments.

    Dsp core running nothing or just while 1 loop. In this condition arm is able to read/write the data to dsp over hpi, also read/write operation are correct and consistent.

    A data pattern was written to dsp's ddr known location and on arm side in while loop around 500msecs delay, the data was read. when dsp core is idle or running just while 1 loop, we see the data read from dsp is correct. This we have testing for more than an hour.

    say, when dsp core is accessing the ddr, like, writing and reading from ddr. we see inconsistency in the reading the data from arm. say my data pattern was 0x2222222211111111 , 64 bit pattern. we see randomly read as 0x222222211110000 or 0x2222222200000000 or 0x1111000000000000 or 1111111100000000.Sometimes while writing the data, we see the data written from arm is not been latched on dsp's ddr memory location.

    Here gpmc is configured as Non Multiplexed Address Data 16-Bit Device. gmpc timing for read cycle is 110 ns and write cycle is 60 ns secs. I have tried by improving the read/write cycle upto max. i.e 310ns. But still, no luck. The behaviour is same, not able to read consistently, when dsp core is accessing ddr.


  • Thanks for the update.
    Based on the description, I feel it can still be some sort of software/hardware synchronization issue when DSP is accessing DDR as I wouldn't have expected data corruption.
    Is HRDY etc hooked up and behaving well ?
    Are you aware of the following wiki
    processors.wiki.ti.com/.../AM335x_GPMC_to_OMAP-L13x_HPI_Interface

    To rule out traffic contention issues
    1) Try to put HPI at highest priority 0 (via MSTPRIx) registers , this will give higher priority for ARM accesses to DDR over HPI compared to DSP
    2) Ensure that DDR BPRIO/PBBR setting is reduced from default 0xff to something lower like 0x20.
    3) Try to do a temp test, if feasible, to see what happens if DSP accesses are routed to Shared RAM instead of DDR or if ARM accesses are routed to Shared RAM instead of DDR.

    Regards
    Mukul
  • Hi Mukul,


    Thanks for advising on making arm access as highest priority by writing to MSTPRIx register. I need to try that. 

    By the way, about the link. Yes i have seen the link earlier. Infact, my timings for read and write cycle is almost similar to which the link suggests.


    About HRDY., it is connect to gpmc_wait0. But I have observed that, hrdy signals are observed only on writing to HPIA registers and 2 half or higher order bits on read cycle. I dont see these signals when I do write operations. Even TRM explains the same. So I thought of not using hrdy signal.

    If you insist me to use the hrdy(gpmc_wait0) signal ? Should I need to configure the gpmc_wait0 only for read access ?

    Please let me know ur thoughts....


    Regards

    Chaitanya

  • Hello Chaitanya,

    The UHPI_HRDY signal is used for handshaking between the DSP and the host under both read and write cycles. So i would suggest to configure UHPI_HRDY on both cycles to avoid data lost.

    To debug on the DDR access issue further, we would like to know the result of the tests suggested by Mukul.

    Regards,
    Senthil
  • To rule out traffic contention issues, tests conducted:
    1) Try to put HPI at highest priority 0 (via MSTPRIx) registers , this will give higher priority for ARM accesses to DDR over HPI compared to DSP
    2) Ensure that DDR BPRIO/PBBR setting is reduced from default 0xff to something lower like 0x20.
    3) Try to do a temp test, if feasible, to see what happens if DSP accesses are routed to Shared RAM instead of DDR or if ARM accesses are routed to Shared RAM instead of DDR.

    1) checked the MSTPRI2 register bit 20-22 UHPI priority, its seems to be 0 by default. 0 says highest priority and 7 says lowest priority . In my case its 0 by default

    2) DDR PBBR is set to less 0x20. but no change in behaviour

    3) We have been doing this as a workaround. Since writing  to internal ram of dsp doesn’t have problem, we face issues only with ddr access. Also we try to transfer about 1Mbytes/sec, so it not possible for us to use the internal ram, since its only 256kbytes.

    I have used the hrdy signals earlier and tried. but that did not improve or helped me in resolving the issues.

    As per TRM,  Depending on the type of write operation (HPID without autoincrementing, HPIA, HPIC, or HPID with autoincrementing) and the state of the FIFO, transitions on UHPI_HRDY may or may not occur. Based on this, I asked, will the hrdy really help me in write operations. Since i dont see the hrdy occurring on write operations.


    Regards,

    Chaitanya.R

  • Chaitanya

    Yes, HRDY should absolutely be used for both reads and writes. Even if you don’t see ‘not ready’ on writes in some experiments, it’s still possible that the internal FIFO can become full (e.g., due to arbitration or bandwidth congestion on the DDR) and hrdy says ‘not ready’.
    It is this signal that tells the host (AM335x) that c674x UHPI is not ready to take any data, if it is busy - if AM335x ARM writes w/o notion of that , I wonder if data corruption can happen etc.

    What speed are you running your DDR interface, is it robust? Can you try a lower frequency to see if helps or hurts the configuration.

    FYI, as per the user guide UHPI MSTPRI bits by default should be set to a value of 6. So I don't know why you are seeing 0, is your software configuring this differently? I am also assuming you are accessing this register in priv mode (DSP in supervisor mode) - do you see non-zero values in other fields of MSTPRI2 register (most of the bit field except 18:16 and 6:4 should have a non zero value of 4 or 6 , as that is the default priority of the masters controlled by this register).

    Regards
    Mukul
  • Sorry about MSTPRI2 register value. even on my side default value is 6. I had read it wrong. I changed it to 0, but i see the isssue occurs more frequently. Its actually worsening the problem. I tried making it to 7, but it behaved the same way, as 6.

    On ddr frequency, currently i'm using at 150mhz. I'm using ddr2 on the device. i can try by reducing it.
  • Chaitanya,

    When you try reducing the DDR frequency and test, please configure the UHPI_HRDY also in your code for write cycle and check the behavior.

    Regards,
    Senthil
  • I tried reducing the ddr frequency from 150mhz to 132Mhz and 102mhz. but no change, also what I observed was, when ddr frequency was reduced, i used to see some corrupted data read randomly from dsp ddr . When ddr configured for 150mhz, I don’t see issues in reading.

    Meanwhile, I have configured the gpmc side to use the gpmc_wait for both read and write operation.

  • can u tell me, is hpi_hcs line edge triggered  or level triggered ?

  • Chaitanya,

    I do not able to find this information, but i think it should be level triggered.

    Regards,
    Senthil
  • Hi Serdar,

    It seems to us some synchronization issue when accessing the DDR by both the cores.

    To narrow down this problem, we shall derive test cases like below. This will help to identify a sequence of read/ write pattern
    by both the cores.

    Test case : 1
    ==============

    Lock the DDR memory location for exclusive access ( Using mechanisms like Mutex / semaphore ).
    Write a value by DSP to a particular memory location of DDR.Let us name the write value as 'O'
    Unlock the DDR memory location for exclusive access

    Lock the DDR memory location for exclusive access
    Read the value by DSP to the same particular memory location of DDR. Let us name the read value as 'A'
    Unlock the DDR memory location for exclusive access.

    Lock the DDR memory location for exclusive access
    Read the value by ARM through HPI to the same particular memory location of DDR. Let us name the read value as 'B'
    Unlock the DDR memory location for exclusive access.

    Check whether O == A or O == B or A == B

    Test case : 2
    ==============


    Lock the DDR memory location for exclusive access ( Using mechanisms like Mutex / semaphore ).
    Write a value by ARM through HPI to a particular memory location of DDR.Let us name the write value as 'P'
    Unlock the DDR memory location for exclusive access

    Lock the DDR memory location for exclusive access
    Read the value by ARM through HPI to the same particular memory location of DDR. Let us name the read value as 'Q'
    Unlock the DDR memory location for exclusive access.

    Lock the DDR memory location for exclusive access
    Read the value by DSP to the same particular memory location of DDR. Let us name the read value as 'R'
    Unlock the DDR memory location for exclusive access.

    Check whether P == Q or P == R or Q == R

    Repeat this test multiple times by changing the memory locations of DDR and changing the ARM and DSP read/ write sequence and check whether the values o == A == B == P == Q == R

    And also, please make sure that they have enabled the cache and there is no problem of cache coherency.

    Regards,
    Shankari and Titus
  • Hi Chaitanya, Serdar
    Shankari and team do have the additional information/details that you shared with the field team in December time frame.
    Their suggestions are based on parsing the information that was shared.

    I am also interested in understanding , if you have done any additional testing and/or have additional observations/data points to share that might help us diagnose this issue further.

    Regards
    Mukul
  • Hi Mukul,
    Chaitanya and me working on the problem. He develops ARM side and I develop DSP side.
    Beside this thread I have one more. I added some experiments. It may be good to understand better.
    e2e.ti.com/.../1737987

    Regards,
    Serdar