• State
  • Locked Locked
  • Replies 22 replies
  • Subscribers 30 subscribers
  • Views 2077 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMS570LC43xx interconnect global error and SDC MMR access

Shuozhi Yang
Intellectual 440 points
Other Parts Discussed in Thread: HALCOGEN

Hi,

I recently encountered this weird issue. When I enabled the internal EEPROM emulator flash bank access and regularly wrote to it, I got ESM error and the flag tells me it's the interconnect subsystem global error. So I tried to read SDC MMR registers to figure out what is going on and I found I cannot access the SDC MMR registers, neither from GDB or from C program.

Any help is appreciated!

Thank you!

  • 0
    TI__Guru**** 190806 points
    Hi Shuozhi,
    What do you mean that you cannot access the SDC MMR registers? Are you getting a data abort? If this is the case please check the MPU setting to make sure that the SDC MMR starting at 0xFF00_0000 is included as a valid region.

    Can you also tell me how you wrote to the EEPROM? Are you using the F021 Flash EEPROM Emulation Library to write to the flash bank?
  • 0 in reply to Charles Tsai
    Intellectual 440 points
    Hi Charles,

    From GDB, it prompts the error message "Cannot access address 0xFA000000", and in C code I got a data abort (I will check MPU setting on this).

    I am using F021 Flash API to write to the EEPROM bank. Basically I use it as a flash with finer bank width.
  • 0 in reply to Shuozhi Yang
    TI__Guru**** 190806 points
    HI Shuozhi,
    Yes, I think the reason is because the SDC MMR is not declared as a valid region. If you check the data fault status register and data fault address register in the CPU it will indicate a MPU fault.

    Once you can access the register please tell me which error flags are set. Personally I have not seen an interconnect global error flagged during a Flash write operation though. There are some errata related to global error. I wonder if you are hitting the errata.
  • 0 in reply to Charles Tsai
    Intellectual 440 points

    Hi Charles,

    I have enabled accessing SDC MMR registers in the MPU setttings and now I can pull the error flags from them.

    Here are the register dump from the C code:
    0x00: sdc_status = 0x00000010 --> Global Error
    0x04: sdc_control = 0x0
    0x08: err_generic_parity = 0x0
    0x0C: err_unexpected_trans = 0x0
    0x10: err_trans_id = 0x00000010 --> Cortex-R5F CPU master
    0x14: err_trans_signature = 0x0
    0x18: err_trans_type = 0x0
    0x1C: err_user_parity = 0x0
    0x20: serr_unexpected_mid = 0x0
    0x24: serr_addr_decode = 0x0
    0x28: serr_user_parity = 0x0

    Can you please provide me some insight explanation about this erorr?

    Thank you very much!


    Shuzohi

  • 0 in reply to Charles Tsai
    Intellectual 440 points
    Updates:

    I have tried to trigger the global error multiple times and I also occasionally got "Cortex-R5F CPU master" error flag on different error status registers, eg.
    0x08: err_generic_parity = 0x00000010
    0x10: err_trans_id = 0x00000010
    0x14: err_trans_signature = 0x00000010

    Also, I actually reordered the bytes when I post the messages. When I read the register, either from GDB or from C code, I actually read 0x10000000 on these registers, which maps to reserved section. I thought it was because these registers can only be accessed in 32-bit and it's a big endian system, but would love to have some confirmation from you.
  • 0 in reply to Shuozhi Yang
    TI__Guru**** 190806 points

    Hi Shuozhi,

     Please see below errata about byte swap. So what you see is an known errata.

    DEVICE#51 Values in the Interconnect status registers are left-shifted left 24 bits

    Severity 4-Low

    Expected Behavior Reading the interconnect status register is expected to return the proper value in bits 7

    through 0 as documented.

    Issue With the issue, the values of the interconnect registers bits 7 through 0 appear in bit locations 31 through

    24.

    Condition Any read of the interconnect registers (addresses 0xFA000000 through 0xFA00002B)

    Implication(s) Byte reads from the specified byte location returns zero.

    Workaround(s) Software is needed to right-shift the register by 24 bits.

    Back to you original problem as to what is the reason that these errors are detected, at the moment I do not have an answer as this is the first time I have seen this problem when writing to the EEprom bank.

    Can you please tell me a few settings of yours:

     1. what is the device frequency you are operating (GCLK, HCLK and VCLK)?

     2. Are you using the DMA or other bus masters at the time when these errors are detected?

     3. What is the MPU setting for the EEprom bank?

     We have an errata that generates false interconnect global error under certain conditions. In this errata, the false global error is generated depending on the MPU attributes of the memory and whether or not DMA is operating. I wonder if your observed problem could be also explained by this errata. Please refer to DEVICE#49 errata below.

    DEVICE#49 False interconnect safety checker error flag

    Severity 3-Medium

    Expected Behavior CPU Interconnect Subsystem - Global error flag (ESM group 1 channel 52) should not be set

    for untaken speculative data fetches.

    Issue CPU Interconnect Subsystem - Global error flag (ESM group 1 channel 52) may be set for untaken

    speculative data fetches.

    Condition All the following conditions have to occur.

    (1)The memory attribute is configured as normal.

    (2)There is a speculative fetch outside the valid memory range.

    (3)Multiple bus masters are configured for use.

    Implication(s) CPU Interconnect Subsystem - Global error flag (ESM group 1 channel 52) is falsely set.

    Workaround(s) Enable CPU MPU to block any access (real and speculative) outside the valid memory range.

  • 0 in reply to Charles Tsai
    Intellectual 440 points
    Hi Charles,

    Thank you for your prompt response!

    I have seen this errata when I was trying to debug myself, but I did not see it fit in the issue I have.

    To your questions:
    1. GCLK = 300MHz, HCLK = 150MHz, VCLK = 75MHz;

    2. No, I am not using DMA

    3. EEPROM MPU settings are:
    - privileged/user read/write
    - non-executable

    This global error happens regardless how frequent I program the EEPROM. If I program 128 bytes every 1ms, it will occur in about 5 seconds, while if I program the EEPROM 128 bytes every second, it will happen in 15 seconds to 5 minutes.
  • 0 in reply to Shuozhi Yang
    TI__Guru**** 190806 points
    Hi Shuozhi,
    Thank you for providing these info. The frequency looks normal to me. We can exclude the frequency and DMA as a possible cause for the problem.

    Are you running code out of flash when you program the EEprom or you are running the API out of the RAM? Can you also give the MPU region size for the flash memory? The LC4357 has 4MB of flash memory and 512k of SRAM. What are the MPU region sizes for these two memories?

    Finally, can you confirm if you are not programming the EEPROM then you do not encounter the interconnect global error? Also even with the global error asserted, does it impact the completion of the EEPROM programming in any way?

    Sorry for asking many questions as this is the first time I have seen this error being reported by anyone.
  • 0 in reply to Charles Tsai
    Intellectual 440 points

    I am running the EEPROM flashing API out of the RAM.

    In MPU setting, we mark 4MB for flash memory while for EEPROM we actually mark 8MB starting from 0xF0000000, which includes Flash OTP, ECC and EEPROM according to SPNU563 (TRM) 2.2.2 Memory Map Table.

    The 8MB size is a little too much for this region, and I don't know if it could cause any problem.

    The system has been running for couple of months and we have never seen the interconnect global error. It just happened since we enabled the EEPROM and started to do some programming stress/timing test. When the global error occurs, the EEPROM programming seems to be interrupted in the middle, eg. randomly, 16/32/64 bytes programmed out of 128 bytes and followed by unprogrammed 0xff.

  • 0 in reply to Shuozhi Yang
    TI__Guru**** 190806 points
    Hi Shuozhi,
    What about the MPU type for the EEPROM bank? Is it declared as NORMAL or DEVICE or Strongly Ordered? If it is not strongly-ordered can you configure to strong-ordered?
  • 0 in reply to Charles Tsai
    Intellectual 440 points
    The EEPROM MPU type is NORMAL. I have tried Strongly Ordered and still got the same global error.
  • 0 in reply to Charles Tsai
    Intellectual 440 points
    I am wondering if you can reproduce this error on your system? I have the Hercules devkit: www.ti.com/.../tmdx570ls04hdk

    What I do in my test is to write random data to EEPORM, 128 bytes at a time; when it reaches the end of the EEPROM, it wraps back to the base and continues the same sequence. During test, whenever it switches sector it checks if the sector is programmed and erases the sector if it is.

    I tried programming every millisecond to every second and the global error occurred in 5 seconds to couple of minutes for sure.
  • 0 in reply to Shuozhi Yang
    TI__Guru**** 190806 points

    Hi Shuozhi,

      I think I will need you to run several experiments. First, do you have cache enabled? If it is enabled, can you disable it? Secondly, you should be able to run the flash API from flash bank0 instead of RAM since you are programming the EEPROM which is a different flash bank from bank0. Can you also try these two experiments?

      You mentioned that you don't have the DMA. Do you have any other masters such as HTU, FTU, EMAC operating?

  • 0 in reply to Charles Tsai
    Intellectual 440 points

    Hi Charles,

    Yes, I have tried disabling cache for EEPROM, and got the same error.

    I do have EMAC operating. It still happened when I disabled EMAC.

    Calling flash API from flash still gave me the same error.

  • 0 in reply to Shuozhi Yang
    TI__Guru**** 190806 points
    Hi Shuozhi,
    Thank you for running these experiments. Have you tried to single step and be able to narrow down the instruction that caused the global error to assert or this can only be reproduced in run mode?

    Do you have another LC4357 device that you can try and reproduce the same failure?
  • 0 in reply to Charles Tsai
    Intellectual 440 points
    Hi Charles,

    I tried three boards. It happened to all three.

    It is very hard to reproduce the error in single step debugging. It occurs randomly in time between couple of seconds and couple of minutes. As I said if I increased the programming frequency (every 1ms), it will occur sooner, which seems to me a bus collision situation where more frequent flashing programming increases the interconnect system bandwidth usage.

    So, what is an interconnect system global error from the hardware side? I guess the interconnect system has an arbiter/buffering kind of system that coordinate the traffic on the processor bus. Please correct me if I have the wrong concept.

    Thank you!


    Shuozhi
  • 0 in reply to Shuozhi Yang
    TI__Guru**** 190806 points
    Hi Shuozhi,
    The interconnect has built in hardware checker. Every transaction issued by the CPU onto the interconnect is being monitored by the the hardware checker. The error is showing that a mismatch is detected.

    Can you try to limit to one sector, will you see the same problem? Repeat programming for sector and erase.

    Earlier you said that you have been using the device for months without problem. During these times without problem were you writing to the EEPROM? If you did, did you remember what is the difference compared to now?
  • 0 in reply to Charles Tsai
    Intellectual 440 points
    I tried 1 sector, program 128 bytes at a time till the end of the sector and then erase it and start programming again from the start. The global error still happens.

    This is the first time we started to utilize the EEPROM region in the flash memory. In the past couple of months we only used the flash memory to stored firmware and erase/update the flash when we do in-application firmware update, which seems to be fine (no stress test). Now we want to use the EEPROM region to save some dynamic settings that may be updated frequently under certain conditions, so that's when the error got exposed.
  • 0 in reply to Shuozhi Yang
    TI__Guru**** 190806 points
    Hi Shuozhi,
    From my understanding you are not using the FEE driver provided by HalCoGen but rather the flash API to program the EEPROM bank. Is this correct?

    TI provides the FEE driver that allows you to emulate EEPROM style of writes and reads. Please take a look at the below wiki link for further details if you have not used it before. HalcoGen also provide example FEE code. Please go the HalCoGen->Help->Help Topics->Examples->example_TI_Fee_Write_Read.c for illustration.

    processors.wiki.ti.com/.../TI_FEE

    This is not to say we are forgetting about your problem. It is just that I'm kind of running out of ideas as to what happens at the moment. How small can you cut down your project and can still reproduce the problem? Can you also create this as a CCS project so I can take a look. In the meantime I will suggest you to use the TI FEE driver for managing writing to EEPROM.
  • 0 in reply to Charles Tsai
    Intellectual 440 points

    Hi Charles,

    Before switching to FEE, I did the my test on flash memory. I wrote to flash bank2 every 1ms to the end of the sector and erase whole sector and continue programming it, 128 bytes at a time. So far, I have not seen similar issue with flash bank2 yet.

    I have tried adding delay to each stage of EEPROM programming just in case the FSM does not reflect the correct ready state, but I still got the same error. I think if this is a underlying interconnect system problem/errata, it is hard to avoid by using any higher level wrapper library like FEE.

    For now, I will use some spare sectors in flash bank2 to save my runtime data. If you have any further information, please do let me know.

    Thank you very much for your help!

    Best

    Shuozhi

  • 0 in reply to Shuozhi Yang
    TI__Guru**** 190806 points
    Hi Shuozhi,
    I'm sorry for you to go through these troubles. You might have run into a real issue that we have not known so far. Please do try to use the FEE to see if it makes a difference. Can you please also send us your CCS project to a minimum configuration (for example, with basically the CPU writing to the EEPROM and nothing else like EMAC or other operating peripherals) that can still reproduce the problem. We will need to involve our design team to understand the root cause and perhaps issue an errata and workaround if we confirm the problem.
  • 0 in reply to Charles Tsai
    Intellectual 440 points
    Hi Charles,

    I will try to generate the smallest project with HALCoGen to reproduce the interconnect global error.

    Thank you!


    Shuozhi