RM57 Peripherals Become Read-Only

HI Garry,

  It is a bit weird that you can write to external SRAM but not internal peripheral registers. Architectural-wise the Cortex-R5F CPU core has two buses, the AXI-M and the PP-AXI. The AXI-M is used to interface with the memories such as L2 flash, L2 SRAM and external memory via EMIF. The PP-AXI is used to interface with the on-chip peripheral. When you use the JTAG to access the internal system resources it can be done via the CPU or via the DAP AHB-AP. I believe you are currently using the CPU to access the system resources during debug state, correct?

  When you said you can't write to the registers, I would like you to try several things.

  1. Can you do a continuous refresh to see after you write the register is refreshed with new value?

  2. I guess you write to the register via the register browswer window, correct? Can you open the memory browser window and go the respective peripheral address and write there? Please force a refresh too. Will it make a difference?

  3. Can you write to the internal SRAM starting at 0x08000000?

  4. If the above three aren't successful, can you use the DAP AHB-AP to write the same registers? This will give us some hints what is going on.

  5. If you simply do a system reset, not power up reset and not power-cycle, can you repeat the above steps?

Hi Charles:

Thank you for your reply and suggestions.  

Yes, my JTAG is connected directly to the MCU JTAG pins.  I am using the XDS2xx USB Probe.  Thus, we are using the MCU to access the system via JTAG.

Here are my findings:

1. I have turned on "continuous refresh" and the values do not change - no fix

2. I have tried writing with both Register Browser, and Memory Browser, same results - no change with forced refresh

3. I can write to internal memory at 0x08000000 okay - works

4. How do I use the DAP AHB-AP to access the system?

5. If I do a "system reset", not "CPU Reset" from the "chip icon", then I get the same result - no access.

So, if you have some more suggestions, please let me know.  It looks like the AXI-M is working but the PP-AXI is not working.

Thanks,

Garry.

Hi Charles:

I am looking for a way to use the SW-DP via CCS, but it seems to be hidden.

I know with using ST MIcro MCUs, there is a choice for using SW-DP or JTAG access, but I find no such choice here.

So, please let me know how to switch modes with CCS and XDS2xx device.

Thanks,
Garry.

Hi Garry,

The steps to do #4 is follows.

  1. In the debug window as shown below do a right-click to select 'Show all cores'

 2. You will now see additionally the IcePick, Dap.

 3. Select the DAP with a left click first

 4. If you have the Memory Browser open you can select the System_view. Go to a peripheral address and see if you can write to it. 

Hi Garry,
RM57 does not support SW-DP. Only JTAG.

Hi Garry,  

  You can find below block diagram in the device datasheet. Here you will find a bus (PP-AXI) from the Cortex-R5 to the Peripheral Interconnect Subsystem. You will find also a bus (AXI-M) connecting the CPU to the CPU Interconnect Subsystem. There is also the DAP in the diagram. The DAP has access to all resources in the device. If you are successful to use the DAP to access registers then the problem is somehow on the PP-AXI. As you can see under the Peripheral Interconnect Subsystem there are PCR1, PCR2 and PCR3. The CCMR5 is under PCR1. If you can't access CCMR5 then you most likely can't access other modules under PCR1. However, you can try to access modules under PCR3. Pick one module like NHET or GIO and see if you can write something to them. Based on your information we can further isolate the area of the problem. 

  

Hi Charles:

I followed your steps for this DP access, but still I cannot change the value of 0 that is located at 0xFFFFF604. Am I supposed to be able to change it? I think so.

I have re-confirmed that I can change internal RAM (0x08000000) and external SRAM (0x64000000).

I will now try your next suggestions.

Garry.

Charles:
I think it was a wrong decision by TI to remove SW-DP as the "backup" option for debugging.
Garry.

Charles:

Thank you for the diagram.

I have tried to modify GPIOB registers (bits 7-0) for DIR, DOUT, PSEL, and PDIS but they do not change.

I think my Peripheral Interconnect System is locked-up, possibly a silicon bug.

So, my question is: What can disable access all peripherals, or make them read-only? What control register, or what bit, can do this?

Let's keep going until we get to the bottom of this. I am stuck with my progress, and my deadline for code completion is upon me, so please throw some more TI resources at this problem.

Thanks,
Garry.

Hi Garry,

 I can't comment on the product definition to not include SW debug interface. But with the problem you are seeing I don't think it will make a difference if you have the SW DP or not. The SW-DP will be in parallel with the JTAG-DP inside the DAP. The problem you are seeing is not between any DP to AP because you can still read/write to the memory. This means that the path  JTAG-DP and APB-AP is proper. Having  SW-DP as the front end DP will not matter in your specific situation.

Garry,
For the GIO register you need to make sure you have released the GIO from reset by setting the RESET bit to 1 in the GIOGCR0 register. With the module in reset you can't write to any registers.

When you said you can read but not write, what values do you read? Do they still retain some values other than zeros?

What about your CPU MPU setting for the peripheral region? Did they somehow change to read-only? I doubt this is the case because you said you do not see any abort exception. But it is worth to check.

If something locks up then a reset should recover. But you said it does not change. Let's do another experiment. Do a system reset and place a breakpoint at 0x0. User the debugger to wrie to some registers. Can you succeed? Or the problem only occurs after you run some code?

Before the problem occured, did it work before? Did you change something in your code and suddenly the problem occured?

What is the clock frequency (GCLK, HCLK and VCLK) at the point when bad things happen? The peripheral operates off of the VCLK which is limited to 110MHz. If you exceed this limit by a big margin then it is possible the write will fail.

These are the things I can think of to check further.

Charles:

I see from the RM57 Tech Ref Man, section 5.3.5 (Reset Management), that the PMM can generate at Peripheral Reset (nPRST). Could this be the cause?

I also see from 5.3.3 that "TI can disable a power domain with a reset configuration word in flash bank 0". Is it possible that CCS created a corrupted flash image that is disabling the peripherals.

My system was working for the last 2 weeks, but recently when I flashed it again, then this problem surfaced. The incidence of the problem appears to coincide with the most recent debug session (and corresponding flash of code). I did not change source code, but I think the image was built again before flashing into MCU.

Now, the debug works, the code view works, and the code runs except it fails because peripherals are not working. I even did power-down the power-on, and it still does not fix the problem. Most likely something in flash is turning off the peripherals. How can this happen?

Where should I be looking to check for these possibilities?

Thanks,
Garry.

Garry,
I think our threads cross each other. Can you read my reply and try a few things I suggested. The GIO and CCM are in always on power domain. I don't think it is the power domain. If you access something in a turned-off powe domain you will get abort but this is not what you see, right?

One more thing I want to confirm with you. When you said you can't write. Is it that only in debug mode where you can't write using the register and memory browser? Can you code write to the registers?

Charles:

The application has been working for many weeks, so I don't think that I did anything to the code to make this happen. However, I did observe that the RESET bit in the GIOGCR0 register was zero, meaning that it was in reset state, correct?

The GIOGCR0 value is definitely zero (like most other registers that I am trying to change). So, I change GIOGCR0 to value 1 and that value is accepted. Now, I try some other GIO registers (DIR, DOUT, etc), but they do not change - they remain at zero.

I have been looking in the Tech Ref Manual for other possible reasons why all peripherals are not accessible, which is were I found out about the "peripheral reset" idea.

Sorry if I missed answering some of your question, but I may have tried them but failed to respond because they did not fix the problem.

I will go back over the chat session and see if I missed anything.

Thanks for your help.

Garry.

Charles:

In answer to your question about not being able to write, yes it is in debug mode (both peripheral and memory browser). No, the code write to the register does not work either.

Start-up self-test code that used to work has now failed. The self test writes to CCMR5 then reads back to confirm, then fails the test because the programmed write did not occur.

Garry.

Charles:

In answer to your question about break point at address 0, I have tried that with "CPU Reset" feature, then while at the break point, I cannot write to any peripheral. Memory data write is okay. All this occurs before any code runs, so nothing in the code locks out access to the peripherals.

In answer to your question about clock frequency, I have not changed any frequency, and while sitting at CPU reset, I believe the clock should be at start-up frequency since the PLL has not been activated.

There are no Exceptions Thrown by the core, neither Data nor PreFetch Aborts. The code runs until it fails self-test because peripherals are not programmable.

I think the MCU has locked up somehow. I believe I can fix it by re-flashing a few times, but I don't want to do that until we can find the source of this failure.

Perhaps you could ask Christian or some other people for their suggestions.

This may be a serious design flaw in the silicon, so I am not prepared to cover it up. We must find the reason so we can move forward.

Thanks for your help, so let's keep going at it until we fix it. BTW, I am on Pacific Time zone,
Garry.

Charles:

FYI, here are screen captures of the SYS registers that I have in this state:

Sys_Reg_Images.7z (231.1 KB) => files.secureserver.net/0sDDhh4ASeokzo

Perhaps something has been flipped in here.

Garry.

Garry,

Garry Anderson said:

The GIOGCR0 value is definitely zero (like most other registers that I am trying to change). So, I change GIOGCR0 to value 1 and that value is accepted. Now, I try some other GIO registers (DIR, DOUT, etc), but they do not change - they remain at zero.

  This is interesting. You are able to write to GIOGCR0 but not other GIO registers. If you can write to the GIOGCRO0 then it means the path from CPU->CPU Interconnect Subsystem->PCR3->GIO is working. Can you try other module like NHET peripheral or NHET RAM?

Garry Anderson said:

I have been looking in the Tech Ref Manual for other possible reasons why all peripherals are not accessible, which is were I found out about the "peripheral reset" idea.

Can you check the CLKCNTL register in the SYS module. Bit 8 needs to be 1 to release peripherals from reset. What is it? Please provide the entire register value. Please also provide HCLKCNTL register value in the SYS frame. I want to know your GCLK:HCLK:VCLK ratio.

Garry Anderson said:

In answer to your question about break point at address 0, I have tried that with "CPU Reset" feature, then while at the break point, I cannot write to any peripheral. Memory data write is okay. All this occurs before any code runs, so nothing in the code locks out access to the peripherals.

CPU reset only resets the CPU and nothing else. If somewhere other than the CPU is bad, a CPU reset will have no effect. You must try the system reset which reset the entire device.

[Edit] I'm not sure how you generate the CPU reset. If you simply click the CPU reset in CCS, then it is not even real CPU reset. The debugger simply move the PC to 0x0. Nothing inside the CPU is reset. Again, please try system reset and breakpoint at 0x0.

Garry Anderson said:

In answer to your question about clock frequency, I have not changed any frequency, and while sitting at CPU reset, I believe the clock should be at start-up frequency since the PLL has not been activated.

No, if you simply assert CPU reset then only the CPU is reset. CPU reset will not reset the PLL. Again try system reset.

Garry Anderson said:

There are no Exceptions Thrown by the core, neither Data nor PreFetch Aborts. The code runs until it fails self-test because peripherals are not programmable.

ok. What happen if you simply comment out the selftest for the sake of debugging. Will it make a difference?

Garry Anderson said:

I think the MCU has locked up somehow. I believe I can fix it by re-flashing a few times, but I don't want to do that until we can find the source of this failure.

Do you have another board where you can flash the same code and see if you have the same problem? This way we know if the problem is repeatable on another device. 

Garry Anderson said:

Perhaps you could ask Christian or some other people for their suggestions.

I will ask around for suggestions. Personally this is the first time I have seen this problem. 

Garry Anderson said:

This may be a serious design flaw in the silicon, so I am not prepared to cover it up. We must find the reason so we can move forward.

Thanks for your help, so let's keep going at it until we fix it. BTW, I am on Pacific Time zone,

No one is covering up anything. If it is truly a bug then we will issue errata and provide a possible workaround.

I'm in central time, Houston TX. 

The CLKCNTRL looks fine where it shows the VCLK is divide by 2 from HCLK and the peripheral is out of reset.

You did not show the SYS2 frame. I want to see the HCLKCNTL register content.

You have enabled ECLK and your current ECLK is VCLK / 6. Can you measure ECLK frequency on a scope?

Charles:

In answer to your questions, here is what I found using Debug Mode of CCS via JTAG:

1. I can change NHET1-GblCtrl value from 0 to 1 and back to 0 again, so that appears to work.

2. I cannot figure out how to use NHET RAM, so I am going to have to skip that step.

3. In answer to your question about ClkCntl value it is 0x01010100, indicating that PENA = 1

4. Using System Reset (from CCS menu) with my first break point left over at address 0, the execution stops at 0 (as expected), but what am I supposed to check? I checked the CCMR5 register, while in this state, and it still cannot be modified at address 0xFFFFF604 memory browser or peripheral register mode.

5. Running from 0 after system reset hits my second break point at the code that attempts to write value of 6 into CCMR5, but that write does not work either. Also, using memory browser or peripheral registers at this break point does not modify the CCMR5 value.

6. Your suggestion about flashing the code on another board can be tried later because the other boards are being used by other team members. The board I am using is new, and has been working for weeks. My code has been working for weeks until recently.

7. Commenting out the self-test will mean that I have to re-build and re-flash, so that is not a good idea at the moment. You keep forgetting that the code has been working for weeks, and the self-test has been working. It is best that we keep the system in this state until we figure out what is going on.

8. Yes I have a scope, but where can I measure ECLK? On our board we have something called ECLK1 coming from ball A12 going to a "modbus_mcu_eclk" signal. Will that work? It is on the bottom of the board and difficult to reach, but if necessary I can try to measure it.

9. Here is the Sys2 Reg capture => https://files.secureserver.net/0sU6Ih1ALktwLD

Hope these answers help,

Garry Anderson.

1. I can change NHET1-GblCtrl value from 0 to 1 and back to 0 again, so that appears to work.

Charles>> So writing to peripherals in PCR3 is ok.

2. I cannot figure out how to use NHET RAM, so I am going to have to skip that step.

Charles>> NHET RAM is memory mapped and starts at 0xFF46_0000. 

3. In answer to your question about ClkCntl value it is 0x01010100, indicating that PENA = 1

Charles>> The ClkCntl looks fine to me. 

4. Using System Reset (from CCS menu) with my first break point left over at address 0, the execution stops at 0 (as expected), but what am I supposed to check? I checked the CCMR5 register, while in this state, and it still cannot be modified at address 0xFFFFF604 memory browser or peripheral register mode.

Charles>> While you are in debug state halted at 0x0 can you write to other CCMR5 registers at 0xFFFFF60C, or 0xFFFFF614?

5. Running from 0 after system reset hits my second break point at the code that attempts to write value of 6 into CCMR5, but that write does not work either. Also, using memory browser or peripheral registers at this break point does not modify the CCMR5 value.

Charles >> Again can you try to write registers at 0xFFFFF60C, or 0xFFFFF614? Can you also try to write registers in the SYS module? Both the SYS and CCMR5 are under the PCR1. 

6. Your suggestion about flashing the code on another board can be tried later because the other boards are being used by other team members. The board I am using is new, and has been working for weeks. My code has been working for weeks until recently.

Charles>> This is where my colleague and I are having hard time with when it was working for weeks and without re-flash the code does not work anymore.  I would really like you to have a chance to run your existing identical code on another board. It will tell a lot of information. If the other board is working fine then it will lead me to think it is a hardware (could be the MCU or the board) issue of your current board. If it fails  then it could be either hardware or software related.  

Can you read the several addresses below? 

  0xF0080140 - This the system configuration stored in the OTP

  0xFFFFF500 - This is the ESM module. All errors are recorded in this module. 

  0xFFFF1000 - 0xFFFF15DC - These are the PCR1 registers. They control peripheral powerdown, peripheral protection, masterID control.

  As I mentioned in the past threads, if there is a violation due to illegal access (peripheral protection, masterID violation) you should get an abort. But you said you do not get any abort exception.  

7. Commenting out the self-test will mean that I have to re-build and re-flash, so that is not a good idea at the moment. You keep forgetting that the code has been working for weeks, and the self-test has been working. It is best that we keep the system in this state until we figure out what is going on.

Charles>> Fine. We need to at least try another board. 

8. Yes I have a scope, but where can I measure ECLK? On our board we have something called ECLK1 coming from ball A12 going to a "modbus_mcu_eclk" signal. Will that work? It is on the bottom of the board and difficult to reach, but if necessary I can try to measure it.

Charles>> Yes, please probe A12. Your SYS frame dump shows that the ECLK is divide by 6 from VCLK. Based on your ECLK frequency we can deduce the VCLK, HCLK and GCLK. I'm also interested to know what your Oscillator frequency. 

9. Here is the Sys2 Reg capture

Charles>> The HCKCNTL looks fine. It is divide by 2 from GCLK. 

HI Charles:

Thanks for getting back to me.

FYI, when I left the project on Friday, I left all my equipment powered and running over the weekend (with continuous-refresh turned on) so that I could continue this morning. However, upon arriving this morning I was greeted with Red Error messages about disconnect from XDS2xx device. So, I tried the usual tricks like unplugging the USB cable to re-assert the Win7 Driver, shutting down CCS and re-starting, then at some point I got connected again. During this process, when I had to restart CCS, the compiler cleaned the project then rebuilt, then flashed again. I confirmed the points that I had sent earlier today, that we were still in the same situation, cannot write to CCMR5 register. So, a CCS re-start and a re-flash did not fix the problem. Interesting that a re-flash did not fix the problems I have been experiencing.

Once I was connected again, after having checked that the problem still existed, I started following the steps outlined in your email, and when I got to point #4, my debugger suddenly lost communications with the XDS2xx device again, but this time while I was using it (I had continuous-refresh still running), so there were no power glitches with the equipment; CCS just disconnected from the XDS2xx device. I don't know why this happened, but most likely something flaky with the DAP inside the MCU I am thinking. After many attempts to re-connect (like this morning), this time I was unable to re-establish communication with XDS2xx device, so finally I had no other choice but to power down my board with the MCU on board. Now, after that power cycle, everything is working correctly again. I have not made any code changes, but we have lost the evidence of the problem. The board and application runs fine again.

So, now I am convinced that the MCU was "confused" internally and that System Reset did not fix it and re-flashing did not fix it. It looks like I could not communicate with the DAP when it was "confused", and that he CCMR5 register (and others) could not be accessed (programatically and DAP access) during its confused state.

At this point, my only conclusion is the this MCU is generally unreliable or, somehow, a specific sequence of flashing, debug connection, continuous-refresh of variables, etc. puts the MCU into a "confused" state that prevents writing to some registers.

So for now, we have to put this analysis on pause, until my MCU gets confused again. I will make another post when that happens, then provide you with answers to your most recent questions.

Garry Anderson.

Hi Garry,
When you are disconnected it is possible that the internal Voltage monitor detects an out of regulation. When a voltage out of regulation is detected it will pull an internal power up reset and this is why it is possible the debugger can lose connection. A system reset will reset majority of the device but not entirely. Things like PLL, OSC and debug logic are only reset by a power up reset, not system reset. Do you have external voltage supervisor? Can you interrogate your external voltage supervisor to see if any voltage out of regulation is also detected? I don't know for sure how the MCU puts itself into the 'confused' state but I think the power supply is something I will recommend that you monitor before the next occurrence of the problem.

Do you have other emulator other than XDS2xx? Next time if the same problem occurs can you use a different emulator to see if you can connect?

Garry,

Was your device at all abnormally warm or hot when the error condition was there?

Have you checked all the IOs and power rails to make sure there's no severe overshoot or undershoot?

Charles & Anthony:

Thanks for the suggestions about power supply. I did not design the hardware; that was done by another team at the company. From what I can see, the power supply design is very good and stable, but I will see if there is some way to monitor it. The external PS monitor IC is called UCD9090 device, but I don't know how it works except to provide reset signal to MCU for watchdog timeout or bad voltages. But, if the voltages are out-of-bounds, then the MCU would be reset and I would not be able to debug. I will get back on this point after I have spoken to hardware design team.

Anthony:

I did not "feel" the MCU for temperature when it was confused. I will do that next time it hangs up. I am quite confident that the hardware design team has done a good job with PCB layout and I/O protection. This is a multi-layer PCB with internal power and ground planes, approximately 10 x 15 inches with hundreds of components on board. The power rails appear to be clean, and the signal waveforms are good.

Garry.

Garry,

I'm wondering if some transient like a severe overshoot or an ESD event might have induced a state that stayed there until the power was removed. I wouldn't assume that the PS monitor would always catch this and anyway if it did it would only be able to reset but in this case reset doesn't recover as you have observed.