AM5708: A15, dsp1 hangs if try to access any register of IEP of PRU-ICSS1: omap_l3_noc 44000000.ocp: L3 application error: target 5 mod:1 (unclearable)

Paritosh Dixit

Part Number: AM5708

Board: Custom board with AM5708

Processor SDK Linux RT 5.02

Our FW uses 64-bit PRU-ICSS1 Industrial Ethernet Timer. It is observed that dsp1 FW hangs while trying to initialize this timer via its registers. At the same time, we see this error:

[ 51.532367] omap_l3_noc 44000000.ocp: L3 application error: target 5 mod:1 (unclearable)
[ 51.540529] omap_l3_noc 44000000.ocp: L3 debug error: target 5 mod:1 (unclearable)

Basically, trying to access any register of IEP of PRU-ICSS1 hangs a core (dsp1 or A15).

To investigate this issue, we removed our FW / code completely, i.e., no FW running on dsp1, ipu1, pruss1. Also, no custom application running on A15/Linux. At this moment, if we access any register of IEP of PRU-ICSS1 using devmem2, it hangs the A15, and we have to power cycle the board.

devmem2 0x4B22E000 <<-- This command will hang A15

PRUSS1 CLK registers values on a affected board:

CM_L4PER2_CLKSTCTRL: Register : 0x4A0098FC, value: 0x00016100

CM_L4PER2_PRUSS1_CLKCTRL: Register : 0x4A009718, value: 0x00000002

Please note that:

We observed this issue on only few boards in production (Approximately 2 or 3 boards out of 10). Other boards work fine.

Issue is only observed after power cycle / hard reset: When we power cycle the affected board, issue is observed 100% of the times. When we reboot the board from Linux, issue is not observed, i.e., we can access IEP registers without any issue.

Please let us know how to diagnose this issue.

over 3 years ago

0 Vineet Roy over 3 years ago

TI__Mastermind 37435 points

Hi Paritosh,

Sorry for the delay. Please help me reproduce the issue

You mentioned that devmem2 0x4B22E000 hangs the A15 core but you also say that if you reboot the board from Linux you do not see this issue. How do I run devmem2 without rebooting on Linux ?

I didn't understand.

Do you mean that if you power cycle and let Linux boot then you see the issue but if you issue "reboot" command within Linux then you don't see the problem ?

Regards

Vineet

0 Paritosh Dixit over 3 years ago in reply to Vineet Roy

Intellectual 720 points

Hi Vineet,

Here are two scenarios:

1. Hard reset case

- Power cycle the relay

- after relay boots up, run devmem2 0x4B22E000

- A15 hangs

2. Soft reset case

- Power cycle the relay

- after relay boots up, reboot relay using reboot command (this does warm reset on the SoC/board)

- after relay boots up, run devmem2 0x4B22E000

- A15 does not hang. Start our FW on dsp1, pruss1, A15. Everything works fine.

Thanks,

Paritosh

0 Vineet Roy over 3 years ago in reply to Paritosh Dixit

TI__Mastermind 37435 points

Hi Paritosh,

I checked the Hard reset case on AM572 IDK with latest SDK and I do not see the issue (The value read back is 0x00000550). I have removed the PRU Firmware so it does not load but driver will still do some IEP configurations I guess.

Have you completely disabled all PRU initialization in your case ?

I will check with SDK 5.02 now

Regards

Vineet

0 Paritosh Dixit over 3 years ago in reply to Vineet Roy

Intellectual 720 points

Hi Vineet,

We removed PRU FW loading by removing PRU FW from /lib/firmware. We still see the issue (A15 hanging when accessing PRUSS1 IEP register).

Is there any other thing we can try diagnose issue in HW or FW?

Thanks,
Paritosh

0 Tom Schultz over 3 years ago in reply to Vineet Roy

TI__Expert 3895 points

Hi Vineet, what factors are you aware of that can cause a processor to hang? Can you separately list HW and SW factors that could result in this? Conversely, what has to be in place to allow the DSP core to operate properly? Thanks.

0 Vineet Roy over 3 years ago in reply to Paritosh Dixit

TI__Mastermind 37435 points

Hi Paritosh, Tom,

Sorry for the delay, I have so far been enable to reproduce this issue. I was looking for additional IDK's (other than those on my desk) and I couldn't simulate it on them.

I am now discussing this issue with the HW team on any potential race conditions involving IEP and PRU-ICSS which can cause such a crash, but it will be hard to debug the issue without reproducing this.

I will get back with the next debug steps on this thread.

Regards

Vineet

0 Vineet Roy over 3 years ago in reply to Vineet Roy

TI__Mastermind 37435 points

Hi Paritosh,

Are you using both PRU-ICSS1 and PRU-ICSS2 in your system ?

Regards

Vineet

0 Paritosh Dixit over 3 years ago in reply to Vineet Roy

Intellectual 720 points

Yes Vineet. Our system is using both both PRU-ICSS1 and PRU-ICSS2.

In the test, even if I removed FWs of PRU-ICSS2 (though both are enabled in PINMUX).

0 Vineet Roy over 3 years ago in reply to Paritosh Dixit

TI__Mastermind 37435 points

Hi Paritosh,

Thanks for confirming. Tom and our team will get in touch with you for the debug. I will leave the ticket open for a few more days but we will take it offline.

Regards

Vineet

0 Vineet Roy over 3 years ago in reply to Vineet Roy

TI__Mastermind 37435 points

Closing the ticket as debug is ongoing offline

Regards

Vineet

Processors

Processors forum

AM5708: A15, dsp1 hangs if try to access any register of IEP of PRU-ICSS1: omap_l3_noc 44000000.ocp: L3 application error: target 5 mod:1 (unclearable)