RTOS/AM5728: EtherCAT Master lockup issue

Ernest Cho

Part Number: AM5728

Tool/software: TI-RTOS

Hi all,

I am following up the customer lock up issue on the AM5728 MPU0 for their EtherCAT master board.

At that time, I tried to attach the MPU0 core but I can't attach the JTAG after lockup happened.
But at that time, JTAG message shows -1323 which noted pipeline stalled.

Customer and I are investigating the root cause, and checking the address right after -1323 @ 0x_xxxx_xxxx.
I think the address specified in the JTAG error log and its nearby address could be a key points.
The address is not fixed but it shows some functions so far.

At first, the address pointed terminal message function, so we removed UART terminal messages and its function.
And then it pointed cache_invalidate() function which is related OCMC shared memory. (actually they do process many data from PCIe and MPU0).
Sometimes, the address points at the customer's 3rd party's library, the address point out to the jump command.

One of question is,
What does pipeline stall means, why this pipeline stall issue happened and when?

Could you please help me with your opinion?

Best Regards,
Ernest Cho.

over 6 years ago

0 Biser Gatchev-XID over 6 years ago

TI__Guru**** 393215 points

The RTOS team have been notified. They will respond here.

0 Paula Carrillo over 6 years ago

TI__Mastermind 32515 points

Hi Ernest, which EtherCAT master was implemented? which Processor SDK version and tools are you using? could you please share the logs when the error happens?

thank you,
Paula

0 Ernest Cho over 6 years ago in reply to Paula Carrillo

TI__Intellectual 2961 points

Hello Paula,

Sorry for late response.

Customer use Acontis EtherCAT master library which is PDK version 1.0.2.
So Processor SDK version is v02.00.02.

Now they request new library for latest SDK version, but current they uses that version.

For the log, as I mentioned earlier, we don't have logs.
We just catched JTAG attach error message as below :

Debugger error: CortexA15_0: Trouble Halting Target CPU: (Error -1323 @ 0xoooo oooo) Device failed to enter debug/halt mode because pipeline is stalled. Power-cycle the board. If error persists, confirm configuration and/or try more reliable JTAG settings (e.g. lower TCLK). (Emulation package 6.0.504.1)

So, I want to know in more detail about pipeline stall?

Best Regards,
Ernest Cho

0 Paula Carrillo over 6 years ago in reply to Ernest Cho

TI__Mastermind 32515 points

Hi Ernest, let me get the right eyes to take a look on this issue.

thank you,
Paula

0 Ernest Cho over 6 years ago in reply to Paula Carrillo

TI__Intellectual 2961 points

Paula,

Ok. My question was the meaning of the "pipeline stall" and when this mostly happen. It would be helpful for identify the reason.

Any clue would be helpful.

Ernest

0 Paula Carrillo over 6 years ago in reply to Ernest Cho

TI__Mastermind 32515 points

Hi Ernest, is this happening on a custom board or in TI's AM5728 IDK board?
thank you,
Paula

0 Ernest Cho over 6 years ago in reply to Paula Carrillo

TI__Intellectual 2961 points

It happens on a customer board.
Their board is based on the idk.

0 Paula Carrillo over 6 years ago in reply to Ernest Cho

TI__Mastermind 32515 points

Hi Ernest, I got some comments from our experts about "pipeline stall" meaning in your customer case.

It seems that the system is “locked up” in this case and from the error message below the A15 is “locked up” as well. The A15 is effectively hung waiting for an instruction to complete—and because of this we are unable to put the CPU into a state that would allow debug.

Do you know if C66x’s is used in the customer’s system?. This question, is because C66x's debug logic includes some hooks to break-into the C66x’s CPU in cases where it is hung. We could also use the C66x to probe the system to figure out if part of the SOC infrastructure is hung.

Please let us know, thank you,
Paula

0 Ernest Cho over 6 years ago in reply to Paula Carrillo

TI__Intellectual 2961 points

Paula,
Yes, c66x can attach the debugger.
Customer currently use only MPU0.
We assume the address next to the 1323 error is the code and it cause the hang. But it is not a fixed address so far.
We attached c66x with debugger before but we didn't found useful data from MPU0.
How we found useful data using c66x jtag attached from MPU0?

Thanks.

0 Paula Carrillo over 6 years ago in reply to Ernest Cho

TI__Mastermind 32515 points

Hi Ernest, In this particular case, the C66x can’t be used to debug the A15. However, depending on how badly the SOC is locked up, we could use the C66x to access SOC resources to determine if they are functional, what state they are in or otherwise may provide some insight into what went wrong.

I will ask to move this thread to CCS forum, as they can help you better to debug this hang.

thank you,
Paula

0 desouza over 6 years ago in reply to Paula Carrillo

TI__Guru**** 168267 points

Ernest,

Paula already send additional details about the error itself, which is also described at the reference below:
software-dl.ti.com/.../ccsv7_debugging_jtag_connectivity_issues.html

In addition to the approach suggested, there is also a possibility to have complete access to the device memory via the Debug Access Port or DAP. Please check the short clip below for details.
https://youtu.be/-yGmq_VKvTQ

Due to the random nature of these lockup scenarios, it is very difficult to have a systematic approach to perform a "post mortem" analysis and therefore our options are quite limited.

I could also suggest you to enable the Exception traps, but since the processor is halted during a pipeline error, there is a chance the exception was not reached yet - thus the trap will never be activated.

If you are interested, check the thread below where I cover some additional debug details:
e2e.ti.com/.../2481873

Hope this helps,
Rafael

0 Ernest Cho over 6 years ago in reply to desouza

TI__Intellectual 2961 points

Hi Rafael,

Thank you for your reply.
I already sent those to my customer.
I will get back with customer's opinion.

Thank you.
Ernest Cho

0 Ernest Cho over 5 years ago in reply to Ernest Cho

TI__Intellectual 2961 points

Hello Rafael and Paula

Customer move the PDK source version from v1.0.2 to v1.0.9 and tested around 1week.
They said the issue is not reproduced so far.
Some restriction to handle between cache_invalidate and write_back function are added and we assume this could prevent the issue.

Anyway, Thank you so far.

0 Paula Carrillo over 5 years ago in reply to Ernest Cho

TI__Mastermind 32515 points

Ernest, thanks for let us know
Paula

Processors

Processors forum

RTOS/AM5728: EtherCAT Master lockup issue