AM5748: Watchdog Timer Does Not Trigger on System Hang

Hello,

Does the customer have an E2E account?

I will look into this and likely not have an answer until next week.

-Josue

Field Apps has confirmed that the customer can access this thread. 

TI-CSC said:

Both AM5748 cores is running its own watchdog process that handles a different watchdog timer in case one of the cores hang when running a process/kernel thread.

Can you expand on this sentence? Does this refer to separate A15 cores or another processor in the SoC?

Could you please share what SDK release(s) are being used? 

How are your Watchdogs enabled?

-Josue

There are 3 watchdog timers in the Linux kernel: /dev/watchdog, /dev/watchdog0, /dev/watchdog1. Not sure why I can't really do anything with /dev/watchdog0. So, we have these 2 processes running:

/bin/watchdog /dev/watchdog - A15 core0
/bin/watchdog /dev/watchdog1 - A15 core1

We are not using your SDK. But we did start off from this SDK ti-processor-sdk-linux-am57xx-hs-evm-08_02_01_00.

Watchdogs are enabled as stated above.

Maynard Cabiente ,

So is your kernel based on Linux kernel 5.10?

What address are the watchdogs mapped to? One of them is not accessible because it is reserved for the security purposes. So only one available is WD_TIMER2 as stated in TRM.

How are these setup in DTS?

Here is a thread that has some clues to a possible implementation:

 RE: AM5728: Issues with WDT enabled in U-boot 

-Josue

Our Linux kernel is 6.12.34.

This is the list of the watchdog devices and their identity.

# ls -lt /dev/watchdog*
crw------- 1 root root 248, 1 Dec 31 1969 /dev/watchdog1
crw------- 1 root root 10, 130 Dec 31 1969 /dev/watchdog
crw------- 1 root root 248, 0 Dec 31 1969 /dev/watchdog0

# cat /sys/class/watchdog/watchdog0/identity
Software Watchdog
# cat /sys/class/watchdog/watchdog1/identity
OMAP Watchdog

I have to check if /dev/watchdog is somehow connected tto /dev/watchdog0, which is only a Sofware Watchdog.

In terms of kernel DTS, this is the only thing I found about watchdog entry included in our board DTS from "dra74-ipu-dsp-common.dtsi".

&dsp2 {
mboxes = <&mailbox6 &mbox_dsp2_ipc3x>;
ti,timers = <&timer6>;
ti,watchdog-timers = <&timer13>;
};

Thanks for the link. I already have the watchdog configured in U-Boot and working. Our issue is in the kernel and when a software reboot is issued and then system hangs forever.

In regards to other WATCHDOG TIMERS in AM5748, the TRM said that there is one timer and watchdog timer per MPU core. What is the MPU core? Is this the A15 core0 and core1? If it is, how do I access both watchdog timers?

There is also a Secure Watchdog Timer (as you have mentioned). We do use the Secure-Boot Option of the AM5748 chip. Is there a way for us to use this in our application so that we actually have 2 hardware Wathdog TImers and not rely on 1 Sofware Watchdog Timer?

-Maynard

Maynard Cabiente ,

The Secure watchdog is not open for modification. I believe the secure documentation only goes over how to modify its response through ISW config knobs.

Wrt watchdogs in Linux, I do not have a ready example and this would take me a little bit of time.

I currently do not have bandwidth to help here. Ill try to respond by next week.

Meanwhile see the following thread:  Linux/AM5718: WDTimer stops at kernel start 

-Josue

We do have watchdog timers setup in Linux. Unless you actually have to do something for the MPU watchdog timers, then watchdog timers are already supported in Linux.

Again, our issue is that the watchdog timer does not reset the AM5748 when it hangs after a software reboot. We need to know if there is a hardware fix to this or if there is something that we need to do in software so that the watchdog timer asserts the RSTOUTn pin so that it resets the system.

Regards,
Maynard

Hi Maynard,

what do you see in the kernel log after you issue reboot command? could you share the log? perhaps alongside the boot log to see WD initialization etc

thanks

Michael

Unfortunately, there is no log after issuing the reboot command as seen below.

# reboot
[hangs here with no output]

During boot-up, this is the log regarding watchdog timers:

[ 1.105987] softdog: initialized. soft_noboot=0 soft_margin=60 sec soft_panic=0 (nowayout=1)
[ 1.114685] softdog: soft_reboot_cmd=<not set> soft_active_on_boot=0

[ 1.711364] OMAP GPIO hardware version 0.1
[ 1.718841] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec

-Maynard

that's not a lot to go by

From what I am able to gather the likely reason is a "deep" kernel panic due to possible unhandled memory fault that could potentially mess up the WD. 

did you try increasing verbosity (verbosity=7) in command line arguments? could potentially produce some useful kernel messages 

some people suggested adding "panic=60" to the command line arguments to force hard reset on timeout (60sec).

the best (IMO) thing you could  do to pinpoint the issue would be to hook-up JTAG and read WD registers in the hung state (and for quick comparison in operational state). if the WD config is "healthy" in both cases the issue is more likely to be HW related. otherwise it would be indicative of whatever made kernel freeze also messed up WD configuration.  I would reiterate that reviewing schematic is very useful to rule out possible HW issues. 

regards,

Michael

Maynard Cabiente ,

I will revisit this thread next week.

Thank you for your patience!

-Josue

 Maynard Cabiente ,

From the threads I shared, there is som evidence that the WDT is disabled along the way to Linux boot for development purposes. Have you made any changes to u-boot or Linux in order to make the watchdogs available?

Have you tried the following: https://unix.stackexchange.com/questions/21335/how-do-i-cause-a-watchdog-reset-of-my-embedded-linux-device
    

Hi Josue,

There is no issue with the setup of watchdog(s) (hardware and software) in our Linux system. I have tested both where the watchdog timer actually triggers like putting a while(1) loop in a kernel driver or using sysrq trigger.

Again, the issue is that the watchdog timer should reset the system if there is a system or kernel hang in our system when this happens:

"We have an FPGA connected to PCIe that does DMA to the CPU memory and is used for transferring video data. Whenever the FPGA is running and data transfers maybe occurring and we just call the Linux reboot command, the system will hang and never reboots."

The watchdog timer should have save us in this situation but it is not triggering the reset.

Regards,
Maynard

Maynard,

I think I finally understand, so the watchdog in Linux is actually working, 

Maynard Cabiente said:

I have tested both where the watchdog timer actually triggers like putting a while(1) loop in a kernel driver or using sysrq trigger.

It is just in this specific scenario

Maynard Cabiente said:

"We have an FPGA connected to PCIe that does DMA to the CPU memory and is used for transferring video data. Whenever the FPGA is running and data transfers maybe occurring and we just call the Linux reboot command, the system will hang and never reboots

that you enter into a hang and the WDT does not actually trigger the reboot as expected.

Sorry the lag in my understanding. This was not clear to me.

At this point, the advice from Michael Shklyarman is the way to go.

-Josue