AM623: U-Boot crashes when RTI watchdog is enabled

Hi Aaron,

I will try reproducing the enabling of watchdog and get back with my findings in 1-2 working days.

Regards,
Aparna

Hi Aaron,

Apologies for the delay!

I was not able to test this in the past few days. I will work on this and post an update by EOD tomorrow. 

Thank you for your patience.

Regards,
Aparna

Hi Aaron,

This error is not reproducible on TI EVM. Could you provide a diff of the changes you have in the U-boot source code apart from whats provided in the default SDK release.

Regards,
Aparna

Hi Aparna,

I'm Arnaud, working with Aaron. Aaron is in PTO at the moment and I took over this investigtation.

I dive a little bit in the crash report and I'm able to sched a bit more light.

First, our board is using an AM623x chip (AM6232 in my case, but we also have some AM6231 around).
This is probably why you're not able to reproduce the crash with the EVM. Is your EVM running a AM625 ? Or do you have the ability to put a AM623 in them?

The crash happens in:

```
U-Boot 2023.04-gd74d0993e2 (Oct 18 2023 - 17:49:46 +0000)

SoC: AM62X SR1.0 HS-FS
Model: XXXXXXX
DRAM: 1 GiB
Core: 63 devices, 30 uclasses, devicetree: separate
WDT: Started watchdog@e000000 without servicing (60s timeout)
WDT: Started watchdog@e010000 without servicing (60s timeout)
"Synchronous Abort" handler, esr 0x96000010
elr: 0000000080852bbc lr : 0000000080852bb8 (reloc)
elr: 00000000bff61bbc lr : 00000000bff61bb8
x0 : 00000000bdef4520 x1 : 000000000e020000
x2 : 0000000000000000 x3 : 00000000bff61ba4
x4 : 00000000bdedff38 x5 : 0000000000000010
x6 : 000000000000003c x7 : 000000000000000a
x8 : 00000000bdee1190 x9 : 0000000000000008
x10: 0000000000001f50 x11: 00000000bdedfcac
x12: 0000000000001f1c x13: 0000000000000000
x14: 00000000bdee1190 x15: 0000000000000002
x16: 00000000bff5db68 x17: 0000000000000000
x18: 00000000bdeefd90 x19: 000000000000ea60
x20: 00000000bdef2c70 x21: 00000000bffb924e
x22: 00000000000003e8 x23: 00000000deadbeef
x24: 0000000000000009 x25: 0000000000000000
x26: 00000000800c24ac x27: 0000000000000000
x28: 0000000000000000 x29: 00000000bdedff20

Code: f9000bf3 aa0103f3 97ff7f20 f9400001 (b9409021)
Resetting CPU ...
```

According to
```
└─> cat ../build/xxxxxxxx_a53_defconfig/u-boot.sym | sort | grep 0000000080852
...
0000000080852b40 l F .text_rest 0000000000000064 rti_wdt_reset
0000000080852ba4 l F .text_rest 00000000000000c8 rti_wdt_start
0000000080852c6c l F .text_rest 0000000000000084 rti_wdt_probe
...
```
we're crashing in 0000000080852ba4+0x14 in rti_wdt_start().

Adding a quick printf in this method.

Looks like we're calling the method three times but we only have two cores in this chip, so only two watchdogs.

A couple more printf are telling me we're crashing in `if (readl(priv->regs + RTIDWDCTRL) == WDENABLE_KEY)`, at the 3rd invocatoin (for `main_rti2` which is likely not present in our chip).

I added the following line in our "custommachine.dts":

```
&cbass_main {
/delete-node/ watchdog@e010000;
/delete-node/ watchdog@e020000;
/delete-node/ watchdog@e030000;
/delete-node/ watchdog@e0f0000;
};
```

and this is now avoiding the crash problem.

Two questions:

Q1:

Our custommachine.dts is doing an `#include "k3-am625.dtsi` and is then trying to remove all the HW non-applicable to AM623.
And this is tedious. We're discovering problems one after another, crash after crash.
Is there a proper "dtsi" in u-boot-ti-staging that we should be using for an AM623x chip ?
Is there some application notes regarding u-boot and kernel for an AM62x chip that is not 625 ?

Q2:

Now that it's not crashing. I still can't configure and trigger the watchdog with the `wdt` command.
I need to dive in Aaron's notes. I believe he was successful with those commands on am62xx-evm (but I'm not sure).
Do you have an idea of what's the problem could be?

```
U-Boot 2023.04-gd74d0993e2 (Oct 18 2023 - 17:49:46 +0000)

SoC: AM62X SR1.0 HS-FS
Model: Custom
DRAM: 1 GiB
Core: 59 devices, 30 uclasses, devicetree: separate
rti_wdt_start( dev=00000000bdef2b30 ), priv=00000000bdef4310
WDT: Started watchdog@e000000 without servicing (60s timeout)
MMC: mmc@fa10000: 0
Loading Environment from FAT... OK
Loading Environment from nowhere... OK
In: serial@2800000
Out: serial@2800000
Err: serial@2800000
Net: eth0: ethernet@8000000port@1
Autoboot in 4 seconds

=> wdt list
watchdog@e000000 (rti_wdt)

=> wdt dev watchdog@e000000

=> wdt start 30000
rti_wdt_start( dev=00000000bdef2b30 ), priv=00000000bdef4310
Starting watchdog timer failed (-16)

=> wdt expire
rti_wdt_start( dev=00000000bdef2b30 ), priv=00000000bdef4310
Expiring watchdog timer failed (-16)

=> wdt help
wdt - Watchdog sub-system

Usage:
wdt list - list watchdog devices
wdt dev [<name>] - get/set current watchdog device
wdt start <timeout ms> [flags] - start watchdog timer
wdt stop - stop watchdog timer
wdt reset - reset watchdog timer
wdt expire [flags] - expire watchdog timer immediately
```

Looking for how to attach (upload) our custommachine.dts into this E2E forum.

I cannot find a way to upload a file. Is there one?
Copying the whole DTS here. Loosing the indentation though :(.

/*
* Custom board
*
* Copyright (C) 2023 ...
*
* based on TI's AM625 SK:
* Copyright (C) 2021-2022 Texas Instruments Incorporated - https://www.ti.com/
*
* This file has been created as a clone of `arch/arm/dts/k3-am625-sk.dts`.
* We then started stripping the unnecessary node from the tree, such as:
* - `ospi0` (the NOR flash)
* - multimedia nodes were already removed
* - `mmc1`, `sdhci1` nodes (as there's no SD card)
* - `i2c1` and `i2c2` buses
* - the `cpu2` and `cpu3`
* ...
*/

/dts-v1/;

#include <dt-bindings/leds/common.h>
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/net/ti-dp83867.h>
#include "k3-am625.dtsi"

/ {
compatible = "ti,am625-sk", "ti,am625";
model = "CUSTOM";

aliases {
serial2 = &main_uart0;
mmc0 = &sdhci0;
mmc2 = &sdhci2;
ethernet0 = &cpsw_port1;
ethernet1 = &cpsw_port2;
usb0 = &usb0;
usb1 = &usb1;
rtc0 = &mcp7940n;
};

// The k3-am625.dtsi is declaring 4 cores, the chip on custom board only has two
// deleting cores/cpus 2 and 3
cpus {
cpu-map {
cluster0 {
/delete-node/ core2;
/delete-node/ core3;
};
};
/delete-node/ cpu@2;
/delete-node/ cpu@3;
};

chosen {
stdout-path = "serial2:115200n8";
bootargs = "console=ttyS2,115200n8 earlycon=ns16550a,mmio32,0x02800000";
};

opp-table {
/* Add 1.4GHz OPP for am625-sk board. Requires VDD_CORE to be at 0.85V */
opp-1400000000 {
opp-hz = /bits/ 64 <1400000000>;
opp-supported-hw = <0x01 0x0004>;
clock-latency-ns = <6000000>;
};
};

memory@80000000 {
device_type = "memory";
/* 1G RAM */
reg = <0x00000000 0x80000000 0x00000000 0x40000000>;

};

reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;

ramoops@9ca00000 {
compatible = "ramoops";
reg = <0x00 0x9ca00000 0x00 0x00100000>;
record-size = <0x8000>;
console-size = <0x8000>;
ftrace-size = <0x00>;
pmsg-size = <0x8000>;
};

secure_tfa_ddr: tfa@9e780000 {
reg = <0x00 0x9e780000 0x00 0x80000>;
alignment = <0x1000>;
no-map;
};

secure_ddr: optee@9e800000 {
reg = <0x00 0x9e800000 0x00 0x01800000>; /* for OP-TEE */
alignment = <0x1000>;
no-map;
};

wkup_r5fss0_core0_dma_memory_region: r5f-dma-memory@9db00000 {
compatible = "shared-dma-pool";
reg = <0x00 0x9db00000 0x00 0xc00000>;
no-map;
};
};

vmain_pd: regulator-0 {
/* TPS65988 PD CONTROLLER OUTPUT */
compatible = "regulator-fixed";
regulator-name = "vmain_pd";
regulator-min-microvolt = <5000000>;
regulator-max-microvolt = <5000000>;
regulator-always-on;
regulator-boot-on;
};

vcc_5v0: regulator-1 {
/* Output of LM34936 */
compatible = "regulator-fixed";
regulator-name = "vcc_5v0";
regulator-min-microvolt = <5000000>;
regulator-max-microvolt = <5000000>;
vin-supply = <&vmain_pd>;
regulator-always-on;
regulator-boot-on;
};

vcc_3v3_sys: regulator-2 {
/* output of LM61460-Q1 */
compatible = "regulator-fixed";
regulator-name = "vcc_3v3_sys";
regulator-min-microvolt = <3300000>;
regulator-max-microvolt = <3300000>;
vin-supply = <&vmain_pd>;
regulator-always-on;
regulator-boot-on;
};

// GPIO0.31 is controlling PMIC_VSEL_SD, allowing to switch between 1.8V and 3.3V
vdd_sd_dv: regulator-4 {
/* Output of TLV71033 */
compatible = "regulator-gpio";
regulator-name = "tlv71033";
pinctrl-names = "default";
pinctrl-0 = <&vdd_sd_dv_pins_default>;
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <3300000>;
regulator-boot-on;
vin-supply = <&vcc_5v0>;
gpios = <&main_gpio0 31 GPIO_ACTIVE_HIGH>;
states = <1800000 0x0>,
<3300000 0x1>;
};

leds {
compatible = "gpio-leds";
pinctrl-names = "default";
pinctrl-0 = <&usr_led_pins_default>;

led-0 {
label = "am62-sk:green:heartbeat";
gpios = <&main_gpio1 49 GPIO_ACTIVE_HIGH>;
linux,default-trigger = "heartbeat";
function = LED_FUNCTION_HEARTBEAT;
default-state = "off";
};
};
};

&main_pmx0 {
main_uart0_pins_default: main-uart0-pins-default {
pinctrl-single,pins = <
AM62X_IOPAD(0x1c8, PIN_INPUT, 0) /* (D14) UART0_RXD */
AM62X_IOPAD(0x1cc, PIN_OUTPUT, 0) /* (E14) UART0_TXD */
AM62X_IOPAD(0x01d0, PIN_INPUT, 0) /* (A15) UART0_CTSn */
AM62X_IOPAD(0x01d4, PIN_OUTPUT, 0) /* (B15) UART0_RTSn */
>;
};

main_i2c0_pins_default: main-i2c0-pins-default {
pinctrl-single,pins = <
AM62X_IOPAD(0x1e0, PIN_INPUT_PULLUP, 0) /* (B16) I2C0_SCL */
AM62X_IOPAD(0x1e4, PIN_INPUT_PULLUP, 0) /* (A16) I2C0_SDA */
>;
};

main_mmc0_pins_default: main-mmc0-pins-default {
pinctrl-single,pins = <
AM62X_IOPAD(0x220, PIN_INPUT, 0) /* (Y3) MMC0_CMD */
AM62X_IOPAD(0x218, PIN_INPUT, 0) /* (AB1) MMC0_CLK */
AM62X_IOPAD(0x214, PIN_INPUT, 0) /* (AA2) MMC0_DAT0 */
AM62X_IOPAD(0x210, PIN_INPUT, 0) /* (AA1) MMC0_DAT1 */
AM62X_IOPAD(0x20c, PIN_INPUT, 0) /* (AA3) MMC0_DAT2 */
AM62X_IOPAD(0x208, PIN_INPUT, 0) /* (Y4) MMC0_DAT3 */
AM62X_IOPAD(0x204, PIN_INPUT, 0) /* (AB2) MMC0_DAT4 */
AM62X_IOPAD(0x200, PIN_INPUT, 0) /* (AC1) MMC0_DAT5 */
AM62X_IOPAD(0x1fc, PIN_INPUT, 0) /* (AD2) MMC0_DAT6 */
AM62X_IOPAD(0x1f8, PIN_INPUT, 0) /* (AC2) MMC0_DAT7 */
>;
};

main_mmc1_pins_default: main-mmc1-pins-default {
pinctrl-single,pins = <
AM62X_IOPAD(0x23c, PIN_INPUT, 0) /* (A21) MMC1_CMD */
AM62X_IOPAD(0x234, PIN_INPUT, 0) /* (B22) MMC1_CLK */
AM62X_IOPAD(0x230, PIN_INPUT, 0) /* (A22) MMC1_DAT0 */
AM62X_IOPAD(0x22c, PIN_INPUT, 0) /* (B21) MMC1_DAT1 */
AM62X_IOPAD(0x228, PIN_INPUT, 0) /* (C21) MMC1_DAT2 */
AM62X_IOPAD(0x224, PIN_INPUT, 0) /* (D22) MMC1_DAT3 */
AM62X_IOPAD(0x240, PIN_INPUT, 0) /* (D17) MMC1_SDCD */
>;
};

usr_led_pins_default: usr-led-pins-default {
pinctrl-single,pins = <
AM62X_IOPAD(0x244, PIN_OUTPUT, 7) /* (C17) MMC1_SDWP.GPIO1_49 */
>;
};

main_mdio1_pins_default: main-mdio1-pins-default {
pinctrl-single,pins = <
AM62X_IOPAD(0x160, PIN_OUTPUT, 0) /* (AD24) MDIO0_MDC */
AM62X_IOPAD(0x15c, PIN_INPUT, 0) /* (AB22) MDIO0_MDIO */
>;
};

main_rmii1_pins_default: main-rmii1-pins-default {
status = "okay";
pinctrl-single,pins = <
AM62X_IOPAD(0x0130, PIN_INPUT, 1) /* (AE19) RGMII1_TXC.RMII1_CRS_DV */
AM62X_IOPAD(0x0148, PIN_INPUT, 1) /* (AD17) RGMII1_RXC.RMII1_REF_CLK */
AM62X_IOPAD(0x014c, PIN_INPUT, 1) /* (AB17) RGMII1_RD0.RMII1_RXD0 */
AM62X_IOPAD(0x0150, PIN_INPUT, 1) /* (AC17) RGMII1_RD1.RMII1_RXD1 */
AM62X_IOPAD(0x0144, PIN_INPUT, 1) /* (AE17) RGMII1_RX_CTL.RMII1_RX_ER */
AM62X_IOPAD(0x0134, PIN_OUTPUT, 1) /* (AE20) RGMII1_TD0.RMII1_TXD0 */
AM62X_IOPAD(0x0138, PIN_OUTPUT, 1) /* (AD20) RGMII1_TD1.RMII1_TXD1 */
AM62X_IOPAD(0x012c, PIN_OUTPUT, 1) /* (AD19) RGMII1_TX_CTL.RMII1_TX_EN */
AM62X_IOPAD(0x0154, PIN_INPUT_PULLUP, 7) /* (AB16) RGMII1_RD2.GPIO0_83 */
>;
};

main_rmii2_pins_default: main-rmii2-pins-default {
status = "okay";
pinctrl-single,pins = <
AM62X_IOPAD(0x0168, PIN_INPUT, 1) /* (AE21) RGMII2_TXC.RMII2_CRS_DV */
AM62X_IOPAD(0x0180, PIN_INPUT, 1) /* (AD23) RGMII2_RXC.RMII2_REF_CLK */
AM62X_IOPAD(0x0184, PIN_INPUT, 1) /* (AE23) RGMII2_RD0.RMII2_RXD0 */
AM62X_IOPAD(0x0188, PIN_INPUT, 1) /* (AB20) RGMII2_RD1.RMII2_RXD1 */
AM62X_IOPAD(0x017c, PIN_INPUT, 1) /* (AD22) RGMII2_RX_CTL.RMII2_RX_ER */
AM62X_IOPAD(0x016c, PIN_OUTPUT, 1) /* (Y18) RGMII2_TD0.RMII2_TXD0 */
AM62X_IOPAD(0x0170, PIN_OUTPUT, 1) /* (AA18) RGMII2_TD1.RMII2_TXD1 */
AM62X_IOPAD(0x0164, PIN_OUTPUT, 1) /* (AA19) RGMII2_TX_CTL.RMII2_TX_EN */
AM62X_IOPAD(0x0158, PIN_INPUT_PULLUP, 7) /* (AA15) RGMII1_RD3.GPIO0_84 */
>;
};

vdd_sd_dv_pins_default: vdd-sd-dv-pins-default {
pinctrl-single,pins = <
AM62X_IOPAD(0x07c, PIN_OUTPUT, 7) /* (P25) GPMC0_CLK.GPIO0_31 */
>;
};

main_usb1_pins_default: main-usb1-pins-default {
pinctrl-single,pins = <
AM62X_IOPAD(0x0258, PIN_OUTPUT, 0) /* (F18) USB1_DRVVBUS */
>;
};
};

&wkup_uart0 {
/* WKUP UART0 is used by DM firmware */
status = "reserved";
};

&main_uart0 {
status = "okay";
pinctrl-names = "default";
pinctrl-0 = <&main_uart0_pins_default>;
};

&main_uart1 {
/* Main UART1 is used by TIFS firmware */
status = "reserved";
};

&main_i2c0 {
status = "okay";
pinctrl-names = "default";
pinctrl-0 = <&main_i2c0_pins_default>;
clock-frequency = <400000>;

mcp7940n: mcp7940n@6f {
compatible = "microchip,mcp7940x";
reg = <0x6f>;
};
};

&main_i2c1 {
status = "disabled";
};

&cbass_main {
/delete-node/ watchdog@e010000;
/delete-node/ watchdog@e020000;
/delete-node/ watchdog@e030000;
/delete-node/ watchdog@e0f0000;
};

&sdhci0 {
status = "okay";
pinctrl-names = "default";
pinctrl-0 = <&main_mmc0_pins_default>;
ti,driver-strength-ohm = <50>;
disable-wp;
};

&sdhci1 {
/* SD/MMC */
status = "disabled";
};

&cpsw3g {
pinctrl-names = "default";
pinctrl-0 = <&main_rmii1_pins_default
&main_rmii2_pins_default>;
};

&cpsw_port1 {
phy-mode = "rmii";
phy-handle = <&cpsw3g_phy0>;
};

&cpsw_port2 {
phy-mode = "rmii";
phy-handle = <&cpsw3g_phy1>;
};

&cpsw3g_mdio {
status = "okay";
pinctrl-names = "default";
pinctrl-0 = <&main_mdio1_pins_default>;

cpsw3g_phy0: ethernet-phy@0 {
reg = <0>;
};

cpsw3g_phy1: ethernet-phy@1 {
reg = <1>;
};
};

&mailbox0_cluster0 {
mbox_m4_0: mbox-m4-0 {
ti,mbox-rx = <0 0 0>;
ti,mbox-tx = <1 0 0>;
};
};

&usbss0 {
status = "okay";
ti,vbus-divider;
};

&usbss1 {
status = "okay";
};

&usb0 {
/* Using CONFIG_LPP_USB_RECOVERY to determine if the USB device should be host or
peripheral mode. This U-Boot config should only be present in USBMSC defconfigs. */
#ifdef CONFIG_LPP_USB_RECOVERY
dr_mode = "host";
#else
dr_mode = "peripheral";
#endif
};

&usb1 {
dr_mode = "host";
pinctrl-names = "default";
pinctrl-0 = <&main_usb1_pins_default>;
};

```

Hello,

Yes, you can upload a file like this:

It is a bit tricky, you have to click on the grey "upload" word to select a file for upload:

You can also put terminal output in the code block so it doesn't take up as much space:

Regards,

Nick

Hello Arnold,

Apologies for the delayed responses, I have been on vacation and Aparna will also be out the rest of the week. Please keep in mind that I have NOT tried using watchdog during uboot, so take these responses with a grain of salt.

Watchdog is already enabled in your boot log

Taking a look at your uboot boot log, it looks like the watchdog is actually getting enabled before you reach the uboot command line:

rti_wdt_start( dev=00000000bdef2b30 ), priv=00000000bdef4310
WDT: Started watchdog@e000000 without servicing (60s timeout)

This also makes sense when you look at the Linux error code 16 (EBUSY - device or resource busy). I would expect to see this error if you were trying to configure or start a watchdog that was already running:

=> wdt start 30000
rti_wdt_start( dev=00000000bdef2b30 ), priv=00000000bdef4310
Starting watchdog timer failed (-16)

=> wdt expire
rti_wdt_start( dev=00000000bdef2b30 ), priv=00000000bdef4310
Expiring watchdog timer failed (-16)

I assume that "without servicing" means that there is not any software that is programmed to pet the watchdog. If that is true, then it should be very easy to check that the watchdog is actually running.

wdt expire returning error 16 is interesting. It could be that specific command is not compatible with the TI watchdog, or maybe you need to use a specific flag to get it to work properly.

Ok, so... How do I check that the watchdog is actually working? 

I assume that "started watchdog without servicing" means that the watchdog is counting down from 60 seconds, but uboot has NOT started any code that would pet the watchdog before the watchdog expires.

The quick test

If that is true, it should be easy to test if the watchdog is working as expected. Just enter the uboot terminal and... wait for 60 seconds. Did the processor reset? Great, your watchdog is working as expected. If not, then you'll need to do some additional debug.

A bit more granular - checking the watchdog registers 

You can use the md command to read watchdog register values and see what is going on. For more information, try
=> help md

CFG_WDSTATUS - watchdog status register
More information here:
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1370422/am62p-am62p/5253338#5253338

CFG_DWDCNTR -  Digital Watchdog Down Counter

You should see this number decreasing over time if the watchdog is actually running. You can get the number of seconds remaining by dividing the counter value by the frequency, 32,768 Hz.

Regards,

Nick