Part Number: AM5718
Other Parts Discussed in Thread: DRA722
Hello,
We have examples of both the AM5718 and 5748 boards which occasionally experience SDHCI failures on mmc1 which corresponds to device tree overlay &mmc2. Note that we boot all the way into the login prompt before these errors occur, and in some cases they are on reboot before halting.
We are using an EMMC chip MTFC32GAZAQHD and running a 5.10 kernel, I have tried the following:
Disable BKOPS entirely
Disable EMMC caching
Limiting to SD HS mode via no-1-8-v
Reduce the bus width to 1
Applying IRQ locking patches from the upstream v6.2-rc4...601363cc08da25747feb87c55573dd54de91d66a (https://lore.kernel.org/linux-arm-kernel/20230308144105.di552lbogqv2s7fk@mraw.org/#r)
Here are the device tree overlays we have:
|
&mmc1 { vmmc-supply = <&ldo1_reg>;
&mmc2 {
bus-width = <1>; /delete-property/ mmc-hs200-1_8v;
|
This still occurs periodically. What other problems might be occuring here?
Note that I added a bit of extra logging to the kernel, including a stack trace at the watchdog error. The watchdog error is our biggest lead right now but I haven't tracked that down yet. It occurs only spuriously and does not correspond 100% to the SDHCI error
| Stopping Session c2 of user root. [ OK ] Removed slice system-getty.slice. [ OK ] Removed slice system-sshd.slice. [ OK ] Stopped target Network (Pre). [ OK ] Stopped target Host and Network Name Lookups. [ OK ] Stopped target Timers. [ OK ] Stopped Daily Cleanup of Temporary Directories. [ OK ] Stopped target Hardware activated USB gadget. Stopping Load/Save Random Seed... Stopping Network Name Resolution... Stopping weston - Running aGTX... [ OK ] Stopped OpenSSH Key Generation. [ OK ] Stopped Network Name Resolution. [ OK ] Stopped Serial Getty on ttyS0. [ OK ] Stopped weston - Running as GTX. [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Session c1 of user gtx. [ OK ] Stopped Session c2 of user root. [ OK ] Removed slice system-serial\x2dgetty.slice. [ OK ] Stopped target Graphical Interface. [ OK ] Stopped target Multi-User System. [ OK ] Stopped target Login Prompts. Stopping Launch GTX application service... [ OK ] Stopped IPv6 Packet Filtering Framework. [ OK ] Stopped IPv4 Packet Filtering Framework. Stopping Login Service... Stopping thermal-zone-init.service... Stopping User Manager for UID 0... Stopping User Manager for UID 1000... [ OK ] Stopped thermal-zone-init.service. Stopping sshd.service... [ OK ] Stopped Login Service. [ OK ] Stopped User Manager for UID 1000. [ OK ] Stopped User Manager for UID 0. [ OK ] Stopped sshd.service. [ OK ] Stopped Launch GTX application service. [ OK ] Stopped mount service. Stopping detect model service... Stopping rc.pvr.service... Stopping User Runtime Directory /run/user/0... Stopping User Runtime Directory /run/user/1000... [ OK ] Unmounted /run/user/0. [ OK ] Unmounted /run/user/1000. [ OK ] Stopped detect model service. [ OK ] Stopped rc.pvr.service. [ OK ] Stopped User Runtime Directory /run/user/0. [ OK ] Stopped User Runtime Directory /run/user/1000. [ OK ] Removed slice User Slice of UID 0. [ OK ] Removed slice User Slice of UID 1000. Stopping D-Bus System Message Bus... Stopping Permit User Sessions... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped Permit User Sessions. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Sockets. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Load Kernel Modules. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Unmounting /media/ram... Unmounting /mnt/software... Unmounting /mnt/storage... Unmounting Temporary Directory (/tmp)... Unmounting /var/volatile... [ OK ] Unmounted /media/ram. [ OK ] Unmounted /mnt/software. [ OK ] Unmounted /mnt/storage. [ OK ] Unmounted Temporary Directory (/tmp). [ OK ] Unmounted /var/volatile. [ OK ] Stopped target Local File Systems (Pre). [ OK ] Stopped target Swap. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped Remount Root and Kernel File Systems. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Reached target Shutdown. [ OK ] Reached target Final Step. [ OK ] Started Reboot. [ OK ] Reached target Reboot. [ 358.388580] watchdog: watchdog0: watchdog did not stop!! [ 358.394042] CPU: 0 PID: 1 Comm: systemd Tainted: G O 5.10.168-1-ctx #77 [ 358.402099] Hardware name: Generic DRA74X (Flattened Device Tree) [ 358.408294] [<c020eac4>] (unwind_backtrace) from [<c0209d44>] (show_stack+0x10/0x14) [ 358.416198] [<c0209d44>] (show_stack) from [<c0c258c0>] (dump_stack+0x94/0xa8) [ 358.423522] [<c0c258c0>] (dump_stack) from [<c09b58f4>] (watchdog_release+0x160/0x18c) [ 358.431579] [<c09b58f4>] (watchdog_release) from [<c036b45c>] (__fput+0x78/0x234) [ 358.439208] [<c036b45c>] (__fput) from [<c0248b48>] (task_work_run+0x90/0xbc) [ 358.446472] [<c0248b48>] (task_work_run) from [<c0209a6c>] (do_work_pending+0x54c/0x554) [ 358.454681] [<c0209a6c>] (do_work_pending) from [<c02000c4>] (slow_work_pending+0xc/0x20) [ 358.463012] Exception stack(0xc1897fb0 to 0xc1897ff8) [ 358.468139] 7fa0: 00000000 b6f7d0a4 00000000 00000000 [ 358.476440] 7fc0: 00000004 b6f7d5a0 00000000 00000006 055d4a80 00000000 000ac1b8 00112240 [ 358.484771] 7fe0: 00000006 bec11ba0 b6cc0ef5 b6c41ae6 80000030 00000004 [ 358.516601] systemd-shutdown[1]: Syncing filesystems and block devices. [ 358.530181] systemd-shutdown[1]: Sending SIGTERM to remaining processes... [ 358.544158] systemd-journald[146]: Received SIGTERM from PID 1 (systemd-shutdow). [ 358.571014] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 358.583618] systemd-shutdown[1]: Hardware watchdog 'OMAP Watchdog', version 0 [ 358.593872] systemd-shutdown[1]: Unmounting file systems. [ 358.602172] [1044]: Remounting '/' read-only in with options '(null)'. [ 358.621887] EXT4-fs (mmcblk1p3): re-mounted. Opts: (null) [ 358.630187] systemd-shutdown[1]: All filesystems unmounted. [ 358.635955] systemd-shutdown[1]: Deactivating swaps. [ 358.642486] systemd-shutdown[1]: All swaps deactivated. [ 358.648101] systemd-shutdown[1]: Detaching loop devices. [ 358.657592] systemd-shutdown[1]: All loop devices detached. [ 358.663391] systemd-shutdown[1]: Detaching DM devices. [ 358.669036] systemd-shutdown[1]: All DM devices detached. [ 358.674560] systemd-shutdown[1]: All filesystems, swaps, loop devices and DM devices detached. [ 358.698669] systemd-shutdown[1]: Syncing filesystems and block devices. [ 358.708801] systemd-shutdown[1]: Rebooting. [ 358.717834] mmc1: ===== EMMC EXT_CSD INFO ===== [ 358.724029] mmc1: CACHE_CTRL [33]: 0x03 00 [ 358.729858] mmc1: BKOPS_STATUS [246]: 0x00 00 [ 358.735870] mmc1: BKOPS_EN [163]: 0x00 00 00 [ 358.741821] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 01 [ 358.748748] mmc1: EXCEPTION_STATUS [54]: 0x01 00 [ 358.754974] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 0x39473030 54513936 ] [ 358.765533] mmc1: NOTE 0 calling suspend [ 358.771057] mmc1: ===== EMMC EXT_CSD INFO ===== [ 358.777130] mmc1: CACHE_CTRL [33]: 0x03 00 [ 358.782775] mmc1: BKOPS_STATUS [246]: 0x00 00 [ 358.788604] mmc1: BKOPS_EN [163]: 0x00 00 00 [ 358.794281] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 01 [ 358.801055] mmc1: EXCEPTION_STATUS [54]: 0x01 00 [ 358.807159] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 0x39473030 54513936 ] [ 358.824066] mmc1: 0 err = mmc_poweroff_notify(host->card, notify_type) [ 358.832092] mmc1: 0 mmc_power_off [ 358.838134] mmc1: 0 OUT mmc_release_host [ 369.137695] mmc1: Timeout waiting for hardware cmd interrupt. [ 369.144866] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 369.152709] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 369.160583] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 369.168457] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 369.176300] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 369.184173] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 369.192047] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 369.199890] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 369.207702] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 369.215515] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 369.223297] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 369.231079] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 369.238861] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 369.246643] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 369.254394] mmc1: sdhci: Host ctl2: 0x00000000 [ 369.260101] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 369.267852] Present State: 0x01f00000 [ 369.272766] DAT Line Active: NO [ 369.277282] Write Transfer: IDLE [ 369.281890] Read Transfer: IDLE [ 369.286376] DAT[3:0] Line Signal Level: 0xf [ 369.291931] mmc1: ===== EMMC EXT_CSD INFO ===== [ 369.297698] mmc1: CACHE_CTRL [33]: 0x03 [ 369.302734] mmc1: BKOPS_STATUS [246]: 0x00 [ 369.308013] mmc1: BKOPS_EN [163]: 0x00 [ 369.312896] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 369.318359] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 369.324523] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 369.332275] mmc1: sdhci: ============================================ [ 369.440155] mmc1: Reset 0x2 never completed. [ 369.445556] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 369.453155] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 369.460754] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 369.468383] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 369.475982] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 369.483581] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 369.491210] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 369.498809] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 369.506439] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 369.514038] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 369.521636] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 369.529266] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 369.536865] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 369.544494] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 369.552093] mmc1: sdhci: Host ctl2: 0x00000000 [ 369.557678] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 369.565246] Present State: 0x01f00000 [ 369.570037] DAT Line Active: NO [ 369.574493] Write Transfer: IDLE [ 369.579010] Read Transfer: IDLE [ 369.583435] DAT[3:0] Line Signal Level: 0xf [ 369.588897] mmc1: ===== EMMC EXT_CSD INFO ===== [ 369.594573] mmc1: CACHE_CTRL [33]: 0x03 [ 369.599548] mmc1: BKOPS_STATUS [246]: 0x00 [ 369.604766] mmc1: BKOPS_EN [163]: 0x00 [ 369.609649] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 369.615112] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 369.621307] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 369.629089] mmc1: sdhci: ============================================ [ 369.736663] mmc1: Reset 0x4 never completed. [ 369.742034] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 369.749633] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 369.757263] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 369.764862] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 369.772491] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 369.780120] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 369.787719] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 369.795349] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 369.802947] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 369.810546] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 369.818145] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 369.825744] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 369.833374] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 369.841003] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 369.848602] mmc1: sdhci: Host ctl2: 0x00000000 [ 369.854156] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 369.861755] Present State: 0x01f00000 [ 369.866546] DAT Line Active: NO [ 369.870971] Write Transfer: IDLE [ 369.875488] Read Transfer: IDLE [ 369.879913] DAT[3:0] Line Signal Level: 0xf [ 369.885375] mmc1: ===== EMMC EXT_CSD INFO ===== [ 369.891052] mmc1: CACHE_CTRL [33]: 0x03 [ 369.896026] mmc1: BKOPS_STATUS [246]: 0x00 [ 369.901245] mmc1: BKOPS_EN [163]: 0x00 [ 369.906127] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 369.911621] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 369.917785] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 369.925537] mmc1: sdhci: ============================================ [ 380.017669] mmc1: Timeout waiting for hardware cmd interrupt. [ 380.024566] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 380.032165] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 380.039794] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 380.047424] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 380.055023] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 380.062652] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 380.070251] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 380.077880] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 380.085479] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 380.093109] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 380.100708] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 380.108306] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 380.115936] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 380.123565] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 380.131164] mmc1: sdhci: Host ctl2: 0x00000000 [ 380.136749] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 380.144317] Present State: 0x01f00000 [ 380.149108] DAT Line Active: NO [ 380.153533] Write Transfer: IDLE [ 380.158081] Read Transfer: IDLE [ 380.162506] DAT[3:0] Line Signal Level: 0xf [ 380.167999] mmc1: ===== EMMC EXT_CSD INFO ===== [ 380.173675] mmc1: CACHE_CTRL [33]: 0x03 [ 380.178619] mmc1: BKOPS_STATUS [246]: 0x00 [ 380.183868] mmc1: BKOPS_EN [163]: 0x00 [ 380.188751] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 380.194244] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 380.200439] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 380.208190] mmc1: sdhci: ============================================ [ 380.315826] mmc1: Reset 0x2 never completed. [ 380.321197] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 380.328796] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 380.336395] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 380.344024] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 380.351654] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 380.359252] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 380.366882] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 380.374481] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 380.382110] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 380.389709] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 380.397308] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 380.404907] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 380.412536] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 380.420135] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 380.427734] mmc1: sdhci: Host ctl2: 0x00000000 [ 380.433319] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 380.440917] Present State: 0x01f00000 [ 380.445678] DAT Line Active: NO [ 380.450134] Write Transfer: IDLE [ 380.454650] Read Transfer: IDLE [ 380.459075] DAT[3:0] Line Signal Level: 0xf [ 380.464569] mmc1: ===== EMMC EXT_CSD INFO ===== [ 380.470214] mmc1: CACHE_CTRL [33]: 0x03 [ 380.475189] mmc1: BKOPS_STATUS [246]: 0x00 [ 380.480407] mmc1: BKOPS_EN [163]: 0x00 [ 380.485290] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 380.490783] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 380.496978] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 380.504730] mmc1: sdhci: ============================================ [ 380.612304] mmc1: Reset 0x4 never completed. [ 380.617675] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 380.625274] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 380.632904] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 380.640502] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 380.648132] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 380.655731] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 380.663360] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 380.670959] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 380.678558] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 380.686157] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 380.693786] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 380.701385] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 380.709014] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 380.716644] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 380.724243] mmc1: sdhci: Host ctl2: 0x00000000 [ 380.729797] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 380.737365] Present State: 0x01f00000 [ 380.742156] DAT Line Active: NO [ 380.746582] Write Transfer: IDLE [ 380.751129] Read Transfer: IDLE [ 380.755554] DAT[3:0] Line Signal Level: 0xf [ 380.761016] mmc1: ===== EMMC EXT_CSD INFO ===== [ 380.766693] mmc1: CACHE_CTRL [33]: 0x03 [ 380.771667] mmc1: BKOPS_STATUS [246]: 0x00 [ 380.776885] mmc1: BKOPS_EN [163]: 0x00 [ 380.781768] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 380.787231] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 380.793426] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 380.801177] mmc1: sdhci: ============================================ [ 390.897705] mmc1: Timeout waiting for hardware cmd interrupt. [ 390.904602] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 390.912200] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 390.919830] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 390.927429] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 390.935058] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 390.942657] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 390.950286] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 390.957916] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 390.965484] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 390.973114] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 390.980712] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 390.988311] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 390.995941] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 391.003570] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 391.011169] mmc1: sdhci: Host ctl2: 0x00000000 [ 391.016754] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 391.024322] Present State: 0x01f00000 [ 391.029113] DAT Line Active: NO [ 391.033569] Write Transfer: IDLE [ 391.038085] Read Transfer: IDLE [ 391.042541] DAT[3:0] Line Signal Level: 0xf [ 391.048004] mmc1: ===== EMMC EXT_CSD INFO ===== [ 391.053680] mmc1: CACHE_CTRL [33]: 0x03 [ 391.058654] mmc1: BKOPS_STATUS [246]: 0x00 [ 391.063873] mmc1: BKOPS_EN [163]: 0x00 [ 391.068756] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 391.074249] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 391.080413] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 391.088195] mmc1: sdhci: ============================================ [ 391.196044] mmc1: Reset 0x2 never completed. [ 391.201446] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 391.209045] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 391.216644] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 391.224273] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 391.231872] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 391.239501] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 391.247100] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 391.254730] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 391.262329] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 391.269927] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 391.277557] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 391.285156] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 391.292785] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 391.300415] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 391.308013] mmc1: sdhci: Host ctl2: 0x00000000 [ 391.313568] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 391.321166] Present State: 0x01f00000 [ 391.325958] DAT Line Active: NO [ 391.330383] Write Transfer: IDLE [ 391.334930] Read Transfer: IDLE [ 391.339355] DAT[3:0] Line Signal Level: 0xf [ 391.344848] mmc1: ===== EMMC EXT_CSD INFO ===== [ 391.350494] mmc1: CACHE_CTRL [33]: 0x03 [ 391.355468] mmc1: BKOPS_STATUS [246]: 0x00 [ 391.360717] mmc1: BKOPS_EN [163]: 0x00 [ 391.365570] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 391.371063] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 391.377227] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 391.385009] mmc1: sdhci: ============================================ [ 391.492584] mmc1: Reset 0x4 never completed. [ 391.497955] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 391.505554] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 391.513153] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 391.520782] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 391.528381] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 391.536010] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 391.543609] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 391.551239] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 391.558837] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 391.566436] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 391.574066] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 391.581634] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 391.589263] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 391.596893] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 391.604492] mmc1: sdhci: Host ctl2: 0x00000000 [ 391.610046] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 391.617645] Present State: 0x01f00000 [ 391.622406] DAT Line Active: NO [ 391.626861] Write Transfer: IDLE [ 391.631378] Read Transfer: IDLE [ 391.635803] DAT[3:0] Line Signal Level: 0xf [ 391.641296] mmc1: ===== EMMC EXT_CSD INFO ===== [ 391.646972] mmc1: CACHE_CTRL [33]: 0x03 [ 391.651916] mmc1: BKOPS_STATUS [246]: 0x00 [ 391.657165] mmc1: BKOPS_EN [163]: 0x00 [ 391.662048] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 391.667541] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 391.673706] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 391.681457] mmc1: sdhci: ============================================ [ 401.777709] mmc1: Timeout waiting for hardware cmd interrupt. [ 401.784606] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 401.792205] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 401.799804] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 401.807434] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 401.815063] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 401.822662] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 401.830291] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 401.837921] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 401.845520] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 401.853118] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 401.860717] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 401.868347] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 401.875946] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 401.883575] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 401.891174] mmc1: sdhci: Host ctl2: 0x00000000 [ 401.896759] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 401.904327] Present State: 0x01f00000 [ 401.909118] DAT Line Active: NO [ 401.913574] Write Transfer: IDLE [ 401.918090] Read Transfer: IDLE [ 401.922515] DAT[3:0] Line Signal Level: 0xf [ 401.928009] mmc1: ===== EMMC EXT_CSD INFO ===== [ 401.933685] mmc1: CACHE_CTRL [33]: 0x03 [ 401.938659] mmc1: BKOPS_STATUS [246]: 0x00 [ 401.943878] mmc1: BKOPS_EN [163]: 0x00 [ 401.948760] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 401.954254] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 401.960418] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 401.968200] mmc1: sdhci: ============================================ [ 402.076110] mmc1: Reset 0x2 never completed. [ 402.081512] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 402.089080] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 402.096710] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 402.104309] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 402.111938] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 402.119537] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 402.127166] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 402.134796] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 402.142395] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 402.149993] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 402.157592] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 402.165222] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 402.172821] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 402.180480] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 402.188079] mmc1: sdhci: Host ctl2: 0x00000000 [ 402.193634] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 402.201202] Present State: 0x01f00000 [ 402.205993] DAT Line Active: NO [ 402.210449] Write Transfer: IDLE [ 402.214965] Read Transfer: IDLE [ 402.219390] DAT[3:0] Line Signal Level: 0xf [ 402.224884] mmc1: ===== EMMC EXT_CSD INFO ===== [ 402.230529] mmc1: CACHE_CTRL [33]: 0x03 [ 402.235504] mmc1: BKOPS_STATUS [246]: 0x00 [ 402.240753] mmc1: BKOPS_EN [163]: 0x00 [ 402.245605] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 402.251098] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 402.257293] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 402.265045] mmc1: sdhci: ============================================ [ 402.372650] mmc1: Reset 0x4 never completed. [ 402.378021] mmc1: sdhci: ============ SDHCI REGISTER DUMP =========== [ 402.385620] mmc1: sdhci: Sys addr: 0x00000000 | Version: 0x00003302 [ 402.393218] mmc1: sdhci: Blk size: 0x00000000 | Blk cnt: 0x00000000 [ 402.400848] mmc1: sdhci: Argument: 0x00010000 | Trn mode: 0x00000000 [ 402.408447] mmc1: sdhci: Present: 0x01f00000 | Host ctl: 0x00000000 [ 402.416076] mmc1: sdhci: Power: 0x00000000 | Blk gap: 0x00000000 [ 402.423675] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000000 [ 402.431304] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000 [ 402.438903] mmc1: sdhci: Int enab: 0x007f0003 | Sig enab: 0x007f0003 [ 402.446533] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000 [ 402.454132] mmc1: sdhci: Caps: 0x25e90080 | Caps_1: 0x00000f77 [ 402.461730] mmc1: sdhci: Cmd: 0x00000d1a | Max curr: 0x00000000 [ 402.469360] mmc1: sdhci: Resp[0]: 0x00000000 | Resp[1]: 0x00000000 [ 402.476959] mmc1: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000 [ 402.484558] mmc1: sdhci: Host ctl2: 0x00000000 [ 402.490142] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000 [ 402.497711] Present State: 0x01f00000 [ 402.502502] DAT Line Active: NO [ 402.506958] Write Transfer: IDLE [ 402.511474] Read Transfer: IDLE [ 402.515899] DAT[3:0] Line Signal Level: 0xf [ 402.521392] mmc1: ===== EMMC EXT_CSD INFO ===== [ 402.527069] mmc1: CACHE_CTRL [33]: 0x03 [ 402.532012] mmc1: BKOPS_STATUS [246]: 0x00 [ 402.537261] mmc1: BKOPS_EN [163]: 0x00 [ 402.542114] mmc1: EXCEPTION_STATUS [54]: 0x01 [ 402.547607] mmc1: EXT_CSD_BKOPS_SUPPORT [0x1f6]: 0x00 [ 402.553802] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 402.561553] mmc1: sdhci: ============================================ [ 402.569763] mmc1: card 0001 removed |
# cat /sys/kernel/debug/mmc1/ios clock: 52000000 Hz vdd: 21 (3.3 ~ 3.4 V) bus mode: 2 (push-pull) chip select: 0 (don't care) power mode: 2 (on) bus width: 3 (8 bits) timing spec: 1 (mmc high-speed) signal voltage: 0 (3.30 V) driver type: 0 (driver type B)\ |
The errata i887 is taken into account as the frequency is less than 64 MHz. I'll try decreasing the frequency even more to 25MHz now and update tomorrow as this is spurious so it will take some time to determine if this is the fix.
Decompiled device tree portions relevant to MMC:
|
ocp { segment@0 { target-module@2000 {
scm_conf@0 { pbias_regulator@e00 { pbias_mmc_omap5 { pinmux@1400 { mmc1_pins_default { mmc1_pins_sdr12 { mmc1_pins_hs { mmc1_pins_sdr25 { mmc1_pins_sdr50 { mmc1_pins_ddr50 { mmc1_pins_sdr104 { mmc2_pins_default { mmc2_pins_hs { mmc2_pins_ddr_3_3v_rev11 { mmc2_pins_ddr_1_8v_rev11 { mmc2_pins_ddr_rev20 { mmc2_pins_hs200 { mmc4_pins_default { mmc4_pins_hs { mmc3_pins_default { mmc3_pins_hs { mmc3_pins_sdr12 { mmc3_pins_sdr25 { mmc3_pins_sdr50 { mmc4_pins_sdr12 { mmc4_pins_sdr25 { ... target-module@2000 { target-module@8000 { l3init-clkctrl@20 { ... interconnect@48000000 { segment@0 { ... i2c@0 { tps659038_pmic { regulators { ... ldo1 { }; target-module@5d000 { gpio@0 { mmc@0 { mmc@0 { mmc@0 { target-module@d1000 { mmc@0 { mmc1_iodelay_ddr_rev11_conf { mmc1_iodelay_ddr50_rev20_conf { mmc1_iodelay_sdr104_rev11_conf { mmc1_iodelay_sdr104_rev20_conf { mmc2_iodelay_hs200_rev11_conf { mmc2_iodelay_hs200_rev20_conf { mmc2_iodelay_ddr_3_3v_rev11_conf { mmc2_iodelay_ddr_1_8v_rev11_conf { mmc3_iodelay_manual1_conf { mmc4_iodelay_ds_rev11_conf { mmc4_iodelay_ds_rev20_conf { mmc4_iodelay_sdr12_hs_sdr25_rev11_conf { mmc4_iodelay_sdr12_hs_sdr25_rev20_conf { fixedregulator-vdd_3v3 { }; |
|
# mmc extcsd read /dev/mmcblk1 Card Supported Command sets [S_CMD_SET: 0x01]
|
Full boot log (in this one, it failed around login prompt time, as opposed to on shutdown as before. This was on the first boot of a new RFS, and when it fails to shut down it's after a new RFS install but before the reboot onto it (still on the old partition).
Note, I added some calls to print_info in :
_mmc_suspend
_mmc_resume
_mmc_hw_reset
So, the initial EXT_CSD output is not an error, it is new printing
Also, the "sync-clocks.service" failure happens consistently even when no SDHCI errors occur, so while I'm still tracking that down, I don't think this is the direct relation.
Boot fails on first boot of new RFS, if I power cycle the board afterwards it boots:
|
Starting kernel ... [ 0.000000] Booting Linux on physical CPU 0x0 Welcome to Linux! [ 9.058380] systemd[1]: Set hostname to <am57xx-comp>. [ 10.447570] ext4 filesystem being remounted at / supports timestamps until 2038 (0x7fffffff) _____ _____ _ _ Arago Project am57xx-comp [ 23.187805] davinci-mcasp8 48464000.mcasp: set PDIR +a0000000 +14000000... Arago 2021.09 am57xx-comp ttyS0 am57xx-comp login: [ 23.217895] davinci-mcasp8 48464000.mcasp: bus clock slave am57xx-comp login: root |
Fails after installing the new RFS but not yet switching partitions, if I power cycle it boots on new RFS
|
Starting kernel ... [ 0.000000] Booting Linux on physical CPU 0x0 Welcome to Linux! [ 9.058715] systemd[1]: Set hostname to <am57xx-comp>. [ 20.948089] mmc1: EXT_CSD_FIRMWARE_VERSION [0xfe]: 0x00000000 00000000 [ 0x39473030 54513936 ] |