This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM623: stuck in SD card after boot-up

Part Number: AM623
Other Parts Discussed in Thread: AM625

Dear Champs,

My customer is working to bring-up their custom board and succeed to boot-up using OSPI flash memory, but faced stuck on SD Card with below log.

Could you please check below log and provide your opinion what should be checked?

~~~

ERROR:   Unhandled External Abort received on 0x80000000 from S-EL1

ERROR:   exception reason=1 syndrome=0x92000010

Unhandled Exception from EL1

~~~~

Full log is as below.

U-Boot SPL 2021.01 (Apr 19 2023 - 09:26:54 +0900)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.5.3--v08.05.03 (Chill Capybar')
SPL initial stack usage: 13424 bytes
Trying to boot from MMC2
Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was also encrypted
Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was also encrypted
Loading Environment from MMC... *** Warning - No MMC card found, using default environment

Starting ATF on ARM64 core...

NOTICE:  BL31: v2.7(release):v2.7.0-359-g1309c6c805-dirty
NOTICE:  BL31: Built : 11:48:12, Dec 14 2022
I/TC:
I/TC: OP-TEE version: 3.19.0-15-gd6c5d0037 (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10))) #1 Wed Dec 14 11:52:03 UTC 2022 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.5.3--v08.05.03 (Chill Capybar')
I/TC: HUK Initialized
I/TC: Activated SA2UL device
I/TC: Fixing SA2UL firewall owner for GP device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.01 (Apr 19 2023 - 09:26:20 +0900)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.5.3--v08.05.03 (Chill Capybar')
Trying to boot from MMC2
i2c_write: error waiting for data ACK (status=0x116)
pca953x gpio@22: Error reading output register


U-Boot 2021.01 (Apr 19 2023 - 09:26:20 +0900)

SoC:   AM62X SR1.0 GP
Model: Texas Instruments AM625 SK
Timed out in wait_for_event: status=0000
Check if pads/pull-ups of bus are properly configured
EEPROM not available at 0x50, trying to read at 0x51
Timed out in wait_for_event: status=0000
Check if pads/pull-ups of bus are properly configured
Reading on-board EEPROM at 0x51 failed 1
DRAM:  2 GiB
MMC:   mmc@fa10000: 0, mmc@fa00000: 1, mmc@fa20000: 2
Loading Environment from MMC... Card did not respond to voltage select! : -110
*** Warning - No block device, using default environment

In:    serial@2800000
Out:   serial@2800000
Err:   serial@2800000
EEPROM not available at 0x50, trying to read at 0x51
Reading on-board EEPROM at 0x51 failed 1
Net:   Could not get PHY for ethernet@8000000port@1: addr 0
am65_cpsw_nuss_port ethernet@8000000port@1: phy_connect() failed
No ethernet found.

[common/autoboot.c:bootdelay_process:357] bootcmd: [run distro_bootcmd]
[common/autoboot.c:abortboot_single_key:254] TRACE:
Hit any key to stop autoboot:  0
i2c_write: error waiting for data ACK (status=0x116)
pca953x gpio@22: Error reading output register
switch to partitions #0, OK
mmc1 is current device
i2c_write: error waiting for data ACK (status=0x116)
pca953x gpio@22: Error reading output register
SD/MMC found on device 1
Failed to load 'boot.scr'
46 bytes read in 4 ms (10.7 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
Running uenvcmd ...
KISAN u-boot-0.0.1 2023-03-23
19440128 bytes read in 848 ms (21.9 MiB/s)
35215 bytes read in 8 ms (4.2 MiB/s)
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Device Tree to 000000008fef4000, end 000000008fffffff ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 5.10.153 (root@ubuntu18) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 9.2.1 20191025, GNU ld (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 2.33.1.20191209) #10 SMP PREEMPT Wed Apr 19 09:59:52 KST 2023
[    0.000000] Machine model: Texas Instruments AM625 SK
[    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009c800000, size 3 MiB
[    0.000000] OF: reserved mem: initialized node ipc-memories@9c800000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009cb00000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node m4f-dma-memory@9cb00000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009cc00000, size 14 MiB
[    0.000000] OF: reserved mem: initialized node m4f-memory@9cc00000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009da00000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@9da00000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009db00000, size 12 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@9db00000, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x000000009c7fffff]
[    0.000000]   node   0: [mem 0x000000009c800000-0x000000009e6fffff]
[    0.000000]   node   0: [mem 0x000000009e700000-0x000000009e77ffff]
[    0.000000]   node   0: [mem 0x000000009e780000-0x000000009fffffff]
[    0.000000]   node   0: [mem 0x00000000a0000000-0x00000000ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000] On node 0 totalpages: 32768
[    0.000000]   DMA zone: 32 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 32768 pages, LIFO batch:3
[    0.000000] cma: Reserved 512 MiB at 0x00000000c0000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 2 pages/cpu s50008 r8192 d72872 u131072
[    0.000000] pcpu-alloc: s50008 r8192 d72872 u131072 alloc=2*65536
[    0.000000] pcpu-alloc: [0] 0 [0] 1
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: ARM erratum 845719
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] Built 1 zonelists, mobility grouping off.  Total pages: 32736
[    0.000000] Kernel command line: console=ttyS2,115200n8 consoleblank=0 earlycon=ns16550a,mmio32,0x02800000 root=/dev/mmcblk1p2 rw rootfstype=ext4 rootwait loglevel=8
[    0.000000] Dentry cache hash table entries: 262144 (order: 5, 2097152 bytes, linear)
[    0.000000] Inode-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 1489664K/2097152K available (10944K kernel code, 1320K rwdata, 4416K rodata, 1856K init, 711K bss, 83200K reserved, 524288K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
[    0.000000]  Trampoline variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001880000
[    0.000000] ITS [mem 0x01820000-0x0182ffff]
[    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
[    0.000000] ITS@0x0000000001820000: allocated 1048576 Devices @a0800000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x00000000a02c0000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x00000000a02d0000
[    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
[    0.000004] sched_clock: 56 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
[    0.008473] Console: colour dummy device 80x25
[    0.013062] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000)
[    0.023743] pid_max: default: 32768 minimum: 301
[    0.028533] LSM: Security Framework initializing
[    0.033332] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.040927] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.051150] rcu: Hierarchical SRCU implementation.
[    0.056382] Platform MSI: msi-controller@1820000 domain created
[    0.062639] PCI/MSI: /bus@f0000/interrupt-controller@1800000/msi-controller@1820000 domain created
[    0.071867] EFI services will not be available.
[    0.076784] smp: Bringing up secondary CPUs ...
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
[    0.090160] Detected VIPT I-cache on CPU1
[    0.090198] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000
[    0.090214] GICv3: CPU1: using allocated LPI pending table @0x00000000a02e0000
[    0.090279] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.090396] smp: Brought up 1 node, 2 CPUs
[    0.119781] SMP: Total of 2 processors activated.
[    0.124594] CPU features: detected: 32-bit EL0 Support
[    0.129865] CPU features: detected: CRC32 instructions
[    0.142362] CPU: All CPU(s) started at EL2
[    0.146569] alternatives: patching kernel code
[    0.152236] devtmpfs: initialized
[    0.160944] KASLR disabled due to lack of seed
[    0.165746] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.175721] futex hash table entries: 512 (order: -1, 32768 bytes, linear)
[    0.184010] pinctrl core: initialized pinctrl subsystem
[    0.189991] DMI not present or invalid.
[    0.194686] NET: Registered protocol family 16
[    0.205271] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations
[    0.212950] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.220973] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.229650] thermal_sys: Registered thermal governor 'step_wise'
[    0.229655] thermal_sys: Registered thermal governor 'power_allocator'
[    0.236407] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.250097] ASID allocator initialised with 65536 entries
[    0.275927] HugeTLB registered 16.0 GiB page size, pre-allocated 0 pages
[    0.282804] HugeTLB registered 512 MiB page size, pre-allocated 0 pages
[    0.289567] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.298230] cryptd: max_cpu_qlen set to 1000
[    0.306037] k3-chipinfo 43000014.chipid: Family:AM62X rev:SR1.0 JTAGID[0x0bb7e02f] Detected
[    0.315085] vcc_5v0: supplied by vmain_pd
[    0.319626] vcc_3v3_sys: supplied by vmain_pd
[    0.325060] iommu: Default domain type: Translated
[    0.330451] SCSI subsystem initialized
[    0.334697] mc: Linux media interface: v0.10
[    0.339091] videodev: Linux video capture interface: v2.00
[    0.344763] pps_core: LinuxPPS API ver. 1 registered
[    0.349839] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti giometti@linux.it
[    0.359188] PTP clock support registered
[    0.363225] EDAC MC: Ver: 3.0.0
[    0.367089] omap-mailbox 29000000.mailbox: omap mailbox rev 0x66fc9100
[    0.374239] FPGA manager framework
[    0.377817] Advanced Linux Sound Architecture Driver Initialized.
[    0.384993] clocksource: Switched to clocksource arch_sys_counter
[    0.391669] VFS: Disk quotas dquot_6.6.0
[    0.395779] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[    0.408265] NET: Registered protocol family 2
[    0.413020] IP idents hash table entries: 32768 (order: 2, 262144 bytes, linear)
[    0.421867] tcp_listen_portaddr_hash hash table entries: 4096 (order: 0, 65536 bytes, linear)
[    0.430694] TCP established hash table entries: 16384 (order: 1, 131072 bytes, linear)
[    0.438893] TCP bind hash table entries: 16384 (order: 2, 262144 bytes, linear)
[    0.446610] TCP: Hash tables configured (established 16384 bind 16384)
[    0.453470] UDP hash table entries: 2048 (order: 0, 65536 bytes, linear)
[    0.460397] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes, linear)
[    0.467895] NET: Registered protocol family 1
[    0.472897] RPC: Registered named UNIX socket transport module.
[    0.478981] RPC: Registered udp transport module.
[    0.483792] RPC: Registered tcp transport module.
[    0.488600] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.495196] PCI: CLS 0 bytes, default 64
[    0.499915] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
[    0.512418] Initialise system trusted keyrings
[    0.517202] workingset: timestamp_bits=46 max_order=15 bucket_order=0
[    0.527766] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.534385] NFS: Registering the id_resolver key type
[    0.539600] Key type id_resolver registered
[    0.543888] Key type id_legacy registered
[    0.548046] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.554902] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.562657] 9p: Installing v9fs 9p2000 file system support
[    0.603867] Key type asymmetric registered
[    0.608064] Asymmetric key parser 'x509' registered
[    0.613091] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    0.620657] io scheduler mq-deadline registered
[    0.625289] io scheduler kyber registered
[    0.631255] pinctrl-single 4084000.pinctrl: 34 pins, size 136
[    0.637587] pinctrl-single f4000.pinctrl: 171 pins, size 684
[    0.650560] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
[    0.669991] brd: module loaded
[    0.679267] loop: module loaded
[    0.683305] megasas: 07.714.04.00-rc1
[    0.690606] tun: Universal TUN/TAP device driver, 1.6
[    0.696317] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[    0.702735] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.708831] sky2: driver version 1.30
[    0.713258] VFIO - User Level meta-driver version: 0.3
[    0.718976] mousedev: PS/2 mouse device common for all mice
[    0.725440] i2c /dev entries driver
[    0.730556] sdhci: Secure Digital Host Controller Interface driver
[    0.736906] sdhci: Copyright(c) Pierre Ossman
[    0.741652] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.748288] ledtrig-cpu: registered to indicate activity on CPUs
[    0.754772] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[    0.762756] optee: probing for conduit method.
[    0.767355] optee: revision 3.19 (d6c5d003)
[    0.767616] optee: dynamic shared memory is enabled
[    0.777585] optee: initialized driver
[    0.783019] NET: Registered protocol family 17
[    0.787717] 9pnet: Installing 9P2000 support
[    0.792157] Key type dns_resolver registered
[    0.796736] Loading compiled-in X.509 certificates
[    0.810522] ti-sci 44043000.system-controller: ti,ctx-memory-region is required for suspend but not provided.
[    0.820703] ti-sci 44043000.system-controller: ti_sci_init_suspend failed, mem suspend will be non-functional.
[    0.830941] ti-sci 44043000.system-controller: ABI: 3.1 (firmware rev 0x0008 '8.5.3--v08.05.03 (Chill Capybar')
[    0.891563] omap-gpmc 3b000000.memory-controller: GPMC revision 6.0
[    0.898071] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[    0.907081] tps65219 0-0030: Failed to request IRQ 0 for tps65219_irq: -22
[    0.914282] tps65219: probe of 0-0030 failed with error -22
[    0.920058] omap_i2c 20000000.i2c: bus 0 rev0.12 at 400 kHz
[    0.926959] omap_i2c 20010000.i2c: bus 1 rev0.12 at 100 kHz
[    0.933327] ti-sci-intr 4210000.interrupt-controller: Interrupt Router 5 domain created
[    0.941682] ti-sci-intr bus@f0000:interrupt-controller@a00000: Interrupt Router 3 domain created
[    0.950914] ti-sci-inta 48000000.interrupt-controller: Interrupt Aggregator domain 28 created
[    0.959901] ti-udma 485c0100.dma-controller: Number of rings: 82
[    0.967908] ti-udma 485c0100.dma-controller: Channels: 48 (bchan: 18, tchan: 12, rchan: 18)
[    0.978930] ti-udma 485c0000.dma-controller: Number of rings: 150
[    0.988682] ti-udma 485c0000.dma-controller: Channels: 35 (tchan: 20, rchan: 15)
[    0.998831] printk: console [ttyS2] disabled
[    1.003281] 2800000.serial: ttyS2 at MMIO 0x2800000 (irq = 27, base_baud = 3000000) is a 8250
[    1.012046] printk: console [ttyS2] enabled
[    1.012046] printk: console [ttyS2] enabled
[    1.020489] printk: bootconsole [ns16550a0] disabled
[    1.020489] printk: bootconsole [ns16550a0] disabled
[    1.032565] [drm] Initialized tidss 1.0.0 20180215 for 30200000.dss on minor 0
[    1.046051] rtc-ti-k3 2b1f0000.rtc: registered as rtc0
[    1.051253] rtc-ti-k3 2b1f0000.rtc: setting system clock to 1970-01-01T00:00:05 UTC (5)
[    1.164149] mmc0: CQHCI version 5.10
[    1.168852] mmc2: CQHCI version 5.10
[    1.178757] vdd_mmc1: supplied by vcc_3v3_sys
[    1.184936] debugfs: Directory 'pd:53' with parent 'pm_genpd' already present!
[    1.192371] debugfs: Directory 'pd:52' with parent 'pm_genpd' already present!
[    1.192495] mmc1: CQHCI version 5.10
[    1.199621] debugfs: Directory 'pd:51' with parent 'pm_genpd' already present!
[    1.210735] debugfs: Directory 'pd:182' with parent 'pm_genpd' already present!
[    1.215769] mmc0: SDHCI controller on fa10000.mmc [fa10000.mmc] using ADMA 64-bit
[    1.229149] ALSA device list:
[    1.232176]   No soundcards found.
[    1.249000] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
[    1.305875] mmc1: new ultra high speed DDR50 SDHC card at address 59b4
[    1.313082] mmcblk1: mmc1:59b4 00000 29.8 GiB
[    1.320462]  mmcblk1: p1 p2 p3
[    2.689230] sdhci-am654 fa20000.mmc: Power on failed
[    2.724804] mmc2: SDHCI controller on fa20000.mmc [fa20000.mmc] using ADMA 64-bit
[    2.783248] EXT4-fs (mmcblk1p2): mounted filesystem with ordered data mode. Opts: (null)
[    2.791437] VFS: Mounted root (ext4 filesystem) on device 179:2.
[    2.801354] devtmpfs: mounted
[    2.805323] Freeing unused kernel memory: 1856K
[    2.809884] Run /sbin/init as init process
[    2.813973]   with arguments:
[    2.816928]     /sbin/init
[    2.819629]   with environment:
[    2.822764]     HOME=/
[    2.825117]     TERM=linux
INIT: version 2.96 booting
Starting udev
Starting version 244.5+
[    3.657473] random: crng init done

ERROR:   Unhandled External Abort received on 0x80000000 from S-EL1
ERROR:   exception reason=1 syndrome=0x92000010
Unhandled Exception from EL1
x0             = 0xffff800012280090
x1             = 0xffff00002279ee00
x2             = 0x0000000000000000
x3             = 0xffff000025652b00
x4             = 0xffff00002107ae10
x5             = 0xffff80001117448c
x6             = 0xffff80001228ffff
x7             = 0x0000000000000020
x8             = 0xffff80000fffffff
x9             = 0x00007ffffbda0000
x10            = 0xffff0000205b0e00
x11            = 0xffff8000122a0000
x12            = 0xffff00007fff0008
x13            = 0x0000000000000000
x14            = 0x000000000e030000
x15            = 0xffff8000112c0000
x16            = 0x0000000000000000
x17            = 0x0000000000000000
x18            = 0x0140000000000000
x19            = 0xffff000023148280
x20            = 0xffff00002107ac10
x21            = 0xffff000023148290
x22            = 0xffff00002107ac00
x23            = 0xffff000023148320
x24            = 0x0000000000000032
x25            = 0xffff80001123cab0
x26            = 0x0000000000000003
x27            = 0xffff8000100f5d78
x28            = 0xffff000025f80000
x29            = 0xffff800011c4f920
x30            = 0xffff800009000330
scr_el3        = 0x000000000000073d
sctlr_el3      = 0x0000000030cd183f
cptr_el3       = 0x0000000000000000
tcr_el3        = 0x0000000080803520
daif           = 0x00000000000002c0
mair_el3       = 0x00000000004404ff
spsr_el3       = 0x0000000080000005
elr_el3        = 0xffff800009000340
ttbr0_el3      = 0x000000009e7910c0
esr_el3        = 0x0000000092000010
far_el3        = 0xffff800012280090
spsr_el1       = 0x0000000060000000
elr_el1        = 0x0000000000457368
spsr_abt       = 0x0000000000000000
spsr_und       = 0x0000000000000000
spsr_irq       = 0x0000000000000000
spsr_fiq       = 0x0000000000000000
sctlr_el1      = 0x0000000034d4d91d
actlr_el1      = 0x0000000000000000
cpacr_el1      = 0x0000000000300000
csselr_el1     = 0x0000000000000000
sp_el1         = 0xffff800011c4f920
esr_el1        = 0x0000000056000000
ttbr0_el1      = 0x00000000a1ba3a00
ttbr1_el1      = 0x0094000082f60000
mair_el1       = 0x000c0400bb44ffff
amair_el1      = 0x0000000000000000
tcr_el1        = 0x00000032f5d07590
tpidr_el1      = 0xffff80006eef0000
tpidr_el0      = 0x0000fffbbf736660
tpidrro_el0    = 0x0000000000000000
par_el1        = 0x0000000000000000
mpidr_el1      = 0x0000000080000000
afsr0_el1      = 0x0000000000000000
afsr1_el1      = 0x0000000000000000
contextidr_el1 = 0x0000000000000000
vbar_el1       = 0xffff800010010800
cntp_ctl_el0   = 0x0000000000000005
cntp_cval_el0  = 0x000000006419d30d
cntv_ctl_el0   = 0x0000000000000000
cntv_cval_el0  = 0x0000000000000000
cntkctl_el1    = 0x00000000000000d6
sp_el0         = 0x000000009e78b440
isr_el1        = 0x0000000000000040
dacr32_el2     = 0x0000000000000000
ifsr32_el2     = 0x0000000000000000
cpuectlr_el1   = 0x0000000000000040
cpumerrsr_el1  = 0x000000000004062b
l2merrsr_el1   = 0x0000000011045120
cpuactlr_el1   = 0x00001000090ca000
[   14.416992] mmc1: Timeout waiting for hardware interrupt.
[   14.422399] mmc1: sdhci: ============ SDHCI REGISTER DUMP ===========
[   14.428825] mmc1: sdhci: Sys addr:  0x00000008 | Version:  0x00001004
[   14.435248] mmc1: sdhci: Blk size:  0x00007200 | Blk cnt:  0x00000000
[   14.441672] mmc1: sdhci: Argument:  0x00050140 | Trn mode: 0x0000003b
[   14.448096] mmc1: sdhci: Present:   0x01ff0000 | Host ctl: 0x0000001f
[   14.454519] mmc1: sdhci: Power:     0x0000000f | Blk gap:  0x00000080
[   14.460942] mmc1: sdhci: Wake-up:   0x00000000 | Clock:    0x00000207
[   14.467365] mmc1: sdhci: Timeout:   0x00000000 | Int stat: 0x00000003
[   14.473788] mmc1: sdhci: Int enab:  0x03ff008b | Sig enab: 0x03ff008b
[   14.480210] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000001
[   14.486633] mmc1: sdhci: Caps:      0x3de8c801 | Caps_1:   0x18002407
[   14.493056] mmc1: sdhci: Cmd:       0x0000123a | Max curr: 0x00000000
[   14.499478] mmc1: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0x00ee9d7f
[   14.505901] mmc1: sdhci: Resp[2]:   0x325b5900 | Resp[3]:  0x00000900
[   14.512323] mmc1: sdhci: Host ctl2: 0x0000000c
[   14.516753] mmc1: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x00000000a287020c
[   14.523869] mmc1: sdhci: ============================================
[   25.208982] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[   25.215072] rcu:     0-...0: (1 GPs behind) idle=31a/1/0x4000000000000000 softirq=459/460 fqs=2611
[   25.223838]  (detected by 1, t=5252 jiffies, g=-455, q=6657)
[   25.229482] Task dump for CPU 0:
[   25.232700] task:systemd-udevd   state:R  running task     stack:    0 pid:  192 ppid:   172 flags:0x0000002a
[   25.242600] Call trace:
[   25.245049]  __switch_to+0xc0/0x118
[   25.248530]  lock_timer_base+0x64/0xa8
[   25.252268]  try_to_del_timer_sync+0x2c/0x90
[   25.256525]  del_timer_sync+0x38/0x60
[   25.260180]  of_find_property+0x48/0x78
[   25.264007]  string_nocheck+0x60/0x78
[   25.267656]  string+0x5c/0x70
[   25.270615]  alloc_vmap_area.isra.0+0x98/0x7e8
[   25.275046]  __get_vm_area_node.isra.0+0x9c/0x180
[   25.279737]  0xffff000025f80000

They connected SD card to mmc1 and I also attached their HW schematic in below.

Thanks and Best Regards,

SI.

  • Hi SI,

    The log shows the customer uses Processor SDK v8.5. Can you please ask the customer to test SDK v8.6? There is an important MMC driver fix in the SDK8.6 kernel.

  • Hi Bin Liu,

    even with Processor SDK v8.5, they faced same error as below.

    ERROR:   Unhandled External Abort received on 0x80000000 from S-EL1

    ERROR:   exception reason=1 syndrome=0x92000010

    Unhandled Exception from EL1

    After some debugging, they found it was caused by K3-RTI-WATCHDOG driver module  and it seemed it was crashed when wdt->base + RTIDWDCTRL register in below.

    ~/drivers/watchdog/rti_wdt.c

    static int rti_wdt_probe(struct platform_device *pdev)

    {

                :

                :

         if (readl(wdt->base + RTIDWDCTRL) == WDENABLE_KEY) {

            u32 time_left_ms;

            u64 heartbeat_ms;

            u32 wsize;

     

    When they disabled this rti-watchdog driver, there was no issue.

    Could you please check if there is known issue in RTI_WDT driver of AM62x?

    Thanks and Best Regards,

    SI.

  • Hi SI,

    Sorry for my late response.

    Can you please add the following in your board dts to disable rti15 to see if it solves the kernel crash?

    &main_rti15 {
        status = "disabled";
    };

  • Hi Bin,

    Yes. it solves the kernel crash when 'K3 RTI-Windowed Watchdog Timer (WWDT) driver' was disabled.

    is there any artifacts in their custom board when  'K3 RTI-Windowed Watchdog Timer (WWDT) driver' was disabled?

    Do you think this is normal situation?

    Thanks and Best Regards,

    SI.

  • Hi SI,

    What I asked is slightly different from what the customer did - the customer disabled the WWDT driver which disables all the 5 RTI timers, but what I asked is to keep kernel WWDT driver enabled, but only disable main_rti15 in kernel device tree, so that other 4 RTI timers are still functional.

    If you check the AM62x RTM Table 4-136 RTI integration Attributes, RTI0~4 power domains are PD_A53_*, but RTI15 power domain is PD_GPU. Since AM623 doesn't have a GPU, I doubt the PD_GPU power domain or RTI15 timer exists. So the test of disabling main_rti15 in dts but keeping kernel RTI driver enabled would prove my suspicion is correct or not.

  • Hi Bin,

    They tried it and found there was no issue with main_rti0 and main_rti1, but still an issue when one of others is enabled as below.

    // NOTE::2023-05-09

    // https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1218039/am623-stuck-in-sd-card-after-boot-up/4622186#4622186

    &main_rti0 {    // watchdog@e000000

        status = "okay";

    };

     

    &main_rti1 {    // watchdog@e010000

        status = "okay";

    };

     

    &main_rti2 {    // watchdog@e020000

        status = "disabled";

        // Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP

    };

     

    &main_rti3 {    // watchdog@e030000

        status = "disabled";

        // Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP

    };

     

    &main_rti15 {   // watchdog@e0f0000

        status = "disabled";

        // Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP

    };

    Error logs per each are in below.

    * error logs when main_rti2 enabled.

    [    1.219023] rti-wdt e000000.watchdog: Trying to check register RTIDWDCTRL...

    [    1.226094] rti-wdt e000000.watchdog: Trying to read register RTIDWDCTRL... OK.

    [    1.234353] rti-wdt e010000.watchdog: Trying to check register RTIDWDCTRL...

    [    1.241426] rti-wdt e010000.watchdog: Trying to read register RTIDWDCTRL... OK.

    [    1.249509] rti-wdt e020000.watchdog: Trying to check register RTIDWDCTRL...

    [    1.256576] Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP

    [    1.264041] Modules linked in:

    [    1.267095] CPU: 1 PID: 62 Comm: kworker/1:2 Not tainted 5.10.168-g2c23e6c538 #62

    [    1.274558] Hardware name: Texas Instruments AM625 SK (DT)

    [    1.280045] Workqueue: events deferred_probe_work_func

    [    1.285174] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)

    [    1.291169] pc : rti_wdt_probe+0x298/0x4f8

    [    1.295253] lr : rti_wdt_probe+0x290/0x4f8

    [    1.299335] sp : ffff80001165bb70

    [    1.302637] x29: ffff80001165bb70 x28: ffff8000110f7000

    [    1.307939] x27: ffff800011433cc8 x26: ffff0000003054c8

    [    1.313239] x25: ffff8000111c4808 x24: 0000000000000022

    [    1.318540] x23: ffff0000015fd720 x22: ffff0000015fd690

    [    1.323840] x21: ffff000000291000 x20: ffff000000291010

    [    1.329141] x19: ffff0000015fd680 x18: 0000000000000010

    [    1.334441] x17: 0000000000000000 x16: 0000000000000001

    [    1.339741] x15: ffff000000c21390 x14: 00000000000000df

    [    1.345042] x13: ffff000000c21390 x12: 00000000ffffffea

    [    1.350343] x11: ffff8000111807d0 x10: ffff800011168790

    [    1.355643] x9 : ffff8000111687e8 x8 : 0000000000017fe8

    [    1.360943] x7 : c0000000ffffefff x6 : 0000000000000001

    [    1.366244] x5 : 0000000000000000 x4 : 0000000000000000

    [    1.371544] x3 : 00000000ffffffff x2 : e67aa454c6135400

    [    1.376844] x1 : e67aa454c6135400 x0 : ffff8000115a5090

    [    1.382145] Call trace:

    [    1.384584]  rti_wdt_probe+0x298/0x4f8

    [    1.388324]  platform_drv_probe+0x54/0xa8

    [    1.392321]  really_probe+0xec/0x400

    [    1.395885]  driver_probe_device+0x58/0xb8

    [    1.399969]  __device_attach_driver+0xb8/0xe0

    [    1.404315]  bus_for_each_drv+0x78/0xc8

    [    1.408138]  __device_attach+0xf8/0x188

    [    1.411961]  device_initial_probe+0x14/0x20

    [    1.416133]  bus_probe_device+0x9c/0xa8

    [    1.419959]  deferred_probe_work_func+0x88/0xc0

    [    1.424481]  process_one_work+0x1a0/0x348

    [    1.428479]  worker_thread+0x1f8/0x440

    [    1.432218]  kthread+0x140/0x160

    [    1.435436]  ret_from_fork+0x10/0x30

    [    1.439004] Code: 9121c021 9409a41a f9400260 91024000 (b9400000)

    [    1.445086] ---[ end trace fdcf0fbef253d7c3 ]---

    * error logs When main_rti3_enabled.

    [    1.219273] rti-wdt e000000.watchdog: Trying to check register RTIDWDCTRL...

    [    1.226336] rti-wdt e000000.watchdog: Trying to read register RTIDWDCTRL... OK.

    [    1.234594] rti-wdt e010000.watchdog: Trying to check register RTIDWDCTRL...

    [    1.241659] rti-wdt e010000.watchdog: Trying to read register RTIDWDCTRL... OK.

    [    1.249720] rti-wdt e030000.watchdog: Trying to check register RTIDWDCTRL...

    [    1.256786] Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP

    [    1.264251] Modules linked in:

    [    1.267305] CPU: 0 PID: 62 Comm: kworker/0:2 Not tainted 5.10.168-g2c23e6c538 #62

    [    1.274767] Hardware name: Texas Instruments AM625 SK (DT)

    [    1.280255] Workqueue: events deferred_probe_work_func

    [    1.285384] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)

    [    1.291378] pc : rti_wdt_probe+0x298/0x4f8

    [    1.295463] lr : rti_wdt_probe+0x290/0x4f8

    [    1.299546] sp : ffff80001165bb70

    [    1.302848] x29: ffff80001165bb70 x28: ffff8000110f7000

    [    1.308150] x27: ffff8000112fbcc8 x26: ffff0000003065c8

    [    1.313450] x25: ffff8000111c4808 x24: 0000000000000022

    [    1.318750] x23: ffff000001632320 x22: ffff000001632290

    [    1.324050] x21: ffff000000297000 x20: ffff000000297010

    [    1.329351] x19: ffff000001632280 x18: 0000000000000010

    [    1.334651] x17: 0000000000000000 x16: 0000000000000001

    [    1.339951] x15: ffff000000c13010 x14: 00000000000000df

    [    1.345251] x13: ffff000000c13010 x12: 00000000ffffffea

    [    1.350552] x11: ffff8000111807d0 x10: ffff800011168790

    [    1.355852] x9 : ffff8000111687e8 x8 : 0000000000017fe8

    [    1.361153] x7 : c0000000ffffefff x6 : 0000000000000001

    [    1.366452] x5 : 0000000000000000 x4 : 0000000000000000

    [    1.371753] x3 : 00000000ffffffff x2 : a75441360e57aa00

    [    1.377053] x1 : a75441360e57aa00 x0 : ffff8000115a5090

    [    1.382354] Call trace:

    [    1.384793]  rti_wdt_probe+0x298/0x4f8

    [    1.388535]  platform_drv_probe+0x54/0xa8

    [    1.392532]  really_probe+0xec/0x400

    [    1.396097]  driver_probe_device+0x58/0xb8

    [    1.400181]  __device_attach_driver+0xb8/0xe0

    [    1.404526]  bus_for_each_drv+0x78/0xc8

    [    1.408350]  __device_attach+0xf8/0x188

    [    1.412174]  device_initial_probe+0x14/0x20

    [    1.416345]  bus_probe_device+0x9c/0xa8

    [    1.420170]  deferred_probe_work_func+0x88/0xc0

    [    1.424692]  process_one_work+0x1a0/0x348

    [    1.428691]  worker_thread+0x1f8/0x440

    [    1.432429]  kthread+0x140/0x160

    [    1.435648]  ret_from_fork+0x10/0x30

    [    1.439215] Code: 9121c021 9409a41a f9400260 91024000 (b9400000)

    [    1.445297] ---[ end trace 1ff3ae781e450004 ]---

    * error logs When main_rti15_enabled.

    [    1.219010] rti-wdt e000000.watchdog: Trying to check register RTIDWDCTRL...

    [    1.226074] rti-wdt e000000.watchdog: Trying to read register RTIDWDCTRL... OK.

    [    1.234332] rti-wdt e010000.watchdog: Trying to check register RTIDWDCTRL...

    [    1.241401] rti-wdt e010000.watchdog: Trying to read register RTIDWDCTRL... OK.

    [    1.249463] rti-wdt e0f0000.watchdog: Trying to check register RTIDWDCTRL...

    [    1.256526] Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP

    [    1.263991] Modules linked in:

    [    1.267047] CPU: 0 PID: 104 Comm: kworker/0:2 Not tainted 5.10.168-g2c23e6c538 #62

    [    1.274597] Hardware name: Texas Instruments AM625 SK (DT)

    [    1.280084] Workqueue: events deferred_probe_work_func

    [    1.285213] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)

    [    1.291208] pc : rti_wdt_probe+0x298/0x4f8

    [    1.295293] lr : rti_wdt_probe+0x290/0x4f8

    [    1.299375] sp : ffff80001178bb70

    [    1.302677] x29: ffff80001178bb70 x28: ffff8000110f7000

    [    1.307978] x27: ffff8000112fbcc8 x26: ffff000000dc34c8

    [    1.313279] x25: ffff8000111c4808 x24: 0000000000000022

    [    1.318579] x23: ffff000001620d20 x22: ffff000001620c90

    [    1.323880] x21: ffff000000297000 x20: ffff000000297010

    [    1.329180] x19: ffff000001620c80 x18: 0000000000000010

    [    1.334480] x17: 0000000000000000 x16: 0000000000000001

    [    1.339780] x15: ffff000000383010 x14: 00000000000000df

    [    1.345081] x13: ffff000000383010 x12: 00000000ffffffea

    [    1.350381] x11: ffff8000111807d0 x10: ffff800011168790

    [    1.355681] x9 : ffff8000111687e8 x8 : 0000000000017fe8

    [    1.360982] x7 : c0000000ffffefff x6 : 0000000000000001

    [    1.366282] x5 : 0000000000000000 x4 : 0000000000000000

    [    1.371582] x3 : 00000000ffffffff x2 : d20d7cd810ac9c00

    [    1.376882] x1 : d20d7cd810ac9c00 x0 : ffff8000115a5090

    [    1.382183] Call trace:

    [    1.384622]  rti_wdt_probe+0x298/0x4f8

    [    1.388362]  platform_drv_probe+0x54/0xa8

    [    1.392359]  really_probe+0xec/0x400

    [    1.395923]  driver_probe_device+0x58/0xb8

    [    1.400007]  __device_attach_driver+0xb8/0xe0

    [    1.404352]  bus_for_each_drv+0x78/0xc8

    [    1.408176]  __device_attach+0xf8/0x188

    [    1.411999]  device_initial_probe+0x14/0x20

    [    1.416171]  bus_probe_device+0x9c/0xa8

    [    1.419996]  deferred_probe_work_func+0x88/0xc0

    [    1.424519]  process_one_work+0x1a0/0x348

    [    1.428517]  worker_thread+0x1f8/0x440

    [    1.432256]  kthread+0x140/0x160

    [    1.435476]  ret_from_fork+0x10/0x30

    [    1.439043] Code: 9121c021 9409a41a f9400260 91024000 (b9400000)

    [    1.445125] ---[ end trace 2120fbd3e2ac8130 ]---

    Thanks and Best Regards,

    SI.

  • Hi SI,

    Thanks for the detailed test, I appreciate it.

    They tried it and found there was no issue with main_rti0 and main_rti1, but still an issue when one of others is enabled as below.

    Does the customer use AM6232 which has dual A53 cores?

  • Yes. They are using AM6232 as below.

    AM6232ATCGGAALW

    Thanks and Best Regards,

    SI.

  • Hi SI,

    Thanks for confirming.

    Since AM6232 does not have main_rti2, main_rti3, and main_rti15, the corresponding DT nodes should be disabled in the board device tree file.