Other Parts Discussed in Thread: AM3352
We have a custom board using the AM3352 Processor and we are having intermittent issues during boot that seem to occur when various partitions of the EMMC are mounted. Sometimes the system boots fine and will run indefinitely.
Boards built with Samsung eMMC never fail but boards build with Toshiba and KIOXIA flash have the problem.
Observances:
- Occurs during boot or reboot. Soft reset or hard reset.
- Seem to happen when various partitions of the eMMC are mounted during the boot process.
- Once the system is running it is stable.
- System can reboot successfully over 20 times before a failure occurs. Intermittent.
- Failure error varies and sometime the system just halts with no error. Inconsistent.
KIOXIA analyzed the flash and observed that during the failure the DAT lines drop from 3.3V to 2.4V. We have tested configuring the DAT lines as Pullup, Pulldown, & Pullup/Pulldown disabled in U-Boot and the Kernel, but the failure still occurs. The suspicion is that something is briefly pulling down the DAT lines during boot.
Please let me know how I can debug this issue and if there is any more information you need from me.
Kernel:
Linux 5.10.100-g75c81c5c2a8b #3 PREEMPT Fri Mar 10 11:22:09 PST 2023 armv7l armv7l armv7l GNU/Linux
U-boot:
U-Boot SPL 2021.01-g95dff484-dirty (Apr 27 2023 - 15:17:09 -0500)
Trying to boot from MMC2
U-Boot 2021.01-g95dff484-dirty (Apr 27 2023 - 15:17:09 -0500)
Example Failure:
U-Boot SPL 2021.01-gf3d2ad47-dirty (Feb 24 2023 - 14:41:49 -0600)
Trying to boot from MMC2
U-Boot 2021.01-gf3d2ad47-dirty (Feb 24 2023 - 14:41:49 -0600)
CPU : AM335X-GP rev 2.1
Model: TI AM335x EVMB
DRAM: 1 GiB
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
Loading Environment from MMC... OK
<ethaddr> not set. Validating first E-fuse MAC
Net: eth2: ethernet@4a100000
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc1(part 0) is current device
Scanning mmc 1:1...
Scanning disk mmc@48060000.blk...
Disk mmc@48060000.blk not ready
Scanning disk mmc@481d8000.blk...
Found 5 disks
No EFI system partition
BootOrder not defined
EFI boot manager: Cannot load any image
switch to partitions #0, OK
mmc1(part 0) is current device
SD/MMC found on device 1
4174336 bytes read in 348 ms (11.4 MiB/s)
63463 bytes read in 7 ms (8.6 MiB/s)
## Flattened Device Tree blob at 88000000
Booting using the fdt blob at 0x88000000
Loading Device Tree to 8ffed000, end 8ffff7e6 ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.10.100-g3c283e3cb13b-dirty (aaron@aaron-VirtualBox) (arm-liteon-linux-gnueabi-gcc (GCC) 9.3.0, GNU ld (GNU Binutils) 2.34.0.20200220) #17 PREEMPT Tue Feb 21 14:55:28 CST 2023
[ 0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: TI AM335x EVMB
[ 0.000000] Memory policy: Data cache writeback
[ 0.000000] efi: UEFI not found.
[ 0.000000] cma: Reserved 48 MiB at 0xbd000000
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000080000000-0x00000000afefffff]
[ 0.000000] HighMem [mem 0x00000000aff00000-0x00000000bfffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000080000000-0x00000000afefffff]
[ 0.000000] node 0: [mem 0x00000000b0000000-0x00000000bfffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000bfffffff]
[ 0.000000] CPU: All CPU(s) started in SVC mode.
[ 0.000000] AM335X ES2.1 (neon)
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 260162
[ 0.000000] Kernel command line: console=ttyO0,115200n8 root=PARTUUID=00000000-02 rw rootfstype=ext4 rootwait
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 974876K/1047552K available (8192K kernel code, 672K rwdata, 2672K rodata, 1024K init, 268K bss, 23524K reserved, 49152K cma-reserved, 212992K highmem)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] Trampoline variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] IRQ: Found an INTC at 0x(ptrval) (revision 5.0) with 128 interrupts
[ 0.000000] random: get_random_bytes called from start_kernel+0x30c/0x4c8 with crng_init=0
[ 0.000000] TI gptimer clocksource: always-on /ocp/interconnect@44c00000/segment@200000/target-module@31000
[ 0.000013] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
[ 0.000038] clocksource: dmtimer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[ 0.000497] TI gptimer clockevent: 24000000 Hz at /ocp/interconnect@48000000/segment@0/target-module@40000
[ 0.001879] Console: colour dummy device 80x30
[ 0.001930] WARNING: Your 'console=ttyO0' has been replaced by 'ttyS0'
[ 0.001942] This ensures that you still see kernel messages. Please
[ 0.001952] update your kernel commandline.
[ 0.002023] Calibrating delay loop... 597.60 BogoMIPS (lpj=2988032)
[ 0.060525] pid_max: default: 32768 minimum: 301
[ 0.060778] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[ 0.060807] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[ 0.061956] CPU: Testing write buffer coherency: ok
[ 0.062051] CPU0: Spectre v2: using BPIALL workaround
[ 0.063138] Setting up static identity map for 0x80100000 - 0x80100060
[ 0.063336] rcu: Hierarchical SRCU implementation.
[ 0.063462] EFI services will not be available.
[ 0.063967] devtmpfs: initialized
[ 0.078791] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
[ 0.079128] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.079160] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[ 0.083592] pinctrl core: initialized pinctrl subsystem
[ 0.084803] DMI not present or invalid.
[ 0.085400] NET: Registered protocol family 16
[ 0.088316] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.108712] l3-aon-clkctrl:0000:0: failed to disable
[ 0.110294] thermal_sys: Registered thermal governor 'fair_share'
[ 0.110308] thermal_sys: Registered thermal governor 'bang_bang'
[ 0.110329] thermal_sys: Registered thermal governor 'step_wise'
[ 0.110341] thermal_sys: Registered thermal governor 'user_space'
[ 0.110843] cpuidle: using governor ladder
[ 0.110894] cpuidle: using governor menu
[ 1.762730] No ATAGs?
[ 1.762768] hw-breakpoint: debug architecture 0x4 unsupported.
[ 1.797453] iommu: Default domain type: Translated
[ 1.799290] SCSI subsystem initialized
[ 1.799996] pps_core: LinuxPPS API ver. 1 registered
[ 1.800018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 1.800053] PTP clock support registered
[ 1.800144] EDAC MC: Ver: 3.0.0
[ 1.803547] clocksource: Switched to clocksource dmtimer
[ 1.814747] NET: Registered protocol family 2
[ 1.815073] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[ 1.816727] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 1.816788] TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear)
[ 1.816904] TCP bind hash table entries: 8192 (order: 3, 32768 bytes, linear)
[ 1.817017] TCP: Hash tables configured (established 8192 bind 8192)
[ 1.817191] UDP hash table entries: 512 (order: 1, 8192 bytes, linear)
[ 1.817234] UDP-Lite hash table entries: 512 (order: 1, 8192 bytes, linear)
[ 1.817405] NET: Registered protocol family 1
[ 1.818161] RPC: Registered named UNIX socket transport module.
[ 1.818183] RPC: Registered udp transport module.
[ 1.818195] RPC: Registered tcp transport module.
[ 1.818205] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 1.818230] PCI: CLS 0 bytes, default 64
[ 1.819542] hw perfevents: enabled with armv7_cortex_a8 PMU driver, 5 counters available
[ 1.821224] Initialise system trusted keyrings
[ 1.821644] workingset: timestamp_bits=14 max_order=18 bucket_order=4
[ 1.827454] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 1.828477] NFS: Registering the id_resolver key type
[ 1.828539] Key type id_resolver registered
[ 1.828553] Key type id_legacy registered
[ 1.828673] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 1.828691] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[ 1.828746] ntfs: driver 2.1.32 [Flags: R/O].
[ 1.829533] Key type asymmetric registered
[ 1.829560] Asymmetric key parser 'x509' registered
[ 1.829641] bounce: pool size: 64 pages
[ 1.829715] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[ 1.829732] io scheduler mq-deadline registered
[ 1.829746] io scheduler kyber registered
[ 2.422283] omap-mailbox 480c8000.mailbox: omap mailbox rev 0x400
[ 2.473609] random: fast init done
[ 2.584897] debugfs: Directory '49000000.dma' with parent 'dmaengine' already present!
[ 2.584953] edma 49000000.dma: TI EDMA DMA engine driver
[ 2.644710] pinctrl-single 44e10800.pinmux: 142 pins, size 568
[ 2.720341] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
[ 2.725538] omap8250 44e09000.serial: rts_gpiod 00000000
[ 2.726795] 44e09000.serial: ttyS0 at MMIO 0x44e09000 (irq = 20, base_baud = 3000000) is a 8250
[ 3.395573] printk: console [ttyS0] enabled
[ 3.405691] omap_rng 48310000.rng: Random Number Generator ver. 20
[ 3.413713] random: crng init done
[ 3.429171] brd: module loaded
[ 3.441813] loop: module loaded
[ 3.462557] CAN device driver interface
[ 3.468340] c_can_platform 481cc000.can: c_can_platform device registered (regs=c762f957, irq=41)
[ 3.478740] c_can_platform 481d0000.can: c_can_platform device registered (regs=e2ef5dc1, irq=42)
[ 3.553643] davinci_mdio 4a101000.mdio: davinci mdio revision 1.6, bus freq 1000000
[ 3.572779] davinci_mdio 4a101000.mdio: phy[1]: device 4a101000.mdio:01, driver unknown
[ 3.581056] davinci_mdio 4a101000.mdio: phy[2]: device 4a101000.mdio:02, driver unknown
[ 3.589881] cpsw-switch 4a100000.switch: initialized cpsw ale version 1.4
[ 3.596848] cpsw-switch 4a100000.switch: ALE Table size 1024
[ 3.602669] cpsw-switch 4a100000.switch: cpts: overflow check period 500 (jiffies)
[ 3.610370] cpsw-switch 4a100000.switch: CPTS: ref_clk_freq:250000000 calc_mult:2147483648 calc_shift:29 error:0 nsec/sec
[ 3.621501] cpsw-switch 4a100000.switch: Detected MACID = 40:bd:32:c7:6e:ab
[ 3.628566] cpsw-switch 4a100000.switch: Detected MACID = 40:bd:32:c7:6e:ad
[ 3.637613] cpsw-switch 4a100000.switch: initialized (regs 0x4a100000, pool size 256) hw_ver:0019010C 1.12 (0)
[ 3.649850] omap_rtc 44e3e000.rtc: already running
[ 3.655952] omap_rtc 44e3e000.rtc: registered as rtc0
[ 3.661083] omap_rtc 44e3e000.rtc: setting system clock to 2023-03-17T19:18:44 UTC (1679080724)
[ 3.670309] i2c /dev entries driver
[ 3.677408] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 3.686510] cpuidle: enable-method property 'ti,am3352' found operations
[ 3.694317] sdhci: Secure Digital Host Controller Interface driver
[ 3.700536] sdhci: Copyright(c) Pierre Ossman
[ 3.706656] sdhci-pltfm: SDHCI platform and OF driver helper
[ 3.715630] sdhci-omap 481d8000.mmc: supply vqmmc not found, using dummy regulator
[ 3.724705] ledtrig-cpu: registered to indicate activity on CPUs
[ 3.748626] NET: Registered protocol family 10
[ 3.764825] Segment Routing with IPv6
[ 3.768629] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 3.781685] NET: Registered protocol family 17
[ 3.793617] can: controller area network core
[ 3.798110] NET: Registered protocol family 29
[ 3.802578] can: raw protocol
[ 3.813557] can: broadcast manager protocol
[ 3.817780] can: netlink gateway - max_hops=1
[ 3.822760] Key type dns_resolver registered
[ 3.839396] omap_voltage_late_init: Voltage driver support not added
[ 3.847141] Loading compiled-in X.509 certificates
[ 3.866493] OMAP GPIO hardware version 0.1
[ 3.871945] mmc1: SDHCI controller on 481d8000.mmc [481d8000.mmc] using External DMA
[ 3.884775] pca953x 0-0020: supply vcc not found, using dummy regulator
[ 3.891709] pca953x 0-0020: using no AI
[ 3.930146] tps65910 0-002d: No interrupt support, no core IRQ
[ 3.938499] vrtc: supplied by vbat
[ 3.945646] vio: supplied by vbat
[ 3.951090] vdd_mpu: supplied by vbat
[ 3.956763] vdd_core: supplied by vbat
[ 3.963372] vdig1: supplied by vbat
[ 3.968504] vdig2: supplied by vbat
[ 3.973571] vpll: supplied by vbat
[ 3.978453] vdac: supplied by vbat
[ 3.983439] vaux1: supplied by vbat
[ 3.988617] vaux2: supplied by vbat
[ 3.993823] mmc1: new high speed MMC card at address 0001
[ 3.999960] vaux33: supplied by vbat
[ 4.004082] mmcblk1: mmc1:0001 004GA0 3.69 GiB
[ 4.009017] mmcblk1boot0: mmc1:0001 004GA0 partition 1 2.00 MiB
[ 4.016252] mmcblk1boot1: mmc1:0001 004GA0 partition 2 2.00 MiB
[ 4.022760] vmmc: supplied by vbat
[ 4.026277] mmcblk1rpmb: mmc1:0001 004GA0 partition 3 512 KiB, chardev (244:0)
[ 4.035331] vbb: supplied by vbat
[ 4.039553] omap_i2c 44e0b000.i2c: bus 0 rev0.11 at 400 kHz
[ 4.046972] mmcblk1: p1 p2 p3 p4
[ 4.065432] omap_reset_deassert: timedout waiting for gfx:0
[ 4.071149] ti-sysc: probe of 5600fe00.target-module failed with error -110
[ 4.079260] omap8250 48022000.serial: rts_gpiod 69faf621
[ 4.086927] 48022000.serial: ttyS1 at MMIO 0x48022000 (irq = 27, base_baud = 3000000) is a 8250
[ 4.097413] omap8250 48024000.serial: rts_gpiod ea84067f
[ 4.105039] 48024000.serial: ttyS2 at MMIO 0x48024000 (irq = 28, base_baud = 3000000) is a 8250
[ 4.115432] omap8250 481a6000.serial: rts_gpiod 221ffc21
[ 4.122891] 481a6000.serial: ttyS3 at MMIO 0x481a6000 (irq = 38, base_baud = 3000000) is a 8250
[ 4.135445] omap_gpio 44e07000.gpio: Could not set line 6 debounce to 200000 microseconds (-22)
[ 4.148541] sdhci-omap 48060000.mmc: Got CD GPIO
[ 4.153500] sdhci-omap 48060000.mmc: supply vqmmc not found, using dummy regulator
[ 4.219813] mmc0: SDHCI controller on 48060000.mmc [48060000.mmc] using External DMA
[ 4.240460] EXT4-fs (mmcblk1p2): mounted filesystem with ordered data mode. Opts: (null)
[ 4.248878] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 4.256280] devtmpfs: mounted
[ 4.260753] Freeing unused kernel memory: 1024K
[ 4.265781] Run /sbin/init as init process
INIT: version 2.96 booting
[ 4.848270] usbcore: registered new interface driver usbfs
[ 4.857402] usbcore: registered new interface driver hub
[ 4.862811] usbcore: registered new device driver usb
[ 4.935726] usbcore: registered new interface driver usb-storage
[ 4.985698] am335x-phy-driver 47401300.usb-phy: supply vcc not found, using dummy regulator
[ 5.004716] am335x-phy-driver 47401b00.usb-phy: supply vcc not found, using dummy regulator
[ 5.016874] musb-hdrc musb-hdrc.0: MUSB HDRC host driver
[ 5.022263] musb-hdrc musb-hdrc.0: new USB bus registered, assigned bus number 1
[ 5.033757] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[ 5.042064] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 5.058875] usb usb1: Product: MUSB HDRC host driver
[ 5.063877] usb usb1: Manufacturer: Linux 5.10.100-g3c283e3cb13b-dirty musb-hcd
[ 5.071216] usb usb1: SerialNumber: musb-hdrc.0
[ 5.076418] hub 1-0:1.0: USB hub found
[ 5.080231] hub 1-0:1.0: 1 port detected
[ 5.107400] musb-hdrc musb-hdrc.1: MUSB HDRC host driver
[ 5.112785] musb-hdrc musb-hdrc.1: new USB bus registered, assigned bus number 2
[ 5.136956] EXT4-fs (mmcblk1p2): re-mounted. Opts: (null)
[ 5.145099] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[ 5.153420] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 5.162274] usb usb2: Product: MUSB HDRC host driver
[ 5.168411] usb usb2: Manufacturer: Linux 5.10.100-g3c283e3cb13b-dirty musb-hcd
[ 5.182636] usb usb2: SerialNumber: musb-hdrc.1
[ 5.193478] hub 2-0:1.0: USB hub found
[ 5.204442] hub 2-0:1.0: 1 port detected
[ 5.550927] 8<--- cut here ---
[ 5.554178] Unable to handle kernel paging request at virtual address 35410000
[ 5.561439] pgd = a0f81092
[ 5.564174] [35410000] *pgd=00000000
[ 5.567775] Internal error: Oops: 5 [#1] PREEMPT ARM
[ 5.572758] Modules linked in: phy_am335x phy_am335x_control phy_generic usb_storage musb_dsps musb_hdrc udc_core usbcore usb_common
[ 5.584762] CPU: 0 PID: 135 Comm: sh Not tainted 5.10.100-g3c283e3cb13b-dirty #17
[ 5.592274] Hardware name: Generic AM33XX (Flattened Device Tree)
[ 5.598404] PC is at copy_page+0x10/0x70
[ 5.602351] LR is at v6_copy_user_highpage_nonaliasing+0xbc/0x170
[ 5.608466] pc : [<c048ea30>] lr : [<c0116064>] psr: a0010113
[ 5.614757] sp : c224fe78 ip : 80010193 fp : c16cee00
[ 5.620000] r10: 00000800 r9 : c2232780 r8 : c0115fa8
[ 5.625243] r7 : c2232780 r6 : 35410000 r5 : efe75eb8 r4 : bfe2c000
[ 5.631795] r3 : c224e000 r2 : 0000001f r1 : 35410000 r0 : bfe2c000
[ 5.638349] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 5.645513] Control: 10c5387d Table: 822d8019 DAC: 00000051
[ 5.651280] Process sh (pid: 135, stack limit = 0x9b095b77)
[ 5.656873] Stack: (0xc224fe78 to 0xc2250000)
[ 5.661248] fe60: bfe2c000 c0116064
[ 5.669462] fe80: c224fef0 efe75eb8 efe76240 c0202128 c0d78798 c01ae7e8 00000021 c16cee40
[ 5.677676] fea0: 01200000 c21f5f70 00000000 c0d0d0d8 c21f5f70 00000255 000fa000 c224ffb0
[ 5.685889] fec0: 00000040 00000c73 ffffe000 00000800 00000255 c020790c c21f61d0 c222e140
[ 5.694103] fee0: 00000000 00000000 c224ff60 c014d2c4 c2232780 00000255 00100cca 000000da
[ 5.702317] ff00: 000fa000 c22d8000 c22d8000 bc5d83cf 00000000 efe76260 ffefe3e8 c16cee40
[ 5.710531] ff20: 00000000 c0d03048 fffff000 c224ffb0 000fa734 0000081f c16cee00 c2232780
[ 5.718744] ff40: c16cee40 00000800 00000255 c011100c c224e000 00000078 00130338 c0125e90
[ 5.726959] ff60: 01200000 c0d07c5c 0000081f 000fa734 c224ffb0 c0110efc 000d3c8c 00101174
[ 5.735173] ff80: 00130338 c01112e8 00000000 00000000 00000000 00000000 00054a94 60010010
[ 5.743387] ffa0: ffffffff 10c5387d 10c53c7d c0100e7c 00000016 be86a890 00000000 00000000
[ 5.751601] ffc0: 0000009d 000fa724 000ffe28 00000008 00000000 000d3c8c 00101174 00130338
[ 5.759814] ffe0: 00130280 be86a800 00054a80 00054a94 60010010 ffffffff 00000000 00000000
[ 5.768037] Code: e92d4010 f5d1f000 f5d1f040 e3a0201f (e8b15018)
[ 5.774195] ---[ end trace 53e80d0e85bd0d5a ]---
[ 5.778834] note: sh[135] exited with preempt_count 2
/etc/rcS.d/S37populate-volatile.sh: line 268: 135 Segmentation fault sh "${ROOT_DIR}/etc/volatile.cache"
