Other Parts Discussed in Thread: AM625
Tool/software:
Hello
We are experiencing some failures in climate chamber tests with our customized design of the AM62P (XAM62P54AUMHAAMH) at low temperatures. This has happened with all seven of the pieces we have tested so far.
The failures occurs only at temperatures below 0°C. The modules either stalls and crashes, or print a lot of kernel dumps.
Linux version: 6.6.84-7.3.0-devel
SYSFW: 4.0 (firmware rev 0x000b '11.0.7--v11.00.07 (Fancy Rat)')
For example, it stalls and crashes during boot.
Climate Chamber: TempEvent T/180/70a/3 Start Temperature: -13.8�C End Temperature: -11.6�C Measured Temperature: -13.8�C Shelf Temperature: -13.2�C Shelf Name: Shelf_33 Module Serial Number: 15671821 Module Location: Shelf 1, Carrier Board 5 Test duration: 391.55s Test Log ******** 22:44:50.81 10->11 Waited 300.00s 22:39:50.70 9->10 Set Reset Low 22:39:50.59 8->9 Power Off 22:39:50.48 3->8 Not found, time out: "CPU stress" 22:38:20.40 2->3 Message found: "U-Boot" 22:38:19.92 1->2 Set Reset High 22:38:19.81 0->1 Power On Search Strings ************** PASS : U-Boot PASS : Starting kernel PASS : critical temperature reached PASS : hwclock 1 failed PASS : hwclock 2 failed PASS : I/O error PASS : Internal error PASS : memtester failed PASS : FAILURE FAIL : Done. PASS : bonnie\+\+ failed FAIL : Waiting for power-cycle N/A : Device life time estimation type B N/A : Device life time estimation type A N/A : Pre EOL information FAIL : CPU stress test, TestResult: PASS FAIL : Ethernet test, TestResult: PASS FAIL : memory test, TestResult: PASS FAIL : eMMC test, TestResult: PASS Terminal ******** U-Boot SPL 2024.04-ti-gcedb677ccf6e (Apr 25 2025 - 10:55:48 +0000) SYSFW ABI: 4.0 (firmware rev 0x000b '11.0.7--v11.00.07 (Fancy Rat)') SPL initial stack usage: 13568 bytes Trying to boot from MMC1 Authentication passed Authentication passed Authentication passed Loading Environment from nowhere... OK init_env from device 9 not supported! Authentication passed Authentication passed Starting ATF on ARM64 core... NOTICE: BL31: v2.12.0(release):11.00.09-2-gce454ea3b-dirty NOTICE: BL31: Built : 12:14:11, Apr 16 2025 U-Boot SPL 2024.04-ti-gcedb677ccf6e (Apr 25 2025 - 10:55:48 +0000) SYSFW ABI: 4.0 (firmware rev 0x000b '11.0.7--v11.00.07 (Fancy Rat)') SPL initial stack usage: 1872 bytes HW CFG: 0x00 Trying to boot from MMC1 Authentication passed Authentication passed U-Boot 2024.04-ti-gcedb677ccf6e (Apr 25 2025 - 10:55:48 +0000) SoC: AM62PX SR1.0 HS-FS DRAM: 2 GiB Core: 149 devices, 31 uclasses, devicetree: separate MMC: mmc@fa10000: 0, mmc@fa00000: 1 Loading Environment from MMC... OK MISSING TORADEX CARRIER CONFIG BLOCKS In: serial@2800000 Out: serial@2800000 Err: serial@2800000 Model: Toradex 0099 Verdin AM62P Quad 2GB WB IT V1.0A Serial#: 15671821 am65_cpsw_nuss ethernet@8000000: K3 CPSW: nuss_ver: 0x6BA01903 cpsw_ver: 0x6BA81903 ale_ver: 0x00290105 Ports:2 Setting variant to wifi Net: eth0: ethernet@8000000port@1 [PRIME]Could not get PHY for mdio@f00: addr 7 am65_cpsw_nuss_port ethernet@8000000port@2: phy_connect() failed Hit any key to stop autoboot: 1 0 7[r[999;999H[6n8Card did not respond to voltage select! : -110 No EFI system partition No EFI system partition Failed to persist EFI variables No EFI system partition Failed to persist EFI variables No EFI system partition Failed to persist EFI variables ** Booting bootflow '<NULL>' with efi_mgr Loading Boot0000 'mmc 0' failed EFI boot manager: Cannot load any image Boot failed (err=-14) Card did not respond to voltage select! : -110 ** Booting bootflow 'mmc@fa10000.bootdev.part_1' with script Loading DeviceTree: k3-am62p5-verdin-wifi-dev.dtb 73915 bytes read in 13 ms (5.4 MiB/s) 84 bytes read in 11 ms (6.8 KiB/s) Working FDT set to 90200000 Applying Overlay: verdin-am62p_dsi-to-hdmi_overlay.dtbo 3639 bytes read in 12 ms (295.9 KiB/s) Applying Overlay: verdin-am62p_spidev_overlay.dtbo 564 bytes read in 12 ms (45.9 KiB/s) 7916958 bytes read in 65 ms (116.2 MiB/s) Bootargs: root=PARTUUID=92b064bd-02 ro rootwait console=tty1 console=ttyS2,115200 Uncompressing Kernel Image to 0 ## Flattened Device Tree blob at 90200000 Booting using the fdt blob at 0x90200000 Working FDT set to 90200000 Loading Device Tree to 00000000fce65000, end 00000000fce9afff ... OK Working FDT set to fce65000 Starting kernel ... [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034] [ 0.000000] Linux version 6.6.84-7.3.0-devel (oe-user@oe-host) (aarch64-tdx-linux-gcc (GCC) 13.3.0, GNU ld (GNU Binutils) 2.42.0.20240723) #1 SMP PREEMPT Mon Apr 28 09:45:20 UTC 2025 [ 0.000000] KASLR disabled due to lack of seed [ 0.000000] Machine model: Toradex Verdin AM62P WB on Verdin Development Board [ 0.000000] efi: UEFI not found. [ 0.000000] OF: reserved mem: 0x0000000080000000..0x000000008007ffff (512 KiB) nomap non-reusable tfa@80000000 [ 0.000000] Reserved memory: created DMA memory pool at 0x000000009c900000, size 30 MiB [ 0.000000] OF: reserved mem: initialized node r5f-dma-memory@9c900000, compatible id shared-dma-pool [ 0.000000] OF: reserved mem: 0x000000009c900000..0x000000009e6fffff (30720 KiB) nomap non-reusable r5f-dma-memory@9c900000 [ 0.000000] OF: reserved mem: 0x000000009e800000..0x000000009fffffff (24576 KiB) nomap non-reusable optee@9e800000 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000080000000-0x00000000ffffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000080000000-0x000000008007ffff] [ 0.000000] node 0: [mem 0x0000000080080000-0x000000009c8fffff] [ 0.000000] node 0: [mem 0x000000009c900000-0x000000009e6fffff] [ 0.000000] node 0: [mem 0x000000009e700000-0x000000009e7fffff] [ 0.000000] node 0: [mem 0x000000009e800000-0x000000009fffffff] [ 0.000000] node 0: [mem 0x00000000a0000000-0x00000000ffffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000ffffffff] [ 0.000000] cma: Reserved 410 MiB at 0x00000000e3400000 on node -1 [ 0.000000] psci: probing for conduit method from DT. [ 0.000000] psci: PSCIv1.1 detected in firmware. [ 0.000000] psci: Using standard PSCI v0.2 function IDs [ 0.000000] psci: Trusted OS migration not required [ 0.000000] psci: SMC Calling Convention v1.5 [ 0.000000] percpu: Embedded 20 pages/cpu s42856 r8192 d30872 u81920 [ 0.000000] Detected VIPT I-cache on CPU0 [ 0.000000] CPU features: detected: GIC system register CPU interface [ 0.000000] alternatives: applying boot alternatives [ 0.000000] Kernel command line: root=PARTUUID=92b064bd-02 ro rootwait console=tty1 console=ttyS2,115200 [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 516096 [ 0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off [ 0.000000] software IO TLB: area num 4. [ 0.000000] software IO TLB: mapped [mem 0x00000000df400000-0x00000000e3400000] (64MB) [ 0.000000] Memory: 1495296K/2097152K available (11392K kernel code, 1230K rwdata, 3532K rodata, 2560K init, 458K bss, 182016K reserved, 419840K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] rcu: Preemptible hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4. [ 0.000000] Trampoline variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode [ 0.000000] GICv3: 256 SPIs implemented [ 0.000000] GICv3: 0 Extended SPIs implemented [ 0.000000] Root IRQ handler: gic_handle_irq [ 0.000000] GICv3: GICv3 features: 16 PPIs [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001880000 [ 0.000000] ITS [mem 0x01820000-0x0182ffff] [ 0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS [ 0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19 [ 0.000000] ITS@0x0000000001820000: allocated 524288 Devices @80c00000 (flat, esz 8, psz 64K, shr 0) [ 0.000000] ITS: using cache flushing for cmd queue [ 0.000000] GICv3: using LPI property table @0x0000000080440000 [ 0.000000] GIC: using cache flushing for LPI property table [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000080450000 [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. [ 0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys). [ 0.000000] clocksource: arch_sys_counter: mask: 0x3ffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns [ 0.000001] sched_clock: 58 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns [ 0.000422] Console: colour dummy device 80x25 [ 0.000433] printk: console [tty1] enabled [ 0.000957] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000) [ 0.000980] pid_max: default: 32768 minimum: 301 [ 0.001139] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.001163] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.002805] RCU Tasks: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=4. [ 0.002893] RCU Tasks Trace: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=4. [ 0.003075] rcu: Hierarchical SRCU implementation. [ 0.003088] rcu: Max phase no-delay instances is 1000. [ 0.003787] Platform MSI: msi-controller@1820000 domain created [ 0.004119] PCI/MSI: /bus@f0000/interrupt-controller@1800000/msi-controller@1820000 domain created [ 0.004229] EFI services will not be available. [ 0.004549] smp: Bringing up secondary CPUs ... [ 0.005143] Detected VIPT I-cache on CPU1 [ 0.005216] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000 [ 0.005232] GICv3: CPU1: using allocated LPI pending table @0x0000000080460000 [ 0.005282] CPU1: Booted secondary processor 0x0000000001 [0x410fd034] [ 0.005936] Detected VIPT I-cache on CPU2 [ 0.005991] GICv3: CPU2: found redistributor 2 region 0:0x00000000018c0000 [ 0.006005] GICv3: CPU2: using allocated LPI pending table @0x0000000080470000 [ 0.006035] CPU2: Booted secondary processor 0x0000000002 [0x410fd034] [ 0.006609] Detected VIPT I-cache on CPU3 [ 0.006662] GICv3: CPU3: found redistributor 3 region 0:0x00000000018e0000 [ 0.006676] GICv3: CPU3: using allocated LPI pending table @0x0000000080480000 [ 0.006704] CPU3: Booted secondary processor 0x0000000003 [0x410fd034] [ 0.006774] smp: Brought up 1 node, 4 CPUs [ 0.006882] SMP: Total of 4 processors activated. [ 0.006893] CPU features: detected: 32-bit EL0 Support [ 0.006903] CPU features: detected: CRC32 instructions [ 0.006965] CPU: All CPU(s) started at EL2 [ 0.006976] alternatives: applying system-wide alternatives [ 0.008608] devtmpfs: initialized [ 0.017364] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.017415] futex hash table entries: 1024 (order: 4, 65536 bytes, linear) [ 0.029588] pinctrl core: initialized pinctrl subsystem [ 0.030199] DMI not present or invalid. [ 0.030834] NET: Registered PF_NETLINK/PF_ROUTE protocol family [ 0.031685] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations [ 0.031801] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.031927] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.032476] thermal_sys: Registered thermal governor 'step_wise' [ 0.032484] thermal_sys: Registered thermal governor 'user_space' [ 0.032499] thermal_sys: Registered thermal governor 'power_allocator' [ 0.032542] cpuidle: using governor menu [ 0.032660] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers. [ 0.032746] ASID allocator initialised with 65536 entries [ 0.039740] /bus@f0000/interrupt-controller@1800000: Fixed dependency cycle(s) with /bus@f0000/interrupt-controller@1800000 [ 0.039913] /bus@f0000/i2c@20000000/hdmi@48: Fixed dependency cycle(s) with /bus@f0000/dsi@30500000 [ 0.039939] /bus@f0000/i2c@20000000/hdmi@48: Fixed dependency cycle(s) with /hdmi-connector [ 0.040271] /bus@f0000/dss@30220000: Fixed dependency cycle(s) with /bus@f0000/dsi@30500000 [ 0.040315] /bus@f0000/dsi@30500000: Fixed dependency cycle(s) with /bus@f0000/i2c@20000000/hdmi@48 [ 0.040339] /bus@f0000/dsi@30500000: Fixed dependency cycle(s) with /bus@f0000/dss@30220000 [ 0.046837] /bus@f0000/i2c@20000000/hdmi@48: Fixed dependency cycle(s) with /bus@f0000/dsi@30500000 [ 0.046885] /bus@f0000/i2c@20000000/hdmi@48: Fixed dependency cycle(s) with /hdmi-connector [ 0.050915] /bus@f0000/dss@30220000: Fixed dependency cycle(s) with /bus@f0000/dsi@30500000 [ 0.051197] /bus@f0000/dss@30220000: Fixed dependency cycle(s) with /bus@f0000/dsi@30500000 [ 0.051271] /bus@f0000/i2c@20000000/hdmi@48: Fixed dependency cycle(s) with /bus@f0000/dsi@30500000 [ 0.051352] /bus@f0000/dsi@30500000: Fixed dependency cycle(s) with /bus@f0000/i2c@20000000/hdmi@48 [ 0.051378] /bus@f0000/dsi@30500000: Fixed dependency cycle(s) with /bus@f0000/dss@30220000 [ 0.051980] /bus@f0000/usb@f900000/usb@31000000: Fixed dependency cycle(s) with /connector [ 0.052054] /connector: Fixed dependency cycle(s) with /bus@f0000/usb@f900000/usb@31000000 [ 0.053739] Modules: 27936 pages in range for non-PLT usage [ 0.053748] Modules: 519456 pages in range for PLT usage [ 0.054436] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages [ 0.054465] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page [ 0.054478] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages [ 0.054489] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page [ 0.054501] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages [ 0.054512] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page [ 0.054523] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages [ 0.054535] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page [ 0.055954] k3-chipinfo 43000014.chipid: Family:AM62PX rev:SR1.0 JTAGID[0x0bb9d02f] Detected [ 0.056412] iommu: Default domain type: Translated [ 0.056436] iommu: DMA domain TLB invalidation policy: strict mode [ 0.056715] SCSI subsystem initialized [ 0.056861] usbcore: registered new interface driver usbfs [ 0.056894] usbcore: registered new interface driver hub [ 0.056937] usbcore: registered new device driver usb [ 0.057652] mc: Linux media interface: v0.10 [ 0.057698] videodev: Linux video capture interface: v2.00 [ 0.057733] pps_core: LinuxPPS API ver. 1 registered [ 0.057743] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 0.057766] PTP clock support registered [ 0.058019] scmi_core: SCMI protocol bus registered [ 0.058357] Advanced Linux Sound Architecture Driver Initialized. [ 0.059642] clocksource: Switched to clocksource arch_sys_counter [ 0.066218] NET: Registered PF_INET protocol family [ 0.066406] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) [ 0.068037] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) [ 0.068117] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) [ 0.068141] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) [ 0.068261] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) [ 0.068699] TCP: Hash tables configured (established 16384 bind 16384) [ 0.068833] UDP hash table entries: 1024 (order: 3, 32768 bytes, linear) [ 0.068899] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes, linear) [ 0.069082] NET: Registered PF_UNIX/PF_LOCAL protocol family [ 0.069542] RPC: Registered named UNIX socket transport module. [ 0.069565] RPC: Registered udp transport module. [ 0.069574] RPC: Registered tcp transport module. [ 0.069583] RPC: Registered tcp-with-tls transport module. [ 0.069593] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.069615] PCI: CLS 0 bytes, default 64 [ 0.070822] Initialise system trusted keyrings [ 0.071038] workingset: timestamp_bits=46 max_order=19 bucket_order=0 [ 0.071558] NFS: Registering the id_resolver key type [ 0.071594] Key type id_resolver registered [ 0.071605] Key type id_legacy registered [ 0.071668] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 0.071682] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering... [ 0.071840] Key type asymmetric registered [ 0.071853] Asymmetric key parser 'x509' registered [ 0.071902] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245) [ 0.071921] io scheduler mq-deadline registered [ 0.071932] io scheduler kyber registered [ 0.071967] io scheduler bfq registered [ 0.075447] pinctrl-single 4084000.pinctrl: 34 pins, size 136 [ 0.076456] pinctrl-single f4000.pinctrl: 171 pins, size 684 [ 0.120920] Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled [ 0.128364] usbcore: registered new interface driver usb-storage [ 0.128604] i2c_dev: i2c /dev entries driver [ 0.131124] sdhci: Secure Digital Host Controller Interface driver [ 0.131150] sdhci: Copyright(c) Pierre Ossman [ 0.131387] sdhci-pltfm: SDHCI platform and OF driver helper [ 0.132082] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping .... [ 0.132525] usbcore: registered new interface driver usbhid [ 0.132539] usbhid: USB HID core driver [ 0.134185] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available [ 0.136594] NET: Registered PF_PACKET protocol family [ 0.136699] Key type dns_resolver registered [ 0.142598] Loading compiled-in X.509 certificates [ 0.155145] ti-sci 44043000.system-controller: ABI: 4.0 (firmware rev 0x000b '11.0.7--v11.00.07 (Fancy Rat)') [ 0.207330] omap_i2c 4900000.i2c: bus 3 rev0.12 at 100 kHz [ 0.208572] /bus@f0000/bus@b00000/i2c@2b200000/pmic@30: Fixed dependency cycle(s) with /bus@f0000/bus@b00000/i2c@2b200000/pmic@30/regulators/buck2 [ 0.208624] /bus@f0000/bus@b00000/i2c@2b200000/pmic@30: Fixed dependency cycle(s) with /bus@f0000/bus@b00000/i2c@2b200000/pmic@30/regulators/buck1 [ 0.235810] rtc-ds1307 0-0032: oscillator failed, set time! [ 0.236053] rtc-ds1307 0-0032: registered as rtc0 [ 0.236254] rtc-ds1307 0-0032: hctosys: unable to read the hardware clock [ 0.236873] at24 0-0050: supply vcc not found, using dummy regulator [ 0.237514] at24 0-0050: 256 byte 24c02 EEPROM, writable, 16 bytes/write [ 0.237587] omap_i2c 2b200000.i2c: bus 0 rev0.12 at 400 kHz [ 0.239284] /bus@f0000/dsi@30500000: Fixed dependency cycle(s) with /bus@f0000/i2c@20000000/hdmi@48 [ 0.239440] /hdmi-connector: Fixed dependency cycle(s) with /bus@f0000/i2c@20000000/hdmi@48 [ 0.239605] /bus@f0000/i2c@20000000/hdmi@48: Fixed dependency cycle(s) with /bus@f0000/dsi@30500000 [ 0.239757] /bus@f0000/i2c@20000000/hdmi@48: Fixed dependency cycle(s) with /hdmi-connector [ 0.240775] at24 1-0057: supply vcc not found, using dummy regulator [ 0.264243] at24 1-0057: 256 byte 24c02 EEPROM, writable, 16 bytes/write [ 0.264321] omap_i2c 20000000.i2c: bus 1 rev0.12 at 100 kHz [ 0.265620] omap_i2c 20010000.i2c: bus 2 rev0.12 at 100 kHz [ 0.266435] omap_i2c 20030000.i2c: bus 4 rev0.12 at 100 kHz [ 0.266663] ti-sci-intr 4210000.interrupt-controller: Interrupt Router 5 domain created [ 0.266837] ti-sci-intr bus@f0000:interrupt-controller@a00000: Interrupt Router 3 domain created [ 0.267093] debugfs: File ':bus@f0000:bus@48000000:interrupt-controller@48000000' in directory 'domains' already present! [ 0.267149] ti-sci-inta 48000000.interrupt-controller: Interrupt Aggregator domain 28 created [ 0.267432] debugfs: File ':bus@f0000:bus@4e000000:interrupt-controller@4e400000' in directory 'domains' already present! [ 0.267469] ti-sci-inta 4e400000.interrupt-controller: Interrupt Aggregator domain 200 created [ 0.269123] ti-udma 485c0100.dma-controller: Number of rings: 82 [ 0.272272] ti-udma 485c0100.dma-controller: Channels: 48 (bchan: 18, tchan: 12, rchan: 18) [ 0.275071] ti-udma 485c0000.dma-controller: Number of rings: 150 [ 0.280886] ti-udma 485c0000.dma-controller: Channels: 35 (tchan: 20, rchan: 15) [ 0.283044] ti-udma 4e230000.dma-controller: Number of rings: 6 [ 0.283786] ti-udma 4e230000.dma-controller: Channels: 6 (bchan: 0, tchan: 0, rchan: 6) [ 0.285595] 4a00000.serial: ttyS3 at MMIO 0x4a00000 (irq = 302, base_baud = 3000000) is a 8250 [ 0.287166] 2800000.serial: ttyS2 at MMIO 0x2800000 (irq = 303, base_baud = 3000000) is a 8250 [ 0.287235] printk: console [ttyS2] enabled [ 1.795215] 2810000.serial: ttyS0 at MMIO 0x2810000 (irq = 304, base_baud = 3000000) is a 8250 [ 1.804706] omap8250 2860000.serial: PM domain pd:158 will not be powered off [ 1.812646] 2860000.serial: ttyS4 at MMIO 0x2860000 (irq = 305, base_baud = 3000000) is a 8250 [ 1.821405] serial serial0: tty port ttyS4 registered [ 1.867645] davinci_mdio 8000f00.mdio: davinci mdio revision 17.7, bus freq 1000000 [ 1.876723] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA01903, cpsw version 0x6BA81903 Ports: 3 quirks:00000006 [ 1.889616] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.5 [ 1.896756] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512 [ 1.903320] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010d, freq:500000000, add_val:1 pps:0 [ 1.917357] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19 [ 1.930947] cpufreq: cpufreq_online: CPU0: Running at unlisted initial frequency: 1200000 KHz, changing to: 1250000 KHz [ 1.943360] mmc0: CQHCI version 5.10 [ 1.943603] mmc2: CQHCI version 5.10 [ 1.983650] mmc0: SDHCI controller on fa10000.mmc [fa10000.mmc] using ADMA 64-bit [ 1.991184] mmc2: SDHCI controller on fa20000.mmc [fa20000.mmc] using ADMA 64-bit [ 1.991650] davinci_mdio 8000f00.mdio: davinci mdio revision 17.7, bus freq 1000000 [ 2.010291] mdio_bus 8000f00.mdio: MDIO device at address 7 is missing. [ 2.016931] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867 [ 2.026390] sdhci-am654 fa00000.mmc: Got CD GPIO [ 2.029083] clk: Disabling unused clocks [ 2.035267] mmc1: CQHCI version 5.10 [ 2.041518] ALSA device list: [ 2.044519] No soundcards found. [ 2.054292] mmc2: new ultra high speed SDR104 SDIO card at address 0001 [ 2.086976] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit [ 2.094691] Waiting for root device PARTUUID=92b064bd-02... [ 2.136215] random: crng init done [ 2.163715] mmc0: Command Queue Engine enabled [ 2.168285] mmc0: new HS200 MMC card at address 0001 [ 2.176889] mmcblk0: mmc0:0001 S40016 14.6 GiB [ 2.184621] mmcblk0: p1 p2 [ 2.188365] mmcblk0boot0: mmc0:0001 S40016 4.00 MiB [ 2.195463] mmcblk0boot1: mmc0:0001 S40016 4.00 MiB [ 2.202082] mmcblk0rpmb: mmc0:0001 S40016 4.00 MiB, chardev (242:0) [ 2.220545] EXT4-fs (mmcblk0p2): mounted filesystem 45bb78a6-8bb1-4ecc-9029-32f075855806 ro with ordered data mode. Quota mode: disabled. [ 2.232991] VFS: Mounted root (ext4 filesystem) readonly on device 179:2. [ 2.241432] devtmpfs: mounted [ 2.245067] Freeing unused kernel memory: 2560K [ 2.249667] Run /sbin/init as init process [ 2.393276] systemd[1]: System time before build time, advancing clock. [ 2.416099] systemd[1]: Inserted module 'autofs4' [ 2.482220] NET: Registered PF_INET6 protocol family [ 2.488637] Segment Routing with IPv6 [ 2.492370] In-situ OAM (IOAM) with IPv6 [ 2.520846] systemd[1]: systemd 255.17^ running in system mode (+PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP -GCRYPT -GNUTLS -OPENSSL +ACL +BLKID -CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 -BZIP2 -LZ4 -XZ -ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified) [ 2.552966] systemd[1]: Detected architecture arm64. Welcome to [1mTDX Wayland with XWayland 7.3.0-devel-202505+build.7 (scarthgap)[0m! [ 2.576309] systemd[1]: Hostname set to <verdin-am62p-15671821>. [ 6.011629] sched: RT throttling activated [ 25.699719] rcu: INFO: rcu_preempt self-detected stall on CPU [ 25.705477] rcu: 0-....: (5167 ticks this GP) idle=b4fc/1/0x4000000000000000 softirq=166/167 fqs=2624 [ 25.714775] rcu: (t=5253 jiffies g=-791 q=28 ncpus=4) [ 25.719908] CPU: 0 PID: 80 Comm: irq/498-mmc1 Not tainted 6.6.84-7.3.0-devel #1 [ 25.727208] Hardware name: Toradex Verdin AM62P WB on Verdin Development Board (DT) [ 25.734852] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 25.741804] pc : __memset+0xf0/0x188 [ 25.745382] lr : update_sd_lb_stats.constprop.0+0x68/0x91c [ 25.750867] sp : ffffffc080003b40 [ 25.754172] x29: ffffffc080003be0 x28: ffffffc080003db0 x27: 0000000000000000 [ 25.761303] x26: 0000000000000000 x25: ffffff807fb61a90 x24: 0000000000000004 [ 25.768433] x23: ffffffc081138940 x22: ffffffc080f54100 x21: ffffffc080f54100 [ 25.775563] x20: ffffff8000623060 x19: ffffffc080003db0 x18: ffffffc080003d08 [ 25.782692] x17: ffffffc0811f55a0 x16: 00000000461ca29f x15: 000000007d87b00e [ 25.789821] x14: 0000000000000000 x13: 000000000000012d x12: 0000000000000000 [ 25.796950] x11: 0000000000000b2d x10: 0000000000000002 x9 : 0000000000000000 [ 25.804080] x8 : ffffffc080003b90 x7 : 0000000000000000 x6 : 000000000000003f [ 25.811209] x5 : 0000000000000040 x4 : 0000000000000008 x3 : ffffff8000622600 [ 25.818337] x2 : 0000000000000048 x1 : 0000000000000000 x0 : ffffffc080003b88 [ 25.825467] Call trace: [ 25.827907] __memset+0xf0/0x188 [ 25.831132] find_busiest_group+0x4c/0x46c [ 25.835223] load_balance+0x144/0xaa8 [ 25.838881] rebalance_domains+0x268/0x384 [ 25.842972] run_rebalance_domains+0x4c/0x74 [ 25.847237] handle_softirqs+0xf8/0x230 [ 25.851068] __do_softirq+0x14/0x20 [ 25.854551] ____do_softirq+0x10/0x1c [ 25.858207] call_on_irq_stack+0x24/0x4c [ 25.862123] do_softirq_own_stack+0x1c/0x2c [ 25.866301] irq_exit_rcu+0x84/0xc0 [ 25.869784] el1_interrupt+0x38/0x68 [ 25.873356] el1h_64_irq_handler+0x18/0x24 [ 25.877447] el1h_64_irq+0x64/0x68 [ 25.880842] _raw_spin_unlock_irqrestore+0x8/0x44 [ 25.885541] sdhci_thread_irq+0x24/0xcc [ 25.889374] irq_thread_fn+0x2c/0xa8 [ 25.892945] irq_thread+0x160/0x248 [ 25.896428] kthread+0x110/0x114 [ 25.899652] ret_from_fork+0x10/0x20 [ 26.596386] platform 2b300050.target-module: deferred probe pending ***** UART ERROR **** UART Error Code: -1073807253 VISA Read in ModuleProcess.vi:1380011->ClimateChamberControl.viVISA Source: COM7 **********************
For example, it begins to print the kernel dumps during runtime, but still manages to finish the test. However, it can also crash during runtime.
From what I could see in the logs, most of the traces pointed to the SD card interface mmc1. Therefore, I reran the tests with the interface disabled and could not see any issues. I could also see no issues when an SD card was inserted. However, neither of these workarounds is a permanent solution for us.
Do you have any idea what is triggering the issues that seem to be strongly related to the MMC1 interface?
From a hardware point of view, the four data lanes and the CMD lane are pulled up to the 3.3 V/1.8 V voltage rail. Due to some mechanical issues, the 49.9 kΩ MMC1_CLK pull-down resistor was not assembled. After identifying this issue, we reran the test with the resistor assembled, but the issue is still there.
We would still like to know what issues could be triggered by not having the MMC1_CLK pull-down, and whether it would also be safe to not assemble it so that we would not need to redesign our board.
Please let me know if you need any further information.
Regards
Franz
