AM625: Boot Stall and Crashes

Part Number: AM625
Other Parts Discussed in Thread: SK-AM62-LP, SK-AM62B-P1, SK-AM62

Hello,

This is a follow-up of previous threads and discussion from months ago:

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1268222/am625-boot-stall-and-rcu-preempt-stall

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1309832/am625-crash-internal-error-synchronous-external-abort-in-mcan-driver-at-low-temperatures-20-degrees-celsius/4978691 

Software/Hardware configuration:

- TI SDK 09.02.01.09

- Custom HW (various configurations of memory size and memory vendor, different SOC variants: AM6254, AM6231, AM6252)

- Kernel sources: https://git.toradex.com/cgit/linux-toradex.git/log/?h=toradex_ti-linux-6.1.y 

- U-Boot sources: https://git.toradex.com/cgit/u-boot-toradex.git/log/?h=toradex_ti-u-boot-2023.04 

- The DDR was validated both internally by us and by the memory vendor on its premises. We have variants with memory with different sizes and vendors and they are all affected by the same issues

- We have variant with 0.75V and 0.85V VDD_CORE, no differences on the behavior because of this

During the testing of our software/hardware, we faced some issues with the boot stalling in different ways. They all happen below -20 Celsius degrees.

1. system stalls with the following error.

[   86.779029] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[   86.785173] rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-0): P186/1:b..l
[   86.792151] 	(detected by 0, t=18276 jiffies, g=6889, q=115 ncpus=1)
[   86.798506] task:systemd-udevd   state:R  running task     stack:0     pid:186   ppid:173    flags:0x00000a04
[   86.808424] Call trace:
[   86.810870]  __switch_to+0xd4/0x130
[   86.814374]  __schedule+0x230/0x664
[   86.817863]  preempt_schedule_irq+0x44/0xd0
[   86.822047]  el1_interrupt+0x4c/0x70
[   86.825626]  el1h_64_irq_handler+0x18/0x2c
[   86.829725]  el1h_64_irq+0x64/0x68
[   86.833130]  set_root+0x64/0xfc
[   86.836275]  nd_jump_root+0xb8/0xf0
[   86.839764]  path_init+0x2d0/0x380
[   86.843167]  path_parentat+0x20/0xb0
[   86.846743]  __filename_parentat+0xa4/0x1c0
[   86.850927]  do_unlinkat+0x68/0x270
[   86.854416]  __arm64_sys_unlinkat+0x40/0x84
[   86.858600]  invoke_syscall+0x48/0x114
[   86.862352]  el0_svc_common.constprop.0+0xd4/0xfc
[   86.867057]  do_el0_svc+0x20/0x30
[   86.870372]  el0_svc+0x28/0xa0
[   86.873428]  el0t_64_sync_handler+0xbc/0x140
[   86.877701]  el0t_64_sync+0x18c/0x190
[   86.881364] rcu: rcu_preempt kthread timer wakeup didn't happen for 18273 jiffies! g6889 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[   86.892749] rcu: 	Possible timer handling issue on cpu=0 timer-softirq=2071
[   86.899705] rcu: rcu_preempt kthread starved for 18274 jiffies! g6889 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
[   86.910136] rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[   86.919260] rcu: RCU grace-period kthread stack dump:
[   86.924305] task:rcu_preempt     state:I stack:0     pid:14    ppid:2      flags:0x00000008
[   86.932656] Call trace:
[   86.935099]  __switch_to+0xd4/0x130
[   86.938587]  __schedule+0x230/0x664
[   86.942074]  schedule+0x5c/0xcc
[   86.945216]  schedule_timeout+0x84/0xfc
[   86.949055]  rcu_gp_fqs_loop+0x12c/0x46c
[   86.952983]  rcu_gp_kthread+0x128/0x164
[   86.956820]  kthread+0x10c/0x110
[   86.960051]  ret_from_fork+0x10/0x20
[   86.963627] rcu: Stack dump where RCU GP kthread last ran:
[   86.969111] CPU: 0 PID: 181 Comm: systemd-udevd Not tainted 6.1.80+git.0d9f1748ac17 #1
[   86.977026] Hardware name: Toradex Verdin AM62 WB on Verdin Development Board (DT)
[   86.984591] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   86.991550] pc : iomap_read_reg+0x10/0x30 [m_can_platform]
[   86.997049] lr : m_can_get_berr_counter+0x3c/0x10c [m_can]
[   87.002548] sp : ffff8000099735f0
[   87.005858] x29: ffff8000099735f0 x28: 0000000000000240 x27: ffff800000cca1a0
[   87.012998] x26: ffff000003f9e000 x25: ffff000003f9e240 x24: 0000000000000000
[   87.020137] x23: 0000000000000000 x22: ffff0000002ae410 x21: ffff0000035f8000
[   87.027276] x20: ffff0000035f8980 x19: ffff80000997362c x18: 0000000000000000
[   87.034414] x17: ffff800014add000 x16: ffff800008000000 x15: 0000000000000000
[   87.041552] x14: 000000000000022f x13: 0000000000000001 x12: 0000000000000001
[   87.048691] x11: 0000000000000000 x10: 00000000000009b0 x9 : ffff8000099731e0
[   87.055829] x8 : ffff00001da1b180 x7 : 0000010000108100 x6 : ffff0000002ae590
[   87.062968] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[   87.070106] x2 : ffff800000ca4000 x1 : 0000000000000040 x0 : 000000000071ff00
[   87.077244] Call trace:
[   87.079687]  iomap_read_reg+0x10/0x30 [m_can_platform]
[   87.084832]  can_fill_info+0x108/0x524 [can_dev]
[   87.089477]  rtnl_fill_ifinfo+0x844/0x11b0
[   87.093578]  rtnl_getlink+0x23c/0x424
[   87.097239]  rtnetlink_rcv_msg+0x130/0x3a0
[   87.101334]  netlink_rcv_skb+0x60/0x130
[   87.105174]  rtnetlink_rcv+0x18/0x2c
[   87.108755]  netlink_unicast+0x2f0/0x360
[   87.112677]  netlink_sendmsg+0x1b0/0x420
[   87.116600]  __sys_sendto+0x138/0x174
[   87.120265]  __arm64_sys_sendto+0x28/0x3c
[   87.124275]  invoke_syscall+0x48/0x114
[   87.128024]  el0_svc_common.constprop.0+0xd4/0xfc
[   87.132729]  do_el0_svc+0x20/0x30
[   87.136044]  el0_svc+0x28/0xa0
[   87.139100]  el0t_64_sync_handler+0xbc/0x140
[   87.143372]  el0t_64_sync+0x18c/0x190

Complete logs:

24-04-17_22-12-36-Fail.txt
Climate Chamber:       KWP 120
Start Temperature:     -29.2�C
End Temperature:       -28.3�C
Measured Temperature:  -29.2�C
Shelf Temperature:     -28.5�C
Shelf Name:            Shelf_33
Module Serial Number:  15313618
Module Location:       Shelf 1, Carrier Board 5
Test duration:         157.60s

Test Log
********
22:12:35.84 13->14 Waited 60.00s
22:11:35.75 12->13 Set Reset Low
22:11:35.68 11->12 Power Off
22:11:35.62 5->11 Not found, time out: "login:"
22:10:05.48 4->5 Message found: "Starting kernel"
22:10:00.04 3->4 Message found: "U-Boot"
22:09:59.93 2->3 Waited 1.00s
22:09:58.79 1->2 Set Reset High
22:09:58.68 0->1 Power On

Search Strings
**************
PASS : U-Boot
PASS : Less than 64MB RAM detected
PASS : Booting from mmc failed!
PASS : Starting kernel
PASS : mmcblk0: mmc0
PASS : cut here

FAIL : rcu: INFO: rcu_preempt detected
PASS : rcu: INFO: rcu_preempt self-detected stall
PASS : Internal error
PASS : I/O error
FAIL : login:
FAIL : I2C test, TestResult: PASS
FAIL : RTC test, TestResult: PASS
FAIL : Audio test, TestResult: PASS
FAIL : Wifi test, TestResult: PASS
FAIL : Bluetooth test, TestResult: PASS
FAIL : CPU stress test, TestResult: PASS
PASS : critical temperature reached
FAIL : Ethernet test, TestResult: PASS
FAIL : connected with
FAIL : RAM test, TestResult: PASS
PASS : memtester failed
FAIL : memtester version
PASS : FAILURE
PASS : running CQE recovery
FAIL : eMMC Nand test, TestResult: PASS
PASS : bonnie\+\+ failed
FAIL : Waiting for power-cycle
0 : OAK ERROR
1 : UART ERROR
OK : registered as rtc0
OK : registered as rtc1
OK : xception stack
N/A : Device life time estimation type B 
N/A : Pre EOL information
2 : USB hub found
0 : xception stack
PASS : page allocation failure
PASS : hwclock

Terminal
********
�
U-Boot SPL 2024.04-00035-gfc942d8282-dirty (Apr 16 2024 - 11:52:32 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)')
SPL initial stack usage: 13368 bytes
Trying to boot from MMC1
Authentication passed
Authentication passed
Authentication passed
Authentication passed
Authentication passed
Starting ATF on ARM64 core...

NOTICE:  BL31: v2.10.0	(release):v2.10.0-340-g4da4a1a61
NOTICE:  BL31: Built : 11:30:01, Feb  8 2024
I/TC: 
I/TC: OP-TEE version: 4.1.0-71-g439c5ecbb (gcc version 12.3.1 20230626 (Arm GNU Toolchain 12.3.Rel1 (Build arm-12.35))) #5 Fri Feb 23 13:00:52 UTC 2024 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: GIC redistributor base address not provided
I/TC: Assuming default GIC group status and modifier
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)')
I/TC: HUK Initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2024.04-00035-gfc942d8282-dirty (Apr 16 2024 - 11:47:25 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)')
SPL initial stack usage: 1840 bytes
Trying to boot from MMC1
Authentication passed
Authentication passed


U-Boot 2024.04-00035-gfc942d8282-dirty (Apr 16 2024 - 11:47:25 +0200)

SoC:   AM62X SR1.0 HS-FS
DRAM:  512 MiB
Core:  138 devices, 29 uclasses, devicetree: separate
MMC:   mmc@fa10000: 0, mmc@fa00000: 1
Loading Environment from MMC... OK
MISSING TORADEX CARRIER CONFIG BLOCKS
In:    serial@2800000
Out:   serial@2800000
Err:   serial@2800000
Model: Toradex 0072 Verdin AM62 Solo 512MB WB IT V1.1B
Serial#: 15313618
am65_cpsw_nuss ethernet@8000000: K3 CPSW: nuss_ver: 0x6BA01103 cpsw_ver: 0x6BA81103 ale_ver: 0x00290105 Ports:2 mdio_freq:1000000
Setting variant to wifi
Net:   
Warning: ethernet@8000000port@1 MAC addresses don't match:
Address in ROM is		64:1c:10:2a:76:c6
Address in environment is	00:14:2d:e9:aa:d2
eth0: ethernet@8000000port@1 [PRIME]Could not get PHY for ethernet@8000000port@1: addr 7
am65_cpsw_nuss_port ethernet@8000000port@2: phy_connect() failed

Hit any key to stop autoboot:  1  0 
MMC: no card present
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
6003 bytes read in 15 ms (390.6 KiB/s)
## Executing script at 90280000
Loading DeviceTree: k3-am625-verdin-wifi-dev.dtb
70049 bytes read in 16 ms (4.2 MiB/s)
82 bytes read in 14 ms (4.9 KiB/s)
Working FDT set to 90200000
Applying Overlay: verdin-am62_dsi-to-hdmi_overlay.dtbo
3050 bytes read in 15 ms (198.2 KiB/s)
Applying Overlay: verdin-am62_spidev_overlay.dtbo
560 bytes read in 15 ms (36.1 KiB/s)
7861931 bytes read in 76 ms (98.7 MiB/s)
Bootargs: root=PARTUUID=f360aa55-02 ro rootwait console=tty1 console=ttyS2,115200 consoleblank=0 earlycon=ns16550a,mmio32,0x02800000
   Uncompressing Kernel Image to 0
## Flattened Device Tree blob at 90200000
   Booting using the fdt blob at 0x90200000
Working FDT set to 90200000
ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4)
ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4)
   Loading Device Tree to 0000000098ed5000, end 0000000098f09fff ... OK
Working FDT set to 98ed5000

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 6.1.80+git.0d9f1748ac17 (oe-user@oe-host) (aarch64-tdx-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Thu Mar 28 13:48:11 UTC 2024
[    0.000000] Machine model: Toradex Verdin AM62 WB on Verdin Development Board
[    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009db00000, size 12 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@9db00000, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x000000009fffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x000000009dafffff]
[    0.000000]   node   0: [mem 0x000000009db00000-0x000000009e6fffff]
[    0.000000]   node   0: [mem 0x000000009e700000-0x000000009e77ffff]
[    0.000000]   node   0: [mem 0x000000009e780000-0x000000009fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000009fffffff]
[    0.000000] cma: Reserved 128 MiB at 0x0000000090e00000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.4
[    0.000000] percpu: Embedded 20 pages/cpu s41064 r8192 d32664 u81920
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: ARM erratum 845719
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 129024
[    0.000000] Kernel command line: root=PARTUUID=f360aa55-02 ro rootwait console=tty1 console=ttyS2,115200 consoleblank=0 earlycon=ns16550a,mmio32,0x02800000
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 326452K/524288K available (11072K kernel code, 1354K rwdata, 3784K rodata, 2176K init, 422K bss, 66764K reserved, 131072K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=1.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001880000
[    0.000000] ITS [mem 0x01820000-0x0182ffff]
[    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
[    0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
[    0.000000] ITS@0x0000000001820000: allocated 524288 Devices @80800000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x0000000080030000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000080040000
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0x3ffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
[    0.000001] sched_clock: 58 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
[    0.008741] Console: colour dummy device 80x25
[    0.013998] printk: console [tty1] enabled
[    0.018258] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000)
[    0.028968] pid_max: default: 32768 minimum: 301
[    0.033771] LSM: Security Framework initializing
[    0.038658] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.046167] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.056427] cblist_init_generic: Setting adjustable number of callback queues.
[    0.063928] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.070273] cblist_init_generic: Setting adjustable number of callback queues.
[    0.077696] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.084180] rcu: Hierarchical SRCU implementation.
[    0.089112] rcu: 	Max phase no-delay instances is 1000.
[    0.095333] Platform MSI: msi-controller@1820000 domain created
[    0.102075] EFI services will not be available.
[    0.106997] smp: Bringing up secondary CPUs ...
[    0.111667] smp: Brought up 1 node, 1 CPU
[    0.115785] SMP: Total of 1 processors activated.
[    0.120617] CPU features: detected: 32-bit EL0 Support
[    0.125892] CPU features: detected: CRC32 instructions
[    0.131247] CPU: All CPU(s) started at EL2
[    0.135451] alternatives: applying system-wide alternatives
[    0.142834] devtmpfs: initialized
[    0.158651] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.168727] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[    0.182721] pinctrl core: initialized pinctrl subsystem
[    0.189133] DMI not present or invalid.
[    0.194026] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.201430] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.208807] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.216851] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.225077] audit: initializing netlink subsys (disabled)
[    0.231398] thermal_sys: Registered thermal governor 'step_wise'
[    0.231411] thermal_sys: Registered thermal governor 'power_allocator'
[    0.237604] audit: type=2000 audit(0.148:1): state=initialized audit_enabled=0 res=1
[    0.252296] cpuidle: using governor menu
[    0.256580] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.263608] ASID allocator initialised with 65536 entries
[    0.290281] KASLR disabled due to lack of seed
[    0.303826] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.310960] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page
[    0.317395] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages
[    0.324354] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page
[    0.330781] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.337739] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
[    0.344165] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages
[    0.351122] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
[    0.359487] k3-chipinfo 43000014.chipid: Family:AM62X rev:SR1.0 JTAGID[0x0bb7e02f] Detected
[    0.369941] iommu: Default domain type: Translated 
[    0.375013] iommu: DMA domain TLB invalidation policy: strict mode 
[    0.381818] SCSI subsystem initialized
[    0.386153] usbcore: registered new interface driver usbfs
[    0.391852] usbcore: registered new interface driver hub
[    0.397333] usbcore: registered new device driver usb
[    0.403190] pps_core: LinuxPPS API ver. 1 registered
[    0.408292] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.417668] PTP clock support registered
[    0.421892] EDAC MC: Ver: 3.0.0
[    0.426568] FPGA manager framework
[    0.430217] Advanced Linux Sound Architecture Driver Initialized.
[    0.437814] clocksource: Switched to clocksource arch_sys_counter
[    0.444498] VFS: Disk quotas dquot_6.6.0
[    0.448589] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.463527] NET: Registered PF_INET protocol family
[    0.469038] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.477741] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.486421] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.494378] TCP established hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.502365] TCP bind hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.509964] TCP: Hash tables configured (established 4096 bind 4096)
[    0.516621] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.523343] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.530665] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.537040] RPC: Registered named UNIX socket transport module.
[    0.543174] RPC: Registered udp transport module.
[    0.548003] RPC: Registered tcp transport module.
[    0.552828] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.559439] NET: Registered PF_XDP protocol family
[    0.565276] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
[    0.576109] Initialise system trusted keyrings
[    0.581058] workingset: timestamp_bits=46 max_order=17 bucket_order=0
[    0.594210] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.601028] NFS: Registering the id_resolver key type
[    0.606329] Key type id_resolver registered
[    0.610625] Key type id_legacy registered
[    0.614824] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.621702] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.685005] Key type asymmetric registered
[    0.689221] Asymmetric key parser 'x509' registered
[    0.694341] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    0.702143] io scheduler mq-deadline registered
[    0.706816] io scheduler kyber registered
[    0.716168] pinctrl-single 4084000.pinctrl: 34 pins, size 136
[    0.723664] pinctrl-single f4000.pinctrl: 171 pins, size 684
[    0.739628] Serial: 8250/16550 driver, 6 ports, IRQ sharing enabled
[    0.759811] brd: module loaded
[    0.768904] loop: module loaded
[    0.777614] tun: Universal TUN/TAP device driver, 1.6
[    0.784036] VFIO - User Level meta-driver version: 0.3
[    0.790746] usbcore: registered new interface driver usb-storage
[    0.797906] i2c_dev: i2c /dev entries driver
[    0.805040] sdhci: Secure Digital Host Controller Interface driver
[    0.811447] sdhci: Copyright(c) Pierre Ossman
[    0.816361] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.823008] ledtrig-cpu: registered to indicate activity on CPUs
[    0.829415] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[    0.836562] usbcore: registered new interface driver usbhid
[    0.842285] usbhid: USB HID core driver
[    0.847709] optee: probing for conduit method.
I/TC: Reserved shared memory is enabled
I/TC: Dynamic shared memory is enabled
I/TC: Normal World virtualization support is disabled
I/TC: Asynchronous notifications are disabled
[    0.852348] optee: revision 4.1 (439c5ecb)
[    0.869055] optee: dynamic shared memory is enabled
[    0.878624] optee: initialized driver
[    0.884857] Initializing XFRM netlink socket
[    0.889476] NET: Registered PF_PACKET protocol family
[    0.894803] Key type dns_resolver registered
[    0.899734] registered taskstats version 1
[    0.903977] Loading compiled-in X.509 certificates
[    0.923699] ti-sci 44043000.system-controller: ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)')
[    1.015166] omap_i2c 4900000.i2c: bus 3 rev0.12 at 100 kHz
[    1.023018] i2c 0-0030: Fixed dependency cycle(s) with /bus@f0000/i2c@20000000/pmic@30/regulators/buck2
[    1.064072] input: tps65219-pwrbutton as /devices/platform/bus@f0000/20000000.i2c/i2c-0/0-0030/tps65219-pwrbutton.2.auto/input/input0
[    1.078851] rtc-ds1307 0-0032: oscillator failed, set time!
[    1.084853] rtc-ds1307 0-0032: registered as rtc0
[    1.089861] rtc-ds1307 0-0032: hctosys: unable to read the hardware clock
[    1.099624] at24 0-0050: supply vcc not found, using dummy regulator
[    1.106713] at24 0-0050: 256 byte 24c02 EEPROM, writable, 16 bytes/write
[    1.113668] omap_i2c 20000000.i2c: bus 0 rev0.12 at 400 kHz
[    1.122378] pca953x 1-0021: supply vcc not found, using dummy regulator
[    1.129424] pca953x 1-0021: using no AI
[    1.153963] pca953x 1-0021: failed writing register
[    1.159208] pca953x: probe of 1-0021 failed with error -121
[    1.165743] at24 1-0057: supply vcc not found, using dummy regulator
[    1.173087] at24 1-0057: 256 byte 24c02 EEPROM, writable, 16 bytes/write
[    1.180058] omap_i2c 20010000.i2c: bus 1 rev0.12 at 100 kHz
[    1.187228] omap_i2c 20020000.i2c: bus 2 rev0.12 at 100 kHz
[    1.194156] omap_i2c 20030000.i2c: bus 4 rev0.12 at 100 kHz
[    1.200251] ti-sci-intr 4210000.interrupt-controller: Interrupt Router 5 domain created
[    1.208680] ti-sci-intr bus@f0000:interrupt-controller@a00000: Interrupt Router 3 domain created
[    1.218050] ti-sci-inta 48000000.interrupt-controller: Interrupt Aggregator domain 28 created
[    1.231961] ti-udma 485c0100.dma-controller: Number of rings: 82
[    1.241098] ti-udma 485c0100.dma-controller: Channels: 48 (bchan: 18, tchan: 12, rchan: 18)
[    1.253263] ti-udma 485c0000.dma-controller: Number of rings: 150
[    1.264895] ti-udma 485c0000.dma-controller: Channels: 35 (tchan: 20, rchan: 15)
[    1.276269] 4a00000.serial: ttyS3 at MMIO 0x4a00000 (irq = 291, base_baud = 3000000) is a 8250
[    1.286834] 2800000.serial: ttyS2 at MMIO 0x2800000 (irq = 292, base_baud = 3000000) is a 8250
[    1.295832] printk: console [ttyS2] enabled
[    1.295832] printk: console [ttyS2] enabled
[    1.304335] printk: bootconsole [ns16550a0] disabled
[    1.304335] printk: bootconsole [ns16550a0] disabled
[    1.315862] 2810000.serial: ttyS0 at MMIO 0x2810000 (irq = 293, base_baud = 3000000) is a 8250
[    1.325583] omap8250 2850000.serial: PM domain pd:156 will not be powered off
[    1.333146] 2850000.serial: ttyS4 at MMIO 0x2850000 (irq = 294, base_baud = 3000000) is a 8250
[    1.341983] serial serial0: tty port ttyS4 registered
[    1.348868] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
[    1.393827] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
[    1.402951] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA01103, cpsw version 0x6BA81103 Ports: 3 quirks:00000006
[    1.416017] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.5
[    1.423205] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512
[    1.430149] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:0
[    1.440812] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[    1.458918] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller
[    1.464520] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 1
[    1.472354] xhci-hcd xhci-hcd.3.auto: USB3 root hub has no ports
[    1.478380] xhci-hcd xhci-hcd.3.auto: hcc params 0x0258fe6d hci version 0x110 quirks 0x0000008020010010
[    1.487875] xhci-hcd xhci-hcd.3.auto: irq 299, io mem 0x31100000
[    1.495008] hub 1-0:1.0: USB hub found
[    1.498876] hub 1-0:1.0: 1 port detected
[    1.635583] mmc0: CQHCI version 5.10
[    1.642498] gpio-488 (PCIE_1_RESET#): hogged as output/low
[    1.650560] gpio-427 (CTRL_SLEEP_MOCI#): hogged as output/high
[    1.667963] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
[    1.687190] mmc0: SDHCI controller on fa10000.mmc [fa10000.mmc] using ADMA 64-bit
[    1.713815] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
[    1.726778] mdio_bus 8000f00.mdio: MDIO device at address 7 is missing.
[    1.733476] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867
[    1.745358] sdhci-am654 fa00000.mmc: Got CD GPIO
[    1.751185] sdhci-am654 fa20000.mmc: allocated mmc-pwrseq
[    1.757925] debugfs: Directory 'pd:182' with parent 'pm_genpd' already present!
[    1.766019] mmc1: CQHCI version 5.10
[    1.769949] usb 1-1: new high-speed USB device number 2 using xhci-hcd
[    1.776786] mmc2: CQHCI version 5.10
[    1.787674] input: gpio-keys as /devices/platform/gpio-keys/input/input1
[    1.794814] mmc0: Command Queue Engine enabled
[    1.799325] mmc0: new HS200 MMC card at address 0001
[    1.807267] mmcblk0: mmc0:0001 M04A11 3.64 GiB 
[    1.815734]  mmcblk0: p1 p2
[    1.820300] mmcblk0boot0: mmc0:0001 M04A11 4.00 MiB 
[    1.827956] mmcblk0boot1: mmc0:0001 M04A11 4.00 MiB 
[    1.833042] mmc2: SDHCI controller on fa20000.mmc [fa20000.mmc] using ADMA 64-bit
[    1.842782] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
[    1.850985] mmcblk0rpmb: mmc0:0001 M04A11 4.00 MiB, chardev (241:0)
[    1.857436] ALSA device list:
[    1.861226]   No soundcards found.
[    1.872204] EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem
[    1.879658] EXT4-fs (mmcblk0p2): write access will be enabled during recovery
[    1.917200] mmc2: new ultra high speed DDR50 SDIO card at address 0001
[    1.929960] EXT4-fs (mmcblk0p2): recovery complete
[    1.939642] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Quota mode: none.
[    1.948436] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[    1.957168] devtmpfs: mounted
[    1.961742] Freeing unused kernel memory: 2176K
[    1.966514] Run /sbin/init as init process
[    2.006634] hub 1-1:1.0: USB hub found
[    2.011231] hub 1-1:1.0: 4 ports detected
[    2.102682] systemd[1]: System time before build time, advancing clock.
[    2.149156] NET: Registered PF_INET6 protocol family
[    2.155807] Segment Routing with IPv6
[    2.159737] In-situ OAM (IOAM) with IPv6
[    2.188938] systemd[1]: systemd 250.5+ running in system mode (+PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP -GCRYPT -GNUTLS -OPENSSL +ACL +BLKID -CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -BZIP2 -LZ4 -XZ -ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=hybrid)
[    2.221111] systemd[1]: Detected architecture arm64.

Welcome to TDX Wayland with XWayland 6.6.0+build.12 (kirkstone)!

[    2.271706] systemd[1]: Hostname set to <verdin-am62-15313618>.
[    2.357979] usb 1-1.4: new high-speed USB device number 3 using xhci-hcd
[    2.894162] systemd[1]: Queued start job for default target Graphical Interface.
[    2.953744] systemd[1]: Created slice Slice /system/getty.
[  OK  ] Created slice Slice /system/getty.
[    2.976737] systemd[1]: Created slice Slice /system/modprobe.
[  OK  ] Created slice Slice /system/modprobe.
[    3.000622] systemd[1]: Created slice Slice /system/serial-getty.
[  OK  ] Created slice Slice /system/serial-getty.
[    3.024175] systemd[1]: Created slice User and Session Slice.
[  OK  ] Created slice User and Session Slice.
[    3.046545] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[    3.070547] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Started Forward Password R…uests to Wall Directory Watch.
[    3.094609] systemd[1]: Reached target Host and Network Name Lookups.
[  OK  ] Reached target Host and Network Name Lookups.
[    3.118055] systemd[1]: Reached target Path Units.
[  OK  ] Reached target Path Units.
[    3.138407] systemd[1]: Reached target Remote File Systems.
[  OK  ] Reached target Remote File Systems.
[    3.162210] systemd[1]: Reached target Slice Units.
[  OK  ] Reached target Slice Units.
[    3.182196] systemd[1]: Reached target Swaps.
[  OK  ] Reached target Swaps.
[    3.244264] systemd[1]: Listening on RPCbind Server Activation Socket.
[  OK  ] Listening on RPCbind Server Activation Socket.
[    3.266502] systemd[1]: Reached target RPC Port Mapper.
[  OK  ] Reached target RPC Port Mapper.
[    3.291812] systemd[1]: Listening on Syslog Socket.
[  OK  ] Listening on Syslog Socket.
[    3.314946] systemd[1]: Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[    3.340011] systemd[1]: Listening on Journal Audit Socket.
[  OK  ] Listening on Journal Audit Socket.
[    3.363293] systemd[1]: Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket (/dev/log).
[    3.387496] systemd[1]: Listening on Journal Socket.
[  OK  ] Listening on Journal Socket.
[    3.412464] systemd[1]: Listening on Network Service Netlink Socket.
[  OK  ] Listening on Network Service Netlink Socket.
[    3.436099] systemd[1]: Listening on udev Control Socket.
[  OK  ] Listening on udev Control Socket.
[    3.459534] systemd[1]: Listening on udev Kernel Socket.
[  OK  ] Listening on udev Kernel Socket.
[    3.485090] systemd[1]: Listening on User Database Manager Socket.
[  OK  ] Listening on User Database Manager Socket.
[    3.534563] systemd[1]: Mounting Huge Pages File System...
         Mounting Huge Pages File System...
[    3.574641] systemd[1]: Mounting POSIX Message Queue File System...
         Mounting POSIX Message Queue File System...
[    3.618600] systemd[1]: Mounting Kernel Debug File System...
         Mounting Kernel Debug File System...
[    3.637393] systemd[1]: Kernel Trace File System was skipped because of a failed condition check (ConditionPathExists=/sys/kernel/tracing).
[    3.674808] systemd[1]: Mounting Temporary Directory /tmp...
         Mounting Temporary Directory /tmp...
[    3.722804] systemd[1]: Starting Create List of Static Device Nodes...
         Starting Create List of Static Device Nodes...
[    3.770812] systemd[1]: Starting Load Kernel Module configfs...
         Starting Load Kernel Module configfs...
[    3.818819] systemd[1]: Starting Load Kernel Module drm...
         Starting Load Kernel Module drm...
[    3.855089] systemd[1]: Starting Load Kernel Module fuse...
         Starting Load Kernel Module fuse...
[    3.880254] fuse: init (API version 7.37)
[    3.906760] systemd[1]: Starting RPC Bind...
         Starting RPC Bind...
[    3.945471] systemd[1]: Starting File System Check on Root Device...
         Starting File System Check on Root Device...
[    3.999047] systemd[1]: Starting Journal Service...
         Starting Journal Service...
[    4.055247] systemd[1]: Starting Load Kernel Modules...
         Starting Load Kernel Modules...
[    4.108670] systemd[1]: Starting Generate network units from Kernel command line...
         Starting Generate network …ts from Kernel command line...
[    4.174316] systemd[1]: Starting Coldplug All udev Devices...
         Starting Coldplug All udev Devices...
[    4.252030] systemd[1]: Started RPC Bind.
[  OK  ] Started RPC Bind.
[    4.291107] systemd[1]: Mounted Huge Pages File System.
[  OK  ] Mounted Huge Pages File System.
[    4.323111] systemd[1]: Mounted POSIX Message Queue File System.
[  OK  ] Mounted POSIX Message Queue File System.
[    4.359027] systemd[1]: Mounted Kernel Debug File System.
[  OK  ] Mounted Kernel Debug File System.
[    4.391037] systemd[1]: Mounted Temporary Directory /tmp.
[  OK  ] Mounted Temporary Directory /tmp.
[    4.424539] systemd[1]: Finished Create List of Static Device Nodes.
[  OK  ] Finished Create List of Static Device Nodes.
[    4.463771] systemd[1]: modprobe@configfs.service: Deactivated successfully.
[    4.490396] systemd[1]: Finished Load Kernel Module configfs.
[  OK  ] Finished Load Kernel Module configfs.
[    4.527063] systemd[1]: Started Journal Service.
[  OK  ] Started Journal Service.
[  OK  ] Finished Load Kernel Module drm.
[  OK  ] Finished Load Kernel Module fuse.
[  OK  ] Finished File System Check on Root Device.
[  OK  ] Finished Load Kernel Modules.
[  OK  ] Finished Generate network units from Kernel command line.
         Mounting FUSE Control File System...
         Mounting Kernel Configuration File System...
         Starting Remount Root and Kernel File Systems...
         Starting Apply Kernel Variables...
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Mounted Kernel Configuration File System.
[  OK  ] Finished Apply Kernel Variables.
[    4.986677] EXT4-fs (mmcblk0p2): re-mounted. Quota mode: none.
[  OK  ] Finished Remount Root and Kernel File Systems.
         Starting Flush Journal to Persistent Storage...
         Starting Create Static Device Nodes in /dev...
[    5.122533] systemd-journald[154]: Received client request to flush runtime journal.
[  OK  ] Finished Flush Journal to Persistent Storage.
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Preparation for Local File Systems.
         Mounting /var/volatile...
[    5.300098] audit: type=1334 audit(1651167748.196:2): prog-id=5 op=LOAD
[    5.325154] audit: type=1334 audit(1651167748.208:3): prog-id=6 op=LOAD
         Starting Rule-based Manage…for Device Events and Files...
[  OK  ] Mounted /var/volatile.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Create Volatile Files and Directories.
         Starting Network Time Synchronization...
         Starting Record System Boot/Shutdown in UTMP...
[  OK  ] Started Rule-based Manager for Device Events and Files.
[  OK  ] Finished Record System Boot/Shutdown in UTMP.
[  OK  ] Finished Coldplug All udev Devices.
         Starting Wait for udev To …plete Device Initialization...
[    6.449919] random: crng init done
[  OK  ] Finished Load/Save Random Seed.
[  OK  ] Started Network Time Synchronizatio[    6.945149] systemd-journald[154]: Oldest entry in /run/log/journal/98d6394cd75d4f2dad68076443517daa/system.journal is older than the configured file retention duration (1month), suggesting rotation.
n.
[  OK  ] Reached target System Time Set.
[    7.020112] systemd-journald[154]: /run/log/journal/98d6394cd75d4f2dad68076443517daa/system.journal: Journal header limits reached or header out-of-date, rotating.
[    7.521756] lm75 0-0048: supply vs not found, using dummy regulator
[    7.541716] omap8250 2b300000.serial: No clock speed specified: using default: 48000000
[    7.575821] 2b300000.serial: ttyS1 at MMIO 0x2b300000 (irq = 485, base_baud = 3000000) is a 8250
[    7.631547] hwmon hwmon1: temp1_input not attached to any thermal zone
[    7.656301] ina2xx 1-0040: error configuring the device: -121
[    7.665554] nau8822 1-001a: Failed to issue reset: -121
[    7.707843] lm75 0-0048: hwmon1: sensor 'tmp1075'
[    7.721699] nau8822: probe of 1-001a failed with error -121
[    7.744113] CAN device driver interface
[    7.766785] lm75 1-004f: supply vs not found, using dummy regulator
[    7.853274] hwmon hwmon2: temp1_input not attached to any thermal zone
[    7.866875] m_can_platform 4e08000.can: m_can device registered (irq=0, version=32)
[    7.929951] lm75 1-004f: hwmon2: sensor 'tmp75c'
[  OK  ] Found device /dev/ttyS2.
[    7.970693] m_can_platform 20701000.can: m_can device registered (irq=487, version=32)
[    8.333694] Bluetooth: Core ver 2.22
[    8.373929] k3-m4-rproc 5000000.m4fss: device does not reserved memory regions, ret = -22
[    8.423953] NET: Registered PF_BLUETOOTH protocol family
[    8.471681] k3-m4-rproc 5000000.m4fss: reserved memory init failed, ret = -22
[    8.492952] Bluetooth: HCI device and connection manager initialized
[    8.547050] remoteproc remoteproc0: releasing 5000000.m4fss
[    8.555637] Bluetooth: HCI socket layer initialized
[    8.611257] Bluetooth: L2CAP socket layer initialized
[    8.616886] k3-m4-rproc: probe of 5000000.m4fss failed with error -22
[    8.686371] Bluetooth: SCO socket layer initialized
[    8.818317] rtc-ti-k3 2b1f0000.rtc: registered as rtc1
[    8.858490] platform 78000000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1)
[    9.037121] platform 78000000.r5f: configured R5F for IPC-only mode
[    9.117604] platform 78000000.r5f: device does not have reserved memory regions, ret = -22
[    9.139396] lt8912 1-0048: supply vdd not found, using dummy regulator
[    9.150509] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    9.158689] lt8912 1-0048: supply vccmipirx not found, using dummy regulator
[    9.168500] lt8912 1-0048: supply vccsysclk not found, using dummy regulator
[    9.185702] lt8912 1-0048: supply vcclvdstx not found, using dummy regulator
[    9.202141] lt8912 1-0048: supply vcchdmitx not found, using dummy regulator
[    9.218109] lt8912 1-0048: supply vcclvdspll not found, using dummy regulator
[    9.234095] lt8912 1-0048: supply vcchdmipll not found, using dummy regulator
[    9.273170] k3_r5_rproc bus@f0000:bus@b00000:r5fss@78000000: reserved memory init failed, ret = -22
[    9.291153] [drm] Initialized tidss 1.0.0 20180215 for 30200000.dss on minor 0
[    9.306164] tidss 30200000.dss: [drm] Cannot find any crtc or sizes
[    9.326420] tidss 30200000.dss: [drm] Cannot find any crtc or sizes
[    9.363223] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    9.408365] remoteproc remoteproc0: releasing 78000000.r5f
[    9.437554] cfg80211: Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
[    9.456495] k3_r5_rproc bus@f0000:bus@b00000:r5fss@78000000: k3_r5_cluster_rproc_init failed, ret = -22
[    9.582017] k3_r5_rproc: probe of bus@f0000:bus@b00000:r5fss@78000000 failed with error -22
[   10.061498] mwifiex_sdio mmc2:0001:1: info: FW download over, size 570976 bytes
[*     ] A start job is running for Wait for…ice Initialization (2s / 2min 57s)
[   11.477920] mwifiex_sdio mmc2:0001:1: WLAN FW is active
[   11.531305] mwifiex_sdio mmc2:0001:1: Unknown api_id: 5
[   11.575400] mwifiex_sdio mmc2:0001:1: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84) 
[   11.589950] mwifiex_sdio mmc2:0001:1: driver_version = mwifiex 1.0 (16.92.21.p84) 
M
[**    ] A start job is running for Wait for…ice Initialization (2s / 2min 57s)
M
[***   ] A start job is running for Wait for…ice Initialization (3s / 2min 57s)
M
[ ***  ] A start job is running for Wait for…ice Initialization (3s / 2min 57s)
M
[  *** ] A start job is running for Wait for…ice Initialization (4s / 2min 57s)
M
[   ***] A start job is running for Wait for…ice Initialization (4s / 2min 57s)
[   86.779029] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[   86.785173] rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-0): P186/1:b..l
[   86.792151] 	(detected by 0, t=18276 jiffies, g=6889, q=115 ncpus=1)
[   86.798506] task:systemd-udevd   state:R  running task     stack:0     pid:186   ppid:173    flags:0x00000a04
[   86.808424] Call trace:
[   86.810870]  __switch_to+0xd4/0x130
[   86.814374]  __schedule+0x230/0x664
[   86.817863]  preempt_schedule_irq+0x44/0xd0
[   86.822047]  el1_interrupt+0x4c/0x70
[   86.825626]  el1h_64_irq_handler+0x18/0x2c
[   86.829725]  el1h_64_irq+0x64/0x68
[   86.833130]  set_root+0x64/0xfc
[   86.836275]  nd_jump_root+0xb8/0xf0
[   86.839764]  path_init+0x2d0/0x380
[   86.843167]  path_parentat+0x20/0xb0
[   86.846743]  __filename_parentat+0xa4/0x1c0
[   86.850927]  do_unlinkat+0x68/0x270
[   86.854416]  __arm64_sys_unlinkat+0x40/0x84
[   86.858600]  invoke_syscall+0x48/0x114
[   86.862352]  el0_svc_common.constprop.0+0xd4/0xfc
[   86.867057]  do_el0_svc+0x20/0x30
[   86.870372]  el0_svc+0x28/0xa0
[   86.873428]  el0t_64_sync_handler+0xbc/0x140
[   86.877701]  el0t_64_sync+0x18c/0x190
[   86.881364] rcu: rcu_preempt kthread timer wakeup didn't happen for 18273 jiffies! g6889 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[   86.892749] rcu: 	Possible timer handling issue on cpu=0 timer-softirq=2071
[   86.899705] rcu: rcu_preempt kthread starved for 18274 jiffies! g6889 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
[   86.910136] rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[   86.919260] rcu: RCU grace-period kthread stack dump:
[   86.924305] task:rcu_preempt     state:I stack:0     pid:14    ppid:2      flags:0x00000008
[   86.932656] Call trace:
[   86.935099]  __switch_to+0xd4/0x130
[   86.938587]  __schedule+0x230/0x664
[   86.942074]  schedule+0x5c/0xcc
[   86.945216]  schedule_timeout+0x84/0xfc
[   86.949055]  rcu_gp_fqs_loop+0x12c/0x46c
[   86.952983]  rcu_gp_kthread+0x128/0x164
[   86.956820]  kthread+0x10c/0x110
[   86.960051]  ret_from_fork+0x10/0x20
[   86.963627] rcu: Stack dump where RCU GP kthread last ran:
[   86.969111] CPU: 0 PID: 181 Comm: systemd-udevd Not tainted 6.1.80+git.0d9f1748ac17 #1
[   86.977026] Hardware name: Toradex Verdin AM62 WB on Verdin Development Board (DT)
[   86.984591] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   86.991550] pc : iomap_read_reg+0x10/0x30 [m_can_platform]
[   86.997049] lr : m_can_get_berr_counter+0x3c/0x10c [m_can]
[   87.002548] sp : ffff8000099735f0
[   87.005858] x29: ffff8000099735f0 x28: 0000000000000240 x27: ffff800000cca1a0
[   87.012998] x26: ffff000003f9e000 x25: ffff000003f9e240 x24: 0000000000000000
[   87.020137] x23: 0000000000000000 x22: ffff0000002ae410 x21: ffff0000035f8000
[   87.027276] x20: ffff0000035f8980 x19: ffff80000997362c x18: 0000000000000000
[   87.034414] x17: ffff800014add000 x16: ffff800008000000 x15: 0000000000000000
[   87.041552] x14: 000000000000022f x13: 0000000000000001 x12: 0000000000000001
[   87.048691] x11: 0000000000000000 x10: 00000000000009b0 x9 : ffff8000099731e0
[   87.055829] x8 : ffff00001da1b180 x7 : 0000010000108100 x6 : ffff0000002ae590
[   87.062968] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[   87.070106] x2 : ffff800000ca4000 x1 : 0000000000000040 x0 : 000000000071ff00
[   87.077244] Call trace:
[   87.079687]  iomap_read_reg+0x10/0x30 [m_can_platform]
[   87.084832]  can_fill_info+0x108/0x524 [can_dev]
[   87.089477]  rtnl_fill_ifinfo+0x844/0x11b0
[   87.093578]  rtnl_getlink+0x23c/0x424
[   87.097239]  rtnetlink_rcv_msg+0x130/0x3a0
[   87.101334]  netlink_rcv_skb+0x60/0x130
[   87.105174]  rtnetlink_rcv+0x18/0x2c
[   87.108755]  netlink_unicast+0x2f0/0x360
[   87.112677]  netlink_sendmsg+0x1b0/0x420
[   87.116600]  __sys_sendto+0x138/0x174
[   87.120265]  __arm64_sys_sendto+0x28/0x3c
[   87.124275]  invoke_syscall+0x48/0x114
[   87.128024]  el0_svc_common.constprop.0+0xd4/0xfc
[   87.132729]  do_el0_svc+0x20/0x30
[   87.136044]  el0_svc+0x28/0xa0
[   87.139100]  el0t_64_sync_handler+0xbc/0x140
[   87.143372]  el0t_64_sync+0x18c/0x190
M
[    **] A start job is running for Wait for…itialization (1min 18s / 2min 57s)
[   87.171051] platform sound: deferred probe pending
M
[  OK  ] Listening on Load/Save RF …itch Status /dev/rfkill Watch.
         Starting Load/Save RF Kill Switch Status...
[  OK  ] Started Load/Save RF Kill Switch Status.
[  OK  ] Created slice Slice /system/systemd-fsck.
[  OK  ] Found device /dev/mmcblk0p1.
[  OK  ] Finished Wait for udev To Complete Device Initialization.
[   88.788341] audit: type=1334 audit(1713384460.956:4): prog-id=7 op=LOAD
[   88.799379] audit: type=1334 audit(1713384460.968:5): prog-id=8 op=LOAD
[  OK  ] Started Hardware RNG Entropy Gatherer Daemon.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Logs some system s…istics to the systemd journal.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timer Units.
[  OK  ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Listening on dropbear.socket.
[  OK  ] Listening on GPS (Global P…ioning System) Daemon Sockets.
         Starting Weston socket...
         Starting File System Check on /dev/mmcblk0p1...
[  OK  ] Listening on Weston socket.
[  OK  ] Reached target Socket Units.
[  OK  ] Reached target Basic System.
         Starting Bluetooth service...
[  OK  ] Started Kernel Logging Service.
[  OK  ] Started System Logging Service.
         Starting D-Bus System Message Bus...
         Starting show continuously some info about the som...
         Starting IPv6 Packet Filtering Framework...
         Starting IPv4 Packet Filtering Framework...
         Starting Telephony service...
[   89.645375] audit: type=1334 audit(1713384461.812:6): prog-id=9 op=LOAD
[   89.682739] audit: type=1334 audit(1713384461.824:7): prog-id=10 op=LOAD
         Starting User Login Management...
[  OK  ] Started start the tests in climatic chamber.
[  OK  ] Started Timestamping service.
         Starting Load default USB gadget schema g1.schema...
[  OK  ] Started D-Bus System Message Bus.
[   90.069121] using random self ethernet address
[   90.101441] using random host ethernet address
[  OK  ] Finished File System Check on /dev/mmcblk0p1.
[  OK  ] Finished IPv6 Packet Filtering Framework.
[  OK  ] Finished IPv4 Packet Filtering Framework.
[  OK  ] Reached target Preparation for Network.

***** UART ERROR ****
UART Error Code: -1073807253
VISA Read in ModuleProcess.vi:1380011->ClimateChamberControl.viVISA Source: COM16

**********************

2. Boot stall without any message.

Log is attached, however from what I can tell there is no hint of what's going on

24-04-17_18-29-29-Fail.txt
Climate Chamber:       KWP 120
Start Temperature:     -34.6�C
End Temperature:       -35.5�C
Measured Temperature:  -34.6�C
Shelf Temperature:     -32.7�C
Shelf Name:            Shelf_33
Module Serial Number:  15313618
Module Location:       Shelf 1, Carrier Board 5
Test duration:         158.31s

Test Log
********
18:29:29.12 13->14 Waited 60.00s
18:28:28.90 12->13 Set Reset Low
18:28:28.79 11->12 Power Off
18:28:28.68 5->11 Not found, time out: "login:"
18:26:58.48 4->5 Message found: "Starting kernel"
18:26:52.46 3->4 Message found: "U-Boot"
18:26:52.45 2->3 Waited 1.00s
18:26:51.32 1->2 Set Reset High
18:26:51.24 0->1 Power On

Search Strings
**************
PASS : U-Boot
PASS : Less than 64MB RAM detected
PASS : Booting from mmc failed!
PASS : Starting kernel
PASS : mmcblk0: mmc0
PASS : cut here

PASS : rcu: INFO: rcu_preempt detected
PASS : rcu: INFO: rcu_preempt self-detected stall
PASS : Internal error
PASS : I/O error
FAIL : login:
FAIL : I2C test, TestResult: PASS
FAIL : RTC test, TestResult: PASS
FAIL : Audio test, TestResult: PASS
FAIL : Wifi test, TestResult: PASS
FAIL : Bluetooth test, TestResult: PASS
FAIL : CPU stress test, TestResult: PASS
PASS : critical temperature reached
FAIL : Ethernet test, TestResult: PASS
FAIL : connected with
FAIL : RAM test, TestResult: PASS
PASS : memtester failed
FAIL : memtester version
PASS : FAILURE
PASS : running CQE recovery
FAIL : eMMC Nand test, TestResult: PASS
PASS : bonnie\+\+ failed
FAIL : Waiting for power-cycle
0 : OAK ERROR
1 : UART ERROR
OK : registered as rtc0
OK : registered as rtc1
OK : xception stack
N/A : Device life time estimation type B 
N/A : Pre EOL information
2 : USB hub found
0 : xception stack
PASS : page allocation failure
PASS : hwclock

Terminal
********
�
U-Boot SPL 2024.04-00035-gfc942d8282-dirty (Apr 16 2024 - 11:52:32 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)')
SPL initial stack usage: 13368 bytes
Trying to boot from MMC1
Authentication passed
Authentication passed
Authentication passed
Authentication passed
Authentication passed
Starting ATF on ARM64 core...

NOTICE:  BL31: v2.10.0	(release):v2.10.0-340-g4da4a1a61
NOTICE:  BL31: Built : 11:30:01, Feb  8 2024
I/TC: 
I/TC: OP-TEE version: 4.1.0-71-g439c5ecbb (gcc version 12.3.1 20230626 (Arm GNU Toolchain 12.3.Rel1 (Build arm-12.35))) #5 Fri Feb 23 13:00:52 UTC 2024 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: GIC redistributor base address not provided
I/TC: Assuming default GIC group status and modifier
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)')
I/TC: HUK Initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2024.04-00035-gfc942d8282-dirty (Apr 16 2024 - 11:47:25 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)')
SPL initial stack usage: 1840 bytes
Trying to boot from MMC1
Authentication passed
Authentication passed


U-Boot 2024.04-00035-gfc942d8282-dirty (Apr 16 2024 - 11:47:25 +0200)

SoC:   AM62X SR1.0 HS-FS
DRAM:  512 MiB
Core:  138 devices, 29 uclasses, devicetree: separate
MMC:   mmc@fa10000: 0, mmc@fa00000: 1
Loading Environment from MMC... OK
MISSING TORADEX CARRIER CONFIG BLOCKS
In:    serial@2800000
Out:   serial@2800000
Err:   serial@2800000
Model: Toradex 0072 Verdin AM62 Solo 512MB WB IT V1.1B
Serial#: 15313618
am65_cpsw_nuss ethernet@8000000: K3 CPSW: nuss_ver: 0x6BA01103 cpsw_ver: 0x6BA81103 ale_ver: 0x00290105 Ports:2 mdio_freq:1000000
Setting variant to wifi
Net:   
Warning: ethernet@8000000port@1 MAC addresses don't match:
Address in ROM is		64:1c:10:2a:76:c6
Address in environment is	00:14:2d:e9:aa:d2
eth0: ethernet@8000000port@1 [PRIME]Could not get PHY for ethernet@8000000port@1: addr 7
am65_cpsw_nuss_port ethernet@8000000port@2: phy_connect() failed

Hit any key to stop autoboot:  1  0 
MMC: no card present
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
6003 bytes read in 15 ms (390.6 KiB/s)
## Executing script at 90280000
Loading DeviceTree: k3-am625-verdin-wifi-dev.dtb
70049 bytes read in 16 ms (4.2 MiB/s)
82 bytes read in 15 ms (4.9 KiB/s)
Working FDT set to 90200000
Applying Overlay: verdin-am62_dsi-to-hdmi_overlay.dtbo
3050 bytes read in 15 ms (198.2 KiB/s)
Applying Overlay: verdin-am62_spidev_overlay.dtbo
560 bytes read in 16 ms (34.2 KiB/s)
7861931 bytes read in 76 ms (98.7 MiB/s)
Bootargs: root=PARTUUID=f360aa55-02 ro rootwait console=tty1 console=ttyS2,115200 consoleblank=0 earlycon=ns16550a,mmio32,0x02800000
   Uncompressing Kernel Image to 0
## Flattened Device Tree blob at 90200000
   Booting using the fdt blob at 0x90200000
Working FDT set to 90200000
ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4)
ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4)
   Loading Device Tree to 0000000098ed5000, end 0000000098f09fff ... OK
Working FDT set to 98ed5000

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 6.1.80+git.0d9f1748ac17 (oe-user@oe-host) (aarch64-tdx-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Thu Mar 28 13:48:11 UTC 2024
[    0.000000] Machine model: Toradex Verdin AM62 WB on Verdin Development Board
[    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009db00000, size 12 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@9db00000, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x000000009fffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x000000009dafffff]
[    0.000000]   node   0: [mem 0x000000009db00000-0x000000009e6fffff]
[    0.000000]   node   0: [mem 0x000000009e700000-0x000000009e77ffff]
[    0.000000]   node   0: [mem 0x000000009e780000-0x000000009fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000009fffffff]
[    0.000000] cma: Reserved 128 MiB at 0x0000000090e00000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.4
[    0.000000] percpu: Embedded 20 pages/cpu s41064 r8192 d32664 u81920
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: ARM erratum 845719
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 129024
[    0.000000] Kernel command line: root=PARTUUID=f360aa55-02 ro rootwait console=tty1 console=ttyS2,115200 consoleblank=0 earlycon=ns16550a,mmio32,0x02800000
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 326452K/524288K available (11072K kernel code, 1354K rwdata, 3784K rodata, 2176K init, 422K bss, 66764K reserved, 131072K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=1.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001880000
[    0.000000] ITS [mem 0x01820000-0x0182ffff]
[    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
[    0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
[    0.000000] ITS@0x0000000001820000: allocated 524288 Devices @80800000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x0000000080030000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000080040000
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0x3ffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
[    0.000001] sched_clock: 58 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
[    0.008742] Console: colour dummy device 80x25
[    0.014002] printk: console [tty1] enabled
[    0.018262] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000)
[    0.028973] pid_max: default: 32768 minimum: 301
[    0.033774] LSM: Security Framework initializing
[    0.038662] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.046171] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.056437] cblist_init_generic: Setting adjustable number of callback queues.
[    0.063936] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.070279] cblist_init_generic: Setting adjustable number of callback queues.
[    0.077702] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.084181] rcu: Hierarchical SRCU implementation.
[    0.089112] rcu: 	Max phase no-delay instances is 1000.
[    0.095319] Platform MSI: msi-controller@1820000 domain created
[    0.102059] EFI services will not be available.
[    0.106992] smp: Bringing up secondary CPUs ...
[    0.111659] smp: Brought up 1 node, 1 CPU
[    0.115776] SMP: Total of 1 processors activated.
[    0.120607] CPU features: detected: 32-bit EL0 Support
[    0.125884] CPU features: detected: CRC32 instructions
[    0.131240] CPU: All CPU(s) started at EL2
[    0.135444] alternatives: applying system-wide alternatives
[    0.142818] devtmpfs: initialized
[    0.158662] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.168738] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[    0.182772] pinctrl core: initialized pinctrl subsystem
[    0.189190] DMI not present or invalid.
[    0.194079] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.201486] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.208868] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.216913] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.225140] audit: initializing netlink subsys (disabled)
[    0.231468] thermal_sys: Registered thermal governor 'step_wise'
[    0.231482] thermal_sys: Registered thermal governor 'power_allocator'
[    0.237675] audit: type=2000 audit(0.148:1): state=initialized audit_enabled=0 res=1
[    0.252368] cpuidle: using governor menu
[    0.256651] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.263676] ASID allocator initialised with 65536 entries
[    0.290297] KASLR disabled due to lack of seed
[    0.303853] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.310985] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page
[    0.317422] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages
[    0.324381] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page
[    0.330807] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.337764] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
[    0.344190] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages
[    0.351148] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
[    0.359513] k3-chipinfo 43000014.chipid: Family:AM62X rev:SR1.0 JTAGID[0x0bb7e02f] Detected
[    0.369953] iommu: Default domain type: Translated 
[    0.375021] iommu: DMA domain TLB invalidation policy: strict mode 
[    0.381822] SCSI subsystem initialized
[    0.386153] usbcore: registered new interface driver usbfs
[    0.391850] usbcore: registered new interface driver hub
[    0.397333] usbcore: registered new device driver usb
[    0.403196] pps_core: LinuxPPS API ver. 1 registered
[    0.408298] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.417674] PTP clock support registered
[    0.421897] EDAC MC: Ver: 3.0.0
[    0.426561] FPGA manager framework
[    0.430206] Advanced Linux Sound Architecture Driver Initialized.
[    0.437794] clocksource: Switched to clocksource arch_sys_counter
[    0.444485] VFS: Disk quotas dquot_6.6.0
[    0.448574] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.463560] NET: Registered PF_INET protocol family
[    0.469066] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.477771] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.486457] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.494413] TCP established hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.502401] TCP bind hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.509996] TCP: Hash tables configured (established 4096 bind 4096)
[    0.516653] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.523375] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.530710] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.537085] RPC: Registered named UNIX socket transport module.
[    0.543216] RPC: Registered udp transport module.
[    0.548045] RPC: Registered tcp transport module.
[    0.552870] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.559480] NET: Registered PF_XDP protocol family
[    0.565319] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
[    0.576123] Initialise system trusted keyrings
[    0.581093] workingset: timestamp_bits=46 max_order=17 bucket_order=0
[    0.594170] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.601074] NFS: Registering the id_resolver key type
[    0.606369] Key type id_resolver registered
[    0.610665] Key type id_legacy registered
[    0.614859] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.621737] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.682736] Key type asymmetric registered
[    0.686951] Asymmetric key parser 'x509' registered
[    0.692072] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    0.699876] io scheduler mq-deadline registered
[    0.704549] io scheduler kyber registered
[    0.714092] pinctrl-single 4084000.pinctrl: 34 pins, size 136
[    0.721439] pinctrl-single f4000.pinctrl: 171 pins, size 684
[    0.737297] Serial: 8250/16550 driver, 6 ports, IRQ sharing enabled
[    0.757528] brd: module loaded
[    0.766705] loop: module loaded
[    0.775454] tun: Universal TUN/TAP device driver, 1.6
[    0.781863] VFIO - User Level meta-driver version: 0.3
[    0.788491] usbcore: registered new interface driver usb-storage
[    0.795622] i2c_dev: i2c /dev entries driver
[    0.802829] sdhci: Secure Digital Host Controller Interface driver
[    0.809215] sdhci: Copyright(c) Pierre Ossman
[    0.814135] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.820781] ledtrig-cpu: registered to indicate activity on CPUs
[    0.827195] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[    0.834338] usbcore: registered new interface driver usbhid
[    0.840065] usbhid: USB HID core driver
[    0.845479] optee: probing for conduit method.
I/TC: Reserved shared memory is enabled
I/TC: Dynamic shared memory is enabled
I/TC: Normal World virtualization support is disabled
I/TC: Asynchronous notifications are disabled
[    0.850116] optee: revision 4.1 (439c5ecb)
[    0.866813] optee: dynamic shared memory is enabled
[    0.876378] optee: initialized driver
[    0.882761] Initializing XFRM netlink socket
[    0.887336] NET: Registered PF_PACKET protocol family
[    0.892634] Key type dns_resolver registered
[    0.897563] registered taskstats version 1
[    0.901816] Loading compiled-in X.509 certificates
[    0.921082] ti-sci 44043000.system-controller: ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)')
[    1.013314] omap_i2c 4900000.i2c: bus 3 rev0.12 at 100 kHz
[    1.021160] i2c 0-0030: Fixed dependency cycle(s) with /bus@f0000/i2c@20000000/pmic@30/regulators/buck2
[    1.064086] input: tps65219-pwrbutton as /devices/platform/bus@f0000/20000000.i2c/i2c-0/0-0030/tps65219-pwrbutton.2.auto/input/input0
[    1.078857] rtc-ds1307 0-0032: oscillator failed, set time!
[    1.084870] rtc-ds1307 0-0032: registered as rtc0
[    1.089875] rtc-ds1307 0-0032: hctosys: unable to read the hardware clock
[    1.099594] at24 0-0050: supply vcc not found, using dummy regulator
[    1.106699] at24 0-0050: 256 byte 24c02 EEPROM, writable, 16 bytes/write
[    1.113665] omap_i2c 20000000.i2c: bus 0 rev0.12 at 400 kHz
[    1.122388] pca953x 1-0021: supply vcc not found, using dummy regulator
[    1.129429] pca953x 1-0021: using no AI
[    1.153942] pca953x 1-0021: failed writing register
[    1.159185] pca953x: probe of 1-0021 failed with error -121
[    1.165720] at24 1-0057: supply vcc not found, using dummy regulator
[    1.173052] at24 1-0057: 256 byte 24c02 EEPROM, writable, 16 bytes/write
[    1.180025] omap_i2c 20010000.i2c: bus 1 rev0.12 at 100 kHz
[    1.187181] omap_i2c 20020000.i2c: bus 2 rev0.12 at 100 kHz
[    1.194111] omap_i2c 20030000.i2c: bus 4 rev0.12 at 100 kHz
[    1.200205] ti-sci-intr 4210000.interrupt-controller: Interrupt Router 5 domain created
[    1.208636] ti-sci-intr bus@f0000:interrupt-controller@a00000: Interrupt Router 3 domain created
[    1.218007] ti-sci-inta 48000000.interrupt-controller: Interrupt Aggregator domain 28 created
[    1.231963] ti-udma 485c0100.dma-controller: Number of rings: 82
[    1.241104] ti-udma 485c0100.dma-controller: Channels: 48 (bchan: 18, tchan: 12, rchan: 18)
[    1.253256] ti-udma 485c0000.dma-controller: Number of rings: 150
[    1.264900] ti-udma 485c0000.dma-controller: Channels: 35 (tchan: 20, rchan: 15)
[    1.276261] 4a00000.serial: ttyS3 at MMIO 0x4a00000 (irq = 291, base_baud = 3000000) is a 8250
[    1.286803] 2800000.serial: ttyS2 at MMIO 0x2800000 (irq = 292, base_baud = 3000000) is a 8250
[    1.295801] printk: console [ttyS2] enabled
[    1.295801] printk: console [ttyS2] enabled
[    1.304305] printk: bootconsole [ns16550a0] disabled
[    1.304305] printk: bootconsole [ns16550a0] disabled
[    1.315813] 2810000.serial: ttyS0 at MMIO 0x2810000 (irq = 293, base_baud = 3000000) is a 8250
[    1.325528] omap8250 2850000.serial: PM domain pd:156 will not be powered off
[    1.333090] 2850000.serial: ttyS4 at MMIO 0x2850000 (irq = 294, base_baud = 3000000) is a 8250
[    1.341943] serial serial0: tty port ttyS4 registered
[    1.348852] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
[    1.393804] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
[    1.402914] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA01103, cpsw version 0x6BA81103 Ports: 3 quirks:00000006
[    1.415992] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.5
[    1.423182] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512
[    1.430115] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:0
[    1.440788] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[    1.458899] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller
[    1.464499] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 1
[    1.472331] xhci-hcd xhci-hcd.3.auto: USB3 root hub has no ports
[    1.478355] xhci-hcd xhci-hcd.3.auto: hcc params 0x0258fe6d hci version 0x110 quirks 0x0000008020010010
[    1.487847] xhci-hcd xhci-hcd.3.auto: irq 299, io mem 0x31100000
[    1.494975] hub 1-0:1.0: USB hub found
[    1.498843] hub 1-0:1.0: 1 port detected
[    1.639647] mmc0: CQHCI version 5.10
[    1.645286] gpio-488 (PCIE_1_RESET#): hogged as output/low
[    1.654044] gpio-427 (CTRL_SLEEP_MOCI#): hogged as output/high
[    1.671552] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
[    1.690765] mmc0: SDHCI controller on fa10000.mmc [fa10000.mmc] using ADMA 64-bit
[    1.717790] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
[    1.731009] mdio_bus 8000f00.mdio: MDIO device at address 7 is missing.
[    1.737725] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867
[    1.749853] sdhci-am654 fa00000.mmc: Got CD GPIO
[    1.755590] sdhci-am654 fa20000.mmc: allocated mmc-pwrseq
[    1.762660] debugfs: Directory 'pd:182' with parent 'pm_genpd' already present!
[    1.770061] usb 1-1: new high-speed USB device number 2 using xhci-hcd
[    1.777282] mmc1: CQHCI version 5.10
[    1.781229] mmc2: CQHCI version 5.10
[    1.790433] input: gpio-keys as /devices/platform/gpio-keys/input/input1
[    1.801065] mmc0: Command Queue Engine enabled
[    1.806019] mmc0: new HS200 MMC card at address 0001
[    1.812685] mmcblk0: mmc0:0001 M04A11 3.64 GiB 
[    1.821196]  mmcblk0: p1 p2
[    1.825647] mmcblk0boot0: mmc0:0001 M04A11 4.00 MiB 
[    1.830724] mmc2: SDHCI controller on fa20000.mmc [fa20000.mmc] using ADMA 64-bit
[    1.840451] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
[    1.849028] mmcblk0boot1: mmc0:0001 M04A11 4.00 MiB 
[    1.855721] ALSA device list:
[    1.859032]   No soundcards found.
[    1.863725] mmcblk0rpmb: mmc0:0001 M04A11 4.00 MiB, chardev (241:0)
[    1.877754] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Quota mode: none.
[    1.886544] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[    1.894806] devtmpfs: mounted
[    1.899448] Freeing unused kernel memory: 2176K
[    1.904237] Run /sbin/init as init process
[    1.935781] mmc2: new ultra high speed DDR50 SDIO card at address 0001
[    1.974596] hub 1-1:1.0: USB hub found
[    1.979376] hub 1-1:1.0: 4 ports detected
[    2.044626] systemd[1]: System time before build time, advancing clock.
[    2.090508] NET: Registered PF_INET6 protocol family
[    2.097294] Segment Routing with IPv6
[    2.101112] In-situ OAM (IOAM) with IPv6
[    2.129609] systemd[1]: systemd 250.5+ running in system mode (+PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP -GCRYPT -GNUTLS -OPENSSL +ACL +BLKID -CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -BZIP2 -LZ4 -XZ -ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=hybrid)
[    2.161915] systemd[1]: Detected architecture arm64.

Welcome to TDX Wayland with XWayland 6.6.0+build.12 (kirkstone)!

[    2.215565] systemd[1]: Hostname set to <verdin-am62-15313618>.
[    2.341972] usb 1-1.4: new high-speed USB device number 3 using xhci-hcd
[    2.817374] systemd[1]: Queued start job for default target Graphical Interface.
[    2.877711] systemd[1]: Created slice Slice /system/getty.
[  OK  ] Created slice Slice /system/getty.
[    2.900740] systemd[1]: Created slice Slice /system/modprobe.
[  OK  ] Created slice Slice /system/modprobe.
[    2.924602] systemd[1]: Created slice Slice /system/serial-getty.
[  OK  ] Created slice Slice /system/serial-getty.
[    2.948185] systemd[1]: Created slice User and Session Slice.
[  OK  ] Created slice User and Session Slice.
[    2.970517] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[    2.994551] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Started Forward Password R…uests to Wall Directory Watch.
[    3.018620] systemd[1]: Reached target Host and Network Name Lookups.
[  OK  ] Reached target Host and Network Name Lookups.
[    3.042022] systemd[1]: Reached target Path Units.
[  OK  ] Reached target Path Units.
[    3.062290] systemd[1]: Reached target Remote File Systems.
[  OK  ] Reached target Remote File Systems.
[    3.086176] systemd[1]: Reached target Slice Units.
[  OK  ] Reached target Slice Units.
[    3.106204] systemd[1]: Reached target Swaps.
[  OK  ] Reached target Swaps.
[    3.174555] systemd[1]: Listening on RPCbind Server Activation Socket.
[  OK  ] Listening on RPCbind Server Activation Socket.
[    3.198487] systemd[1]: Reached target RPC Port Mapper.
[  OK  ] Reached target RPC Port Mapper.
[    3.223769] systemd[1]: Listening on Syslog Socket.
[  OK  ] Listening on Syslog Socket.
[    3.246940] systemd[1]: Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[    3.272058] systemd[1]: Listening on Journal Audit Socket.
[  OK  ] Listening on Journal Audit Socket.
[    3.295278] systemd[1]: Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket (/dev/log).
[    3.319902] systemd[1]: Listening on Journal Socket.
[  OK  ] Listening on Journal Socket.
[    3.346342] systemd[1]: Listening on Network Service Netlink Socket.
[  OK  ] Listening on Network Service Netlink Socket.
[    3.373173] systemd[1]: Listening on udev Control Socket.
[  OK  ] Listening on udev Control Socket.
[    3.396158] systemd[1]: Listening on udev Kernel Socket.
[  OK  ] Listening on udev Kernel Socket.
[    3.420658] systemd[1]: Listening on User Database Manager Socket.
[  OK  ] Listening on User Database Manager Socket.
[    3.470927] systemd[1]: Mounting Huge Pages File System...
         Mounting Huge Pages File System...
[    3.515082] systemd[1]: Mounting POSIX Message Queue File System...
         Mounting POSIX Message Queue File System...
[    3.562672] systemd[1]: Mounting Kernel Debug File System...
         Mounting Kernel Debug File System...
[    3.581340] systemd[1]: Kernel Trace File System was skipped because of a failed condition check (ConditionPathExists=/sys/kernel/tracing).
[    3.614544] systemd[1]: Mounting Temporary Directory /tmp...
         Mounting Temporary Directory /tmp...
[    3.662799] systemd[1]: Starting Create List of Static Device Nodes...
         Starting Create List of Static Device Nodes...
[    3.714853] systemd[1]: Starting Load Kernel Module configfs...
         Starting Load Kernel Module configfs...
[    3.766948] systemd[1]: Starting Load Kernel Module drm...
         Starting Load Kernel Module drm...
[    3.814988] systemd[1]: Starting Load Kernel Module fuse...
         Starting Load Kernel Module fuse...
[    3.837090] fuse: init (API version 7.37)
[    3.874358] systemd[1]: Starting RPC Bind...
         Starting RPC Bind...
[    3.918981] systemd[1]: Starting File System Check on Root Device...
         Starting File System Check on Root Device...
[    3.971035] systemd[1]: Starting Journal Service...
         Starting Journal Service...
[    4.027301] systemd[1]: Starting Load Kernel Modules...
         Starting Load Kernel Modules...
[    4.080708] systemd[1]: Starting Generate network units from Kernel command line...
         Starting Generate network …ts from Kernel command line...
[    4.147233] systemd[1]: Starting Coldplug All udev Devices...
         Starting Coldplug All udev Devices...
[    4.217049] systemd[1]: Started RPC Bind.
[  OK  ] Started RPC Bind.
[    4.238553] systemd[1]: Mounted Huge Pages File System.
[  OK  ] Mounted Huge Pages File System.
[    4.279047] systemd[1]: Mounted POSIX Message Queue File System.
[  OK  ] Mounted POSIX Message Queue File System.
[    4.315020] systemd[1]: Mounted Kernel Debug File System.
[  OK  ] Mounted Kernel Debug File System.
[    4.346964] systemd[1]: Mounted Temporary Directory /tmp.
[  OK  ] Mounted Temporary Directory /tmp.
[    4.380497] systemd[1]: Finished Create List of Static Device Nodes.
[  OK  ] Finished Create List of Static Device Nodes.
[    4.419684] systemd[1]: modprobe@configfs.service: Deactivated successfully.
[    4.446457] systemd[1]: Finished Load Kernel Module configfs.
[  OK  ] Finished Load Kernel Module configfs.
[    4.487009] systemd[1]: Started Journal Service.
[  OK  ] Started Journal Service.
[  OK  ] Finished Load Kernel Module drm.
[  OK  ] Finished Load Kernel Module fuse.
[  OK  ] Finished File System Check on Root Device.
[  OK  ] Finished Load Kernel Modules.
[  OK  ] Finished Generate network units from Kernel command line.
         Mounting FUSE Control File System...
         Mounting Kernel Configuration File System...
         Starting Remount Root and Kernel File Systems...
         Starting Apply Kernel Variables...
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Mounted Kernel Configuration File System.
[  OK  ] Finished Apply Kernel Variables.
[    4.952190] EXT4-fs (mmcblk0p2): re-mounted. Quota mode: none.
[  OK  ] Finished Remount Root and Kernel File Systems.
         Starting Flush Journal to Persistent Storage...
         Starting Create Static Device Nodes in /dev...
[    5.094565] systemd-journald[155]: Received client request to flush runtime journal.
[  OK  ] Finished Flush Journal to Persistent Storage.
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Preparation for Local File Systems.
         Mounting /var/volatile...
[    5.275993] audit: type=1334 audit(1651167748.228:2): prog-id=5 op=LOAD
[    5.298902] audit: type=1334 audit(1651167748.236:3): prog-id=6 op=LOAD
         Starting Rule-based Manage…for Device Events and Files...
[  OK  ] Mounted /var/volatile.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Create Volatile Files and Directories.
         Starting Network Time Synchronization...
         Starting Record System Boot/Shutdown in UTMP...
[  OK  ] Started Rule-based Manager for Device Events and Files.
[  OK  ] Finished Record System Boot/Shutdown in UTMP.
[  OK  ] Finished Coldplug All udev Devices.
         Starting Wait for udev To …plete Device Initialization...
[    6.389871] random: crng init done
[  OK  ] Finished Load/Save Random Seed.
[  OK  ] Started Network Time Synchronizatio[    6.932942] systemd-journald[155]: Oldest entry in /run/log/journal/98d6394cd75d4f2dad68076443517daa/system.journal is older than the configured file retention duration (1month), suggesting rotation.
n.
[  OK  ] Reached target System Time Set.
[    7.006135] systemd-journald[155]: /run/log/journal/98d6394cd75d4f2dad68076443517daa/system.journal: Journal header limits reached or header out-of-date, rotating.
[    7.504857] omap8250 2b300000.serial: No clock speed specified: using default: 48000000
[    7.513100] lm75 0-0048: supply vs not found, using dummy regulator
[    7.524666] 2b300000.serial: ttyS1 at MMIO 0x2b300000 (irq = 485, base_baud = 3000000) is a 8250
[    7.598553] hwmon hwmon1: temp1_input not attached to any thermal zone
[    7.634646] ina2xx 1-0040: error configuring the device: -121
[    7.671330] lm75 0-0048: hwmon1: sensor 'tmp1075'
[    7.707383] nau8822 1-001a: Failed to issue reset: -121
[    7.724486] lm75 1-004f: supply vs not found, using dummy regulator
[    7.771641] nau8822: probe of 1-001a failed with error -121
[    7.789487] CAN device driver interface
[    7.830726] hwmon hwmon2: temp1_input not attached to any thermal zone
[    7.879127] lm75 1-004f: hwmon2: sensor 'tmp75c'
[    7.903803] m_can_platform 4e08000.can: m_can device registered (irq=0, version=32)
[  OK  ] Found device /dev/ttyS2.
[    8.018690] m_can_platform 20701000.can: m_can device registered (irq=489, version=32)
[    8.357560] Bluetooth: Core ver 2.22
[    8.361443] k3-m4-rproc 5000000.m4fss: device does not reserved memory regions, ret = -22
[    8.398193] NET: Registered PF_BLUETOOTH protocol family
[    8.442828] k3-m4-rproc 5000000.m4fss: reserved memory init failed, ret = -22
[    8.469919] Bluetooth: HCI device and connection manager initialized
[    8.530610] remoteproc remoteproc0: releasing 5000000.m4fss
[    8.553175] Bluetooth: HCI socket layer initialized
[    8.589020] k3-m4-rproc: probe of 5000000.m4fss failed with error -22
[    8.604202] Bluetooth: L2CAP socket layer initialized
[    8.680960] Bluetooth: SCO socket layer initialized
[    8.782120] rtc-ti-k3 2b1f0000.rtc: registered as rtc1
[    8.881623] platform 78000000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1)
[    9.037641] platform 78000000.r5f: configured R5F for IPC-only mode
[    9.100848] platform 78000000.r5f: device does not have reserved memory regions, ret = -22
[    9.135323] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    9.144308] lt8912 1-0048: supply vdd not found, using dummy regulator
[    9.158229] lt8912 1-0048: supply vccmipirx not found, using dummy regulator
[    9.174207] lt8912 1-0048: supply vccsysclk not found, using dummy regulator
[    9.191784] lt8912 1-0048: supply vcclvdstx not found, using dummy regulator
[    9.206175] lt8912 1-0048: supply vcchdmitx not found, using dummy regulator
[    9.227389] lt8912 1-0048: supply vcclvdspll not found, using dummy regulator
[    9.241153] lt8912 1-0048: supply vcchdmipll not found, using dummy regulator
[    9.272838] k3_r5_rproc bus@f0000:bus@b00000:r5fss@78000000: reserved memory init failed, ret = -22
[    9.299259] [drm] Initialized tidss 1.0.0 20180215 for 30200000.dss on minor 0
[    9.313316] tidss 30200000.dss: [drm] Cannot find any crtc or sizes
[    9.334329] tidss 30200000.dss: [drm] Cannot find any crtc or sizes
[    9.375274] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    9.409068] remoteproc remoteproc0: releasing 78000000.r5f
[    9.418689] cfg80211: Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
[    9.486254] k3_r5_rproc bus@f0000:bus@b00000:r5fss@78000000: k3_r5_cluster_rproc_init failed, ret = -22
[    9.589986] k3_r5_rproc: probe of bus@f0000:bus@b00000:r5fss@78000000 failed with error -22
[   10.043966] mwifiex_sdio mmc2:0001:1: info: FW download over, size 570976 bytes
[*     ] A start job is running for Wait for…ice Initialization (2s / 2min 57s)
[   11.465914] mwifiex_sdio mmc2:0001:1: WLAN FW is active
[   11.510599] mwifiex_sdio mmc2:0001:1: Unknown api_id: 5
M
[**    ] A start job is running for Wait for…ice Initialization (2s / 2min 57s)
[   11.564064] mwifiex_sdio mmc2:0001:1: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84) 
[   11.582133] mwifiex_sdio mmc2:0001:1: driver_version = mwifiex 1.0 (16.92.21.p84) 
M
[***   ] A start job is running for Wait for…ice Initialization (3s / 2min 57s)
M
[ ***  ] A start job is running for Wait for…ice Initialization (3s / 2min 57s)
M
[  *** ] A start job is running for Wait for…ice Initialization (4s / 2min 57s)
M
[   ***] A start job is running for Wait for…ice Initialization (4s / 2min 57s)

***** UART ERROR ****
UART Error Code: -1073807253
VISA Read in ModuleProcess.vi:1380011->ClimateChamberControl.viVISA Source: COM16

**********************

  • Hello 

    Similar to previous requests a few months ago, we will need your help to peel the onion a bit more and see if you can show this with a simpler test and localized to specific modules (CAN?) or others being what is sensitive to the failures etc.

    1) I want to double confirm that you are using the latest SDK and specifically SYSFW that has the fixes listed 2.1. Release Notes — Processor SDK AM62x Documentation (SYSFW-7056) 

    2) You have eliminated the issues you previously had due to DDR and eMMC failures / instability 

    3) What have you done to narrow the issue down further? Does every board, every variant fail. Any study done to see if the failure follows the device or the board or a mix of both? ABA swaps?

    4) What is the % failing vs passing boards

    5) Have you reconfirmed that OPPs are correct and you are not running 1.4GHz etc at 0.75V 

    6) What is the correlation of your temp chamber temperature to what on die sensors report as the die temperature

    7) Does the failure happen at negative temperature only - everything fine at high temp? 

    8) Are the failures sensitive to a given temperature or happen when you initialize at a different temperature but do the stress test or temp chamber test at a different temperature?

  • 1) I want to double confirm that you are using the latest SDK and specifically SYSFW that has the fixes listed 2.1. Release Notes — Processor SDK AM62x Documentation (SYSFW-7056) 

    Ok, it seems we made a mistake here. The last round of tests was executed with a manually built image to add patches for the U-Boot eMMC bug (https://lore.kernel.org/u-boot/20240418190100.1289150-1-jm@ti.com/), and an old SYSFW was integrated there by mistake.

    We will re-execute the test and update you in a few days.

  • thanks, that will be very helpful 

  • 1) I want to double confirm that you are using the latest SDK and specifically SYSFW that has the fixes listed 2.1. Release Notes — Processor SDK AM62x Documentation (SYSFW-7056) 

    After fixing a mistake we did re-run all our test with latest SYSFW. We could not see any improvements.

    See updated logs.

    24-04-30_17-01-57-Fail.txt
    Climate Chamber:       KWP 120
    Start Temperature:     -40.3�C
    End Temperature:       -39.8�C
    Measured Temperature:  -40.3�C
    Shelf Temperature:     -39.5�C
    Shelf Name:            Shelf_19
    Module Serial Number:  15313474
    Module Location:       Shelf 1, Carrier Board 5
    Test duration:         127.82s
    
    Test Log
    ********
    17:01:56.91 13->14 Waited 30.00s
    17:01:26.76 12->13 Set Reset Low
    17:01:26.65 11->12 Power Off
    17:01:26.55 5->11 Not found, time out: "login:"
    16:59:56.34 4->5 Message found: "Starting kernel"
    16:59:50.92 3->4 Message found: "U-Boot"
    16:59:50.81 2->3 Waited 1.00s
    16:59:49.64 1->2 Set Reset High
    16:59:49.53 0->1 Power On
    
    Search Strings
    **************
    PASS : U-Boot
    PASS : Less than 64MB RAM detected
    PASS : Booting from mmc failed!
    PASS : Starting kernel
    PASS : mmcblk0: mmc0
    PASS : cut here
    
    PASS : rcu: INFO: rcu_preempt detected
    PASS : rcu: INFO: rcu_preempt self-detected stall
    PASS : Internal error
    PASS : I/O error
    FAIL : login:
    PASS : critical temperature reached
    FAIL : RAM test, TestResult: PASS
    PASS : memtester failed
    FAIL : memtester version
    PASS : FAILURE
    PASS : running CQE recovery
    FAIL : Waiting for power-cycle
    0 : OAK ERROR
    0 : UART ERROR
    N/A : Device life time estimation type B 
    N/A : Pre EOL information
    2 : USB hub found
    0 : xception stack
    PASS : page allocation failure
    PASS : hwclock
    
    Terminal
    ********
    
    U-Boot SPL 2023.04-ti-gca551722e0b5 (Mar 22 2024 - 08:10:43 +0000)
    SYSFW ABI: 3.1 (firmware rev 0x0009 '9.2.7--v09.02.07 (Kool Koala)')
    SPL initial stack usage: 13408 bytes
    Trying to boot from MMC1
    Authentication passed
    Authentication passed
    Authentication passed
    Authentication passed
    Authentication passed
    Starting ATF on ARM64 core...
    
    NOTICE:  BL31: v2.10.0(release):v2.10.0-367-g00f1ec6b87-dirty
    NOTICE:  BL31: Built : 16:09:05, Feb  9 2024
    
    U-Boot SPL 2023.04-ti-gca551722e0b5 (Mar 22 2024 - 08:10:43 +0000)
    SYSFW ABI: 3.1 (firmware rev 0x0009 '9.2.7--v09.02.07 (Kool Koala)')
    SPL initial stack usage: 1856 bytes
    Trying to boot from MMC1
    Authentication passed
    Authentication passed
    
    
    U-Boot 2023.04-ti-gca551722e0b5 (Mar 22 2024 - 08:10:43 +0000)
    
    SoC:   AM62X SR1.0 HS-FS
    DRAM:  1 GiB
    Core:  143 devices, 31 uclasses, devicetree: separate
    MMC:   mmc@fa10000: 0, mmc@fa00000: 1
    Loading Environment from MMC... OK
    In:    serial@2800000
    Out:   serial@2800000
    Err:   serial@2800000
    Model: Toradex 0075 Verdin AM62 Dual 1GB WB IT V1.1B
    Serial#: 15313474
    MISSING TORADEX CARRIER CONFIG BLOCKS
    am65_cpsw_nuss ethernet@8000000: K3 CPSW: nuss_ver: 0x6BA01103 cpsw_ver: 0x6BA81103 ale_ver: 0x00290105 Ports:2 mdio_freq:1000000
    Setting variant to wifi
    Net:   
    Warning: ethernet@8000000port@1 MAC addresses don't match:
    Address in ROM is		1c:63:49:1f:df:7c
    Address in environment is	00:14:2d:e9:aa:42
    eth0: ethernet@8000000port@1 [PRIME]Could not get PHY for ethernet@8000000port@1: addr 7
    am65_cpsw_nuss_port ethernet@8000000port@2: phy_connect() failed
    
    Hit any key to stop autoboot:  1  0 
    MMC: no card present
    switch to partitions #0, OK
    mmc0(part 0) is current device
    Scanning mmc 0:1...
    Found U-Boot script /boot.scr
    6003 bytes read in 15 ms (390.6 KiB/s)
    ## Executing script at 90280000
    Loading DeviceTree: k3-am625-verdin-wifi-dev.dtb
    70049 bytes read in 17 ms (3.9 MiB/s)
    82 bytes read in 14 ms (4.9 KiB/s)
    Working FDT set to 90200000
    Applying Overlay: verdin-am62_dsi-to-hdmi_overlay.dtbo
    3050 bytes read in 15 ms (198.2 KiB/s)
    Applying Overlay: verdin-am62_spidev_overlay.dtbo
    560 bytes read in 15 ms (36.1 KiB/s)
    7861931 bytes read in 71 ms (105.6 MiB/s)
    Bootargs: root=PARTUUID=016de4a6-02 ro rootwait console=tty1 console=ttyS2,115200 consoleblank=0 earlycon=ns16550a,mmio32,0x02800000
       Uncompressing Kernel Image
    ## Flattened Device Tree blob at 90200000
       Booting using the fdt blob at 0x90200000
    Working FDT set to 90200000
    ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4)
    ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
    ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4)
       Loading Device Tree to 0000000098eac000, end 0000000098ee0fff ... OK
    Working FDT set to 98eac000
    
    Starting kernel ...
    
    [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
    [    0.000000] Linux version 6.1.80+git.0d9f1748ac17 (oe-user@oe-host) (aarch64-tdx-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Thu Mar 28 13:48:11 UTC 2024
    [    0.000000] Machine model: Toradex Verdin AM62 WB on Verdin Development Board
    [    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
    [    0.000000] printk: bootconsole [ns16550a0] enabled
    [    0.000000] efi: UEFI not found.
    [    0.000000] Reserved memory: created DMA memory pool at 0x000000009db00000, size 12 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-memory@9db00000, compatible id shared-dma-pool
    [    0.000000] Zone ranges:
    [    0.000000]   DMA      [mem 0x0000000080000000-0x00000000bfffffff]
    [    0.000000]   DMA32    empty
    [    0.000000]   Normal   empty
    [    0.000000] Movable zone start for each node
    [    0.000000] Early memory node ranges
    [    0.000000]   node   0: [mem 0x0000000080000000-0x000000009dafffff]
    [    0.000000]   node   0: [mem 0x000000009db00000-0x000000009e6fffff]
    [    0.000000]   node   0: [mem 0x000000009e700000-0x000000009e77ffff]
    [    0.000000]   node   0: [mem 0x000000009e780000-0x000000009fffffff]
    [    0.000000]   node   0: [mem 0x00000000a0000000-0x00000000bfffffff]
    [    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000bfffffff]
    [    0.000000] cma: Reserved 128 MiB at 0x00000000b6c00000
    [    0.000000] psci: probing for conduit method from DT.
    [    0.000000] psci: PSCIv1.1 detected in firmware.
    [    0.000000] psci: Using standard PSCI v0.2 function IDs
    [    0.000000] psci: Trusted OS migration not required
    [    0.000000] psci: SMC Calling Convention v1.4
    [    0.000000] percpu: Embedded 20 pages/cpu s41064 r8192 d32664 u81920
    [    0.000000] Detected VIPT I-cache on CPU0
    [    0.000000] CPU features: detected: GIC system register CPU interface
    [    0.000000] CPU features: detected: ARM erratum 845719
    [    0.000000] alternatives: applying boot alternatives
    [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258048
    [    0.000000] Kernel command line: root=PARTUUID=016de4a6-02 ro rootwait console=tty1 console=ttyS2,115200 consoleblank=0 earlycon=ns16550a,mmio32,0x02800000
    [    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
    [    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
    [    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
    [    0.000000] Memory: 840660K/1048576K available (11072K kernel code, 1354K rwdata, 3784K rodata, 2176K init, 422K bss, 76844K reserved, 131072K cma-reserved)
    [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
    [    0.000000] rcu: Preemptible hierarchical RCU implementation.
    [    0.000000] rcu: 	RCU event tracing is enabled.
    [    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
    [    0.000000] 	Trampoline variant of Tasks RCU enabled.
    [    0.000000] 	Tracing variant of Tasks RCU enabled.
    [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
    [    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
    [    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
    [    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
    [    0.000000] GICv3: 256 SPIs implemented
    [    0.000000] GICv3: 0 Extended SPIs implemented
    [    0.000000] Root IRQ handler: gic_handle_irq
    [    0.000000] GICv3: GICv3 features: 16 PPIs
    [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001880000
    [    0.000000] ITS [mem 0x01820000-0x0182ffff]
    [    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
    [    0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
    [    0.000000] ITS@0x0000000001820000: allocated 524288 Devices @80800000 (flat, esz 8, psz 64K, shr 0)
    [    0.000000] ITS: using cache flushing for cmd queue
    [    0.000000] GICv3: using LPI property table @0x0000000080030000
    [    0.000000] GIC: using cache flushing for LPI property table
    [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000080040000
    [    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
    [    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
    [    0.000000] clocksource: arch_sys_counter: mask: 0x3ffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
    [    0.000001] sched_clock: 58 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
    [    0.008753] Console: colour dummy device 80x25
    [    0.014024] printk: console [tty1] enabled
    [    0.018282] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000)
    [    0.028992] pid_max: default: 32768 minimum: 301
    [    0.033794] LSM: Security Framework initializing
    [    0.038700] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
    [    0.046303] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
    [    0.056757] cblist_init_generic: Setting adjustable number of callback queues.
    [    0.064254] cblist_init_generic: Setting shift to 1 and lim to 1.
    [    0.070598] cblist_init_generic: Setting adjustable number of callback queues.
    [    0.078023] cblist_init_generic: Setting shift to 1 and lim to 1.
    [    0.084522] rcu: Hierarchical SRCU implementation.
    [    0.089456] rcu: 	Max phase no-delay instances is 1000.
    [    0.095650] Platform MSI: msi-controller@1820000 domain created
    [    0.102391] EFI services will not be available.
    [    0.107418] smp: Bringing up secondary CPUs ...
    [    0.112980] Detected VIPT I-cache on CPU1
    [    0.113137] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000
    [    0.113160] GICv3: CPU1: using allocated LPI pending table @0x0000000080050000
    [    0.113234] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
    [    0.113395] smp: Brought up 1 node, 2 CPUs
    [    0.142898] SMP: Total of 2 processors activated.
    [    0.147730] CPU features: detected: 32-bit EL0 Support
    [    0.153028] CPU features: detected: CRC32 instructions
    [    0.158378] CPU: All CPU(s) started at EL2
    [    0.162581] alternatives: applying system-wide alternatives
    [    0.170278] devtmpfs: initialized
    [    0.186795] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
    [    0.196860] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
    [    0.210879] pinctrl core: initialized pinctrl subsystem
    [    0.217240] DMI not present or invalid.
    [    0.222131] NET: Registered PF_NETLINK/PF_ROUTE protocol family
    [    0.229754] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
    [    0.237159] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
    [    0.245230] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
    [    0.253463] audit: initializing netlink subsys (disabled)
    [    0.259346] audit: type=2000 audit(0.164:1): state=initialized audit_enabled=0 res=1
    [    0.259985] thermal_sys: Registered thermal governor 'step_wise'
    [    0.267315] thermal_sys: Registered thermal governor 'power_allocator'
    [    0.273740] cpuidle: using governor menu
    [    0.284728] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
    [    0.291770] ASID allocator initialised with 65536 entries
    [    0.319416] KASLR disabled due to lack of seed
    [    0.332904] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
    [    0.339915] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page
    [    0.346344] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages
    [    0.353302] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page
    [    0.359727] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
    [    0.366683] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
    [    0.373107] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages
    [    0.380063] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
    [    0.388678] k3-chipinfo 43000014.chipid: Family:AM62X rev:SR1.0 JTAGID[0x0bb7e02f] Detected
    [    0.399153] iommu: Default domain type: Translated 
    [    0.404188] iommu: DMA domain TLB invalidation policy: strict mode 
    [    0.411015] SCSI subsystem initialized
    [    0.415383] usbcore: registered new interface driver usbfs
    [    0.421067] usbcore: registered new interface driver hub
    [    0.426551] usbcore: registered new device driver usb
    [    0.432434] pps_core: LinuxPPS API ver. 1 registered
    [    0.437538] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
    [    0.446923] PTP clock support registered
    [    0.451152] EDAC MC: Ver: 3.0.0
    [    0.455873] FPGA manager framework
    [    0.459511] Advanced Linux Sound Architecture Driver Initialized.
    [    0.467149] clocksource: Switched to clocksource arch_sys_counter
    [    0.473773] VFS: Disk quotas dquot_6.6.0
    [    0.477868] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
    [    0.493717] NET: Registered PF_INET protocol family
    [    0.499039] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
    [    0.508241] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
    [    0.516913] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
    [    0.524884] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
    [    0.532910] TCP bind hash table entries: 8192 (order: 6, 262144 bytes, linear)
    [    0.540697] TCP: Hash tables configured (established 8192 bind 8192)
    [    0.547433] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
    [    0.554269] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
    [    0.561740] NET: Registered PF_UNIX/PF_LOCAL protocol family
    [    0.568218] RPC: Registered named UNIX socket transport module.
    [    0.574340] RPC: Registered udp transport module.
    [    0.579168] RPC: Registered tcp transport module.
    [    0.583992] RPC: Registered tcp NFSv4.1 backchannel transport module.
    [    0.590605] NET: Registered PF_XDP protocol family
    [    0.596609] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
    [    0.607524] Initialise system trusted keyrings
    [    0.612424] workingset: timestamp_bits=46 max_order=18 bucket_order=0
    [    0.625488] squashfs: version 4.0 (2009/01/31) Phillip Lougher
    [    0.632401] NFS: Registering the id_resolver key type
    [    0.637668] Key type id_resolver registered
    [    0.641964] Key type id_legacy registered
    [    0.646161] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
    [    0.653038] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
    [    0.714970] Key type asymmetric registered
    [    0.719187] Asymmetric key parser 'x509' registered
    [    0.724266] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
    [    0.732098] io scheduler mq-deadline registered
    [    0.736759] io scheduler kyber registered
    [    0.746207] pinctrl-single 4084000.pinctrl: 34 pins, size 136
    [    0.753441] pinctrl-single f4000.pinctrl: 171 pins, size 684
    [    0.769493] Serial: 8250/16550 driver, 6 ports, IRQ sharing enabled
    [    0.790202] brd: module loaded
    [    0.799995] loop: module loaded
    [    0.808699] tun: Universal TUN/TAP device driver, 1.6
    [    0.815079] VFIO - User Level meta-driver version: 0.3
    [    0.821694] usbcore: registered new interface driver usb-storage
    [    0.828821] i2c_dev: i2c /dev entries driver
    [    0.836131] sdhci: Secure Digital Host Controller Interface driver
    [    0.842518] sdhci: Copyright(c) Pierre Ossman
    [    0.847657] sdhci-pltfm: SDHCI platform and OF driver helper
    [    0.854416] ledtrig-cpu: registered to indicate activity on CPUs
    [    0.860874] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
    [    0.868050] usbcore: registered new interface driver usbhid
    [    0.873807] usbhid: USB HID core driver
    [    0.879274] optee: probing for conduit method.
    [    0.883891] optee: revision 4.1 (012cdca4)
    [    0.884340] optee: dynamic shared memory is enabled
    [    0.894127] optee: initialized driver
    [    0.900508] Initializing XFRM netlink socket
    [    0.904992] NET: Registered PF_PACKET protocol family
    [    0.910296] Key type dns_resolver registered
    [    0.915268] registered taskstats version 1
    [    0.919510] Loading compiled-in X.509 certificates
    [    0.938252] ti-sci 44043000.system-controller: ABI: 3.1 (firmware rev 0x0009 '9.2.7--v09.02.07 (Kool Koala)')
    [    1.004135] ti-sci-clk 44043000.system-controller:clock-controller: recalc-rate failed for dev=81, clk=20, ret=-19
    [    1.041801] omap_i2c 4900000.i2c: bus 3 rev0.12 at 100 kHz
    [    1.049659] i2c 0-0030: Fixed dependency cycle(s) with /bus@f0000/i2c@20000000/pmic@30/regulators/buck2
    [    1.093611] input: tps65219-pwrbutton as /devices/platform/bus@f0000/20000000.i2c/i2c-0/0-0030/tps65219-pwrbutton.2.auto/input/input0
    [    1.108385] rtc-ds1307 0-0032: oscillator failed, set time!
    [    1.114398] rtc-ds1307 0-0032: registered as rtc0
    [    1.119401] rtc-ds1307 0-0032: hctosys: unable to read the hardware clock
    [    1.129188] at24 0-0050: supply vcc not found, using dummy regulator
    [    1.136292] at24 0-0050: 256 byte 24c02 EEPROM, writable, 16 bytes/write
    [    1.143269] omap_i2c 20000000.i2c: bus 0 rev0.12 at 400 kHz
    [    1.152026] pca953x 1-0021: supply vcc not found, using dummy regulator
    [    1.159026] pca953x 1-0021: using no AI
    [    1.183311] pca953x 1-0021: failed writing register
    [    1.188571] pca953x: probe of 1-0021 failed with error -121
    [    1.195078] at24 1-0057: supply vcc not found, using dummy regulator
    [    1.202431] at24 1-0057: 256 byte 24c02 EEPROM, writable, 16 bytes/write
    [    1.209401] omap_i2c 20010000.i2c: bus 1 rev0.12 at 100 kHz
    [    1.216705] omap_i2c 20020000.i2c: bus 2 rev0.12 at 100 kHz
    [    1.223701] omap_i2c 20030000.i2c: bus 4 rev0.12 at 100 kHz
    [    1.229781] ti-sci-intr 4210000.interrupt-controller: Interrupt Router 5 domain created
    [    1.238221] ti-sci-intr bus@f0000:interrupt-controller@a00000: Interrupt Router 3 domain created
    [    1.247555] ti-sci-inta 48000000.interrupt-controller: Interrupt Aggregator domain 28 created
    [    1.261569] ti-udma 485c0100.dma-controller: Number of rings: 82
    [    1.270702] ti-udma 485c0100.dma-controller: Channels: 48 (bchan: 18, tchan: 12, rchan: 18)
    [    1.282862] ti-udma 485c0000.dma-controller: Number of rings: 150
    [    1.294503] ti-udma 485c0000.dma-controller: Channels: 35 (tchan: 20, rchan: 15)
    [    1.305943] 4a00000.serial: ttyS3 at MMIO 0x4a00000 (irq = 291, base_baud = 3000000) is a 8250
    [    1.316417] 2800000.serial: ttyS2 at MMIO 0x2800000 (irq = 292, base_baud = 3000000) is a 8250
    [    1.325393] printk: console [ttyS2] enabled
    [    1.325393] printk: console [ttyS2] enabled
    [    1.333870] printk: bootconsole [ns16550a0] disabled
    [    1.333870] printk: bootconsole [ns16550a0] disabled
    [    1.345377] 2810000.serial: ttyS0 at MMIO 0x2810000 (irq = 293, base_baud = 3000000) is a 8250
    [    1.355064] omap8250 2850000.serial: PM domain pd:156 will not be powered off
    [    1.362611] 2850000.serial: ttyS4 at MMIO 0x2850000 (irq = 294, base_baud = 3000000) is a 8250
    [    1.371414] serial serial0: tty port ttyS4 registered
    [    1.378234] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
    [    1.423159] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [    1.432298] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA01103, cpsw version 0x6BA81103 Ports: 3 quirks:00000006
    [    1.445305] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.5
    [    1.452460] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512
    [    1.459426] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:0
    [    1.470282] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
    [    1.486913] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller
    [    1.492532] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 1
    [    1.500386] xhci-hcd xhci-hcd.3.auto: USB3 root hub has no ports
    [    1.506414] xhci-hcd xhci-hcd.3.auto: hcc params 0x0258fe6d hci version 0x110 quirks 0x0000008020010010
    [    1.515907] xhci-hcd xhci-hcd.3.auto: irq 299, io mem 0x31100000
    [    1.523058] hub 1-0:1.0: USB hub found
    [    1.526888] hub 1-0:1.0: 1 port detected
    [    1.640592] gpio-488 (PCIE_1_RESET#): hogged as output/low
    [    1.640750] mmc0: CQHCI version 5.10
    [    1.648877] gpio-427 (CTRL_SLEEP_MOCI#): hogged as output/high
    [    1.665561] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
    [    1.689896] mmc0: SDHCI controller on fa10000.mmc [fa10000.mmc] using ADMA 64-bit
    [    1.711156] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [    1.723908] mdio_bus 8000f00.mdio: MDIO device at address 7 is missing.
    [    1.730606] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867
    [    1.742508] sdhci-am654 fa00000.mmc: Got CD GPIO
    [    1.742509] sdhci-am654 fa20000.mmc: allocated mmc-pwrseq
    [    1.743028] mmc1: CQHCI version 5.10
    [    1.753003] mmc2: CQHCI version 5.10
    [    1.761185] debugfs: Directory 'pd:182' with parent 'pm_genpd' already present!
    [    1.771770] input: gpio-keys as /devices/platform/gpio-keys/input/input1
    [    1.781903] mmc0: new HS200 MMC card at address 0001
    [    1.784106] ti-sci-clk 44043000.system-controller:clock-controller: is_prepared failed for dev=81, clk=20, ret=-19
    [    1.786965] usb 1-1: new high-speed USB device number 2 using xhci-hcd
    [    1.802590] ALSA device list:
    [    1.804854] mmcblk0: mmc0:0001 Q2J55L 7.09 GiB 
    [    1.806742]   No soundcards found.
    [    1.811937] mmc2: SDHCI controller on fa20000.mmc [fa20000.mmc] using ADMA 64-bit
    [    1.822907] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
    [    1.831277]  mmcblk0: p1 p2
    [    1.835096] mmcblk0boot0: mmc0:0001 Q2J55L 16.0 MiB 
    [    1.841703] mmcblk0boot1: mmc0:0001 Q2J55L 16.0 MiB 
    [    1.847997] mmcblk0rpmb: mmc0:0001 Q2J55L 4.00 MiB, chardev (241:0)
    [    1.860467] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Quota mode: none.
    [    1.869147] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
    [    1.877970] devtmpfs: mounted
    [    1.882522] Freeing unused kernel memory: 2176K
    [    1.887239] Run /sbin/init as init process
    [    1.909936] mmc2: new ultra high speed DDR50 SDIO card at address 0001
    [    2.002614] hub 1-1:1.0: USB hub found
    [    2.006618] hub 1-1:1.0: 4 ports detected
    [    2.034122] systemd[1]: System time before build time, advancing clock.
    [    2.083655] NET: Registered PF_INET6 protocol family
    [    2.090570] Segment Routing with IPv6
    [    2.094370] In-situ OAM (IOAM) with IPv6
    [    2.128623] systemd[1]: systemd 250.5+ running in system mode (+PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP -GCRYPT -GNUTLS -OPENSSL +ACL +BLKID -CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -BZIP2 -LZ4 -XZ -ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=hybrid)
    [    2.161120] systemd[1]: Detected architecture arm64.
    
    Welcome to TDX Wayland with XWayland 6.6.0+build.12 (kirkstone)!
    
    [    2.232628] systemd[1]: Hostname set to <verdin-am62-15313474>.
    [    2.367297] usb 1-1.4: new high-speed USB device number 3 using xhci-hcd
    [    2.725818] systemd[1]: Queued start job for default target Graphical Interface.
    [    2.783105] systemd[1]: Created slice Slice /system/getty.
    [  OK  ] Created slice Slice /system/getty.
    [    2.805770] systemd[1]: Created slice Slice /system/modprobe.
    [  OK  ] Created slice Slice /system/modprobe.
    [    2.830020] systemd[1]: Created slice Slice /system/serial-getty.
    [  OK  ] Created slice Slice /system/serial-getty.
    [    2.853668] systemd[1]: Created slice User and Session Slice.
    [  OK  ] Created slice User and Session Slice.
    [    2.876003] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
    [  OK  ] Started Dispatch Password …ts to Console Directory Watch.
    [    2.899810] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
    [  OK  ] Started Forward Password R…uests to Wall Directory Watch.
    [    2.924265] systemd[1]: Reached target Host and Network Name Lookups.
    [  OK  ] Reached target Host and Network Name Lookups.
    [    2.947382] systemd[1]: Reached target Path Units.
    [  OK  ] Reached target Path Units.
    [    2.967571] systemd[1]: Reached target Remote File Systems.
    [  OK  ] Reached target Remote File Systems.
    [    2.987545] systemd[1]: Reached target Slice Units.
    [  OK  ] Reached target Slice Units.
    [    3.007936] systemd[1]: Reached target Swaps.
    [  OK  ] Reached target Swaps.
    [    3.088862] systemd[1]: Listening on RPCbind Server Activation Socket.
    [  OK  ] Listening on RPCbind Server Activation Socket.
    [    3.111892] systemd[1]: Reached target RPC Port Mapper.
    [  OK  ] Reached target RPC Port Mapper.
    [    3.138514] systemd[1]: Listening on Syslog Socket.
    [  OK  ] Listening on Syslog Socket.
    [    3.161139] systemd[1]: Listening on initctl Compatibility Named Pipe.
    [  OK  ] Listening on initctl Compatibility Named Pipe.
    [    3.187083] systemd[1]: Listening on Journal Audit Socket.
    [  OK  ] Listening on Journal Audit Socket.
    [    3.209695] systemd[1]: Listening on Journal Socket (/dev/log).
    [  OK  ] Listening on Journal Socket (/dev/log).
    [    3.234041] systemd[1]: Listening on Journal Socket.
    [  OK  ] Listening on Journal Socket.
    [    3.259103] systemd[1]: Listening on Network Service Netlink Socket.
    [  OK  ] Listening on Network Service Netlink Socket.
    [    3.290235] systemd[1]: Listening on udev Control Socket.
    [  OK  ] Listening on udev Control Socket.
    [    3.313760] systemd[1]: Listening on udev Kernel Socket.
    [  OK  ] Listening on udev Kernel Socket.
    [    3.338092] systemd[1]: Listening on User Database Manager Socket.
    [  OK  ] Listening on User Database Manager Socket.
    [    3.395983] systemd[1]: Mounting Huge Pages File System...
             Mounting Huge Pages File System...
    [    3.427650] systemd[1]: Mounting POSIX Message Queue File System...
             Mounting POSIX Message Queue File System...
    [    3.479925] systemd[1]: Mounting Kernel Debug File System...
             Mounting Kernel Debug File System...
    [    3.504713] systemd[1]: Kernel Trace File System was skipped because of a failed condition check (ConditionPathExists=/sys/kernel/tracing).
    [    3.547958] systemd[1]: Mounting Temporary Directory /tmp...
             Mounting Temporary Directory /tmp...
    [    3.584124] systemd[1]: Starting Create List of Static Device Nodes...
             Starting Create List of Static Device Nodes...
    [    3.618798] systemd[1]: Starting Load Kernel Module configfs...
             Starting Load Kernel Module configfs...
    [    3.664800] systemd[1]: Starting Load Kernel Module drm...
             Starting Load Kernel Module drm...
    [    3.695791] systemd[1]: Starting Load Kernel Module fuse...
             Starting Load Kernel Module fuse...
    [    3.714300] fuse: init (API version 7.37)
    [    3.727483] systemd[1]: Starting RPC Bind...
             Starting RPC Bind...
    [    3.772255] systemd[1]: Starting File System Check on Root Device...
             Starting File System Check on Root Device...
    [    3.805807] systemd[1]: Starting Journal Service...
             Starting Journal Service...
    [    3.824161] systemd[1]: Starting Load Kernel Modules...
             Starting Load Kernel Modules...
    [    3.852541] systemd[1]: Starting Generate network units from Kernel command line...
             Starting Generate network …ts from Kernel command line...
    [    3.924288] systemd[1]: Starting Coldplug All udev Devices...
             Starting Coldplug All udev Devices...
    [    3.960046] systemd[1]: Started RPC Bind.
    [  OK  ] Started RPC Bind.
    [    3.980400] systemd[1]: Mounted Huge Pages File System.
    [  OK  ] Mounted Huge Pages File System.
    [    4.008267] systemd[1]: Started Journal Service.
    [  OK  ] Started Journal Service.
    [  OK  ] Mounted POSIX Message Queue File System.
    [  OK  ] Mounted Kernel Debug File System.
    [  OK  ] Mounted Temporary Directory /tmp.
    [  OK  ] Finished Create List of Static Device Nodes.
    [  OK  ] Finished Load Kernel Module configfs.
    [  OK  ] Finished Load Kernel Module drm.
    [  OK  ] Finished Load Kernel Module fuse.
    [  OK  ] Finished File System Check on Root Device.
    [  OK  ] Finished Load Kernel Modules.
    [  OK  ] Finished Generate network units from Kernel command line.
             Mounting FUSE Control File System...
             Mounting Kernel Configuration File System...
             Starting Remount Root and Kernel File Systems...
             Starting Apply Kernel Variables...
    [    4.435355] EXT4-fs (mmcblk0p2): re-mounted. Quota mode: none.
    [  OK  ] Mounted FUSE Control File System.
    [  OK  ] Mounted Kernel Configuration File System.
    [  OK  ] Finished Remount Root and Kernel File Systems.
    [  OK  ] Finished Apply Kernel Variables.
             Starting Flush Journal to Persistent Storage...
    [    4.568336] systemd-journald[165]: Received client request to flush runtime journal.
             Starting Create Static Device Nodes in /dev...
    [  OK  ] Finished Flush Journal to Persistent Storage.
    [  OK  ] Finished Create Static Device Nodes in /dev.
    [  OK  ] Reached target Preparation for Local File Systems.
             Mounting /var/volatile...
    [    4.729921] audit: type=1334 audit(1651167747.692:2): prog-id=5 op=LOAD
    [    4.738042] audit: type=1334 audit(1651167747.700:3): prog-id=6 op=LOAD
             Starting Rule-based Manage…for Device Events and Files...
    [  OK  ] Finished Coldplug All udev Devices.
    [  OK  ] Mounted /var/volatile.
             Starting Wait for udev To …plete Device Initialization...
             Starting Load/Save Random Seed...
    [  OK  ] Reached target Local File Systems.
             Starting Create Volatile Files and Directories...
    [  OK  ] Started Rule-based Manager for Device Events and Files.
    [  OK  ] Finished Create Volatile Files and Directories.
             Starting Network Time Synchronization...
             Starting Record System Boot/Shutdown in UTMP...
    [  OK  ] Finished Record System Boot/Shutdown in UTMP.
    [  OK  ] Started Network Time Synchronizatio[    5.493236] systemd-journald[165]: Oldest entry in /run/log/journal/8fcb2eb825ae4b84b7d3d65de2fa836f/system.journal is older than the configured file retention duration (1month), suggesting rotation.
    n.
    [    5.521664] systemd-journald[165]: /run/log/journal/8fcb2eb825ae4b84b7d3d65de2fa836f/system.journal: Journal header limits reached or header out-of-date, rotating.
    [  OK  ] Reached target System Time Set.
    [    5.703644] random: crng init done
    [    5.704887] omap8250 2b300000.serial: No clock speed specified: using default: 48000000
    [    5.718564] 2b300000.serial: ttyS1 at MMIO 0x2b300000 (irq = 485, base_baud = 3000000) is a 8250
    [  OK  ] Finished Load/Save Random Seed.
    [    5.797244] lm75 0-0048: supply vs not found, using dummy regulator
    [    5.808512] ina2xx 1-0040: error configuring the device: -121
    [    5.833158] hwmon hwmon1: temp1_input not attached to any thermal zone
    [    5.844155] nau8822 1-001a: Failed to issue reset: -121
    [    5.850055] lm75 0-0048: hwmon1: sensor 'tmp1075'
    [    5.868502] nau8822: probe of 1-001a failed with error -121
    [    5.883880] CAN device driver interface
    [    5.901392] lm75 1-004f: supply vs not found, using dummy regulator
    [    5.907643] m_can_platform 4e08000.can: m_can device registered (irq=0, version=32)
    [    5.930504] hwmon hwmon2: temp1_input not attached to any thermal zone
    [    5.937536] m_can_platform 20701000.can: m_can device registered (irq=486, version=32)
    [    5.938639] lm75 1-004f: hwmon2: sensor 'tmp75c'
    [  OK  ] Found device /dev/ttyS2.
    [    6.256785] k3-m4-rproc 5000000.m4fss: device does not reserved memory regions, ret = -22
    [    6.270433] k3-m4-rproc 5000000.m4fss: reserved memory init failed, ret = -22
    [    6.281451] Bluetooth: Core ver 2.22
    [    6.282822] remoteproc remoteproc0: releasing 5000000.m4fss
    [    6.293835] k3-m4-rproc: probe of 5000000.m4fss failed with error -22
    [    6.308101] NET: Registered PF_BLUETOOTH protocol family
    [    6.334951] rtc-ti-k3 2b1f0000.rtc: registered as rtc1
    [    6.342148] Bluetooth: HCI device and connection manager initialized
    [    6.352458] Bluetooth: HCI socket layer initialized
    [    6.357729] Bluetooth: L2CAP socket layer initialized
    [    6.363415] Bluetooth: SCO socket layer initialized
    [    6.467573] cfg80211: Loading compiled-in X.509 certificates for regulatory database
    [    6.511810] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
    [    6.528438] cfg80211: Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
    [    6.597826] pvrsrvkm: loading out-of-tree module taints kernel.
    [    6.634863] platform 78000000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1)
    [    6.667635] platform 78000000.r5f: configured R5F for IPC-only mode
    [    6.678608] platform 78000000.r5f: device does not have reserved memory regions, ret = -22
    [    6.680808] lt8912 1-0048: supply vdd not found, using dummy regulator
    [    6.699995] k3_r5_rproc bus@f0000:bus@b00000:r5fss@78000000: reserved memory init failed, ret = -22
    [    6.702406] lt8912 1-0048: supply vccmipirx not found, using dummy regulator
    [    6.710521] remoteproc remoteproc0: releasing 78000000.r5f
    [    6.721921] k3_r5_rproc bus@f0000:bus@b00000:r5fss@78000000: k3_r5_cluster_rproc_init failed, ret = -22
    [    6.732972] lt8912 1-0048: supply vccsysclk not found, using dummy regulator
    [    6.736641] PVR_K:  193: Device: fd00000.gpu
    [    6.745704] lt8912 1-0048: supply vcclvdstx not found, using dummy regulator
    [    6.747686] k3_r5_rproc: probe of bus@f0000:bus@b00000:r5fss@78000000 failed with error -22
    [    6.760725] lt8912 1-0048: supply vcchdmitx not found, using dummy regulator
    [    6.774641] lt8912 1-0048: supply vcclvdspll not found, using dummy regulator
    [    6.780230] PVR_K:  193: Read BVNC 33.15.11.3 from HW device registers
    [    6.788722] lt8912 1-0048: supply vcchdmipll not found, using dummy regulator
    [    6.813237] PVR_K:  193: RGX Device registered with BVNC 33.15.11.3
    [    6.831807] [drm] Initialized pvr 23.3.6512818 20170530 for fd00000.gpu on minor 0
    [    6.854251] [drm] Initialized tidss 1.0.0 20180215 for 30200000.dss on minor 1
    [    6.864289] tidss 30200000.dss: [drm] Cannot find any crtc or sizes
    [    6.879082] tidss 30200000.dss: [drm] Cannot find any crtc or sizes
    [    7.144434] mwifiex_sdio mmc2:0001:1: info: FW download over, size 570976 bytes
    [    8.555276] mwifiex_sdio mmc2:0001:1: WLAN FW is active
    [    8.587903] mwifiex_sdio mmc2:0001:1: Unknown api_id: 5
    [    8.622133] mwifiex_sdio mmc2:0001:1: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84) 
    [    8.632030] mwifiex_sdio mmc2:0001:1: driver_version = mwifiex 1.0 (16.92.21.p84) 
    [*     ] A start job is running for Wait for…ice Initialization (2s / 2min 58s)
    
    

    24-04-30_20-19-09-Fail.txt
    Climate Chamber:       KWP 120
    Start Temperature:     -30.5�C
    End Temperature:       -30.0�C
    Measured Temperature:  -30.5�C
    Shelf Temperature:     -30.7�C
    Shelf Name:            Shelf_19
    Module Serial Number:  15313465
    Module Location:       Shelf 1, Carrier Board 3
    Test duration:         127.94s
    
    Test Log
    ********
    20:19:08.80 13->14 Waited 30.00s
    20:18:38.58 12->13 Set Reset Low
    20:18:38.47 11->12 Power Off
    20:18:38.36 5->11 Not found, time out: "login:"
    20:17:08.28 4->5 Message found: "Starting kernel"
    20:17:02.66 3->4 Message found: "U-Boot"
    20:17:02.55 2->3 Waited 1.00s
    20:17:01.41 1->2 Set Reset High
    20:17:01.30 0->1 Power On
    
    Search Strings
    **************
    PASS : U-Boot
    PASS : Less than 64MB RAM detected
    PASS : Booting from mmc failed!
    PASS : Starting kernel
    PASS : mmcblk0: mmc0
    PASS : cut here
    
    FAIL : rcu: INFO: rcu_preempt detected
    PASS : rcu: INFO: rcu_preempt self-detected stall
    FAIL : Internal error
    PASS : I/O error
    FAIL : login:
    PASS : critical temperature reached
    FAIL : RAM test, TestResult: PASS
    PASS : memtester failed
    FAIL : memtester version
    PASS : FAILURE
    PASS : running CQE recovery
    FAIL : Waiting for power-cycle
    0 : OAK ERROR
    0 : UART ERROR
    N/A : Device life time estimation type B 
    N/A : Pre EOL information
    2 : USB hub found
    0 : xception stack
    PASS : page allocation failure
    PASS : hwclock
    
    Terminal
    ********
    
    U-Boot SPL 2023.04-ti-gca551722e0b5 (Mar 22 2024 - 08:10:43 +0000)
    SYSFW ABI: 3.1 (firmware rev 0x0009 '9.2.7--v09.02.07 (Kool Koala)')
    SPL initial stack usage: 13408 bytes
    Trying to boot from MMC1
    Authentication passed
    Authentication passed
    Authentication passed
    Authentication passed
    Authentication passed
    Starting ATF on ARM64 core...
    
    NOTICE:  BL31: v2.10.0(release):v2.10.0-367-g00f1ec6b87-dirty
    NOTICE:  BL31: Built : 16:09:05, Feb  9 2024
    
    U-Boot SPL 2023.04-ti-gca551722e0b5 (Mar 22 2024 - 08:10:43 +0000)
    SYSFW ABI: 3.1 (firmware rev 0x0009 '9.2.7--v09.02.07 (Kool Koala)')
    SPL initial stack usage: 1856 bytes
    Trying to boot from MMC1
    Authentication passed
    Authentication passed
    
    
    U-Boot 2023.04-ti-gca551722e0b5 (Mar 22 2024 - 08:10:43 +0000)
    
    SoC:   AM62X SR1.0 HS-FS
    DRAM:  1 GiB
    Core:  143 devices, 31 uclasses, devicetree: separate
    MMC:   mmc@fa10000: 0, mmc@fa00000: 1
    Loading Environment from MMC... OK
    In:    serial@2800000
    Out:   serial@2800000
    Err:   serial@2800000
    Model: Toradex 0075 Verdin AM62 Dual 1GB WB IT V1.1B
    Serial#: 15313465
    MISSING TORADEX CARRIER CONFIG BLOCKS
    am65_cpsw_nuss ethernet@8000000: K3 CPSW: nuss_ver: 0x6BA01103 cpsw_ver: 0x6BA81103 ale_ver: 0x00290105 Ports:2 mdio_freq:1000000
    Setting variant to wifi
    Net:   
    Warning: ethernet@8000000port@1 MAC addresses don't match:
    Address in ROM is		1c:63:49:1f:df:7a
    Address in environment is	00:14:2d:e9:aa:39
    eth0: ethernet@8000000port@1 [PRIME]Could not get PHY for ethernet@8000000port@1: addr 7
    am65_cpsw_nuss_port ethernet@8000000port@2: phy_connect() failed
    
    Hit any key to stop autoboot:  1  0 
    MMC: no card present
    switch to partitions #0, OK
    mmc0(part 0) is current device
    Scanning mmc 0:1...
    Found U-Boot script /boot.scr
    6003 bytes read in 15 ms (390.6 KiB/s)
    ## Executing script at 90280000
    Loading DeviceTree: k3-am625-verdin-wifi-dev.dtb
    70049 bytes read in 17 ms (3.9 MiB/s)
    82 bytes read in 13 ms (5.9 KiB/s)
    Working FDT set to 90200000
    Applying Overlay: verdin-am62_dsi-to-hdmi_overlay.dtbo
    3050 bytes read in 15 ms (198.2 KiB/s)
    Applying Overlay: verdin-am62_spidev_overlay.dtbo
    560 bytes read in 15 ms (36.1 KiB/s)
    7861931 bytes read in 69 ms (108.7 MiB/s)
    Bootargs: root=PARTUUID=0c22282f-02 ro rootwait console=tty1 console=ttyS2,115200 consoleblank=0 earlycon=ns16550a,mmio32,0x02800000
       Uncompressing Kernel Image
    ## Flattened Device Tree blob at 90200000
       Booting using the fdt blob at 0x90200000
    Working FDT set to 90200000
    ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4)
    ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
    ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4)
       Loading Device Tree to 0000000098eac000, end 0000000098ee0fff ... OK
    Working FDT set to 98eac000
    
    Starting kernel ...
    
    [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
    [    0.000000] Linux version 6.1.80+git.0d9f1748ac17 (oe-user@oe-host) (aarch64-tdx-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Thu Mar 28 13:48:11 UTC 2024
    [    0.000000] Machine model: Toradex Verdin AM62 WB on Verdin Development Board
    [    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
    [    0.000000] printk: bootconsole [ns16550a0] enabled
    [    0.000000] efi: UEFI not found.
    [    0.000000] Reserved memory: created DMA memory pool at 0x000000009db00000, size 12 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-memory@9db00000, compatible id shared-dma-pool
    [    0.000000] Zone ranges:
    [    0.000000]   DMA      [mem 0x0000000080000000-0x00000000bfffffff]
    [    0.000000]   DMA32    empty
    [    0.000000]   Normal   empty
    [    0.000000] Movable zone start for each node
    [    0.000000] Early memory node ranges
    [    0.000000]   node   0: [mem 0x0000000080000000-0x000000009dafffff]
    [    0.000000]   node   0: [mem 0x000000009db00000-0x000000009e6fffff]
    [    0.000000]   node   0: [mem 0x000000009e700000-0x000000009e77ffff]
    [    0.000000]   node   0: [mem 0x000000009e780000-0x000000009fffffff]
    [    0.000000]   node   0: [mem 0x00000000a0000000-0x00000000bfffffff]
    [    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000bfffffff]
    [    0.000000] cma: Reserved 128 MiB at 0x00000000b6c00000
    [    0.000000] psci: probing for conduit method from DT.
    [    0.000000] psci: PSCIv1.1 detected in firmware.
    [    0.000000] psci: Using standard PSCI v0.2 function IDs
    [    0.000000] psci: Trusted OS migration not required
    [    0.000000] psci: SMC Calling Convention v1.4
    [    0.000000] percpu: Embedded 20 pages/cpu s41064 r8192 d32664 u81920
    [    0.000000] Detected VIPT I-cache on CPU0
    [    0.000000] CPU features: detected: GIC system register CPU interface
    [    0.000000] CPU features: detected: ARM erratum 845719
    [    0.000000] alternatives: applying boot alternatives
    [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258048
    [    0.000000] Kernel command line: root=PARTUUID=0c22282f-02 ro rootwait console=tty1 console=ttyS2,115200 consoleblank=0 earlycon=ns16550a,mmio32,0x02800000
    [    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
    [    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
    [    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
    [    0.000000] Memory: 840660K/1048576K available (11072K kernel code, 1354K rwdata, 3784K rodata, 2176K init, 422K bss, 76844K reserved, 131072K cma-reserved)
    [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
    [    0.000000] rcu: Preemptible hierarchical RCU implementation.
    [    0.000000] rcu: 	RCU event tracing is enabled.
    [    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
    [    0.000000] 	Trampoline variant of Tasks RCU enabled.
    [    0.000000] 	Tracing variant of Tasks RCU enabled.
    [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
    [    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
    [    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
    [    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
    [    0.000000] GICv3: 256 SPIs implemented
    [    0.000000] GICv3: 0 Extended SPIs implemented
    [    0.000000] Root IRQ handler: gic_handle_irq
    [    0.000000] GICv3: GICv3 features: 16 PPIs
    [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001880000
    [    0.000000] ITS [mem 0x01820000-0x0182ffff]
    [    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
    [    0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
    [    0.000000] ITS@0x0000000001820000: allocated 524288 Devices @80800000 (flat, esz 8, psz 64K, shr 0)
    [    0.000000] ITS: using cache flushing for cmd queue
    [    0.000000] GICv3: using LPI property table @0x0000000080030000
    [    0.000000] GIC: using cache flushing for LPI property table
    [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000080040000
    [    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
    [    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
    [    0.000000] clocksource: arch_sys_counter: mask: 0x3ffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
    [    0.000001] sched_clock: 58 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
    [    0.008749] Console: colour dummy device 80x25
    [    0.014017] printk: console [tty1] enabled
    [    0.018275] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000)
    [    0.028985] pid_max: default: 32768 minimum: 301
    [    0.033788] LSM: Security Framework initializing
    [    0.038696] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
    [    0.046301] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
    [    0.056767] cblist_init_generic: Setting adjustable number of callback queues.
    [    0.064268] cblist_init_generic: Setting shift to 1 and lim to 1.
    [    0.070617] cblist_init_generic: Setting adjustable number of callback queues.
    [    0.078039] cblist_init_generic: Setting shift to 1 and lim to 1.
    [    0.084535] rcu: Hierarchical SRCU implementation.
    [    0.089469] rcu: 	Max phase no-delay instances is 1000.
    [    0.095666] Platform MSI: msi-controller@1820000 domain created
    [    0.102435] EFI services will not be available.
    [    0.107471] smp: Bringing up secondary CPUs ...
    [    0.113030] Detected VIPT I-cache on CPU1
    [    0.113193] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000
    [    0.113215] GICv3: CPU1: using allocated LPI pending table @0x0000000080050000
    [    0.113289] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
    [    0.113449] smp: Brought up 1 node, 2 CPUs
    [    0.142956] SMP: Total of 2 processors activated.
    [    0.147788] CPU features: detected: 32-bit EL0 Support
    [    0.153088] CPU features: detected: CRC32 instructions
    [    0.158446] CPU: All CPU(s) started at EL2
    [    0.162648] alternatives: applying system-wide alternatives
    [    0.170329] devtmpfs: initialized
    [    0.186867] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
    [    0.196931] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
    [    0.210942] pinctrl core: initialized pinctrl subsystem
    [    0.217309] DMI not present or invalid.
    [    0.222196] NET: Registered PF_NETLINK/PF_ROUTE protocol family
    [    0.229804] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
    [    0.237205] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
    [    0.245278] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
    [    0.253512] audit: initializing netlink subsys (disabled)
    [    0.259394] audit: type=2000 audit(0.164:1): state=initialized audit_enabled=0 res=1
    [    0.260030] thermal_sys: Registered thermal governor 'step_wise'
    [    0.267365] thermal_sys: Registered thermal governor 'power_allocator'
    [    0.273782] cpuidle: using governor menu
    [    0.284777] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
    [    0.291816] ASID allocator initialised with 65536 entries
    [    0.319413] KASLR disabled due to lack of seed
    [    0.332917] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
    [    0.339921] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page
    [    0.346349] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages
    [    0.353306] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page
    [    0.359731] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
    [    0.366687] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
    [    0.373111] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages
    [    0.380067] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
    [    0.388698] k3-chipinfo 43000014.chipid: Family:AM62X rev:SR1.0 JTAGID[0x0bb7e02f] Detected
    [    0.399208] iommu: Default domain type: Translated 
    [    0.404244] iommu: DMA domain TLB invalidation policy: strict mode 
    [    0.411068] SCSI subsystem initialized
    [    0.415418] usbcore: registered new interface driver usbfs
    [    0.421102] usbcore: registered new interface driver hub
    [    0.426588] usbcore: registered new device driver usb
    [    0.432468] pps_core: LinuxPPS API ver. 1 registered
    [    0.437573] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
    [    0.446954] PTP clock support registered
    [    0.451184] EDAC MC: Ver: 3.0.0
    [    0.455900] FPGA manager framework
    [    0.459541] Advanced Linux Sound Architecture Driver Initialized.
    [    0.467178] clocksource: Switched to clocksource arch_sys_counter
    [    0.473797] VFS: Disk quotas dquot_6.6.0
    [    0.477893] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
    [    0.493733] NET: Registered PF_INET protocol family
    [    0.499051] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
    [    0.508255] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
    [    0.516932] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
    [    0.524902] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
    [    0.532931] TCP bind hash table entries: 8192 (order: 6, 262144 bytes, linear)
    [    0.540711] TCP: Hash tables configured (established 8192 bind 8192)
    [    0.547443] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
    [    0.554285] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
    [    0.561761] NET: Registered PF_UNIX/PF_LOCAL protocol family
    [    0.568240] RPC: Registered named UNIX socket transport module.
    [    0.574362] RPC: Registered udp transport module.
    [    0.579189] RPC: Registered tcp transport module.
    [    0.584014] RPC: Registered tcp NFSv4.1 backchannel transport module.
    [    0.590627] NET: Registered PF_XDP protocol family
    [    0.596633] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
    [    0.607542] Initialise system trusted keyrings
    [    0.612440] workingset: timestamp_bits=46 max_order=18 bucket_order=0
    [    0.625488] squashfs: version 4.0 (2009/01/31) Phillip Lougher
    [    0.632393] NFS: Registering the id_resolver key type
    [    0.637646] Key type id_resolver registered
    [    0.641942] Key type id_legacy registered
    [    0.646135] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
    [    0.653013] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
    [    0.714491] Key type asymmetric registered
    [    0.718707] Asymmetric key parser 'x509' registered
    [    0.723786] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
    [    0.731621] io scheduler mq-deadline registered
    [    0.736280] io scheduler kyber registered
    [    0.745689] pinctrl-single 4084000.pinctrl: 34 pins, size 136
    [    0.752881] pinctrl-single f4000.pinctrl: 171 pins, size 684
    [    0.768909] Serial: 8250/16550 driver, 6 ports, IRQ sharing enabled
    [    0.789653] brd: module loaded
    [    0.799454] loop: module loaded
    [    0.808158] tun: Universal TUN/TAP device driver, 1.6
    [    0.814523] VFIO - User Level meta-driver version: 0.3
    [    0.821119] usbcore: registered new interface driver usb-storage
    [    0.828256] i2c_dev: i2c /dev entries driver
    [    0.835537] sdhci: Secure Digital Host Controller Interface driver
    [    0.841910] sdhci: Copyright(c) Pierre Ossman
    [    0.847024] sdhci-pltfm: SDHCI platform and OF driver helper
    [    0.853791] ledtrig-cpu: registered to indicate activity on CPUs
    [    0.860254] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
    [    0.867452] usbcore: registered new interface driver usbhid
    [    0.873193] usbhid: USB HID core driver
    [    0.878631] optee: probing for conduit method.
    [    0.883269] optee: revision 4.1 (012cdca4)
    [    0.883696] optee: dynamic shared memory is enabled
    [    0.893509] optee: initialized driver
    [    0.899876] Initializing XFRM netlink socket
    [    0.904352] NET: Registered PF_PACKET protocol family
    [    0.909650] Key type dns_resolver registered
    [    0.914592] registered taskstats version 1
    [    0.918845] Loading compiled-in X.509 certificates
    [    0.937579] ti-sci 44043000.system-controller: ABI: 3.1 (firmware rev 0x0009 '9.2.7--v09.02.07 (Kool Koala)')
    [    1.003548] ti-sci-clk 44043000.system-controller:clock-controller: recalc-rate failed for dev=81, clk=20, ret=-19
    [    1.041100] omap_i2c 4900000.i2c: bus 3 rev0.12 at 100 kHz
    [    1.048962] i2c 0-0030: Fixed dependency cycle(s) with /bus@f0000/i2c@20000000/pmic@30/regulators/buck2
    [    1.093618] input: tps65219-pwrbutton as /devices/platform/bus@f0000/20000000.i2c/i2c-0/0-0030/tps65219-pwrbutton.2.auto/input/input0
    [    1.108381] rtc-ds1307 0-0032: oscillator failed, set time!
    [    1.114405] rtc-ds1307 0-0032: registered as rtc0
    [    1.119405] rtc-ds1307 0-0032: hctosys: unable to read the hardware clock
    [    1.129165] at24 0-0050: supply vcc not found, using dummy regulator
    [    1.136260] at24 0-0050: 256 byte 24c02 EEPROM, writable, 16 bytes/write
    [    1.143248] omap_i2c 20000000.i2c: bus 0 rev0.12 at 400 kHz
    [    1.152042] pca953x 1-0021: supply vcc not found, using dummy regulator
    [    1.159040] pca953x 1-0021: using no AI
    [    1.183341] pca953x 1-0021: failed writing register
    [    1.188592] pca953x: probe of 1-0021 failed with error -121
    [    1.195089] at24 1-0057: supply vcc not found, using dummy regulator
    [    1.202448] at24 1-0057: 256 byte 24c02 EEPROM, writable, 16 bytes/write
    [    1.209424] omap_i2c 20010000.i2c: bus 1 rev0.12 at 100 kHz
    [    1.216726] omap_i2c 20020000.i2c: bus 2 rev0.12 at 100 kHz
    [    1.223715] omap_i2c 20030000.i2c: bus 4 rev0.12 at 100 kHz
    [    1.229795] ti-sci-intr 4210000.interrupt-controller: Interrupt Router 5 domain created
    [    1.238227] ti-sci-intr bus@f0000:interrupt-controller@a00000: Interrupt Router 3 domain created
    [    1.247569] ti-sci-inta 48000000.interrupt-controller: Interrupt Aggregator domain 28 created
    [    1.261541] ti-udma 485c0100.dma-controller: Number of rings: 82
    [    1.270686] ti-udma 485c0100.dma-controller: Channels: 48 (bchan: 18, tchan: 12, rchan: 18)
    [    1.282823] ti-udma 485c0000.dma-controller: Number of rings: 150
    [    1.294473] ti-udma 485c0000.dma-controller: Channels: 35 (tchan: 20, rchan: 15)
    [    1.305894] 4a00000.serial: ttyS3 at MMIO 0x4a00000 (irq = 291, base_baud = 3000000) is a 8250
    [    1.316342] 2800000.serial: ttyS2 at MMIO 0x2800000 (irq = 292, base_baud = 3000000) is a 8250
    [    1.325303] printk: console [ttyS2] enabled
    [    1.325303] printk: console [ttyS2] enabled
    [    1.333786] printk: bootconsole [ns16550a0] disabled
    [    1.333786] printk: bootconsole [ns16550a0] disabled
    [    1.345276] 2810000.serial: ttyS0 at MMIO 0x2810000 (irq = 293, base_baud = 3000000) is a 8250
    [    1.354951] omap8250 2850000.serial: PM domain pd:156 will not be powered off
    [    1.362505] 2850000.serial: ttyS4 at MMIO 0x2850000 (irq = 294, base_baud = 3000000) is a 8250
    [    1.371324] serial serial0: tty port ttyS4 registered
    [    1.378155] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
    [    1.423187] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [    1.432323] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA01103, cpsw version 0x6BA81103 Ports: 3 quirks:00000006
    [    1.445324] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.5
    [    1.452479] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512
    [    1.459436] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:0
    [    1.470254] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
    [    1.486884] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller
    [    1.492555] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 1
    [    1.500433] xhci-hcd xhci-hcd.3.auto: USB3 root hub has no ports
    [    1.506461] xhci-hcd xhci-hcd.3.auto: hcc params 0x0258fe6d hci version 0x110 quirks 0x0000008020010010
    [    1.515955] xhci-hcd xhci-hcd.3.auto: irq 299, io mem 0x31100000
    [    1.523133] hub 1-0:1.0: USB hub found
    [    1.526954] hub 1-0:1.0: 1 port detected
    [    1.640353] mmc0: CQHCI version 5.10
    [    1.640379] gpio-488 (PCIE_1_RESET#): hogged as output/low
    [    1.651954] gpio-427 (CTRL_SLEEP_MOCI#): hogged as output/high
    [    1.668015] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
    [    1.685314] mmc0: SDHCI controller on fa10000.mmc [fa10000.mmc] using ADMA 64-bit
    [    1.711182] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [    1.724038] mdio_bus 8000f00.mdio: MDIO device at address 7 is missing.
    [    1.730735] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867
    [    1.742556] sdhci-am654 fa00000.mmc: Got CD GPIO
    [    1.748505] debugfs: Directory 'pd:182' with parent 'pm_genpd' already present!
    [    1.748512] sdhci-am654 fa20000.mmc: allocated mmc-pwrseq
    [    1.748915] mmc1: CQHCI version 5.10
    [    1.759409] input: gpio-keys as /devices/platform/gpio-keys/input/input1
    [    1.765180] mmc2: CQHCI version 5.10
    [    1.780159] ti-sci-clk 44043000.system-controller:clock-controller: is_prepared failed for dev=81, clk=20, ret=-19
    [    1.790603] usb 1-1: new high-speed USB device number 2 using xhci-hcd
    [    1.799203] mmc0: new HS200 MMC card at address 0001
    [    1.802710] ALSA device list:
    [    1.805213] mmcblk0: mmc0:0001 Q2J55L 7.09 GiB 
    [    1.807134]   No soundcards found.
    [    1.815433]  mmcblk0: p1 p2
    [    1.818298] mmc2: SDHCI controller on fa20000.mmc [fa20000.mmc] using ADMA 64-bit
    [    1.825983] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
    [    1.834475] mmcblk0boot0: mmc0:0001 Q2J55L 16.0 MiB 
    [    1.841240] mmcblk0boot1: mmc0:0001 Q2J55L 16.0 MiB 
    [    1.847510] mmcblk0rpmb: mmc0:0001 Q2J55L 4.00 MiB, chardev (241:0)
    [    1.860128] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Quota mode: none.
    [    1.868889] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
    [    1.878894] devtmpfs: mounted
    [    1.883605] Freeing unused kernel memory: 2176K
    [    1.888341] Run /sbin/init as init process
    [    1.910166] mmc2: new ultra high speed DDR50 SDIO card at address 0001
    [    2.002628] hub 1-1:1.0: USB hub found
    [    2.006808] hub 1-1:1.0: 4 ports detected
    [    2.035528] systemd[1]: System time before build time, advancing clock.
    [    2.083391] NET: Registered PF_INET6 protocol family
    [    2.090295] Segment Routing with IPv6
    [    2.094070] In-situ OAM (IOAM) with IPv6
    [    2.122827] systemd[1]: systemd 250.5+ running in system mode (+PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP -GCRYPT -GNUTLS -OPENSSL +ACL +BLKID -CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -BZIP2 -LZ4 -XZ -ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=hybrid)
    [    2.155438] systemd[1]: Detected architecture arm64.
    
    Welcome to TDX Wayland with XWayland 6.6.0+build.12 (kirkstone)!
    
    [    2.228263] systemd[1]: Hostname set to <verdin-am62-15313465>.
    [    2.359320] usb 1-1.4: new high-speed USB device number 3 using xhci-hcd
    [    2.470632] omap8250 2b300000.serial: No clock speed specified: using default: 48000000
    [    2.479887] 2b300000.serial: ttyS1 at MMIO 0x2b300000 (irq = 485, base_baud = 3000000) is a 8250
    [    2.723474] systemd[1]: Queued start job for default target Graphical Interface.
    [    2.783027] systemd[1]: Created slice Slice /system/getty.
    [  OK  ] Created slice Slice /system/getty.
    [    2.805795] systemd[1]: Created slice Slice /system/modprobe.
    [  OK  ] Created slice Slice /system/modprobe.
    [    2.830061] systemd[1]: Created slice Slice /system/serial-getty.
    [  OK  ] Created slice Slice /system/serial-getty.
    [    2.853678] systemd[1]: Created slice User and Session Slice.
    [  OK  ] Created slice User and Session Slice.
    [    2.876018] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
    [  OK  ] Started Dispatch Password …ts to Console Directory Watch.
    [    2.899837] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
    [  OK  ] Started Forward Password R…uests to Wall Directory Watch.
    [    2.924356] systemd[1]: Reached target Host and Network Name Lookups.
    [  OK  ] Reached target Host and Network Name Lookups.
    [    2.947407] systemd[1]: Reached target Path Units.
    [  OK  ] Reached target Path Units.
    [    2.967578] systemd[1]: Reached target Remote File Systems.
    [  OK  ] Reached target Remote File Systems.
    [    2.987546] systemd[1]: Reached target Slice Units.
    [  OK  ] Reached target Slice Units.
    [    3.007998] systemd[1]: Reached target Swaps.
    [  OK  ] Reached target Swaps.
    [    3.088802] systemd[1]: Listening on RPCbind Server Activation Socket.
    [  OK  ] Listening on RPCbind Server Activation Socket.
    [    3.111924] systemd[1]: Reached target RPC Port Mapper.
    [  OK  ] Reached target RPC Port Mapper.
    [    3.138587] systemd[1]: Listening on Syslog Socket.
    [  OK  ] Listening on Syslog Socket.
    [    3.161189] systemd[1]: Listening on initctl Compatibility Named Pipe.
    [  OK  ] Listening on initctl Compatibility Named Pipe.
    [    3.185461] systemd[1]: Listening on Journal Audit Socket.
    [  OK  ] Listening on Journal Audit Socket.
    [    3.209817] systemd[1]: Listening on Journal Socket (/dev/log).
    [  OK  ] Listening on Journal Socket (/dev/log).
    [    3.234080] systemd[1]: Listening on Journal Socket.
    [  OK  ] Listening on Journal Socket.
    [    3.259087] systemd[1]: Listening on Network Service Netlink Socket.
    [  OK  ] Listening on Network Service Netlink Socket.
    [    3.286443] systemd[1]: Listening on udev Control Socket.
    [  OK  ] Listening on udev Control Socket.
    [    3.309472] systemd[1]: Listening on udev Kernel Socket.
    [  OK  ] Listening on udev Kernel Socket.
    [    3.334053] systemd[1]: Listening on User Database Manager Socket.
    [  OK  ] Listening on User Database Manager Socket.
    [    3.391973] systemd[1]: Mounting Huge Pages File System...
             Mounting Huge Pages File System...
    [    3.423627] systemd[1]: Mounting POSIX Message Queue File System...
             Mounting POSIX Message Queue File System...
    [    3.471945] systemd[1]: Mounting Kernel Debug File System...
             Mounting Kernel Debug File System...
    [    3.496747] systemd[1]: Kernel Trace File System was skipped because of a failed condition check (ConditionPathExists=/sys/kernel/tracing).
    [    3.552106] systemd[1]: Mounting Temporary Directory /tmp...
             Mounting Temporary Directory /tmp...
    [    3.588341] systemd[1]: Starting Create List of Static Device Nodes...
             Starting Create List of Static Device Nodes...
    [    3.640291] systemd[1]: Starting Load Kernel Module configfs...
             Starting Load Kernel Module configfs...
    [    3.665050] systemd[1]: Starting Load Kernel Module drm...
             Starting Load Kernel Module drm...
    [    3.692559] systemd[1]: Starting Load Kernel Module fuse...
             Starting Load Kernel Module fuse...
    [    3.715711] fuse: init (API version 7.37)
    [    3.724796] systemd[1]: Starting RPC Bind...
             Starting RPC Bind...
    [    3.751502] systemd[1]: Starting File System Check on Root Device...
             Starting File System Check on Root Device...
    [    3.784184] systemd[1]: Starting Journal Service...
             Starting Journal Service...
    [    3.810158] systemd[1]: Starting Load Kernel Modules...
             Starting Load Kernel Modules...
    [    3.840122] systemd[1]: Starting Generate network units from Kernel command line...
             Starting Generate network …ts from Kernel command line...
    [    3.867988] systemd[1]: Starting Coldplug All udev Devices...
             Starting Coldplug All udev Devices...
    [    3.897270] systemd[1]: Started RPC Bind.
    [  OK  ] Started RPC Bind.
    [    3.915731] systemd[1]: Mounted Huge Pages File System.
    [  OK  ] Mounted Huge Pages File System.
    [    3.940332] systemd[1]: Mounted POSIX Message Queue File System.
    [  OK  ] Mounted POSIX Message Queue File System.
    [    3.968320] systemd[1]: Mounted Kernel Debug File System.
    [  OK  ] Mounted Kernel Debug File System.
    [    3.992271] systemd[1]: Started Journal Service.
    [  OK  ] Started Journal Service.
    [  OK  ] Mounted Temporary Directory /tmp.
    [  OK  ] Finished Create List of Static Device Nodes.
    [  OK  ] Finished Load Kernel Module configfs.
    [  OK  ] Finished Load Kernel Module drm.
    [  OK  ] Finished Load Kernel Module fuse.
    [  OK  ] Finished File System Check on Root Device.
    [  OK  ] Finished Load Kernel Modules.
    [  OK  ] Finished Generate network units from Kernel command line.
             Mounting FUSE Control File System...
             Mounting Kernel Configuration File System...
             Starting Remount Root and Kernel File Systems...
             Starting Apply Kernel Variables...
    [  OK  ] Mounted FUSE Control File System.
    [  OK  ] Mounted Kernel Configuration File System.
    [    4.350811] EXT4-fs (mmcblk0p2): re-mounted. Quota mode: none.
    [  OK  ] Finished Apply Kernel Variables.
    [  OK  ] Finished Remount Root and Kernel File Systems.
             Starting Flush Journal to Persistent Storage...
    [    4.438803] systemd-journald[165]: Received client request to flush runtime journal.
             Starting Create Static Device Nodes in /dev...
    [  OK  ] Finished Flush Journal to Persistent Storage.
    [  OK  ] Finished Create Static Device Nodes in /dev.
    [  OK  ] Reached target Preparation for Local File Systems.
    [    4.552287] audit: type=1334 audit(1651167747.516:2): prog-id=5 op=LOAD
    [    4.563843] audit: type=1334 audit(1651167747.524:3): prog-id=6 op=LOAD
             Starting Rule-based Manage…for Device Events and Files...
    [  OK  ] Finished Coldplug All udev Devices.
             Starting Wait for udev To …plete Device Initialization...
    [  OK  ] Started Rule-based Manager for Device Events and Files.
             Mounting /var/volatile...
    [  OK  ] Mounted /var/volatile.
             Starting Load/Save Random Seed...
    [  OK  ] Reached target Local File Systems.
             Starting Create Volatile Files and Directories...
    [  OK  ] Finished Create Volatile Files and Directories.
             Starting Network Time Synchronization...
             Starting Record System Boot/Shutdown in UTMP...
    [    5.171564] random: crng init done
    [  OK  ] Finished Load/Save Random Seed.
    [  OK  ] Finished Record System Boot/Shutdown in UTMP.
    [    5.272290] CAN device driver interface
    [    5.332653] lm75 0-0048: supply vs not found, using dummy regulator
    [    5.363867] hwmon hwmon1: temp1_input not attached to any thermal zone
    [    5.372125] lm75 0-0048: hwmon1: sensor 'tmp1075'
    [    5.401785] lm75 1-004f: supply vs not found, using dummy regulator
    [    5.444043] m_can_platform 4e08000.can: m_can device registered (irq=0, version=32)
    [    5.455505] ina2xx 1-0040: error configuring the device: -121
    [    5.468703] nau8822 1-001a: Failed to issue reset: -121
    [    5.474789] hwmon hwmon2: temp1_input not attached to any thermal zone
    [    5.480196] nau8822: probe of 1-001a failed with error -121
    [    5.482432] lm75 1-004f: hwmon2: sensor 'tmp75c'
    [    5.518461] m_can_platform 20701000.can: m_can device registered (irq=487, version=32)
    [  OK  ] Found device /dev/ttyS2.
    [    5.749016] systemd-journald[165]: Oldest entry in /run/log/journal/35a1c7ac7be84291a78cad9d2f797988/system.journal is older than the configured file retention duration (1month), suggesting rotation.
    [  OK  ] Started Network Time Synchronizatio[    5.774759] systemd-journald[165]: /run/log/journal/35a1c7ac7be84291a78cad9d2f797988/system.journal: Journal header limits reached or header out-of-date, rotating.
    n.
    [  OK  ] Reached target System Time Set.
    [    5.857262] Bluetooth: Core ver 2.22
    [    5.881979] NET: Registered PF_BLUETOOTH protocol family
    [    5.904783] k3-m4-rproc 5000000.m4fss: device does not reserved memory regions, ret = -22
    [    5.905330] Bluetooth: HCI device and connection manager initialized
    [    5.940510] Bluetooth: HCI socket layer initialized
    [    5.946955] k3-m4-rproc 5000000.m4fss: reserved memory init failed, ret = -22
    [    5.955837] Bluetooth: L2CAP socket layer initialized
    [    5.983424] Bluetooth: SCO socket layer initialized
    [    5.993744] remoteproc remoteproc0: releasing 5000000.m4fss
    [    6.002797] k3-m4-rproc: probe of 5000000.m4fss failed with error -22
    [    6.067773] cfg80211: Loading compiled-in X.509 certificates for regulatory database
    [    6.108827] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
    [    6.116670] cfg80211: Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
    [    6.160786] rtc-ti-k3 2b1f0000.rtc: registered as rtc1
    [    6.166389] platform 78000000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1)
    [    6.185562] platform 78000000.r5f: configured R5F for IPC-only mode
    [    6.226409] platform 78000000.r5f: device does not have reserved memory regions, ret = -22
    [    6.234871] k3_r5_rproc bus@f0000:bus@b00000:r5fss@78000000: reserved memory init failed, ret = -22
    [    6.244057] remoteproc remoteproc0: releasing 78000000.r5f
    [    6.250010] k3_r5_rproc bus@f0000:bus@b00000:r5fss@78000000: k3_r5_cluster_rproc_init failed, ret = -22
    [    6.271855] k3_r5_rproc: probe of bus@f0000:bus@b00000:r5fss@78000000 failed with error -22
    [    6.290250] pvrsrvkm: loading out-of-tree module taints kernel.
    [    6.358951] lt8912 1-0048: supply vdd not found, using dummy regulator
    [    6.365964] lt8912 1-0048: supply vccmipirx not found, using dummy regulator
    [    6.375030] lt8912 1-0048: supply vccsysclk not found, using dummy regulator
    [    6.384069] lt8912 1-0048: supply vcclvdstx not found, using dummy regulator
    [    6.395849] lt8912 1-0048: supply vcchdmitx not found, using dummy regulator
    [    6.408396] lt8912 1-0048: supply vcclvdspll not found, using dummy regulator
    [    6.419692] lt8912 1-0048: supply vcchdmipll not found, using dummy regulator
    [    6.456913] PVR_K:  188: Device: fd00000.gpu
    [    6.469022] [drm] Initialized tidss 1.0.0 20180215 for 30200000.dss on minor 0
    [    6.480559] tidss 30200000.dss: [drm] Cannot find any crtc or sizes
    [    6.498752] tidss 30200000.dss: [drm] Cannot find any crtc or sizes
    [    6.543166] PVR_K:  188: Read BVNC 33.15.11.3 from HW device registers
    [    6.566169] PVR_K:  188: RGX Device registered with BVNC 33.15.11.3
    [    6.595807] [drm] Initialized pvr 23.3.6512818 20170530 for fd00000.gpu on minor 1
    [    6.720253] mwifiex_sdio mmc2:0001:1: info: FW download over, size 570976 bytes
    [    8.135330] mwifiex_sdio mmc2:0001:1: WLAN FW is active
    [    8.178063] mwifiex_sdio mmc2:0001:1: Unknown api_id: 5
    [    8.213237] mwifiex_sdio mmc2:0001:1: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84) 
    [    8.221519] mwifiex_sdio mmc2:0001:1: driver_version = mwifiex 1.0 (16.92.21.p84) 
    [*     ] A start job is running for Wait for…ice Initialization (2s / 2min 58s)
    [   87.013061] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
    [   87.019219] rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-1): P501/1:b..l
    [   87.026212] 	(detected by 1, t=19643 jiffies, g=1813, q=259 ncpus=2)
    [   87.032583] task:sed             state:R  running task     stack:0     pid:501   ppid:499    flags:0x00000a0c
    [   87.042508] Call trace:
    [   87.044951]  __switch_to+0xd4/0x130
    [   87.048452]  __schedule+0x230/0x664
    [   87.051937]  preempt_schedule_irq+0x44/0xd0
    [   87.056116]  el1_interrupt+0x4c/0x70
    [   87.059690]  el1h_64_irq_handler+0x18/0x2c
    [   87.063783]  el1h_64_irq+0x64/0x68
    [   87.067183]  __memcg_kmem_charge_page+0xbc/0x280
    [   87.071798]  __alloc_pages+0x1d0/0xc80
    [   87.075545]  get_zeroed_page+0x24/0x50
    [   87.079291]  __pud_alloc+0x40/0x150
    [   87.082776]  __handle_mm_fault+0x1ac/0xc10
    [   87.086867]  handle_mm_fault+0xec/0x280
    [   87.090697]  do_page_fault+0x124/0x3d0
    [   87.094445]  do_translation_fault+0xac/0xc0
    [   87.098623]  do_mem_abort+0x44/0x94
    [   87.102107]  el1_abort+0x40/0x6c
    [   87.105332]  el1h_64_sync_handler+0xd8/0xe4
    [   87.109512]  el1h_64_sync+0x64/0x68
    [   87.112995]  __arch_clear_user+0x10/0x80
    [   87.116915]  load_elf_binary+0x674/0x144c
    [   87.120922]  bprm_execve+0x21c/0x55c
    [   87.124493]  do_execveat_common.isra.0+0x198/0x240
    [   87.129278]  __arm64_sys_execve+0x40/0x54
    [   87.133280]  invoke_syscall+0x48/0x114
    [   87.137026]  el0_svc_common.constprop.0+0xd4/0xfc
    [   87.141724]  do_el0_svc+0x20/0x30
    [   87.145035]  el0_svc+0x28/0xa0
    [   87.148086]  el0t_64_sync_handler+0xbc/0x140
    [   87.152351]  el0t_64_sync+0x18c/0x190
    [   87.156011] rcu: rcu_preempt kthread timer wakeup didn't happen for 19644 jiffies! g1813 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
    [   87.167403] rcu: 	Possible timer handling issue on cpu=1 timer-softirq=611
    [   87.174285] rcu: rcu_preempt kthread starved for 19680 jiffies! g1813 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
    [   87.184549] rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
    [   87.193674] rcu: RCU grace-period kthread stack dump:
    [   87.198718] task:rcu_preempt     state:R  running task     stack:0     pid:14    ppid:2      flags:0x00000008
    [   87.208630] Call trace:
    [   87.211071]  __switch_to+0xd4/0x130
    [   87.214561]  __schedule+0x230/0x664
    [   87.218045]  schedule+0x5c/0xcc
    [   87.221180]  schedule_timeout+0x84/0xfc
    [   87.225014]  rcu_gp_fqs_loop+0x12c/0x46c
    [   87.228937]  rcu_gp_kthread+0x128/0x164
    [   87.232769]  kthread+0x10c/0x110
    [   87.235996]  ret_from_fork+0x10/0x20
    [   87.239571] rcu: Stack dump where RCU GP kthread last ran:
    [   87.245052] CPU: 1 PID: 188 Comm: systemd-udevd Tainted: G           O       6.1.80+git.0d9f1748ac17 #1
    [   87.254436] Hardware name: Toradex Verdin AM62 WB on Verdin Development Board (DT)
    [   87.261995] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
    [   87.268947] pc : el1_abort+0x30/0x6c
    [   87.272522] lr : el1_abort+0x24/0x6c
    [   87.276093] sp : ffff80000998b460
    [   87.279400] x29: ffff80000998b460 x28: ffff000002083900 x27: ffff800000c421a0
    [   87.286532] x26: ffff000002645800 x25: ffff000002645a40 x24: 0000000000000000
    [   87.293664] x23: 0000000060000005 x22: ffff800000c6600c x21: ffff8000098e1040
    [   87.300795] x20: 0000000096000010 x19: ffff80000998b4a0 x18: 0000000000000000
    [   87.307925] x17: 0000000000000000 x16: 0000000000000000 x15: 0000675aa1a3cc8c
    [   87.315055] x14: 000000000000022c x13: 0000000000000000 x12: 0000000000000000
    [   87.322186] x11: 0000000000000000 x10: 00000000000009b0 x9 : ffff80000998b1e0
    [   87.329316] x8 : ffff00003fd98180 x7 : 0001001000000020 x6 : ffff000000c0d190
    [   87.336446] x5 : ffff80000998b4a0 x4 : 0000002000000000 x3 : 0000000000000025
    [   87.343577] x2 : 0000002200000000 x1 : 0000000096000010 x0 : 0000000000000000
    [   87.350708] Call trace:
    [   87.353148]  el1_abort+0x30/0x6c
    [   87.356373]  el1h_64_sync_handler+0xd8/0xe4
    [   87.360553]  el1h_64_sync+0x64/0x68
    [   87.364036]  iomap_read_reg+0xc/0x30 [m_can_platform]
    [   87.369092]  can_fill_info+0x108/0x524 [can_dev]
    [   87.373727]  rtnl_fill_ifinfo+0x844/0x11b0
    [   87.377821]  rtnl_getlink+0x23c/0x424
    [   87.381478]  rtnetlink_rcv_msg+0x130/0x3a0
    [   87.385568]  netlink_rcv_skb+0x60/0x130
    [   87.389401]  rtnetlink_rcv+0x18/0x2c
    [   87.392975]  netlink_unicast+0x2f0/0x360
    [   87.396893]  netlink_sendmsg+0x1b0/0x420
    [   87.400811]  __sys_sendto+0x138/0x174
    [   87.404471]  __arm64_sys_sendto+0x28/0x3c
    [   87.408475]  invoke_syscall+0x48/0x114
    [   87.412221]  el0_svc_common.constprop.0+0xd4/0xfc
    [   87.416919]  do_el0_svc+0x20/0x30
    [   87.420228]  el0_svc+0x28/0xa0
    [   87.423279]  el0t_64_sync_handler+0xbc/0x140
    [   87.427544]  el0t_64_sync+0x18c/0x190
    [   87.431373] Bluetooth: hci0: Opcode 0x0c03 failed: -110
    M
    [**    ] A start job is running fo[   87.449023] platform sound: deferred probe pending
    r Wait for…itialization (1min 21s / 2min 58s)
    [   87.449187] Bluetooth: hci0: Setting wake-up method failed (-22)
    [   87.477228] Internal error: synchronous external abort: 0000000096000010 [#1] PREEMPT SMP
    [   87.485471] Modules linked in: pru_rproc(+) irq_pruss_intc btnxpuart snd_soc_simple_card snd_soc_simple_card_utils crct10dif_ce pvrsrvkm(O) display_connector mwifiex_sdio mwifiex ti_k3_r5_remoteproc virtio_rpmsg_bus rpmsg_ns rtc_ti_k3 cfg80211 ti_k3_m4_remoteproc ti_k3_common bluetooth ecdh_generic ecc snd_soc_davinci_mcasp rfkill snd_soc_ti_udma snd_soc_ti_edma snd_soc_ti_sdma tidss sa2ul mcrc drm_dma_helper pruss snd_soc_nau8822 ina2xx tc358768 lontium_lt8912b drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops lm75 m_can_platform m_can pwm_tiehrpwm can_dev spi_omap2_mcspi optee_rng rng_core libcomposite fuse drm drm_panel_orientation_quirks ipv6
    [   87.543791] CPU: 1 PID: 188 Comm: systemd-udevd Tainted: G           O       6.1.80+git.0d9f1748ac17 #1
    [   87.553197] Hardware name: Toradex Verdin AM62 WB on Verdin Development Board (DT)
    [   87.560768] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
    [   87.567737] pc : iomap_read_reg+0xc/0x30 [m_can_platform]
    [   87.573169] lr : m_can_get_berr_counter+0x3c/0x10c [m_can]
    [   87.578684] sp : ffff80000998b5f0
    [   87.581998] x29: ffff80000998b5f0 x28: 0000000000000240 x27: ffff800000c421a0
    [   87.589145] x26: ffff000002645800 x25: ffff000002645a40 x24: 0000000000000000
    [   87.596294] x23: 0000000000000000 x22: ffff000000c0d010 x21: ffff000002728000
    [   87.603436] x20: ffff000002728980 x19: ffff80000998b62c x18: 0000000000000000
    [   87.610582] x17: 0000000000000000 x16: 0000000000000000 x15: 0000675aa1a3cc8c
    [   87.617729] x14: 000000000000022c x13: 0000000000000000 x12: 0000000000000000
    [   87.624871] x11: 0000000000000000 x10: 00000000000009b0 x9 : ffff80000998b1e0
    [   87.632014] x8 : ffff00003fd98180 x7 : 0001001000000020 x6 : ffff000000c0d190
    [   87.639159] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
    [   87.646304] x2 : ffff800000c66000 x1 : 0000000000000040 x0 : ffff8000098e1040
    [   87.653450] Call trace:
    [   87.655905]  iomap_read_reg+0xc/0x30 [m_can_platform]
    [   87.660983]  can_fill_info+0x108/0x524 [can_dev]
    [   87.665649]  rtnl_fill_ifinfo+0x844/0x11b0
    [   87.669763]  rtnl_getlink+0x23c/0x424
    [   87.673422]  rtnetlink_rcv_msg+0x130/0x3a0
    [   87.677513]  netlink_rcv_skb+0x60/0x130
    [   87.681349]  rtnetlink_rcv+0x18/0x2c
    [   87.684925]  netlink_unicast+0x2f0/0x360
    [   87.688843]  netlink_sendmsg+0x1b0/0x420
    [   87.692761]  __sys_sendto+0x138/0x174
    [   87.696422]  __arm64_sys_sendto+0x28/0x3c
    [   87.700427]  invoke_syscall+0x48/0x114
    [   87.704174]  el0_svc_common.constprop.0+0xd4/0xfc
    [   87.708872]  do_el0_svc+0x20/0x30
    [   87.712182]  el0_svc+0x28/0xa0
    [   87.715239]  el0t_64_sync_handler+0xbc/0x140
    [   87.719504]  el0t_64_sync+0x18c/0x190
    [   87.723177] Code: bad PC value
    [   87.726234] ---[ end trace 0000000000000000 ]---
    [   87.731027] remoteproc remoteproc0: 30074000.pru is available
    M
    [  OK  ] Listening on Load/Save RF …[   87.773378] remoteproc remoteproc1: 30078000.pru is available
    itch Status /dev/rfkill Watch.
             Starting Load/Save RF Kill Switch Status...
    [  OK  ] Created slice Slice /system/systemd-fsck.
    [  OK  ] Found device /dev/mmcblk0p1.
             Starting File System Check on /dev/mmcblk0p1...
    [  OK  ] Finished File System Check on /dev/mmcblk0p1.
             Mounting /media/BOOT-mmcblk0p1...
    [  OK  ] Mounted /media/BOOT-mmcblk0p1.
             Mounting /boot...
    [  OK  ] Mounted /boot.
    [   89.447285] Bluetooth: hci0: command 0xfc53 tx timeout
    
    

    2) You have eliminated the issues you previously had due to DDR and eMMC failures / instability 

    Correct. We integrated the latest WIP you have for the eMMC missing tuning algorithm. 

    In addition to this, I would also like to mention that the DDR and eMMC issues were affecting only a specific variant of our module, while the issue that is described here is affecting all the variants we have.

    3) What have you done to narrow the issue down further? Does every board, every variant fail. Any study done to see if the failure follows the device or the board or a mix of both? ABA swaps?

    We did run a test with the CAN interfaces disabled (`module_blacklist=m_can,m_can_platform` kernel command line arguments), and with this change, we were not able to reproduce this issue. This is a test we just executed yesterday (this is new information).

    4) What is the % failing vs passing boards

    All the boards are failing, below -30°C failure rate is at ~30% on our latest test. See attached graph.

    5) Have you reconfirmed that OPPs are correct and you are not running 1.4GHz etc at 0.75V 

    Yes, I can reconfirm this. I can also confirm that we have boards with 0.85V VDD core failing.

    6) What is the correlation of your temp chamber temperature to what on die sensors report as the die temperature

    We do not have a direct measurement of this, from some indirect measure at -40°C T air of the climate chamber, the SOC temperature sensor reports a temperature of around -32°C after boot

    7) Does the failure happen at negative temperature only - everything fine at high temp? 

    Correct, everything thing is fine above -20°C.

    8) Are the failures sensitive to a given temperature or happen when you initialize at a different temperature but do the stress test or temp chamber test at a different temperature?

    The failures happen only during boot, before any stress test can be executed. The temperature at which the failures happen is the pretty much the same temperature that was present at boot for the same reason.

    I would also ask you to have a look at my analysis in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1309832/am625-crash-internal-error-synchronous-external-abort-in-mcan-driver-at-low-temperatures-20-degrees-celsius/4978691 

    Quoted here is the relevant part:

    """

    Such issue would happen in `m_can_get_berr_counter()` when calling `__m_can_get_berr_counter()` without calling `m_can_clk_start()`, IOW without enabling the clocks. I did look at the code and I was not able to spot any bug in the m_can driver that would justify such behavior.

    ...

    We were not able to reproduce the issue running continuosly `ip -det link show can0` after the system was properly booted at room temperature.

    The only thing I was able to think is that the issue could be because `m_can_runtime_resume()` returns before the clocks are enabled, and maybe this could be related to some kind of race condition with the DM firmware that is running on the cortex R5? 

    """

  • Hi 

    Thanks for a detailed response and the additional battery of test. Much appreciated

    Here are my quick take aways ( please see if you agree)

    1) Temperature sensitivity seems to be around CAN functionality only?

    2) The failure rate is 30% so unclear whether it is the board, SOC or still any marginality in the software driver (i doubt that for now) 

    3) Have you done enough to ensure there is no marginality in functionality on DDR ( even though it looks like a CAN issue). What memory do you use on these boards?

    Is there any way to also look at doing ABA swap to see if the failure follows the device or the board?

    I will also point this thread to the MCAN driver owners to see if they any pointers from the additional information you provided in the logs and your previous posts. 

  • Additionally is there any way to look at the CAN bus activity on the pins, anyway for you to instrument that the data transfers/baud rates are all intact in working vs non working scenario?

  • 1) Temperature sensitivity seems to be around CAN functionality only?

    Yes. We did run stress test with CAN disabled and everything is running stable

    2) The failure rate is 30% so unclear whether it is the board, SOC or still any marginality in the software driver (i doubt that for now) 

    My suggestion/hypothesis already shared in the original e2e thread is that the clock takes some time from when the driver enables it to the time is enabled/stable.

    3) Have you done enough to ensure there is no marginality in functionality on DDR ( even though it looks like a CAN issue). What memory do you use on these boards?

    Yes, all the extensive memory test we did run so far are passing. We have multiple memory used (Micron, Nanya in various size and temperature range variants).

    Is there any way to also look at doing ABA swap to see if the failure follows the device or the board?

    All the board we have are affected, no matter which memory, eMMC, VDD_Core or SOC variant we use. We have lot of combinations and they all fail.

    Additionally is there any way to look at the CAN bus activity on the pins, anyway for you to instrument that the data transfers/baud rates are all intact in working vs non working scenario?

    The crash is happening at probe time. There is nothing going on on the CAN pins at that time. This is way before any baudrate is configured.

  • I did click by mistake on the "This resolved my issue" button, but this is not the case. maybe you can correct this and mark it again as not solved.

  • I will also point this thread to the MCAN driver owners to see if they any pointers from the additional information you provided in the logs and your previous posts. 

    it seems more of an issue on the CAN clock than on the actual MCAN driver, at least from the kind of crash we are experiencing. 

  • Yes I was going to ask if you can also please look at your CAN transceiver temperature  specifications etc. I agree this is leaning to a hardware like problem, I will bring this to attention of the hardware experts too. 

  • Yes I was going to ask if you can also please look at your CAN transceiver temperature  specifications etc.

    Yes, the issue is just going to be the same even without a transceiver, you do not need a transceiver at all to see this issue.

    I wonder if you could reproduce the issue by enabling both the MAIN CAN0 and the MCU CAN0 on the TI AM62 SK board. I do not know if the SK board is fine at -30 degrees Celsius, however.

  • Have you tried loop back test to rule out external component dependencies? Our EVM is not rated to work at -30 C, it is for functional evaluation of software and features  not a reference design to work across -40 to 125C. 

    We need your help to peel the onion further. Is ABA test planned?

  • Have you tried loop back test to rule out external component dependencies?

    What do you mean exactly here? All of the components we use are rated down to -40 Celsius degrees, if you have any specific concerns about some components we can try to do some specific checks on those.

    As I already mentioned the issue happens when the MCAN driver is probed, we are not using CAN in those failing tests.

    We need your help to peel the onion further. Is ABA test planned?

    I am not understanding what you are asking exactly, but we are here to help get to a solution together. What do you want to be tested exactly?

  • sorry i am reading this fast and I guess ABA swap does not make sense as you are saying 100% boards fail, but failure rate is 30% per board. 

    We may also need to focus on what you pointed to earlier  `m_can_runtime_resume()` returns before the clocks are enabled, and maybe this could be related to some kind of race condition with the SYSFW clock enable/disable? Did you try adding any wait states anywhere to see if that helps?  

  • We may also need to focus on what you pointed to earlier  `m_can_runtime_resume()` returns before the clocks are enabled, and maybe this could be related to some kind of race condition with the SYSFW clock enable/disable? Did you try adding any wait states anywhere to see if that helps?  

    Yes, after mentioning this to you we decided to prepare a test with some delay added, we'll have results in one or two days.

  • Some updates

    Yes, after mentioning this to you we decided to prepare a test with some delay added, we'll have results in one or two days.

    Adding a 10ms delay after the CAN clock is enabled seems to have a positive effect, occurrences rate is reduced from 30% to 12%. We are going to increase the delay and re-execute the test. These test results while promising are not yet conclusive.

    In addition to that, we ran one more test with a different set of boards (with a different hardware configuration), and also there, after disabling the CAN interfaces (`module_blacklist=m_can,m_can_platform`), the boards are running stable across the whole temperature range. 

    If you have any suggestions to narrow down the issue let me know.

  • Hi Fd,

    Could you test the following, try disabling main_mcan0 and enable only mcu_mcan0. Then test to see if this still gives the boot failure. Then disable  mcu_mcan0 and enable only main_mcan0 and compare results.

    ~ Judith

  • Hello ,

    Could you test the following, try disabling main_mcan0 and enable only mcu_mcan0. Then test to see if this still gives the boot failure. Then disable  mcu_mcan0 and enable only main_mcan0 and compare results.

    We'll test this tomorrow, however, I have little expectation on this to provide any useful feedback. The issue here has been present since a long time, before the actual Linux m_can driver was extended to enable interrupt-less MCAN devices as required on the MCU_MCAN[01] in the TI AM62 SOC.

  • Hi fd,

    Understood, then it is unlikely that the timer polling support is the issue here, still please test this once.

    Thank you,

    ~ Judith

  • Adding a 10ms delay after the CAN clock is enabled seems to have a positive effect, occurrences rate is reduced from 30% to 12%. We are going to increase the delay and re-execute the test. These test results while promising are not yet conclusive.

    Along these lines I'd also try more drastic measures as this, removing suspend/resume PM....

    $ git diff
    diff --git a/drivers/net/can/m_can/m_can_platform.c b/drivers/net/can/m_can/m_can_platform.c
    index 01c363d20d3f..28a4d5107f39 100644
    --- a/drivers/net/can/m_can/m_can_platform.c
    +++ b/drivers/net/can/m_can/m_can_platform.c
    @@ -220,7 +220,6 @@ static int __maybe_unused m_can_runtime_resume(struct device *dev)
     static const struct dev_pm_ops m_can_pmops = {
            SET_RUNTIME_PM_OPS(m_can_runtime_suspend,
                               m_can_runtime_resume, NULL)
    -       SET_SYSTEM_SLEEP_PM_OPS(m_can_suspend, m_can_resume)
     };
    
     static const struct of_device_id m_can_of_table[] = {

    ...and if this doesn't help even more drastic as this, removing all PM...

    $ git diff
    diff --git a/drivers/net/can/m_can/m_can_platform.c b/drivers/net/can/m_can/m_can_platform.c
    index 01c363d20d3f..622c90c1218f 100644
    --- a/drivers/net/can/m_can/m_can_platform.c
    +++ b/drivers/net/can/m_can/m_can_platform.c
    @@ -233,7 +232,6 @@ static struct platform_driver m_can_plat_driver = {
            .driver = {
                    .name = KBUILD_MODNAME,
                    .of_match_table = m_can_of_table,
    -               .pm     = &m_can_pmops,
            },
            .probe = m_can_plat_probe,
            .remove = m_can_plat_remove,

    Regards, Andreas

  • Along these lines I'd also try more drastic measures as this, removing suspend/resume PM....

    I do not think that this makes any difference. The issue happens during the drive probe(), and the probe is way earlier than any actual suspend/resume is coming into play.

    Or am I missing something?

  • We did run a test with a delay of 50ms after enabling the CAN clocks and with such a change the system is running stable with no failures.

    --- a/drivers/net/can/m_can/m_can_platform.c
    +++ b/drivers/net/can/m_can/m_can_platform.c
    @@ -5,6 +5,7 @@
     //
     // Copyright (C) 2018-19 Texas Instruments Incorporated - http://www.ti.com/
     
    +#include <linux/delay.h>
     #include <linux/hrtimer.h>
     #include <linux/platform_device.h>
     #include <linux/phy/phy.h>
    @@ -213,7 +214,9 @@ static int __maybe_unused m_can_runtime_resume(struct device *dev)
            err = clk_prepare_enable(mcan_class->cclk);
            if (err)
                    clk_disable_unprepare(mcan_class->hclk);
    -
    +       printk("delay 50ms....");
    +       mdelay(50);
    +       printk("end delay....");
            return err;
     }

  • Hi fd,

    We did run a test with a delay of 50ms after enabling the CAN clocks and with such a change the system is running stable with no failures.

    That's great, thanks for confirming. That also means removing some of that PM code altogether (earlier suggested experiment) would most likely also work around the issue, but probably in a cleaner way, until we understand the root cause and validate that a delay is indeed a valid fix (and if so, how long).

    Now we need to figure out on our end here what's really causing this. I'd think the clock enable etc. calls would be synchronous, and only return once the clocks are enabled and settled, something that needs to be confirmed to see if this is at play here. Or something else.

    Regards, Andreas

  • I'd think the clock enable etc. calls would be synchronous

    Yes, I can confirm that the Linux clock API is supposed to be synchronous.

    Now we need to figure out on our end here what's really causing this

    Let me know if and how we can help here.

    As a minimum, we do still plan to run the test that suggested, maybe the issue is specific to one of the two (main/mcu) domains.

  • Yes, I can confirm that the Linux clock API is supposed to be synchronous.

    Those calls should propagate all the way down to the device manager (DM) firmware, that controls the clock tree/muxes/PLLs, etc. So at the end it'll depend on what this firmware is doing, so I'll be having a look at that piece. I don't expect any surprises there really as for example the PLL code always waits for "lock" before proceeding but it doesn't hurt to double check. Of course it could be something else too, perhaps a timing dependency that exhibits itself because of a certain clock ratio (CAN speed / clocks vs system clock, etc.) in your specific setup. It'll require a bit of detective work.

    Regards, Andreas

  • just 2 more cents on this discussion. with this additional delay, we are completely delaying the m_can initialization. Maybe we should keep the option open that something else is not ready yet (power?).

  • (sorry I accidentally marked the above post as 'answer', which is not something I can undo)

    Maybe we should keep the option open that something else is not ready yet (power?).

    In case you haven't tried this (I don't have all the history here), can you load the CAN kernel module at runtime (using modprobe), to see what this does?

    Regards, Andreas

  • In case you haven't tried this (I don't have all the history here), can you load the CAN kernel module at runtime (using modprobe), to see what this does?

    I was thinking more at the power-domains, e.g. for main_mcan0 `power-domains = <&k3_pds 98 TI_SCI_PD_EXCLUSIVE>`. Maybe the clock is ready, but the domain is not powered yet. And with this delay, we might be implicitly working around this issue. Loading the module later will not change this.

    With that said we have not tried the specific test you mentioned, it's not really trivial to set this up since the kernel just autoloads the module.

    From the timing point of view, the module loading is already happening relatively late in the boot phase, with the userspace systemd init sequence going on, more than 5 seconds into the boot.

    If you still think this is an important test to do we can try to figure out some way to test it.

  • One way to confirm this weird hypothesis would be to add the delay _before_ enabling the clock, not after. We'll try to test this.

  • All this could also be caused by some general race condition outside the actual clock and power domain enablement itself. From the exception log it looks like netlink drivers are trying to access a CAN register which isn't ready yet, which usually means due to clock or power domain concerns.

    In this context one experiment to do would be to limit the system to only use one CPU, this way things are happen in a more "orderly" manner. You can do this as follows:

    # In U-Boot, populate an extra Kernel command line parameter
    => env set optargs maxcpus=1
    => env print optargs
    optargs=maxcpus=1
    => boot
    
    <...snip...>
    
    # Check to make sure that parameter made it to the Kernel...
    # (yes, it did)
    [    0.000000] Kernel command line: console=ttyS2,115200n8 maxcpus=1 earlycon=ns16550a,mmio32,0x02800000 mtdparts=spi-nand0:512k(ospi_nand.tiboot3),2m(ospi_nand.tispl),4m(ospi_nand.u-boot),256k(ospi_nand.env),25
    6k(ospi_nand.env.backup),98048k@32m(ospi_nand.rootfs),256k@130816k(ospi_nand.phypattern) root=PARTUUID=eb24392b-02 rw rootfstype=ext4 rootwait
    
    <...snip...>
    
    # Finally, also double-check at Linux prompt...
    root@am62pxx-evm:~# nproc
    1

    Regards, Andreas

  • In this context one experiment to do would be to limit the system to only use one CPU,

    We reproduced this issue already with single-core AM6231 SOC, therefore I do not think this test is required.

  • Some updates after executing more tests.

    Could you test the following, try disabling main_mcan0 and enable only mcu_mcan0. Then test to see if this still gives the boot failure. Then disable  mcu_mcan0 and enable only main_mcan0 and compare results.

    : we tested this and the behavior is the same as before. Whenever any of the CAN interfaces (main OR mcu domain) is enabled the issue is present.

    One way to confirm this weird hypothesis would be to add the delay _before_ enabling the clock, not after. We'll try to test this.

    We tested this, adding a 50ms delay _before_ enabling the CAN clocks solves the issue. So it seems that something needs some delays, but this something is not the clock. One option would be the power domain of the CAN interface, but maybe you have other ideas to explore.

    Just a reminder, this issue is happening only below -20 degrees Celsius (or at least, we were not able to reproduce it in any other condition), so IMO we should be focusing on something timing-related that could be slower when the temperature is "low".

    cc: 

  • Hi fd,

    Thanks for confirming.

    ~ Judith

  • Hi Fd
    Is it possible for you to get us a register dump for PLL (main PLL0 sourcing the clocks to CAN) , CAN and PSC registers for a failing vs passing use-case. 

    Regards

    Mukul 

  • Hello

    Is it possible for you to get us a register dump for PLL (main PLL0 sourcing the clocks to CAN) , CAN and PSC registers for a failing vs passing use-case. 

    I'm not sure how to achieve this. When it fails the system crashes, so there is not much way to execute anything afterward.

    With that said, in case I am misunderstanding your request, share a code snippet/patch that would print this information on the console with printk() and we can apply the patch and run it on our test bench. If you are thinking at using a debugger this is a setup we have not easily available (we would need to investigate some options, but I cannot see any simple solution).

  • Hi Fd,

    I will prepare a kernel patch to dump the necessary registers. I am currently working on another customer issues in parallel, so the progress might be a bit slower here.

  • Fd

    Sorry a few more suggestion talking to one of the software developers focussed on LPM

    Can we try to set inactivity timeout to -1. That'll disable runtime PM

    Alternatively,  try increasing the inactivity timeout to 50ms, rather than the delay. It would ensure that driver doesn't suspend until after 50ms from last activity.

    Again these are just experiments , if easier for you to quickly try, that may continue to help us narrow down the issue.

    I am also evaluating to see if we can just do a quick test on our EVM under thermostream  at negative temperature , but may take a couple of days. 

  • One more update from our last test execution, we did run a test with Linux Kernel v6.9.0 and the issue is still present.

  • One more update from our last test execution, we did run a test with Linux Kernel v6.9.0 and the issue is still present.

    Great, thanks for doing this test.

    Also quick update from my side. End of last week I was trying to re-create the issue by doing some modifications to the drivers with the hopes to make it more likely to fail (at least so I thought) but wasn't able to make much progress. I'm on business travel this week so I will not be able to debug this further but I will be able to prioritize this again starting Monday next week.

    Regards, Andreas

  • Hi Fd,

    Before we dump the PLL registers, can you please change the MCAN refclk parent in DTS to see if the crash still happens without the 50ms delay?

    Here is the reference of DTS change for MCAN refclk:

    diff --git a/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi b/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi
    index 90657f5f532c..fd7b2fc4401a 100644
    --- a/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi
    +++ b/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi
    @@ -614,3 +614,9 @@ &mcu_gpio0 {
     &mcu_gpio_intr {
            status = "reserved";
     };
    +
    +&main_mcan0 {
    +       assigned-clocks = <&k3_clks 98 1>;
    +       assigned-clock-parents = <&k3_clks 98 5>;
    +       status = "okay";
    +};

  • The MCAN refclk parent 5 frequency is 25MHz, (the default parent is 80MHz), so I am not sure if the MCAN0 controller can still talk to the CAN devices on the bus, but I just want to see if the crash happens or not during MCAN initialization with different refclk parent.

  • Additional update, we did a quick test on a single EVM, running a thermostream at -40C on the DUT,  10 out 10 boot /reboot did not show any failures with the CAN. In coming days we may do another run with some automation and on die sensor report out on temperature, but initial results don't match your 30% failure. 

    Have you compared and contrasted the differences on the board/software that maybe different between EVM vs your boards ?

  • Hello Mukul,

    Additional update, we did a quick test on a single EVM, running a thermostream at -40C on the DUT,  10 out 10 boot /reboot did not show any failures with the CAN

    Normally we run the test for at least 1 hour with 7 boards in parallel. Maybe the 30% reproducibility rate is not equally distributed and just 10 reboots might not be enough. can you add more details here?

    With that said I realized that the issue is not happening during the driver probe, but probably the first time that systemd is doing some kind of enumeration on the network devices. I assume this to be the same on your software, but I might be wrong (and of course, you need to enable the MCAN devices in the DT, but this you know, just writing it since the issue is tricky and it's better to be sure we are not getting lost on some stupid detail).

    Have you compared and contrasted the differences on the board/software that maybe different between EVM vs your boards ?

    On the Linux Kernel sources side, there is no real significant difference, and the issue was reproduced with a vanilla Linux 6.9.0 without any software change at all. Of course, the DTS file is different, and the kernel config is also slightly different (some driver is enabled) and the userspace init is also different. Overall nothing specific that I would look into.

    We did a lot of verification on the clock and power of the board and we have not seen anything anomalous there. If there is any specific signal (power, clock, whatever) that might specifically affect the CAN device just let me know and we'll double-check it. I'd like to write once more that as soon as we disable the CAN everything is running rock solid.

    We'll have soon some updates on the test you and Bin suggested.

  • Normally we run the test for at least 1 hour with 7 boards in parallel. Maybe the 30% reproducibility rate is not equally distributed and just 10 reboots might not be enough. can you add more details here?

    The 30% is the average failure rate over the 7 boards combined. It's not equally distributed. Some of the modules show the problem less, others more often. The tests were normally run for +18h to have enough boot cycles (~1000) to get representative results. Only on the latest test with only mcu_mcan or main_mcan enabled, or with Linux 6.9.0 the test time was shorter because the problem was there from the beginning, and so there was no point in running it longer.

    Expect the results of the proposed test by Bin today or tomorrow at the latest.

  • Do you have the ability to enable the on die temperature sensor to see what is the corresponding SOC temperature relative to your temperature chamber

    your older posts had the following 

    Climate Chamber: KWP 120
    Start Temperature: -40.3�C
    End Temperature: -39.8�C
    Measured Temperature: -40.3�C
    Shelf Temperature: -39.5�C
    Shelf Name: Shelf_19
    Module Serial Number: 15313474
    Module Location: Shelf 1, Carrier Board 5
    Test duration: 127.82s

    Is the issue reproducible with warm reset or do you power cycle the board?

    we ran another test with about 50 reboots and yet no failure, of course single EVM. Ice formation on the board , so not planning to run long hour test yet. 

  • Hi Bin,

    I have made the changes requested in our DTS and verified the changes in /sys/kernel/debug/clk/clk_summary.

     clk:98:5                             0        0        0    25000000          0     0  50000         Y
        clk:98:1                          0        0        0    25000000          0     0  50000         Y

    To be clear, I also removed mcu_mcan0 from the DTS to focus on main_mcan0, as you only send changes for main_mcan.

    The problem still exists and occurs at the same point. The failure rate was lower because the problem only occurred on one of the seven modules in this test. I believe that the fact that only one of the two CAN interfaces was available also makes a difference to the failure rate.

    Do you have the ability to enable the on die temperature sensor to see what is the corresponding SOC temperature relative to your temperature chamber

    your older posts had the following 

    Climate Chamber: KWP 120
    Start Temperature: -40.3�C
    End Temperature: -39.8�C
    Measured Temperature: -40.3�C
    Shelf Temperature: -39.5�C
    Shelf Name: Shelf_19
    Module Serial Number: 15313474
    Module Location: Shelf 1, Carrier Board 5
    Test duration: 127.82s

    We don't read the temperature when the problem occurs, only after a successful boot. From the cycles before and after a crash, the SoC temperature sensors report a temperature between -26°C and -32°C (different from module to module and also depending on how long we let the modules cool down after a cycle). The temperatures you have shown are all reference temperatures in our climate chamber.

    Is the issue reproducible with warm reset or do you power cycle the board?

    We do power cycle.

    Can we try to set inactivity timeout to -1. That'll disable runtime PM

    Just to double check on this one. From what I've read here, the autosuspend_delay_ms needs therefore to be set to -1. Is that correct? Just to check that I test the correct settings.


    diff --git a/drivers/net/can/m_can/m_can_platform.c b/drivers/net/can/m_can/m_can_platform.c
    index 01c363d20d3f..91a20eb3d9c2 100644
    --- a/drivers/net/can/m_can/m_can_platform.c
    +++ b/drivers/net/can/m_can/m_can_platform.c
    @@ -154,6 +154,9 @@ static int m_can_plat_probe(struct platform_device *pdev)
            platform_set_drvdata(pdev, mcan_class);
     
            pm_runtime_enable(mcan_class->dev);
    +       printk("Disabling PM Runtime");
    +       pm_runtime_set_autosuspend_delay(mcan_class->dev, -1);
    +       pm_runtime_use_autosuspend(mcan_class->dev);
            ret = m_can_class_register(mcan_class);
            if (ret)
                    goto out_runtime_disable;
    

    root@verdin-am62-15355727:~# cat /sys/devices/platform/bus\@f0000/20701000.can/power/*
    -1
    auto
    351651
    active
    0
    

    Thanks,

    Franz

  • I found this rcu_preempt call trace in the logs from the last test, if it's of any help. It managed to recover and boot, rather than just crashing as usual (something that has been seen in the past, but is rare for it to recover).

    [    5.419275] CAN device driver interface
    [  OK  ] Found device /dev/ttyS2.
    [    5.518926] m_can_platform 20701000.can: m_can device registered (irq=490, version=32)
    [    5.675688] Bluetooth: Core ver 2.22
    [    5.706332] NET: Registered PF_BLUETOOTH protocol family
    [    5.724814] Bluetooth: HCI device and connection manager initialized
    [    5.780995] k3-m4-rproc 5000000.m4fss: device does not reserved memory regions, ret = -22
    [    5.839607] Bluetooth: HCI socket layer initialized
    [    5.846090] systemd-journald[166]: Oldest entry in /run/log/journal/d00310a14e344f619c21685c1baa4fef/system.journal is older than the configured file retention duration (1month), suggesting rotation.
    [    5.851086] k3-m4-rproc 5000000.m4fss: reserved memory init failed, ret = -22
    [    5.865163] Bluetooth: L2CAP socket layer initialized
    [  OK  ] Started Network Time Synchronizatio[    5.880191] systemd-journald[166]: /run/log/journal/d00310a14e344f619c21685c1baa4fef/system.journal: Journal header limits reached or header out-of-date, rotating.
    n.
    [  OK  ] Reached target System Time Set.
    [    5.928415] systemd-journald[166]: Oldest entry in /run/log/journal/d00310a14e344f619c21685c1baa4fef/system.journal is older than the configured file retention duration (1month), suggesting rotation.
    [    5.955110] Bluetooth: SCO socket layer initialized
    [    5.967373] systemd-journald[166]: /run/log/journal/d00310a14e344f619c21685c1baa4fef/system.journal: Journal header limits reached or header out-of-date, rotating.
    [    5.982222] remoteproc remoteproc0: releasing 5000000.m4fss
    [    5.982264] k3-m4-rproc: probe of 5000000.m4fss failed with error -22
    [    6.132507] cfg80211: Loading compiled-in X.509 certificates for regulatory database
    [    6.142535] rtc-ti-k3 2b1f0000.rtc: registered as rtc1
    [    6.152875] platform 78000000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1)
    [    6.190091] platform 78000000.r5f: configured R5F for IPC-only mode
    [    6.197583] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
    [    6.204401] platform 78000000.r5f: device does not have reserved memory regions, ret = -22
    [    6.213720] cfg80211: Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
    [    6.222416] k3_r5_rproc bus@f0000:bus@b00000:r5fss@78000000: reserved memory init failed, ret = -22
    [    6.232111] remoteproc remoteproc0: releasing 78000000.r5f
    [    6.238979] k3_r5_rproc bus@f0000:bus@b00000:r5fss@78000000: k3_r5_cluster_rproc_init failed, ret = -22
    [    6.249375] k3_r5_rproc: probe of bus@f0000:bus@b00000:r5fss@78000000 failed with error -22
    [    6.283486] pvrsrvkm: loading out-of-tree module taints kernel.
    [    6.342544] lt8912 1-0048: supply vdd not found, using dummy regulator
    [    6.351699] lt8912 1-0048: supply vccmipirx not found, using dummy regulator
    [    6.361915] lt8912 1-0048: supply vccsysclk not found, using dummy regulator
    [    6.373716] lt8912 1-0048: supply vcclvdstx not found, using dummy regulator
    [    6.385568] PVR_K:  188: Device: fd00000.gpu
    [    6.390439] lt8912 1-0048: supply vcchdmitx not found, using dummy regulator
    [    6.398947] lt8912 1-0048: supply vcclvdspll not found, using dummy regulator
    [    6.415096] PVR_K:  188: Read BVNC 33.15.11.3 from HW device registers
    [    6.422348] lt8912 1-0048: supply vcchdmipll not found, using dummy regulator
    [    6.436695] PVR_K:  188: RGX Device registered with BVNC 33.15.11.3
    [    6.475119] [drm] Initialized tidss 1.0.0 20180215 for 30200000.dss on minor 1
    [    6.482969] [drm] Initialized pvr 23.3.6512818 20170530 for fd00000.gpu on minor 0
    [    6.484460] tidss 30200000.dss: [drm] Cannot find any crtc or sizes
    [    6.500017] tidss 30200000.dss: [drm] Cannot find any crtc or sizes
    [    6.786402] mwifiex_sdio mmc2:0001:1: info: FW download over, size 570976 bytes
    [    8.199307] mwifiex_sdio mmc2:0001:1: WLAN FW is active
    [    8.240413] mwifiex_sdio mmc2:0001:1: Unknown api_id: 5
    [*     ] A start job is running for Wait for…ice Initialization (2s / 2min 58s)
    M
    [  OK  ] Listening on Load/Save RF …itch Status /dev/rfkill Watch.
    [    8.281250] mwifiex_sdio mmc2:0001:1: info: MWIFIEX VERSION: mwifiex 1.0 (16.92.21.p84) 
    [    8.293371] mwifiex_sdio mmc2:0001:1: driver_version = mwifiex 1.0 (16.92.21.p84) 
             Starting Load/Save RF Kill Switch Status...
    [   29.339185] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
    [   58.650207] remoteproc remoteproc0: 30074000.pru is available
    [   58.653103] rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-1): P184/1:b..l
    [   58.653124] 	(detected by 0, t=12581 jiffies, g=1753, q=434 ncpus=2)
    [   58.653134] task:systemd-udevd   state:R  running task     stack:0     pid:184   ppid:1      flags:0x00000a04
    [   58.653147] Call trace:
    [   58.653151]  __switch_to+0xd4/0x130
    [   58.653172]  __schedule+0x230/0x664
    [   58.691519]  preempt_schedule+0x4c/0x70
    [   58.691526]  _raw_spin_unlock_irqrestore+0x30/0x50
    [   58.691536]  __wake_up_common_lock+0x90/0xd4
    [   58.691546]  __wake_up_sync_key+0x20/0x30
    [   58.691553]  sock_def_readable+0x40/0x80
    [   58.691565]  __netlink_sendskb+0x6c/0xa0
    [   58.691576]  netlink_unicast+0x138/0x360
    [   58.691582]  netlink_sendmsg+0x1b0/0x420
    [   58.691589]  ____sys_sendmsg+0x228/0x290
    [   58.691596]  ___sys_sendmsg+0x84/0xe4
    [   58.691603]  __sys_sendmsg+0x6c/0xd0
    [   58.691609]  __arm64_sys_sendmsg+0x24/0x30
    [   58.691617]  invoke_syscall+0x48/0x114
    [   58.691627]  el0_svc_common.constprop.0+0xd4/0xfc
    [   58.691634]  do_el0_svc+0x20/0x30
    [   58.691640]  el0_svc+0x28/0xa0
    [   58.691650]  el0t_64_sync_handler+0xbc/0x140
    [   58.691658]  el0t_64_sync+0x18c/0x190
    [**    ] (1 of 2) A start job is running for… RF Kill Switch Status (50s / 30s)
    [   58.778725] remoteproc remoteproc1: 30078000.pru is available
    M
    [FAILED] Failed to start Load/Save R[   58.797733] platform sound: deferred probe pending
    F Kill Switch Status.
    See 'systemctl status systemd-rfkill.service' for details.
             Starting Load/Save RF Kill Switch Status...
    [  OK  ] Started Load/Save RF Kill Switch Status.
    [  OK  ] Created slice Slice /system/systemd-fsck.
    [  OK  ] Found device /dev/mmcblk0p1.
    [  OK  ] Finished Wait for udev To Complete Device Initialization.

  • Just to double check on this one. From what I've read here, the autosuspend_delay_ms needs therefore to be set to -1. Is that correct? Just to check that I test the correct settings

    Yes this is the correct settings. 

    Is your last post where you are able to recover from crash, done with Bin's changes or also with runtime suspend disable?

  • Is your last post where you are able to recover from crash, done with Bin's changes or also with runtime suspend disable?

    The log from my last post was only with Bin's changes.

    In the meantime, I've tested with runtime suspend disabled, both CAN interfaces enabled again in the DTS of course, and it hasn't shown any problems yet.