This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM6442: Booting issue

Part Number: AM6442


Hi,

In our boot sequence the R5 starts working before the A53 and the images are stored in the EMMC.

We noticed  that when the A53 completes its startup actions it stuck and it’s impossible communicate with it.

 

When the A53 loads the R5 FW and the images are stored in the SD card there is no issue.

  • Hello Eyal,
    Will you provide more info:
    - SDK used?
    - HW borad info?
    - Any logs?

    Some good reference on AM64x Linux SDK
    - AM64x Linux SDK
    hwww.ti.com/.../PROCESSOR-SDK-LINUX-AM64X
    - AM64x Linux SDK user guide
    software-dl.ti.com/.../Overview.html

    Best,
    -Hong

  • Hi,

    SDK - 08_02_00_31

    HW board – AM64 EVB like the broken devices we sent you

    Log - attached

  • Hello Eyal,
    From the log, it seems like exception was correlated with rproc process.
    I'm passing it to my colleague for his comments.
    [ 10.318128] Call trace:
    [ 10.320315] rproc_handle_resources.constprop.0+0x94/0x168
    [ 10.325196] rproc_boot+0x334/0x670
    [ 10.328302] rproc_add+0x88/0x158
    [ 10.331263] k3_r5_probe+0x698/0xcf0 [ti_k3_r5_remoteproc]
    Best,
    -Hong

  • Hello Eyal,

    I assume the customer is using SBL EMMC boot as described here:
    https://software-dl.ti.com/mcu-plus-sdk/esd/AM64X/08_02_00_31/exports/docs/api_guide_am64x/EXAMPLES_DRIVERS_SBL_EMMC_LINUX.html

    It looks like SBL is trying (and failing) to initialize non-Linux cores:

    Boot-mode enabled, Entering Bootloading process...
    
    R5_0_0 Validation
    Booting application...
    
    R5_0_1 Validation
    Failed to boot application. No valid Firmware exist.
    
    R5_1_0 Validation
    Failed to boot application. No valid Firmware exist.
    
    R5_1_1 Validation
    Failed to boot application. No valid Firmware exist.
    
    Cortex M4 Validation
    Failed to boot application. No valid Firmware exist.
    
    SBL image Validation
    Loading images.......
    Image loading done, switching to application ...
    Starting linux and RTOS/Baremetal applications

    My guess is that this puts the remote cores into an unknown state. And then when Linux tries to initialize the remote cores, something bad happens:

    [    8.911094] platform 78000000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1)
    [    9.103297] k3-m4-rproc 5000000.m4fss: assigned reserved memory node m4f-dma-memory@a4000000
    [    9.115008] platform 78000000.r5f: configured R5F for IPC-only mode
    [    9.151503] k3-m4-rproc 5000000.m4fss: configured M4 for IPC-only mode
    [    9.157596] remoteproc remoteproc1: 5000000.m4fss is available
    [    9.163207] remoteproc remoteproc1: attaching to 5000000.m4fss
    [    9.174588] k3-m4-rproc 5000000.m4fss: M4 initialized in IPC-only mode
    [    9.180010] platform 78000000.r5f: assigned reserved memory node r5f-dma-memory@a0000000
    [    9.193592] remoteproc remoteproc1: remote processor 5000000.m4fss is now attached
    [    9.225179] remoteproc remoteproc0: 78000000.r5f is available
    [    9.245134] remoteproc remoteproc0: attaching to 78000000.r5f
    [    9.255314] platform 78000000.r5f: R5F core initialized in IPC-only mode
    [    9.268563]  remoteproc0#vdev0buffer: assigned reserved memory node r5f-dma-memory@a0000000
    [    9.281633] virtio_rpmsg_bus virtio0: rpmsg host is online
    ...
    [    9.376257]  remoteproc0#vdev0buffer: registered virtio0 (type 7)
    [    9.398280] remoteproc remoteproc0: remote processor 78000000.r5f is now attached
    [    9.415661] platform 78200000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1)
    [    9.430104] TI DP83869 0.1:03: attached PHY driver [TI DP83869] (mii_bus:phy_addr=0.1:03, irq=POLL)
    [    9.445232] am65-cpsw-nuss 8000000.ethernet eth1: Link is Down
    [    9.483186] platform 78200000.r5f: configured R5F for IPC-only mode
    [    9.513610] m_can_platform 20701000.can: m_can device registered (irq=36, version=32)
    [    9.529387] m_can_platform 20711000.can: m_can device registered (irq=38, version=32)
    [    9.551206] platform 78200000.r5f: assigned reserved memory node r5f-dma-memory@a1000000
    [    9.636196] remoteproc remoteproc2: 78200000.r5f is available
    [    9.689570] remoteproc remoteproc2: attaching to 78200000.r5f

    My guess is that the SBL should either
    1) initialize the remote cores (which requires the multicore app images to be bundled together as per the SBL EMMC boot documentation), or
    2) not try to initialize the remote cores to begin with.

    Is customer even using the remote cores? If they are not using the remote cores, they can disable the remote cores in Linux by following this FAQ:
    https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1195052/faq-am62x-am64x-how-to-disable-unused-cores

    Regards,

    Nick

  • Hello Prashant,

    Reassigning to you to comment on SBL.

    1) Can we configure SBL so that it does not try (and fail) to initialize certain remote cores? If so, how do you do that?

    2) Does this behavior look familiar to you?

    Thanks,

    Nick

  • Hi Nick & EyalCo,

    Processor SDK does have a nice way of disabling cores. However, I wish I could say the same for MCU PLUS SDK. Usually, the SBLs in MCU PLUS SDK initializes a core only if there is a RPRC for the respective core in the Multicore Appimage. But, it seems SBL_EMMC_LINUX isn't following this. SBL_EMMC_LINUX initializes each core irrespective of Multicore Appimage. If it does not find RPRC for a core, then it resets the core to the default entry point.

    Disabling a core in SBL

    SBL first loads RPRC for a core & then run it. It does so by calling the APIs Bootloader_loadCpuBootloader_runCpu respectively. Commenting or deleting the calls to these APIs for a core is equivalent to leaving the core untouched.

  • About the issue, I also see the same Call Traces as contained in the logs shared by the customer. However, I am still able to interact with Linux on UART.

    In any case, disabling the remote cores in Linux by following the FAQ shared by Nick and then replacing the .dtb file on eMMC with the updated .dtb file makes Linux not booting Remote Cores and also not reporting any Call Trace logs.

    Attaching logs for reference:

    DMSC Firmware Version 8.4.7--v08.04.07 (Jolly Jellyfi
    DMSC Firmware revision 0x8
    DMSC ABI revision 3.1
    
    [BOOTLOADER_PROFILE] Boot Media       : undefined
    [BOOTLOADER_PROFILE] Boot Image Size  : 0 KB
    [BOOTLOADER_PROFILE] Cores present    :
    [BOOTLOADER PROFILE] System_init                      :      32363us
    [BOOTLOADER PROFILE] Drivers_open                     :     803389us
    [BOOTLOADER PROFILE] Board_driversOpen                :          0us
    [BOOTLOADER PROFILE] App_loadImages                   :     107225us
    [BOOTLOADER_PROFILE] SBL Total Time Taken             :    1107350us
    
    Image loading done, switching to application ...
    Starting linux and RTOS/Baremetal applications
    
    NOTICE:  BL31: v2.5(release):08.03.00.002-dirty
    NOTICE:  BL31: Built : 17:38:37, May  5 2022
    
    U-Boot SPL 2021.01-g44a87e3ab8 (May 05 2022 - 17:42:22 +0000)
    SYSFW ABI: 3.1 (firmware rev 0x0008 '8.4.7--v08.04.07 (Jolly Jellyfi')
    Trying to boot from MMC1
    
    
    U-Boot 2021.01-g44a87e3ab8 (May 05 2022 - 17:42:22 +0000)
    
    SoC:   AM64X SR1.0
    Model: Texas Instruments AM642 EVM
    Board: AM64-GPEVM rev E2
    DRAM:  2 GiB
    NAND:  0 MiB
    MMC:   mmc@fa10000: 0, mmc@fa00000: 1
    In:    serial@2800000
    Out:   serial@2800000
    Err:   serial@2800000
    Net:   eth0: ethernet@8000000port@1
    Hit any key to stop autoboot:  0
    => setenv mmcdev 0
    => setenv bootpart 0:1
    => boot
    switch to partitions #0, OK
    mmc0(part 0) is current device
    SD/MMC found on device 0
    Failed to load 'boot.scr'
    ** Unrecognized filesystem type **
    19276288 bytes read in 104 ms (176.8 MiB/s)
    41599 bytes read in 3 ms (13.2 MiB/s)
    ## Flattened Device Tree blob at 88000000
       Booting using the fdt blob at 0x88000000
       Loading Device Tree to 000000008fff2000, end 000000008ffff27e ... OK
    
    Starting kernel ...
    
    [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
    [    0.000000] Linux version 5.10.100-g7a7a3af903 (oe-user@oe-host) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 9.2.1 20191025, GNU ld (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 2.33.1.20191209) #1 SMP PREEMPT Thu May 5 17:46:38 UTC 2022
    [    0.000000] Machine model: Texas Instruments AM642 EVM
    [    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
    [    0.000000] printk: bootconsole [ns16550a0] enabled
    [    0.000000] efi: UEFI not found.
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a0000000, size 1 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a0000000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a0100000, size 15 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-memory@a0100000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a1000000, size 1 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a1000000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a1100000, size 15 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-memory@a1100000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a2000000, size 1 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a2000000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a2100000, size 15 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-memory@a2100000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a3000000, size 1 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a3000000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a3100000, size 15 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-memory@a3100000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a4000000, size 1 MiB
    [    0.000000] OF: reserved mem: initialized node m4f-dma-memory@a4000000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a4100000, size 15 MiB
    [    0.000000] OF: reserved mem: initialized node m4f-memory@a4100000, compatible id shared-dma-pool
    [    0.000000] Zone ranges:
    [    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
    [    0.000000]   DMA32    empty
    [    0.000000]   Normal   empty
    [    0.000000] Movable zone start for each node
    [    0.000000] Early memory node ranges
    [    0.000000]   node   0: [mem 0x0000000080000000-0x000000009e7fffff]
    [    0.000000]   node   0: [mem 0x000000009e800000-0x00000000a57fffff]
    [    0.000000]   node   0: [mem 0x00000000a5800000-0x00000000ffffffff]
    [    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000ffffffff]
    [    0.000000] cma: Reserved 512 MiB at 0x00000000c0000000
    [    0.000000] psci: probing for conduit method from DT.
    [    0.000000] psci: PSCIv1.1 detected in firmware.
    [    0.000000] psci: Using standard PSCI v0.2 function IDs
    [    0.000000] psci: Trusted OS migration not required
    [    0.000000] psci: SMC Calling Convention v1.2
    [    0.000000] percpu: Embedded 2 pages/cpu s49880 r8192 d73000 u131072
    [    0.000000] Detected VIPT I-cache on CPU0
    [    0.000000] CPU features: detected: ARM erratum 845719
    [    0.000000] CPU features: detected: GIC system register CPU interface
    [    0.000000] Built 1 zonelists, mobility grouping off.  Total pages: 32736
    [    0.000000] Kernel command line: console=ttyS2,115200n8 earlycon=ns16550a,mmio32,0x02800000 mtdparts=fc40000.spi.0:1m(ospi.tiboot3),2m(ospi.tispl),4m(ospi.u-boot),256k(ospi.env),256k(ospi.env.backup),57088k@8m(ospi.rootfs),256k(ospi.phypattern) root=PARTUUID=1abf439a-01 rw rootfstype=ext4 rootwait
    [    0.000000] Dentry cache hash table entries: 262144 (order: 5, 2097152 bytes, linear)
    [    0.000000] Inode-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
    [    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
    [    0.000000] Memory: 1432384K/2097152K available (10880K kernel code, 1288K rwdata, 4352K rodata, 1856K init, 751K bss, 140480K reserved, 524288K cma-reserved)
    [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
    [    0.000000] rcu: Preemptible hierarchical RCU implementation.
    [    0.000000] rcu:     RCU event tracing is enabled.
    [    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
    [    0.000000]  Trampoline variant of Tasks RCU enabled.
    [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
    [    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
    [    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
    [    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
    [    0.000000] GICv3: 256 SPIs implemented
    [    0.000000] GICv3: 0 Extended SPIs implemented
    [    0.000000] GICv3: Distributor has no Range Selector support
    [    0.000000] GICv3: 16 PPIs implemented
    [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001840000
    [    0.000000] ITS [mem 0x01820000-0x0182ffff]
    [    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
    [    0.000000] ITS@0x0000000001820000: allocated 1048576 Devices @a6000000 (flat, esz 8, psz 64K, shr 0)
    [    0.000000] ITS: using cache flushing for cmd queue
    [    0.000000] GICv3: using LPI property table @0x00000000a58c0000
    [    0.000000] GIC: using cache flushing for LPI property table
    [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x00000000a58d0000
    [    0.000000] random: get_random_bytes called from start_kernel+0x31c/0x4c4 with crng_init=0
    [    0.000000] arch_timer: cp15 timer(s) running at 225.00MHz (phys).
    [    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x33e45310bc, max_idle_ns: 440795207391 ns
    [    0.000006] sched_clock: 56 bits at 225MHz, resolution 4ns, wraps every 4398046511101ns
    [    0.008573] Console: colour dummy device 80x25
    [    0.013173] Calibrating delay loop (skipped), value calculated using timer frequency.. 450.00 BogoMIPS (lpj=900000)
    [    0.023860] pid_max: default: 32768 minimum: 301
    [    0.028685] LSM: Security Framework initializing
    [    0.033507] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
    [    0.041103] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
    [    0.051707] rcu: Hierarchical SRCU implementation.
    [    0.057015] Platform MSI: msi-controller@1820000 domain created
    [    0.063400] PCI/MSI: /bus@f4000/interrupt-controller@1800000/msi-controller@1820000 domain created
    [    0.072694] EFI services will not be available.
    [    0.077713] smp: Bringing up secondary CPUs ...
    [    0.083464] Detected VIPT I-cache on CPU1
    [    0.083508] GICv3: CPU1: found redistributor 1 region 0:0x0000000001860000
    [    0.083523] GICv3: CPU1: using allocated LPI pending table @0x00000000a58e0000
    [    0.083593] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
    [    0.083750] smp: Brought up 1 node, 2 CPUs
    [    0.113168] SMP: Total of 2 processors activated.
    [    0.117987] CPU features: detected: 32-bit EL0 Support
    [    0.123272] CPU features: detected: CRC32 instructions
    [    0.138053] CPU: All CPU(s) started at EL2
    [    0.142270] alternatives: patching kernel code
    [    0.148233] devtmpfs: initialized
    [    0.160079] KASLR disabled due to lack of seed
    [    0.164973] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
    [    0.174956] futex hash table entries: 512 (order: -1, 32768 bytes, linear)
    [    0.184271] pinctrl core: initialized pinctrl subsystem
    [    0.190365] DMI not present or invalid.
    [    0.195079] NET: Registered protocol family 16
    [    0.209050] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations
    [    0.216498] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
    [    0.224535] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
    [    0.233379] thermal_sys: Registered thermal governor 'step_wise'
    [    0.233387] thermal_sys: Registered thermal governor 'power_allocator'
    [    0.240138] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
    [    0.253850] ASID allocator initialised with 65536 entries
    [    0.293889] HugeTLB registered 16.0 GiB page size, pre-allocated 0 pages
    [    0.300777] HugeTLB registered 512 MiB page size, pre-allocated 0 pages
    [    0.307546] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
    [    0.316679] cryptd: max_cpu_qlen set to 1000
    [    0.325610] k3-chipinfo 43000014.chipid: Family:AM64X rev:SR1.0 JTAGID[0x0bb3802f] Detected
    [    0.334982] vsys_5v0: supplied by evm_12v0
    [    0.339879] vsys_3v3: supplied by evm_12v0
    [    0.344730] vddb_3v3_display: supplied by vsys_3v3
    [    0.351250] iommu: Default domain type: Translated
    [    0.356729] SCSI subsystem initialized
    [    0.361338] mc: Linux media interface: v0.10
    [    0.365744] videodev: Linux video capture interface: v2.00
    [    0.371462] pps_core: LinuxPPS API ver. 1 registered
    [    0.376541] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
    [    0.385893] PTP clock support registered
    [    0.389945] EDAC MC: Ver: 3.0.0
    [    0.394141] omap-mailbox 29020000.mailbox: omap mailbox rev 0x66fc9100
    [    0.401107] omap-mailbox 29040000.mailbox: omap mailbox rev 0x66fc9100
    [    0.407945] omap-mailbox 29060000.mailbox: omap mailbox rev 0x66fc9100
    [    0.415557] FPGA manager framework
    [    0.419150] Advanced Linux Sound Architecture Driver Initialized.
    [    0.426822] clocksource: Switched to clocksource arch_sys_counter
    [    0.433634] VFS: Disk quotas dquot_6.6.0
    [    0.437795] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
    [    0.452950] NET: Registered protocol family 2
    [    0.457703] IP idents hash table entries: 32768 (order: 2, 262144 bytes, linear)
    [    0.466979] tcp_listen_portaddr_hash hash table entries: 4096 (order: 0, 65536 bytes, linear)
    [    0.475907] TCP established hash table entries: 16384 (order: 1, 131072 bytes, linear)
    [    0.484172] TCP bind hash table entries: 16384 (order: 2, 262144 bytes, linear)
    [    0.491978] TCP: Hash tables configured (established 16384 bind 16384)
    [    0.498917] UDP hash table entries: 2048 (order: 0, 65536 bytes, linear)
    [    0.505880] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes, linear)
    [    0.513443] NET: Registered protocol family 1
    [    0.518591] RPC: Registered named UNIX socket transport module.
    [    0.524692] RPC: Registered udp transport module.
    [    0.529539] RPC: Registered tcp transport module.
    [    0.534353] RPC: Registered tcp NFSv4.1 backchannel transport module.
    [    0.540956] PCI: CLS 0 bytes, default 64
    [    0.545897] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
    [    0.559809] Initialise system trusted keyrings
    [    0.564674] workingset: timestamp_bits=46 max_order=15 bucket_order=0
    [    0.577509] squashfs: version 4.0 (2009/01/31) Phillip Lougher
    [    0.584321] NFS: Registering the id_resolver key type
    [    0.589545] Key type id_resolver registered
    [    0.593825] Key type id_legacy registered
    [    0.598018] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
    [    0.604878] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
    [    0.612688] 9p: Installing v9fs 9p2000 file system support
    [    0.672046] Key type asymmetric registered
    [    0.676269] Asymmetric key parser 'x509' registered
    [    0.681332] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
    [    0.688900] io scheduler mq-deadline registered
    [    0.693537] io scheduler kyber registered
    [    0.700640] pinctrl-single f4000.pinctrl: 180 pins, size 720
    [    0.707848] pinctrl-single a40000.timesync-router: 512 pins, size 2048
    [    0.726566] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
    [    0.753668] brd: module loaded
    [    0.766011] loop: module loaded
    [    0.770255] megasas: 07.714.04.00-rc1
    [    0.779622] tun: Universal TUN/TAP device driver, 1.6
    [    0.785726] igbvf: Intel(R) Gigabit Virtual Function Network Driver
    [    0.792158] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
    [    0.798272] sky2: driver version 1.30
    [    0.803537] VFIO - User Level meta-driver version: 0.3
    [    0.810061] i2c /dev entries driver
    [    0.815199] sdhci: Secure Digital Host Controller Interface driver
    [    0.821546] sdhci: Copyright(c) Pierre Ossman
    [    0.826911] sdhci-pltfm: SDHCI platform and OF driver helper
    [    0.834303] ledtrig-cpu: registered to indicate activity on CPUs
    [    0.840972] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
    [    0.849284] optee: probing for conduit method.
    [    0.853930] optee: revision 3.12 (3d47a131)
    [    0.854649] optee: initialized driver
    [    0.865628] NET: Registered protocol family 17
    [    0.870413] 9pnet: Installing 9P2000 support
    [    0.874865] Key type dns_resolver registered
    [    0.879519] Loading compiled-in X.509 certificates
    [    0.902420] ti-sci 44043000.dmsc: ABI: 3.1 (firmware rev 0x0008 '8.4.7--v08.04.07 (Jolly Jellyfi')
    [    0.963152] random: fast init done
    [    0.972785] omap-gpmc 3b000000.memory-controller: GPMC revision 6.0
    [    0.979287] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
    [    0.988100] omap_i2c 20000000.i2c: bus 0 rev0.12 at 100 kHz
    [    0.996261] pca953x 1-0022: supply vcc not found, using dummy regulator
    [    1.003234] pca953x 1-0022: using AI
    [    1.060943] Console: switching to mono frame buffer device 12x2
    [    1.095932] ssd1307fb 1-003c: fb0: Solomon SSD1307 framebuffer device registered, using 192 bytes of video memory
    [    1.106690] omap_i2c 20010000.i2c: bus 1 rev0.12 at 400 kHz
    [    1.113884] omap_i2c 20020000.i2c: bus 2 rev0.12 at 100 kHz
    [    1.121166] omap_i2c 20030000.i2c: bus 3 rev0.12 at 100 kHz
    [    1.127507] ti-sci-intr bus@f4000:bus@4000000:interrupt-controller1: Interrupt Router 5 domain created
    [    1.137452] ti-sci-intr bus@f4000:interrupt-controller0: Interrupt Router 3 domain created
    [    1.146535] ti-sci-inta 48000000.interrupt-controller: Interrupt Aggregator domain 28 created
    [    1.168919] j721e-pcie f102000.pcie: host bridge /bus@f4000/pcie@f102000 ranges:
    [    1.176615] j721e-pcie f102000.pcie:       IO 0x0068001000..0x0068010fff -> 0x0068001000
    [    1.184919] j721e-pcie f102000.pcie:      MEM 0x0068011000..0x006fffffff -> 0x0068011000
    [    1.193228] j721e-pcie f102000.pcie:   IB MEM 0x0000000000..0x0fffffffff -> 0x0000000000
    [    2.205685] j721e-pcie f102000.pcie: PCI host bridge to bus 0000:00
    [    2.212133] pci_bus 0000:00: root bus resource [bus 00-ff]
    [    2.217754] pci_bus 0000:00: root bus resource [io  0x0000-0xffff] (bus address [0x68001000-0x68010fff])
    [    2.227457] pci_bus 0000:00: root bus resource [mem 0x68011000-0x6fffffff]
    [    2.234539] pci 0000:00:00.0: [104c:b010] type 01 class 0x060400
    [    2.240707] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0xfffffffff 64bit pref]
    [    2.248272] pci 0000:00:00.0: supports D1
    [    2.252380] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
    [    2.261177] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
    [    2.272127] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
    [    2.272175] pci 0000:00:00.0: BAR 0: no space for [mem size 0x1000000000 64bit pref]
    [    2.272183] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x1000000000 64bit pref]
    [    2.272193] pci 0000:00:00.0: PCI bridge to [bus 01]
    [    2.272811] pcieport 0000:00:00.0: PME: Signaling with IRQ 44
    [    2.273699] ti-bcdma 485c0100.dma-controller: Number of rings: 68
    [    2.275431] ti-bcdma 485c0100.dma-controller: Channels: 24 (bchan: 12, tchan: 6, rchan: 6)
    [    2.325018] ti-pktdma 485c0000.dma-controller: Number of rings: 288
    [    2.341358] ti-pktdma 485c0000.dma-controller: Channels: 44 (tchan: 29, rchan: 15)
    [    2.353206] printk: console [ttyS2] disabled
    [    2.357694] 2800000.serial: ttyS2 at MMIO 0x2800000 (irq = 16, base_baud = 3000000) is a 8250
    [    2.366465] printk: console [ttyS2] enabled
    [    2.366465] printk: console [ttyS2] enabled
    [    2.374921] printk: bootconsole [ns16550a0] disabled
    [    2.374921] printk: bootconsole [ns16550a0] disabled
    [    2.390379] spi-nor spi0.0: s28hs512t (65536 Kbytes)
    [    2.395426] 7 cmdlinepart partitions found on MTD device fc40000.spi.0
    [    2.401952] Creating 7 MTD partitions on "fc40000.spi.0":
    [    2.407353] 0x000000000000-0x000000100000 : "ospi.tiboot3"
    [    2.414391] 0x000000100000-0x000000300000 : "ospi.tispl"
    [    2.421192] 0x000000300000-0x000000700000 : "ospi.u-boot"
    [    2.428120] 0x000000700000-0x000000740000 : "ospi.env"
    [    2.434733] 0x000000740000-0x000000780000 : "ospi.env.backup"
    [    2.441895] 0x000000800000-0x000003fc0000 : "ospi.rootfs"
    [    2.448712] 0x000003fc0000-0x000004000000 : "ospi.phypattern"
    [    2.514840] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [    2.524846] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867
    [    2.533107] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA00903, cpsw version 0x6BA80903 Ports: 3 quirks:00000002
    [    2.546551] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 16
    [    2.553428] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.4
    [    2.560584] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512
    [    2.567334] pps pps0: new PPS source ptp0
    [    2.571829] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:1
    [    2.585655] am65-cpts 39000000.cpts: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:0
    [    2.597615] gpio-mux mux-controller: 2-way mux-controller registered
    [    2.603616] mmc1: CQHCI version 5.10
    [    2.617196] vdd_mmc1: supplied by vsys_3v3
    [    2.632052] mmc0: CQHCI version 5.10
    [    2.632191] debugfs: Directory 'pd:114' with parent 'pm_genpd' already present!
    [    2.642907] mmc1: SDHCI controller on fa10000.mmc [fa10000.mmc] using ADMA 64-bit
    [    2.657528] ALSA device list:
    [    2.660546]   No soundcards found.
    [    2.745231] mmc1: Command Queue Engine enabled
    [    2.749715] mmc1: new HS400 MMC card at address 0001
    [    2.755525] mmcblk1: mmc1:0001 S0J56X 14.8 GiB
    [    2.760329] mmcblk1boot0: mmc1:0001 S0J56X partition 1 31.5 MiB
    [    2.766468] mmcblk1boot1: mmc1:0001 S0J56X partition 2 31.5 MiB
    [    2.772644] mmcblk1rpmb: mmc1:0001 S0J56X partition 3 4.00 MiB, chardev (237:0)
    [    2.781873]  mmcblk1: p1
    [    4.159113] sdhci-am654 fa00000.mmc: Power on failed
    [    4.194744] mmc0: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
    [    4.269561] EXT4-fs (mmcblk1p1): 1 orphan inode deleted
    [    4.274882] EXT4-fs (mmcblk1p1): recovery complete
    [    4.285239] EXT4-fs (mmcblk1p1): mounted filesystem with ordered data mode. Opts: (null)
    [    4.293475] VFS: Mounted root (ext4 filesystem) on device 179:1.
    [    4.301124] devtmpfs: mounted
    [    4.305093] Freeing unused kernel memory: 1856K
    [    4.309676] Run /sbin/init as init process
    [    4.378427] systemd[1]: System time before build time, advancing clock.
    [    4.422107] NET: Registered protocol family 10
    [    4.428063] Segment Routing with IPv6
    [    4.445887] systemd[1]: systemd 244 running in system mode. (+PAM -AUDIT -SELINUX +IMA -APPARMOR -SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=hybrid)
    [    4.468070] systemd[1]: Detected architecture arm64.
    
    Welcome to Arago 2021.09!
    
    [    4.503759] systemd[1]: Set hostname to <am64xx-evm>.
    [    4.849767] systemd[1]: /lib/systemd/system/startwlansta.service:7: Unknown key name 'After' in section 'Service', ignoring.
    [    4.862705] systemd[1]: /lib/systemd/system/startwlanap.service:7: Unknown key name 'After' in section 'Service', ignoring.
    [    5.064083] random: systemd: uninitialized urandom read (16 bytes read)
    [    5.071161] systemd[1]: system-getty.slice: unit configures an IP firewall, but the local system does not support BPF/cgroup firewalling.
    [    5.083549] systemd[1]: (This warning is only shown for the first unit using IP firewalling.)
    [    5.095997] systemd[1]: Created slice system-getty.slice.
    [  OK  ] Created slice system-getty.slice.
    [    5.119079] random: systemd: uninitialized urandom read (16 bytes read)
    [    5.127641] systemd[1]: Created slice system-serial\x2dgetty.slice.
    [  OK  ] Created slice system-serial\x2dgetty.slice.
    [    5.155093] random: systemd: uninitialized urandom read (16 bytes read)
    [    5.163504] systemd[1]: Created slice system-syslog\x2dng.slice.
    [  OK  ] Created slice system-syslog\x2dng.slice.
    [    5.188828] systemd[1]: Created slice User and Session Slice.
    [  OK  ] Created slice User and Session Slice.
    [    5.211524] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
    [  OK  ] Started Dispatch Password …ts to Console Directory Watch.
    [    5.235421] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
    [  OK  ] Started Forward Password R…uests to Wall Directory Watch.
    [    5.259386] systemd[1]: Reached target Paths.
    [  OK  ] Reached target Paths.
    [    5.279131] systemd[1]: Reached target Remote File Systems.
    [  OK  ] Reached target Remote File Systems.
    [    5.299114] systemd[1]: Reached target Slices.
    [  OK  ] Reached target Slices.
    [    5.319157] systemd[1]: Reached target Swap.
    [  OK  ] Reached target Swap.
    [    5.340305] systemd[1]: Listening on RPCbind Server Activation Socket.
    [  OK  ] Listening on RPCbind Server Activation Socket.
    [    5.363157] systemd[1]: Reached target RPC Port Mapper.
    [  OK  ] Reached target RPC Port Mapper.
    [    5.391774] systemd[1]: Listening on Process Core Dump Socket.
    [  OK  ] Listening on Process Core Dump Socket.
    [    5.415532] systemd[1]: Listening on initctl Compatibility Named Pipe.
    [  OK  ] Listening on initctl Compatibility Named Pipe.
    [    5.447963] systemd[1]: Condition check resulted in Journal Audit Socket being skipped.
    [    5.457015] systemd[1]: Listening on Journal Socket (/dev/log).
    [  OK  ] Listening on Journal Socket (/dev/log).
    [    5.479914] systemd[1]: Listening on Journal Socket.
    [  OK  ] Listening on Journal Socket.
    [    5.500165] systemd[1]: Listening on Network Service Netlink Socket.
    [  OK  ] Listening on Network Service Netlink Socket.
    [    5.523933] systemd[1]: Listening on udev Control Socket.
    [  OK  ] Listening on udev Control Socket.
    [    5.547653] systemd[1]: Listening on udev Kernel Socket.
    [  OK  ] Listening on udev Kernel Socket.
    [    5.577614] systemd[1]: Mounting Huge Pages File System...
             Mounting Huge Pages File System...
    [    5.605320] systemd[1]: Mounting POSIX Message Queue File System...
             Mounting POSIX Message Queue File System...
    [    5.634090] systemd[1]: Mounting Kernel Debug File System...
             Mounting Kernel Debug File System...
    [    5.659763] systemd[1]: Mounting Temporary Directory (/tmp)...
             Mounting Temporary Directory (/tmp)...
    [    5.686328] systemd[1]: Starting Create list of static device nodes for the current kernel...
             Starting Create list of st…odes for the current kernel...
    [    5.718149] systemd[1]: Starting RPC Bind...
             Starting RPC Bind...
    [    5.735458] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
    [    5.753348] systemd[1]: Starting Journal Service...
             Starting Journal Service...
    [    5.783862] systemd[1]: Starting Load Kernel Modules...
             Starting Load Kernel Modules...
    [    5.812796] cryptodev: loading out-of-tree module taints kernel.
    [    5.825693] systemd[1]: Starting Remount Root and Kernel File Systems...
    [    5.835897] cryptodev: driver 1.10 loaded.
             Starting Remount Root and Kernel File Systems...
    [    5.867501] systemd[1]: Starting udev Coldplug all Devices...
             Starting udev Coldplug all Devices...
    [    5.901051] EXT4-fs (mmcblk1p1): re-mounted. Opts: (null)
    [    5.911319] systemd[1]: Started RPC Bind.
    [  OK  ] Started RPC Bind.
    [    5.923715] systemd[1]: Started Journal Service.
    [  OK  ] Started Journal Service.
    [  OK  ] Mounted Huge Pages File System.
    [  OK  ] Mounted POSIX Message Queue File System.
    [  OK  ] Mounted Kernel Debug File System.
    [  OK  ] Mounted Temporary Directory (/tmp).
    [  OK  ] Started Create list of sta… nodes for the current kernel.
    [  OK  ] Started Load Kernel Modules.
    [  OK  ] Started Remount Root and Kernel File Systems.
             Mounting Kernel Configuration File System...
             Starting Flush Journal to Persistent Storage...
    [    6.156705] systemd-journald[156]: Received client request to flush runtime journal.
             Starting Apply Kernel Variables...
             Starting Create Static Device Nodes in /dev...
    [  OK  ] Mounted Kernel Configuration File System.
    [  OK  ] Started Flush Journal to Persistent Storage.
    [  OK  ] Started Apply Kernel Variables.
    [  OK  ] Started Create Static Device Nodes in /dev.
    [  OK  ] Reached target Local File Systems (Pre).
             Mounting /media/ram...
             Mounting /var/volatile...
             Starting udev Kernel Device Manager...
    [  OK  ] Mounted /media/ram.
    [  OK  ] Mounted /var/volatile.
             Starting Load/Save Random Seed...
    [  OK  ] Reached target Local File Systems.
             Starting Create Volatile Files and Directories...
    [  OK  ] Started Create Volatile Files and Directories.
    [  OK  ] Started udev Kernel Device Manager.
             Starting Network Time Synchronization...
             Starting Update UTMP about System Boot/Shutdown...
    [  OK  ] Started udev Coldplug all Devices.
             Starting udev Wait for Complete Device Initialization...
    [  OK  ] Started Update UTMP about System Boot/Shutdown.
    [  OK  ] Started Network Time Synchronization.
    [  OK  ] Reached target System Time Set.
    [  OK  ] Reached target System Time Synchronized.
    [    7.292787] CAN device driver interface
    [    7.445374] 93xx46 spi1.0: 16-bit eeprom
    [  OK  ] Created slice system-systemd\x2dbacklight.slice.
             Starting Load/Save Screen …ess of backlight:ssd1307fb0...
    [  OK  ] Started Load/Save Screen B…tness of backlight:ssd1307fb0.
    [    8.461881] davinci_mdio 300b2400.mdio: davinci mdio revision 1.7, bus freq 1000000
    [    8.467021] k3_r5_rproc bus@f4000:r5fss@78000000: MCU cluster requires both R5F cores to be enabled, num_cores = 0
    [    8.585612] davinci_mdio 300b2400.mdio: phy[15]: device 300b2400.mdio:0f, driver TI DP83869
    [    8.589557] k3_r5_rproc bus@f4000:r5fss@78400000: MCU cluster requires both R5F cores to be enabled, num_cores = 0
    [    9.054926] random: crng init done
    [    9.058346] random: 7 urandom warning(s) missed due to ratelimiting
    [  OK  ] Started Load/Save Random Seed.
    [    9.087977] m_can_platform 20701000.can: m_can device registered (irq=36, version=32)
    [    9.110886] m_can_platform 20711000.can: m_can device registered (irq=38, version=32)
    [    9.242241] remoteproc remoteproc0: 30034000.pru is available
    [    9.323660] remoteproc remoteproc1: 30004000.rtu is available
    [    9.392706] remoteproc remoteproc2: 3000a000.txpru is available
    [    9.468969] remoteproc remoteproc3: 30038000.pru is available
    [    9.523569] remoteproc remoteproc4: 30006000.rtu is available
    [    9.608283] remoteproc remoteproc5: 3000c000.txpru is available
    [    9.634611] remoteproc remoteproc6: 300b4000.pru is available
    [    9.701011] remoteproc remoteproc7: 30084000.rtu is available
    [    9.775881] remoteproc remoteproc8: 3008a000.txpru is available
    [    9.802070] remoteproc remoteproc9: 300b8000.pru is available
    [    9.847299] remoteproc remoteproc10: 30086000.rtu is available
    [    9.848138] remoteproc remoteproc11: 3008c000.txpru is available
    [   11.553592] omap_rng 40910000.rng: Random Number Generator ver. 241b34c
    [   11.557663] icssg-prueth icssg1-eth: TI PRU ethernet driver initialized: single EMAC mode
    [   11.930623] usbcore: registered new interface driver usbfs
    [   11.992831] usbcore: registered new interface driver hub
    [   12.024408] usbcore: registered new device driver usb
    [  OK  ] Started udev Wait for Complete Device Initialization.
    [  OK  ] Started Hardware RNG Entropy Gatherer Daemon.
    [  OK  ] Reached target System Initialization.
    [  OK  ] Started Daily rotation of log files.
    [  OK  ] Started Timer service to update the IP on OLED each 10s.
    [  OK  ] Started Daily Cleanup of Temporary Directories.
    [  OK  ] Reached target Timers.
    [  OK  ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
    [  OK  ] Listening on D-Bus System Message Bus Socket.
    [  OK  ] Listening on dropbear.socket.
    [  OK  ] Reached target Sockets.
    [  OK  ] Reached target Basic System.
    [  OK  ] Started Job spooling tools.
    [  OK  ] Started Periodic Command Scheduler.
    [  OK  ] Started D-Bus System Message Bus.
             Starting Ethernet Bridge Filtering Tables...
             Starting Print notice about GPLv3 packages...
             Starting IPv6 Packet Filtering Framework...
             Starting IPv4 Packet Filtering Framework...
    [  OK  ] Started irqbalance daemon.
             Starting Reboot and dump vmcore via kexec...
             Starting Matrix GUI...
             Starting startwlanap...
             Starting startwlansta...
             Starting System Logger Daemon "default" instance...
             Starting Login Service...
    [  OK  ] Started TEE Supplicant.
             Starting telnetd.service...
    [  OK  ] Started Ethernet Bridge Filtering Tables.
    [  OK  ] Started IPv6 Packet Filtering Framework.
    [  OK  ] Started IPv4 Packet Filtering Framework.
    [   13.090172] cfg80211: Loading compiled-in X.509 certificates for regulatory database
    [  OK  ] Started Reboot and dump vmcore via kexec.
    [  OK  ] Started Matrix GUI.
    [  OK  ] Started startwlanap.
    [  OK  ] Started telnetd.service.
    [  OK  ] Started startwlansta.
    [   13.196323] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
    [  OK  ] Reached target Network (Pre).
    [  OK  ] Listening on Load/Save RF …itch Status /dev/rfkill Watch.
             Starting syslog.service...
             Starting Network Service...
    [  OK  ] Started System Logger Daemon "default" instance.
    [  OK  ] Started Network Service.
    [   13.739849] remoteproc remoteproc6: powering up 300b4000.pru
    [   13.746106] remoteproc remoteproc6: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, size 37264
    [   13.746138] remoteproc remoteproc6: unsupported resource 5
    [   13.746169] remoteproc remoteproc6: remote processor 300b4000.pru is now up
    [   13.746205] remoteproc remoteproc7: powering up 30084000.rtu
    [   13.751502] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, size 30832
    [   13.751553] remoteproc remoteproc7: remote processor 30084000.rtu is now up
    [   13.751592] remoteproc remoteproc8: powering up 3008a000.txpru
    [   13.755123] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, size 36656
    [   13.755181] remoteproc remoteproc8: remote processor 3008a000.txpru is now up
    [   13.762332] pps pps1: new PPS source ptp2
    [   13.794135] TI DP83869 300b2400.mdio:0f: attached PHY driver [TI DP83869] (mii_bus:phy_addr=300b2400.mdio:0f, irq=POLL)
    [   13.794431] net eth2: started
    [   13.883056] TI DP83869 0.1:03: attached PHY driver [TI DP83869] (mii_bus:phy_addr=0.1:03, irq=POLL)
    [   13.887176] am65-cpsw-nuss 8000000.ethernet eth1: Link is Down
    [   13.938476] TI DP83867 8000f00.mdio:00: attached PHY driver [TI DP83867] (mii_bus:phy_addr=8000f00.mdio:00, irq=POLL)
    [   13.941377] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
             Starting Wait for Network to be Configured...
             Starting Network Name Resolution...
    [  OK  ] Started syslog.service.
    [  OK  ] Started Login Service.
    [   14.355112] startwlansta.sh[678]: Wi-Fi user configuration file missing, exit
    [  OK  ] Started Network Name Resolution.
    [  OK  ] Reached target Network.
    [  OK  ] Reached target Host and Network Name Lookups.
             Starting Avahi mDNS/DNS-SD Stack...
             Starting Enable and configure wl18xx bluetooth stack...
    [  OK  ] Started NFS status monitor for NFSv2/3 locking..
             Starting Simple Network Ma…ent Protocol (SNMP) Daemon....
             Starting Permit User Sessions...
    [  OK  ] Started Vsftpd ftp daemon.
    [  OK  ] Started Enable and configure wl18xx bluetooth stack.
    [  OK  ] Started Permit User Sessions.
    [  OK  ] Started Avahi mDNS/DNS-SD Stack.
    [  OK  ] Started Getty on tty1.
    [  OK  ] Started Serial Getty on ttyS1.
    [  OK  ] Started Serial Getty on ttyS2.
    [  OK  ] Reached target Login Prompts.
             Starting Synchronize System and HW clocks...
    [FAILED] Failed to start Synchronize System and HW clocks.
    See 'systemctl status sync-clocks.service' for details.
    [  OK  ] Started Simple Network Man…ement Protocol (SNMP) Daemon..
    ***************************************************************
    ***************************************************************
    NOTICE: This file system contains the following GPLv3 packages:
            autoconf
            bash
            bc
            binutils
            cifs-utils
            coreutils-stdbuf
            coreutils
            cpio
            cpp-symlinks
            cpp
            dosfstools
            elfutils
            g++-symlinks
            g++
            gawk
            gcc-symlinks
            gcc
            gdb
            gdbserver
            gettext
            gstreamer1.0-libav
            gzip
            hidapi
            less
            libasm1
            libbfd
            libdw1
            libelf1
            libgdbm-compat4
            libgdbm6
            libgettextlib
            libgettextsrc
            libgmp10
            libidn2-0
            libmpc3
            libmpfr6
            libreadline8
            libunistring2
            m4
            make
            nettle
            parted
            tar
            which
    
    If you do not wish to distribute GPLv3 components please remove
    the above packages prior to distribution.  This can be done using
    the opkg remove command.  i.e.:
        opkg remove <package>
    Where <package> is the name printed in the list above
    
    NOTE: If the package is a dependency of another package you
          will be notified of the dependent packages.  You should
          use the --force-removal-of-dependent-packages option to
          also remove the dependent packages as well
    ***************************************************************
    ***************************************************************
    [  OK  ] Started Print notice about GPLv3 packages.
    
     _____                    _____           _         _
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
                  |___|                    |___|
    
    Arago Project http://arago-project.org am64xx-evm ttyS2
    
    Arago 2021.09 am64xx-evm ttyS2
    
    am64xx-evm login: rot
    Password:
    
    Login incorrect
    am64xx-evm login: root
    [   33.866876] vdd_mmc1: disabling
    root@am64xx-evm:~# dmesg | grep remoteproc
    [    9.242241] remoteproc remoteproc0: 30034000.pru is available
    [    9.323660] remoteproc remoteproc1: 30004000.rtu is available
    [    9.392706] remoteproc remoteproc2: 3000a000.txpru is available
    [    9.468969] remoteproc remoteproc3: 30038000.pru is available
    [    9.523569] remoteproc remoteproc4: 30006000.rtu is available
    [    9.608283] remoteproc remoteproc5: 3000c000.txpru is available
    [    9.634611] remoteproc remoteproc6: 300b4000.pru is available
    [    9.701011] remoteproc remoteproc7: 30084000.rtu is available
    [    9.775881] remoteproc remoteproc8: 3008a000.txpru is available
    [    9.802070] remoteproc remoteproc9: 300b8000.pru is available
    [    9.847299] remoteproc remoteproc10: 30086000.rtu is available
    [    9.848138] remoteproc remoteproc11: 3008c000.txpru is available
    [   13.739849] remoteproc remoteproc6: powering up 300b4000.pru
    [   13.746106] remoteproc remoteproc6: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, size 37264
    [   13.746138] remoteproc remoteproc6: unsupported resource 5
    [   13.746169] remoteproc remoteproc6: remote processor 300b4000.pru is now up
    [   13.746205] remoteproc remoteproc7: powering up 30084000.rtu
    [   13.751502] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, size 30832
    [   13.751553] remoteproc remoteproc7: remote processor 30084000.rtu is now up
    [   13.751592] remoteproc remoteproc8: powering up 3008a000.txpru
    [   13.755123] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, size 36656
    [   13.755181] remoteproc remoteproc8: remote processor 3008a000.txpru is now up
    root@am64xx-evm:~#

    Regards,

    Prashant

  • Hi EyalCo,

    You may suggest the same as above to the customer and let us know if they still face issues.

    Regards,

    Prashant

  • Hi Prashant, see their reply:


    First Issue:

    I have disabled R5 cores, and M4 core in the Device Tree.
    The call trace issue is solved. However this does not resolve our issue of using the RPmsg functionality between A53 and R5.


    I believe the issue is that Linux does not recognize or initialize any r5 cores (because we disabled in DTB),

    so the kernel drivers required for R5 communication are not started.


    from dmesg (in this case I enabled only r5f0_0):
    [    7.745977] k3_r5_rproc bus@f4000:r5fss@78000000: MCU cluster requires both R5F cores to be enabled, num_cores = 1

    [    8.157637] k3_r5_rproc bus@f4000:r5fss@78400000: MCU cluster requires both R5F cores to be enabled, num_cores = 0

     

    /sys/class/remoteproc/ is also empty and does not contain any information about the R5 cores.


    Is the RPmsg functionality supposed to work if we disable the R5 cores in DTS?

    Attached files: dmesg_disabled_cores.txt, lsmod_disabled_cores.txt

    Second Issue:

    When we run our R5 firmware configured to use UART 0 for output – then RPMsg communication works between R5 and A53
    however it causes the console terminal to freeze (because they are sharing the same UART)

     

    When we run our R5 firmware configured to use UART 1 for output – then the firmware fails at some point before RPmsg communication is made.

     

    When debugging this issue I have decided to check the DTS.

    I have modified the DTS arch/arm64/boot/dts/ti/k3-am642-evm.dts:

    Changing the main_uart1 to be “disabled” seems to have resolved the issue, and our RPmsg firmware does not freeze anymore.

    Can you comment on this fix? Does it seem correct?
     Is main_uart1 supposed to be disabled so the firmware can use it without freezing, or is there a change we are missing in our firmware?7043.lsmod_disabled_cores.txt

    8765.dmesg_disabled_cores.txt

  • Hello Eyal,

    RPMsg will not work unless the remote core is enabled in the Linux devicetree. So we can either patch SBL boot so that it no longer puts remote cores into a bad state, or your customers can switch to a "regular" Linux SPL boot (as documented here: https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/08_05_00_21/exports/docs/linux/Foundational_Components/U-Boot/UG-General-Info.html ).

    Is there a specific reason your customer is using SBL boot if they aren't initializing the remote cores?

    1) please give me the link to the Jira bug you filed against SBL_EMMC_LINUX

    2) please post the changes you made to SBL (and point to any documentation about rebuilding SBL) so that Eyal's customers can try to disable remote cores on their side.

    Thanks,

    Nick

  • As far as UART, make sure only one core is asking SYSFW for ownership of a single UART instance. If multiple cores request the same UART, you will run into issues.

    Regards,

    Nick

  • Eyal,

    Clarifying the customer's response

    This is my interpretation of what is meant in your response.
    https://e2e.ti.com/support/processors-group/processors---internal/f/processors---internal-forum/1194822/am6442-booting-issue/4526114#4526114 
    Please correct me if I got anything wrong:

    "First Issue": talking about MCU+ SBL boot, where remote cores are disabled in the Linux devicetree. RPMsg is not working, since Linux RemoteProc driver does not know that the remote cores are running.

    "Second issue": is this talking about the default Linux SDK SPL boot? (since RPMsg is working).

    More information about UART 

    Keep in mind that as soon as a Linux devicetree node is defined, it is enabled by default, and Linux will try to use it. Thus, if you want the remote cores to use a peripheral (like UART1), you have to make sure it is specifically disabled in the board-level Linux devicetree.

    However... I would expect "disabled" and "reserved" to do the same thing, so that is curious that the customer is seeing different behavior. I did find one reference to SYSFW using UART1 on AM64x, but I don't see how the Linux devicetree status would impact that: https://lore.kernel.org/lkml/16ad1a82-1e99-2957-720f-08d1238edcf4@ti.com/raw . I'll check with the developers.

    Regards,

    Nick

  • Eyal,

    Final question: Is the customer debugging the R5F application in CCS during this process? (as documented here: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1188177/faq-am62x-am64x-how-to-use-ccs-to-debug-a-running-m4f-core-that-was-started-by-linux ). If so, could they check to see where in the R5F code the code is getting stuck during initialization?

    My suspicion is that the R5F is requesting the UART1, and SYSFW is just never giving it the UART since the peripheral was already claimed by something else, so the software hangs. But it would be helpful to know for sure.

    Regards,

    Nick

  • Call Trace Issue -
    I did not understand your comment regarding the remote cores.
    Why do you say that the SBL puts the remote cores in a bad state?

     

    We are using SBL boot, so we need the r5 cores to be initialized in SBL, so the firmware can start running quickly.


    When Linux boots, it also tries to initialize the cores, which results in the errors we saw (call trace issue and core already initialized issue)

    If we disable R5 cores in Linux DTS like you suggest, then the RPmsg feature won’t work.

    So I believe the solution we need is  to keep R5 cores enabled in DTS, but avoid linux initializing them.


    UART Issue –

    Only 1 single r5 core is using UART, since we are testing our firmware only on a single r5 core right now.
    So like I mentioned previously –

    R5f0_0 core firmware is trying to access UART 1 and failing.
    I have changed &main_uart1 from status reserved to status disabled (in DTS), and that seemed to have solve the problem.

    If this solution seems good to TI engineers, then we will stay with it.

     

    Is there a specific reason you are using SBL boot if you aren't initializing the remote cores? -

    SBL is initializing the remote cores.

    The problem arises because linux tries to initialize them a second time.
    If we disable r5 cores in the DTS, then the RPmsg functionality won’t work, I presume because the kernel drivers won’t see the r5 devices in the filesystem.

     

    It’s not in my department, but we are using SBL for 2 reasons:

    1. Start time for r5 firmware is faster in SBL than SPL.
    2. Issue with LWIP functionality being initialized twice (I am less familiar with this issue)
  • 1) SBL is currently FAILING to initialize the remote cores (as of the information I was provided). That is your problem, not the Linux boot afterwards. If the remote cores were successfully initialized by SBL, then Linux will just set up RPMsg and continue with the Linux boot.

    Look at the very first lines of your SBL boot output from the previous response:
    https://e2e.ti.com/support/processors-group/processors---internal/f/processors---internal-forum/1194822/am6442-booting-issue/4506427#4506427

    R5_0_1 Validation
    Failed to boot application. No valid Firmware exist.
    
    R5_1_0 Validation
    Failed to boot application. No valid Firmware exist.
    
    R5_1_1 Validation
    Failed to boot application. No valid Firmware exist.
    
    Cortex M4 Validation
    Failed to boot application. No valid Firmware exist.

    When SBL is unable to find a remote core binary to load, and then tries to load a nonexistent binary anyway, it puts the remote cores into a bad state. Then Linux sees that there is no firmware running on the binary, Linux tries to initialize the remote cores, and then runs into an error because of the bad state.

    So first, you need to make sure that the remote core binaries you want SBL to load are properly packaged into the SBL. Reference "Understanding the boot flow and bootloaders": https://software-dl.ti.com/mcu-plus-sdk/esd/AM64X/08_05_00_24/exports/docs/api_guide_am64x/BOOTFLOW_GUIDE.html

    2) UART1: "disabled" vs "reserved" in the Linux tree should not make a difference 

    The developers confirmed that from a Linux standpoint, both of those statuses mean that Linux just ignores the peripheral. So changing the status from one to the other should not make a difference.

    Are you sure there is nothing else that you changed between the working case and the not working case?

    3) Enable the remote cores in the Linux devicetree. Otherwise RPMsg will not work 

    As discussed in point 1), the Linux RemoteProc driver checks to see if a remote core is currently running before it tries to load any firmware. If Linux sees that the remote core is in a "running" state, then it just sets up RPMsg based on information provided by the remote core, and then moves on with the kernel boot. You do not need to do anything special to try to modify the default boot behavior.

    Regards,

    Nick

  • For testing purposes right now, we only have firmware installed and running on R5_0_0

    So the errors for missing firmware on the other cores is expected.

     

    We should still be able to work with RPmsg with only firmware on R5_0_0.

     

    So keeping this in mind, we still encounter the issue that:

    1. If we let SBL initialize R5_0_0 firmware, and boot into linux, we get the call trace issue (however RPmsg works)
      2. If we let SBL initialize R5_0_0 firmware , and I disable R5 cores in the linux DTS, then there is no call trace issue, but RPmsg does not work. (presumably linux does not know r5 cores exist and the required RPmsg kernel drivers don’t load)
  • please note this thread

  • Please confirm the exact settings you are doing for each behavior.

    For example,

    1) if you are only packaging the R5F_0_0 firmware into the SBL, and none of the other remote core firmwares, then I would expect that all remote cores other than R5F_0_0 are put into a bad state by SBL. Thus, your Linux devicetree file would have to match: for R5F_0_0 set status = okay to allow RPMsg, and set status = disabled for all other remote cores to prevent issues.

    Or,

    2) you could provide valid firmware for all remote cores in SBL, so that all remote core firmwares are properly booted. Then you can simply leave all remote cores as enabled in the Linux devicetree.

    Do you still see issues when doing 1) or 2)?

    Regards,

    Nick

  • Hi Nick,

    1. When I enabled only r5f_0_0 in the DTS I got the following error. I have also mentioned this issue about 1 week ago, and it is written in the responses above. Can you explain what we are seeing? 
    I presume enabling only 1 core in the DTS will not work due to this error.

    From dmesg (in this case I enabled only r5f0_0):
    [    7.745977] k3_r5_rproc bus@f4000:r5fss@78000000: MCU cluster requires both R5F cores to be enabled, num_cores = 1
    [    8.157637] k3_r5_rproc bus@f4000:r5fss@78400000: MCU cluster requires both R5F cores to be enabled, num_cores = 0

    2. I will have to ask the team that is writing the R5 firmware.
    Right now they do not support all cores.

  • Hello Ben,

    Ahh, it looks like the R5F subsystem is being configured to have both cores running. So when only one core is enabled, the subsystem complains. If you only want to use one R5F core, please configure the subsystem for "single-CPU mode". Reference the Linux driver bindings documentation:
    /Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml

    Regards,

    NIck

  • Hi Nick,

    I am using the Documentation i found here:
    https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml?h=ti-linux-5.10.y

    # Optional properties:
    # --------------------
    
      ti,cluster-mode:
        $ref: /schemas/types.yaml#/definitions/uint32
        description: |
          Configuration Mode for the Dual R5F cores within the R5F cluster.
          Should be either a value of 1 (LockStep mode) or 0 (Split mode) on
          most SoCs (AM65x, J721E, J7200, J721s2), default is LockStep mode if
          omitted; and should be either a value of 0 (Split mode) or 2
          (Single-CPU mode) on AM64x SoCs, default is Split mode if omitted.
    

    Following the DTS here:
    https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/arch/arm64/boot/dts/ti/k3-am64-main.dtsi?h=ti-linux-5.10.y

    It seems there are 2 lines I need to change
    From:
    ti,cluster-mode = <0>;

    To:
    ti,cluster-mode = <2>;

    Let me know if you notice anything else.
    The solution seems like it should work, but I have not tested yet.
    I am on vacation for the entire week, skiing in Austria.
    So it will be some time before I update on this solution.

    Thanks,
    Ben.

  • Hello Ben,

    That looks correct to me. Have fun in the mountains! Let us know if there are followup questions once you get back.

    Regards,

    Nick