This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/PROCESSOR-SDK-AM65X: icssg_prueth kernel panic

Part Number: PROCESSOR-SDK-AM65X


Tool/software: Linux

Dear TI team,

we're getting kernel panics from the icssg_prueth kernel driver whenever we have more than one PRU ethernet interface with an active link (attached to the same switch).

I've attached the serial console output from one test run where we've been using the prebuilt Linux images that come with the processor SDK Linux for the AM65x version 05.02, but the same problem occurs when using a kernel that we've built ourselves with Yocto, or when using a custom kernel (based on the same branch and commit as the processor SDK kernel) on our custom hardware.

The problem only appears when we have more than one Ethernet interface attached. It could be anything from "right after boot" to "after a few minutes" with no obvious relation to what we're doing, i.e. at some point the system crashes even if we're not doing anything, but it also "survives" a flood ping for several minutes.

There is a known issue "LCPD-14189" in the release notes but that doesn't contain a whole lot of information. Is this the same issue we're seeing? Is there a workaround available or a fix planned?

Regards,

Dominic

þ
U-Boot SPL 2018.01-g313dcd69c2 (Dec 16 2018 - 05:23:30)
SYSFW ABI: 2.5 (firmware rev 0x0012 '18.10.0-v2018.10 (Curious Crow)')
Trying to boot from MMC2
Starting ATF on ARM64 core...

NOTICE:  BL31: v1.5(release):ti2018.04-rc1
NOTICE:  BL31: Built : 04:37:22, Dec 16 2018
I/TC:
I/TC: OP-TEE version: 3.2.0-81-g48952f9-dev #1 Sun Dec 16 05:14:09 UTC 2018 aarch64
I/TC: Initialized

U-Boot SPL 2018.01-g313dcd69c2 (Dec 16 2018 - 05:15:53)
Trying to boot from MMC2


U-Boot 2018.01-g313dcd69c2 (Dec 16 2018 - 05:15:53 +0000)

Model: Texas Instruments AM654 Base Board
DRAM:  4 GiB
MMC:   sdhci@0: 0, sdhci@0: 1
*** Warning - bad CRC, using default environment

In:    serial@2800000
Out:   serial@2800000
Err:   serial@2800000
GUID Partition Table Header signature is wrong: 0x0 != 0x5452415020494645
part_get_info_efi: *** ERROR: Invalid GPT ***
GUID Partition Table Header signature is wrong: 0x0 != 0x5452415020494645
part_get_info_efi: *** ERROR: Invalid Backup GPT ***
detected AM6-IDKAPPEVM
detected SER-PCIE2LEVM
Net:
Warning: cpsw_nuss@046000000 using MAC address from ROM
eth0: cpsw_nuss@046000000
Hit any key to stop autoboot:  0
GUID Partition Table Header signature is wrong: 0x0 != 0x5452415020494645
part_get_info_efi: *** ERROR: Invalid GPT ***
GUID Partition Table Header signature is wrong: 0x0 != 0x5452415020494645
part_get_info_efi: *** ERROR: Invalid Backup GPT ***
bad MBR sector signature 0x0000
switch to partitions #0, OK
mmc1 is current device
SD/MMC found on device 1
** Unable to read file boot.scr **
1011 bytes read in 8 ms (123 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
Running uenvcmd ...
Using board_name = am65x
11685896 bytes read in 1020 ms (10.9 MiB/s)
59908 bytes read in 17 ms (3.4 MiB/s)
9204 bytes read in 9 ms (998 KiB/s)
2084 bytes read in 6 ms (338.9 KiB/s)
## Flattened Device Tree blob at 82000000
   Booting using the fdt blob at 0x82000000
   reserving fdt memory region: addr=82000000 size=10f000
   Loading Device Tree to 00000000fddb2000, end 00000000fdec3fff ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.79-gbde58ab01e (oe-user@oe-host) (gcc version 7.2.1 20171011 (Linaro GCC 7.2-2017.11)) #1 SMP PREEMPT Thu Dec 20 03:19:19 UTC 2018
[    0.000000] Boot CPU: AArch64 Processor [410fd034]
[    0.000000] Machine model: Texas Instruments AM654 Base Board
[    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
[    0.000000] bootconsole [ns16550a0] enabled
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009b000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f1-dma-memory@9b000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009b100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f1-memory@9b100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009c000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f0-dma-memory@9c000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009c100000, size 7 MiB
[    0.000000] OF: reserved mem: initialized node r5f0-memory@9c100000, compatible id shared-dma-pool
[    0.000000] cma: Reserved 512 MiB at 0x00000000c0000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] percpu: Embedded 2 pages/cpu @ffff80087feb0000 s38936 r8192 d83944 u131072
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for ARM erratum 845719
[    0.000000] Speculative Store Bypass Disable mitigation not required
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 64704
[    0.000000] Kernel command line: console=ttyS2,115200n8 earlycon=ns16550a,mmio32,0x02800000 mtdparts=47040000.ospi.0:512k(ospi.tiboot3),2m(ospi.tispl),4m(ospi.u-boot),128k(ospi.env),128k(ospi.env.backup),1m(ospi.sysfw),-@8m(ospi.rootfs) root=PARTUUID=e25a6b67-02 rw rootfstype=ext4 rootwait
[    0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes)
[    0.000000] Dentry cache hash table entries: 524288 (order: 6, 4194304 bytes)
[    0.000000] Inode-cache hash table entries: 262144 (order: 5, 2097152 bytes)
[    0.000000] software IO TLB [mem 0xf9db0000-0xfddb0000] (64MB) mapped at [ffff800079db0000-ffff80007ddaffff]
[    0.000000] Memory: 3529664K/4145152K available (7422K kernel code, 660K rwdata, 2816K rodata, 448K init, 654K bss, 91200K reserved, 524288K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7bdfffff0000   (126847 GB)
[    0.000000]       .text : 0xffff000008080000 - 0xffff0000087c0000   (  7424 KB)
[    0.000000]     .rodata : 0xffff0000087c0000 - 0xffff000008a90000   (  2880 KB)
[    0.000000]       .init : 0xffff000008a90000 - 0xffff000008b00000   (   448 KB)
[    0.000000]       .data : 0xffff000008b00000 - 0xffff000008ba5008   (   661 KB)
[    0.000000]        .bss : 0xffff000008ba5008 - 0xffff000008c48a60   (   655 KB)
[    0.000000]     fixed   : 0xffff7fdffe7b0000 - 0xffff7fdffec00000   (  4416 KB)
[    0.000000]     PCI I/O : 0xffff7fdffee00000 - 0xffff7fdfffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffff7fe000000000 - 0xffff800000000000   (   128 GB maximum)
[    0.000000]               0xffff7fe000000000 - 0xffff7fe002200000   (    34 MB actual)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff800880000000   ( 34816 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000]  RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.
[    0.000000]  Tasks RCU enabled.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: no VLPI support, no direct LPI support
[    0.000000] ITS [mem 0x01820000-0x0182ffff]
[    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
[    0.000000] ITS@0x0000000001820000: allocated 1048576 Devices @8c0800000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GIC: using LPI property table @0x00000008c00b0000
[    0.000000] ITS: Allocated 1792 chunks for LPIs
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001880000
[    0.000000] CPU0: using LPI pending table @0x00000008c00c0000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
[    0.000006] sched_clock: 56 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
[    0.008635] Console: colour dummy device 80x25
[    0.013216] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000)
[    0.023906] pid_max: default: 32768 minimum: 301
[    0.028756] Security Framework initialized
[    0.033012] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes)
[    0.039892] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes)
[    0.063310] ASID allocator initialised with 32768 entries
[    0.068947] Hierarchical SRCU implementation.
[    0.081527] PCI/MSI: /soc0/interrupt-controller@1800000/gic-its@1000000 domain created
[    0.089931] Platform MSI: /soc0/interrupt-controller@1800000/gic-its@1000000 domain created
[    0.106515] smp: Bringing up secondary CPUs ...
[    0.139642] Detected VIPT I-cache on CPU1
[    0.139680] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000
[    0.139727] CPU1: using LPI pending table @0x00000008c0360000
[    0.139772] CPU1: Booted secondary processor [410fd034]
[    0.196659] Detected VIPT I-cache on CPU2
[    0.196698] GICv3: CPU2: found redistributor 100 region 0:0x00000000018c0000
[    0.196748] CPU2: using LPI pending table @0x00000008c03d0000
[    0.196796] CPU2: Booted secondary processor [410fd034]
[    0.225073] Detected VIPT I-cache on CPU3
[    0.225100] GICv3: CPU3: found redistributor 101 region 0:0x00000000018e0000
[    0.225147] CPU3: using LPI pending table @0x00000008c0430000
[    0.225178] CPU3: Booted secondary processor [410fd034]
[    0.225312] smp: Brought up 1 node, 4 CPUs
[    0.296968] SMP: Total of 4 processors activated.
[    0.301785] CPU features: detected feature: GIC system register CPU interface
[    0.309089] CPU features: detected feature: 32-bit EL0 Support
[    0.315059] CPU features: detected feature: Kernel page table isolation (KPTI)
[    0.346368] CPU: All CPU(s) started at EL2
[    0.350589] alternatives: patching kernel code
[    0.356559] devtmpfs: initialized
[    0.369766] random: get_random_u32 called from bucket_table_alloc+0x108/0x258 with crng_init=0
[    0.379390] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.389381] futex hash table entries: 1024 (order: 1, 131072 bytes)
[    0.397209] pinctrl core: initialized pinctrl subsystem
[    0.403201] NET: Registered protocol family 16
[    0.409820] vdso: 2 pages (1 code @ ffff0000087d0000, 1 data @ ffff000008b20000)
[    0.423413] DMA: preallocated 256 KiB pool for atomic allocations
[    0.434540] davinci_gpio 600000.main_gpio0: IRQ not populated, err = -517
[    0.441705] davinci_gpio 601000.main_gpio1: IRQ not populated, err = -517
[    0.448831] davinci_gpio 42110000.: IRQ not populated, err = -517
[    0.466574] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.473457] HugeTLB registered 512 MiB page size, pre-allocated 0 pages
[    0.482774] omap_i2c 42120000.i2c: could not find pctldev for node /soc0/pinmux@4301c000/wkup_i2c0_pins_default, deferring probe
[    0.494689] omap_i2c 40b00000.i2c: could not find pctldev for node /soc0/pinmux@4301c000/mcu_i2c0_pins_default, deferring probe
[    0.506467] omap_i2c 2000000.i2c: could not find pctldev for node /soc0/pinmux@11c2e8/main_i2c0_pins_default, deferring probe
[    0.518072] omap_i2c 2010000.i2c: could not find pctldev for node /soc0/pinmux@11c2e8/main_i2c1_pins_default, deferring probe
[    0.529668] omap_i2c 2020000.i2c: could not find pctldev for node /soc0/pinmux@11c000/main_i2c2_pins_default, deferring probe
[    0.541381] media: Linux media interface: v0.10
[    0.546046] Linux video capture interface: v2.00
[    0.550808] pps_core: LinuxPPS API ver. 1 registered
[    0.555890] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.565249] PTP clock support registered
[    0.569284] EDAC MC: Ver: 3.0.0
[    0.573429] Advanced Linux Sound Architecture Driver Initialized.
[    0.580398] clocksource: Switched to clocksource arch_sys_counter
[    0.586811] VFS: Disk quotas dquot_6.6.0
[    0.590936] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[    0.604909] NET: Registered protocol family 2
[    0.609894] TCP established hash table entries: 32768 (order: 2, 262144 bytes)
[    0.617602] TCP bind hash table entries: 32768 (order: 3, 524288 bytes)
[    0.624956] TCP: Hash tables configured (established 32768 bind 32768)
[    0.631795] UDP hash table entries: 2048 (order: 0, 65536 bytes)
[    0.638067] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes)
[    0.644936] NET: Registered protocol family 1
[    0.649897] RPC: Registered named UNIX socket transport module.
[    0.655973] RPC: Registered udp transport module.
[    0.660789] RPC: Registered tcp transport module.
[    0.665603] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.677383] workingset: timestamp_bits=46 max_order=16 bucket_order=0
[    0.689269] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.695839] NFS: Registering the id_resolver key type
[    0.701097] Key type id_resolver registered
[    0.705389] Key type id_legacy registered
[    0.709503] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.716562] 9p: Installing v9fs 9p2000 file system support
[    0.726704] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    0.734292] io scheduler noop registered
[    0.738419] io scheduler cfq registered (default)
[    0.743240] io scheduler mq-deadline registered
[    0.747879] io scheduler kyber registered
[    0.754224] pinctrl-single 11c000.pinmux: 185 pins at pa ffff000009cac000 size 740
[    0.762149] pinctrl-single 11c2e8.pinmux: 9 pins at pa ffff000009ccc2e8 size 36
[    0.769872] pinctrl-single 4301c000.pinmux: 70 pins at pa ffff000009cec000 size 280
[    0.785079] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
[    0.794492] console [ttyS2] disabled
[    0.798216] 2800000.serial: ttyS2 at MMIO 0x2800000 (irq = 14, base_baud = 3000000) is a 8250
[    0.807019] console [ttyS2] enabled
[    0.807019] console [ttyS2] enabled
[    0.814065] bootconsole [ns16550a0] disabled
[    0.814065] bootconsole [ns16550a0] disabled
[    0.834627] loop: module loaded
[    0.839681] libphy: Fixed MDIO Bus: probed
[    0.844395] tun: Universal TUN/TAP device driver, 1.6
[    0.849642] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[    0.857470] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.863435] sky2: driver version 1.30
[    0.868413] VFIO - User Level meta-driver version: 0.3
[    0.874295] i2c /dev entries driver
[    0.878737] sdhci: Secure Digital Host Controller Interface driver
[    0.884916] sdhci: Copyright(c) Pierre Ossman
[    0.889300] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.896255] ledtrig-cpu: registered to indicate activity on CPUs
[    0.902350] ti-sci soc0:dmsc: invalid resource
[    0.907546] optee: probing for conduit method from DT.
I/TC: Dynamic shared memory is disabled
[    0.916715] optee: initialized driver
[    0.921664] NET: Registered protocol family 17
[    0.926199] 9pnet: Installing 9P2000 support
[    0.930526] Key type dns_resolver registered
[    0.935432] registered taskstats version 1
[    0.940140] davinci_gpio 600000.main_gpio0: IRQ not populated, err = -517
[    0.947311] davinci_gpio 601000.main_gpio1: IRQ not populated, err = -517
[    0.954314] davinci_gpio 42110000.: IRQ not populated, err = -517
[    0.965987] ti-sci soc0:dmsc: invalid resource
[    0.970634] ti-sci soc0:dmsc: ABI: 2.5 (firmware rev 0x0012 '18.10.0-v2018.10 (Curious Crow)')
[    0.996666] random: fast init done
[    1.318950] davinci_gpio 600000.main_gpio0: IRQ not populated, err = -517
[    1.326126] davinci_gpio 601000.main_gpio1: IRQ not populated, err = -517
[    1.333130] davinci_gpio 42110000.: IRQ not populated, err = -517
[    1.340715] omap_i2c 42120000.i2c: bus 0 rev0.12 at 400 kHz
[    1.347090] omap_i2c 40b00000.i2c: bus 1 rev0.12 at 400 kHz
[    1.380973] omap_i2c 2000000.i2c: bus 2 rev0.12 at 400 kHz
[    1.387157] omap_i2c 2010000.i2c: bus 3 rev0.12 at 400 kHz
[    1.393314] omap_i2c 2020000.i2c: bus 4 rev0.12 at 400 kHz
[    1.400336] ti-sci-inta 33d00000.interrupt-controller: ti,sci-dst-irq-type resource type ids not available
[    1.414274] keystone-pcie 5500000.pcie: Linked as a consumer to phy-900000.serdes.2
[    1.421993] keystone-pcie 5500000.pcie: Linked as a consumer to phy-910000.serdes.3
[    1.435127] OF: PCI: host bridge /soc0/pcie@5500000 ranges:
[    1.440723] OF: PCI:    IO 0x10020000..0x1002ffff -> 0x00000000
[    1.446649] OF: PCI:   MEM 0x10030000..0x17ffffff -> 0x10030000
[    2.452844] keystone-pcie 5500000.pcie: phy link never came up
[    2.458804] keystone-pcie 5500000.pcie: PCI host bridge to bus 0000:00
[    2.465337] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.470824] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    2.477002] pci_bus 0000:00: root bus resource [mem 0x10030000-0x17ffffff]
[    2.484292] pci 0000:00:00.0: BAR 6: assigned [mem 0x10030000-0x1003ffff pref]
[    2.491521] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    2.496902] pcieport 0000:00:00.0: limiting MRRS to 256
[    2.502344] pcieport 0000:00:00.0: Signaling PME with IRQ 31
[    2.508103] pcieport 0000:00:00.0: AER enabled with IRQ 31
[    2.514594] k3-navss-ringacc 3c000000.ringacc: Ring Accelerator probed rings:818, gp-rings[304,464] sci-dev-id:187
[    2.524946] k3-navss-ringacc 3c000000.ringacc: dma-ring-reset-quirk: enabled
[    2.531993] k3-navss-ringacc 3c000000.ringacc: RA Proxy rev. 66344100, num_proxies:64
[    2.540016] k3-navss-ringacc 2b800000.ringacc: Ring Accelerator probed rings:286, gp-rings[96,255] sci-dev-id:195
[    2.550272] k3-navss-ringacc 2b800000.ringacc: dma-ring-reset-quirk: enabled
[    2.557318] k3-navss-ringacc 2b800000.ringacc: RA Proxy rev. 66344100, num_proxies:64
[    2.566893] cadence-qspi 47040000.ospi: mt35xu512 (65536 Kbytes)
[    2.572928] 7 cmdlinepart partitions found on MTD device 47040000.ospi.0
[    2.579625] Creating 7 MTD partitions on "47040000.ospi.0":
[    2.585211] 0x000000000000-0x000000080000 : "ospi.tiboot3"
[    2.591547] 0x000000080000-0x000000280000 : "ospi.tispl"
[    2.597487] 0x000000280000-0x000000680000 : "ospi.u-boot"
[    2.603483] 0x000000680000-0x0000006a0000 : "ospi.env"
[    2.609229] 0x0000006a0000-0x0000006c0000 : "ospi.env.backup"
[    2.615564] 0x0000006c0000-0x0000007c0000 : "ospi.sysfw"
[    2.621451] 0x000000800000-0x000004000000 : "ospi.rootfs"
[    2.736635] cadence-qspi 47040000.ospi: Cadence QSPI NOR probe failed -517
[    2.745210] spi spi2.0: not using DMA for McSPI (-19)
[    2.750681] m25p80 spi2.0: n25q128a13 (16384 Kbytes)
[    2.756916] am65-cpsw-nuss 46000000.ethernet: rx-flow-id-base is not set -22
[    2.764099] am65-cpsw-nuss 46000000.ethernet: initializing am65 cpsw nuss version 0x6BA07100, cpsw version 0x6BA80100 Ports: 2
[    2.775692] UDMA has not been proped
[    2.779331] am65-cpsw-nuss 46000000.ethernet: Failed to request tx dma channel -517
[    2.788980] am65-cpts 310d0000.cpts: CPTS ver 0x4e8a2109, freq:200000000, add_val:4
[    2.798902] mmc0: Unknown controller version (4). You may experience problems.
[    2.848408] mmc0: SDHCI controller on 4f80000.sdhci [4f80000.sdhci] using ADMA 64-bit
[    2.861071] mmc1: Unknown controller version (4). You may experience problems.
[    2.912425] mmc1: SDHCI controller on 4fa0000.sdhci [4fa0000.sdhci] using ADMA 64-bit
[    2.924774] mmc0: new HS200 MMC card at address 0001
[    2.930150] mmcblk0: mmc0:0001 R1J56L 14.7 GiB
[    2.935103] mmcblk0boot0: mmc0:0001 R1J56L partition 1 8.00 MiB
[    2.944119] mmcblk0boot1: mmc0:0001 R1J56L partition 2 8.00 MiB
[    2.951573] mmcblk0rpmb: mmc0:0001 R1J56L partition 3 4.00 MiB
[    2.991537] omap-mailbox 31f80000.mailbox: omap mailbox rev 0x66fc5900
[    2.998100] mmc1: new high speed SDHC card at address 59b4
[    3.003880] mmcblk1: mmc1:59b4       14.9 GiB
[    3.009182] omap-mailbox 31f81000.mailbox: omap mailbox rev 0x66fc5900
[    3.015881]  mmcblk1: p1 p2
[    3.020266] ti-udma 31150000.udmap: Channels: 267 (tchan: 120, echan: 32, rchan: 150, rflow: 300)
[    3.044309] ti-udma 285c0000.udmap: Channels: 96 (tchan: 48, echan: 0, rchan: 48, rflow: 96)
[    3.059301] cadence-qspi 47040000.ospi: mt35xu512 (65536 Kbytes)
[    3.065341] 7 cmdlinepart partitions found on MTD device 47040000.ospi.0
[    3.072039] Creating 7 MTD partitions on "47040000.ospi.0":
[    3.077616] 0x000000000000-0x000000080000 : "ospi.tiboot3"
[    3.083926] 0x000000080000-0x000000280000 : "ospi.tispl"
[    3.089899] 0x000000280000-0x000000680000 : "ospi.u-boot"
[    3.096010] 0x000000680000-0x0000006a0000 : "ospi.env"
[    3.101727] 0x0000006a0000-0x0000006c0000 : "ospi.env.backup"
[    3.108083] 0x0000006c0000-0x0000007c0000 : "ospi.sysfw"
[    3.114003] 0x000000800000-0x000004000000 : "ospi.rootfs"
[    3.122303] am65-cpsw-nuss 46000000.ethernet: rx-flow-id-base is not set -22
[    3.129492] am65-cpsw-nuss 46000000.ethernet: initializing am65 cpsw nuss version 0x6BA07100, cpsw version 0x6BA80100 Ports: 2
[    3.147263] am65-cpsw-nuss 46000000.ethernet: set new flow-id-base 48
[    3.158568] am65-cpsw-nuss 46000000.ethernet: CPTS ver 0x4e8a2109, freq:200000000, add_val:4
[    3.167037] am65-cpsw-nuss 46000000.ethernet: initialized cpsw ale version 0.4
[    3.174256] am65-cpsw-nuss 46000000.ethernet: ALE Table size 64
[    3.216412] am65-cpsw-nuss 46000000.ethernet: davinci mdio revision 9.7, bus freq 1000000
[    3.224586] libphy: 46000000.ethernet: probed
[    3.232017] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    3.239035] hctosys: unable to open rtc device (rtc0)
[    3.341458] ALSA device list:
[    3.344442]   No soundcards found.
[    3.409589] EXT4-fs (mmcblk1p2): mounted filesystem with ordered data mode. Opts: (null)
[    3.417753] VFS: Mounted root (ext4 filesystem) on device 179:130.
[    3.426108] devtmpfs: mounted
[    3.429286] Freeing unused kernel memory: 448K
[    3.769226] systemd[1]: System time before build time, advancing clock.
[    3.870530] NET: Registered protocol family 10
[    3.876115] Segment Routing with IPv6
[    3.927910] systemd[1]: systemd 234 running in system mode. (+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN default-hierarchy=hybrid)
[    3.949081] systemd[1]: Detected architecture arm64.

Welcome to Arago 2018.10!

[    3.969830] systemd[1]: Set hostname to <am65xx-evm>.
[    4.330602] random: systemd: uninitialized urandom read (16 bytes read)
[    4.337449] systemd[1]: Listening on udev Kernel Socket.
[  OK  ] Listening on udev Kernel Socket.
[    4.352585] random: systemd: uninitialized urandom read (16 bytes read)
[    4.360984] systemd[1]: Created slice User and Session Slice.
[  OK  ] Created slice User and Session Slice.
[    4.380524] random: systemd: uninitialized urandom read (16 bytes read)
[    4.393321] systemd[1]: Listening on Process Core Dump Socket.
[  OK  ] Listening on Process Core Dump Socket.
[    4.413549] systemd[1]: Created slice System Slice.
[  OK  ] Created slice System Slice.
[    4.428779] systemd[1]: Listening on Syslog Socket.
[  OK  ] Listening on Syslog Socket.
[    4.444582] systemd[1]: Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket (/dev/log).
[    4.461394] systemd[1]: Created slice system-getty.slice.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Listening on Journal Socket.
         Mounting Kernel Debug File System...
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Created slice system-serial\x2dgetty.slice.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Reached target Swap.
[  OK  ] Reached target Slices.
         Starting Load Kernel Modules...
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[    4.594917] cryptodev: loading out-of-tree module taints kernel.
[    4.604427] cryptodev: driver 1.9 loaded.
[  OK  ] Reached target Paths.
         Mounting Huge Pages File System...
         Mounting POSIX Message Queue File System...
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
         Starting Create list of required st…ce nodes for the current kernel...
[  OK  ] Listening on Network Service Netlink Socket.
[  OK  ] Reached target Remote File Systems.
         Starting Remount Root and Kernel File Systems...
[    4.743752] EXT4-fs (mmcblk1p2): re-mounted. Opts: (null)
         Mounting Temporary Directory (/tmp)...
         Starting Journal Service...
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted Temporary Directory (/tmp).
[  OK  ] Started Load Kernel Modules.
[  OK  ] Started Create list of required sta…vice nodes for the current kernel.
[  OK  ] Started Remount Root and Kernel File Systems.
[  OK  ] Started Journal Service.
         Starting Create System Users...
         Starting Rebuild Hardware Database...
         Starting Flush Journal to Persistent Storage...
         Starting Apply Kernel Variables...
         Mounting Kernel Configuration File System...
[  OK  ] Mounted Kernel Configuration File System.
[  OK  ] Started Create System Users.
[  OK  ] Started Apply Kernel Variables.
[    5.045043] systemd-journald[151]: Received request to flush runtime journal from PID 1
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Flush Journal to Persistent Storage.
[  OK  ] Started Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
         Mounting /var/volatile...
         Mounting /media/ram...
         Starting udev Kernel Device Manager...
[  OK  ] Mounted /var/volatile.
[  OK  ] Mounted /media/ram.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems.
         Starting Rebuild Dynamic Linker Cache...
         Starting Create Volatile Files and Directories...
         Starting Rebuild Journal Catalog...
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Network Time Synchronization...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started Rebuild Journal Catalog.
[  OK  ] Started Rebuild Dynamic Linker Cache.
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Started Network Time Synchronization.
[  OK  ] Reached target System Time Synchronized.
[  OK  ] Started Rebuild Hardware Database.
         Starting udev Coldplug all Devices...
         Starting Update is Completed...
[  OK  ] Started Update is Completed.
[  OK  ] Started udev Coldplug all Devices.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Listening on dropbear.socket.
[  OK  ] Started Daily rotation of log files.
[  OK  ] Reached target Timers.
         Starting Network Service...
[  OK  ] Listening on RPCbind Server Activation Socket.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[    7.924889] input: gpio-decoder as /devices/platform/gpio-decoder/input/input1
[  OK  ] Reached target Basic System.
[  OK  ] Started strongSwan IPsec IKEv1/IKEv2 daemon using ipsec.conf.
         Starting Login Service...
         Starting rc.pvr.service...
         Starting TI IPC Daemon...
         Starting RPC Bind Service...
         Starting Print notice about GPLv3 packages...
[    8.104702] [drm] Initialized pvr 1.17.4948957 20110701 for 7000000.gpu on minor 0
[  OK  ] Started System Logging Service.
[  OK  ] Started Periodic Command Scheduler.
[    8.165829] CAN device driver interface
[  OK  ] Started Job spooling tools.
[  OK  ] Started Kernel Logging Service.
[  OK  ] Reached target Containers.
[  OK  ] Started Hardware RNG Entropy Gatherer Daemon.
[  OK  ] Started D-Bus System Message Bus.
[    8.222155] m_can 40528000.mcan: m_can device registered (irq=10, version=32)
[    8.248631] m_can 40568000.mcan: m_can device registered (irq=12, version=32)
[    8.294305] random: crng init done
[    8.297962] random: 7 urandom warning(s) missed due to ratelimiting
[    8.352761] PVR_K: UM DDK-(4948957) and KM DDK-(4948957) match. [ OK ]
[    8.397475] omap_rng 4e10000.trng: Random Number Generator ver. 230b34c
[    8.416797] pruss b000000.icssg: creating PRU cores and other child platform devices
[    8.429176] remoteproc remoteproc0: b034000.pru is available
[    8.435084] pru-rproc b034000.pru: PRU rproc node /soc0/pruss_soc_bus@b026004/icssg@0/pru@34000 probed successfully
[    8.437484] at24 0-0050: 131072 byte 24c1024 EEPROM, writable, 1 bytes/write
[    8.462671] remoteproc remoteproc1: b004000.rtu is available
[    8.470175] pru-rproc b004000.rtu: PRU rproc node /soc0/pruss_soc_bus@b026004/icssg@0/rtu@4000 probed successfully
[    8.489998] remoteproc remoteproc2: b038000.pru is available
[    8.498556] pru-rproc b038000.pru: PRU rproc node /soc0/pruss_soc_bus@b026004/icssg@0/pru@38000 probed successfully
[    8.512120] remoteproc remoteproc3: b006000.rtu is available
[    8.517914] pru-rproc b006000.rtu: PRU rproc node /soc0/pruss_soc_bus@b026004/icssg@0/rtu@6000 probed successfully
[    8.572446] davinci_mdio b032400.mdio: davinci mdio revision 1.7, bus freq 1000000
[    8.580088] libphy: b032400.mdio: probed
[    8.676656] davinci_mdio b032400.mdio: phy[0]: device b032400.mdio:00, driver TI DP83867
[    8.686302] davinci_mdio b032400.mdio: phy[3]: device b032400.mdio:03, driver TI DP83867
[    8.696255] pruss b100000.icssg: creating PRU cores and other child platform devices
[    8.706513] remoteproc remoteproc4: b134000.pru is available
[    8.712251] pru-rproc b134000.pru: PRU rproc node /soc0/pruss_soc_bus@b126004/icssg@0/pru@34000 probed successfully
[    8.723440] remoteproc remoteproc5: b104000.rtu is available
[    8.729188] pru-rproc b104000.rtu: PRU rproc node /soc0/pruss_soc_bus@b126004/icssg@0/rtu@4000 probed successfully
[    8.740290] remoteproc remoteproc6: b138000.pru is available
[    8.746025] pru-rproc b138000.pru: PRU rproc node /soc0/pruss_soc_bus@b126004/icssg@0/pru@38000 probed successfully
[    8.757207] remoteproc remoteproc7: b106000.rtu is available
[    8.762944] pru-rproc b106000.rtu: PRU rproc node /soc0/pruss_soc_bus@b126004/icssg@0/rtu@6000 probed successfully
[    8.789114] icssg-prueth pruss0_eth: TI PRU ethernet driver initialized: dual EMAC mode
[    8.812480] davinci_mdio b132400.mdio: davinci mdio revision 1.7, bus freq 1000000
[    8.820097] libphy: b132400.mdio: probed
[    8.832483] davinci_mdio b132400.mdio: phy[0]: device b132400.mdio:00, driver TI DP83867
[    8.842030] davinci_mdio b132400.mdio: phy[3]: device b132400.mdio:03, driver TI DP83867
[    8.851991] pruss b200000.icssg: creating PRU cores and other child platform devices
[    8.864218] remoteproc remoteproc8: b234000.pru is available
[    8.870265] pru-rproc b234000.pru: PRU rproc node /soc0/pruss_soc_bus@b226004/icssg@0/pru@34000 probed successfully
[    8.882904] remoteproc remoteproc9: b204000.rtu is available
[    8.888767] pru-rproc b204000.rtu: PRU rproc node /soc0/pruss_soc_bus@b226004/icssg@0/rtu@4000 probed successfully
[    8.900609] remoteproc remoteproc10: b238000.pru is available
[    8.906576] pru-rproc b238000.pru: PRU rproc node /soc0/pruss_soc_bus@b226004/icssg@0/pru@38000 probed successfully
[    8.920517] remoteproc remoteproc11: b206000.rtu is available
[    8.929990] pru-rproc b206000.rtu: PRU rproc node /soc0/pruss_soc_bus@b226004/icssg@0/rtu@6000 probed successfully
[    8.944189] icssg-prueth pruss1_eth: TI PRU ethernet driver initialized: dual EMAC mode
[    9.036513] davinci_mdio b232400.mdio: davinci mdio revision 1.7, bus freq 1000000
[    9.044356] libphy: b232400.mdio: probed
[    9.103271] davinci_mdio b232400.mdio: phy[0]: device b232400.mdio:00, driver TI DP83867
[    9.111662] davinci_mdio b232400.mdio: phy[3]: device b232400.mdio:03, driver TI DP83867
[    9.131411] k3_r5_rproc soc0:mcu_r5f_cluster@41000000: creating child devices for R5F cores
[    9.151034] icssg-prueth pruss2_eth: TI PRU ethernet driver initialized: dual EMAC mode
[    9.161425] platform 41000000.r5f: boot_vector = 0x41c00000, cfg = 0x1903 ctrl = 0x0 stat = 0x104
[    9.177020] platform 41000000.r5f: assigned reserved memory node r5f0-dma-memory@9c000000
[    9.188192] remoteproc remoteproc12: 41000000.r5f is available
[    9.413186] remoteproc remoteproc12: powering up 41000000.r5f
[    9.419184] remoteproc remoteproc12: Booting fw image am65x-mcu-r5f0-fw, size 3864336
[    9.435687] platform 41000000.r5f: booting R5F core using boot addr = 0x0
[    9.446558] remoteproc remoteproc12: registered virtio0 (type 7)
[    9.454132] remoteproc remoteproc12: remote processor 41000000.r5f is now up
[  OK  ] Started TI IPC Daemon.
[  OK  ] Started RPC Bind Service.
[    9.478330] NET: Registered protocol family 15
[  OK  ] Found device /dev/ttyS2.
[  OK  ] Started Network Service.
[    9.547443] am65-cpsw-nuss 46000000.ethernet: Phy mode driver DT not found
[  OK  ] Started Login Service.
[  OK  ] Reached target Network.
         Starting Permit User Sessions...
[    9.576373] TI DP83867 k3-cpsw-mdio:00: attached PHY driver [TI DP83867] (mii_bus:phy_addr=k3-cpsw-mdio:00, irq=POLL)
         Starting Simple Network Management Protocol (SNMP) Daemon....
[    9.591094] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
         Starting Lightning Fast Webserver With Light System Requirements...
         Starting Network Name Resolution...
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Serial Getty on ttyS2.
[  OK  ] Started Getty on tty1.
[  OK  ] Reached target Login Prompts.
         Starting Synchronize System and HW clocks...
[FAILED] Failed to start Synchronize System and HW clocks.
See 'systemctl status sync-clocks.service' for details.
[  OK  ] Started Lightning Fast Webserver With Light System Requirements.
[    9.938336] Initializing XFRM netlink socket
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Started NFS status monitor for NFSv2/3 locking..
[   10.192249] remoteproc remoteproc0: powering up b034000.pru
[   10.231033] remoteproc remoteproc0: Booting fw image ti-pruss/am65x-pru0-prueth-fw.elf, size 15120
[   10.242754] remoteproc remoteproc0: remote processor b034000.pru is now up
[   10.252656] remoteproc remoteproc1: powering up b004000.rtu
[   10.265687] remoteproc remoteproc1: Booting fw image ti-pruss/am65x-rtu0-prueth-fw.elf, size 16552
[   10.276320] remoteproc remoteproc1: remote processor b004000.rtu is now up
[   10.287610] icssg-prueth pruss0_eth: setting rx flow id 150
[   10.297778] net eth1: started
[   10.301422] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   10.331919] virtio_rpmsg_bus virtio0: rpmsg host is online
[   10.337657] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d
[   10.357963] remoteproc remoteproc6: powering up b138000.pru
[   10.376343] remoteproc remoteproc6: Booting fw image ti-pruss/am65x-pru1-prueth-fw.elf, size 15040
[   10.388098] remoteproc remoteproc6: remote processor b138000.pru is now up
[   10.395765] remoteproc remoteproc7: powering up b106000.rtu
[   10.431522] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-rtu1-prueth-fw.elf, size 16568
[   10.441561] remoteproc remoteproc7: remote processor b106000.rtu is now up
[   10.450621] icssg-prueth pruss1_eth: setting rx flow id 151
[   10.457513] net eth4: started
[   10.460898] IPv6: ADDRCONF(NETDEV_UP): eth4: link is not ready
[   10.461682] usbcore: registered new interface driver usbfs
[   10.461729] usbcore: registered new interface driver hub
[   10.461860] usbcore: registered new device driver usb
[   10.489655] remoteproc remoteproc4: powering up b134000.pru
[   10.495649] remoteproc remoteproc4: Booting fw image ti-pruss/am65x-pru0-prueth-fw.elf, size 15120
[   10.504857] remoteproc remoteproc4: remote processor b134000.pru is now up
[   10.511860] remoteproc remoteproc5: powering up b104000.rtu
[   10.517605] remoteproc remoteproc5: Booting fw image ti-pruss/am65x-rtu0-prueth-fw.elf, size 16552
[   10.526810] remoteproc remoteproc5: remote processor b104000.rtu is now up
[   10.535774] icssg-prueth pruss1_eth: setting rx flow id 152
[   10.542484] net eth3: started
[   10.545756] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[   10.557795] remoteproc remoteproc2: powering up b038000.pru
[   10.563553] remoteproc remoteproc2: Booting fw image ti-pruss/am65x-pru1-prueth-fw.elf, size 15040
[   10.572591] remoteproc remoteproc2: remote processor b038000.pru is now up
[   10.579530] remoteproc remoteproc3: powering up b006000.rtu
[   10.585381] remoteproc remoteproc3: Booting fw image ti-pruss/am65x-rtu1-prueth-fw.elf, size 16568
[   10.594404] remoteproc remoteproc3: remote processor b006000.rtu is now up
[   10.597765] am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
[   10.609265] icssg-prueth pruss0_eth: setting rx flow id 153
[   10.615934] net eth2: started
[   10.619184] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[   10.647427] remoteproc remoteproc10: powering up b238000.pru
[   10.653658] remoteproc remoteproc10: Booting fw image ti-pruss/am65x-pru1-prueth-fw.elf, size 15040
[   10.662853] remoteproc remoteproc10: remote processor b238000.pru is now up
[   10.669934] remoteproc remoteproc11: powering up b206000.rtu
[   10.675981] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[   10.675987] remoteproc remoteproc11: Booting fw image ti-pruss/am65x-rtu1-prueth-fw.elf, size 16568
[   10.676021] remoteproc remoteproc11: remote processor b206000.rtu is now up
[   10.678041] icssg-prueth pruss2_eth: setting rx flow id 154
[   10.679121] net eth6: started
[   10.679800] IPv6: ADDRCONF(NETDEV_UP): eth6: link is not ready
[   10.712467] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
[   10.720937] xhci-hcd xhci-hcd.0.auto: hcc params 0x0298fe6d hci version 0x110 quirks 0x00010010
[   10.729784] xhci-hcd xhci-hcd.0.auto: irq 298, io mem 0x04010000
[   10.736033] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[   10.736366] remoteproc remoteproc8: powering up b234000.pru
[   10.736521] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-pru0-prueth-fw.elf, size 15120
[   10.736560] remoteproc remoteproc8: remote processor b234000.pru is now up
[   10.736578] remoteproc remoteproc9: powering up b204000.rtu
[   10.736645] remoteproc remoteproc9: Booting fw image ti-pruss/am65x-rtu0-prueth-fw.elf, size 16552
[   10.736672] remoteproc remoteproc9: remote processor b204000.rtu is now up
[   10.738687] icssg-prueth pruss2_eth: setting rx flow id 155
[   10.739782] net eth5: started
[   10.739987] IPv6: ADDRCONF(NETDEV_UP): eth5: link is not ready
[   10.800263] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   10.807517] usb usb1: Product: xHCI Host Controller
[   10.812447] usb usb1: Manufacturer: Linux 4.14.79-gbde58ab01e xhci-hcd
[   10.819017] usb usb1: SerialNumber: xhci-hcd.0.auto
[   10.824737] hub 1-0:1.0: USB hub found
[   10.828642] hub 1-0:1.0: 1 port detected
[   10.833290] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[   10.838871] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
[   10.846610] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0  SuperSpeed
[   10.853355] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[   10.861620] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[   10.868443] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   10.875731] usb usb2: Product: xHCI Host Controller
[   10.880643] usb usb2: Manufacturer: Linux 4.14.79-gbde58ab01e xhci-hcd
[   10.887205] usb usb2: SerialNumber: xhci-hcd.0.auto
[   10.892835] hub 2-0:1.0: USB hub found
[   10.896664] hub 2-0:1.0: 1 port detected
[  OK  ] Started Simple Network Management Protocol (SNMP) Daemon..
[   11.082463] NET: Registered protocol family 44
[   11.645458] FAT-fs (mmcblk1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[   12.513969] icssg-prueth pruss1_eth eth4: Link is Up - 1Gbps/Full - flow control rx/tx
[   12.522012] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
[   12.581985] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
[   12.582663] icssg-prueth pruss1_eth eth3: Link is Up - 1Gbps/Full - flow control rx/tx
[   12.773951] icssg-prueth pruss2_eth eth5: Link is Up - 1Gbps/Full - flow control rx/tx
[   12.782001] IPv6: ADDRCONF(NETDEV_CHANGE): eth5: link becomes ready
[  OK  ] Started rc.pvr.service.
         Starting weston.service...
[  OK  ] Started weston.service.
         Starting Matrix GUI...
         Starting telnetd.service...
[  OK  ] Started telnetd.service.
         Starting thttpd.service...
[  OK  ] Started Matrix GUI.
[  OK  ] Started thttpd.service.
         Starting rng-tools.service...

 _____                    _____           _         _
|  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
|     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
|__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
              |___|                    |___|

Arago Project http://arago-project.org am65xx-evm ttyS2

Arago 2018.10 am65xx-evm ttyS2

am65xx-evm login: ***************************************************************
***************************************************************
NOTICE: This file system contains the following GPLv3 packages:
        autoconf
        bash
        binutils
        cifs-utils
        cpio
        cpp-symlinks
        cpp
        dosfstools
        elfutils
        g++-symlinks
        g++
        gawk
        gcc-symlinks
        gcc
        gdb
        gdbserver
        gettext
        glmark2
        gstreamer1.0-libav
        gzip
        hidapi
        libdw1
        libelf1
        libgdbm-compat4
        libgdbm4
        libgettextlib
        libgettextsrc
        libgmp10
        libidn11
        libmpc3
        libmpfr4
        libreadline7
        libunistring2
        m4
        make
        nettle
        parted
        python3-pycairo
        which

If you do not wish to distribute GPLv3 components please remove
the above packages prior to distribution.  This can be done using
the opkg remove command.  i.e.:
    opkg remove <package>
Where <package> is the name printed in the list above

NOTE: If the package is a dependency of another package you
      will be notified of the dependent packages.  You should
      use the --force-removal-of-dependent-packages option to
      also remove the dependent packages as well
***************************************************************
***************************************************************

 _____                    _____           _         _
|  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
|     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
|__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
              |___|                    |___|

Arago Project http://arago-project.org am65xx-evm ttyS2

Arago 2018.10 am65xx-evm ttyS2

am65xx-evm login: root
root@am65xx-evm:~# ifconfig
eth0      Link encap:Ethernet  HWaddr F4:84:4C:ED:34:C5
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth1      Link encap:Ethernet  HWaddr 70:FF:76:1D:57:BA
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth2      Link encap:Ethernet  HWaddr 70:FF:76:1D:57:BB
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth3      Link encap:Ethernet  HWaddr 70:FF:76:1D:57:BC
          inet6 addr: fe80::72ff:76ff:fe1d:57bc%4915360/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:31 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:10450 (10.2 KiB)  TX bytes:5228 (5.1 KiB)

eth4      Link encap:Ethernet  HWaddr 70:FF:76:1D:57:BD
          inet6 addr: fe80::72ff:76ff:fe1d:57bd%4915360/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:31 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:10450 (10.2 KiB)  TX bytes:5330 (5.2 KiB)

eth5      Link encap:Ethernet  HWaddr 70:FF:76:1D:52:9A
          inet6 addr: fe80::72ff:76ff:fe1d:529a%4915360/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:28 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:9448 (9.2 KiB)  TX bytes:5492 (5.3 KiB)

eth6      Link encap:Ethernet  HWaddr 70:FF:76:1D:52:9B
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1%4915360/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:171 errors:0 dropped:0 overruns:0 frame:0
          TX packets:171 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13236 (12.9 KiB)  TX bytes:13236 (12.9 KiB)

root@am65xx-evm:~# [  461.157758] Unable to handle kernel paging request at virtual address 68ead29b10
[  461.165171] Mem abort info:
[  461.167957]   Exception class = DABT (current EL), IL = 32 bits
[  461.173885]   SET = 0, FnV = 0
[  461.176950]   EA = 0, S1PTW = 0
[  461.180081] Data abort info:
[  461.182975]   ISV = 0, ISS = 0x00000046
[  461.186824]   CM = 0, WnR = 1
[  461.189808] user pgtable: 64k pages, 48-bit VAs, pgd = ffff8008411f9400
[  461.196435] [00000068ead29b10] *pgd=00000008c4bf0003, *pud=00000008c4bf0003, *pmd=0000000000000000
[  461.205410] Internal error: Oops: 96000046 [#1] PREEMPT SMP
[  461.210980] Modules linked in: sha512_arm64 md5 des_generic cbc rpmsg_proto xhci_plat_hcd xhci_hcd usbcore virtio_rpmsg_bus rpmsg_core dwc3 udc_core ti_am335x_adc usb_common kfifo_buf xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo ti_k3_r5_remoteproc pci_endpoint_test at24 pruss_soc_bus omap_rng rng_core sa2ul authenc phy_omap_usb2 m_can can_dev ti_am335x_tscadc dwc3_of_simple pvrsrvkm(O) icssg_prueth pru_rproc pruss pruss_intc crc32_ce crct10dif_ce remoteproc gpio_decoder input_polldev sch_fq_codel cryptodev(O) ipv6
[  461.259323] CPU: 0 PID: 7 Comm: ksoftirqd/0 Tainted: G           O    4.14.79-gbde58ab01e #1
[  461.267742] Hardware name: Texas Instruments AM654 Base Board (DT)
[  461.273908] task: ffff8008401c4800 task.stack: ffff000008e80000
[  461.279845] PC is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
[  461.285938] LR is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
[  461.292017] pc : [<ffff000001492620>] lr : [<ffff000001492620>] pstate: 60000145
[  461.299394] sp : ffff000008e8fc60
[  461.302699] x29: ffff000008e8fc60 x28: ffff0000087e0110
[  461.308001] x27: ffff800842ccd9c0 x26: 00000068ead29b00
[  461.313303] x25: ffff80084004d010 x24: ffff800842ccdae0
[  461.318604] x23: 0000000000000040 x22: 0000000040000272
[  461.323907] x21: ffff800842ccd900 x20: ffff800842ccd000
[  461.329210] x19: 0000000000000000 x18: 0000ffffe178b49d
[  461.334512] x17: 0000ffff8cd4d2d0 x16: ffff0000087e18d8
[  461.339813] x15: 0000000000000000 x14: 0000000000000000
[  461.345114] x13: 0000000000000000 x12: ffff800845dbbb80
[  461.350416] x11: 0000000000000001 x10: 00000000ffffffef
[  461.355719] x9 : 0000000000000000 x8 : 0000000000000000
[  461.361020] x7 : 0000400000000000 x6 : ffff80087ff30000
[  461.366323] x5 : 0000000000000000 x4 : ffff800845dbbbb0
[  461.371624] x3 : ffff800845dbbb90 x2 : 0000000000000000
[  461.376926] x1 : 0000000000000040 x0 : 0000000000000000
[  461.382231] Process ksoftirqd/0 (pid: 7, stack limit = 0xffff000008e80000)
[  461.389088] Call trace:
[  461.391529] Exception stack(0xffff000008e8fb20 to 0xffff000008e8fc60)
[  461.397957] fb20: 0000000000000000 0000000000000040 0000000000000000 ffff800845dbbb90
[  461.405772] fb40: ffff800845dbbbb0 0000000000000000 ffff80087ff30000 0000400000000000
[  461.413586] fb60: 0000000000000000 0000000000000000 00000000ffffffef 0000000000000001
[  461.421401] fb80: ffff800845dbbb80 0000000000000000 0000000000000000 0000000000000000
[  461.429214] fba0: ffff0000087e18d8 0000ffff8cd4d2d0 0000ffffe178b49d 0000000000000000
[  461.437029] fbc0: ffff800842ccd000 ffff800842ccd900 0000000040000272 0000000000000040
[  461.444844] fbe0: ffff800842ccdae0 ffff80084004d010 00000068ead29b00 ffff800842ccd9c0
[  461.452659] fc00: ffff0000087e0110 ffff000008e8fc60 ffff000001492620 ffff000008e8fc60
[  461.460473] fc20: ffff000001492620 0000000060000145 ffff800842ccd900 0000000040000272
[  461.468289] fc40: ffffffffffffffff ffff000008449a60 ffff000008e8fc60 ffff000001492620
[  461.476115] [<ffff000001492620>] emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
[  461.483246] [<ffff00000869decc>] net_rx_action+0xf4/0x2b0
[  461.488639] [<ffff0000080811ec>] __do_softirq+0x12c/0x228
[  461.494028] [<ffff0000080aca60>] run_ksoftirqd+0x40/0x68
[  461.499333] [<ffff0000080ca490>] smpboot_thread_fn+0x158/0x240
[  461.505156] [<ffff0000080c66a4>] kthread+0x12c/0x130
[  461.510111] [<ffff000008084690>] ret_from_fork+0x10/0x18
[  461.515413] Code: f94037a5 f9400300 aa0503e1 95bee851 (f9000b54)
[  461.521502] ---[ end trace 3c4c93623230318a ]---
[  461.526110] Kernel panic - not syncing: Fatal exception in interrupt
[  461.532449] SMP: stopping secondary CPUs
[  461.536365] Kernel Offset: disabled
[  461.539844] CPU features: 0x080200c
[  461.543321] Memory Limit: none
[  461.546369] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

Broadcast message from systemd-journald@am65xx-evm (Sun 2018-12-16 03:30:05 UTC):

kernel[276]: [  461.205410] Internal error: Oops: 96000046 [#1] PREEMPT SMP

  • Hello Dominic,

    The AM65x PRUETH PRU firmware and Linux driver are under active development. We are still working through bugs, so expect PRUETH behavior to improve going forward. The soon-to-be-released Linux Processor SDK 5.3 will have some bug fixes. AM65x will continue to see bug fixes in later SDKs.

    LCPD-14189 also details a situation that results in kernel crashes, but I suspect that you might be experiencing a different issue. Let's take a look so we know if I need to file a separate bug report.

    Are your ethernet ports on the same subnet? If so, I would expect to see errors if you had not bonded the ports. Reference the "network topologies" subsection of TI's documentation for PRU-ICSS Ethernet.

    Regards,

    Nick

  • Hello Nick,

    the three Ethernet ports are on different subnets, e.g. 192.168.4.0/24, 192.168.5.0/24 and 192.168.6.0/24. This is of course no meaningful usecase, but it is a simple setup that quickly exhibits the kernel panic. I fully understand the problems indicated in the note in the Processor SDK docs Dual MAC Mode chapter, but even in that case the driver shouldn't cause a kernel panic.

    The kernel panic is always at the same location but the offending address is seemingly "random".

    I read on the forum that there's a new processor SDK RTOS likely to be released by the end of this week - do you know if there'll be a new Linux SDK some time soon, too?

    Regards,

    Dominic
  • Hello Dominic,

    You are correct, we should not see kernel panics based on what you are doing.

    The Linux SDK will come out soon (around the same time as the RTOS SDK).

    Can you retry on Linux SDK 5.3 when it comes out? If you still see issues on SDK 5.3, then we can discuss the details so that our development team can replicate your results and fix it in the future.

    Regards,
    Nick
  • Hello Dominic,

    FYI, Linux SDK 5.3 for AM65xx is available for download here: software-dl.ti.com/.../index_FDS.html

    Regards,
    Nick
  • Hello Dominic,

    I am going to close the thread for record keeping purposes. Please keep us updated by replying to the thread when you are able to try out SDK 5.3.

    Regards,
    Nick
  • Hello Nick,

    we've tested again with processor SDK 05.03, both with the "vanilla" Linux and the "RT" kernel, and the problem still exists in both variants.

    We've further noticed that the driver crashes when copying some files via scp from our host. It appears to depend on exactly what we're trying to copy, i.e. we get the crash when copying our /lib/modules/ tree, but not when copying some similarly sized random files.

    I've attached a couple of log files that show crashes on the IDK for both scenarios that we found.

    "multi_eth" means we had three PRU ethernet interfaces (eth1-eth3) connected to the same switch with separate IP networks assigned to each interface. On several occasions it crashed even before we had a chance to assign an IP address.

    "scp" means we had only one PRU ethernet interface (eth1) connected to our host via a switch.

     _____                    _____           _         _
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
                  |___|                    |___|
    
    Arago Project http://arago-project.org am65xx-evm ttyS2
    
    Arago 2018.10 am65xx-evm ttyS2
    
    am65xx-evm login: root
    root@am65xx-evm:~# ifconfig eth1[   34.649896] Unable to handle kernel paging request at virtual address 4a8cab484
    [   34.649900] Unable to handle kernel paging request at virtual address 4a542a4d8
    [   34.649909] Mem abort info:
    [   34.649912]   Exception class = DABT (current EL), IL = 32 bits
    [   34.649915]   SET = 0, FnV = 0
    [   34.649917]   EA = 0, S1PTW = 0
    [   34.649919] Data abort info:
    [   34.649922]   ISV = 0, ISS = 0x00000046
    [   34.649924]   CM = 0, WnR = 1
    [   34.649930] user pgtable: 64k pages, 48-bit VAs, pgd = ffff800879cfd600
    [   34.649933] [00000004a542a4d8] *pgd=00000008f4a10003, *pud=00000008f4a10003, *pmd=0000000000000000
    [   34.649945] Internal error: Oops: 96000046 [#1] PREEMPT SMP
    [   34.649948] Modules linked in: rpmsg_proto sha512_arm64 sha1_generic md5 virtio_rpmsg_bus rpmsg_core des_generic cbc xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo xhci_plat_hcd xhci_hcd usbcore crc32_ce ti_k3_r5_remoteproc dwc3 udc_core usb_common ti_am335x_adc kfifo_buf at24 pruss_soc_bus pci_endpoint_test pvrsrvkm(O) omap_rng sa2ul rng_core authenc phy_omap_usb2 m_can can_dev ti_am335x_tscadc dwc3_of_simple icssg_prueth pru_rproc pruss pruss_intc gpio_decoder remoteproc input_polldev sch_fq_codel cryptodev(O) ipv6
    [   34.650035] CPU: 0 PID: 780 Comm: irq/371-pruss0_ Tainted: G           O    4.14.79-rt47-ga72bf1418c #1
    [   34.650037] Hardware name: Texas Instruments AM654 Base Board (DT)
    [   34.650041] task: ffff80087a4e1b00 task.stack: ffff000012440000
    [   34.650069] PC is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [   34.650090] LR is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [   34.650093] pc : [<ffff000001062a68>] lr : [<ffff000001062a68>] pstate: 60000145
    [   34.650095] sp : ffff00001244fbf0
    [   34.650097] x29: ffff00001244fbf0 x28: ffff0000087b0110
    [   34.650103] x27: ffff8008781d8a40 x26: 00000004a542a4c8
    [   34.650108] x25: ffff80087c072410 x24: ffff8008781d8ba0
    [   34.650113] x23: 0000000000000040 x22: 0000000040000160
    [   34.650118] x21: ffff8008781d8980 x20: ffff8008781d8000
    [   34.650123] x19: 0000000000000000 x18: 00000000000001d9
    [   34.650128] x17: ffff0000087b1888 x16: ffff0000087b1888
    [   34.650133] x15: 0000000000000000 x14: 0000000000000000
    [   34.650138] x13: 0000000000000000 x12: ffff80087433bc00
    [   34.650142] x11: 0000000000000001 x10: 00000000fffffff4
    [   34.650147] x9 : 0000000000000000 x8 : 0000000000000000
    [   34.650152] x7 : 0008000000000000 x6 : 0000000099f81925
    [   34.650157] x5 : 0000000000000000 x4 : ffff80087433bc38
    [   34.650162] x3 : ffff80087433bc10 x2 : 0000000000000000
    [   34.650167] x1 : 0000000000000040 x0 : 0000000000000000
    [   34.650173] Process irq/371-pruss0_ (pid: 780, stack limit = 0xffff000012440000)
    [   34.650177] Call trace:
    [   34.650182] Exception stack(0xffff00001244fab0 to 0xffff00001244fbf0)
    [   34.650187] faa0:                                   0000000000000000 0000000000000040
    [   34.650192] fac0: 0000000000000000 ffff80087433bc10 ffff80087433bc38 0000000000000000
    [   34.650197] fae0: 0000000099f81925 0008000000000000 0000000000000000 0000000000000000
    [   34.650203] fb00: 00000000fffffff4 0000000000000001 ffff80087433bc00 0000000000000000
    [   34.650208] fb20: 0000000000000000 0000000000000000 ffff0000087b1888 ffff0000087b1888
    [   34.650213] fb40: 00000000000001d9 0000000000000000 ffff8008781d8000 ffff8008781d8980
    [   34.650218] fb60: 0000000040000160 0000000000000040 ffff8008781d8ba0 ffff80087c072410
    [   34.650224] fb80: 00000004a542a4c8 ffff8008781d8a40 ffff0000087b0110 ffff00001244fbf0
    [   34.650229] fba0: ffff000001062a68 ffff00001244fbf0 ffff000001062a68 0000000060000145
    [   34.650235] fbc0: ffff8008781d8980 0000000040000160 ffffffffffffffff ffff000008428c28
    [   34.650238] fbe0: ffff00001244fbf0 ffff000001062a68
    [   34.650259] [<ffff000001062a68>] emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [   34.650274] [<ffff00000866c70c>] net_rx_action+0x18c/0x320
    [   34.650286] [<ffff0000080a1c44>] do_current_softirqs+0x1b4/0x248
    [   34.650292] [<ffff0000080a1d2c>] __local_bh_enable+0x54/0x68
    [   34.650302] [<ffff0000080eeb24>] irq_forced_thread_fn+0x64/0x90
    [   34.650308] [<ffff0000080eee0c>] irq_thread+0xfc/0x198
    [   34.650314] [<ffff0000080bc1a4>] kthread+0x12c/0x130
    [   34.650321] [<ffff000008084720>] ret_from_fork+0x10/0x18
    [   34.650330] Code: f94037a5 f9400300 aa0503e1 95cf23b7 (f9000b54)
    [   34.650335] ---[ end trace 0000000000000002 ]---
    [   34.650380] genirq: exiting task "irq/371-pruss0_" (780) is an active IRQ thread (irq 371)
    [   34.650509] NOHZ: local_softirq_pending 80
    [   34.654257] NOHZ: local_softirq_pending 80
    [   34.658244] NOHZ: local_softirq_pending 80
    [   34.662245] NOHZ: local_softirq_pending 80
    [   34.666242] NOHZ: local_softirq_pending 80
    [   34.670245] NOHZ: local_softirq_pending 80
    [   34.674244] NOHZ: local_softirq_pending 80
    [   34.677385] NOHZ: local_softirq_pending 80
    [   34.678261] NOHZ: local_softirq_pending 80
    [   34.682247] NOHZ: local_softirq_pending 80
    [   35.086331] Mem abort info:
    [   35.089116]   Exception class = DABT (current EL), IL = 32 bits
    [   35.095021]   SET = 0, FnV = 0
    [   35.098065]   EA = 0, S1PTW = 0
    [   35.101196] Data abort info:
    [   35.104067]   ISV = 0, ISS = 0x00000046
    [   35.107892]   CM = 0, WnR = 1
    [   35.110852] user pgtable: 64k pages, 48-bit VAs, pgd = ffff80087b04d200
    [   35.117452] [00000004a8cab484] *pgd=00000008fb790003, *pud=00000008fb790003, *pmd=0000000000000000
    [   35.126400] Internal error: Oops: 96000046 [#2] PREEMPT SMP
    [   35.126403] Modules linked in: rpmsg_proto sha512_arm64 sha1_generic md5 virtio_rpmsg_bus rpmsg_core des_generic cbc xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo xhci_plat_hcd xhci_hcd usbcore crc32_ce ti_k3_r5_remoteproc dwc3 udc_core usb_common ti_am335x_adc kfifo_buf at24 pruss_soc_bus pci_endpoint_test pvrsrvkm(O) omap_rng sa2ul rng_core authenc phy_omap_usb2 m_can can_dev ti_am335x_tscadc dwc3_of_simple icssg_prueth pru_rproc pruss pruss_intc gpio_decoder remoteproc input_polldev sch_fq_codel cryptodev(O) ipv6
    [   35.126490] CPU: 1 PID: 773 Comm: irq/367-pruss1_ Tainted: G      D    O    4.14.79-rt47-ga72bf1418c #1
    [   35.126492] Hardware name: Texas Instruments AM654 Base Board (DT)
    [   35.126496] task: ffff80087df32880 task.stack: ffff000012320000
    [   35.126524] PC is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [   35.126542] LR is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [   35.126545] pc : [<ffff000001062a68>] lr : [<ffff000001062a68>] pstate: 60000145
    [   35.126547] sp : ffff00001232fbf0
    [   35.126549] x29: ffff00001232fbf0 x28: ffff0000087b0110
    [   35.126555] x27: ffff80087c8bca40 x26: 00000004a8cab474
    [   35.126560] x25: ffff80087c072010 x24: ffff80087c8bcba0
    [   35.126565] x23: 0000000000000040 x22: 0000000040000160
    [   35.126570] x21: ffff80087c8bc980 x20: ffff80087c8bc000
    [   35.126575] x19: 0000000000000000 x18: 0000000000000007
    [   35.126580] x17: 0000000000000001 x16: ffff0000087b1888
    [   35.126585] x15: 0000000000000000 x14: 0000000000000000
    [   35.126590] x13: 0000000000000000 x12: ffff80087420e100
    [   35.126595] x11: 0000000000000001 x10: 00000000fffffff4
    [   35.126600] x9 : 0000000000000000 x8 : 0000000000000000
    [   35.126604] x7 : 0008000000000000 x6 : 0000000099f81925
    [   35.126609] x5 : 0000000000000000 x4 : ffff80087420e138
    [   35.126614] x3 : ffff80087420e110 x2 : 0000000000000000
    [   35.126619] x1 : 0000000000000040 x0 : 0000000000000000
    [   35.126625] Process irq/367-pruss1_ (pid: 773, stack limit = 0xffff000012320000)
    [   35.126627] Call trace:
    [   35.126631] Exception stack(0xffff00001232fab0 to 0xffff00001232fbf0)
    [   35.126636] faa0:                                   0000000000000000 0000000000000040
    [   35.126641] fac0: 0000000000000000 ffff80087420e110 ffff80087420e138 0000000000000000
    [   35.126646] fae0: 0000000099f81925 0008000000000000 0000000000000000 0000000000000000
    [   35.126652] fb00: 00000000fffffff4 0000000000000001 ffff80087420e100 0000000000000000
    [   35.126657] fb20: 0000000000000000 0000000000000000 ffff0000087b1888 0000000000000001
    [   35.126662] fb40: 0000000000000007 0000000000000000 ffff80087c8bc000 ffff80087c8bc980
    [   35.126667] fb60: 0000000040000160 0000000000000040 ffff80087c8bcba0 ffff80087c072010
    [   35.126673] fb80: 00000004a8cab474 ffff80087c8bca40 ffff0000087b0110 ffff00001232fbf0
    [   35.126678] fba0: ffff000001062a68 ffff00001232fbf0 ffff000001062a68 0000000060000145
    [   35.126684] fbc0: ffff80087c8bc980 0000000040000160 ffffffffffffffff ffff000008428c28
    [   35.126687] fbe0: ffff00001232fbf0 ffff000001062a68
    [   35.126707] [<ffff000001062a68>] emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [   35.126718] [<ffff00000866c70c>] net_rx_action+0x18c/0x320
    [   35.126728] [<ffff0000080a1c44>] do_current_softirqs+0x1b4/0x248
    [   35.126735] [<ffff0000080a1d2c>] __local_bh_enable+0x54/0x68
    [   35.126743] [<ffff0000080eeb24>] irq_forced_thread_fn+0x64/0x90
    [   35.126749] [<ffff0000080eee0c>] irq_thread+0xfc/0x198
    [   35.126755] [<ffff0000080bc1a4>] kthread+0x12c/0x130
    [   35.126761] [<ffff000008084720>] ret_from_fork+0x10/0x18
    [   35.126768] Code: f94037a5 f9400300 aa0503e1 95cf23b7 (f9000b54)
    [   35.454892] ---[ end trace 0000000000000003 ]---
    [   35.454935] genirq: exiting task "irq/367-pruss1_" (773) is an active IRQ thread (irq 367)
    
    Broadcast message from systemd-journald@am65xx-evm (Sat 2019-04-06 01:41:14 UTC):
    
    kernel[690]: [   34.649945] Internal error: Oops: 96000046 [#1] PREEMPT SMP
    
    
    Broadcast message from systemd-journald@am65xx-evm (Sat 2019-04-06 01:41:15 UTC):
    
    kernel[690]: [   34.650173] Process irq/371-pruss0_ (pid: 780, stack limit = 0xffff000012440000)
    
    
    Broadcast message from systemd-journald@am65xx-evm (Sat 2019-04-06 01:41:15 UTC):
    
    kernel[690]: [   34.650330] Code: f94037a5 f9400300 aa0503e1 95cf23b7 (f9000b54)
    
    
    Broadcast message from systemd-journald@am65xx-evm (Sat 2019-04-06 01:41:15 UTC):
    
    kernel[690]: [   35.126400] Internal error: Oops: 96000046 [#2] PREEMPT SMP
    
    
    Broadcast message from systemd-journald@am65xx-evm (Sat 2019-04-06 01:41:15 UTC):
    
    kernel[690]: [   35.126625] Process irq/367-pruss1_ (pid: 773, stack limit = 0xffff000012320000)
    
    
    Broadcast message from systemd-journald@am65xx-evm (Sat 2019-04-06 01:41:15 UTC):
    
    kernel[690]: [   35.126768] Code: f94037a5 f9400300 aa0503e1 95cf23b7 (f9000b54)
    
    
    eth1      Link encap:Ethernet  HWaddr 70:FF:76:1D:57:BA
              inet6 addr: fe80::72ff:76ff:fe1d:57ba%3361668527/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:115 errors:0 dropped:0 overruns:0 frame:0
              TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:10766 (10.5 KiB)  TX bytes:3376 (3.2 KiB)
    
    root@am65xx-evm:~#
     _____                    _____           _         _
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
                  |___|                    |___|
    
    Arago Project http://arago-project.org am65xx-evm ttyS2
    
    Arago 2018.10 am65xx-evm ttyS2
    
    am65xx-evm login: [   28.073786] Unable to handle kernel paging request at virtual address 44a8b5504
    [   28.081110] Mem abort info:
    [   28.083896]   Exception class = DABT (current EL), IL = 32 bits
    [   28.089843]   SET = 0, FnV = 0
    [   28.092908]   EA = 0, S1PTW = 0
    [   28.096038] Data abort info:
    [   28.098923]   ISV = 0, ISS = 0x00000045
    [   28.102770]   CM = 0, WnR = 1
    [   28.105749] user pgtable: 64k pages, 48-bit VAs, pgd = ffff800842d5ac00
    [   28.112365] [000000044a8b5504] *pgd=0000000000000000, *pud=0000000000000000
    [   28.119334] Internal error: Oops: 96000045 [#1] PREEMPT SMP
    [   28.124896] Modules linked in: sha512_arm64 md5 des_generic cbc xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 xhci_plat_hcd xhci_hcd ah4 usbcore rpmsg_proto virtio_rpmsg_bus af_key rpmsg_core xfrm_algo ti_am335x_adc dwc3 kfifo_buf udc_core usb_common crc32_ce crct10dif_ce ti_k3_r5_remoteproc pci_endpoint_test omap_rng pvrsrvkm(O) rng_core at24 pruss_soc_bus sa2ul authenc phy_omap_usb2 m_can can_dev dwc3_of_simple ti_am335x_tscadc icssg_prueth pru_rproc pruss pruss_intc gpio_decoder remoteproc input_polldev sch_fq_codel cryptodev(O) ipv6
    [   28.173232] CPU: 0 PID: 238 Comm: systemd-network Tainted: G           O    4.14.79-ge669d52447 #1
    [   28.182170] Hardware name: Texas Instruments AM654 Base Board (DT)
    [   28.188336] task: ffff800841e83000 task.stack: ffff00000d980000
    [   28.194267] PC is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [   28.200362] LR is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [   28.206440] pc : [<ffff000001092b00>] lr : [<ffff000001092b00>] pstate: 60000145
    [   28.213816] sp : ffff00000800fdb0
    [   28.217120] x29: ffff00000800fdb0 x28: ffff0000087e0110
    [   28.222422] x27: ffff800842e199c0 x26: 000000044a8b54f4
    [   28.227725] x25: ffff80084004d010 x24: ffff800842e19af0
    [   28.233026] x23: 0000000000000040 x22: 0000000040000272
    [   28.238329] x21: ffff800842e19900 x20: ffff800842e19000
    [   28.243630] x19: 0000000000000000 x18: 0000ffffe135bcbe
    [   28.248932] x17: 0000ffff9f504538 x16: 0000aaaad607f650
    [   28.254233] x15: 00002b09e4000000 x14: 0000000000000000
    [   28.259534] x13: 0000000000000000 x12: ffff800846115080
    [   28.264835] x11: 0000000000000001 x10: 00000000fffffff9
    [   28.270136] x9 : 0000000000000000 x8 : 0000000000000000
    [   28.275437] x7 : 0100000000000000 x6 : ffff80087ff30000
    [   28.280739] x5 : 0000000000000000 x4 : ffff8008461150b0
    [   28.286042] x3 : ffff800846115090 x2 : 0000000000000000
    [   28.291344] x1 : 0000000000000040 x0 : 0000000000000000
    [   28.296646] Process systemd-network (pid: 238, stack limit = 0xffff00000d980000)
    [   28.304025] Call trace:
    [   28.306466] Exception stack(0xffff00000800fc70 to 0xffff00000800fdb0)
    [   28.312893] fc60:                                   0000000000000000 0000000000000040
    [   28.320706] fc80: 0000000000000000 ffff800846115090 ffff8008461150b0 0000000000000000
    [   28.328519] fca0: ffff80087ff30000 0100000000000000 0000000000000000 0000000000000000
    [   28.336332] fcc0: 00000000fffffff9 0000000000000001 ffff800846115080 0000000000000000
    [   28.344145] fce0: 0000000000000000 00002b09e4000000 0000aaaad607f650 0000ffff9f504538
    [   28.351960] fd00: 0000ffffe135bcbe 0000000000000000 ffff800842e19000 ffff800842e19900
    [   28.359775] fd20: 0000000040000272 0000000000000040 ffff800842e19af0 ffff80084004d010
    [   28.367589] fd40: 000000044a8b54f4 ffff800842e199c0 ffff0000087e0110 ffff00000800fdb0
    [   28.375403] fd60: ffff000001092b00 ffff00000800fdb0 ffff000001092b00 0000000060000145
    [   28.383216] fd80: ffff800842e19900 0000000040000272 0000ffffffffffff ffff000008449a60
    [   28.391029] fda0: ffff00000800fdb0 ffff000001092b00
    [   28.395913] [<ffff000001092b00>] emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [   28.403043] [<ffff00000869df54>] net_rx_action+0xf4/0x2b0
    [   28.408435] [<ffff0000080811ec>] __do_softirq+0x12c/0x228
    [   28.413825] [<ffff0000080acd10>] irq_exit+0xc8/0xf8
    [   28.418694] [<ffff0000080fd8c0>] __handle_domain_irq+0x60/0xb0
    [   28.424515] [<ffff000008080fc4>] gic_handle_irq+0x7c/0x178
    [   28.429987] Exception stack(0xffff00000d98fec0 to 0xffff00000d990000)
    [   28.436414] fec0: 0000000000000001 0000000000000000 0000ffff9f5d69f8 0000000000000000
    [   28.444227] fee0: 0000aaaad605d0a8 0000aaaad60550f8 0000ffffe135be80 00000000ffffffff
    [   28.452042] ff00: 0000ffffe135bf50 ffffff80fffffff0 0000ffffe135bf60 0000ffffe135bf60
    [   28.459857] ff20: 11ab00000200d246 98e9152867dd7aee 0000000000000001 00002b09e4000000
    [   28.467672] ff40: 0000aaaad607f650 0000ffff9f504538 0000ffffe135bcbe 0000aaaafb692070
    [   28.475486] ff60: 0000000000000260 0000000000000000 0000aaaafb692080 0000aaaafb6523b0
    [   28.483299] ff80: 0000000000000000 0000000000000000 0000000000000000 0000ffff9f5d8000
    [   28.491113] ffa0: 0000ffff9f5d69a0 0000ffffe135bed0 0000aaaad602b878 0000ffffe135bed0
    [   28.498927] ffc0: 0000ffff9f5016d4 0000000060000000 0000000000000013 00000000ffffffff
    [   28.506741] ffe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [   28.514557] [<ffff000008083050>] el0_irq_naked+0x4c/0x54
    [   28.519859] Code: f94037a5 f9400300 aa0503e1 95cee719 (f9000b54)
    [   28.525946] ---[ end trace 3fd13b8604747d0a ]---
    [   28.530553] Kernel panic - not syncing: Fatal exception in interrupt
    [   28.536892] SMP: stopping secondary CPUs
    [   28.540807] Kernel Offset: disabled
    [   28.544286] CPU features: 0x080200c
    [   28.547763] Memory Limit: none
    [   28.550812] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
    root@am65xx-evm:~# scp -r ibv@172.26.131.50:~/scp_test .
    ibv@172.26.131.50's password:
    nfs_layout_flexfiles.ko                                                                                  0%    0     0.0KB/s   --:-- ETA[   68.131836] skbuff: skb_over_panic: text:ffff0000010e2674 len:1788 put:1788 head:ffff80084553c000 data:ffff80084553c082 tail:0x77e end:0x680 dev:eth1
    [   68.145310] ------------[ cut here ]------------
    [   68.149922] kernel BUG at /oe/bld/build-AARCH64_1/arago-tmp-external-linaro-toolchain/work-shared/am65xx-evm/kernel-source/net/core/skbuff.c:104!
    [   68.162937] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
    [   68.168411] Modules linked in: sha512_arm64 md5 des_generic cbc xhci_plat_hcd xhci_hcd usbcore rpmsg_proto xfrm_user virtio_rpmsg_bus xfrm4_tunnel tunnel4 dwc3 rpmsg_core ipcomp udc_core xfrm_ipcomp usb_common ti_am335x_adc kfifo_buf esp4 ah4 af_key xfrm_algo crc32_ce crct10dif_ce ti_k3_r5_remoteproc pci_endpoint_test pvrsrvkm(O) at24 pruss_soc_bus sa2ul authenc omap_rng rng_core phy_omap_usb2 m_can can_dev ti_am335x_tscadc dwc3_of_simple icssg_prueth pru_rproc pruss pruss_intc gpio_decoder remoteproc input_polldev sch_fq_codel cryptodev(O) ipv6
    [   68.216763] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O    4.14.79-gbde58ab01e #1
    [   68.225008] Hardware name: Texas Instruments AM654 Base Board (DT)
    [   68.231175] task: ffff000008b3b280 task.stack: ffff000008b00000
    [   68.237091] PC is at skb_panic+0x48/0x50
    [   68.241003] LR is at skb_panic+0x48/0x50
    [   68.244916] pc : [<ffff000008683bb0>] lr : [<ffff000008683bb0>] pstate: 20000145
    [   68.252292] sp : ffff00000800fd80
    [   68.255596] x29: ffff00000800fd90 x28: ffff0000087e0110
    [   68.260900] x27: ffff8008425919c0 x26: ffff80084391ef00
    [   68.266203] x25: ffff800842ef2300 x24: ffff800842591ae0
    [   68.271506] x23: 0000000000000040 x22: 00000000000006fc
    [   68.276809] x21: ffff800842591900 x20: ffff800842591000
    [   68.282111] x19: 0000000000000000 x18: 0000000000000010
    [   68.287414] x17: 0000ffff7efacac0 x16: ffff0000081bf7e0
    [   68.292716] x15: ffffffffffffffff x14: 7665642030383678
    [   68.298017] x13: 303a646e65206537 x12: ffff000008b33a30
    [   68.303320] x11: ffff000008478a98 x10: 3438303038666666
    [   68.308621] x9 : 0000000000000016 x8 : 3030633335353438
    [   68.313922] x7 : 303038666666663a x6 : 00000000000001dc
    [   68.319223] x5 : 0000000000000001 x4 : 0000000000000004
    [   68.324524] x3 : 0000000000000000 x2 : 0000000000000040
    [   68.329827] x1 : ffff000008b3b280 x0 : 0000000000000089
    [   68.335130] Process swapper/0 (pid: 0, stack limit = 0xffff000008b00000)
    [   68.341815] Call trace:
    [   68.344254] Exception stack(0xffff00000800fc40 to 0xffff00000800fd80)
    [   68.350682] fc40: 0000000000000089 ffff000008b3b280 0000000000000040 0000000000000000
    [   68.358497] fc60: 0000000000000004 0000000000000001 00000000000001dc 303038666666663a
    [   68.366312] fc80: 3030633335353438 0000000000000016 3438303038666666 ffff000008478a98
    [   68.374125] fca0: ffff000008b33a30 303a646e65206537 7665642030383678 ffffffffffffffff
    [   68.381940] fcc0: ffff0000081bf7e0 0000ffff7efacac0 0000000000000010 0000000000000000
    [   68.389755] fce0: ffff800842591000 ffff800842591900 00000000000006fc 0000000000000040
    [   68.397568] fd00: ffff800842591ae0 ffff800842ef2300 ffff80084391ef00 ffff8008425919c0
    [   68.405383] fd20: ffff0000087e0110 ffff00000800fd90 ffff000008683bb0 ffff00000800fd80
    [   68.413198] fd40: ffff000008683bb0 0000000020000145 ffff0000010e2674 00000000000006fc
    [   68.421013] fd60: 0000ffffffffffff ffff80084553c000 ffff00000800fd90 ffff000008683bb0
    [   68.428829] [<ffff000008683bb0>] skb_panic+0x48/0x50
    [   68.433784] [<ffff000008685a98>] pskb_put+0x0/0x38
    [   68.438594] [<ffff0000010e2674>] emac_napi_rx_poll+0x1b4/0x2b0 [icssg_prueth]
    [   68.445720] [<ffff00000869decc>] net_rx_action+0xf4/0x2b0
    [   68.451111] [<ffff0000080811ec>] __do_softirq+0x12c/0x228
    [   68.456500] [<ffff0000080acd10>] irq_exit+0xc8/0xf8
    [   68.461370] [<ffff0000080fd8c0>] __handle_domain_irq+0x60/0xb0
    [   68.467191] [<ffff000008080fc4>] gic_handle_irq+0x7c/0x178
    [   68.472663] Exception stack(0xffff000008b0fdf0 to 0xffff000008b0ff30)
    [   68.479089] fde0:                                   0000000000000000 00008008773c0000
    [   68.486903] fe00: 0000000000000000 ffff000008af3930 ffff000008af3908 ffff000008b0ff20
    [   68.494717] fe20: 00008008773c0000 0000000000000000 0000000000000002 ffff000008b0feb0
    [   68.502532] fe40: 0000000000000920 0000000000000000 0000000000000001 0000000000000000
    [   68.510347] fe60: 0000ffff7ef00af0 0000ffff7eef3cc8 ffff0000081bf7e0 0000ffff7efacac0
    [   68.518162] fe80: 0000000000000523 ffff000008b33ac4 ffff000008b33a30 ffff000008af0018
    [   68.525976] fea0: ffff000008b3b280 ffff80087ffff480 ffff000008ae0028 0000000000000000
    [   68.533791] fec0: 00000000ffef4ce0 0000000000000400 0000000080a90018 ffff000008b0ff30
    [   68.541605] fee0: ffff000008084e2c ffff000008b0ff30 ffff000008084e30 0000000060000145
    [   68.549419] ff00: 0000000000000000 00000000ffef4ce0 ffffffffffffffff ffff000008123f84
    [   68.557231] ff20: ffff000008b0ff30 ffff000008084e30
    [   68.562098] [<ffff000008082a30>] el1_irq+0xb0/0x124
    [   68.566968] [<ffff000008084e30>] arch_cpu_idle+0x10/0x18
    [   68.572269] [<ffff0000080e5ee0>] do_idle+0xd8/0x118
    [   68.577135] [<ffff0000080e60b4>] cpu_startup_entry+0x24/0x28
    [   68.582784] [<ffff00000879611c>] rest_init+0xcc/0xd8
    [   68.587743] [<ffff000008a90b50>] start_kernel+0x378/0x38c
    [   68.593133] Code: a90023eb aa0a03e1 912b6120 97e9e379 (d4210000)
    [   68.599219] ---[ end trace a3f5975970c27302 ]---
    [   68.603825] Kernel panic - not syncing: Fatal exception in interrupt
    [   68.610165] SMP: stopping secondary CPUs
    [   68.614079] Kernel Offset: disabled
    [   68.617559] CPU features: 0x080200c
    [   68.621036] Memory Limit: none
    [   68.624084] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
    root@am65xx-evm:~# scp -r ibv@172.26.131.50:/home/ibv/WORKSPACE/linux-am65x/modules/lib/modules/4.14.79-rt47-56977-g34cee38-dirty/kernel .
    
    Host '172.26.131.50' is not in the trusted hosts file.
    (ecdsa-sha2-nistp256 fingerprint md5 7a:08:b0:9a:b7:6c:f3:28:14:84:d8:cf:fd:86:5d:14)
    Do you want to continue connecting? (y/n) y
    ibv@172.26.131.50's password:
    nfs_layout_flexfiles.ko                                                                                100% 1188KB   1.2MB/s   00:00
    fuse.ko                                                                                                100% 2264KB   2.2MB/s   00:00
    cuse.ko                                                                                                100%  329KB 328.8KB/s   00:00
    overlay.ko                                                                                             100% 1364KB   1.3MB/s   00:00
    cifs.ko                                                                                                 78%   10MB   9.8MB/s   00:00 ETA[  115.814771] Unable to handle kernel paging request at virtual address 3fffff381000ab
    [  115.822539] Mem abort info:
    [  115.825335]   Exception class = DABT (current EL), IL = 32 bits
    [  115.831263]   SET = 0, FnV = 0
    [  115.834313]   EA = 0, S1PTW = 0
    [  115.837449] Data abort info:
    [  115.840327]   ISV = 0, ISS = 0x00000044
    [  115.844157]   CM = 0, WnR = 1
    [  115.847122] [003fffff381000ab] address between user and kernel address ranges
    [  115.854250] Internal error: Oops: 96000044 [#1] PREEMPT SMP
    [  115.859811] Modules linked in: sha512_arm64 md5 des_generic cbc xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo xhci_plat_hcd xhci_hcd rpmsg_proto virtio_rpmsg_bus rpmsg_core usbcore dwc3 udc_core ti_am335x_adc usb_common kfifo_buf crc32_ce crct10dif_ce ti_k3_r5_remoteproc pci_endpoint_test pruss_soc_bus omap_rng pvrsrvkm(O) sa2ul at24 rng_core authenc phy_omap_usb2 m_can icssg_prueth ti_am335x_tscadc can_dev dwc3_of_simple pru_rproc pruss gpio_decoder pruss_intc remoteproc input_polldev sch_fq_codel cryptodev(O) ipv6
    [  115.908150] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O    4.14.79-ge669d52447 #1
    [  115.916395] Hardware name: Texas Instruments AM654 Base Board (DT)
    [  115.922563] task: ffff000008b3b280 task.stack: ffff000008b00000
    [  115.928496] PC is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [  115.934591] LR is at emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [  115.940669] pc : [<ffff000001312b00>] lr : [<ffff000001312b00>] pstate: 60000145
    [  115.948047] sp : ffff00000800fdb0
    [  115.951351] x29: ffff00000800fdb0 x28: ffff0000087e0110
    [  115.956654] x27: ffff8008429ea9c0 x26: 013fffff3810009b
    [  115.961957] x25: ffff80084004d410 x24: ffff8008429eaaf0
    [  115.967258] x23: 0000000000000040 x22: 0000000040000132
    [  115.972559] x21: ffff8008429ea900 x20: ffff8008429ea000
    [  115.977862] x19: 0000000000000000 x18: 00000000bdd3d36e
    [  115.983163] x17: 0000ffffb5f02548 x16: ffff0000081d6010
    [  115.988464] x15: 00ae5757f9ae5757 x14: 0000000000000000
    [  115.993765] x13: 0000000000000000 x12: ffff800844397580
    [  115.999068] x11: 0000000000000001 x10: 00000000ffffffec
    [  116.004369] x9 : 0000000000000000 x8 : 0000000000000000
    [  116.009671] x7 : 0000080000000000 x6 : ffff80087ff30000
    [  116.014972] x5 : 0000000000000000 x4 : ffff8008443975d8
    [  116.020273] x3 : ffff800844397590 x2 : 0000000000000000
    [  116.025576] x1 : 0000000000000040 x0 : 0000000000000000
    [  116.030879] Process swapper/0 (pid: 0, stack limit = 0xffff000008b00000)
    [  116.037564] Call trace:
    [  116.040003] Exception stack(0xffff00000800fc70 to 0xffff00000800fdb0)
    [  116.046430] fc60:                                   0000000000000000 0000000000000040
    [  116.054243] fc80: 0000000000000000 ffff800844397590 ffff8008443975d8 0000000000000000
    [  116.062056] fca0: ffff80087ff30000 0000080000000000 0000000000000000 0000000000000000
    [  116.069870] fcc0: 00000000ffffffec 0000000000000001 ffff800844397580 0000000000000000
    [  116.077684] fce0: 0000000000000000 00ae5757f9ae5757 ffff0000081d6010 0000ffffb5f02548
    [  116.085498] fd00: 00000000bdd3d36e 0000000000000000 ffff8008429ea000 ffff8008429ea900
    [  116.093313] fd20: 0000000040000132 0000000000000040 ffff8008429eaaf0 ffff80084004d410
    [  116.101126] fd40: 013fffff3810009b ffff8008429ea9c0 ffff0000087e0110 ffff00000800fdb0
    [  116.108940] fd60: ffff000001312b00 ffff00000800fdb0 ffff000001312b00 0000000060000145
    [  116.116753] fd80: ffff8008429ea900 0000000040000132 0000ffffffffffff ffff000008449a60
    [  116.124565] fda0: ffff00000800fdb0 ffff000001312b00
    [  116.129449] [<ffff000001312b00>] emac_napi_rx_poll+0x160/0x2b0 [icssg_prueth]
    [  116.136579] [<ffff00000869df54>] net_rx_action+0xf4/0x2b0
    [  116.141972] [<ffff0000080811ec>] __do_softirq+0x12c/0x228
    [  116.147361] [<ffff0000080acd10>] irq_exit+0xc8/0xf8
    [  116.152231] [<ffff0000080fd8c0>] __handle_domain_irq+0x60/0xb0
    [  116.158052] [<ffff000008080fc4>] gic_handle_irq+0x7c/0x178
    [  116.163524] Exception stack(0xffff000008b0fdf0 to 0xffff000008b0ff30)
    [  116.169950] fde0:                                   0000000000000000 00008008773c0000
    [  116.177764] fe00: 0000000000000000 ffff000008af3930 ffff000008af3908 ffff000008b0ff20
    [  116.185578] fe20: 00008008773c0000 0000000000000000 0000000000000002 ffff000008b0feb0
    [  116.193393] fe40: 0000000000000920 0000000000000001 0000000000000001 0000000000000001
    [  116.201207] fe60: 00000000eff38bce 00ae5757f9ae5757 ffff0000081d6010 0000ffffb5f02548
    [  116.209022] fe80: 00000000bdd3d36e ffff000008b33ac4 ffff000008b33a30 ffff000008af0018
    [  116.216836] fea0: ffff000008b3b280 ffff80087ffff480 ffff000008ae0028 0000000000000000
    [  116.224651] fec0: 00000000ffef3ce0 0000000000000400 0000000080a90018 ffff000008b0ff30
    [  116.232466] fee0: ffff000008084e2c ffff000008b0ff30 ffff000008084e30 0000000060000145
    [  116.240281] ff00: 0000000000000000 00000000ffef3ce0 ffffffffffffffff ffff000008123f84
    [  116.248093] ff20: ffff000008b0ff30 ffff000008084e30
    [  116.252960] [<ffff000008082a30>] el1_irq+0xb0/0x124
    [  116.257827] [<ffff000008084e30>] arch_cpu_idle+0x10/0x18
    [  116.263128] [<ffff0000080e5ee0>] do_idle+0xd8/0x118
    [  116.267995] [<ffff0000080e60b4>] cpu_startup_entry+0x24/0x28
    [  116.273645] [<ffff00000879621c>] rest_init+0xcc/0xd8
    [  116.278603] [<ffff000008a90b50>] start_kernel+0x378/0x38c
    [  116.283993] Code: f94037a5 f9400300 aa0503e1 95c4e719 (f9000b54)
    [  116.290080] ---[ end trace d7c82d781fd228d2 ]---
    [  116.294688] Kernel panic - not syncing: Fatal exception in interrupt
    [  116.301027] SMP: stopping secondary CPUs
    [  116.304943] Kernel Offset: disabled
    [  116.308423] CPU features: 0x080200c
    [  116.311900] Memory Limit: none
    [  116.314949] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
    

    The release notes say that at least LCPD-14189 "ICSSG Ethernet Kernel Panic" still exists. Can you tell us a bit more about that known issue? Is there a schedule for when the PRU ethernet interfaces are expected to be usable?

    Regards,

    Dominic

    EDIT:

    One thing regarding the two "scp" panic logs: They show different locations for the panic, the RT log shows the panic as a BUG at skbuff.c:104 and the vanialla log show the panic as a pagic request failure somewhere in emac_napi_rx_poll, but we've had both kinds of crashes for both vanilla and for RT. Just wanted to point that out so that you don't draw any wrong conclusions.

  • Dominic, thank you for your detailed notes. Taking a look now, this might take a couple of days.

    Regards,
    Nick
  • Dominic, I am sorry for the delayed response. I am trying to replicate the kernel panic with multiple ethernet connections now.

    1) Was this test on the EVM with the 4 Ethernet port daughter card attached (the IDK)? Or on the EVM without the ethernet daughter card (the GP EVM)? If using the daughter card, which ports were tested?

    2) At the moment I am just letting the boards sit to see what happens. In the initial post there was not a specific action that caused the kernel panic, let me know if the issue occurred randomly only during specific tasks (e.g., if it didn't happen every iperf test, but only happened during iperf tests).

    3)
    I have not yet seen any panics, but I have not been testing for long. current setup:
    AM65xx GP EVM 1, SDK 5.3
    eth0: subnet 1, NFS connection to PC (CPSW port)
    eth1: subnet 2, direct connection to EVM 2 eth1
    eth2: subnet 3, direct connection to EVM 2 eth2

    AM65xx GP EVM 2, SDK 5.3
    eth0: subnet 1, NFS connection to PC (CPSW port)
    eth1: subnet 2, direct connection to EVM 1 eth1
    eth2: subnet 3, direct connection to EVM 1 eth2


    Was this your hardware setup to test kernel panics with multiple PRUETH connections? If not, please correct me:
    AM65xx GP EVM, SDK 5.3
    eth0: subnet 1, connected to PC port0 through switch (CPSW port)
    eth1: subnet 2, connected to PC port1 through switch
    eth2: subnet 3, connected to PC port2 through switch

    Regards,
    Nick
  • Hello Nick,

    thanks for getting back to me on this.

    1) We're always using the IDK daughter card.

    2) This happens without doing anything specifically, sometimes right during booting, and sometimes after as long as a few minutes. Usually the kernel crashed after ~10 minutes.

    3) We were using two of the ethernet ports on the IDK daughter card labeled with ETH2/ETH3 and one of the PRU (not CPSW) ethernet ports on the main board. Under linux these showed up as eth3, eth4 and eth5. All three ports were connected to the same switch along with the host computer. There was no explicit traffic to any of the ports, but of course it's likely that there were some broadcasts.

    Regards,

    Dominic

  • Hello Dominic,

    Update: I am able to replicate your kernel panic with multiple ethernet ports connected on SDK 5.3.

    If I do nothing but boot the AM65xx IDK and type ifconfig to check ports, I can get a kernel panic as long as 2 conditions are met: 1) there are 2 or more AM65xx ethernet ports that are plugged into the external switch, and 2) at least one of those ethernet ports must be one of the PRU Ethernet ports on the main board.

    I did not observe any kernel panics when I only used 2 or 3 ports on the IDK Ethernet daughter card. If the multi-port kernel panic is gating development, I would suggest trying that setup.

    I am checking now to see if that issue is fixed in the next SDK release. I will also see if I can replicate your file transfer issue.

    Regards,
    Nick

  • Hello Dominic,

    Update #2: I am able to replicate your scp kernel panic on SDK 5.3, but I only get the panic sometimes. I have only observed the panic on the upper PRU port on the base board, but since the panic is only happening for me a fraction of the time on that port, it is possible that I just did not run enough iterations on other ports to see the panic.

    reboot
    ifconfig ethG up 192.168.1.160
    scp -r linuxPC@192.168.1.100:/nfs/am65xx_05.03/lib/modules/4.14.79-ge669d52447/kernel .

    sometimes this gets me a panic, most of the time it does not. Are you able to reliably reproduce the panic? Is there a single file that always produces a panic for you?

    So far, the 3 port setup has run for an hour and a half without any kernel panics on the next SDK. I will let it continue to run over the weekend.

    Update 2019-05-13: The 3 port setup ran an entire weekend without any kernel panics. It looks like this issue will be addressed in the next SDK

    Regards,
    Nick

  • Hello Nick,
    I'm not sure why you don't see the "3 port panics" as often as we do. On our custom board we also get these panics, and the custom board uses only PRG0 and PRG1, i.e. the same ports as on the IDK daughter card. I guess it really depends on the subtle differences in the network traffic, e.g. packet sizes and timing.
    But it's good to hear that this doesn't happen with the "next SDK". I read in another thread that this next release is scheduled for June/July - do you know if the necessary patches are already in the TI public GIT repository? It would be great if we could test with these changes before the release is out. It's not just that the panics are interfering with development, but also that we need a production-ready software platform. Spending ~3 months between finding and reporting an issue and having a chance to test potential fixes is going to become a problem.

    Regarding the SCP issue:
    We get the panic 100% when copying a particular /lib/modules tree. The kernel usually crashes within the first few files (4-5 maybe). Is there a way I can send you these files without putting them here for everyone?

    Best Regards,
    Dominic
  • Hello Dominic,

    To clarify: I am able to consistently reproduce the 2 and 3 port panics on SDK 5.3, as long as I meet the 2 conditions I mention in this post. The SCP issue is the one I am having trouble reproducing consistently on SDK 5.3.

    On SCP: I will reach out to you separately to get your test files.

    Other notes on the multiport panic: I do not know why daughter card PRU ports do not cause a panic, while the PRU ports on the main board cause panics. Perhaps that ICSSG required slightly different firmware that will be fixed in the next SDK? Maybe there is a board layout or part selection difference that mattered? I am not sure that I will get an explanation, but if I do I will post follow-up information here. (especially if there is a hardware design component).

    I will create a second post that explains the public repos.

    Regards,

    Nick

  • On repos:

    There are two primary TI git repos: ti-linux-kernel, and processor-sdk-linux. ti-linux-kernel is for TI's development team. processor-sdk-linux is used in our Processor SDKs, and is for TI customers. You can clone processor-sdk-linux from git://git.ti.com/processor-sdk/processor-sdk-linux.git .

    As a general rule, I do not support ti-linux-kernel on this forum. That is because processor-sdk-linux adds a bunch of patches on top of ti-linux-kernel. The prueth driver in particular has big differences between ti-linux-kernel and processor-sdk-linux.

    On getting new prueth code to test:

    At the moment, we are still ironing out prueth bugs in the TI development branch. We have not yet created a new branch in processor-sdk-linux for the next SDK, so there is not a public version of prueth available with all the additional SDK patches yet. In addition to the updated prueth driver code, you would need the updated PRU_ICSSG prueth firmware. I am not sure where to get that at the moment.

    So I do not have an answer right now for you on if there is a good way to get you updated software before the SDK release. We can address that question later.

    Regards,
    Nick
  • Hello Dominic,

    FYI I sent a friendship request to connect through e2e.

    Regards,
    Nick
  • FYI, the next SDK is slated for release in early July.

    Regards,
    Nick
  • Hello Dominic,

    I am going to close the thread on my end. Feel free to reach back out if you want to continue the conversation.

    Regards,

    Nick