This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5708: AM5708 PRU Ethernet Independent Mode reception issue

Part Number: AM5708
Other Parts Discussed in Thread: AM5728

Hi:

    I have a custom AM5708 based board with SDK6.02 Linux RT.

The board has 3 ethernet interfaces:

       - eth0: connected to GMAC 

      -  eth1: connected to prueth pruss2_eth

      - eth2: connected prueth pruss2_eth 

I have configured eth1 and eth2 to work independently (see attached script "dual.sh").prueth.c.txtcrash_billa5.txtcrash_trafico_vault_killer.pcapng.txt

dual.sh.txt
#!/bin/bash

echo "Get MAC & IPs information ..."
MAC_A="00:a0:f4:de:ad:11"
MAC_B="00:a0:f4:de:ad:22"
IP_A="192.168.33.187"
MASK_A="255.255.255.0"
GW_A="192.168.33.1"
IP_B="192.168.34.188"
MASK_B="255.255.255.0"
GW_B="192.168.34.1"
sleep 1

echo "Set LLA interface down .."
/sbin/ifconfig bond0 down
/sbin/ip link delete bond0
/sbin/rmmod bonding
sleep 1

echo "Set PRP interface down .."
/sbin/ifconfig prp0 down
/sbin/ip link delete prp0
sleep 1

echo "Set HSR interface down .."
/sbin/ifconfig hsr0 down
/sbin/ip link delete hsr0
sleep 1

echo "Set RSTP interface down .."
/sbin/ifconfig br0 down
/sbin/ip link delete br0
sleep 1

echo "Set DUAL interfaces down .."
/sbin/ifconfig eth1 0.0.0.0 down
/sbin/ifconfig eth2 0.0.0.0 down
sleep 1

echo "Set MAC address ${MAC_A} and ${MAC_B} to eth1 & eth2 interfaces .."
/sbin/ifconfig eth1 hw ether ${MAC_A}
/sbin/ifconfig eth2 hw ether ${MAC_B}
sleep 1

echo "Unset PRP-HSR-SW offload ..."
/usr/sbin/ethtool -K eth1 prp-rx-offload off
/usr/sbin/ethtool -K eth2 prp-rx-offload off
/usr/sbin/ethtool -K eth1 hsr-rx-offload off
/usr/sbin/ethtool -K eth2 hsr-rx-offload off
/usr/sbin/ethtool -K eth1 l2-fwd-offload off
/usr/sbin/ethtool -K eth2 l2-fwd-offload off
sleep 1

echo "Set rx parameters ..."
/usr/sbin/ethtool -C eth1 rx-usecs 200
/usr/sbin/ethtool -C eth2 rx-usecs 200
/usr/sbin/ethtool -C eth1 adaptive-rx on
/usr/sbin/ethtool -C eth2 adaptive-rx on
sleep 1\n");

echo "Set eth1 with IP=${IP_A} and eth2 with IP=${IP_B} interfaces up .."
/opt/edge-agent/ipv4-set eth1 MANUAL ip4 ${IP_A}/24 gw ${GW_A}
/sbin/ifconfig eth1 ${IP_A} netmask ${MASK_A} up
sleep 1

/opt/edge-agent/ipv4-set eth2 MANUAL ip4 ${IP_B}/24 gw ${GW_B}
/sbin/ifconfig eth2 ${IP_B} netmask ${MASK_B} up
sleep 1

echo "Set eth0 in allmulticast mode .."
/sbin/ifconfig eth0 allmulti
sleep 1

echo "Set eth1 in allmulticast mode .."
/sbin/ifconfig eth1 allmulti
sleep 1

echo "Set eth2 in allmulticast mode .."
/sbin/ifconfig eth2 allmulti
sleep 1

I have connected eth1 and eth2 to our local network where we have some traffic (see attached "crash_trafico_vault_killer.pcapng").

I boot up the system over NFS with TI Linux over eth0.

Interfaces eth1 and eth2 are configured to accept GOOSE messages (interfaces in allmulticast mode) as GOOSE messages are circulating around the local network.

After some time I have detected something strange in reception, as if it gets corrupted. So I decided to put some printks in A15 "prueth.c" driver in the reception function "emac_rx_packet()" (see attached "prueth.c").

I know that there is a shared memory between A15 prueth.c driver and PRU firmware for buffer descriptors, and that there is s shared memory in OCMC where the incoming frames are placed by PRU firmware for A15 prueth driver to take them out in a FIFO way.

I have detected that sometimes the BD pointer "bd_rd_ptr" that point to BD shared memory has a wrong value so it can make the driver crash.

I tried to workaround this situation to avoid the driver crashing and see in this situation that even reloading the PRU firmware (ifconfig ethx down/up) the issue persists, as if the prueth driver would be unable to recover.

I have made a few tests but with interfaces eth1 and eth2 in PRP mode asit means that there is a different PRU firmware than in dual-independent mode. It hasn´t failed yet but I will continue with the test

Regards

Billa

  • Hi Billa,

    Jose Carlos Billalabeitia said:
    Interfaces eth1 and eth2 are configured to accept GOOSE messages (interfaces in allmulticast mode) as GOOSE messages are circulating around the local network.

    Does the issue only occur when in allmulticast mode?  Have you tried subscribing to specific GOOSE streams instead?

    How long does it take for the issue to show up?  Is the length of time proportional to either the overall traffic or perhaps to the amount of GOOSE traffic?

    Jose Carlos Billalabeitia said:
    After some time I have detected something strange in reception, as if it gets corrupted.

    Is the issue that a single packet sometimes gets corrupted?  Or does the interface get into a state where all packets get corrupted?

    Jose Carlos Billalabeitia said:
    I tried to workaround this situation to avoid the driver crashing and see in this situation that even reloading the PRU firmware (ifconfig ethx down/up) the issue persists, as if the prueth driver would be unable to recover.

    Does rmmod/insmod ti_prueth "fix" the issue?

    Best regards,
    Brad

  • Brad

        Sorry for the late response. The thing is that is is very difficult to reproduce the issue. In fact in the last 15 days we have only seen the issue in a very few occasions, so it is difficult to interact with it.

    Anyway I want you to inform that we have seen in the IDK a crash that may be similar to the Bridge´s.

    I was power cycling the IDK with is console active and the eth2 and eth3 interfaces configured in DHCP. Both interfaces are connected to our Vault that is causing the issue. See attached the crash log file.

    I would like to highlight part of this crash messages, specially:

              [  455.739699] PC is at memcpy+0x244/0x330
              [  455.739702] LR is at   (null)
              ...................................................

             [  455.739841] Backtrace:
             [  455.739884] [<bf349464>] (emac_rx_packet [ti_prueth]) from [<bf349df0>] (emac_rx_thread+0x1bc/0x2cc [ti_prueth])
             [  455.739892]  r10:e7471c88 r9:d1fc9ebc r8:d1e9a040 r7:00000071 r6:d2a1e000 r5:00000000
             [  455.739895]  r4:e7470071
             [  455.739918] [<bf349c34>] (emac_rx_thread [ti_prueth]) from [<c0289f50>] (irq_thread_fn+0x24/0x80)
             [  455.739924]  r10:00000001 r9:c0289f2c r8:00000000 r7:ffffe000 r6:d14f0a64 r5:d1d84b00
             [  455.739927]  r4:d14f0a40
             [  455.739936] [<c0289f2c>] (irq_thread_fn) from [<c028a288>] (irq_thread+0x140/0x22c)
             [  455.739941]  r7:ffffe000 r6:d14f0a64 r5:d14f0a40 r4:d1d84b00

    Regards

    Billa

    crash_IDK.txt
    U-Boot 2019.01-ga141f7abfd (Jan 05 2020 - 10:28:39 +0000)
    
    CPU  : DRA752-GP ES2.0
    Model: TI AM5728 IDK
    Board: AM572x IDK REV 1.3B
    DRAM:  2 GiB
    MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
    Loading Environment from FAT... *** Warning - bad CRC, using default environment
    
    Loading Environment from MMC... OK
    am57x_idk_lcd_detect: Failed to get I2C device 0/56 (ret 1)
    Net:
    Warning: ethernet@48484000 using MAC address from ROM
    eth0: ethernet@48484000
    Hit any key to stop autoboot:  0
    switch to partitions #0, OK
    mmc0 is current device
    SD/MMC found on device 0
    ** Unable to read file boot.scr **
    1490 bytes read in 2 ms (727.5 KiB/s)
    Loaded env from uEnv.txt
    Importing environment from mmc0 ...
    Running uenvcmd ...
    1 bytes read in 4 ms (0 Bytes/s)
    Already setup.
    switch to partitions #0, OK
    mmc0 is current device
    SD/MMC found on device 0
    4370944 bytes read in 196 ms (21.3 MiB/s)
    93055 bytes read in 6 ms (14.8 MiB/s)
    ## Flattened Device Tree blob at 88000000
       Booting using the fdt blob at 0x88000000
       Loading Device Tree to 8ffe6000, end 8ffffb7e ... OK
    
    Starting kernel ...
    
    [    0.000000] Booting Linux on physical CPU 0x0
    [    0.000000] Linux version 4.19.79-rt28-g5baf382c8f (oe-user@oe-host) (gcc version 8.3.0 (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.0
    [    0.000000] CPU: ARMv7 Processor [412fc0f2] revision 2 (ARMv7), cr=30c5387d
    [    0.000000] CPU: div instructions available: patching division code
    [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
    [    0.000000] OF: fdt: Machine model: TI AM5728 IDK
    [    0.000000] Memory policy: Data cache writealloc
    [    0.000000] efi: Getting EFI parameters from FDT:
    [    0.000000] efi: UEFI not found.
    [    0.000000] Reserved memory: created CMA memory pool at 0x0000000095800000, size 56 MiB
    [    0.000000] OF: reserved mem: initialized node ipu2-memory@95800000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created CMA memory pool at 0x0000000099000000, size 64 MiB
    [    0.000000] OF: reserved mem: initialized node dsp1-memory@99000000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x000000009d000000, size 32 MiB
    [    0.000000] OF: reserved mem: initialized node ipu1-memory@9d000000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created CMA memory pool at 0x000000009f000000, size 8 MiB
    [    0.000000] OF: reserved mem: initialized node dsp2-memory@9f000000, compatible id shared-dma-pool
    [    0.000000] cma: Reserved 24 MiB at 0x00000000fe400000
    [    0.000000] OMAP4: Map 0x00000000ffd00000 to (ptrval) for dram barrier
    [    0.000000] DRA752 ES2.0
    [    0.000000] random: get_random_bytes called from start_kernel+0xb0/0x488 with crng_init=0
    [    0.000000] percpu: Embedded 15 pages/cpu s32224 r8192 d21024 u61440
    [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 382848
    [    0.000000] Kernel command line: console=ttyS2,115200n8 root=PARTUUID=987fd2b2-02 rw rootfstype=ext4 rootwait
    [    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
    [    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
    [    0.000000] Memory: 1348360K/1536000K available (10240K kernel code, 343K rwdata, 2748K rodata, 2048K init, 277K bss, 31992K reserved, 155648K cma-reserved, )
    [    0.000000] Virtual kernel memory layout:
    [    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    [    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
    [    0.000000]     vmalloc : 0xe0800000 - 0xff800000   ( 496 MB)
    [    0.000000]     lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
    [    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
    [    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
    [    0.000000]       .text : 0x(ptrval) - 0x(ptrval)   (12256 kB)
    [    0.000000]       .init : 0x(ptrval) - 0x(ptrval)   (2048 kB)
    [    0.000000]       .data : 0x(ptrval) - 0x(ptrval)   ( 344 kB)
    [    0.000000]        .bss : 0x(ptrval) - 0x(ptrval)   ( 278 kB)
    [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
    [    0.000000] rcu: Preemptible hierarchical RCU implementation.
    [    0.000000] rcu:     RCU priority boosting: priority 1 delay 500 ms.
    [    0.000000]  No expedited grace period (rcu_normal_after_boot).
    [    0.000000]  Tasks RCU enabled.
    [    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
    [    0.000000] GIC: Using split EOI/Deactivate mode
    [    0.000000] OMAP clockevent source: timer1 at 32786 Hz
    [    0.000000] arch_timer: cp15 timer(s) running at 6.14MHz (phys).
    [    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x16af5adb9, max_idle_ns: 440795202250 ns
    [    0.000005] sched_clock: 56 bits at 6MHz, resolution 162ns, wraps every 4398046511023ns
    [    0.000011] Switching to timer-based delay loop, resolution 162ns
    [    0.000288] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
    [    0.000291] OMAP clocksource: 32k_counter at 32768 Hz
    [    0.000740] Console: colour dummy device 80x30
    [    0.000878] Calibrating delay loop (skipped), value calculated using timer frequency.. 12.29 BogoMIPS (lpj=61475)
    [    0.000886] pid_max: default: 32768 minimum: 301
    [    0.001023] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
    [    0.001031] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
    [    0.001702] CPU: Testing write buffer coherency: ok
    [    0.001733] CPU0: Spectre v2: using ICIALLU workaround
    [    0.001979] /cpus/cpu@0 missing clock-frequency property
    [    0.002016] /cpus/cpu@1 missing clock-frequency property
    [    0.002029] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
    [    0.059953] Setting up static identity map for 0x80200000 - 0x80200060
    [    0.099947] rcu: Hierarchical SRCU implementation.
    [    0.160444] EFI services will not be available.
    [    0.180068] smp: Bringing up secondary CPUs ...
    [    0.320459] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
    [    0.320464] CPU1: Spectre v2: using ICIALLU workaround
    [    0.320592] smp: Brought up 1 node, 2 CPUs
    [    0.320611] SMP: Total of 2 processors activated (24.59 BogoMIPS).
    [    0.320621] CPU: All CPU(s) started in HYP mode.
    [    0.320629] CPU: Virtualization extensions available.
    [    0.321292] devtmpfs: initialized
    [    0.360477] VFP support v0.3: implementor 41 architecture 4 part 30 variant f rev 0
    [    0.360811] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
    [    0.360833] futex hash table entries: 512 (order: 3, 32768 bytes)
    [    0.363630] pinctrl core: initialized pinctrl subsystem
    [    0.364517] DMI not present or invalid.
    [    0.364991] NET: Registered protocol family 16
    [    0.367758] DMA: preallocated 256 KiB pool for atomic coherent allocations
    [    0.368695] omap_hwmod: l3_main_2 using broken dt data from ocp
    [    0.581388] OMAP GPIO hardware version 0.1
    [    0.614459] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
    [    0.614473] hw-breakpoint: maximum watchpoint size is 8 bytes.
    [    0.616404] OMAP DMA hardware revision 0.0
    [    0.635225] edma 43300000.edma: memcpy is disabled
    [    0.641421] edma 43300000.edma: TI EDMA DMA engine driver
    [    0.654539] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver (LinkedList1/2/3 supported)
    [    0.660837] omap-iommu 40d01000.mmu: 40d01000.mmu registered
    [    0.661291] omap-iommu 40d02000.mmu: 40d02000.mmu registered
    [    0.661866] omap-iommu 58882000.mmu: 58882000.mmu registered
    [    0.662451] omap-iommu 55082000.mmu: 55082000.mmu registered
    [    0.663263] omap-iommu 41501000.mmu: 41501000.mmu registered
    [    0.663730] omap-iommu 41502000.mmu: 41502000.mmu registered
    [    0.664263] iommu: Adding device 58820000.ipu to group 1
    [    0.664484] iommu: Adding device 55020000.ipu to group 2
    [    0.664856] iommu: Adding device 40800000.dsp to group 0
    [    0.665463] iommu: Adding device 41000000.dsp to group 3
    [    0.666942] SCSI subsystem initialized
    [    0.668482] palmas 0-0058: Irq flag is 0x00000004
    [    0.692801] palmas 0-0058: Muxing GPIO 2f, PWM 0, LED 0
    [    0.694705] SMPS12: supplied by regulator-dummy
    [    0.696700] SMPS3: supplied by VMAIN
    [    0.698594] SMPS45: supplied by regulator-dummy
    [    0.700748] SMPS6: supplied by VMAIN
    [    0.702798] SMPS7: supplied by VMAIN
    [    0.704964] SMPS8: supplied by VMAIN
    [    0.706504] SMPS9: supplied by VMAIN
    [    0.707723] LDO1: supplied by VMAIN
    [    0.711580] random: fast init done
    [    0.711817] LDO2: supplied by VMAIN
    [    0.721641] LDO3: supplied by VMAIN
    [    0.731667] LDO4: supplied by VMAIN
    [    0.741705] LDO5: supplied by regulator-dummy
    [    0.742674] LDO6: supplied by regulator-dummy
    [    0.743638] LDO7: supplied by regulator-dummy
    [    0.744600] LDO8: supplied by regulator-dummy
    [    0.745522] LDO9: supplied by VMAIN
    [    0.752056] LDOLN: supplied by VMAIN
    [    0.761707] LDOUSB: supplied by VMAIN
    [    0.775768] omap_i2c 48070000.i2c: bus 0 rev0.12 at 400 kHz
    [    0.776071] media: Linux media interface: v0.10
    [    0.776134] videodev: Linux video capture interface: v2.00
    [    0.776260] pps_core: LinuxPPS API ver. 1 registered
    [    0.776270] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
    [    0.776296] PTP clock support registered
    [    0.776336] EDAC MC: Ver: 3.0.0
    [    0.777557] omap-mailbox 48840000.mailbox: omap mailbox rev 0x400
    [    0.777930] omap-mailbox 48842000.mailbox: omap mailbox rev 0x400
    [    0.781224] Advanced Linux Sound Architecture Driver Initialized.
    [    0.782411] clocksource: Switched to clocksource arch_sys_counter
    [    0.796463] NET: Registered protocol family 2
    [    0.797386] tcp_listen_portaddr_hash hash table entries: 256 (order: 1, 8192 bytes)
    [    0.797418] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
    [    0.797472] TCP bind hash table entries: 4096 (order: 4, 114688 bytes)
    [    0.797603] TCP: Hash tables configured (established 4096 bind 4096)
    [    0.797744] UDP hash table entries: 256 (order: 2, 16384 bytes)
    [    0.797782] UDP-Lite hash table entries: 256 (order: 2, 16384 bytes)
    [    0.798039] NET: Registered protocol family 1
    [    0.818663] RPC: Registered named UNIX socket transport module.
    [    0.818675] RPC: Registered udp transport module.
    [    0.818685] RPC: Registered tcp transport module.
    [    0.818694] RPC: Registered tcp NFSv4.1 backchannel transport module.
    [    0.820236] hw perfevents: no interrupt-affinity property for /pmu, guessing.
    [    0.820560] hw perfevents: enabled with armv7_cortex_a15 PMU driver, 7 counters available
    [    0.822391] Initialise system trusted keyrings
    [    0.822686] workingset: timestamp_bits=14 max_order=19 bucket_order=5
    [    0.832644] squashfs: version 4.0 (2009/01/31) Phillip Lougher
    [    0.843470] NFS: Registering the id_resolver key type
    [    0.843503] Key type id_resolver registered
    [    0.843514] Key type id_legacy registered
    [    0.843590] ntfs: driver 2.1.32 [Flags: R/O].
    [    0.845810] Key type asymmetric registered
    [    0.845826] Asymmetric key parser 'x509' registered
    [    0.846003] bounce: pool size: 64 pages
    [    0.846046] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
    [    0.846060] io scheduler noop registered
    [    0.846072] io scheduler deadline registered
    [    0.846342] io scheduler cfq registered (default)
    [    0.846356] io scheduler mq-deadline registered
    [    0.846368] io scheduler kyber registered
    [    0.854920] pinctrl-single 4a003400.pinmux: 282 pins, size 1128
    [    0.861499] dra7-pcie 51000000.pcie: Linked as a consumer to phy-4a094000.pciephy.1
    [    0.861863] dra7-pcie 51000000.pcie: WA for Errata i870 not applied
    [    0.862029] dra7-pcie 51000000.pcie: host bridge /ocp/axi@0/pcie@51000000 ranges:
    [    0.862071] dra7-pcie 51000000.pcie:    IO 0x20003000..0x20012fff -> 0x00000000
    [    0.862103] dra7-pcie 51000000.pcie:   MEM 0x20013000..0x2fffffff -> 0x20013000
    [    1.862391] dra7-pcie 51000000.pcie: Phy link never came up
    [    1.862667] dra7-pcie 51000000.pcie: PCI host bridge to bus 0000:00
    [    1.862687] pci_bus 0000:00: root bus resource [bus 00-ff]
    [    1.862700] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
    [    1.862712] pci_bus 0000:00: root bus resource [mem 0x20013000-0x2fffffff]
    [    1.868685] PCI: bus0: Fast back to back transfers disabled
    [    1.874226] PCI: bus1: Fast back to back transfers enabled
    [    1.874281] pci 0000:00:00.0: BAR 0: assigned [mem 0x20100000-0x201fffff 64bit]
    [    1.874302] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
    [    1.874957] pcieport 0000:00:00.0: Signaling PME with IRQ 172
    [    1.875270] pcieport 0000:00:00.0: AER enabled with IRQ 172
    [    1.882712] V3_3D: supplied by smps9
    [    1.883049] vtt_fixed: supplied by V3_3D
    [    1.960949] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
    [    1.967083] console [ttyS2] disabled
    [    1.967154] 48020000.serial: ttyS2 at MMIO 0x48020000 (irq = 45, base_baud = 3000000) is a 8250
    [    3.026459] console [ttyS2] enabled
    [    3.033169] omap_rng 48090000.rng: Random Number Generator ver. 20
    [    3.033377] hwrng: no data available
    [    3.044138] omapdss_dss 58000000.dss: Linked as a consumer to regulator.20
    [    3.051261] DSS: OMAP DSS rev 6.1
    [    3.058903] omapdss_hdmi5 58040000.encoder: Linked as a consumer to regulator.13
    [    3.066532] omapdss_hdmi5 58040000.encoder: Dropping the link to regulator.13
    [    3.095921] brd: module loaded
    [    3.112644] loop: module loaded
    [    3.125396] m25p80 spi0.0: s25fl256s1 (32768 Kbytes)
    [    3.130454] 7 fixed-partitions partitions found on MTD device spi0.0
    [    3.136839] Creating 7 MTD partitions on "spi0.0":
    [    3.141656] 0x000000000000-0x000000040000 : "QSPI.SPL"
    [    3.148116] 0x000000040000-0x000000140000 : "QSPI.u-boot"
    [    3.154774] 0x000000140000-0x0000001c0000 : "QSPI.u-boot-spl-os"
    [    3.161993] 0x0000001c0000-0x0000001d0000 : "QSPI.u-boot-env"
    [    3.168968] 0x0000001d0000-0x0000001e0000 : "QSPI.u-boot-env.backup1"
    [    3.176635] 0x0000001e0000-0x0000009e0000 : "QSPI.kernel"
    [    3.183349] 0x0000009e0000-0x000002000000 : "QSPI.file-system"
    [    3.191440] libphy: Fixed MDIO Bus: probed
    [    3.252465] davinci_mdio 48485000.mdio: davinci mdio revision 1.6, bus freq 1000000
    [    3.260161] davinci_mdio 48485000.mdio: detected phy mask fffffffc
    [    3.281179] libphy: 48485000.mdio: probed
    [    3.285215] davinci_mdio 48485000.mdio: phy[0]: device 48485000.mdio:00, driver Micrel KSZ9031 Gigabit PHY
    [    3.294917] davinci_mdio 48485000.mdio: phy[1]: device 48485000.mdio:01, driver Micrel KSZ9031 Gigabit PHY
    [    3.305960] cpsw 48484000.ethernet: Detected MACID = 94:e3:6d:4f:14:d2
    [    3.312605] cpsw 48484000.ethernet: initialized cpsw ale version 1.4
    [    3.318991] cpsw 48484000.ethernet: ALE Table size 1024
    [    3.324289] cpsw 48484000.ethernet: cpts: overflow check period 500 (jiffies)
    [    3.332917] cpsw 48484000.ethernet: cpsw: Detected MACID = 94:e3:6d:4f:14:d3
    [    3.342314] i2c /dev entries driver
    [    3.353055] sdhci: Secure Digital Host Controller Interface driver
    [    3.359266] sdhci: Copyright(c) Pierre Ossman
    [    3.365211] sdhci-pltfm: SDHCI platform and OF driver helper
    [    3.372709] omap_gpio 4805d000.gpio: Could not set line 27 debounce to 200000 microseconds (-22)
    [    3.381536] sdhci-omap 4809c000.mmc: Got CD GPIO
    [    3.386511] sdhci-omap 4809c000.mmc: Linked as a consumer to regulator.26
    [    3.393702] sdhci-omap 4809c000.mmc: Linked as a consumer to regulator.11
    [    3.400713] sdhci-omap 4809c000.mmc: Dropping the link to regulator.11
    [    3.407422] sdhci-omap 4809c000.mmc: Linked as a consumer to regulator.2
    [    3.414490] sdhci-omap 4809c000.mmc: Linked as a consumer to regulator.11
    [    3.421451] sdhci-omap 4809c000.mmc: no pinctrl state for ddr_3_3v mode
    [    3.455123] mmc0: SDHCI controller on 4809c000.mmc [4809c000.mmc] using ADMA
    [    3.463032] sdhci-omap 480b4000.mmc: Linked as a consumer to regulator.2
    [    3.469958] sdhci-omap 480b4000.mmc: Dropping the link to regulator.2
    [    3.476655] sdhci-omap 480b4000.mmc: Linked as a consumer to regulator.2
    [    3.509672] mmc1: SDHCI controller on 480b4000.mmc [480b4000.mmc] using ADMA
    [    3.529778] NET: Registered protocol family 10
    [    3.534298] mmc0: host does not support reading read-only switch, assuming write-enable
    [    3.544005] Segment Routing with IPv6
    [    3.547784] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
    [    3.554521] NET: Registered protocol family 17
    [    3.558520] mmc0: new high speed SDHC card at address aaaa
    [    3.559888] mmcblk0: mmc0:aaaa SC16G 14.8 GiB
    [    3.562690]  mmcblk0: p1 p2
    [    3.576059] Key type dns_resolver registered
    [    3.580550] Registering SWP/SWPB emulation handler
    [    3.585480] omap_voltage_late_init: Voltage driver support not added
    [    3.591988] Power Management for TI OMAP4+ devices.
    [    3.598070] Loading compiled-in X.509 certificates
    [    3.675393] dmm 4e000000.dmm: workaround for errata i878 in use
    [    3.684579] dmm 4e000000.dmm: initialized all PAT entries
    [    3.691783] omapdss_hdmi5 58040000.encoder: Linked as a consumer to regulator.13
    [    3.699418] omapdss_hdmi5 58040000.encoder: Dropping the link to regulator.13
    [    3.708733] omapdss_hdmi5 58040000.encoder: Linked as a consumer to regulator.13
    [    3.716688] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
    [    3.725016] omapdss_dss 58000000.dss: bound 58040000.encoder (ops hdmi5_component_ops)
    [    3.734429] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
    [    3.741074] [drm] No driver support for vblank timestamp query.
    [    3.747051] [drm] Cannot find any crtc or sizes
    [    3.753332] [drm] Initialized omapdrm 1.0.0 20110917 for omapdrm.0 on minor 0
    [    3.761362] hctosys: unable to open rtc device (rtc0)
    [    3.767978] ALSA device list:
    [    3.770960]   No soundcards found.
    [    3.779431] mmc1: new DDR MMC card at address 0001
    [    3.795320] mmcblk1: mmc1:0001 R1J56L 13.8 GiB
    [    3.800513] mmcblk1boot0: mmc1:0001 R1J56L partition 1 4.00 MiB
    [    3.807101] mmcblk1boot1: mmc1:0001 R1J56L partition 2 4.00 MiB
    [    3.813265] mmcblk1rpmb: mmc1:0001 R1J56L partition 3 128 KiB, chardev (244:0)
    [    4.420031] EXT4-fs (mmcblk0p2): recovery complete
    [    4.425760] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
    [    4.433956] VFS: Mounted root (ext4 filesystem) on device 179:2.
    [    4.451916] devtmpfs: mounted
    [    4.457977] Freeing unused kernel memory: 2048K
    [    4.462678] Run /sbin/init as init process
    [    4.791561] systemd[1]: System time before build time, advancing clock.
    [    4.792521] [drm] Cannot find any crtc or sizes
    [    4.841910] systemd[1]: systemd 239 running in system mode. (+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL +)
    [    4.863932] systemd[1]: Detected architecture arm.
    
    Welcome to Arago 2019.11!
    
    [    4.916336] systemd[1]: Set hostname to <am57xx-evm>.
    [    5.186478] systemd[1]: File /lib/systemd/system/systemd-journald.service:36 configures an IP firewall (IPAddressDeny=any), but the local system does not sup.
    [    5.203619] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
    [    5.495917] random: systemd: uninitialized urandom read (16 bytes read)
    [    5.508333] systemd[1]: Created slice system-getty.slice.
    [  OK  ] Created slice system-getty.slice.
    [    5.542688] random: systemd: uninitialized urandom read (16 bytes read)
    [    5.549717] systemd[1]: Listening on initctl Compatibility Named Pipe.
    [  OK  ] Listening on initctl Compatibility Named Pipe.
    [    5.592605] random: systemd: uninitialized urandom read (16 bytes read)
    [    5.600015] systemd[1]: Listening on Journal Socket (/dev/log).
    [  OK  ] Listening on Journal Socket (/dev/log).
    [    5.677769] systemd[1]: Reached target Remote File Systems.
    [  OK  ] Reached target Remote File Systems.
    [    5.713044] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
    [  OK  ] Started Dispatch Password Requests to Console Directory Watch.
    [  OK  ] Listening on Journal Socket.
             Mounting POSIX Message Queue File System...
             Starting Create list of required st▒…ce nodes for the current kernel...
    [  OK  ] Started Hardware RNG Entropy Gatherer Daemon.
    [    5.890491] random: crng init done
    [    5.895042] random: 6 urandom warning(s) missed due to ratelimiting
             Starting Load Kernel Modules...
    [    5.945606] cmemk: loading out-of-tree module taints kernel.
    [  OK  ] Created slice User and Session Slice.
    [    5.956720] CMEMK module: reference Linux version 4.19.79
    [    5.967541] allocated heap buffer 0x40500000 of size 0x100000
    [    5.973353] cmemk initialized
    [  OK  ] Started Forward Password Requests to Wall Directory Watch.
    [    5.987807] cryptodev: driver 1.9 loaded.
             Starting Remount Root and Kernel File Systems...
    [  OK  ] Reached target Paths.
    [    6.073215] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
    [    6.083016] usbcore: registered new interface driver usbfs
    [    6.088629] usbcore: registered new interface driver hub
    [  OK  ] Reached target Swap.
    [    6.094200] usbcore: registered new device driver usb
    [    6.124186] usbcore: registered new interface driver ftdi_sio
    [    6.130037] usbserial: USB Serial support registered for FTDI USB Serial Device
             Mounting Temporary Directory (/tmp)...
    [  OK  ] Listening on Process Core Dump Socket.
    [  OK  ] Reached target Slices.
    [  OK  ] Listening on Network Service Netlink Socket.
             Starting Journal Service...
             Mounting Kernel Debug File System...
    [  OK  ] Listening on udev Control Socket.
    [  OK  ] Listening on udev Kernel Socket.
             Starting udev Coldplug all Devices...
    [  OK  ] Created slice system-serial\x2dgetty.slice.
    [  OK  ] Mounted POSIX Message Queue File System.
    [  OK  ] Started Journal Service.
    [  OK  ] Started Create list of required sta▒…vice nodes for the current kernel.
    [  OK  ] Started Load Kernel Modules.
    [  OK  ] Started Remount Root and Kernel File Systems.
    [  OK  ] Mounted Temporary Directory (/tmp).
    [  OK  ] Mounted Kernel Debug File System.
             Mounting Kernel Configuration File System...
             Starting Apply Kernel Variables...
             Starting Create Static Device Nodes in /dev...
             Starting Flush Journal to Persistent Storage...
    [  OK  ] Mounted Kernel Configuration File System.
    [  OK  ] Started Apply Kernel Variables.
    [  OK  ] Started Create Static Device Nodes in /dev.
    [    7.065624] systemd-journald[142]: Received request to flush runtime journal from PID 1
             Starting udev Kernel Device Manager...
    [  OK  ] Reached target Local File Systems (Pre).
    [  OK  ] Reached target Containers.
             Mounting /var/volatile...
             Mounting /media/ram...
    [  OK  ] Started udev Kernel Device Manager.
    [  OK  ] Started Flush Journal to Persistent Storage.
    [  OK  ] Mounted /var/volatile.
    [  OK  ] Mounted /media/ram.
             Starting Load/Save Random Seed...
    [  OK  ] Reached target Local File Systems.
             Starting Create Volatile Files and Directories...
    [  OK  ] Started Load/Save Random Seed.
    [  OK  ] Started Create Volatile Files and Directories.
             Starting Network Service...
             Starting Network Time Synchronization...
             Starting Update UTMP about System Boot/Shutdown...
    [  OK  ] Started Update UTMP about System Boot/Shutdown.
    [  OK  ] Started Network Service.
             Starting Network Name Resolution...
             Starting Wait for Network to be Configured...
    [  OK  ] Started Network Time Synchronization.
    [  OK  ] Reached target System Time Synchronized.
    [  OK  ] Started Network Name Resolution.
    [  OK  ] Reached target Host and Network Name Lookups.
    [  OK  ] Reached target Network.
    [    8.815207] omap-rproc 58820000.ipu: ignoring dependency for device, assuming no driver
    [    8.861979] omap-rproc 58820000.ipu: ignoring dependency for device, assuming no driver
    [    8.883227] omap-rproc 58820000.ipu: assigned reserved memory node ipu1-memory@9d000000
    [    8.911046] remoteproc remoteproc0: 58820000.ipu is available
    [    8.934638] omap-rproc 55020000.ipu: ignoring dependency for device, assuming no driver
    [    9.027075] omap-rproc 55020000.ipu: ignoring dependency for device, assuming no driver
    [    9.071455] omap-rproc 55020000.ipu: assigned reserved memory node ipu2-memory@95800000
    [    9.118021] remoteproc remoteproc1: 55020000.ipu is available
    [    9.132574] omap-rproc 40800000.dsp: ignoring dependency for device, assuming no driver
    [    9.143421] omap-rproc 40800000.dsp: ignoring dependency for device, assuming no driver
    [    9.152616] omap-rproc 40800000.dsp: assigned reserved memory node dsp1-memory@99000000
    [    9.178433] remoteproc remoteproc2: 40800000.dsp is available
    [    9.197948] omap-rproc 41000000.dsp: ignoring dependency for device, assuming no driver
    [    9.249702] omap-rproc 41000000.dsp: ignoring dependency for device, assuming no driver
    [    9.261614] omap-rproc 41000000.dsp: assigned reserved memory node dsp2-memory@9f000000
    [    9.280935] remoteproc remoteproc3: 41000000.dsp is available
    [    9.355063] remoteproc remoteproc0: powering up 58820000.ipu
    [    9.364369] remoteproc remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 6884184
    [    9.410998] virtio_rpmsg_bus virtio0: rpmsg host is online
    [    9.411267] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d
    [    9.425165] remoteproc remoteproc0: registered virtio0 (type 7)
    [    9.431407] remoteproc remoteproc0: remote processor 58820000.ipu is now up
    [    9.517987] remoteproc remoteproc1: powering up 55020000.ipu
    [    9.525373] remoteproc remoteproc1: Booting fw image dra7-ipu2-fw.xem4, size 3747220
    [    9.560742] omap-iommu 55082000.mmu: 55082000.mmu: version 2.1
    [    9.791681] virtio_rpmsg_bus virtio1: rpmsg host is online
    [    9.794866] virtio_rpmsg_bus virtio1: creating channel rpmsg-rpc addr 0x65
    [    9.795849] virtio_rpmsg_bus virtio1: creating channel rpmsg-rpc addr 0x66
    [  OK  [    9.831049] remoteproc remoteproc1: registered virtio1 (type 7)
    ] Started udev Coldplug all Devices.
    [    9.840419] remoteproc remoteproc1: remote processor 55020000.ipu is now up
    [  OK  ] Reached target System Initialization.
    [  OK  ] Started Daily rotation of log files.
    [  OK  ] Listening on dropbear.socket.
    [  OK  ] Listening on RPCbind Server Activation Socket.
    [  OK  ] Listening on D-Bus System Message Bus Socket.
    [  OK  ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
    [  OK  ] Reached target Sockets.
    [  OK  ] Reached target Basic System.
    [   10.421714] Driver for 1-wire Dallas network protocol.
    [   10.422855] omap_gpio 48057000.gpio: Could not set line 16 debounce to 20000 microseconds (-22)
             Starting Permit User Sessions...
             Starting Login Service...
    [   10.542240] palmas-rtc 48070000.i2c:tps659038@58:tps659038_rtc: rtc core: registered 48070000.i2c:tps659038@58:tps659038_rtc as rtc0
    [   10.592605] omap_hdq 480b2000.1w: OMAP HDQ Hardware Rev 0.:. Driver in Interrupt mode
    [  OK  ] Started Job spooling tools.
    [   10.732865] ov2659 0-0030: Sensor detection failed (3030, 0)
    [   10.743748] omap-des 480a5000.des: OMAP DES hw accel rev: 2.2
    [   10.751910] omap-des 480a5000.des: will run requests pump with realtime priority
    [   10.849660] w1_master_driver w1_bus_master1: Attaching one wire slave 01.000000000000 crc 3d
             Starting Enable and configure wl18xx bluetooth stack...[   10.922588] w1_master_driver w1_bus_master1: Family 1 for 01.000000000000.3d is not registere.
    
    [  OK  ] Started Redis In-Memory Data Store.
    [   11.053115] omap_rtc 48838000.rtc: registered as rtc1
             Starting Avahi mDNS/DNS-SD Stack...
    [  OK  ] Started Daily Cleanup of Temporary Directories.
    [  OK  ] Reached target Timers.
             Starting TI MultiCore Tools Daemon...
             Starting RPC Bind Service...
             Starting Lightning Fast Webserver With Light System Requirements...
             Starting rc.pvr.service...
             Starting Print notice about GPLv3 packages...
    [   11.456967] remoteproc remoteproc2: powering up 40800000.dsp
    [   11.462808] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 20482260
    [   11.471374] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
    [   11.477305] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
             Starting TI IPC Daemon...
    [   11.531236] remoteproc remoteproc3: powering up 41000000.dsp
    [   11.548889] remoteproc remoteproc3: Booting fw image dra7-dsp2-fw.xe66, size 20482260
    [   11.577988] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
    [   11.585199] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
    [   11.616225] virtio_rpmsg_bus virtio2: rpmsg host is online
    [   11.616328] virtio_rpmsg_bus virtio2: creating channel rpmsg-proto addr 0x3d
    [   11.631664] vpe 489d0000.vpe: loading firmware vpdma-1b8.bin
    [   11.656485] remoteproc remoteproc2: registered virtio2 (type 7)
    [   11.742182] remoteproc remoteproc2: remote processor 40800000.dsp is now up
    [   11.813086] virtio_rpmsg_bus virtio3: rpmsg host is online
    [   11.813189] virtio_rpmsg_bus virtio3: creating channel rpmsg-proto addr 0x3d
             Starting Reboot and dump vmcore via kexec...
    [   11.881947] remoteproc remoteproc3: registered virtio3 (type 7)
    [   11.920090] remoteproc remoteproc3: remote processor 41000000.dsp is now up
             Starting Simple Network Management Protocol (SNMP) Daemon....
    [   11.964704] phy phy-4a084000.phy.3: Linked as a consumer to regulator.21
    [   11.983053] [drm] Initialized pvr 1.17.4948957 20110701 for 56000000.gpu on minor 1
    [  OK  ] Started D-Bus System Message Bus.
    [   12.020670] phy phy-4a085000.phy.4: Linked as a consumer to regulator.21
    [   12.023201] vip 48990000.vip: loading firmware vpdma-1b8.bin
    [   12.092499] vip 48990000.vip: VPDMA firmware loaded
    [   12.093217] vpe 489d0000.vpe: Device registered as /dev/video0
    [   12.142684] PVR_K: UM DDK-(4948957) and KM DDK-(4948957) match. [ OK ]
    [  OK  ] Started Periodic Command Scheduler.
    [   12.375369] net eth0: initializing cpsw version 1.15 (0)
    [   12.444099] ahci 4a140000.sata: controller can't do 64bit DMA, forcing 32bit
    [  OK  ] Started Permit User Sessions.
    [   12.465194] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
    [   12.492937] ahci 4a140000.sata: forcing port_map 0x0 -> 0x1
    [   12.523076] Micrel KSZ9031 Gigabit PHY 48485000.mdio:00: attached PHY driver [Micrel KSZ9031 Gigabit PHY] (mii_bus:phy_addr=48485000.mdio:00, irq=POLL)
    [   12.547395] omap-sham 4b101000.sham: hw accel on OMAP rev 4.3
    [   12.557016] ahci 4a140000.sata: AHCI 0001.0300 32 slots 1 ports 3 Gbps 0x1 impl platform mode
    [   12.557030] ahci 4a140000.sata: flags: ncq sntf pm led clo only pmp pio slum part ccc apst
    [   12.631452] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    [   12.648438] omap-aes 4b500000.aes: OMAP AES hw accel rev: 3.3
    [   12.649902] CAN device driver interface
    [   12.663465] omap-aes 4b500000.aes: will run requests pump with realtime priority
    [   12.678714] omap-aes 4b700000.aes: OMAP AES hw accel rev: 3.3
    [   12.685961] omap-aes 4b700000.aes: will run requests pump with realtime priority
    [   12.701957] scsi host0: ahci
    [   12.786822] ata1: SATA max UDMA/133 mmio [mem 0x4a140000-0x4a1410ff] port 0x100 irq 81
    [   12.841910] c_can_platform 4ae3c000.can: c_can_platform device registered (regs=c2ebcaa7, irq=89)
    [  OK  ] Started Enable and configure wl18xx bluetooth stack.
    [  OK  ] Started TI MultiCore Tools Daemon.
    [  OK  ] Started RPC Bind Service.
    [   13.002385] net eth1: initializing cpsw version 1.15 (0)
    [  OK  ] Started Lightning Fast Webserver With Light System Requirements.
    [   13.124689] ata1: SATA link down (SStatus 0 SControl 300)
    [  OK  ] Started rc.pvr.service.
    [  OK  ] Started TI IPC Daemon.
    [   13.153066] Micrel KSZ9031 Gigabit PHY 48485000.mdio:01: attached PHY driver [Micrel KSZ9031 Gigabit PHY] (mii_bus:phy_addr=48485000.mdio:01, irq=POLL)
    [  OK  ] Started Reboot and dump vmcore via kexec.
    [   13.197826] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    [  OK  ] Found device /dev/ttyS2.
    [   14.070557] omap-hdmi-audio omap-hdmi-audio.0.auto: snd-soc-dummy-dai <-> 58040000.encoder mapping ok
    [   14.129615] omap-hdmi-audio omap-hdmi-audio.0.auto: ASoC: no DMI vendor name!
    [  OK  ] Started Simple Network Management Protocol (SNMP) Daemon..
    [   14.492514] davinci_mdio 4b2b2400.mdio: davinci mdio revision 1.6, bus freq 1000000
    [   14.505629] libphy: 4b2b2400.mdio: probed
    [   14.609890] davinci_mdio 4b2b2400.mdio: phy[0]: device 4b2b2400.mdio:00, driver TI TLK10X 10/100 Mbps PHY
    [   14.619515] davinci_mdio 4b2b2400.mdio: phy[1]: device 4b2b2400.mdio:01, driver TI TLK10X 10/100 Mbps PHY
    [   14.726364] remoteproc remoteproc4: 4b234000.pru is available
    [   14.732302] pru-rproc 4b234000.pru: PRU rproc node pru@4b234000 probed successfully
    [   14.791105] remoteproc remoteproc5: 4b238000.pru is available
    [   14.815872] pru-rproc 4b238000.pru: PRU rproc node pru@4b238000 probed successfully
    [   14.895787] remoteproc remoteproc6: 4b2b4000.pru is available
    [   14.921477] pru-rproc 4b2b4000.pru: PRU rproc node pru@4b2b4000 probed successfully
    [   14.943388] remoteproc remoteproc7: 4b2b8000.pru is available
    [   14.949494] pru-rproc 4b2b8000.pru: PRU rproc node pru@4b2b8000 probed successfully
    [   15.257527] prueth pruss2_eth: pruss_fw_drop_untagged_vlan 0
    [   15.263698] prueth pruss2_eth: pruss MC Mask (Port 0) ff:ff:ff:ff:ff:ff
    [   15.288220] prueth pruss2_eth: pruss MC Mask (Port 1) ff:ff:ff:ff:ff:ff
    [   15.385067] prueth pruss2_eth: TI PRU ethernet (type 0) driver initialized
    [   16.814157] NET: Registered protocol family 45
    [   17.924918] rpmsg_rpc virtio1.rpmsg-rpc.-1.101: probing service dce-callback with src 1024 dst 101
    [   17.978076] rpmsg_rpc virtio1.rpmsg-rpc.-1.101: published functions = 4
    [   17.985564] rpmsg_rpc virtio1.rpmsg-rpc.-1.102: probing service rpmsg-dce with src 1025 dst 102
    [   18.028129] rpmsg_rpc virtio1.rpmsg-rpc.-1.102: published functions = 9
    [   18.224286] dwc3 48890000.usb: Failed to get clk 'ref': -2
    [   18.254697] dwc3 488d0000.usb: Failed to get clk 'ref': -2
    [   19.165660] pruss_intc_irq_set_affinity: event 20 not configured: ch = -1, host = -1
    [   19.165802] pruss_intc_irq_set_affinity: event 22 not configured: ch = -1, host = -1
    [   19.165904] pruss_intc_irq_set_affinity: event 26 not configured: ch = -1, host = -1
    [   19.173445] iep ptp bc clkid -1
    [   19.216743] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
    [   19.216782] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 1
    [   19.217531] xhci-hcd xhci-hcd.1.auto: hcc params 0x0220f04c hci version 0x100 quirks 0x0000000002010010
    [   19.230439] xhci-hcd xhci-hcd.1.auto: irq 181, io mem 0x48890000
    [   19.231369] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19
    [   19.231380] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [   19.231388] usb usb1: Product: xHCI Host Controller
    [   19.231396] usb usb1: Manufacturer: Linux 4.19.79-rt28-g5baf382c8f xhci-hcd
    [   19.231403] usb usb1: SerialNumber: xhci-hcd.1.auto
    [   19.233334] hub 1-0:1.0: USB hub found
    [   19.235736] hub 1-0:1.0: 1 port detected
    [   19.236977] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
    [   19.237004] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 2
    [   19.237024] xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0 SuperSpeed
    [   19.238587] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
    [   19.241443] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 4.19
    [   19.241454] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [   19.241462] usb usb2: Product: xHCI Host Controller
    [   19.241470] usb usb2: Manufacturer: Linux 4.19.79-rt28-g5baf382c8f xhci-hcd
    [   19.241477] usb usb2: SerialNumber: xhci-hcd.1.auto
    [   19.244416] hub 2-0:1.0: USB hub found
    [   19.244926] hub 2-0:1.0: 1 port detected
    [   19.533091] remoteproc remoteproc6: powering up 4b2b4000.pru
    [   19.543853] remoteproc remoteproc6: Booting fw image ti-pruss/am57xx-pru0-prueth-fw.elf, size 7124
    [   19.557817] pruss 4b280000.pruss: configured system_events[63-0] = 00000600,04500000
    [   19.565614] pruss 4b280000.pruss: configured intr_channels = 0x000000d5 host_intr = 0x00000155
    [   19.575133] remoteproc remoteproc6: remote processor 4b2b4000.pru is now up
    [   19.586831] net eth2: started
    [   19.590696] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
    [   19.657967] pruss_intc_irq_set_affinity: event 21 not configured: ch = -1, host = -1
    [   19.658135] pruss_intc_irq_set_affinity: event 23 not configured: ch = -1, host = -1
    [   19.658245] pruss_intc_irq_set_affinity: event 27 not configured: ch = -1, host = -1
    [   19.658390] remoteproc remoteproc7: powering up 4b2b8000.pru
    [   19.731420] remoteproc remoteproc7: Booting fw image ti-pruss/am57xx-pru1-prueth-fw.elf, size 7152
    [   19.740924] pruss 4b280000.pruss: configured system_events[63-0] = 00600000,08a00000
    [   19.752048] pruss 4b280000.pruss: configured intr_channels = 0x0000032a host_intr = 0x000002aa
    [   19.769620] remoteproc remoteproc7: remote processor 4b2b8000.pru is now up
    [   19.776670] net eth3: started
    [   19.780256] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
    ***************************************************************
    ***************************************************************
    NOTICE: This file system contains the following GPLv3 packages:
            autoconf
            bash-dev
            bash
            bc
            binutils-dev
            binutils
            bison-dev
            bison
            cifs-utils
            cpio
            cpp-symlinks
            cpp
            dosfstools
            elfutils-dev
            elfutils
            findutils
            g++-symlinks
            g++
            gawk
            gcc-symlinks
            gcc
            gdb
            gdbc6x
            gdbserver
            gettext
            glmark2
            gstreamer1.0-libav
            gzip
            hidapi
            libasm1
            libbfd
            libcairo-perf-utils
            libdw1
            libelf1
            libgdbm-compat4
            libgdbm-dev
            libgdbm6
            libgettextlib
            libgettextsrc
            libgmp10
            libidn2-0
            libmavconn
            libmpc3
            libmpfr6
            libreadline-dev
            libreadline7
            libunistring2
            m4-dev
            m4
            make
            mavlink
            mavros-extras
            mavros-msgs
            mavros
            nettle
            parted
            pdm-anomaly-detection
            socketcan-interface
            swig-dev
            swig
            which
    
    If you do not wish to distribute GPLv3 components please remove
    the above packages prior to distribution.  This can be done using
    the opkg remove command.  i.e.:
        opkg remove <package>
    Where <package> is the name printed in the list above
    
    NOTE: If the package is a dependency of another package you
          will be notified of the dependent packages.  You should
          use the --force-removal-of-dependent-packages option to
          also remove the dependent packages as well
    ***************************************************************
    ***************************************************************
    [  OK  ] Started Print notice about GPLv3 packages.
    [   21.241129] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
    [   21.248615] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
    [   21.471418] prueth pruss2_eth eth3: Link is Up - 100Mbps/Full - flow control off
    [   21.478904] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
             Starting Save/Restore Sound Card State...
    [  OK  ] Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
    [  OK  ] Created slice system-systemd\x2dfsck.slice.
             Starting File System Check on /dev/mmcblk0p1...
             Starting weston.service...
    [  OK  ] Started NFS status monitor for NFSv2/3 locking..
    [  OK  ] Started Serial Getty on ttyS2.
    [  OK  ] Started Getty on tty1.
    [  OK  ] Reached target Login Prompts.
             Starting Synchronize System and HW clocks...
    [  OK  ] Started Save/Restore Sound Card State.
    [  OK  ] Started Synchronize System and HW clocks.
    [  OK  ] Reached target Sound Card.
    [  OK  ] Started Login Service.
    [  OK  ] Started Avahi mDNS/DNS-SD Stack.
    [  OK  ] Started File System Check on /dev/mmcblk0p1.
             Mounting /run/media/mmcblk0p1...
    [  OK  ] Mounted /run/media/mmcblk0p1.
    [  OK  ] Started weston.service.
             Starting Matrix GUI...
             Starting telnetd.service...
    [  OK  ] Started Matrix GUI.
    [  OK  ] Started telnetd.service.
             Starting thttpd.service...
    [  OK  ] Started thttpd.service.
    
     _____                    _____           _         _
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
                  |___|                    |___|
    
    Arago Project http://arago-project.org am57xx-evm ttyS2
    
    Arago 2019.11 am57xx-evm ttyS2
    
    am57xx-evm login: [  128.809039] NET: Registered protocol family 15
    [  128.958801] Initializing XFRM netlink socket
    [  455.739500] Unhandled fault: asynchronous external abort (0x1211) at 0x00000000
    [  455.739503] ------------[ cut here ]------------
    [  455.739507] pgd = f3ed2eed
    [  455.739514] [00000000] *pgd=80000080004003, *pmd=00000000
    [  455.739528] Internal error: : 1211 [#1] PREEMPT SMP ARM
    [  455.739532] Modules linked in: xfrm_user xfrm4_tunnel ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo xhci_plat_hcd xhci_hcd dwc3 udc_core rpmsg_rpc rpmsg_proto)
    [  455.739683] CPU: 1 PID: 956 Comm: irq/210-eth2 Tainted: G           O      4.19.79-rt28-g5baf382c8f #1
    [  455.739687] Hardware name: Generic DRA74X (Flattened Device Tree)
    [  455.739699] PC is at memcpy+0x244/0x330
    [  455.739702] LR is at   (null)
    [  455.739707] pc : [<c09eeec4>]    lr : [<00000000>]    psr: 20010013
    [  455.739711] sp : d1fc9dd0  ip : 00000000  fp : d1fc9e6c
    [  455.739715] r10: e18fe3a0  r9 : 00000000  r8 : 00000000
    [  455.739719] r7 : 00000000  r6 : 00000000  r5 : 00000000  r4 : 00000000
    [  455.739723] r3 : 00000000  r2 : 0000019e  r1 : e18fe3c4  r0 : d0d63d84
    [  455.739728] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
    [  455.739733] Control: 30c5387d  Table: 92159bc0  DAC: 55555555
    [  455.739737] Process irq/210-eth2 (pid: 956, stack limit = 0xc7c821ed)
    [  455.739741] Stack: (0xd1fc9dd0 to 0xd1fca000)
    [  455.739747] 9dc0:                                     d0d63d82 d2bc0840 d2a1e000 d1e9a040
    [  455.739753] 9de0: 00000240 d0d63d82 00000240 bf34961c d1fc9e5c d1fc9e00 c02019f8 bf35915c
    [  455.739759] 9e00: d0d63d80 bf359480 00000000 00000240 bf359480 c1207488 ffffff1d 00000000
    [  455.739765] 9e20: e1900000 00000000 d2a1e580 00000000 d2a1e580 e7470000 e7470071 7567e7b3
    [  455.739770] 9e40: d2a1e000 e7470071 00000000 d2a1e000 00000071 d1e9a040 d1fc9ebc e7471c88
    [  455.739776] 9e60: d1fc9f04 d1fc9e78 bf349df0 bf349470 00000000 00000000 00000240 00000100
    [  455.739781] 9e80: c1207400 d20698c0 bf359480 dff59080 c1207488 bf355b68 bf359480 00000000
    [  455.739786] 9ea0: bf355b64 00000000 bf359480 000005f2 d2a1e580 e7470000 05d40000 00000000
    [  455.739792] 9ec0: 00000000 00000240 00000100 c1207400 d20698c0 7567e7b3 00000004 d14f0a40
    [  455.739797] 9ee0: d1d84b00 d14f0a64 ffffe000 00000000 c0289f2c 00000001 d1fc9f24 d1fc9f08
    [  455.739803] 9f00: c0289f50 bf349c40 d1d84b00 d14f0a40 d14f0a64 ffffe000 d1fc9f74 d1fc9f28
    [  455.739807] 9f20: c028a288 c0289f38 c0dccdb4 c1207488 c12558fd d20698c0 ffffe000 00000000
    [  455.739813] 9f40: c028a070 7567e7b3 c024bb58 d14f0700 d14f0a80 00000000 d1fc8000 d14f0a40
    [  455.739818] 9f60: c028a148 d29a7974 d1fc9fac d1fc9f78 c024c1e4 c028a154 d14f071c d14f071c
    [  455.739823] 9f80: 00000000 d14f0a80 c024c084 00000000 00000000 00000000 00000000 00000000
    [  455.739828] 9fa0: 00000000 d1fc9fb0 c02010e0 c024c090 00000000 00000000 00000000 00000000
    [  455.739833] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [  455.739839] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
    [  455.739841] Backtrace:
    [  455.739884] [<bf349464>] (emac_rx_packet [ti_prueth]) from [<bf349df0>] (emac_rx_thread+0x1bc/0x2cc [ti_prueth])
    [  455.739892]  r10:e7471c88 r9:d1fc9ebc r8:d1e9a040 r7:00000071 r6:d2a1e000 r5:00000000
    [  455.739895]  r4:e7470071
    [  455.739918] [<bf349c34>] (emac_rx_thread [ti_prueth]) from [<c0289f50>] (irq_thread_fn+0x24/0x80)
    [  455.739924]  r10:00000001 r9:c0289f2c r8:00000000 r7:ffffe000 r6:d14f0a64 r5:d1d84b00
    [  455.739927]  r4:d14f0a40
    [  455.739936] [<c0289f2c>] (irq_thread_fn) from [<c028a288>] (irq_thread+0x140/0x22c)
    [  455.739941]  r7:ffffe000 r6:d14f0a64 r5:d14f0a40 r4:d1d84b00
    [  455.739951] [<c028a148>] (irq_thread) from [<c024c1e4>] (kthread+0x160/0x168)
    [  455.739957]  r10:d29a7974 r9:c028a148 r8:d14f0a40 r7:d1fc8000 r6:00000000 r5:d14f0a80
    [  455.739960]  r4:d14f0700
    [  455.739968] [<c024c084>] (kthread) from [<c02010e0>] (ret_from_fork+0x14/0x34)
    [  455.739973] Exception stack(0xd1fc9fb0 to 0xd1fc9ff8)
    [  455.739978] 9fa0:                                     00000000 00000000 00000000 00000000
    [  455.739983] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [  455.739987] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [  455.739993]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c024c084
    [  455.739996]  r4:d14f0a80
    [  455.740003] Code: e1a09829 e189980c e1a0c82c e18cc80e (e8a013f8)
    [  455.740011] ---[ end trace 0000000000000002 ]---
    [  455.740039] genirq: exiting task "irq/210-eth2" (956) is an active IRQ thread (irq 210)
    [  456.165164] Unhandled fault: asynchronous external abort (0x1211) at 0x00000000
    [  456.165169] pgd = f3ed2eed
    [  456.165173] [00000000] *pgd=80000080004003, *pmd=00000000
    [  456.165183] Internal error: : 1211 [#2] PREEMPT SMP ARM
    [  456.165185] Modules linked in: xfrm_user xfrm4_tunnel ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo xhci_plat_hcd xhci_hcd dwc3 udc_core rpmsg_rpc rpmsg_proto)
    [  456.165311] CPU: 1 PID: 1004 Comm: irq/209-eth3 Tainted: G      D    O      4.19.79-rt28-g5baf382c8f #1
    [  456.165315] Hardware name: Generic DRA74X (Flattened Device Tree)
    [  456.165322] PC is at memcpy+0xc8/0x330
    [  456.165343] LR is at emac_rx_packet+0x1dc/0x7d4 [ti_prueth]
    [  456.165347] pc : [<c09eed48>]    lr : [<bf34963c>]    psr: 40060013
    [  456.165350] sp : d148bde4  ip : 00000000  fp : d148be6c
    [  456.165354] r10: e18fe860  r9 : 00000002  r8 : d1e9a040
    [  456.165357] r7 : d2a1e800  r6 : d2a56240  r5 : d1ff9842  r4 : 00000002
    [  456.165360] r3 : e1900000  r2 : 00000000  r1 : e19048c0  r0 : d1ff9844
    [  456.165365] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
    [  456.165370] Control: 30c5387d  Table: 92159bc0  DAC: 55555555
    [  456.165374] Process irq/209-eth3 (pid: 1004, stack limit = 0xdf3099fe)
    [  456.165378] Stack: (0xd148bde4 to 0xd148c000)
    [  456.165384] bde0:          d1ff9844 00000002 bf34963c 00003d64 00000001 350c07a0 bf35915c
    [  456.165390] be00: d1ff9840 bf359498 00000000 00000002 bf359498 c1207488 fffffcfd 00000000
    [  456.165395] be20: e1900000 00000000 d2a1ed80 00000000 d2a1ed80 e7470000 e747010a 7567e7b3
    [  456.165401] be40: d2a1e800 e747010a 00000000 d2a1e800 0000010a d1e9a040 d148bebc e7471ca0
    [  456.165406] be60: d148bf04 d148be78 bf349df0 bf349470 00000000 00000002 00000002 00000000
    [  456.165411] be80: c1207400 d1d96300 bf359498 dff68080 c1207488 bf355b70 bf359480 00000000
    [  456.165416] bea0: bf355b70 00000004 bf359498 000005f2 d2a1ed80 e7470000 0f400000 00000000
    [  456.165422] bec0: 00000002 00000002 00000000 c1207400 d1d96300 7567e7b3 00000004 d146bfc0
    [  456.165427] bee0: d1d84800 d146bfe4 ffffe000 00000000 c0289f2c 00000001 d148bf24 d148bf08
    [  456.165433] bf00: c0289f50 bf349c40 d1d84800 d146bfc0 d146bfe4 ffffe000 d148bf74 d148bf28
    [  456.165438] bf20: c028a288 c0289f38 c0dccdb4 c1207488 c12558fd d1d96300 ffffe000 00000000
    [  456.165443] bf40: c028a070 7567e7b3 c024bb58 d145f280 d146b180 00000000 d148a000 d146bfc0
    [  456.165449] bf60: c028a148 d29a7974 d148bfac d148bf78 c024c1e4 c028a154 d145f29c d145f29c
    [  456.165454] bf80: 00000000 d146b180 c024c084 00000000 00000000 00000000 00000000 00000000
    [  456.165459] bfa0: 00000000 d148bfb0 c02010e0 c024c090 00000000 00000000 00000000 00000000
    [  456.165464] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [  456.165469] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
    [  456.165471] Backtrace:
    [  456.165507] [<bf349464>] (emac_rx_packet [ti_prueth]) from [<bf349df0>] (emac_rx_thread+0x1bc/0x2cc [ti_prueth])
    [  456.165514]  r10:e7471ca0 r9:d148bebc r8:d1e9a040 r7:0000010a r6:d2a1e800 r5:00000000
    [  456.165517]  r4:e747010a
    [  456.165538] [<bf349c34>] (emac_rx_thread [ti_prueth]) from [<c0289f50>] (irq_thread_fn+0x24/0x80)
    [  456.165544]  r10:00000001 r9:c0289f2c r8:00000000 r7:ffffe000 r6:d146bfe4 r5:d1d84800
    [  456.165547]  r4:d146bfc0
    [  456.165555] [<c0289f2c>] (irq_thread_fn) from [<c028a288>] (irq_thread+0x140/0x22c)
    [  456.165561]  r7:ffffe000 r6:d146bfe4 r5:d146bfc0 r4:d1d84800
    [  456.165569] [<c028a148>] (irq_thread) from [<c024c1e4>] (kthread+0x160/0x168)
    [  456.165575]  r10:d29a7974 r9:c028a148 r8:d146bfc0 r7:d148a000 r6:00000000 r5:d146b180
    [  456.165577]  r4:d145f280
    [  456.165584] [<c024c084>] (kthread) from [<c02010e0>] (ret_from_fork+0x14/0x34)
    [  456.165588] Exception stack(0xd148bfb0 to 0xd148bff8)
    [  456.165593] bfa0:                                     00000000 00000000 00000000 00000000
    [  456.165597] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [  456.165602] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [  456.165608]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c024c084
    [  456.165610]  r4:d146b180
    [  456.165617] Code: e8bd01e0 e1b02f82 14d13001 24d14001 (24d1c001)
    [  456.165623] ---[ end trace 0000000000000003 ]---
    [  456.165648] genirq: exiting task "irq/209-eth3" (1004) is an active IRQ thread (irq 209)
    [  456.629733] WARNING: CPU: 0 PID: 31 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x330/0x380
    [  456.638995] 44000000.ocp:L3 Standard Error: MASTER MPU TARGET DSS (Read Link): At Address: 0x000403A0 : Data Access in User mode during Functional access
    [  456.652787] Modules linked in: xfrm_user xfrm4_tunnel ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo xhci_plat_hcd xhci_hcd dwc3 udc_core rpmsg_rpc rpmsg_proto)
    [  456.716320] CPU: 0 PID: 31 Comm: irq/23-l3-app-i Tainted: G      D    O      4.19.79-rt28-g5baf382c8f #1
    [  456.716324] Hardware name: Generic DRA74X (Flattened Device Tree)
    [  456.716326] Backtrace:
    [  456.716339] [<c020c6a8>] (dump_backtrace) from [<c020c9e0>] (show_stack+0x18/0x1c)
    [  456.716346]  r7:c0df43dc r6:60010113 r5:00000000 r4:c1253c9c
    [  456.716355] [<c020c9c8>] (show_stack) from [<c09f09e4>] (dump_stack+0x90/0xa4)
    [  456.716370] [<c09f0954>] (dump_stack) from [<c022d0a8>] (__warn+0xdc/0xf8)
    [  456.716376]  r7:c0df43dc r6:00000009 r5:00000000 r4:dfa3be24
    [  456.716385] [<c022cfcc>] (__warn) from [<c022ccc4>] (warn_slowpath_fmt+0x50/0x6c)
    [  456.716391]  r9:dfa22c40 r8:e0882958 r7:c0df42fc r6:c0df4248 r5:c0df43ac r4:c1207488
    [  456.716402] [<c022cc78>] (warn_slowpath_fmt) from [<c0556aec>] (l3_interrupt_handler+0x330/0x380)
    [  456.716406]  r3:dfa09800 r2:c0df43ac
    [  456.716409]  r5:00000004 r4:80080001
    [  456.716418] [<c05567bc>] (l3_interrupt_handler) from [<c0289eb4>] (irq_forced_thread_fn+0x28/0xa0)
    [  456.716424]  r10:00000001 r9:c0289e8c r8:00000000 r7:ffffe000 r6:dfa09c24 r5:dfa09c00
    [  456.716426]  r4:df877800
    [  456.716434] [<c0289e8c>] (irq_forced_thread_fn) from [<c028a288>] (irq_thread+0x140/0x22c)
    [  456.716438]  r7:ffffe000 r6:dfa09c24 r5:dfa09c00 r4:df877800
    [  456.716447] [<c028a148>] (irq_thread) from [<c024c1e4>] (kthread+0x160/0x168)
    [  456.716452]  r10:df87ba94 r9:c028a148 r8:dfa09c00 r7:dfa3a000 r6:00000000 r5:dfa09c40
    [  456.716455]  r4:df9fa0c0
    [  456.716462] [<c024c084>] (kthread) from [<c02010e0>] (ret_from_fork+0x14/0x34)
    [  456.716465] Exception stack(0xdfa3bfb0 to 0xdfa3bff8)
    [  456.716470] bfa0:                                     00000000 00000000 00000000 00000000
    [  456.716475] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [  456.716479] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [  456.716485]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c024c084
    [  456.716487]  r4:dfa09c40
    [  456.716490] ---[ end trace 0000000000000004 ]---
    
     _____                    _____           _         _
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
                  |___|                    |___|
    
    Arago Project http://arago-project.org am57xx-evm ttyS2
    
    Arago 2019.11 am57xx-evm ttyS2
    
    am57xx-evm login:
     _____                    _____           _         _
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
                  |___|                    |___|
    
    Arago Project http://arago-project.org am57xx-evm ttyS2
    
    Arago 2019.11 am57xx-evm ttyS2
    
    am57xx-evm login:
     _____                    _____           _         _
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
                  |___|                    |___|
    
    Arago Project http://arago-project.org am57xx-evm ttyS2
    
    Arago 2019.11 am57xx-evm ttyS2
    
    am57xx-evm login:
     _____                    _____           _         _
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
                  |___|                    |___|
    
    Arago Project http://arago-project.org am57xx-evm ttyS2
    
    Arago 2019.11 am57xx-evm ttyS2
    
    am57xx-evm login:
     _____                    _____           _         _
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
                  |___|                    |___|
    

  • Billa,

    Jose Carlos Billalabeitia said:
    Both interfaces are connected to our Vault that is causing the issue.

    Have you only seen this issue when talking to the "vault"?

    Jose Carlos Billalabeitia said:
             [  455.739841] Backtrace:
             [  455.739884] [<bf349464>] (emac_rx_packet [ti_prueth]) from [<bf349df0>] (emac_rx_thread+0x1bc/0x2cc [ti_prueth])

    The emac_rx_thread functions is inside the file drivers/net/ethernet/ti/prueth.c.  I'd like to understand the line of code corresponding to this message.  You'll need your corresponding object file in order to decode it.  For example, in my linux root directory of the SDK I can run the following command:

    ../../linux-devkit/sysroots/x86_64-arago-linux/usr/bin/arm-linux-gnueabihf-objdump -r -S -l --disassemble drivers/net/ethernet/ti/prueth.o > drivers/net/ethernet/ti/prueth.disassembly

    That generates a disassembled version of that same file.  I can then open it and find emac_rx_thread:

    00001078 <emac_rx_thread>:
    emac_rx_thread():
    1078: e1a0c00d mov ip, sp
    107c: e92ddff0 push {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
    1080: e24cb004 sub fp, ip, #4
    1084: e24dd064 sub sp, sp, #100 ; 0x64
    1088: e5918540 ldr r8, [r1, #1344] ; 0x540
    108c: e3003000 movw r3, #0
    108c: R_ARM_MOVW_ABS_NC __stack_chk_guard
    1090: e3403000 movt r3, #0

    That starts at 0x1078.  Therefore emac_rx_thread+0x1bc corresponds to 0x1234:

    11e4: R_ARM_CALL arm_heavy_mb
    11e8: e3a03000 mov r3, #0
    11ec: e5843000 str r3, [r4]
    11f0: f57ff04e dsb st
    11f4: ebfffffe bl 0 <arm_heavy_mb>
    11f4: R_ARM_CALL arm_heavy_mb
    11f8: e15b74b6 ldrh r7, [fp, #-70] ; 0xffffffba
    11fc: e1c970b0 strh r7, [r9]
    1200: e1570005 cmp r7, r5
    1204: 0a00001f beq 1288 <emac_rx_thread+0x210>
    1208: e51b3050 ldr r3, [fp, #-80] ; 0xffffffb0
    120c: e0834007 add r4, r3, r7
    1210: e5941000 ldr r1, [r4]
    1214: f57ff04f dsb sy
    1218: e1a0200a mov r2, sl
    121c: e1a00008 mov r0, r8
    1220: ebfffbf0 bl 1e8 <parse_packet_info>
    1224: e51b203c ldr r2, [fp, #-60] ; 0xffffffc4
    1228: e3520000 cmp r2, #0
    122c: 0affffe7 beq 11d0 <emac_rx_thread+0x158>
    1230: e51b3054 ldr r3, [fp, #-84] ; 0xffffffac
    1234: e1520003 cmp r2, r3
    1238: 8affffe4 bhi 11d0 <emac_rx_thread+0x158>

    Is that consistent with what you see?  Or perhaps you have some instrumentation in the code that changes the offsets?

    Best regards,
    Brad

  • Yes, I have only seen this issue when talking to the "vault" so far. But it does not mean that it may happen with other connections. It is not easy to reproduce.

    I am using the default prueth.ko that comes with the IDK.

  •  -- what do you suggest for next steps?

  • Hi Brad,

    I added some debugs on Billa's IDK, in prueth.c driver. Early in the EMAC Rx interrupt thread handler, added check for validity of bd_wr_ptr and bd_rd_ptr.

    /**
     * emac_rx_thread - EMAC Rx interrupt thread handler
     */
    static irqreturn_t emac_rx_thread(int irq, void *dev_id)
    ...
    
            bd_rd_ptr = readw(&queue_desc->rd_ptr);
            bd_wr_ptr = readw(&queue_desc->wr_ptr);
    
            // Check validity of bd_rd_ptr and bd_wr_ptr
            if((bd_wr_ptr < rxqueue->buffer_desc_offset) || (bd_wr_ptr > rxqueue->buffer_desc_end))
            {
                if((++wr_ptr_inv_count) < 100)
                {
                    netdev_warn(ndev,
                        "(%d) Invalid bd_wr_ptr. Queue: %d, BD start: %hu, end: %hu, bd_rd_ptr = %hu, bd_wr_ptr = %hu\n",
                        wr_ptr_inv_count, i, rxqueue->buffer_desc_offset, rxqueue->buffer_desc_end, bd_rd_ptr, bd_wr_ptr);
                }
    
                return IRQ_HANDLED;
            }
            if((bd_rd_ptr < rxqueue->buffer_desc_offset) || (bd_rd_ptr > rxqueue->buffer_desc_end))
            {
                if((++rd_ptr_inv_count) < 100)
                {
                    netdev_warn(ndev,
                        "(%d) Invalid bd_rd_ptr. Queue: %d, BD start: %hu, end: %hu, bd_rd_ptr = %hu, bd_wr_ptr = %hu\n",
                        rd_ptr_inv_count, i, rxqueue->buffer_desc_offset, rxqueue->buffer_desc_end, bd_rd_ptr, bd_wr_ptr);
                }
    
                return IRQ_HANDLED;
            }
    
    ...
    

      

    In the highlighted changes, if bd_wr_ptr or bd_rd_ptr found to be corrupted, ISR returns immediately. This prevents ISR and emac_rx_packet() to NOT update rd_ptr incorrectly. 

    Also, added a check to prevent driver to update update_rd_ptr, if it is found to be out of range.

    - Observed that in quick succession, both bd_wr_ptr  and bd_rd_ptr get corrupted for normal priority host queues (Queue2 for port 1and Queue4 for port2). 

    - Also, observed that driver is not the one updating bad rd_ptr .

    - With changes, driver continues to run without crash, but Ethernet Rx or Tx on both ports completely stops.

    With above observations, it appears that PRUETH FW corrupts bd_wr_ptr and bd_rd_ptr.  (FW is not suppose to be writing rd_ptr !!!) So far, I could not get any clue where emac_MII_Rcv.asm could be updating/corrupting rd_ptr.

    As I understand from PRUETH EMAC FW code, 4 host queues are used this way for Rx processing:

    QoS tagged frames :  Q1 (port1)  and Q3  (port2)

    Normal frames         :  Q2 (port1)  and Q4  (port2)

    For Rx processing, PRUETH FW only updates wr_ptr, and Linux driver only updates rd_ptr of the queue.

  • A typical log from prueth.c debugs:

    [ 101.387769] prueth pruss2_eth eth2: (1) Invalid bd_wr_ptr. Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 1048, bd_wr_ptr = 0
    [ 101.387889] prueth pruss2_eth eth3: (1) Invalid bd_rd_ptr. Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 2, bd_wr_ptr = 2576
    [ 101.410847] prueth pruss2_eth eth2: (2) Invalid bd_wr_ptr. Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 1048, bd_wr_ptr = 0
    [ 101.581899] prueth pruss2_eth eth2: (3) Invalid bd_wr_ptr. Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 1048, bd_wr_ptr = 0
    [ 101.581906] prueth pruss2_eth eth3: (2) Invalid bd_rd_ptr. Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 2, bd_wr_ptr = 2576
    [ 101.636605] prueth pruss2_eth eth2: (4) Invalid bd_wr_ptr. Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 1048, bd_wr_ptr = 0
    [ 101.636612] prueth pruss2_eth eth3: (5) Invalid bd_wr_ptr. Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 2, bd_wr_ptr = 0
    [ 101.776278] prueth pruss2_eth eth2: (6) Invalid bd_wr_ptr. Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 1048, bd_wr_ptr = 0
    [ 101.776285] prueth pruss2_eth eth3: (7) Invalid bd_wr_ptr. Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 2, bd_wr_ptr = 0
    [ 101.785911] prueth pruss2_eth eth3: (8) Invalid bd_wr_ptr. Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 2, bd_wr_ptr = 0
    [ 101.810347] prueth pruss2_eth eth2: (9) Invalid bd_wr_ptr. Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 1048, bd_wr_ptr = 0
    [ 101.855929] prueth pruss2_eth eth2: (10) Invalid bd_wr_ptr. Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 1048, bd_wr_ptr = 0
    [ 101.855936] prueth pruss2_eth eth3: (11) Invalid bd_wr_ptr. Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 2, bd_wr_ptr = 0

  • Please let us know if we can collect any specific diagnostic information from this IDK.

    Other observations:

    Resetting the PRU using the SOFT_RST_N bit of the PRU_CONTROL register, does not resolve issue. 

    Tx frame count in ifconfig and ethtool continues to increase, but other system does not receive these frame (tcpdump on Ubuntu was not receiving any ICMP echo frames from the IDK, when pinging from IDK during the issue). Since, (ethtool) stats are updated in PRU FW. Does that means, PRU FW receives Tx frames from driver but not able to transmit to MII ?

  • Hi Paritosh, Billa

    Let me try and summarize the issue

    1. Test Setup

    eth0(GMAC) <--- NFS

    eth1 and eth 2 (pruss2_eth) <--vault

    eth1 and eth2 are receiving multicast traffic (GOOSE)

    2. Sometimes you observe kernel crash which you have isolated to invalid buffer read pointer (you isolated it by adding debug code in driver)

    3. You see issue only with Dual MAC firmware (not PRP)

    4. The issue does not go away with link reset or PRU reset.

    5. You have managed to reproduce this in the IDK but it's not consistent.

    6. Tx interface counters keep increasing even when the issue happens, indicating that PRU is still receiving messages.

    This is a strange issue indeed. As you rightly pointed out, PRU FW is not supposed to updated BD Read pointer.

    Few things we can try are

    1. Reset the BD pointers and other read/write pointers when this issue happens (sort of a PRU Init) and see if firmware can recover from the issue

    2. Figure out a way to make this more frequent

    3. Can you confirm that the issue does not happen on other firmwares, is it only limited to EMAC ? What about HSR ?

    This does indeed look like an issue with buffer descriptor corruption.

    Regards

    Vineet

  • Hi Vineet,

    Our answers are inline:

    1. [Paritosh] : Test Setup on IDK

    eth2 and eth 3 (pruss2_eth) <--vault

    [Paritosh]: Latest observation on IDK is that GOOSE is not mandatory to reproduce this issue. IDK is configured with default systemd network scripts in /etc/systemd/network/ and eth2/eth3 acquire IP via DHCP.
    eth2/eth3 are NOT configured to receive all multicast frames (ALLMULTI) and NO explicit multicast subscription is done.


    2. Sometimes you observe kernel crash which you have isolated to invalid buffer read pointer (you isolated it by adding debug code in driver)

    [Paritosh]: If ISR code returns, when it sees bd_rd_ptr or bd_wr_ptr as invalid, prueth driver does not crash. It continues to run. But, no Tx/Rx with devices outside IDK.


    3. You see issue only with Dual MAC firmware (not PRP)

    [Paritosh]: Yes, with am57xx-pru0-prueth-fw.elf and am57xx-pru1-prueth-fw.elf


    4. The issue does not go away with link reset or PRU reset.

    [Paritosh]: Correct


    5. You have managed to reproduce this in the IDK but it's not consistent.

    [Paritosh]: With DHCP config mentioned above, it can be reproduced consistently on IDK now. We could not isolate the exact traffic causing it though.


    6. Tx interface counters keep increasing even when the issue happens, indicating that PRU is still receiving messages.

    [Paritosh]: Yes. These counts keep increasing:

    root@am57xx-evm:/lib/firmware/ti-pruss# ethtool -S eth3 | grep tx
    txBcast: 233
    txMcast: 54
    txUcast: 45
    txOctets: 28183
    tx64byte: 234
    tx65_127byte: 75
    tx128_255byte: 7
    tx256_511byte: 16


    This is a strange issue indeed. As you rightly pointed out, PRU FW is not supposed to updated BD Read pointer.

    Few things we can try are

    1. Reset the BD pointers and other read/write pointers when this issue happens (sort of a PRU Init) and see if firmware can recover from the issue

    [Paritosh]: Sure. We will try this. It will be helpful if you can share a SDK 6.02 patch to do this best possible way ?


    2. Figure out a way to make this more frequent

    3. Can you confirm that the issue does not happen on other firmwares, is it only limited to EMAC ? What about HSR ?

    [Paritosh]: So far, we have not seen this with PRP/HSR FW.

  • Hi Paritosh,

    1. Will AM5728 IDK work for reproducing the issue ?

    2. >> Sure. We will try this. It will be helpful if you can share a SDK 6.02 patch to do this best possible way ?

    Duplicate prueth_hostinit() function (drivers/net/ethernet/ti/prueth.c) and remove prueth_mii_init() from it. Call that function whenever the issue happens.

    3. If I configure in DHCP mode and send iperf traffic, will I be able to reproduce the issue ?

    Regards

    Vineet

  • Hi Vineet,

    1. We are able to reproduce it on AM5728 IDK. 

    3. Today, I was able to reproduce with static IPs as well. So, it is not DHCP specific.

    So far, we are able to reproduce it only in vault network with both ports are connected to vault network.

    Today, I tried with only one port connected to vault network, but could not reproduce it.

    I also tried to reproduce in a small local network (disconnected from vault), with iperf and with pcap captured inside the vault, but unable to reproduce it.

    So main recipe to reproduce the issue is vault network and both ports are connected to it. 

    If it helps and possible, we can walk you through setup and issue over a screen-sharing meeting.

  • Hi Vineet,

    Probably, I have located root cause of this crash issue.

    This commit in PRU EMAC FW introduced this issue:

    https://git.ti.com/cgit/keystone-rtos/icss-emac/commit/firmware/icss_dualemac/src?id=4ffd24ffac468ccdd71dad29405d9d0eedf148b9 

    From SDK 6.02 version of emac_MII_Rcv.asm, if I delete these 4 lines of code (i.e. undo above commit), it fixes this issue and BD wr_ptr/rd_ptr corruption is not observed.
    https://git.ti.com/cgit/processor-sdk/pdk/log/packages/ti/drv/icss_emac/firmware/icss_dualemac/src/emac_MII_Rcv.asm?h=release/PROCESSOR-SDK_06.02.00 



    diff --git a/packages/ti/drv/icss_emac/firmware/icss_dualemac/src/emac_MII_Rcv.asm b/packages/ti/drv/icss_emac/firmware/icss_dualemac/src/emac_MII_Rcv.asm index a6a563cd..8ddb70a2 100755 --- a/packages/ti/drv/icss_emac/firmware/icss_dualemac/src/emac_MII_Rcv.asm +++ b/packages/ti/drv/icss_emac/firmware/icss_dualemac/src/emac_MII_Rcv.asm @@ -1080,10 +1080,6 @@ LESS_THAN_32_BYTES_RCVD: .if $defined(PTP) JAL RCV_TEMP_REG_3.w2, FN_TIMESTAMP_GPTP_PACKET - .if $defined("ICSS_DUAL_EMAC_BUILD") - ; For DualEMAC, check here to skip host rcv to properly drop sync frames not from master - QBBC NB_PROCESS_32BYTES_CHECK_FLAGS_QUEUE_NOT_FULL, MII_RCV.rx_flags, host_rcv_flag_shift ; MII_RCV.rx_flags.host_rcv_flag - .endif LDI RCV_TEMP_REG_3.w2, PTP_IPV4_UDP_E2E_ENABLE LBCO &RCV_TEMP_REG_3.b2, ICSS_SHARED_CONST, RCV_TEMP_REG_3.w2, 1

    This commit is in old GIT repo, which was carried forward to new repo (where SDK 6.02 was released).

    Can you please review this commit and provide correct way to "drop sync frames not from master" ?

    emac_MII_Rcv.asm

  • Hi Paritosh,

    Thanks for the effort in catching the issue. You are right that this code can cause BD Rd/Wr pointer issues.

    The four lines were added by mistake by someone. They are not required, we already have code to drop PTP frames which are not from master.

    See the function FN_TIMESTAMP_GPTP_PACKET and the code snippet below

        ;Below is a combined load to save cycles.
        ;R20.b0 tells if DUT is master. R20.b1 tells which port is connected to master
        ;and R20.w2 and R21 tell the MAC ID of the master selected by BMCA
        LBCO    &R20.w0, ICSS_SHARED_CONST, DUT_IS_MASTER_OFFSET, 8
        QBNE    DUT_IS_SLAVE_SYNC, R20.b0, 1    ; exit if DUT is configured as master, also discard frame
        CLR     MII_RCV.rx_flags, MII_RCV.rx_flags, host_rcv_flag_shift    ; Don't forward to host
        SET     R22, R22, PTP_RELEASE_HOST_QUEUE_BIT
        QBA     FN_TIMESTAMP_GPTP_PACKET_EXIT

    The highlighted code already takes care of dropping the packets properly.

    You can keep your changes (removing the commit)

    Since this change does not exist on HSR/PRP you don't see it.

    We have noted this and will fix this internally.

    Regards

    Vineet

  • FYI, this update has been merged to the git.ti.com branch:

    https://git.ti.com/cgit/processor-sdk/pdk/commit/packages/ti/drv/icss_emac?h=legacy_master&id=e7e9dcaae0c6441bd1d7e477ba9c536ea5ceee23

    Please confirm the issue is resolved/closed.

  • Hi Brad, Vineet,

    BD Pointer corruption is again seen on Bridge device on Bilbao vault network. We double checked that PRU EMAC code is correctly patched and updated FW is installed on the DUT.

    Can you please review PRU EMAC FW (Rx and Tx side) for similar code that can cause BD pointer corruption.

    Thanks

    Paritosh

  • Hi Paritosh,

    1. Is it the same setup as before ?

    2. Is the issue seen only on Dual MAC or HSR/PRP too ?

    Regards

    Vineet

  • Hi Vineet,

    1. Yes, it is occurring with the same setup. 

    2. Dual EMAC only, as before.

    Thanks,
    Paritosh

  • Paritosh,

    Has the issue only been observed in conjunction with "the vault"?  What is that device?  Is there a specific type of traffic pattern used in conjunction with this device?  In short, I'm looking for a way to reproduce the issue so we can more effectively debug it.

    Thanks!
    Brad

  • Hi Brad,

    We are trying to isolate scenario in which it is reproduced again.

    - It is reproduced on two Bridge devices in EMAC PRU ETH mode, both ports connected to Bilbao vault network.

    - PTP is not enabled, while goose rx is enabled.

    We have asked the tester to disable goose rx on the affected device and observe if issue reoccurs. 

    Thanks,
    Paritosh

  • Hi Vineet,

    With the latest observations, it looks almost certain that there is another issue in PRUETH FW that causes BD pointer corruptions.

    It takes around 1-2 days to recreate issue on these Bridge units. It takes time to apply a change and wait for issue to reoccur.

    Every time issue occurs, we are able to see logs in prueth.c driver indicating rd/wr ptr corruption:

     

    [74407.323621] prueth pruss2_eth eth2: Invalid bd_wr_ptr (Occurance Count 57691). Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 542, bd_wr_ptr = 0

    [74407.341891] prueth pruss2_eth eth1: Invalid bd_wr_ptr (Occurance Count 57692). Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 542, bd_wr_ptr = 0

    [74407.397449] prueth pruss2_eth eth2: Invalid bd_wr_ptr (Occurance Count 57693). Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 542, bd_wr_ptr = 0

    [74407.411847] prueth pruss2_eth eth2: Invalid bd_wr_ptr (Occurance Count 57694). Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 542, bd_wr_ptr = 0

    [74407.425045] prueth pruss2_eth eth2: Invalid bd_wr_ptr (Occurance Count 57695). Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 542, bd_wr_ptr = 0

    [74407.438243] prueth pruss2_eth eth1: Invalid bd_wr_ptr (Occurance Count 57696). Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 542, bd_wr_ptr = 0

    [74407.480087] prueth pruss2_eth eth2: Invalid bd_wr_ptr (Occurance Count 57697). Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 542, bd_wr_ptr = 0

     

    On another unit:

    [114619.412674] prueth pruss2_eth eth1: Invalid bd_wr_ptr (Occurance Count 4050). Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 0, bd_wr_ptr = 7424

    [114629.370518] prueth pruss2_eth eth2: Invalid bd_wr_ptr (Occurance Count 4051). Queue: 2, BD start: 2576, end: 3348, bd_rd_ptr = 16, bd_wr_ptr = 7424

    [114639.501174] prueth pruss2_eth eth1: Invalid bd_rd_ptr (Occurance Count 4052). Queue: 0, BD start: 1024, end: 1796, bd_rd_ptr = 0, bd_wr_ptr = 1536

    [114650.659613] prueth pruss2_eth eth2: Invalid bd_wr_ptr (Occurance Count 4053). Queue: 3, BD start: 3352, end: 4124, bd_rd_ptr = 52, bd_wr_ptr = 12544

    [114661.355556] prueth pruss2_eth eth1: Invalid bd_wr_ptr (Occurance Count 4054). Queue: 1, BD start: 1800, end: 2572, bd_rd_ptr = 136, bd_wr_ptr = 7424

     

    In the latest test, we disabled GE applications completely on these two Bridge units. That means NO GOOSE Tx/Rx, NO PTP, and NO multicast subscriptions.

    In that config, typical ethtool stats are as shown below (soon after reboot, before issue occurs):

     # ethtool -S eth1    (Similar stats on eth2.)

    NIC statistics:

         txBcast: 3

         txMcast: 12

         txUcast: 0

         txOctets: 1176

         rxBcast: 5178

         rxMcast: 0

         rxUcast: 0

         rxOctets: 401353

         tx64byte: 3

         tx65_127byte: 12

         tx128_255byte: 0

         tx256_511byte: 0

         tx512_1023byte: 0

         tx1024byte: 0

         rx64byte: 0

         rx65_127byte: 5104

         rx128_255byte: 29

         rx256_511byte: 45

         rx512_1023byte: 0

         rx1024byte: 0

         lateColl: 0

         singleColl: 0

         multiColl: 0

         excessColl: 0

         rxMisAlignmentFrames: 0

         stormPrevCounter: 0

         macRxError: 1

         SFDError: 0

         defTx: 0

         macTxError: 0

         rxOverSizedFrames: 0

         rxUnderSizedFrames: 0

         rxCRCFrames: 0

         droppedPackets: 23

         txHWQOverFlow: 0

         txHWQUnderFlow: 0

         emacMulticastDropped: 85847

         emacVlanDropped: 0

    rxMcast is very low and not increasing. Clearly, NO rxMcast activity.

     

    When issue is observed after 1-2 days, couple of interesting observations:

    1. rxMcast is very high and continuously increasing even though no Ethernet communication (no ping to/from). It appears somehow muticast (subscription or allmulti) got enabled as of a result of issue occurrence.
    2. excessColl is increased to count 2 or 3 on both eth1 and eth2.

     

    Whne issue is observed, ethtool stats are as shown below:

    # ethtool -S eth1     (Similar stats on eth2.)

    NIC statistics:

         txBcast: 3

         txMcast: 33

         txUcast: 75

         txOctets: 7530

         rxBcast: 744585

         rxMcast: 5003205

         rxUcast: 223

         rxOctets: 700858018

         tx64byte: 78

         tx65_127byte: 33

         tx128_255byte: 0

         tx256_511byte: 0

         tx512_1023byte: 0

         tx1024byte: 0

         rx64byte: 3

         rx65_127byte: 1715100

         rx128_255byte: 4023758

         rx256_511byte: 7257

         rx512_1023byte: 1895

         rx1024byte: 0

         lateColl: 0

         singleColl: 0

         multiColl: 0

         excessColl: 2

         rxMisAlignmentFrames: 0

         stormPrevCounter: 0

         macRxError: 1

         SFDError: 0

         defTx: 0

         macTxError: 0

         rxOverSizedFrames: 3

         rxUnderSizedFrames: 0

         rxCRCFrames: 0

         droppedPackets: 2290

         txHWQOverFlow: 0

         txHWQUnderFlow: 0

         emacMulticastDropped: 12171667

         emacVlanDropped: 0

     

    ethtool stats on the bridge unit, where issue is not observed (after 1 day on running):

    # ethtool -S eth1         (Similar stats on eth2.)

    NIC statistics:

         txBcast: 3

         txMcast: 33

         txUcast: 8

         txOctets: 3242

         rxBcast: 745940

         rxMcast: 144

         rxUcast: 13

         rxOctets: 57246845

         tx64byte: 11

         tx65_127byte: 33

         tx128_255byte: 0

         tx256_511byte: 0

         tx512_1023byte: 0

         tx1024byte: 0

         rx64byte: 1

         rx65_127byte: 738312

         rx128_255byte: 4263

         rx256_511byte: 3521

         rx512_1023byte: 0

         rx1024byte: 0

         lateColl: 0

         singleColl: 0

         multiColl: 0

         excessColl: 0

         rxMisAlignmentFrames: 0

         stormPrevCounter: 0

         macRxError: 1

         SFDError: 0

         defTx: 0

         macTxError: 0

         rxOverSizedFrames: 3

         rxUnderSizedFrames: 0

         rxCRCFrames: 0

         droppedPackets: 2456

         txHWQOverFlow: 0

         txHWQUnderFlow: 0

         emacMulticastDropped: 12101429

         emacVlanDropped: 0

    rxMcast is very low and not increasing.

     Please see if these observations provide any pointers in the PRU ETH code to analyze BD ptr corruption.

     

     Can you also provide some information related to last code change:

    - We removed this commit, which helped to reduce occurrence of BD ptr corruption. Can you please explain how those 4 lines of code were causing the BD ptr corruption. This will help us to look for similar code patterns.

    -  In function FN_TIMESTAMP_GPTP_PACKET, the host_rcv_flag_shift bit is cleared in MII_RCV.rx_flags, in order to discard a frame and not to forward it to host. Can you please explain code location and control flow, on how exactly this bit causes a frame to be forwarded to host, while clearing it will cause this frame to be discarded.

     

    Thanks,

    Paritosh

  • Hi Paritosh,

    Thanks for the logs. Our PRUETH firmware experts are reviewing them, we will provide the next steps for debugging this issue.

    Thanks & Regards,

    Sunita.

  • Hi Sunita,

    Could this issue be related to multicast and vlan filtering features? Here are latest updates from what we have tried:

    - When multicast filtering and vlan filtering macros are commented in PRU EMAC FW code, issue is not observed for the last 3 days on these Bridge units. We still need to monitor it for next 2-3 days before confirming this.

        506 FB_MULTICAST:
        507     SET MII_RCV.rx_flags , MII_RCV.rx_flags , host_rcv_flag_shift
        508     SET R22 , R22 , RX_MC_FRAME      ; set multi-cast bit to indicate the multicast frame
        509
        510     .if $defined("ICSS_DUAL_EMAC_BUILD") | $defined("ICSS_STP_SWITCH")
        511         ; check if multicast filtering is enabled or not i.e. check MULTICAST_CONTROL_BIT in PRU1 data memory
        512         LDI     RCV_TEMP_REG_3.w0, ICSS_EMAC_FW_MULTICAST_FILTER_CTRL_OFFSET
        513         LBCO    &RCV_TEMP_REG_3.b0, PRU_DMEM_ADDR, RCV_TEMP_REG_3.w0, 1
        514         QBEQ    FB_MULTICAST_CONTINUE, RCV_TEMP_REG_3.b0, ICSS_EMAC_FW_MULTICAST_FILTER_CTRL_DISABLED    ; one byte field : 0 -> multicast filtering disabled | 1 -> multicast filtering enabled
        515
        516         ; search in multicast table to determine if host receive is to be enabled or not for the incoming multicast frame
        517         ;M_MULTICAST_TABLE_SEARCH_OP
        518         ; QBA     FB_MULTICAST_CONTINUE
        519     .endif
        520
        521 FB_MULTICAST_CONTINUE:
        522 FB_UNICAST_CHECK_CT:
        523 FB_BROADCAST_CHECK_CT:
        524     .if $defined("ICSS_DUAL_EMAC_BUILD") | $defined("ICSS_STP_SWITCH")
        525         ; check if VLAN filtering is enabled or not i.e. check VLAN_FLTR_CTRL_SHIFT in VLAN_FLTR_CTRL_BYTE stored in data memory
        526         LDI     RCV_TEMP_REG_3.w0, ICSS_EMAC_FW_VLAN_FILTER_CTRL_BITMAP_OFFSET
        527         LBCO    &RCV_TEMP_REG_3.b0, PRU_DMEM_ADDR, RCV_TEMP_REG_3.w0, 1
        528     ;-----------------------------------------------------------
        529     ;   VLAN_FLTR_CTRL_BYTE  |          |          |           | R13 | RCV_TEMP_REG_3
        530     ;-----------------------------------------------------------
        531         QBBC    FB_SKIP_VLAN_FLTR, RCV_TEMP_REG_3.b0, ICSS_EMAC_FW_VLAN_FILTER_CTRL_ENABLE_BIT   ; one bit field | 0 : VLAN filter disabled | 1 : VLAN filter enabled
        532
        533         ; search in VLAN table to determine if host receive is to be enabled or not for the incoming frame
        534         ;M_VLAN_FLTR_SRCH_OP
        535
        536 FB_SKIP_VLAN_FLTR:
        537     .endif
    

    Only change is that call to macros M_MULTICAST_TABLE_SEARCH_OP and M_VLAN_FLTR_SRCH_OP are commented on lines 517 and 534.

    - After this change, we can also observe that all muticast traffic is accepted, as seen from ethtool stats:

    $ ethtool -S eth2

    ...

    rxMcast: 16976755

    emacMulticastDropped: 0
    emacVlanDropped: 0

    ...

    - In a different test, issue was also observed with PRU EMAC FW code in SDK 5.02 as well  (DEV.PROCESSOR-SDK.05.02.00.07 tag compiled in SDK 6.02)

    While you prepare next steps for debugging, can you provide information that we asked for in last post:

    - We removed this commit, which helped to reduce occurrence of BD ptr corruption. Can you please explain how those 4 lines of code were causing the BD ptr corruption. This will help us to look for similar code patterns.

    -  In function FN_TIMESTAMP_GPTP_PACKET, the host_rcv_flag_shift bit is cleared in MII_RCV.rx_flags, in order to discard a frame and not to forward it to host. Can you please explain code location and control flow, on how exactly this bit causes a frame to be forwarded to host, while clearing it will cause this frame to be discarded.

    Thanks,

    Paritosh

  • Hi Paritosh,

    We found this issue that rd/wr_ptr update is done irrespective host_rcv_flag status in Dual MAC mode of the firmware. This is identified during code review and we have only done limited tests. Please try following patch

    LDI RCV_TEMP_REG_2 , RX_PKT_SIZE_OFFSET
    SBCO &RCV_CONTEXT.byte_cntr, PRU_DMEM_ADDR, RCV_TEMP_REG_2, 2 ; load the byte count in working register

    .endif ;TWO_PORT_CFG
    LB_UPDATE_FOR_HOST_RECEIVE:
    +QBBC LB_RELEASE_HOST_QUEUE, MII_RCV.rx_flags, host_rcv_flag_shift ; skip RX LB pointer updates
    ; Update the Receive Buffer Descriptor
    ; Read the wr_ptr of first buffer descriptor
    .if $defined("TWO_PORT_CFG")
    LBCO &RCV_TEMP_REG_1, PRU1_DMEM_CONST, RCV_CONTEXT.rcv_queue_pointer, 4
    .else
    LBCO &RCV_TEMP_REG_1, ICSS_SHARED_CONST, RCV_CONTEXT.rcv_queue_pointer, 4
    .endif ;TWO_PORT_CFG
    ; clear length field (18..28) and update length with current received frame
    LDI RCV_TEMP_REG_2.w0, 0

    Thanks,

    Roopak

  • Paritosh -- did you get a chance to try this?  How does it look so far?

  • Hi Brad,

    This patch was applied to affected Bridge units and it has been running fine since last 3 days. Patch is included into Bridge PEOS build for validation, that is starting next week.

    Thanks,
    Paritosh

  • Hi Paritosh,

    While reviewing the code we have identified that host_rcv_flag checks are missed in the RX_NB and RX_LB in dual emac build. These checks are required in order to drop a packet cleanly. We have fixed this by adding additional checks to avoid read and write pointer corruption when host_rcv_flag is cleared. Please find the attached patch file for Host Rcv flag checks.

    patch_emac_MII_Rcv.zip

  • Hi Paritosh

    This patch includes previous patch changes so apply the latest patch on base code.

    apply patch at this path : pdk\packages\ti\drv\icss_emac\firmware\icss_dualemac\src

    you can share results once you test the patch.

    Thank you

  • This patch includes previous patch changes so apply the latest patch on base code.

    Roopak -- I don't see either one of the previous patches included.  Can you please clarify what should be used as the basis for this patch?  Should it be the head of this legacy_master branch:

    https://git.ti.com/cgit/processor-sdk/pdk/log/?h=legacy_master

    If that is the case, then Patch 1 would already be part of the baseline code, so perhaps that explains that one?

    For Patch 2, I'm not clear how that one is being incorporated.  I see code that is similar in the updated patch, but not exactly that one.  It's not clear if Patch 2 should also be applied separately, or perhaps the implementation changed and we should be using this updated implementation instead of the original?  Please clarify.

    Thanks!

    Brad

  • Hi Brad,

    I am attaching the fresh patch incorporating all the changes which can be applied on top of legacy master branch( Commit ID : 0c94dbe668e3f185abc61d0f66aff90baf800368). Please find the attached patch file below.

    Apologies for all the confusion.

    Thank you

    Roopak

    legacy_patch_dualemac.zip

  • Roopak -- thanks! It's very clear now.

    Paritosh -- can you please try that out and confirm it's working properly?