TDA4VH-Q1: REPLY: After NFS is mounted, the system occasionally encounters the “serror 0xbf000002” exception.

HI, ti Experts

This thread is created to reply to a previous thread: (+) TDA4VH-Q1: After NFS is mounted, the system occasionally encounters the “serror 0xbf000002” exception. - Processors forum - Processors - TI E2E support forums

Best regards

Alex

HI, ti Experts

We have also encountered MCU2_0 crash issue before, and the Oops information we captured is as follows:

Exception data abort on task context!!!
Exception pc : 0x99fc07e4
Exception sp : 0x99917f60
Current task : Lwip2Enet_RxPacketTask
r0 : 0x00000001  r1 : 0x00000000
r2 : 0x00000001  r3 : 0x00000000
r4 : 0x00000000  r5 : 0x33db7000
r6 : 0x00000001  r7 : 0x00000000
r8 : 0x08080808  r9 : 0x99911d70
r10 : 0x99910d34 r11 : 0x9a080899
r12 : 0x00000008 r14 : 0x99fc07d0
dfsr : 0x00001808 dfar : 0x33db7000
ifsr : 0x00000000 ifar : 0x00000000

We captured the logs of M4 using the same method as above, and the log of M4 is as follows:

FWL Bit  0x1E
Exception addr  0x45B0B000
FWL Exception  0x110E000
 0x10000
 0x33DB7004
 0x0
 0xA8122D4
 0x4

From the information obtained so far, the cause of this issue is the same as the cause of the serror problem.

Please investigate the cause of the problem. Thank you very much.

Best regards

Alex

Hi Alex,

But which SDK release are you using? we dont atleast see these crashes on the EVM, so wondering what's changed? Have you changed anything in SDK? or in the resource manager or in ethernet driver? 

Regards,

Brijesh

Hi, Brijesh

Our use Software  Version  is  SDK 8.6, we use boot model is SPL。

We modified the rm_cfg.c file, and the modified content is as follows: “+” represents the original content of the SDK, and “-” represents our modification.

@@ -187,42 +187,42 @@
                /* Main GPIO Interrupt router */
                {
                        .start_resource = 0,
-                       .num_resource = 24,
+                       .num_resource = 4,
                        .type = RESASG_UTYPE (J784S4_DEV_GPIOMUX_INTRTR0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
-                       .host_id = HOST_ID_ALL,
+                       .host_id = HOST_ID_MAIN_0_R5_0,
                },
                {
-                       .start_resource = 24,
-                       .num_resource = 0,
+                       .start_resource = 4,
+                       .num_resource = 4,
                        .type = RESASG_UTYPE (J784S4_DEV_GPIOMUX_INTRTR0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
                        .host_id = HOST_ID_MAIN_0_R5_2,
                },
                {
-                       .start_resource = 24,
-                       .num_resource = 0,
+                       .start_resource = 8,
+                       .num_resource = 4,
                        .type = RESASG_UTYPE (J784S4_DEV_GPIOMUX_INTRTR0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
                        .host_id = HOST_ID_MAIN_1_R5_0,
                },
                {
-                       .start_resource = 24,
-                       .num_resource = 0,
+                       .start_resource = 12,
+                       .num_resource = 4,
                        .type = RESASG_UTYPE (J784S4_DEV_GPIOMUX_INTRTR0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
                        .host_id = HOST_ID_MAIN_1_R5_2,
                },
                {
-                       .start_resource = 24,
-                       .num_resource = 0,
+                       .start_resource = 16,
+                       .num_resource = 4,
                        .type = RESASG_UTYPE (J784S4_DEV_GPIOMUX_INTRTR0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
                        .host_id = HOST_ID_MAIN_2_R5_0,
                },
                {
-                       .start_resource = 24,
-                       .num_resource = 0,
+                       .start_resource = 20,
+                       .num_resource = 4,
                        .type = RESASG_UTYPE (J784S4_DEV_GPIOMUX_INTRTR0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
                        .host_id = HOST_ID_MAIN_2_R5_2,
@@ -339,21 +339,21 @@
                },
                {
                        .start_resource = 22,
-                       .num_resource = 0,
+                       .num_resource = 6,
                        .type = RESASG_UTYPE (J784S4_DEV_WKUP_GPIOMUX_INTRTR0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
                        .host_id = HOST_ID_A72_3,
                },
                {
-                       .start_resource = 22,
-                       .num_resource = 10,
+                       .start_resource = 28,
+                       .num_resource = 2,
                        .type = RESASG_UTYPE (J784S4_DEV_WKUP_GPIOMUX_INTRTR0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
-                       .host_id = HOST_ID_ALL,
+                       .host_id = HOST_ID_MAIN_0_R5_0,
                },
                {
-                       .start_resource = 32,
-                       .num_resource = 0,
+                       .start_resource = 30,
+                       .num_resource = 2,
                        .type = RESASG_UTYPE (J784S4_DEV_WKUP_GPIOMUX_INTRTR0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
                        .host_id = HOST_ID_MAIN_0_R5_2,
@@ -437,17 +437,17 @@
                /* Main NAVSS Interrupt router */
                {
                        .start_resource = 16,
-                       .num_resource = 50,
+                       .num_resource = 80,
                        .type = RESASG_UTYPE (J784S4_DEV_NAVSS0_INTR_0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
                        .host_id = HOST_ID_A72_2,
                },
                {
-                       .start_resource = 66,
-                       .num_resource = 44,
+                       .start_resource = 96,
+                       .num_resource = 14,
                        .type = RESASG_UTYPE (J784S4_DEV_NAVSS0_INTR_0,
                                        RESASG_SUBTYPE_IR_OUTPUT),
-                       .host_id = HOST_ID_ALL,
+                       .host_id = HOST_ID_A72_3,
                },
                {
                        .start_resource = 110,

All modifications to Linux network drivers are as follows: the content on the left represents the original content of the SDK, and the content on the right represents our modification.

linux/drivers/phy/ti/phy-gmii-sel.c:

linux/drivers/soc/ti/k3-ringacc.c:

linux/drivers/dma/ti/k3-udma.c:

linux/drivers/net/phy/phy_device.c

We haven’t made any modifications to the RTOS network driver.

Best regards

Alex

Hi,

Coming up to speed on this issue, some thoughts based on the provided FWL exception above.

The default firewall settings, privIds etc, for TDA4VH are available at below link:

• J784S4 Firewall Descriptions — TISCI User Guide

The log included above has the following data provided which can be mapped to registers in the TRM XLS.

• Hdr[0] - 0x110E000
• Hdr[1] - 0x10000
• Data[0] - 0x33DB7004
• Data[1] - 0x0
• Data[2]  - 0xA8122D4
• Data[3] - 0x4

This should give additional information as to where/why the error is occurring.  A quick look at Data[2], shows a PRIV Id of 0xD4 (212) which seems to indicate write access from MCU2_0 did not have access.

Given in parallel thread it was mentioned that NB bridge firewalls were modified, and TI has not been able to reproduce the issue, the modified firewall settings should be reviewed.

• "We configured the North Bridge (NB) firewall to allow only MCU1_0 to access the 1GB memory starting from address 0x80000000. After using the firewall configuration, the Universal DMA (UDMA) inside NAVSS is unable to access the memory. "

Regards,

kb

HI, KB

Thank you for your explanation of the meaning of M4 logs.

What you mentioned about ‘Given in parallel thread it was mentioned that NB bridge firewalls were modified, and TI has not been able to reproduce the issue, the modified firewall settings should be reviewed.’ is only our testing demo and has not been applied in actual projects. All relevant modifications we made have been documented in the above statement.

In addition, we found in our tests that on mcu2_0, it is necessary to execute the call of “Sciclient_rmIrqSet” before accessing the 0x33db7000 register. The actual work contents of “Sciclient_rmIrqSet” seem to be closed source, and we currently only know that it configures firewall permissions, but we are not clear about the specific process. You said "TI has not been able to reproduce the issue" but it  is possible that some potential issues in configuring firewall properties may have been exposed in our environment. Can TI help investigate whether there is an issue with configuring firewall permissions?

Best regards

Alex

Hi Alex,

Please be aware of below link, which provides a description of decoding the M4 log, vs explanation above.

• Firewall FAQ — TISCI User Guide

Referencing this link and using the M4 log from above, line 4, would seem to indicate that error 0x1 is occurring, please double check.

If this is the case, next steps would be to read the appropriate firewall, and if indeed it is active and not enabled, try deactivating it running test again.  If test passes, then decision can be made if the firewall should deactivated, or have programming added to it, so it can be active.

The Sciclient_rmIrqSet() implementation is part of the PDK, at path packages/ti/drv/sciclient/src/sciclient/sciclient_rm.c, in the SDK RTOS downloads.

Regarding the GPIO changes shared above, wondering if the moving of GPIO resources for J784S4_DEV_WKUP_GPIOMUX_INTRTR0  from MCU2_0 to A72 is having an impact.  Is it possible to test without those changes and see if issue is reproduced?

Regards,

kb

HI KB:

1. The function Sciclient_rmIrqSet requests MCU1_0 to handle it by proxy, and then the M4 processor continues to handle security. The source code of the configuration of M4’s firewall is closed to us。Could you provide us with a version of M4 firmware that is not configured with a firewall when CALL Sciclient_rmIrqSet?

Sciclient_service(TISCI_MSG_RM_IRQ_SET )[A72/MCU2_0 indirect mode ]   ---->  Sciclient_service[MCU1_0]  ---> Sciclient_serviceSecureProxy(pReqPrm, pRespPrm) --> M4

2. How can I read the configuration information of the 4320U firewall? We have attempted to read it but keep receiving an error indicating that we do not have permission to access the information.

3. GPIO changes. We are trying to roll back the code to reproduce the issue.

Best regards

Alex

(1) The M4 firmware source code is not available for distribution on TDA4VH

(2) The 4320 Channelized Firewall has the below default settings, which channel are is being read?  I would suggest that this be handled on a different e2e thread if further discussion required.  

J784S4 Firewall Descriptions — TISCI User Guide

(3)  Thank you, it is understood that this effort will take some time, hopefully it will show that it is related to issue being seen.

Regards,

kb

hi  KB:

(2) The 4320 Channelized Firewall has the below default settings, which channel are is being read?

We tried to read firewall infomation from MCU1_0 / A72 through the SCI interface

Pseudocode：

firewallreq.fwl_id = 4320;
firewallreq.region = 0， 1， 256; // Inputting both channel number and region number as parameters has been attempted.
firewallreq.n_permission_regs = cfg->n_permission_regs;

ret = Sciclient_firewallGetRegion(&firewallreq,&firewallresp,SCICLIENT_SERVICE_WAIT_FOREVER);

May I ask if there is an issue with the parameters being passed? If so, could you please provide an example of a successful read with the correct parameters? Thank you.

We have another question that we would like to inquire about：

 why the network driver tx&rx needs to perform interrupt disable and enable operations for the DMA, while other modules that use DMA do not require similar operations? Can we avoid disabling and enabling interrupts in the DMA critical functions of the network driver in Linux and PDK?

Most of the issues we are encountering now involve accessing the enable/disable interrupt registers for DMA INTR.

Linux:

am65_cpsw_nuss_tx_irq

     -->disable_irq_nosync(irq)

am65_cpsw_nuss_tx_poll

      ---> enable_irq(tx_chn->irq);

PDK:

Lwip2Enet_notifyRxPackets

     -->EnetDma_disableRxEvent(rx->hFlow)

Lwip2Enet_rxPacketTask

    -->EnetDma_enableRxEvent(rx->hFlow)

Hi,

Can you please open a new thread for the network driver question.

Regarding firewall 4320, the screen shot include above is showing that by default channel 0, and channel 256 are configured to values shown.

The Firewall TISCI Description — TISCI User Guide, describes where the channel number should be specified to readback settings.

Regards,

kb

HI KB:

We have created a new thread to inquire about the network issues.

In addition we have reverted all modifications made to rm_cfg.c, but the issue still persists.

Best regards

Alex

Hi KB:

         When reproducing the serror issue, we used JTAG (CCS) to read the Channelized firewall register configuration information。

M4 log:

/* Properties of channelized firewall at: NAVSS0_UDMASS_INTA_0_UDMASS_INTA0_CFG_GCNTRTI */
#define CSL_CH_FW_NAVSS0_UDMASS_INTA_0_UDMASS_INTA0__CFG__GCNTRTI_ID (4320U)
#define CSL_CH_FW_NAVSS0_UDMASS_INTA_0_UDMASS_INTA0__CFG__GCNTRTI_TYPE (CSL_FW_CHANNEL)
#define CSL_CH_FW_NAVSS0_UDMASS_INTA_0_UDMASS_INTA0__CFG__GCNTRTI_MMR_BASE (0x00000045438000U)

0x33D3C004  --> VINT#(0x34), so VINT#(0x34) channel firewall base：0x45438000 + 0x3c * 0x20 = 0x45438780。The firewall configuration obtained by CCS reading is as follows：

From the read registers, the firewall configuration seems to be correct. However, why produce an M4 firewall exception event when Linux accessing it?

Register interpretation：

0x00000000A : enable

0x00018B88: PRIV-ID 01 (non-secure A72), 8B88(NONSEC_SUPV_READ & WRITE).

Best regards

Alex

HI Alex,

Thank you for updates.

The most recent is showing A72 as priv Id, previous one I had looked at had MCU2_0 as originator.   In both cases the error of '1' (Line 4[16-23]) is logged meaning.

A theory would be that the DMA channel being used is not correct, as the firewall associated with it has not been enabled. 

Regards,

kb

HI KB：

      What are the next steps we can take to troubleshoot this issue?

Regards,

Alex

Hi Alex,

If issue can be reproduce with TI SDK on EVM that would be the ideal, as it enables the TI team to directly recreate and work on the issue.

Will loop in some other folks for thoughts.

Regards,

kb

Hi Alex,

I have not tried NFS with TDA4VH but on the the other TDA4 platforms we have used NFS and not seen the crash above. It wolly help if you can reproduce on EVM.

Best Regards,

Keerthy 

Hi Alex,

Did you get the chance to reproduce this on EVM? It would be really helpful if you could.

Regards,

Brijesh

Hi Brijesh

Currently we are using the Fast Startup solution on the customer board, not the one using systemd. So it takes time to test the NFS mounting function by booting with Tiny File System on the EVM board. In addition, I have two related questions to consult.

1. It is impossible to guarantee that the code on the customer board is exactly the same as that on the EVM. If there is an issue that can be replicated on the customer board but not on the EVM, how does TI typically resolve it?

2. For the current issue, we used the same RM-cfg as EVM, but the problem still occurs. Based on the above analysis, it is quite clear that it is a firewall issue. Does TI have any further troubleshooting ideas?

Best regards

Alex

Hi,

li quan said:

 It is impossible to guarantee that the code on the customer board is exactly the same as that on the EVM. If there is an issue that can be replicated on the customer board but not on the EVM, how does TI typically resolve it?

Please share the complete boot logs. Also are you using ethernet firmware to control CPSW or Linux native driver?

li quan said:

. For the current issue, we used the same RM-cfg as EVM, but the problem still occurs. Based on the above analysis, it is quite clear that it is a firewall issue. Does TI have any further troubleshooting ideas?

If you disable the MCU2_0 firmware can you check if this still happens? If not then we need to see why MCU2_0 is accessing that region which is firewalled.

- Keerthy

HI, Keerthy

The normal boot log and dmesg are shown below:

boot log:
U-Boot SPL 2021.01-svn522639 (Dec 26 2023 - 11:46:05 +0800)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
SPL initial stack usage: 13472 bytes
Trying to boot from SPI
bootpart :[main]
NOTICE:  BL31: v2.8(release):
NOTICE:  BL31: Built : 11:45:33, Dec 26 2023


U-Boot 2021.01-svn522639 (Dec 26 2023 - 11:45:39 +0800), Build: jenkins-AutoElec-cci-Pipeline-13339

SoC:   J784S4 SR1.0 GP
Model: HIKAUTO AE-B50038-S
DRAM:  4 GiB
Flash: 0 Bytes
MMC:   mmc@4fb0000: 1
In:    serial@2880000
Out:   serial@2880000
Err:   serial@2880000
Net:   eth1: ethernet@c200000, eth0: ethernet@c200000port@1
BOOTX: waitting remote boot notify
k3-navss-ringacc ringacc@3c000000: Ring Accelerator probed rings:1024, gp-rings[878,128] sci-dev-id:315
k3-navss-ringacc ringacc@3c000000: dma-ring-reset-quirk: disabled
Listening for TFTP transfer on 192.168.1.130
Load address: 0x82000000
Loading: *
         0 Bytes/s
done
Bytes transferred = 8 (8 hex)
Enter remote boot mode!
cadence_spi spi@47040000: Can't get reset: -2
k3-navss-ringacc ringacc@2b800000: Ring Accelerator probed rings:286, gp-rings[96,20] sci-dev-id:328
k3-navss-ringacc ringacc@2b800000: dma-ring-reset-quirk: disabled
SF: Detected w25q128 with page size 256 Bytes, erase size 64 KiB, total 16 MiB
Exit remote boot mode, do bootx!
Hit ctr+u key to stop autoboot:  0
k3_r5f_rproc r5f@41000000: Core 1 is already in use. No rproc commands work
k3_r5f_rproc r5f@41400000: Core 2 is already in use. No rproc commands work
## Loading kernel from FIT Image at 82000000 ...
   Using 'hikauto-AE-B50038-S-S01-j784s4.dtb' configuration
   Trying 'kernel' kernel subimage
     Description:  Image.lz4 linux-5.10.65 from TI TDA4x
     Type:         Kernel Image
     Compression:  lz4 compressed
     Data Start:   0x82000420
     Data Size:    9278858 Bytes = 8.8 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x84000000
     Entry Point:  0x84000000
     Hash algo:    sha1
     Hash value:   57f41a2491e476ea4ec115f17c45f978de678736
   Verifying Hash Integrity ... sha1+ OK
## Loading init Ramdisk from Legacy Image at 86000000 ...
   Image Name:   ti-image-initramfs-j7-evm
   Image Type:   AArch64 Linux RAMDisk Image (lz4 compressed)
   Data Size:    13020284 Bytes = 12.4 MiB
   Load Address: 00000000
   Entry Point:  00000000
## Loading fdt from FIT Image at 82000000 ...
   Using 'hikauto-AE-B50038-S-S01-j784s4.dtb' configuration
   Trying 'hikauto-AE-B50038-S-S01-j784s4.dtb' fdt subimage
     Description:  hikauto-AE-B50038-S-S01-j784s4
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x828d99b0
     Data Size:    87168 Bytes = 85.1 KiB
     Architecture: AArch64
     Hash algo:    sha1
     Hash value:   6cc28bb2a0d62a32a8a13763c03f19df6670e393
   Verifying Hash Integrity ... sha1+ OK
   Booting using the fdt blob at 0x828d99b0
   Uncompressing Kernel Image
   Using Device Tree in place at 00000000828d99b0, end 00000000828f1e2f

Starting kernel ...

[    0.000000] cma: Failed to reserve 512 MiB



[    0.325129] k3-dsp-rproc 64800000.dsp: configured DSP for IPC-only mode
[    0.331929] k3-dsp-rproc 64800000.dsp: DSP initialized in IPC-only mode
root@j7-evm:~# [    0.339832] k3-dsp-rproc 65800000.dsp: configured DSP for IPC-only mode
[    0.346555] k3-dsp-rproc 65800000.dsp: DSP initialized in IPC-only mode
[    0.353818] k3-dsp-rproc 66800000.dsp: configured DSP for IPC-only mode
[    0.360541] k3-dsp-rproc 66800000.dsp: DSP initialized in IPC-only mode
[    0.367749] k3-dsp-rproc 67800000.dsp: configured DSP for IPC-only mode
[    0.374446] k3-dsp-rproc 67800000.dsp: DSP initialized in IPC-only mode
[    0.384669] platform 41000000.r5f: configured R5F for IPC-only mode
[    0.391167] platform 41000000.r5f: R5F core initialized in IPC-only mode
[    0.399649] platform 5c00000.r5f: configured R5F for IPC-only mode
[    0.406046] platform 5c00000.r5f: R5F core initialized in IPC-only mode
[    0.413342] platform 5d00000.r5f: configured R5F for IPC-only mode
[    0.419671] platform 5d00000.r5f: R5F core initialized in IPC-only mode
[    0.427947] platform 5e00000.r5f: configured R5F for IPC-only mode
[    0.434305] platform 5e00000.r5f: R5F core initialized in IPC-only mode
[    0.441946] platform 5f00000.r5f: configured R5F for IPC-only mode
[    0.448249] platform 5f00000.r5f: R5F core initialized in IPC-only mode
[    0.456552] platform 5900000.r5f: configured R5F for IPC-only mode
[    0.462878] platform 5900000.r5f: R5F core initialized in IPC-only mode
[    0.470188] platform 5a00000.r5f: configured R5F for IPC-only mode
[    0.476485] platform 5a00000.r5f: R5F core initialized in IPC-only mode
[    0.539142] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac6: rpmsg attach_ext - rx_mtu:1522 features:00000007 tx_mtu[0]:2024 flow_idx:114 tx_cpsw_psil_dst_id:51712 mac_addr:70:ff:76:00:b7:7d mac-only:6
[    0.559113] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac7: rpmsg attach_ext - rx_mtu:1522 features:00000007 tx_mtu[0]:2024 flow_idx:115 tx_cpsw_psil_dst_id:51713 mac_addr:70:ff:76:00:88:13 mac-only:7
[    0.578860] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac8: rpmsg attach_ext - rx_mtu:1522 features:00000007 tx_mtu[0]:2024 flow_idx:116 tx_cpsw_psil_dst_id:51714 mac_addr:70:ff:76:00:9e:fc mac-only:8

dmesg:
dmesg
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd080]
[    0.000000] Linux version 5.10.162-g76b3e88d56 (ci@CI-AutoElec-Slave-71-138) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 9.2.1 20191025, GNU ld (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 2.33.1.20191209) #1 SMP PREEMPT Tue Dec 26 11:46:35 CST 2023
[    0.000000] Machine model: HIKAUTO AE-B50038-S
[    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002880000 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000088000000, size 176 MiB
[    0.000000] OF: reserved mem: initialized node dsp-multicore-dma-memory@88000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000093000000, size 16 MiB
[    0.000000] OF: reserved mem: initialized node bsp-multicore-dma-memory@93000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000094000000, size 64 MiB
[    0.000000] OF: reserved mem: initialized node vision-apps-core-heap-memory-lo@94000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000098000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f2_0-dma-memory@98000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000098100000, size 47 MiB
[    0.000000] OF: reserved mem: initialized node r5f2_0-memory@98100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009b000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f1_0-dma-memory@9b000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009b100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f1_0-memory@9b100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009c000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f2_1-dma-memory@9c000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009c100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f2_1-memory@9c100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009d000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f4_0-dma-memory@9d000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009d100000, size 23 MiB
[    0.000000] OF: reserved mem: initialized node r5f4_0-memory@9d100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a0000000, size 16 MiB
[    0.000000] OF: reserved mem: initialized node pcie_memory@a0000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a1000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f3_1-dma-memory@a1000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a1100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f3_1-memory@a1100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a2000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f4_1-dma-memory@a2000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a2100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f4_1-memory@a2100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a3000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f1_1-dma-memory@a3000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a3100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f1_1-memory@a3100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a4000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f3_0-dma-memory@a4000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a4100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f3_0-memory@a4100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000ac000000, size 48 MiB
[    0.000000] OF: reserved mem: initialized node rtos-ipc-memory-region@ac000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000af000000, size 80 MiB
[    0.000000] OF: reserved mem: initialized node vision-apps-dma-memory@af000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000b4000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c71_0-dma-memory@b4000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000b4100000, size 35 MiB
[    0.000000] OF: reserved mem: initialized node c71_0-memory@b4100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000b6400000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c71_1-dma-memory@b6400000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000b6500000, size 35 MiB
[    0.000000] OF: reserved mem: initialized node c71_1-memory@b6500000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000b8800000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c71_2-dma-memory@b8800000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000b8900000, size 35 MiB
[    0.000000] OF: reserved mem: initialized node c71_2-memory@b8900000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000bac00000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c71_3-dma-memory@bac00000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000bad00000, size 35 MiB
[    0.000000] OF: reserved mem: initialized node c71_3-memory@bad00000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: initialized node vision_apps_shared-memories, compatible id dma-heap-carveout
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000880000000, size 750 MiB
[    0.000000] OF: reserved mem: initialized node vision-apps-core-heap-memory-hi@880000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000008d4000000, size 64 MiB
[    0.000000] OF: reserved mem: initialized node pcie-memory-region@8d4000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000008e0000000, size 320 MiB
[    0.000000] OF: reserved mem: initialized node ddr-high-shared-memory@8e0000000, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000008ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x0000000087ffffff]
[    0.000000]   node   0: [mem 0x0000000088000000-0x00000000a4ffffff]
[    0.000000]   node   0: [mem 0x00000000a5000000-0x00000000abffffff]
[    0.000000]   node   0: [mem 0x00000000ac000000-0x00000000bcffffff]
[    0.000000]   node   0: [mem 0x00000000bd000000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000880000000-0x00000008aedfffff]
[    0.000000]   node   0: [mem 0x00000008aee00000-0x00000008d3ffffff]
[    0.000000]   node   0: [mem 0x00000008d4000000-0x00000008d7ffffff]
[    0.000000]   node   0: [mem 0x00000008d8000000-0x00000008dfffffff]
[    0.000000]   node   0: [mem 0x00000008e0000000-0x00000008f3ffffff]
[    0.000000]   node   0: [mem 0x00000008f4000000-0x00000008ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000008ffffffff]
[    0.000000] On node 0 totalpages: 65536
[    0.000000]   DMA32 zone: 32 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 32768 pages, LIFO batch:3
[    0.000000]   Normal zone: 32 pages used for memmap
[    0.000000]   Normal zone: 32768 pages, LIFO batch:3
[    0.000000] cma: Failed to reserve 512 MiB
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 2 pages/cpu s50136 r8192 d72744 u131072
[    0.000000] pcpu-alloc: s50136 r8192 d72744 u131072 alloc=2*65536
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5 [0] 6 [0] 7
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: EL2 vector hardening
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: detected: ARM erratum 1742098
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 65472
[    0.000000] Kernel command line: console=ttyS2,115200n8 quiet uboot_version=v2.7.0 uboot_svnversion=522639 earlycon=ns16550a,mmio32,0x02880000 irqchip.gicv3_pseudo_nmi=1 rproc_rsctable=41000000.r5f:0x8c@0x9b100000;5c00000.r5f:0x8c@0x98100000;5d00000.r5f:0x8c@0x9c100000;5e00000.r5f:0x8c@0xa4100000;5f00000.r5f:0x8c@0xa1100000;5900000.r5f:0x8c@0x9d100000;5a00000.r5f:0x8c@0xa2100000;64800000.dsp:0x98@0xb4100000;65800000.dsp:0x98@0xb6500000;66800000.dsp:0x98@0xb8900000;67800000.dsp:0x98@0xbad00000 bootxpart=main
[    0.000000] Dentry cache hash table entries: 524288 (order: 6, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 5, 2097152 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000fbff0000-0x00000000ffff0000] (64MB)
[    0.000000] Memory: 1644224K/4194304K available (11008K kernel code, 1334K rwdata, 4352K rodata, 1856K init, 703K bss, 2550080K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=8.
[    0.000000] rcu:     RCU priority boosting: priority 1 delay 500 ms.
[    0.000000]  Trampoline variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 960 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001900000
[    0.000000] ITS [mem 0x01820000-0x0182ffff]
[    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
[    0.000000] ITS@0x0000000001820000: allocated 1048576 Devices @8af000000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x00000008aef20000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x00000008aef30000
[    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
[    0.000002] sched_clock: 56 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
[    0.000175] Console: colour dummy device 80x25
[    0.000205] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000)
[    0.000211] pid_max: default: 32768 minimum: 301
[    0.000251] LSM: Security Framework initializing
[    0.000284] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.000294] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.001481] rcu: Hierarchical SRCU implementation.
[    0.001757] Platform MSI: msi-controller@1820000 domain created
[    0.001853] PCI/MSI: /bus@100000/interrupt-controller@1800000/msi-controller@1820000 domain created
[    0.001891] EFI services will not be available.
[    0.002134] smp: Bringing up secondary CPUs ...
[    0.002601] Detected PIPT I-cache on CPU1
[    0.002624] GICv3: CPU1: found redistributor 1 region 0:0x0000000001920000
[    0.002636] GICv3: CPU1: using allocated LPI pending table @0x00000008aef40000
[    0.002670] CPU1: Booted secondary processor 0x0000000001 [0x411fd080]
[    0.003144] Detected PIPT I-cache on CPU2
[    0.003158] GICv3: CPU2: found redistributor 2 region 0:0x0000000001940000
[    0.003170] GICv3: CPU2: using allocated LPI pending table @0x00000008aef50000
[    0.003198] CPU2: Booted secondary processor 0x0000000002 [0x411fd080]
[    0.003663] Detected PIPT I-cache on CPU3
[    0.003678] GICv3: CPU3: found redistributor 3 region 0:0x0000000001960000
[    0.003690] GICv3: CPU3: using allocated LPI pending table @0x00000008aef60000
[    0.003718] CPU3: Booted secondary processor 0x0000000003 [0x411fd080]
[    0.004235] Detected PIPT I-cache on CPU4
[    0.004260] GICv3: CPU4: found redistributor 100 region 0:0x0000000001980000
[    0.004273] GICv3: CPU4: using allocated LPI pending table @0x00000008aef70000
[    0.004309] CPU4: Booted secondary processor 0x0000000100 [0x411fd080]
[    0.004799] Detected PIPT I-cache on CPU5
[    0.004817] GICv3: CPU5: found redistributor 101 region 0:0x00000000019a0000
[    0.004828] GICv3: CPU5: using allocated LPI pending table @0x00000008aef80000
[    0.004856] CPU5: Booted secondary processor 0x0000000101 [0x411fd080]
[    0.005318] Detected PIPT I-cache on CPU6
[    0.005337] GICv3: CPU6: found redistributor 102 region 0:0x00000000019c0000
[    0.005349] GICv3: CPU6: using allocated LPI pending table @0x00000008aef90000
[    0.005378] CPU6: Booted secondary processor 0x0000000102 [0x411fd080]
[    0.005860] Detected PIPT I-cache on CPU7
[    0.005881] GICv3: CPU7: found redistributor 103 region 0:0x00000000019e0000
[    0.005893] GICv3: CPU7: using allocated LPI pending table @0x00000008aefa0000
[    0.005922] CPU7: Booted secondary processor 0x0000000103 [0x411fd080]
[    0.005979] smp: Brought up 1 node, 8 CPUs
[    0.005989] SMP: Total of 8 processors activated.
[    0.005994] CPU features: detected: 32-bit EL0 Support
[    0.005997] CPU features: detected: CRC32 instructions
[    0.017270] CPU: All CPU(s) started at EL2
[    0.017296] alternatives: patching kernel code
[    0.018100] devtmpfs: initialized
[    0.022383] KASLR disabled due to lack of seed
[    0.022486] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.022495] futex hash table entries: 2048 (order: 1, 131072 bytes, linear)
[    0.022692] pinctrl core: initialized pinctrl subsystem
[    0.022911] DMI not present or invalid.
[    0.023307] NET: Registered protocol family 16
[    0.023641] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations
[    0.023747] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.023936] thermal_sys: Registered thermal governor 'step_wise'
[    0.023938] thermal_sys: Registered thermal governor 'power_allocator'
[    0.024154] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.024245] ASID allocator initialised with 65536 entries
[    0.034176] HugeTLB registered 16.0 GiB page size, pre-allocated 0 pages
[    0.034181] HugeTLB registered 512 MiB page size, pre-allocated 0 pages
[    0.034184] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.035430] cryptd: max_cpu_qlen set to 1000
[    0.037446] k3-chipinfo 43000014.chipid: Family:J784S4 rev:SR1.0 JTAGID[0x0bb8002f] Detected
[    0.037664] vsys_3v3: supplied by evm_12v0
[    0.037838] vsys_5v0: supplied by evm_12v0
[    0.037966] vdd_mmc1: supplied by vsys_3v3
[    0.038494] iommu: Default domain type: Translated
[    0.038638] vgaarb: loaded
[    0.038828] SCSI subsystem initialized
[    0.038964] mc: Linux media interface: v0.10
[    0.038974] videodev: Linux video capture interface: v2.00
[    0.038999] pps_core: LinuxPPS API ver. 1 registered
[    0.039002] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.039008] PTP clock support registered
[    0.039026] EDAC MC: Ver: 3.0.0
[    0.039559] FPGA manager framework
[    0.039989] clocksource: Switched to clocksource arch_sys_counter
[    0.040198] VFS: Disk quotas dquot_6.6.0
[    0.040236] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[    0.042639] Carveout Heap: Exported 512 MiB at 0x00000000c1000000
[    0.042699] NET: Registered protocol family 2
[    0.042811] IP idents hash table entries: 65536 (order: 3, 524288 bytes, linear)
[    0.043973] tcp_listen_portaddr_hash hash table entries: 4096 (order: 0, 65536 bytes, linear)
[    0.044027] TCP established hash table entries: 32768 (order: 2, 262144 bytes, linear)
[    0.044130] TCP bind hash table entries: 32768 (order: 3, 524288 bytes, linear)
[    0.044336] TCP: Hash tables configured (established 32768 bind 32768)
[    0.044392] UDP hash table entries: 2048 (order: 0, 65536 bytes, linear)
[    0.044433] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes, linear)
[    0.044547] NET: Registered protocol family 1
[    0.044817] RPC: Registered named UNIX socket transport module.
[    0.044821] RPC: Registered udp transport module.
[    0.044823] RPC: Registered tcp transport module.
[    0.044825] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.044832] PCI: CLS 0 bytes, default 64
[    0.044950] Trying to unpack rootfs image as initramfs...
[    0.104613] Freeing initrd memory: 12608K
[    0.105269] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    0.107636] Initialise system trusted keyrings
[    0.107726] workingset: timestamp_bits=46 max_order=15 bucket_order=0
[    0.109442] NFS: Registering the id_resolver key type
[    0.109454] Key type id_resolver registered
[    0.109457] Key type id_legacy registered
[    0.109483] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.109486] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.109489] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    0.109751] 9p: Installing v9fs 9p2000 file system support
[    0.129103] Key type asymmetric registered
[    0.129107] Asymmetric key parser 'x509' registered
[    0.129124] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 241)
[    0.129128] io scheduler mq-deadline registered
[    0.129130] io scheduler kyber registered
[    0.130243] pinctrl-single 4301c000.pinctrl: 94 pins, size 376
[    0.130363] pinctrl-single 11c000.pinctrl: 72 pins, size 288
[    0.133585] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
[    0.139248] brd: module loaded
[    0.142743] loop: module loaded
[    0.144767] tun: Universal TUN/TAP device driver, 1.6
[    0.145265] VFIO - User Level meta-driver version: 0.3
[    0.145533] i2c /dev entries driver
[    0.146125] sdhci: Secure Digital Host Controller Interface driver
[    0.146127] sdhci: Copyright(c) Pierre Ossman
[    0.146264] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.146662] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[    0.147280] optee: probing for conduit method.
[    0.147286] optee: api uid mismatch
[    0.147291] optee: probe of firmware:optee failed with error -22
[    0.147919] ipip: IPv4 and MPLS over IPv4 tunneling driver
[    0.148141] NET: Registered protocol family 17
[    0.148168] 8021q: 802.1Q VLAN Support v1.8
[    0.148237] 9pnet: Installing 9P2000 support
[    0.148266] Key type dns_resolver registered
[    0.148372] registered taskstats version 1
[    0.148377] Loading compiled-in X.509 certificates
[    0.151472] ti-sci 44083000.system-controller: ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
[    0.163751] ti-sci-intr 310e0000.interrupt-controller: Interrupt Router 283 domain created
[    0.163903] ti-sci-inta 33d00000.msi-controller: Interrupt Aggregator domain 321 created
[    0.165676] k3-ringacc 2b800000.ringacc: Ring Accelerator probed rings:286, gp-rings[96,20] sci-dev-id:328
[    0.165681] k3-ringacc 2b800000.ringacc: dma-ring-reset-quirk: disabled
[    0.165685] k3-ringacc 2b800000.ringacc: RA Proxy rev. 66349100, num_proxies:64
[    0.166998] k3-ringacc 3c000000.ringacc: Ring Accelerator probed rings:1024, gp-rings[878,128] sci-dev-id:315
[    0.167001] k3-ringacc 3c000000.ringacc: dma-ring-reset-quirk: disabled
[    0.167004] k3-ringacc 3c000000.ringacc: RA Proxy rev. 66349100, num_proxies:64
[    0.167408] 2830000.serial: ttyS3 at MMIO 0x2830000 (irq = 16, base_baud = 3000000) is a 8250
[    0.167862] 2880000.serial: ttyS2 at MMIO 0x2880000 (irq = 17, base_baud = 3000000) is a 8250
[    0.167892] printk: console [ttyS2] enabled
[    0.167895] printk: bootconsole [ns16550a0] disabled
[    0.186984] spi-nor spi0.0: w25q128 (16384 Kbytes)
[    0.188758] omap-mailbox 31f80000.mailbox: omap mailbox rev 0x66fca100
[    0.188897] mmc1: CQHCI version 5.10
[    0.189066] omap-mailbox 31f81000.mailbox: omap mailbox rev 0x66fca100
[    0.189371] omap-mailbox 31f82000.mailbox: omap mailbox rev 0x66fca100
[    0.189643] omap-mailbox 31f83000.mailbox: omap mailbox rev 0x66fca100
[    0.189917] omap-mailbox 31f84000.mailbox: omap mailbox rev 0x66fca100
[    0.190183] omap-mailbox 31f85000.mailbox: omap mailbox rev 0x66fca100
[    0.190698] ti-udma 285c0000.dma-controller: Channels: 22 (tchan: 11, rchan: 11, gp-rflow: 8)
[    0.192015] ti-udma 31150000.dma-controller: Channels: 62 (tchan: 29, rchan: 33, gp-rflow: 16)
[    0.198100] debugfs: Directory 'pd:73' with parent 'pm_genpd' already present!
[    0.198120] debugfs: Directory 'pd:72' with parent 'pm_genpd' already present!
[    0.198475] debugfs: Directory 'pd:335' with parent 'pm_genpd' already present!
[    0.198491] debugfs: Directory 'pd:334' with parent 'pm_genpd' already present!
[    0.198542] debugfs: Directory 'pd:333' with parent 'pm_genpd' already present!
[    0.198557] debugfs: Directory 'pd:332' with parent 'pm_genpd' already present!
[    0.199647] am65-cpsw-nuss c200000.ethernet: initializing am65 cpsw nuss version 0x6BA02102, cpsw version 0x6BA82102 Ports: 2 quirks:00000000
[    0.199724] am65-cpsw-nuss c200000.ethernet: initialized cpsw ale version 1.4
[    0.199727] am65-cpsw-nuss c200000.ethernet: ALE Table size 64
[    0.200104] am65-cpsw-nuss c200000.ethernet: CPTS ver 0x4e8a010b, freq:200000000, add_val:4 pps:0
[    0.200961] am65-cpsw-nuss c200000.ethernet: set new flow-id-base 82
[    0.219752] mmc1: SDHCI controller on 4fb0000.mmc [4fb0000.mmc] using ADMA 64-bit
[    0.220159] Freeing unused kernel memory: 1856K
[    0.240018] Run /init as init process
[    0.240020]   with arguments:
[    0.240022]     /init
[    0.240024]   with environment:
[    0.240026]     HOME=/
[    0.240028]     TERM=linux
[    0.240029]     bootxpart=main
[    0.285702] random: dd: uninitialized urandom read (512 bytes read)
[    0.307704] random: dropbear: uninitialized urandom read (32 bytes read)
[    0.321847] am65-cpsw-nuss c200000.ethernet eth2: configuring for fixed/rgmii-rxid link mode
[    0.321906] am65-cpsw-nuss c200000.ethernet eth2: Link is Up - 100Mbps/Full - flow control off
[    0.322433] 8021q: adding VLAN 0 to HW filter on device eth2
[    0.324913] k3-dsp-rproc 64800000.dsp: assigned reserved memory node c71_0-dma-memory@b4000000
[    0.325129] k3-dsp-rproc 64800000.dsp: configured DSP for IPC-only mode
[    0.331802] remoteproc remoteproc0: 64800000.dsp is available
[    0.331834] remoteproc remoteproc0: attaching to 64800000.dsp
[    0.331860] remoteproc remoteproc0: unsupported resource 65538
[    0.331929] k3-dsp-rproc 64800000.dsp: DSP initialized in IPC-only mode
[    0.338539]  remoteproc0#vdev0buffer: assigned reserved memory node c71_0-dma-memory@b4000000
[    0.338766] virtio_rpmsg_bus virtio0: rpmsg host is online
[    0.339204]  remoteproc0#vdev0buffer: registered virtio0 (type 7)
[    0.339208] remoteproc remoteproc0: remote processor 64800000.dsp is now attached
[    0.339722] k3-dsp-rproc 65800000.dsp: assigned reserved memory node c71_1-dma-memory@b6400000
[    0.339832] k3-dsp-rproc 65800000.dsp: configured DSP for IPC-only mode
[    0.341491] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xd
[    0.346473] remoteproc remoteproc1: 65800000.dsp is available
[    0.346502] remoteproc remoteproc1: attaching to 65800000.dsp
[    0.346523] remoteproc remoteproc1: unsupported resource 65538
[    0.346555] k3-dsp-rproc 65800000.dsp: DSP initialized in IPC-only mode
[    0.346635] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0x15
[    0.353221]  remoteproc1#vdev0buffer: assigned reserved memory node c71_1-dma-memory@b6400000
[    0.353456] virtio_rpmsg_bus virtio1: rpmsg host is online
[    0.353469]  remoteproc1#vdev0buffer: registered virtio1 (type 7)
[    0.353473] remoteproc remoteproc1: remote processor 65800000.dsp is now attached
[    0.353718] k3-dsp-rproc 66800000.dsp: assigned reserved memory node c71_2-dma-memory@b8800000
[    0.353818] k3-dsp-rproc 66800000.dsp: configured DSP for IPC-only mode
[    0.353877] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xe
[    0.360447] remoteproc remoteproc2: 66800000.dsp is available
[    0.360471] remoteproc remoteproc2: attaching to 66800000.dsp
[    0.360491] remoteproc remoteproc2: unsupported resource 65538
[    0.360541] k3-dsp-rproc 66800000.dsp: DSP initialized in IPC-only mode
[    0.360795] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xd
[    0.367149]  remoteproc2#vdev0buffer: assigned reserved memory node c71_2-dma-memory@b8800000
[    0.367267] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0x15
[    0.367380] virtio_rpmsg_bus virtio2: rpmsg host is online
[    0.367394]  remoteproc2#vdev0buffer: registered virtio2 (type 7)
[    0.367397] remoteproc remoteproc2: remote processor 66800000.dsp is now attached
[    0.367642] k3-dsp-rproc 67800000.dsp: assigned reserved memory node c71_3-dma-memory@bac00000
[    0.367749] k3-dsp-rproc 67800000.dsp: configured DSP for IPC-only mode
[    0.367778] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xe
[    0.374375] remoteproc remoteproc3: 67800000.dsp is available
[    0.374396] remoteproc remoteproc3: attaching to 67800000.dsp
[    0.374415] remoteproc remoteproc3: unsupported resource 65538
[    0.374446] k3-dsp-rproc 67800000.dsp: DSP initialized in IPC-only mode
[    0.374702] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xd
[    0.381054]  remoteproc3#vdev0buffer: assigned reserved memory node c71_3-dma-memory@bac00000
[    0.381167] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0x15
[    0.381277] virtio_rpmsg_bus virtio3: rpmsg host is online
[    0.381290]  remoteproc3#vdev0buffer: registered virtio3 (type 7)
[    0.381294] remoteproc remoteproc3: remote processor 67800000.dsp is now attached
[    0.381669] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xe
[    0.384046] virtio_rpmsg_bus virtio3: creating channel rpmsg_chrdev addr 0xd
[    0.384171] virtio_rpmsg_bus virtio3: creating channel rpmsg_chrdev addr 0x15
[    0.384558] platform 41000000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1)
[    0.384669] platform 41000000.r5f: configured R5F for IPC-only mode
[    0.386331] virtio_rpmsg_bus virtio3: creating channel ti.ipc4.ping-pong addr 0xe
[    0.390985] platform 41000000.r5f: assigned reserved memory node r5f1_0-dma-memory@9b000000
[    0.391055] remoteproc remoteproc4: 41000000.r5f is available
[    0.391085] remoteproc remoteproc4: attaching to 41000000.r5f
[    0.391167] platform 41000000.r5f: R5F core initialized in IPC-only mode
[    0.397858]  remoteproc4#vdev0buffer: assigned reserved memory node r5f1_0-dma-memory@9b000000
[    0.398071] virtio_rpmsg_bus virtio4: rpmsg host is online
[    0.398422]  remoteproc4#vdev0buffer: registered virtio4 (type 7)
[    0.398426] remoteproc remoteproc4: remote processor 41000000.r5f is now attached
[    0.399649] platform 5c00000.r5f: configured R5F for IPC-only mode
[    0.405867] platform 5c00000.r5f: assigned reserved memory node r5f2_0-dma-memory@98000000
[    0.405958] remoteproc remoteproc5: 5c00000.r5f is available
[    0.405981] remoteproc remoteproc5: attaching to 5c00000.r5f
[    0.406046] platform 5c00000.r5f: R5F core initialized in IPC-only mode
[    0.412655]  remoteproc5#vdev0buffer: assigned reserved memory node r5f2_0-dma-memory@98000000
[    0.412860] virtio_rpmsg_bus virtio5: rpmsg host is online
[    0.413169]  remoteproc5#vdev0buffer: registered virtio5 (type 7)
[    0.413173] remoteproc remoteproc5: remote processor 5c00000.r5f is now attached
[    0.413342] platform 5d00000.r5f: configured R5F for IPC-only mode
[    0.416675] virtio_rpmsg_bus virtio5: creating channel rpmsg_chrdev addr 0xd
[    0.419530] platform 5d00000.r5f: assigned reserved memory node r5f2_1-dma-memory@9c000000
[    0.419584] remoteproc remoteproc6: 5d00000.r5f is available
[    0.419608] virtio_rpmsg_bus virtio5: creating channel rpmsg_chrdev addr 0x15
[    0.419610] remoteproc remoteproc6: attaching to 5d00000.r5f
[    0.419671] platform 5d00000.r5f: R5F core initialized in IPC-only mode
[    0.426279]  remoteproc6#vdev0buffer: assigned reserved memory node r5f2_1-dma-memory@9c000000
[    0.426481] virtio_rpmsg_bus virtio6: rpmsg host is online
[    0.426808]  remoteproc6#vdev0buffer: registered virtio6 (type 7)
[    0.426812] remoteproc remoteproc6: remote processor 5d00000.r5f is now attached
[    0.427947] platform 5e00000.r5f: configured R5F for IPC-only mode
[    0.434138] platform 5e00000.r5f: assigned reserved memory node r5f3_0-dma-memory@a4000000
[    0.434188] remoteproc remoteproc7: 5e00000.r5f is available
[    0.434242] remoteproc remoteproc7: attaching to 5e00000.r5f
[    0.434305] platform 5e00000.r5f: R5F core initialized in IPC-only mode
[    0.440911]  remoteproc7#vdev0buffer: assigned reserved memory node r5f3_0-dma-memory@a4000000
[    0.441033] virtio_rpmsg_bus virtio6: creating channel rpmsg-remote-service addr 0x37
[    0.441118] virtio_rpmsg_bus virtio7: rpmsg host is online
[    0.441434]  remoteproc7#vdev0buffer: registered virtio7 (type 7)
[    0.441438] remoteproc remoteproc7: remote processor 5e00000.r5f is now attached
[    0.441946] platform 5f00000.r5f: configured R5F for IPC-only mode
[    0.448136] platform 5f00000.r5f: assigned reserved memory node r5f3_1-dma-memory@a1000000
[    0.448186] remoteproc remoteproc8: 5f00000.r5f is available
[    0.448203] remoteproc remoteproc8: attaching to 5f00000.r5f
[    0.448249] platform 5f00000.r5f: R5F core initialized in IPC-only mode
[    0.454854]  remoteproc8#vdev0buffer: assigned reserved memory node r5f3_1-dma-memory@a1000000
[    0.455057] virtio_rpmsg_bus virtio8: rpmsg host is online
[    0.455358]  remoteproc8#vdev0buffer: registered virtio8 (type 7)
[    0.455362] remoteproc remoteproc8: remote processor 5f00000.r5f is now attached
[    0.456552] platform 5900000.r5f: configured R5F for IPC-only mode
[    0.461455] virtio_rpmsg_bus virtio4: creating channel rpmsg-remote-service addr 0x37
[    0.462738] platform 5900000.r5f: assigned reserved memory node r5f4_0-dma-memory@9d000000
[    0.462798] remoteproc remoteproc9: 5900000.r5f is available
[    0.462818] remoteproc remoteproc9: attaching to 5900000.r5f
[    0.462878] platform 5900000.r5f: R5F core initialized in IPC-only mode
[    0.469482]  remoteproc9#vdev0buffer: assigned reserved memory node r5f4_0-dma-memory@9d000000
[    0.469684] virtio_rpmsg_bus virtio9: rpmsg host is online
[    0.469990]  remoteproc9#vdev0buffer: registered virtio9 (type 7)
[    0.469994] remoteproc remoteproc9: remote processor 5900000.r5f is now attached
[    0.470188] platform 5a00000.r5f: configured R5F for IPC-only mode
[    0.476372] platform 5a00000.r5f: assigned reserved memory node r5f4_1-dma-memory@a2000000
[    0.476421] remoteproc remoteproc10: 5a00000.r5f is available
[    0.476438] remoteproc remoteproc10: attaching to 5a00000.r5f
[    0.476485] platform 5a00000.r5f: R5F core initialized in IPC-only mode
[    0.483089]  remoteproc10#vdev0buffer: assigned reserved memory node r5f4_1-dma-memory@a2000000
[    0.483289] virtio_rpmsg_bus virtio10: rpmsg host is online
[    0.483595]  remoteproc10#vdev0buffer: registered virtio10 (type 7)
[    0.483599] remoteproc remoteproc10: remote processor 5a00000.r5f is now attached
[    0.487900] virtio_rpmsg_bus virtio8: creating channel rpmsg-remote-service addr 0x37
[    0.510031] virtio_rpmsg_bus virtio7: creating channel rpmsg-remote-service addr 0x37
[    0.516063] virtio_rpmsg_bus virtio10: creating channel rpmsg-remote-service addr 0x37
[    0.537745] virtio_rpmsg_bus virtio5: creating channel ti.ethfw.notifyservice addr 0x1e
[    0.538110] virtio_rpmsg_bus virtio5: creating channel rpmsg-kdrv addr 0x1a
[    0.538594] rpmsg-kdrv-eth-switch rpmsg-kdrv-1-mpu_1_0_ethmac-device-6: Device info: permissions: 3FFFFFFF uart_id: 2
[    0.538598] rpmsg-kdrv-eth-switch rpmsg-kdrv-1-mpu_1_0_ethmac-device-6: FW ver 0.2 (rev 0) 26/Dec/2023 SHA:
[    0.538626] rpmsg-kdrv-eth-switch rpmsg-kdrv-2-mpu_1_0_ethmac-device-7: Device info: permissions: 3FFFFFFF uart_id: 2
[    0.538630] rpmsg-kdrv-eth-switch rpmsg-kdrv-2-mpu_1_0_ethmac-device-7: FW ver 0.2 (rev 0) 26/Dec/2023 SHA:
[    0.538657] rpmsg-kdrv-eth-switch rpmsg-kdrv-3-mpu_1_0_ethmac-device-8: Device info: permissions: 3FFFFFFF uart_id: 2
[    0.538661] rpmsg-kdrv-eth-switch rpmsg-kdrv-3-mpu_1_0_ethmac-device-8: FW ver 0.2 (rev 0) 26/Dec/2023 SHA:
[    0.538693] rpmsg-kdrv-eth-switch rpmsg-kdrv-4-mpu_1_0_ethmac-device-4: Device info: permissions: 3FFFFFFF uart_id: 2
[    0.538697] rpmsg-kdrv-eth-switch rpmsg-kdrv-4-mpu_1_0_ethmac-device-4: FW ver 0.2 (rev 0) 26/Dec/2023 SHA:
[    0.539142] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac6: rpmsg attach_ext - rx_mtu:1522 features:00000007 tx_mtu[0]:2024 flow_idx:114 tx_cpsw_psil_dst_id:51712 mac_addr:70:ff:76:00:b7:7d mac-only:6
[    0.557110] virtio_rpmsg_bus virtio9: creating channel rpmsg-remote-service addr 0x37
[    0.558715] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac6: virt_cpsw_nuss mac loaded
[    0.558719] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac6: rdev_features:00000007 rdev_mtu:1522 flow_id:114 tx_psil_dst_id:4A00 mac_only:6
[    0.558723] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac6: local_mac_addr:00:00:00:00:00:00 rdev_mac_addr:70:ff:76:00:b7:7d
[    0.559113] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac7: rpmsg attach_ext - rx_mtu:1522 features:00000007 tx_mtu[0]:2024 flow_idx:115 tx_cpsw_psil_dst_id:51713 mac_addr:70:ff:76:00:88:13 mac-only:7
[    0.578438] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac7: virt_cpsw_nuss mac loaded
[    0.578443] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac7: rdev_features:00000007 rdev_mtu:1522 flow_id:115 tx_psil_dst_id:4A01 mac_only:7
[    0.578446] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac7: local_mac_addr:00:00:00:00:00:00 rdev_mac_addr:70:ff:76:00:88:13
[    0.578860] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac8: rpmsg attach_ext - rx_mtu:1522 features:00000007 tx_mtu[0]:2024 flow_idx:116 tx_cpsw_psil_dst_id:51714 mac_addr:70:ff:76:00:9e:fc mac-only:8
[    0.598558] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac8: virt_cpsw_nuss mac loaded
[    0.598564] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac8: rdev_features:00000007 rdev_mtu:1522 flow_id:116 tx_psil_dst_id:4A02 mac_only:8
[    0.598567] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac8: local_mac_addr:00:00:00:00:00:00 rdev_mac_addr:70:ff:76:00:9e:fc
[    0.654581] virtio_rpmsg_bus virtio5: creating channel ti.ipc4.ping-pong addr 0xe
[    0.702717] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac8: virt_cpsw_nuss mac started
[    0.833062] pci_epf_test_bind func 0
[    0.833093] j721e-pcie 2910000.pcie-ep: epf space bar-1,phys_addr 0x8d4000000,size 0x2000000
[    0.833099] j721e-pcie 2910000.pcie-ep: epf space bar-2,phys_addr 0x8d6000000,size 0x2000000
[    0.837167] pci_epf_test_bind func 1
[    0.837194] j721e-pcie 2910000.pcie-ep: epf space bar-1,phys_addr 0xa0000000,size 0x1000000
[    0.840898] GTC: loading out-of-tree module taints kernel.
[    0.854103] Successfully initialized hikio module
[    0.984109] pci_epf_test pci_epf_test.0: function 0 ,shmem_iomem 0xffff800138000000,shmem_phymem 0x1a000000,shmem_remotemem 0x8d4000000
[    0.992056] pci_epf_test pci_epf_test.0: function 0 ,shmem_iomem 0xffff80013c000000,shmem_phymem 0x1c000000,shmem_remotemem 0x8d6000000
[    1.055307] PVR_K:  318: Read BVNC 36.53.104.796 from HW device registers
[    1.055318] PVR_K:  318: RGX Device registered BVNC 36.53.104.796 with 1 core in the system
[    1.055548] [drm] Initialized pvr 1.15.6133109 20170530 for 4e20000000.gpu on minor 0
[    1.061800] j721e-cpsw-virt-mac main-r5fss-cpsw9g-virt-mac6: virt_cpsw_nuss mac started
[    1.075215] device eth1 entered promiscuous mode
[    1.090410] device eth3 entered promiscuous mode
[    1.093218] random: pimd: uninitialized urandom read (4 bytes read)
[    1.115470] HAL_GET_PARTITIONS_SIZE1: 80085001
[    1.396008] random: crng init done
[   31.716004] vdd_mmc1: disabling

Now we using ethernet firmware to control CPSW on MCU2_0.

After disable mcu2_0, there is still an issue of serror on Linux.

Best regards

Alex

Hi,

li quan said:

After disable mcu2_0, there is still an issue of serror on Linux.

The issue description is that MCU2_0 is trying to access some firewalled memory right?
I am not sure if you are getting the same exception or a different one after disabling MCU2_0.

Also the logs attached are not clearly showing when the issue happens?

- Keerthy

Hi Brijesh, 

We have successfully reproduced the SError issue on the EVM.Please refer to the attached file for the reproduction method.
If you have any questions, please contact us.

Best regards
Alexreproduce serror exception annex.zip

Hi Alex 

Allow us some time to replicate the issue will update you regarding the process by end of the next week.

Regards
Diwakar

Hi Diwakar

Soft reminder to keep this issue in your schedule. Customer is pushing us hard as it is affecting project SOP deadline. 

Please be sure to replicate the issue on your site and give us updates by EOW.

Regards

Zekun

Hi, Diwakar

How is going with the replication? If you met any problems you can updates here. 

Customer mentioned that this step is not need if we can mount successfully.

Regards

Zekun

Hi Zekun 

We would like to have a call with customer for better understanding on this issue.

Regards
Diwakar

Hi, Diwakar

The attached file is the modified rootfs. Please adjust the etc/init.d/nfs_test.sh script according to your environment. Special attention should be paid to the fact that the ENV parameters of uboot need to be consistent with ours, as we have discovered through testing that this parameter is also relevant.

Additionally, our kernel boot logs and serial logs are also attached for your reference.

Best regards

Alex

tinyrootfs_and_logs.zip

Hi Alex 

Thanks for share the rootfs.

The logs which you share are on the EVM both pass and fail logs ?

Regards
Diwakar

Hi Diwakar

The kernel log indicates a normal startup, while the serial port log is the log recorded when the Serror occurs.

Best regards

Alex

HI Alex 

Are those both logs from TI EVM ?

Regards
Diwakar

Hi Divakar

Those logs are all from the TI EVM. 

May I ask if there are any issues with the current reproduction environment? Can reproduce the issue now?

Best regards

Hi Divakar

May I ask how the current issue is progressing?

Best regards

Alex

Hello,

This thread is assigned to our engineer in the India office. Due to a regional holiday, half of our team is out of the office. Please expect a 1~2 day delay in responses.

Apologies for the delay, and thank you for your patience.

Thanks.

Hi Praveen

Has there been any new progress on the issue?

Best regards

Alex

Hi, Experts

There have been severals days past, may I know the current status of this issue?

Regards

Zekun

Hi Zekun,Alex

Can you please try with the below patch i tried 1000 times so far not able to reproduce the issue.

https://patchwork.kernel.org/project/linux-arm-kernel/list/?series=845384

net-net-ethernet-ti-am65-cpsw-nuss-cleanup-DMA-Channels-before-using-them.patch

Regards
Diwakar

Hi Divakar, Zekun

We have applied this patch in the SDK 8.6, but the SError issue still recurs after testing on the EVM board. Could you please clarify whether this patch is based on SDK 8.6 or another version?

Attached is the reproduction log.

[16:15:04:987]U-Boot SPL 2021.01-g62a9e51344 (Mar 13 2023 - 15:43:18 +0000)
[16:15:05:061]ti_i2c_eeprom_am6_get: Ignoring record id 255
[16:15:05:062]ti_i2c_eeprom_am6_get: Ignoring record id 255
[16:15:05:063]ti_i2c_eeprom_am6_get: Ignoring record id 255
[16:15:05:074]ti_i2c_eeprom_am6_get: Ignoring record id 255
[16:15:05:074]ti_i2c_eeprom_am6_get: Ignoring record id 255
[16:15:05:075]ti_i2c_eeprom_am6_get: Ignoring record id 255
[16:15:05:093]ti_i2c_eeprom_am6_get: Ignoring record id 255
[16:15:05:094]ti_i2c_eeprom_am6_get: Ignoring record id 255
[16:15:05:096]ti_i2c_eeprom_am6_get: Ignoring record id 255
[16:15:05:118]ti_i2c_eeprom_am6_get: Ignoring record id 255
[16:15:05:119]SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
[16:15:05:276]SPL initial stack usage: 13472 bytes
[16:15:05:277]Trying to boot from MMC2
[16:15:05:497]Starting ATF on ARM64 core...
[16:15:05:497]
[16:15:05:512]NOTICE:  BL31: v2.8(release):v2.8-226-g2fcd408bb3-dirty
[16:15:05:513]NOTICE:  BL31: Built : 15:42:56, Mar 13 2023
[16:15:05:524]I/TC: 
[16:15:05:545]I/TC: OP-TEE version: 3.20.0 (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10))) #1 Mon Mar 13 15:43:08 UTC 2023 aarch64
[16:15:05:546]I/TC: WARNING: This OP-TEE configuration might be insecure!
[16:15:05:559]I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
[16:15:05:560]I/TC: Primary CPU initializing
[16:15:05:574]I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
[16:15:05:574]I/TC: HUK Initialized
[16:15:05:575]I/TC: Activated SA2UL device
[16:15:05:585]I/TC: Fixing SA2UL firewall owner for GP device
[16:15:05:595]I/TC: Enabled firewalls for SA2UL TRNG device
[16:15:05:595]I/TC: SA2UL TRNG initialized
[16:15:05:595]I/TC: SA2UL Drivers initialized
[16:15:05:596]I/TC: Primary CPU switching to normal world boot
[16:15:05:687]
[16:15:05:688]U-Boot SPL 2021.01-g62a9e51344 (Mar 13 2023 - 15:43:26 +0000)
[16:15:05:716]SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
[16:15:05:749]Trying to boot from MMC2
[16:15:06:979]
[16:15:06:979]
[16:15:06:979]U-Boot 2021.01-g62a9e51344 (Mar 13 2023 - 15:43:26 +0000)
[16:15:06:979]
[16:15:06:980]SoC:   J784S4 SR1.0 GP
[16:15:06:980]Model: Texas Instruments J784S4 EVM
[16:15:06:994]Board: J784S4X-EVM rev E2
[16:15:06:995]DRAM:  32 GiB
[16:15:07:027]idle-statesFlash: 0 Bytes
[16:15:07:044]MMC:   mmc@4f80000: 0, mmc@4fb0000: 1
[16:15:07:135]Loading Environment from MMC... OK
[16:15:07:135]In:    serial@2880000
[16:15:07:135]Out:   serial@2880000
[16:15:07:136]Err:   serial@2880000
[16:15:07:147]am65_cpsw_nuss ethernet@46000000: K3 CPSW: nuss_ver: 0x6BA02102 cpsw_ver: 0x6BA82102 ale_ver: 0x00293904 Ports:1 mdio_freq:1000000
[16:15:07:160]Unidentified board claims J784S4X-EVM in eeprom header
[16:15:07:161]Net:   eth0: ethernet@46000000port@1
[16:15:07:162]Hit any key to stop autoboot:  0 
[16:15:07:371]switch to partitions #0, OK
[16:15:07:372]mmc1 is current device
[16:15:08:019]SD/MMC found on device 1
[16:15:08:029]Failed to load 'boot.scr'
[16:15:08:029]1331 bytes read in 14 ms (92.8 KiB/s)
[16:15:08:030]Loaded env from uEnv.txt
[16:15:08:030]Importing environment from mmc1 ...
[16:15:08:030]Running uenvcmd ...
[16:15:08:030]1 bytes read in 14 ms (0 Bytes/s)
[16:15:08:030]Already setup.
[16:15:08:030]gpio: pin gpio@22_17 (gpio 188) value is 1
[16:15:08:030]gpio: pin gpio@22_16 (gpio 187) value is 0
[16:15:08:030]gpio: pin gpio@22_13 (gpio 184) value is 1
[16:15:08:030]gpio: pin gpio@22_14 (gpio 185) value is 1
[16:15:08:030]gpio: pin gpio@22_15 (gpio 186) value is 1
[16:15:08:031]k3_r5f_rproc r5f@41000000: Core 1 is already in use. No rproc commands work
[16:15:08:031]k3_r5f_rproc r5f@41400000: Core 2 is already in use. No rproc commands work
[16:15:08:031]597188 bytes read in 50 ms (11.4 MiB/s)
[16:15:08:031]Load Remote Processor 2 with data@addr=0x82000000 597188 bytes: Success!
[16:15:08:031]301996 bytes read in 38 ms (7.6 MiB/s)
[16:15:08:031]Load Remote Processor 3 with data@addr=0x82000000 301996 bytes: Success!
[16:15:08:031]179128 bytes read in 33 ms (5.2 MiB/s)
[16:15:08:032]Load Remote Processor 4 with data@addr=0x82000000 179128 bytes: Success!
[16:15:08:032]179128 bytes read in 32 ms (5.3 MiB/s)
[16:15:08:032]Load Remote Processor 5 with data@addr=0x82000000 179128 bytes: Success!
[16:15:08:032]596856 bytes read in 50 ms (11.4 MiB/s)
[16:15:08:033]Load Remote Processor 6 with data@addr=0x82000000 596856 bytes: Success!
[16:15:08:034]179128 bytes read in 32 ms (5.3 MiB/s)
[16:15:08:041]Load Remote Processor 7 with data@addr=0x82000000 179128 bytes: Success!
[16:15:09:066]14684456 bytes read in 634 ms (22.1 MiB/s)
[16:15:09:076]Load Remote Processor 8 with data@addr=0x82000000 14684456 bytes: Success!
[16:15:09:353]14946600 bytes read in 646 ms (22.1 MiB/s)
[16:15:09:354]Load Remote Processor 9 with data@addr=0x82000000 14946600 bytes: Success!
[16:15:10:100]14553384 bytes read in 629 ms (22.1 MiB/s)
[16:15:10:106]Load Remote Processor 10 with data@addr=0x82000000 14553384 bytes: Success!
[16:15:11:133]14553384 bytes read in 630 ms (22 MiB/s)
[16:15:11:147]Load Remote Processor 11 with data@addr=0x82000000 14553384 bytes: Success!
[16:15:11:479]19079680 bytes read in 808 ms (22.5 MiB/s)
[16:15:11:507]99445 bytes read in 19 ms (5 MiB/s)
[16:15:11:528]13306 bytes read in 14 ms (927.7 KiB/s)
[16:15:11:653]## Flattened Device Tree blob at 88000000
[16:15:11:667]   Booting using the fdt blob at 0x88000000
[16:15:11:703]   Loading Device Tree to 000000008fee4000, end 000000008fffffff ... OK
[16:15:11:703]
[16:15:11:704]Starting kernel ...
[16:15:11:705]
[16:15:12:882]I/TC: Secondary CPU 1 initializing
[16:15:12:892]I/TC: Secondary CPU 1 switching to normal world boot
[16:15:12:892]I/TC: Secondary CPU 2 initializing
[16:15:12:893]I/TC: Secondary CPU 2 switching to normal world boot
[16:15:12:893]I/TC: Secondary CPU 3 initializing
[16:15:12:905]I/TC: Secondary CPU 3 switching to normal world boot
[16:15:12:905]I/TC: Secondary CPU 2 initializing
[16:15:12:906]I/TC: Secondary CPU 2 switching to normal world boot
[16:15:12:916]I/TC: Secondary CPU 3 initializing
[16:15:12:916]I/TC: Secondary CPU 3 switching to normal world boot
[16:15:12:917]I/TC: Secondary CPU 4 initializing
[16:15:12:923]I/TC: Secondary CPU 4 switching to normal world boot
[16:15:12:938]I/TC: Secondary CPU 5 initializing
[16:15:12:939]I/TC: Secondary CPU 5 switching to normal world boot
[16:15:13:043]I/TC: Reserved shared memory is enabled
[16:15:13:064]I/TC: Dynamic shared memory is enabled
[16:15:13:064]I/TC: Normal World virtualization support is disabled
[16:15:13:065]I/TC: Asynchronous notifications are disabled
[16:15:14:199][    1.314382] j721e-pcie 2910000.pcie: Failed to init phy
[16:15:14:265][    1.393004] cadence-qspi 47040000.spi: error -ENODEV: No Rx DMA available
[16:15:14:349][    1.473283] cadence-qspi 47050000.spi: error -ENODEV: No Rx DMA available
[16:15:14:444][    1.563068] am65-cpsw-nuss c200000.ethernet: Use random MAC address
[16:15:16:344][    3.453275] [ZJDBG] am64-cpsw-nuss ------
[16:15:16:375][    3.499048] am65-cpsw-nuss c200000.ethernet: Use random MAC address
[16:15:16:376][    3.507331] [ZJDBG] am64-cpsw-nuss ------
[16:15:16:387][    3.513797] debugfs: Directory 'pd:74' with parent 'pm_genpd' already present!
[16:15:16:395][    3.521034] debugfs: Directory 'pd:73' with parent 'pm_genpd' already present!
[16:15:16:403][    3.528257] debugfs: Directory 'pd:72' with parent 'pm_genpd' already present!
[16:15:16:414][    3.535935] debugfs: Directory 'pd:335' with parent 'pm_genpd' already present!
[16:15:16:428][    3.543243] debugfs: Directory 'pd:333' with parent 'pm_genpd' already present!
[16:15:16:430][    3.550561] debugfs: Directory 'pd:332' with parent 'pm_genpd' already present!
[16:15:16:684]INIT: version 2.96 booting
[16:15:16:813]hwclock: can't open '/dev/misc/rtc': No such file or directory
[16:15:16:814]Mon Mar 13 15:52:52 UTC 2023
[16:15:16:827]hwclock: can't open '/dev/misc/rtc': No such file or directory
[16:15:16:896]INIT: Entering runlevel: 5
[16:15:16:905]Configuring network interfaces... done.
[16:15:16:948][    4.076233] k3-dsp-rproc 64800000.dsp: configured DSP for IPC-only mode
[16:15:16:958][    4.083026] k3-dsp-rproc 64800000.dsp: DSP initialized in IPC-only mode
[16:15:16:965][    4.091735] k3-dsp-rproc 65800000.dsp: configured DSP for IPC-only mode
[16:15:16:975][    4.098455] k3-dsp-rproc 65800000.dsp: DSP initialized in IPC-only mode
[16:15:16:980][    4.107347] k3-dsp-rproc 66800000.dsp: configured DSP for IPC-only mode
[16:15:16:992][    4.114095] k3-dsp-rproc 66800000.dsp: DSP initialized in IPC-only mode
[16:15:16:998][    4.122946] k3-dsp-rproc 67800000.dsp: configured DSP for IPC-only mode
[16:15:17:007][    4.129676] k3-dsp-rproc 67800000.dsp: DSP initialized in IPC-only mode
[16:15:17:022][    4.147486] platform 41000000.r5f: configured R5F for IPC-only mode
[16:15:17:033][    4.154275] platform 41000000.r5f: R5F core initialized in IPC-only mode
[16:15:17:034][    4.163092] platform 5c00000.r5f: configured R5F for IPC-only mode
[16:15:17:049][    4.170021] platform 5c00000.r5f: R5F core initialized in IPC-only mode
[16:15:17:050][    4.177383] platform 5d00000.r5f: configured R5F for IPC-only mode
[16:15:17:064][    4.184234] platform 5d00000.r5f: R5F core initialized in IPC-only mode
[16:15:17:065][    4.193177] platform 5e00000.r5f: configured R5F for IPC-only mode
[16:15:17:080][    4.199883] platform 5e00000.r5f: R5F core initialized in IPC-only mode
[16:15:17:081][    4.207115] platform 5f00000.r5f: configured R5F for IPC-only mode
[16:15:17:090][    4.213808] platform 5f00000.r5f: R5F core initialized in IPC-only mode
[16:15:17:103][    4.222277] platform 5900000.r5f: configured R5F for IPC-only mode
[16:15:17:103][    4.229277] platform 5900000.r5f: R5F core initialized in IPC-only mode
[16:15:17:109][    4.236495] platform 5a00000.r5f: configured R5F for IPC-only mode
[16:15:17:117][    4.243786] platform 5a00000.r5f: R5F core initialized in IPC-only mode
[16:15:17:136]Module                  Size  Used by
[16:15:17:136]ti_k3_r5_remoteproc    28672  0
[16:15:17:136]ti_k3_m4_remoteproc    20480  0
[16:15:17:137]ti_k3_dsp_remoteproc    20480  0
[16:15:17:146]virtio_rpmsg_bus       24576  0
[16:15:17:146]rpmsg_char             20480  0
[16:15:17:146]mount nfs start----------
[16:15:17:147]mount result:---------------------1
[16:15:17:154]Filesystem           1K-blocks      Used Available Use% Mounted on
[16:15:17:166]/dev/root              7390912    420356   6573708   6% /
[16:15:17:166]devtmpfs              13849024         0  13849024   0% /dev
[16:15:17:167]tmpfs                 14767456         8  14767448   0% /run
[16:15:17:177]tmpfs                    51200        40     51160   0% /var/volatile
[16:15:17:180]10.1.14.33:/data1/zhangjing24/nfs
[16:15:17:186]                      20971520  10970112  10001408  52% /mnt/nfs0
[16:15:17:197]mount result:---------------------2
[16:15:17:198]#!/bin/sh
[16:15:17:198]
[16:15:17:198]echo "nfs data write:####################"
[16:15:17:198]rm -rf /mnt/nfs0/tievm/random.bin
[16:15:17:198]dd if=/dev/zero of=/mnt/nfs0/tievm/random.bin bs=1024 count=10240
[16:15:17:209]echo "nfs data write end#################"
[16:15:17:209]
[16:15:17:211]
[16:15:17:227]echo "nfs data read:*******************[    4.338108] SError Interrupt on CPU0, code 0xbf000002 -- SError
[16:15:17:227][    4.338110] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.10.162-g76b3e88d56 #1
[16:15:17:228][    4.338112] Hardware name: Texas Instruments J784S4 EVM (DT)
[16:15:17:240][    4.338113] pstate: 00000085 (nzcv daIf -PAN -UAO -TCO BTYPE=--)
[16:15:17:240][    4.338114] pc : _raw_spin_unlock+0x38/0x48
[16:15:17:242][    4.338115] lr : handle_irq_event+0x40/0xe0
[16:15:17:249][    4.338116] sp : ffff8000112bbed0
[16:15:17:263][    4.338117] x29: ffff8000112bbed0 x28: ffff8000111219c0 
[16:15:17:263][    4.338120] x27: 00000000ffee3504 x26: ffff8000112bc000 
[16:15:17:264][    4.338123] x25: ffff0008f20f2680 x24: ffff0008f36f0060 
[16:15:17:280][    4.338125] x23: ffff8000111219c0 x22: ffff0008f36f0000 
[16:15:17:280][    4.338127] x21: ffff0008f211f8dc x20: ffff0008f211f8dc 
[16:15:17:282][    4.338129] x19: ffff0008f211f800 x18: 0000000000000002 
[16:15:17:293][    4.338131] x17: 000000008a14937f x16: 00000000af384a83 
[16:15:17:293][    4.338133] x15: 0000000000000000 x14: 0000000000000161 
[16:15:17:294][    4.338135] x13: 0000000000000000 x12: 0000000000000001 
[16:15:17:311][    4.338138] x11: 0000000000000040 x10: ffff8000111a0750 
[16:15:17:311][    4.338140] x9 : ffff8000111a0748 x8 : ffff0008f2401940 
[16:15:17:313][    4.338142] x7 : 0000000000000000 x6 : 0000000000000000 
[16:15:17:324][    4.338144] x5 : ffff800f6b443000 x4 : ffff8000112bbf00 
[16:15:17:325][    4.338147] x3 : ffff800f6b443000 x2 : ffff0008f211f800 
[16:15:17:325][    4.338149] x1 : ffff8000111219c0 x0 : 0000000100010001 
[16:15:17:339][    4.338152] Kernel panic - not syncing: Asynchronous SError Interrupt
[16:15:17:340][    4.338153] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.10.162-g76b3e88d56 #1
[16:15:17:353][    4.338154] Hardware name: Texas Instruments J784S4 EVM (DT)
[16:15:17:353][    4.338155] Call trace:
[16:15:17:353][    4.338156]  dump_backtrace+0x0/0x1a0
[16:15:17:354][    4.338157]  show_stack+0x18/0x68
[16:15:17:370][    4.338158]  dump_stack+0xd0/0x12c
[16:15:17:370][    4.338159]  panic+0x16c/0x334
[16:15:17:370][    4.338160]  nmi_panic+0x8c/0x90
[16:15:17:370][    4.338161]  arm64_serror_panic+0x78/0x84
[16:15:17:371][    4.338162]  do_serror+0x38/0x98
[16:15:17:386][    4.338163]  el1_error+0x90/0x110
[16:15:17:386][    4.338164]  _raw_spin_unlock+0x38/0x48
[16:15:17:387][    4.338165]  handle_level_irq+0xb8/0x140
[16:15:17:387][    4.338166]  generic_handle_irq+0x30/0x48
[16:15:17:401][    4.338167]  ti_sci_inta_irq_handler+0xc4/0x160
[16:15:17:401][    4.338168]  generic_handle_irq+0x30/0x48
[16:15:17:401][    4.338169]  __handle_domain_irq+0x64/0xc0
[16:15:17:401][    4.338170]  gic_handle_irq+0x58/0x128
[16:15:17:402][    4.338171]  el1_irq+0xcc/0x180
[16:15:17:421][    4.338172]  arch_cpu_idle+0x18/0x28
[16:15:17:421][    4.338173]  default_idle_call+0x20/0x68
[16:15:17:421][    4.338174]  do_idle+0xc0/0x128
[16:15:17:421][    4.338175]  cpu_startup_entry+0x24/0x60
[16:15:17:422][    4.338176]  rest_init+0xd4/0xe4
[16:15:17:434][    4.338177]  arch_call_rest_init+0x10/0x1c
[16:15:17:434][    4.338178]  start_kernel+0x478/0x4b0
[16:15:17:434][    4.338202] SMP: stopping secondary CPUs
[16:15:17:451][    4.338203] Kernel Offset: disabled
[16:15:17:451][    4.338205] CPU features: 0x28040022,20006008
[16:15:17:451][    4.338205] Memory Limit: none
[16:15:17:451]**"
[16:15:17:451]rm -rf /run/test.bin
[16:15:17:452]dd if=/mnt/nfs0/tievm/Image of=/run/test.bin bs=1024
[16:15:17:462]echo "nfs data read end******************"
[16:15:17:462]
[16:15:17:462]sync
[16:15:17:463]
[16:15:17:479]df

Hi Li 

we have tested the patch on the 9.2 SDK and after that we are not seeing any issue related to the corruption , ideally it should be same for the 8.6SDK.

Can you make sure that the changes are applied properly and what is the address range for which we are getting firewall exception in the TIFS traces.

Regards
Diwakar

Hi Divakar

1. We can confirm that the current patch has been applied to the kernel. The following are the patch and kernel logs.

2. The exception log captured during the occurrence of the SError issue is as follows, and the firewall output is the same as before.

Exception addr 0x45B0B000 FWL Exception 0x110E000 0x10000 0x33D38004 0x0 0x2201 0x4

uart log：

[15:15:43:305]U-Boot SPL 2021.01-dirty (Apr 30 2024 - 14:42:16 +0800)
[15:15:43:385]ti_i2c_eeprom_am6_get: Ignoring record id 255
[15:15:43:396]ti_i2c_eeprom_am6_get: Ignoring record id 255
[15:15:43:396]ti_i2c_eeprom_am6_get: Ignoring record id 255
[15:15:43:400]ti_i2c_eeprom_am6_get: Ignoring record id 255
[15:15:43:413]ti_i2c_eeprom_am6_get: Ignoring record id 255
[15:15:43:413]ti_i2c_eeprom_am6_get: Ignoring record id 255
[15:15:43:416]ti_i2c_eeprom_am6_get: Ignoring record id 255
[15:15:43:427]ti_i2c_eeprom_am6_get: Ignoring record id 255
[15:15:43:428]ti_i2c_eeprom_am6_get: Ignoring record id 255
[15:15:43:432]ti_i2c_eeprom_am6_get: Ignoring record id 255
[15:15:43:450]SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
[15:15:43:609]SPL initial stack usage: 13472 bytes
[15:15:43:610]Trying to boot from MMC2
[15:15:43:834]Starting ATF on ARM64 core...
[15:15:43:835]
[15:15:43:849]NOTICE:  BL31: v2.8(release):v2.8-226-g2fcd408bb3-dirty
[15:15:43:850]NOTICE:  BL31: Built : 15:42:56, Mar 13 2023
[15:15:43:871]I/TC: 
[15:15:43:878]I/TC: OP-TEE version: 3.20.0 (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10))) #1 Mon Mar 13 15:43:08 UTC 2023 aarch64
[15:15:43:880]I/TC: WARNING: This OP-TEE configuration might be insecure!
[15:15:43:892]I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
[15:15:43:902]I/TC: Primary CPU initializing
[15:15:43:911]I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
[15:15:43:911]I/TC: HUK Initialized
[15:15:43:912]I/TC: Activated SA2UL device
[15:15:43:913]I/TC: Fixing SA2UL firewall owner for GP device
[15:15:43:934]I/TC: Enabled firewalls for SA2UL TRNG device
[15:15:43:934]I/TC: SA2UL TRNG initialized
[15:15:43:934]I/TC: SA2UL Drivers initialized
[15:15:43:935]I/TC: Primary CPU switching to normal world boot
[15:15:44:025]
[15:15:44:040]U-Boot SPL 2021.01-g62a9e51344 (Mar 13 2023 - 15:43:26 +0000)
[15:15:44:067]SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
[15:15:44:105]Trying to boot from MMC2
[15:15:45:257]
[15:15:45:257]
[15:15:45:257]U-Boot 2021.01-g62a9e51344 (Mar 13 2023 - 15:43:26 +0000)
[15:15:45:257]
[15:15:45:257]SoC:   J784S4 SR1.0 GP
[15:15:45:272]Model: Texas Instruments J784S4 EVM
[15:15:45:272]Board: J784S4X-EVM rev E2
[15:15:45:273]DRAM:  32 GiB
[15:15:45:321]idle-statesFlash: 0 Bytes
[15:15:45:337]MMC:   mmc@4f80000: 0, mmc@4fb0000: 1
[15:15:45:417]Loading Environment from MMC... OK
[15:15:45:418]In:    serial@2880000
[15:15:45:433]Out:   serial@2880000
[15:15:45:433]Err:   serial@2880000
[15:15:45:433]am65_cpsw_nuss ethernet@46000000: K3 CPSW: nuss_ver: 0x6BA02102 cpsw_ver: 0x6BA82102 ale_ver: 0x00293904 Ports:1 mdio_freq:1000000
[15:15:45:449]Unidentified board claims J784S4X-EVM in eeprom header
[15:15:45:450]Net:   eth0: ethernet@46000000port@1
[15:15:45:464]Hit any key to stop autoboot:  0 
[15:15:45:707]switch to partitions #0, OK
[15:15:45:720]mmc1 is current device
[15:15:45:960]SD/MMC found on device 1
[15:15:45:977]Failed to load 'boot.scr'
[15:15:45:992]1331 bytes read in 14 ms (92.8 KiB/s)
[15:15:45:993]Loaded env from uEnv.txt
[15:15:45:993]Importing environment from mmc1 ...
[15:15:45:993]Running uenvcmd ...
[15:15:46:020]1 bytes read in 14 ms (0 Bytes/s)
[15:15:46:020]Already setup.
[15:15:46:020]gpio: pin gpio@22_17 (gpio 188) value is 1
[15:15:46:020]gpio: pin gpio@22_16 (gpio 187) value is 0
[15:15:46:035]gpio: pin gpio@22_13 (gpio 184) value is 1
[15:15:46:035]gpio: pin gpio@22_14 (gpio 185) value is 1
[15:15:46:036]gpio: pin gpio@22_15 (gpio 186) value is 1
[15:15:46:057]k3_r5f_rproc r5f@41000000: Core 1 is already in use. No rproc commands work
[15:15:46:058]k3_r5f_rproc r5f@41400000: Core 2 is already in use. No rproc commands work
[15:15:46:142]597188 bytes read in 50 ms (11.4 MiB/s)
[15:15:46:157]Load Remote Processor 2 with data@addr=0x82000000 597188 bytes: Success!
[15:15:46:188]301996 bytes read in 38 ms (7.6 MiB/s)
[15:15:46:200]Load Remote Processor 3 with data@addr=0x82000000 301996 bytes: Success!
[15:15:46:252]179128 bytes read in 32 ms (5.3 MiB/s)
[15:15:46:252]Load Remote Processor 4 with data@addr=0x82000000 179128 bytes: Success!
[15:15:46:297]179128 bytes read in 32 ms (5.3 MiB/s)
[15:15:46:298]Load Remote Processor 5 with data@addr=0x82000000 179128 bytes: Success!
[15:15:46:364]596856 bytes read in 51 ms (11.2 MiB/s)
[15:15:46:379]Load Remote Processor 6 with data@addr=0x82000000 596856 bytes: Success!
[15:15:46:411]179128 bytes read in 32 ms (5.3 MiB/s)
[15:15:46:425]Load Remote Processor 7 with data@addr=0x82000000 179128 bytes: Success!
[15:15:47:065]14684456 bytes read in 634 ms (22.1 MiB/s)
[15:15:47:081]Load Remote Processor 8 with data@addr=0x82000000 14684456 bytes: Success!
[15:15:47:737]14946600 bytes read in 646 ms (22.1 MiB/s)
[15:15:47:752]Load Remote Processor 9 with data@addr=0x82000000 14946600 bytes: Success!
[15:15:48:379]14553384 bytes read in 629 ms (22.1 MiB/s)
[15:15:48:392]Load Remote Processor 10 with data@addr=0x82000000 14553384 bytes: Success!
[15:15:49:036]14553384 bytes read in 630 ms (22 MiB/s)
[15:15:49:051]Load Remote Processor 11 with data@addr=0x82000000 14553384 bytes: Success!
[15:15:49:881]19079680 bytes read in 809 ms (22.5 MiB/s)
[15:15:49:911]99445 bytes read in 20 ms (4.7 MiB/s)
[15:15:49:925]13306 bytes read in 14 ms (927.7 KiB/s)
[15:15:50:057]## Flattened Device Tree blob at 88000000
[15:15:50:057]   Booting using the fdt blob at 0x88000000
[15:15:50:089]   Loading Device Tree to 000000008fee4000, end 000000008fffffff ... OK
[15:15:50:090]
[15:15:50:104]Starting kernel ...
[15:15:50:105]
[15:15:51:274]I/TC: Secondary CPU 1 initializing
[15:15:51:283]I/TC: Secondary CPU 1 switching to normal world boot
[15:15:51:283]I/TC: Secondary CPU 2 initializing
[15:15:51:287]I/TC: Secondary CPU 2 switching to normal world boot
[15:15:51:288]I/TC: Secondary CPU 3 initializing
[15:15:51:300]I/TC: Secondary CPU 3 switching to normal world boot
[15:15:51:300]I/TC: Secondary CPU 2 initializing
[15:15:51:305]I/TC: Secondary CPU 2 switching to normal world boot
[15:15:51:318]I/TC: Secondary CPU 3 initializing
[15:15:51:318]I/TC: Secondary CPU 3 switching to normal world boot
[15:15:51:318]I/TC: Secondary CPU 4 initializing
[15:15:51:319]I/TC: Secondary CPU 4 switching to normal world boot
[15:15:51:336]I/TC: Secondary CPU 5 initializing
[15:15:51:337]I/TC: Secondary CPU 5 switching to normal world boot
[15:15:51:448]I/TC: Reserved shared memory is enabled
[15:15:51:470]I/TC: Dynamic shared memory is enabled
[15:15:51:470]I/TC: Normal World virtualization support is disabled
[15:15:51:471]I/TC: Asynchronous notifications are disabled
[15:15:52:590][    1.314724] j721e-pcie 2910000.pcie: Failed to init phy
[15:15:52:669][    1.392948] cadence-qspi 47040000.spi: error -ENODEV: No Rx DMA available
[15:15:52:752][    1.472727] cadence-qspi 47050000.spi: error -ENODEV: No Rx DMA available
[15:15:52:847][    1.563429] am65-cpsw-nuss c200000.ethernet: Use random MAC address
[15:15:54:735][    3.453676] [ZJDBG] am65_cpsw_nuss_register_ndevs:patch test log
[15:15:54:776][    3.499413] am65-cpsw-nuss c200000.ethernet: Use random MAC address
[15:15:54:782][    3.507710] [ZJDBG] am65_cpsw_nuss_register_ndevs:patch test log
[15:15:54:792][    3.516051] debugfs: Directory 'pd:74' with parent 'pm_genpd' already present!
[15:15:54:800][    3.523284] debugfs: Directory 'pd:73' with parent 'pm_genpd' already present!
[15:15:54:809][    3.530507] debugfs: Directory 'pd:72' with parent 'pm_genpd' already present!
[15:15:54:815][    3.538198] debugfs: Directory 'pd:335' with parent 'pm_genpd' already present!
[15:15:54:825][    3.545506] debugfs: Directory 'pd:333' with parent 'pm_genpd' already present!
[15:15:54:832][    3.552824] debugfs: Directory 'pd:332' with parent 'pm_genpd' already present!
[15:15:55:008]INIT: version 2.96 booting
[15:15:55:152]hwclock: can't open '/dev/misc/rtc': No such file or directory
[15:15:55:152]Mon Mar 13 15:52:52 UTC 2023
[15:15:55:153]hwclock: can't open '/dev/misc/rtc': No such file or directory
[15:15:55:216]INIT: Entering runlevel: 5
[15:15:55:252]Configuring network interfaces... done.
[15:15:55:291][    4.008921] k3-dsp-rproc 64800000.dsp: configured DSP for IPC-only mode
[15:15:55:292][    4.015702] k3-dsp-rproc 64800000.dsp: DSP initialized in IPC-only mode
[15:15:55:309][    4.024727] k3-dsp-rproc 65800000.dsp: configured DSP for IPC-only mode
[15:15:55:309][    4.031457] k3-dsp-rproc 65800000.dsp: DSP initialized in IPC-only mode
[15:15:55:323][    4.040380] k3-dsp-rproc 66800000.dsp: configured DSP for IPC-only mode
[15:15:55:323][    4.047121] k3-dsp-rproc 66800000.dsp: DSP initialized in IPC-only mode
[15:15:55:327][    4.055940] k3-dsp-rproc 67800000.dsp: configured DSP for IPC-only mode
[15:15:55:347][    4.062713] k3-dsp-rproc 67800000.dsp: DSP initialized in IPC-only mode
[15:15:55:355][    4.080720] platform 41000000.r5f: configured R5F for IPC-only mode
[15:15:55:359][    4.087503] platform 41000000.r5f: R5F core initialized in IPC-only mode
[15:15:55:371][    4.096297] platform 5c00000.r5f: configured R5F for IPC-only mode
[15:15:55:379][    4.103180] platform 5c00000.r5f: R5F core initialized in IPC-only mode
[15:15:55:387][    4.110609] platform 5d00000.r5f: configured R5F for IPC-only mode
[15:15:55:394][    4.117651] platform 5d00000.r5f: R5F core initialized in IPC-only mode
[15:15:55:411][    4.126194] platform 5e00000.r5f: configured R5F for IPC-only mode
[15:15:55:411][    4.132894] platform 5e00000.r5f: R5F core initialized in IPC-only mode
[15:15:55:425][    4.140142] platform 5f00000.r5f: configured R5F for IPC-only mode
[15:15:55:426][    4.146785] platform 5f00000.r5f: R5F core initialized in IPC-only mode
[15:15:55:435][    4.155153] platform 5900000.r5f: configured R5F for IPC-only mode
[15:15:55:435][    4.162178] platform 5900000.r5f: R5F core initialized in IPC-only mode
[15:15:55:451][    4.169411] platform 5a00000.r5f: configured R5F for IPC-only mode
[15:15:55:451][    4.176385] platform 5a00000.r5f: R5F core initialized in IPC-only mode
[15:15:55:451]Module                  Size  Used by
[15:15:55:474]ti_k3_r5_remoteproc    28672  0
[15:15:55:474]ti_k3_m4_remoteproc    20480  0
[15:15:55:474]ti_k3_dsp_remoteproc    20480  0
[15:15:55:474]virtio_rpmsg_bus       24576  0
[15:15:55:474]rpmsg_char             20480  0
[15:15:55:474]mount nfs start----------
[15:15:55:487]mount result:---------------------1
[15:15:55:488]Filesystem           1K-blocks      Used Available Use% Mounted on
[15:15:55:489]/dev/root              7390912    420372   6573692   6% /
[15:15:55:504]devtmpfs              13849024         0  13849024   0% /dev
[15:15:55:505]tmpfs                 14767456         8  14767448   0% /run
[15:15:55:505]tmpfs                    51200        40     51160   0% /var/volatile
[15:15:55:519]10.1.14.33:/[    4.235894] SError Interrupt on CPU0, code 0xbf000002 -- SError
[15:15:55:521][    4.235896] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.10.162-g76b3e88d56 #1
[15:15:55:535][    4.235897] Hardware name: Texas Instruments J784S4 EVM (DT)
[15:15:55:535][    4.235898] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[15:15:55:536][    4.235899] pc : _raw_spin_unlock_irqrestore+0x3c/0x48
[15:15:55:551][    4.235900] lr : __irq_put_desc_unlock+0x20/0x50
[15:15:55:551][    4.235901] sp : ffff8000112bbd40
[15:15:55:552][    4.235902] x29: ffff8000112bbd40 x28: 0000000000000000 
[15:15:55:563][    4.235906] x27: ffff8000112bbe30 x26: ffff0008f3918120 
[15:15:55:563][    4.235908] x25: ffff0008f36ee000 x24: ffff8000111219c0 
[15:15:55:570][    4.235911] x23: ffff0008f36ef000 x22: ffff0008f3918120 
[15:15:55:581][    4.235913] x21: 000000000000005e x20: 0000000000000001 
[15:15:55:581][    4.235916] x19: ffff0008f217f800 x18: 0000000000000000 
[15:15:55:583][    4.235918] x17: 0000000000000000 x16: 0000000000000000 
[15:15:55:597][    4.235920] x15: 000000000000038d x14: 000000000000038c 
[15:15:55:597][    4.235922] x13: 0000000000000000 x12: ffff80001180d000 
[15:15:55:599][    4.235924] x11: 0000000000000040 x10: ffff8000111a0750 
[15:15:55:616][    4.235926] x9 : ffff8000111a0748 x8 : ffff0008f2451940 
[15:15:55:616][    4.235929] x7 : 0000000000000000 x6 : ffffffffffffffe0 
[15:15:55:617][    4.235931] x5 : ffff0008f2154680 x4 : 0000000000000000 
[15:15:55:629][    4.235933] x3 : ffffffffffffffe0 x2 : 0000000000000000 
[15:15:55:630][    4.235936] x1 : ffff8000111219c0 x0 : 0000000100000101 
[15:15:55:631][    4.235938] Kernel panic - not syncing: Asynchronous SError Interrupt
[15:15:55:644][    4.235940] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.10.162-g76b3e88d56 #1
[15:15:55:644][    4.235941] Hardware name: Texas Instruments J784S4 EVM (DT)
[15:15:55:647][    4.235942] Call trace:
[15:15:55:647][    4.235943]  dump_backtrace+0x0/0x1a0
[15:15:55:660][    4.235944]  show_stack+0x18/0x68
[15:15:55:660][    4.235945]  dump_stack+0xd0/0x12c
[15:15:55:660][    4.235946]  panic+0x16c/0x334
[15:15:55:664][    4.235947]  nmi_panic+0x8c/0x90
[15:15:55:664][    4.235948]  arm64_serror_panic+0x78/0x84
[15:15:55:678][    4.235949]  do_serror+0x38/0x98
[15:15:55:678][    4.235950]  el1_error+0x90/0x110
[15:15:55:679][    4.235952]  _raw_spin_unlock_irqrestore+0x3c/0x48
[15:15:55:680][    4.235952]  enable_irq+0x5c/0xa0
[15:15:55:695][    4.235954]  am65_cpsw_nuss_tx_poll+0x298/0x3a0
[15:15:55:695][    4.235954]  net_rx_action+0x114/0x380
[15:15:55:696][    4.235955]  efi_header_end+0x120/0x268
[15:15:55:697][    4.235956]  irq_exit+0xc0/0xe0
[15:15:55:707][    4.235957]  __handle_domain_irq+0x68/0xc0
[15:15:55:708][    4.235958]  gic_handle_irq+0x58/0x128
[15:15:55:708][    4.235959]  el1_irq+0xcc/0x180
[15:15:55:711][    4.235960]  arch_cpu_idle+0x18/0x28
[15:15:55:723][    4.235961]  default_idle_call+0x20/0x68
[15:15:55:723][    4.235962]  do_idle+0xc0/0x128
[15:15:55:723][    4.235963]  cpu_startup_entry+0x28/0x60
[15:15:55:724][    4.235964]  rest_init+0xd4/0xe4
[15:15:55:727][    4.235965]  arch_call_rest_init+0x10/0x1c
[15:15:55:743][    4.235966]  start_kernel+0x478/0x4b0
[15:15:55:743][    4.235992] SMP: stopping secondary CPUs
[15:15:55:743][    4.235993] Kernel Offset: disabled
[15:15:55:744][    4.235994] CPU features: 0x28040022,20006008
[15:15:55:755][    4.235995] Memory Limit: none
[15:15:55:755]data1/zhangjing24/nfs
[15:15:55:755]                      20971520   8434688  12536832  40% /mnt/nfs0
[15:15:55:759]mount result:---------------------2
[15:15:55:759]#!/bin/sh
[15:15:55:760]
[15:15:55:772]echo "nfs data write:####################"
[15:15:55:772]rm -rf /mnt/nfs0/tievm/random.bin
[15:15:55:772]dd if=/dev/zero of=/mnt/nfs0/tievm/random.bin bs=1024 count=10240
[15:15:55:775]echo "nfs data write end#################"
[15:15:55:775]
[15:15:55:776]
[15:15:55:787]echo "nfs data read:*********************"
[15:15:55:787]rm -rf /run/test.bin
[15:15:55:788]dd if=/mnt/nfs0/tievm/Image of=/run/test.bin bs=1024
[15:15:55:791]echo "nfs data read end******************"
[15:15:55:791]
[15:15:55:791]sync
[15:15:55:791]
[15:15:55:791]df

3. To resolve this issue, besides the network driver patch, is there anything else that needs to be updated in the SDK 8.6 version, such as the firewall firmware?

4. Please confirm if the SError issue does not recur in the SDK 9.2 version. Was the replication environment set up using the tiny rootfs provided before? When we tested with the tiny rootfs, the SDK 8.6 can stably reproduce the issue, and the frequency of reproduction is quite high.

Hi Li 

We have tested a normal NFS boot on 9.1,9.2 SDK with default sdk rootfs not tiny one without this patch we were able to see the firewall exception.

TIFS traces:

A72 logs:

After applying the shared patch we were not able to reproduce the issue .

We will also keep a device under test with 8.6 SDK to see if issue persists on that.Will update you about our finding on 8.6.

Regards
Diwakar

Hi Li 

Also i assume the above patches you have tested on the EVM setup have you tested the same on your hardware setup as well where on board booting another?

Regards
Diwakar

Hi Divakar

Yes, we have also tested it on the product board, and the issue can be reproduced as well.

Best Regards

Hi Li 

One more thing when you are not doing fixed link changes in the dtsi is the issue reproducable ?

We are able to make the setup somewhat similar to your but not seeing any issue so far , The diffrence between your setup and our setup on EVM are:

• We are not using fixed link 
• NFS script changes

  2 
  3 PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
  4 
  5 insmod /lib/modules/5.10.162-g76b3e88d56/kernel/drivers/rpmsg/rpmsg_char.ko
  6 insmod /lib/modules/5.10.162-g76b3e88d56/kernel/drivers/rpmsg/virtio_rpmsg_bus.ko
  7 
  8 insmod /lib/modules/5.10.162-g76b3e88d56/kernel/drivers/remoteproc/ti_k3_dsp_remoteproc.ko
  9 insmod /lib/modules/5.10.162-g76b3e88d56/kernel/drivers/remoteproc/ti_k3_m4_remoteproc.ko
 10 insmod /lib/modules/5.10.162-g76b3e88d56/kernel/drivers/remoteproc/ti_k3_r5_remoteproc.ko
 11 
 12 lsmod
 13 
 14 echo "mount nfs start----------"
 15 ifconfig eth0 hw ether 68:e7:4a:08:15:5f
 16 ifconfig eth0 up
 17 #sleep 4
 18 #udhcpc -i eth0
 19 ifconfig eth0 10.24.68.42  netmask 255.255.254.0
 20 
 21 route add default gw 10.24.68.254
 22 mkdir -p /mnt/nfs0
 23 mount -t nfs -o nolock 10.24.69.226:/home/<unmae>/Desktop/j7ahp/ti-processor-sdk-linux-j784s4-evm-08_06_00_12/targetNFS /mnt/nfs0
 24 
 25 echo "mount result:---------------------1"
 26 df
 27 echo "mount result:---------------------2"
 28 
 29 cat /mnt/nfs0/run_nfs_evm.sh
 30 
 31 dd if=/dev/zero of=/mnt/nfs0/test_zero.bin bs=1M count=10
 32 sync
 33 
 34 reboot
 35 

Regards
Diwakar

Hi Divakar

The issue of fixed-link has been explained before. Our product utilizes two VH chips connected directly through a MAC-MAC network. Therefore, the device tree network needs to be configured in fixed-link mode.

On the EVM board, without the fixed-link mode set, we did not encounter the SError issue. However, after enabling the fixed-link mode, the Linux kernel boot speed is accelerated due to skipping the PHY auto-negotiation process, and the subsequent driver module loading also takes place earlier.

We suspect that during the network data communication process, the concurrent loading of the rpmsg and remoteproc drivers led to the occurrence of the SError issue.

Therefore, it is still recommended to configure it in fixed-link mode for testing. The modification to the nfs script has no impact.

Best Regards

Alex

Hi Alex,

On the EVM, when we make any of the two CPSW ports as fixed-link, We will not be able to get link working as the phy is not boot-strapped.

When you re-created the issue on EVM, were you able to ping and transfer the data or the issue was seen even without making the link working?

Regards,
Tanmay

Hi Tanmay,

Our EVM board is connected to a switch, and after the MCU RGMII port is set to fixed-link mode, it is able to ping each other with the NFS server's PC. NFS can also be mounted successfully. The df command in the nfs_test.sh script can confirm the NFS mount status. As shown below.

[15:15:55:755]data1/zhangjing24/nfs
[15:15:55:755] 20971520 8434688 12536832 40% /mnt/nfs0

You can ask Divakar how the SError issue was reproduced on the EVM board before.

Best Regards

Alex

Hi Alex,

Thanks for the info.

In an earlier response I saw your hypothesis that this might have have something to do with the fixed-link taking less time to link up. But still the issue is seen only when you run your nfs script. Is this assumption correct? If this is correct, then it might be possible that there could be some race conditions seen due to the rproc or rpmsg modules, both of which use the same interrupt router as udma.

Have you tried to not load the modules before you mount the nfs partitions. Can you see if the error is reproduced after this as well.

Regards,
Tanmay

Hi Tanmay,

We compiled the rpmsg and remoteproc drivers into the kernel and avoided network transmission and driver loading, and there were no issues encountered when executing nfs_test.sh on the EVM board. However, using the same strategy, we were still able to reproduce the SError issue on our product, which occurs approximately during network data transmission when the DSP service performs DMA initialization.

The core issue lies in the abnormal interception by the firewall, as previously described. Is it possible to analyze whether there is a problem with the firewall component?

Best Regards

Alex