Hello TI Support team,
We've also observed these mailbox timeout issues, reported in the related thread, for TI SCI FW which causes a system crash. This is reproducible on doing multiple reboots from Linux:
[ 21.285731] ti-sci 44083000.dmsc: Mbox timedout in resp(caller: ti_sci_cmd_set_event_irq+0x44/0x50) [ 21.285737] ti-sci 44083000.dmsc: Mbox timedout in resp(caller: k3_r5_rproc_start+0xb0/0x298 [ti_k3_r5_remoteproc]) [ 21.285744] ti-sci 44083000.dmsc: Mbox send fail -110 [ 21.294775] ti-sci 44083000.dmsc: Mbox send fail -110 [ 21.306236] platform 5e00000.r5f: ti-sci processor set_config failed: -110 [ 21.310228] ti_sci_allocate_event_irq: Event allocation failed from src = 211, index = 448, to dst = 14,irq = 92,via ia_id = 209, vint = 50,global event = 64, status_bit = 0 [ 21.310255] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000030 [ 21.316646] remoteproc remoteproc6: can't start rproc 5e00000.r5f: -110 [ 21.322853] Mem abort info: [ 21.355745] ESR = 0x96000005 [ 21.358796] Exception class = DABT (current EL), IL = 32 bits [ 21.364703] SET = 0, FnV = 0 [ 21.367750] EA = 0, S1PTW = 0 [ 21.370884] Data abort info: [ 21.373757] ISV = 0, ISS = 0x00000005 [ 21.377579] CM = 0, WnR = 0 [ 21.380539] user pgtable: 64k pages, 48-bit VAs, pgdp = (____ptrval____) [ 21.387228] [0000000000000030] pgd=0000000000000000, pud=0000000000000000 [ 21.394004] Internal error: Oops: 96000005 [#1] PREEMPT SMP [ 21.399560] Modules linked in: af_key(+) xfrm_algo rfkill pvrsrvkm(O+) rpmsg_char ti_am335x_adc(+) kfifo_buf cdns3(+) omap_rng(+) rng_core pruss(+) roles udc_core crc32_ce crct10dif_ce j721e_cpsw_virt6 [ 21.448651] Process systemd-udevd (pid: 185, stack limit = 0x(____ptrval____)) [ 21.455853] CPU: 1 PID: 185 Comm: systemd-udevd Tainted: G O 4.19.193 #1 [ 21.463833] Hardware name: Texas Instruments K3 J721E SoC (DT) [ 21.469648] pstate: 60000005 (nZCv daif -PAN -UAO) [ 21.474429] pc : ti_sci_inta_register_event+0xd4/0x1c8 [ 21.479550] lr : ti_sci_inta_register_event+0xd4/0x1c8 [ 21.484670] sp : ffff00000f84f840 [ 21.487969] x29: ffff00000f84f840 x28: 0000000000000000 [ 21.493265] x27: 0000000000000004 x26: 0000000000000000 [ 21.498560] x25: 00000000000000d3 x24: 00000000000001c0 [ 21.503854] x23: ffff00000f84f8a8 x22: ffff000008e93000 [ 21.509148] x21: ffff800841d26b80 x20: 0000000000000000 [ 21.514442] x19: ffff800841d27500 x18: ffff000008eacbe0 [ 21.519736] x17: 0000000000000fff x16: 0000000000000000 [ 21.525030] x15: 0000000000000000 x14: ffff8008208b6c40 [ 21.530324] x13: ffff8008208b6b38 x12: 0000000000000003 [ 21.535618] x11: ffff8008208b6b60 x10: 0000000000000040 [ 21.540912] x9 : ffff000008eacbe0 x8 : ffff800820000248 [ 21.541722] ti-sci 44083000.dmsc: Mbox timedout in resp(caller: ti_sci_cmd_get_clock+0x34/0x40) [ 21.546206] x7 : ffff800820000270 x6 : 0000000000000000 [ 21.546208] x5 : ffff800820000248 x4 : 0000000000000000 [ 21.546210] x3 : 0000000000000000 x2 : 0000000000000000 [ 21.546212] x1 : 0000000000000028 x0 : 0000000000000000 [ 21.546215] Call trace: [ 21.546219] ti_sci_inta_register_event+0xd4/0x1c8 [ 21.554904] ti-sci 44083000.dmsc: Mbox send fail -110 [ 21.560184] k3_nav_udmax_rx_get_irq+0x90/0xf8 [ 21.560193] virt_cpsw_nuss_probe+0x598/0xb30 [j721e_cpsw_virt_mac] [ 21.560197] platform_drv_probe+0x50/0xa0 [ 21.570939] cdns-usb3 6000000.usb: DRD version v1 (ID: 0004024e, rev: 00000200) [ 21.576078] really_probe+0x220/0x2c0 [ 21.576080] driver_probe_device+0x58/0x108 [ 21.576082] __driver_attach+0xec/0xf0 [ 21.576083] bus_for_each_dev+0x74/0xc8 [ 21.576087] driver_attach+0x20/0x28 [ 21.580151] omap_rng 4e10000.trng: Unable to enable the clk: -110 [ 21.583295] bus_add_driver+0x1b8/0x228 [ 21.583297] driver_register+0x60/0x110 [ 21.583300] __platform_driver_register+0x40/0x48 [ 21.583306] virt_cpsw_nuss_driver_init+0x20/0x10000 [j721e_cpsw_virt_mac] [ 21.583310] do_one_initcall+0x48/0x280 [ 21.583313] do_init_module+0x5c/0x1c4 [ 21.583316] load_module+0x1be0/0x2018 [ 21.583320] __se_sys_finit_module+0xc0/0xd8 [ 21.595431] ------------[ cut here ]------------ [ 21.599032] __arm64_sys_finit_module+0x14/0x20 [ 21.603022] clk:264:1 already disabled [ 21.610310] el0_svc_common+0x6c/0x188 [ 21.613976] WARNING: CPU: 0 PID: 179 at /opt/flex/jenkins/mel_elm_async5/myproject/workspace/build_j7-evm-mel_dev_wo_nw_mel_8948_mel_12141/tmp/work-shared/j7-evm-mel/kernel-source/drivers/clk/clk.c:848 [ 21.618120] el0_svc_handler+0x2c/0x88 [ 21.621853] Modules linked in: af_key(+) xfrm_algo rfkill pvrsrvkm(O+) rpmsg_char ti_am335x_adc(+) kfifo_buf cdns3(+) omap_rng(+) rng_core pruss roles udc_core crc32_ce crct10dif_ce j721e_cpsw_virt_ma6 [ 21.625673] el0_svc+0x8/0xe8 [ 21.629233] CPU: 0 PID: 179 Comm: systemd-udevd Tainted: G O 4.19.193 #1 [ 21.635304] Code: 340006e0 2a1403e1 aa1303e0 97f16bf0 (f9401800) [ 21.639123] Hardware name: Texas Instruments K3 J721E SoC (DT) [ 21.642943] ---[ end trace 98ca50d865b031f1 ]--- [ 21.647627] pstate: 40000085 (nZcv daIf -PAN -UAO) [ 21.656782] ti-sci 44083000.dmsc: Mbox timedout in resp(caller: k3_ringacc_ring_cfg+0x1ec/0x3d8) [ 21.658300] pc : clk_core_disable+0x260/0x298 [ 21.658303] lr : clk_core_disable+0x260/0x298 [ 21.658304] sp : ffff00000e0cf9b0 [ 21.658305] x29: ffff00000e0cf9b0 x28: ffff000010330000 [ 21.658308] x27: 0000000000000100 x26: ffff00000e0cfde0 [ 21.658310] x25: ffff000002020190 x24: ffff000002020050 [ 21.658312] x23: ffff800844cd3200 x22: 0000000000000000 [ 21.658314] x21: ffff800843594010 x20: ffff800841d03f00 [ 21.658316] x19: ffff800841d03f00 x18: ffffffffffffffff [ 21.658318] x17: 0000000000000001 x16: 0000000000000007 [ 21.658319] x15: ffff000008e93648 x14: ffff000088fc68df [ 21.658324] x13: ffff000008fc68ed x12: ffff000008eac000 [ 21.662077] ti-sci 44083000.dmsc: RM_RA:Mbox config send fail -110 [ 21.665790] x11: 0000000005f5e0ff x10: ffff000008e93680 [ 21.665792] x9 : 00000000ffffffd0 x8 : ffff0000085a20e0 [ 21.665794] x7 : 5f796c6c61657220 x6 : 0000000000000016 [ 21.665797] x5 : 0000000000000000 x4 : ffff80087fa91818 [ 21.673984] k3-ringacc 3c000000.ringacc: TISCI config ring fail (-110) ring_idx 319 [ 21.674651] x3 : ffff80087fa91818 x2 : 0000000000000007 [ 21.745996] cdns-usb3 6400000.usb: DRD version v1 (ID: 0004024e, rev: 00000200) [ 21.759542] x1 : df6be58aa1b82000 x0 : 0000000000000000 [ 21.759545] Call trace: [ 21.759552] clk_core_disable+0x260/0x298 [ 21.759557] clk_core_disable_lock+0x20/0x38 [ 21.923621] clk_disable+0x1c/0x30 [ 21.927014] omap_rng_probe+0x178/0x484 [omap_rng] [ 21.931790] platform_drv_probe+0x50/0xa0 [ 21.935783] really_probe+0x220/0x2c0 [ 21.939431] driver_probe_device+0x58/0x108 [ 21.943598] __driver_attach+0xec/0xf0 [ 21.947331] bus_for_each_dev+0x74/0xc8 [ 21.951152] driver_attach+0x20/0x28 [ 21.954713] bus_add_driver+0x1b8/0x228 [ 21.958534] driver_register+0x60/0x110 [ 21.962355] __platform_driver_register+0x40/0x48 [ 21.967044] omap_rng_driver_init+0x20/0x10000 [omap_rng] [ 21.972427] do_one_initcall+0x48/0x280 [ 21.976249] do_init_module+0x5c/0x1c4 [ 21.979985] load_module+0x1be0/0x2018 [ 21.983720] __se_sys_finit_module+0xc0/0xd8 [ 21.987974] __arm64_sys_finit_module+0x14/0x20 [ 21.992489] el0_svc_common+0x6c/0x188 [ 21.996223] el0_svc_handler+0x2c/0x88 [ 21.999958] el0_svc+0x8/0xe8 [ 22.002912] ---[ end trace 98ca50d865b031f2 ]--- [ 22.007661] ------------[ cut here ]------------ [ 22.009745] ti-sci 44083000.dmsc: Mbox timedout in resp(caller: ti_sci_cmd_get_device+0x14/0x20) [ 22.012282] clk:264:1 already unprepared [ 22.021492] ti-sci 44083000.dmsc: Mbox send fail -110 [ 22.025124] WARNING: CPU: 0 PID: 179 at /opt/flex/jenkins/mel_elm_async5/myproject/workspace/build_j7-evm-mel_dev_wo_nw_mel_8948_mel_12141/tmp/work-shared/j7-evm-mel/kernel-source/drivers/clk/clk.c:708 [ 22.030413] rtc-ds1307 3-006f: write error -110 [ 22.050541] Modules linked in: af_key(+) xfrm_algo rfkill pvrsrvkm(O+) rpmsg_char ti_am335x_adc(+) kfifo_buf cdns3 omap_rng(+) rng_core pruss roles udc_core crc32_ce crct10dif_ce j721e_cpsw_virt_mac(+6 [ 22.055094] rtc-ds1307 3-006f: read error -22 [ 22.103618] CPU: 0 PID: 179 Comm: systemd-udevd Tainted: G D W O 4.19.193 #1 [ 22.103620] Hardware name: Texas Instruments K3 J721E SoC (DT) [ 22.121753] pstate: 40000005 (nZcv daif -PAN -UAO) [ 22.126531] pc : clk_core_unprepare+0x228/0x258 [ 22.131046] lr : clk_core_unprepare+0x228/0x258 [ 22.135560] sp : ffff00000e0cf9d0 [ 22.138860] x29: ffff00000e0cf9d0 x28: ffff000010330000 [ 22.144155] x27: 0000000000000100 x26: ffff00000e0cfde0 [ 22.149449] x25: ffff000002020190 x24: ffff000002020050 [ 22.154744] x23: ffff800844cd3200 x22: 0000000000000000 [ 22.160038] x21: ffff800843594010 x20: 00000000ffffff92 [ 22.165333] x19: ffff800841d03f00 x18: 0000000000000400 [ 22.170628] x17: 0000000000000000 x16: 0000000000000000 [ 22.175923] x15: 0000000000000400 x14: 0000000000000400 [ 22.181216] x13: 0000000000000000 x12: 0000000000000001 [ 22.186510] x11: 0000000000000000 x10: 00000000000009d0 [ 22.191804] x9 : ffff00000e0cf6b0 x8 : ffff800842abaf30 [ 22.197099] x7 : ffff800843814e00 x6 : ffff80087fa91820 [ 22.202393] x5 : ffff80087fa91820 x4 : 0000000000000000 [ 22.207686] x3 : ffff80087faa8748 x2 : ffff80087fa91820 [ 22.212980] x1 : df6be58aa1b82000 x0 : 0000000000000000 [ 22.218274] Call trace: [ 22.220710] clk_core_unprepare+0x228/0x258 [ 22.224878] clk_unprepare+0x28/0x40 [ 22.228443] omap_rng_probe+0x180/0x484 [omap_rng] [ 22.233220] platform_drv_probe+0x50/0xa0 [ 22.237215] really_probe+0x220/0x2c0 [ 22.240862] driver_probe_device+0x58/0x108 [ 22.245029] __driver_attach+0xec/0xf0 [ 22.248763] bus_for_each_dev+0x74/0xc8 [ 22.252583] driver_attach+0x20/0x28 [ 22.256144] bus_add_driver+0x1b8/0x228 [ 22.259964] driver_register+0x60/0x110 [ 22.263786] __platform_driver_register+0x40/0x48 [ 22.268474] omap_rng_driver_init+0x20/0x10000 [omap_rng] [ 22.273857] do_one_initcall+0x48/0x280 [ 22.277679] do_init_module+0x5c/0x1c4 [ 22.281414] load_module+0x1be0/0x2018 [ 22.285148] __se_sys_finit_module+0xc0/0xd8 [ 22.289402] __arm64_sys_finit_module+0x14/0x20 [ 22.293918] el0_svc_common+0x6c/0x188 [ 22.297652] el0_svc_handler+0x2c/0x88 [ 22.301386] el0_svc+0x8/0xe8 [ 22.304340] ---[ end trace 98ca50d865b031f3 ]--- [ 22.309001] omap_rng 4e10000.trng: initialization failed. [ 22.314456] omap_rng: probe of 4e10000.trng failed with error -110 [ 22.320637] ti-sci 44083000.dmsc: Mbox timedout in resp(caller: ti_sci_cmd_get_device+0x14/0x20) [ 22.320644] ti-sci 44083000.dmsc: Mbox send fail -110 [ 22.322001] NET: Registered protocol family 15 [ 22.341930] remoteproc remoteproc7: b034000.pru is available [ 22.348292] pru-rproc b034000.pru: PRU rproc node pru@b034000 probed successfully [ 22.356071] remoteproc remoteproc8: b004000.rtu is available [ 22.361782] pru-rproc b004000.rtu: PRU rproc node rtu@b004000 probed successfully [ 22.369441] remoteproc remoteproc9: b00a000.txpru is available [ 22.375320] pru-rproc b00a000.txpru: PRU rproc node txpru@b00a000 probed successfully [ 22.383317] remoteproc remoteproc10: b038000.pru is available [ 22.389105] pru-rproc b038000.pru: PRU rproc node pru@b038000 probed successfully [ 22.396726] remoteproc remoteproc11: b006000.rtu is available [ 22.402501] pru-rproc b006000.rtu: PRU rproc node rtu@b006000 probed successfully [ 22.410086] remoteproc remoteproc12: b00c000.txpru is available [ 22.416040] pru-rproc b00c000.txpru: PRU rproc node txpru@b00c000 probed successfully [ 22.424018] remoteproc remoteproc13: b134000.pru is available [ 22.429791] pru-rproc b134000.pru: PRU rproc node pru@b134000 probed successfully [ 22.437445] remoteproc remoteproc14: b104000.rtu is available [ 22.443226] pru-rproc b104000.rtu: PRU rproc node rtu@b104000 probed successfully [ 22.450838] remoteproc remoteproc15: b10a000.txpru is available [ 22.456784] pru-rproc b10a000.txpru: PRU rproc node txpru@b10a000 probed successfully [ 22.464751] remoteproc remoteproc16: b138000.pru is available [ 22.470520] pru-rproc b138000.pru: PRU rproc node pru@b138000 probed successfully [ 22.478230] remoteproc remoteproc17: b106000.rtu is available [ 22.484027] pru-rproc b106000.rtu: PRU rproc node rtu@b106000 probed successfully [ 22.491643] remoteproc remoteproc18: b10c000.txpru is available [ 22.497602] pru-rproc b10c000.txpru: PRU rproc node txpru@b10c000 probed successfully [ 22.643617] usbcore: registered new interface driver usbfs [ 22.649132] usbcore: registered new interface driver hub [ 22.655271] usbcore: registered new device driver usb [ 22.686036] xhci-hcd xhci-hcd.2.auto: xHCI Host Controller [ 22.691536] xhci-hcd xhci-hcd.2.auto: new USB bus registered, assigned bus number 1 [ 22.699235] Initializing XFRM netlink socket [ 22.705393] xhci-hcd xhci-hcd.2.auto: hcc params 0x200073c9 hci version 0x100 quirks 0x0000000000010010 [ 22.714864] xhci-hcd xhci-hcd.2.auto: irq 324, io mem 0x06410000 [ 22.721038] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19 [ 22.729329] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 22.736555] usb usb1: Product: xHCI Host Controller [ 22.741441] usb usb1: Manufacturer: Linux 4.19.193 xhci-hcd [ 22.747009] usb usb1: SerialNumber: xhci-hcd.2.auto [ 22.752621] hub 1-0:1.0: USB hub found [ 22.756407] hub 1-0:1.0: 1 port detected [ 22.760498] xhci-hcd xhci-hcd.2.auto: xHCI Host Controller [ 22.765988] xhci-hcd xhci-hcd.2.auto: new USB bus registered, assigned bus number 2 [ 22.773646] xhci-hcd xhci-hcd.2.auto: Host supports USB 3.0 SuperSpeed [ 22.780194] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM. [ 22.788334] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 4.19 [ 22.796588] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 22.803803] usb usb2: Product: xHCI Host Controller [ 22.808672] usb usb2: Manufacturer: Linux 4.19.193 xhci-hcd [ 22.814239] usb usb2: SerialNumber: xhci-hcd.2.auto [ 22.819344] hub 2-0:1.0: USB hub found [ 22.823111] hub 2-0:1.0: 1 port detected [ 23.017721] usb 1-1: new high-speed USB device number 2 using xhci-hcd [ 23.169990] usb 1-1: New USB device found, idVendor=0451, idProduct=8142, bcdDevice= 1.00 [ 23.178155] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=1 [ 23.185279] usb 1-1: SerialNumber: A4000069DE36 [ 23.190370] hub 1-1:1.0: USB hub found [ 23.194143] hub 1-1:1.0: 4 ports detected [ 28.125721] random: crng init done [ 28.129113] random: 7 urandom warning(s) missed due to ratelimiting [ 31.525728] ti-sci 44083000.dmsc: Mbox timedout in resp(caller: ti_sci_cmd_get_device_resets+0x1c/0x28) [ 31.525732] ti-sci 44083000.dmsc: Mbox timedout in resp(caller: ti_sci_cmd_get_device+0x14/0x20) [ 31.525738] ti-sci 44083000.dmsc: Mbox send fail -110 [ 31.535114] ti-sci 44083000.dmsc: Mbox send fail -110 [ 31.535119] platform 5e00000.r5f: local-reset assert failed, ret = -110 [ 31.545046] pvrsrvkm 4e20000000.gpu: adding gpu_0 device link failed! [ 31.548920] platform 5e00000.r5f: unable to disable cores, ret = -110 ERROR: Unhandled External Abort received on 0x80000001 at EL3! ERROR: exception reason=0 syndrome=0xbf000000 PANIC in EL3. x30 = 0x0000000070004644 x0 = 0x0000000000000000 x1 = 0x0000000000000060 x2 = 0x0000000000000060 x3 = 0x0000000000000000 x4 = 0x0000000000000062 x5 = 0x0000000000000008 x6 = 0xffff000010b2f6e0 x7 = 0xfefefefefefefefe x8 = 0xffffffffffffffff x9 = 0x0000000041023060 x10 = 0x000000000000073d x11 = 0x0101010101010101 x12 = 0x0000000000000000 x13 = 0xffff000002500090 x14 = 0x0000000000000015 x15 = 0x0000000000000006 x16 = 0x0000000000000000 x17 = 0x0000000000000000 x18 = 0xffff800843e69740 x19 = 0x0000000000000000 x20 = 0x00000000bf000000 x21 = 0xffff000008e93000 x22 = 0xffff0000024ff000 x23 = 0x0000000000000000 x24 = 0xffff800844326600 x25 = 0xffff800844326000 x26 = 0xffff00000246e000 x27 = 0x0000000000000100 x28 = 0xffff000012910000 x29 = 0x000000007000b620 scr_el3 = 0x000000000000073d sctlr_el3 = 0x0000000030cd183f cptr_el3 = 0x0000000000000000 tcr_el3 = 0x0000000080803520 daif = 0x00000000000002c0 mair_el3 = 0x00000000004404ff spsr_el3 = 0x00000000a0000005 elr_el3 = 0xffff0000023dc120 ttbr0_el3 = 0x000000007000f9e0 esr_el3 = 0x00000000bf000000 far_el3 = 0x0000000000000000 spsr_el1 = 0x0000000040000005 elr_el1 = 0xffff000008118c68 spsr_abt = 0x0000000000000000 spsr_und = 0x0000000000000000 spsr_irq = 0x0000000000000000 spsr_fiq = 0x0000000000000000 sctlr_el1 = 0x0000000034d5d91d actlr_el1 = 0x0000000000000000 cpacr_el1 = 0x0000000000300000 csselr_el1 = 0x0000000000000000 sp_el1 = 0xffff000010b2f7b0 esr_el1 = 0x0000000056000000 ttbr0_el1 = 0x00000008c17db000 ttbr1_el1 = 0x0842000081090000 mair_el1 = 0x0000bbff440c0400 amair_el1 = 0x0000000000000000 tcr_el1 = 0x00000034f5507510 tpidr_el1 = 0x0000800876c90000 tpidr_el0 = 0x0000ffffbb200700 tpidrro_el0 = 0x0000000000000000 par_el1 = 0x0000000000000000 mpidr_el1 = 0x0000000080000001 afsr0_el1 = 0x0000000000000000 afsr1_el1 = 0x0000000000000000 contextidr_el1 = 0x0000000000000000 vbar_el1 = 0xffff000008081800 cntp_ctl_el0 = 0x0000000000000005 cntp_cval_el0 = 0x00000001c5e8bd93 cntv_ctl_el0 = 0x0000000000000000 cntv_cval_el0 = 0x0000000000000000 cntkctl_el1 = 0x00000000000000d6 sp_el0 = 0x000000007000b620 isr_el1 = 0x0000000000000040 dacr32_el2 = 0x0000000000000000 ifsr32_el2 = 0x0000000000000000 cpuectlr_el1 = 0x0000001b00000040 cpumerrsr_el1 = 0x0000000000000000 l2merrsr_el1 = 0x0000000000000000
Was this ever triaged and fixed? We can not use a later SDK at this point, so it would be nice to have a fix that we can incorporate in 06_02_00_07 SDK.
Thanks