• State Resolved
  • Locked Locked
  • Replies 10 replies
  • Subscribers 99 subscribers
  • Views 529 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

DRA76P: RTOS/DR76P: L3 Custom Error

Woj
Prodigy 235 points
Part Number: DRA76P

Hello TI team,

There always be consecutively "L3 Custom Error" issues when I was trying to run the vis_sdk on our custom board, the dmesg log is:

[ 33.274743] WARNING: CPU: 0 PID: 0 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x318/0x400()
[ 33.274748] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER2_P3 (Read): Data Access in User mode during Functional access
[ 33.274752] Modules linked in: cmemk(O) pvrsrvkm(O) input_polldev ti_fpd3_serdes sd_mod usb_storage scsi_mod
[ 33.274772] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O 4.4.117-gb7a70b4-dirty #4
[ 33.274776] Hardware name: Generic DRA74X (Flattened Device Tree)
[ 33.274779] Backtrace:
[ 33.274787] [<c020c80c>] (dump_backtrace) from [<c020ca08>] (show_stack+0x18/0x1c)
[ 33.274790] r6:60070193 r5:c1241e94 r4:00000000 r3:dc8ba675
[ 33.274803] [<c020c9f0>] (show_stack) from [<c0b16720>] (dump_stack+0x94/0xa8)
[ 33.274808] [<c0b1668c>] (dump_stack) from [<c0232000>] (warn_slowpath_fmt+0xa8/0xec)
[ 33.274812] r6:00000093 r5:c058ec58 r4:c120394c r3:dc8ba675
[ 33.274825] [<c0231f5c>] (warn_slowpath_fmt) from [<c058ec58>] (l3_interrupt_handler+0x318/0x400)
[ 33.274828] r3:eea3a180 r2:c0e2ebcc
[ 33.274835] r7:00000002 r6:eea3a310 r5:c0e2eb0c r4:80080003
[ 33.274848] [<c058e940>] (l3_interrupt_handler) from [<c029be58>] (handle_irq_event_percpu+0xc0/0x2c0)
[ 33.274852] r10:c0c03250 r9:c1200010 r8:00000000 r7:00000017 r6:c120e9ec r5:ee900c60
[ 33.274864] r4:eea3a680
[ 33.274871] [<c029bd98>] (handle_irq_event_percpu) from [<c029c098>] (handle_irq_event+0x40/0x64)
[ 33.274874] r10:c1200000 r9:c1203a4c r8:ee824000 r7:00000001 r6:c120e9ec r5:ee900c60
[ 33.274886] r4:ee900c00
[ 33.274893] [<c029c058>] (handle_irq_event) from [<c029f950>] (handle_fasteoi_irq+0xc0/0x1c4)
[ 33.274897] r6:c120e9ec r5:ee900c60 r4:ee900c00 r3:00000000
[ 33.274910] [<c029f890>] (handle_fasteoi_irq) from [<c029b740>] (__handle_domain_irq+0x74/0xcc)
[ 33.274913] r7:00000001 r6:00000000 r5:00000000 r4:c1085518
[ 33.274926] [<c029b6cc>] (__handle_domain_irq) from [<c02014d8>] (gic_handle_irq+0x40/0x7c)
[ 33.274929] r8:fa213000 r7:fa212000 r6:c1201ee8 r5:fa21200c r4:c12048c0 r3:c1201ee8
[ 33.274943] [<c0201498>] (gic_handle_irq) from [<c020d500>] (__irq_svc+0x40/0x74)
[ 33.274947] Exception stack(0xc1201ee8 to 0xc1201f30)
[ 33.274952] 1ee0: 00000001 00000000 fe600000 00000000 c12039ec c0c03250
[ 33.274957] 1f00: 00000000 c1291878 00000000 c1203a4c c1200000 c1201f44 c1201f24 c1201f38
[ 33.274961] 1f20: c02234e4 c0208bfc 60070013 ffffffff
[ 33.274965] r8:00000000 r7:c1201f1c r6:ffffffff r5:60070013 r4:c0208bfc r3:c02234e4
[ 33.274981] [<c0208bd4>] (arch_cpu_idle) from [<c0288b30>] (default_idle_call+0x28/0x34)
[ 33.274987] [<c0288b08>] (default_idle_call) from [<c0288e8c>] (cpu_startup_entry+0x350/0x408)
[ 33.274993] [<c0288b3c>] (cpu_startup_entry) from [<c0b18944>] (rest_init+0x90/0x94)
[ 33.274996] r7:00000000
[ 33.275003] [<c0b188b4>] (rest_init) from [<c1000e64>] (start_kernel+0x440/0x494)
[ 33.275006] r4:c129c050 r3:dc8ba675
[ 33.275015] [<c1000a24>] (start_kernel) from [<80008090>] (0x80008090)

and finally a "iommu fault" occurred:

[ 34.917400] omap-iommu 40d01000.mmu: iommu fault: da 0x8e1001c0 flags 0x0
[ 34.924227] remoteproc0: crash detected in 40800000.dsp: type mmufault
[ 34.930881] omap-iommu 40d01000.mmu: 40d01000.mmu: errs:0x00000002 da:0x8e1001c0 pgd:0xedfe6384 *pgd:px00000000

Looking into both the RadioApp and the RTOS(/sys/kernel/debug/remoteproc/remoteproc0/trace0_last) log,

it seems that the "L3 Custom Error" is caused by calling GIO_create() to create "/mcasp5" for below reasons:

1. The RadioApp is blocked from waiting the remote core response after sending the IOA_INIT_REQ command.

01-01 00:00:04.473 273 286 I AUDIOMGR: [AMGR_CtrlThread 998] AMGR_sendIoaInit IOA_SOURCE_0
01-01 00:00:04.473 273 286 I AUDIOMGR: [AMGR_sendIoaInit 945] AMGR_sendMcASPInit
01-01 00:00:04.473 273 286 I AUDIOMGR: [AMGR_sendMcASPInit 639] AMGR_sendCmdMsg IOA_INIT_REQ
01-01 00:00:04.473 273 286 I AUDIOMGR: [AMGR_sendMcASPInit 655] osal_msg_wait...

2. The RTOS is blocked after calling GIO_create().

[ 2.538] [IOA_dataHandler][IOA_Sink_0]:
[ 2.539] [IOA_dataHandler][IOA_Sink_0]: IOA_initWait
[ 2.539] [IOA_dataHandler][IOA_Source_0]:
[ 2.539] [IOA_dataHandler][IOA_Source_0]: IOA_initWait
[ 2.543] [IOA_initWait]: IOA_process: msgId 2304
[ 2.543] [IOA_process]: IOA_INIT_REQ
[ 2.543] [IOA_process]: IOA_INIT_REQ IOA_InitMcAsp
[ 2.543] [IOA_InitMcAsp]: IOA_createMCASP: mcaspName: /mcasp5
[ 2.543] [IOA_createMCASP]: GIO_create: /mcasp5

My question is what's the root cause of the "L3 Custom Error" issue, and how to debug/resolve it?

 

  • 0
    TI__Mastermind 42515 points

    Hi Woj,

    Which version is the SDK/kernel you use?

    Also you can check this document for some directions about L3 error:

    Regards,

    Yordan

  • 0 in reply to Yordan Kamenov
    Prodigy 235 points

    Hello Yordan,

    Thanks for your information.

    The version I used is Android 6AO.1.1 + Linux kernel 4.4.117.

    According to the document, there are mainly two directions of the L3 error: due to access clock-gated module or due to wrong access address.

    However, I don't know much about the underlying implement of the vis_sdk，especially the RTOS part.

    All I did is modify the memory configurations of the vis_sdk base on our custom hardware to make sure the remote core image be loaded and run properly.

    Can you please provide some more specific steps to investigate this issue? Or do you need any more information/logs for analysis?

  • 0 in reply to Yordan Kamenov
    Prodigy 235 points

    And the vis_sdk version is 01.50.23.01.

    As I said in the first post, the logs indicate that the RTOS is blocked when calling GIO_create() to create device "/mcasp5" every time, perhaps the L3 custom error is caused in somehow by GIO_create() ?

  • 0 in reply to Woj
    TI__Expert 8250 points

    Hi

    As i understand you are using Android SDK release for the SW running on A15 core.

    Can you clarify on the use case that you are trying to bringup - does it involve early loading of the remote cores from uboot(early boot/late-attach approach)

  • 0 in reply to Srirama Govindarajan
    Prodigy 235 points

    Hello Srirama,

    Yes, Android SDK running on A15, and vis_sdk(Audio post process engine, APPE) running on DSP core.

    No early loading involved, the remote core image is loaded and booted by remoteproc driver of Linux kernel.

  • 0 in reply to Yordan Kamenov
    Prodigy 235 points

    By investigating into the McASP BSP driver, I found the "L3 Custom Error" is caused by the McASPGlobalCtlGet() & McASPRxGlobalCtlGet() call. Inside function mcaspBitSetGblRCtl(), there is a while() loop which calls McASPGlobalCtlGet() to query MCASP_GBLCTLR register status, this is the reason why we see too many "L3 Custom Error" are shown in the dmesg log, which is also shared in my first post:

    "There always be consecutively "L3 Custom Error" issues when I was trying to run the vis_sdk on our custom board"

    When I reduce the while() loop to only three times, the "L3 Custom Error" are shown only three times correspondingly in the dmesg log. Plus the other two McASPGlobalCtlGet() calls and one McASPRxGlobalCtlGet() call before the while() loop, there are total six "L3 Custom Error" should be caused. The dmesg log and RTOS trace log proved this.

    abstracted dmesg log:

    Line 938: [ 4.613913] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER2_P3 (Idle): Data Access in User mode during Functional access
    Line 972: [ 4.626946] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER2_P3 (Idle): Data Access in User mode during Functional access
    Line 1006: [ 4.627276] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER2_P3 (Read): Data Access in User mode during Functional access
    Line 1058: [ 4.627680] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER2_P3 (Read): Data Access in User mode during Functional access
    Line 1143: [ 14.227858] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER2_P3 (Read): Data Access in User mode during Functional access
    Line 1221: [ 23.827860] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER2_P3 (Read): Data Access in User mode during Functional access
    Line 1271: [ 33.427854] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER2_P3 (Read): Data Access in User mode during Functional access

    RTOS trace log:

    [ 2.626] [mcaspBitSetGblRCtl][4449]: McASPGlobalCtlGet tempVal1 0x8c1baf24
    [ 2.626] [mcaspBitSetGblRCtl][4451]: McASPRxGlobalCtlGet tempVal2 0x8b0b52ec
    [ 2.626] [mcaspBitSetGblRCtl][4463]: McASPRxGlobalCtlGet tempVal2 0x1
    [ 2.626] [mcaspBitSetGblRCtl][4466]: McASPRxGlobalCtlSet regs 0x48474000, tempVal2 0x3, bitMaskVal 0x2
    [ 2.627] [mcaspBitSetGblRCtl][4469]: McASPGlobalCtlGet tempVal1 0x8c1baf24
    [ 12.627] [mcaspBitSetGblRCtl][4490]: McASPGlobalCtlGet tempVal1 0x4e20, timeout 20000
    [ 22.627] [mcaspBitSetGblRCtl][4490]: McASPGlobalCtlGet tempVal1 0x2710, timeout 10000
    [ 32.627] [mcaspBitSetGblRCtl][4490]: McA

    The key point is found, but I still don't understand why does the McASPGlobalCtlGet() & McASPRxGlobalCtlGet() call will cause the "L3 Custom Error"? And how to resolve this issue?

  • 0 in reply to Woj
    TI__Prodigy 605 points

    Hi Woj,

    It is perhaps related to the McASP module power state.

    • Please check if the McASP instance of interest is activated in the device tree. This is needed for Linux to bring module out of reset
    • Please confirm that DEV_create() is ran for the "/mcasp5". You can find it in audio_app.c. This is needed to set PWRIDLESYSCONFIG register properly at module init.

    Thanks,

    Sheng

  • 0 in reply to Sheng Zhao
    Prodigy 235 points

    Hello Sheng,

    Thanks for your reply!

    I have already resolved this issue, of which the root cause is the McASP is not enabled properly in the device tree just like the your #1 suggestion said.

    BTW, may I contact to you directly on APPE topics in the future?

  • 0 in reply to Woj
    TI__Mastermind 42515 points

    Hi Woj,

    Glad to hear that the issue is resolved.

    You can send personal message to Sheng by clicking on his name and then on the blue "+ Connect" button at the top right.

    On the other hand I would recommend to post new thread for any new issues/questions you have - this way we can involve proper experts and provide better support.

    Regards,

    Yordan

  • 0 in reply to Woj
    TI__Prodigy 605 points

    Hi Woj,

    I'm glad that your issue was resolved.

    For future APPE topics, please post new threads on E2E and the right expert could assist you.

    Thanks,
    Sheng