This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

[OMAP-L137] ARM9 hangs polling CPSR

Other Parts Discussed in Thread: OMAP-L137

I'm running on an OMAP-L137 processor using DSPLink to coordinate the ARM9 app and the DSP code. I'm seeing the ARM9 stop responding after looping for quite a while successfully.

When I connect to the DSP using CCS, it is looping as expected, patiently waiting for the next request from the ARM9.

The ARM9 code is significantly instrumented with print statements just about everywhere in the App. When the ARM9 is waiting for a DSP Notification, it sits in a small polling loop which includes a print of the last event logged by the DSP.

When the ARM9 code hangs, I can no longer telnet into it, so I can't look at the active processes. The print statements from my ARM9 app stop, suggesting that it is not sitting in an infinite loop in my code. (all infinite loops are instrumented with print statements in my test app).

I tried connecting to it with CCS, and I can see that it is looping (not reset). The assembly language loop is below. All of these addresses are not in the DSP range. The DSPLink reserved areas start at 0xc2000000 (POOLMEM, reset area, etc) That is, this is the memory space reserved for ARM9. (I'm running MV5.0.) Is there anyway for me to tell what code might be sitting in this address when the ARM9 gets stuck? For example, if it is in the printf function, that might be useful to know. Or is it possible to get call stack info? (Currently, CCS says "C-Source Is Not Available" and no memory addresses are given either.) Could it be stuck in DSPLink module itself?

C00393E0 E12FFF35 BLX           0xC0C390BC                     (goes to c003965c)
C00393E4 E5943000 LDR           R3, [R4]
C00393E8 E3130004 TST           R3, #0x4
C00393EC 1A000002 BNE           0xC00393FC
C00393F0 E5943000 LDR           R3, [R4]
C00393F4 E3130008 TST           R3, #0x8
C00393F8 0AFFFFF8 BEQ           0xC00393E0                      (takes this branch)

C003965C E1A0C00D MOV           R12, R13
C0039660 E92DD800 STMFD         R13!, {R11-R12, R14-PC}
C0039664 E24CB004 SUB           R11, R12, #0x4
C0039668 E59F304C LDR           R3, 0xC00396BC
C003966C E5933000 LDR           R3, [R3]
C0039670 E3530000 CMP           R3, #0x0
C0039674 0A000000 BEQ           0xC003967C                       (takes this branch)
C0039678 E89DA800 LDMFD         R13, {R11, R13, PC}
C003967C E10F3000 MRS           R3, CPSR
C0039680 E3833080 ORR           R3, R3, #0x80
C0039684 E121F003 MSR           CPSR_c, R3
C0039688 E3CD2D7F BIC           R2, R13, #0x1FC0
C003968C E3C2203F BIC           R2, R2, #0x3F
C0039690 E5923000 LDR           R3, [R2]
C0039694 E3130004 TST           R3, #0x4
C0039698 1A000003 BNE           0xC00396AC
C003969C E5923000 LDR           R3, [R2]
C00396A0 E3130008 TST           R3, #0x8
C00396A4 1A000000 BNE           0xC00396AC
C00396A8 EB002054 BL            0xC0041800                              (takes this branch)

C00396AC E10F3000 MRS           R3, CPSR
C00396B0 E3C33080 BIC           R3, R3, #0x80
C00396B4 E121F003 MSR           CPSR_c, R3
C00396B8 E89DA800 LDMFD         R13, {R11, R13, PC}            (goes to 0xc00393e4)
C00396BC C03CA238 EORGTS        R10, R12, R8, LSR R2
C00396C0 E1A0C00D MOV           R12, R13
C00396C4 E92DD830 STMFD         R13!, {R4-R5, R11-R12, R14-PC}
C00396C8 E24CB004 SUB           R11, R12, #0x4
C00396CC E3CD4D7F BIC           R4, R13, #0x1FC0
C00396D0 E3C4403F BIC           R4, R4, #0x3F
C00396D4 E594500C LDR           R5, [R4, #0xC]
C00396D8 E284004C ADD           R0, R4, #0x4C
C00396DC E3A01010 MOV           R1, #0x10
C00396E0 EB06123E BL            0xC01BDFE0

C0041800 E3A00000 MOV           R0, #0x0
C0041804 EE111F10 MRC           P15, #0x0, R1, C1, C0, #0x0
C0041808 EE070F9A MCR           P15, #0x0, R0, C7, C10, #0x4
C004180C E3C12A01 BIC           R2, R1, #0x1000
C0041810 EE012F10 MCR           P15, #0x0, R2, C1, C0, #0x0
C0041814 EE070F90 MCR           P15, #0x0, R0, C7, C0, #0x4
C0041818 EE011F10 MCR           P15, #0x0, R1, C1, C0, #0x0
C004181C E1A0F00E MOV           PC, R14                         (R14 = 0xc00396ac)

  • More info: The address where it ends up is not constant, but I reproduced the problem with another run and did "echo t >/proc/sysrq-trigger" followed by dmesg to get the following dump. You will see my app (vqm_results) interacting with dsplinkk followed by "copy_from_user call failed 24" repeated. Any insight into what is happening would help.

    root@172.23.14.212:~# echo t >/proc/sysrq-trigger
    root@172.23.14.212:~# dmesg
    02DE150     0    13      1            14    12 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0066c34>] (worker_thread+0xe8/0x144)
     r4 = 00000000
    [<c0066b4c>] (worker_thread+0x0/0x144) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    kthread       S [c046b340] C02DE150     0    14      1    29     199    13 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0066c34>] (worker_thread+0xe8/0x144)
     r4 = 00000000
    [<c0066b4c>] (worker_thread+0x0/0x144) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    kblockd/0     S [c04ba3a0] C02DE150     0    29     14            36       (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0066c34>] (worker_thread+0xe8/0x144)
     r4 = 00000000
    [<c0066b4c>] (worker_thread+0x0/0x144) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    khubd         S [c0506c20] C02DE150     0    36     14            93    29 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0212c60>] (hub_thread+0xb60/0xc20)
     r4 = C0506C20
    [<c0212100>] (hub_thread+0x0/0xc20) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    pdflush       S [c05719e0] C02DE150     0    93     14            94    36 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0084684>] (pdflush+0xf4/0x250)
     r4 = C0576000
    [<c0084590>] (pdflush+0x0/0x250) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    pdflush       S [c0571720] C02DE150     0    94     14            95    93 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0084684>] (pdflush+0xf4/0x250)
     r4 = C0578000
    [<c0084590>] (pdflush+0x0/0x250) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    kswapd0       S [c0571460] C02DE150     0    95     14            96    94 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0087d1c>] (kswapd+0xac/0x3f8)
     r4 = C0571460
    [<c0087c70>] (kswapd+0x0/0x3f8) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    aio/0         S [c05711a0] C02DE150     0    96     14            97    95 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0066c34>] (worker_thread+0xe8/0x144)
     r4 = 00000000
    [<c0066b4c>] (worker_thread+0x0/0x144) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    cifsoplockd   S [c05d2cc0] C02DE150     0    97     14            98    96 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df170>] (schedule_timeout+0x98/0xc4)
     r4 = 00064471
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c01518fc>] (cifs_oplock_thread+0x1c0/0x1f0)
     r7 = C0451F20  r6 = C05D4000  r5 = C03DDB0C  r4 = 00000000
    [<c015173c>] (cifs_oplock_thread+0x0/0x1f0) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    cifsdnotifyd  S [c05d2a00] C02DE150     0    98     14           202    97 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df170>] (schedule_timeout+0x98/0xc4)
     r4 = 0006459D
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c0151550>] (cifs_dnotify_thread+0x2c/0xb8)
     r7 = 00000002  r6 = C03DDAF8  r5 = C05D6000  r4 = C03DDAF8
    [<c0151524>] (cifs_dnotify_thread+0x0/0xb8) from [<c0069c94>] (kthread+0xe8/0x128)
     r7 = C0451F20  r6 = C0151524  r5 = C05D6000  r4 = 00000000
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    mtdblockd     S [c05066a0] C02DE150     0   199      1           405    14 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c020b274>] (mtd_blktrans_thread+0x128/0x2e0)
     r4 = C05066A0
    [<c020b14c>] (mtd_blktrans_thread+0x0/0x2e0) from [<c0056478>] (do_exit+0x0/0xa44)
    dm_spi.0      S [c0650d00] C02DE150     0   202     14           204    98 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0066c34>] (worker_thread+0xe8/0x144)
     r4 = 00000000
    [<c0066b4c>] (worker_thread+0x0/0x144) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    dm_spi.1      S [c0650780] C02DE150     0   204     14           240   202 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0066c34>] (worker_thread+0xe8/0x144)
     r4 = 00000000
    [<c0066b4c>] (worker_thread+0x0/0x144) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    rpciod/0      S [c0515400] C02DE150     0   240     14           947   204 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0066c34>] (worker_thread+0xe8/0x144)
     r4 = 00000000
    [<c0066b4c>] (worker_thread+0x0/0x144) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    udevd         S [c05156c0] C02DE150     0   405      1           683   199 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 00000008  r6 = 00000000  r5 = C1E33F70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    portmap       S [c0571ca0] C02DE150     0   683      1           729   405 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b1ee0>] (do_sys_poll+0x330/0x45c)
     r7 = C0651D40  r6 = C1E8A000  r5 = FFFFFFFF  r4 = 00000000
    [<c00b1bb0>] (do_sys_poll+0x0/0x45c) from [<c00b2058>] (sys_poll+0x4c/0x54)
    [<c00b200c>] (sys_poll+0x0/0x54) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r5 = 00016720  r4 = 00000002
    inetd         S [c0650200] C02DE150     0   729      1   926     836   683 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 00000005  r6 = 00000000  r5 = C1873F70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    syslogd       S [c06324a0] C02DE150     0   836      1           849   729 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 00000005  r6 = 00000000  r5 = C1E0FF70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    rpc.statd     S [c04d03c0] C02DE150     0   849      1                 836 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 0000000A  r6 = 00000000  r5 = C195BF70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    telnetd       S [c04d0c00] C02DE150     0   926    729   927     950       (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 00000004  r6 = 00000000  r5 = C19B5F70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    bash          S [c04ae380] C02DE150     0   927    926   940               (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c00562c4>] (do_wait+0xc08/0xcec)
     r4 = C04AE380
    [<c00556bc>] (do_wait+0x0/0xcec) from [<c00563e0>] (sys_wait4+0x38/0x40)
    [<c00563a8>] (sys_wait4+0x0/0x40) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r4 = BE94CA84
    gdb           S [c04d0100] C02DE150     0   940    927   949               (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c003b584>] (sys_rt_sigsuspend+0xfc/0x120)
     r4 = C0784000
    [<c003b488>] (sys_rt_sigsuspend+0x0/0x120) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r7 = 000000B3  r6 = 00000000  r5 = 00000000  r4 = 00000001
    vqm_results   S [c05063e0] C02DE150     0   946    940                 949 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df4a0>] (do_nanosleep+0x64/0xa4)
     r4 = C1747F20
    [<c02df43c>] (do_nanosleep+0x0/0xa4) from [<c006d0d8>] (hrtimer_nanosleep+0x44/0x140)
     r7 = 00000000  r6 = 1DCD6500  r5 = C1747F88  r4 = C1747F20
    [<c006d094>] (hrtimer_nanosleep+0x0/0x140) from [<c006d26c>] (sys_nanosleep+0x98/0xac)
    [<c006d1d4>] (sys_nanosleep+0x0/0xac) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r5 = 00000003  r4 = 00098AF8
    vqm_results   S [c06504c0] C02DE150     0   949    940           946       (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = C06504C0
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<bf008000>] (SYNC_WaitSEM+0x320/0x3f0 [dsplinkk])
     r7 = 00000002  r6 = C29BD000  r5 = C11D8000  r4 = C06504C0
    [<bf007ce0>] (SYNC_WaitSEM+0x0/0x3f0 [dsplinkk]) from [<bf005488>] (UEVENT_GetBuf+0xf4/0x21c [dsplinkk])
    [<bf005394>] (UEVENT_GetBuf+0x0/0x21c [dsplinkk]) from [<bf0006ec>] (DRV_Read+0x14/0x18 [dsplinkk])
    [<bf0006d8>] (DRV_Read+0x0/0x18 [dsplinkk]) from [<c009bed0>] (vfs_read+0xd8/0x1b4)
    [<c009bdf8>] (vfs_read+0x0/0x1b4) from [<c009c9f8>] (sys_read+0x94/0x124)
     r7 = 00000007  r6 = 00000018  r5 = C05185A0  r4 = 00000001
    [<c009c964>] (sys_read+0x0/0x124) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r8 = C0037FA8  r7 = 00000003  r6 = 00000000  r5 = 40971684
     r4 = 00000001
    DSPLINK_DPC_0 S [c05d2480] C02DE150     0   947     14           948   240 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df170>] (schedule_timeout+0x98/0xc4)
     r4 = 00064241
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<bf00287c>] (DPC_Callback+0x90/0xa8 [dsplinkk])
     r7 = C1747C98  r6 = 00000002  r5 = C11A0000  r4 = BF023DD0
    [<bf0027ec>] (DPC_Callback+0x0/0xa8 [dsplinkk]) from [<c0069c94>] (kthread+0xe8/0x128)
     r6 = BF0027EC  r5 = C11A0000  r4 = 00000000
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    DSPLINK_DPC_1 S [c0650a40] C02DE150     0   948     14                 947 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df170>] (schedule_timeout+0x98/0xc4)
     r4 = 00064214
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<bf00287c>] (DPC_Callback+0x90/0xa8 [dsplinkk])
     r7 = C1747C98  r6 = 00000002  r5 = C11C6000  r4 = BF023DE8
    [<bf0027ec>] (DPC_Callback+0x0/0xa8 [dsplinkk]) from [<c0069c94>] (kthread+0xe8/0x128)
     r6 = BF0027EC  r5 = C11C6000  r4 = 00000001
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    telnetd       S [c0632a20] C02DE150     0   950    729   951           926 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 00000004  r6 = 00000000  r5 = C11E9F70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    bash          R [c06321e0] C02DE150 [curr]     0   951    950                     (NOTLB)
    [<c003cd30>] (show_stack+0x0/0x48) from [<c004ed3c>] (show_state+0x19c/0x22c)
    [<c004eba0>] (show_state+0x0/0x22c) from [<c01e14c8>] (sysrq_handle_showstate+0x10/0x14)
    [<c01e14b8>] (sysrq_handle_showstate+0x0/0x14) from [<c01e171c>] (__handle_sysrq+0xb4/0x178)
    [<c01e1668>] (__handle_sysrq+0x0/0x178) from [<c00dc4a0>] (write_sysrq_trigger+0x40/0x48)
    [<c00dc460>] (write_sysrq_trigger+0x0/0x48) from [<c009bd1c>] (vfs_write+0xdc/0x1b8)
     r4 = C1C8BA80
    [<c009bc40>] (vfs_write+0x0/0x1b8) from [<c009c8d4>] (sys_write+0x94/0x124)
     r7 = 00000001  r6 = 00000002  r5 = C1C8BA80  r4 = 00000002
    [<c009c840>] (sys_write+0x0/0x124) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r8 = C0037FA8  r7 = 00000004  r6 = 401A5630  r5 = 000B6408
     r4 = 00000002
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    root@172.23.14.212:~# dmesg
    x0/0xfc) from [<c02df170>] (schedule_timeout+0x98/0xc4)
     r4 = 0006459D
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c0151550>] (cifs_dnotify_thread+0x2c/0xb8)
     r7 = 00000002  r6 = C03DDAF8  r5 = C05D6000  r4 = C03DDAF8
    [<c0151524>] (cifs_dnotify_thread+0x0/0xb8) from [<c0069c94>] (kthread+0xe8/0x128)
     r7 = C0451F20  r6 = C0151524  r5 = C05D6000  r4 = 00000000
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    mtdblockd     S [c05066a0] C02DE150     0   199      1           405    14 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c020b274>] (mtd_blktrans_thread+0x128/0x2e0)
     r4 = C05066A0
    [<c020b14c>] (mtd_blktrans_thread+0x0/0x2e0) from [<c0056478>] (do_exit+0x0/0xa44)
    dm_spi.0      S [c0650d00] C02DE150     0   202     14           204    98 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0066c34>] (worker_thread+0xe8/0x144)
     r4 = 00000000
    [<c0066b4c>] (worker_thread+0x0/0x144) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    dm_spi.1      S [c0650780] C02DE150     0   204     14           240   202 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0066c34>] (worker_thread+0xe8/0x144)
     r4 = 00000000
    [<c0066b4c>] (worker_thread+0x0/0x144) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    rpciod/0      S [c0515400] C02DE150     0   240     14           947   204 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c0066c34>] (worker_thread+0xe8/0x144)
     r4 = 00000000
    [<c0066b4c>] (worker_thread+0x0/0x144) from [<c0069c94>] (kthread+0xe8/0x128)
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    udevd         S [c05156c0] C02DE150     0   405      1           683   199 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 00000008  r6 = 00000000  r5 = C1E33F70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    portmap       S [c0571ca0] C02DE150     0   683      1           729   405 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b1ee0>] (do_sys_poll+0x330/0x45c)
     r7 = C0651D40  r6 = C1E8A000  r5 = FFFFFFFF  r4 = 00000000
    [<c00b1bb0>] (do_sys_poll+0x0/0x45c) from [<c00b2058>] (sys_poll+0x4c/0x54)
    [<c00b200c>] (sys_poll+0x0/0x54) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r5 = 00016720  r4 = 00000002
    inetd         S [c0650200] C02DE150     0   729      1   926     836   683 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 00000005  r6 = 00000000  r5 = C1873F70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    syslogd       S [c06324a0] C02DE150     0   836      1           849   729 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 00000005  r6 = 00000000  r5 = C1E0FF70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    rpc.statd     S [c04d03c0] C02DE150     0   849      1                 836 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 0000000A  r6 = 00000000  r5 = C195BF70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    telnetd       S [c04d0c00] C02DE150     0   926    729   927     950       (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 00000004  r6 = 00000000  r5 = C19B5F70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    bash          S [c04ae380] C02DE150     0   927    926   940               (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c00562c4>] (do_wait+0xc08/0xcec)
     r4 = C04AE380
    [<c00556bc>] (do_wait+0x0/0xcec) from [<c00563e0>] (sys_wait4+0x38/0x40)
    [<c00563a8>] (sys_wait4+0x0/0x40) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r4 = BE94CA84
    gdb           S [c04d0100] C02DE150     0   940    927   949               (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c003b584>] (sys_rt_sigsuspend+0xfc/0x120)
     r4 = C0784000
    [<c003b488>] (sys_rt_sigsuspend+0x0/0x120) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r7 = 000000B3  r6 = 00000000  r5 = 00000000  r4 = 00000001
    vqm_results   S [c05063e0] C02DE150     0   946    940                 949 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df4a0>] (do_nanosleep+0x64/0xa4)
     r4 = C1747F20
    [<c02df43c>] (do_nanosleep+0x0/0xa4) from [<c006d0d8>] (hrtimer_nanosleep+0x44/0x140)
     r7 = 00000000  r6 = 1DCD6500  r5 = C1747F88  r4 = C1747F20
    [<c006d094>] (hrtimer_nanosleep+0x0/0x140) from [<c006d26c>] (sys_nanosleep+0x98/0xac)
    [<c006d1d4>] (sys_nanosleep+0x0/0xac) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r5 = 00000003  r4 = 00098AF8
    vqm_results   S [c06504c0] C02DE150     0   949    940           946       (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = C06504C0
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<bf008000>] (SYNC_WaitSEM+0x320/0x3f0 [dsplinkk])
     r7 = 00000002  r6 = C29BD000  r5 = C11D8000  r4 = C06504C0
    [<bf007ce0>] (SYNC_WaitSEM+0x0/0x3f0 [dsplinkk]) from [<bf005488>] (UEVENT_GetBuf+0xf4/0x21c [dsplinkk])
    [<bf005394>] (UEVENT_GetBuf+0x0/0x21c [dsplinkk]) from [<bf0006ec>] (DRV_Read+0x14/0x18 [dsplinkk])
    [<bf0006d8>] (DRV_Read+0x0/0x18 [dsplinkk]) from [<c009bed0>] (vfs_read+0xd8/0x1b4)
    [<c009bdf8>] (vfs_read+0x0/0x1b4) from [<c009c9f8>] (sys_read+0x94/0x124)
     r7 = 00000007  r6 = 00000018  r5 = C05185A0  r4 = 00000001
    [<c009c964>] (sys_read+0x0/0x124) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r8 = C0037FA8  r7 = 00000003  r6 = 00000000  r5 = 40971684
     r4 = 00000001
    DSPLINK_DPC_0 S [c05d2480] C02DE150     0   947     14           948   240 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df170>] (schedule_timeout+0x98/0xc4)
     r4 = 00064241
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<bf00287c>] (DPC_Callback+0x90/0xa8 [dsplinkk])
     r7 = C1747C98  r6 = 00000002  r5 = C11A0000  r4 = BF023DD0
    [<bf0027ec>] (DPC_Callback+0x0/0xa8 [dsplinkk]) from [<c0069c94>] (kthread+0xe8/0x128)
     r6 = BF0027EC  r5 = C11A0000  r4 = 00000000
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    DSPLINK_DPC_1 S [c0650a40] C02DE150     0   948     14                 947 (L-TLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df170>] (schedule_timeout+0x98/0xc4)
     r4 = 00064214
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<bf00287c>] (DPC_Callback+0x90/0xa8 [dsplinkk])
     r7 = C1747C98  r6 = 00000002  r5 = C11C6000  r4 = BF023DE8
    [<bf0027ec>] (DPC_Callback+0x0/0xa8 [dsplinkk]) from [<c0069c94>] (kthread+0xe8/0x128)
     r6 = BF0027EC  r5 = C11C6000  r4 = 00000001
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 00000000
    telnetd       S [c0632a20] C02DE150     0   950    729   951           926 (NOTLB)
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df0fc>] (schedule_timeout+0x24/0xc4)
     r4 = 00000000
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<c00b24e0>] (do_select+0x480/0x4f0)
     r7 = 00000004  r6 = 00000000  r5 = C11E9F70  r4 = 00000000
    [<c00b2060>] (do_select+0x0/0x4f0) from [<c00b285c>] (sys_select+0x30c/0x4c8)
    [<c00b2550>] (sys_select+0x0/0x4c8) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
    bash          R [c06321e0] C02DE150 [curr]     0   951    950                     (NOTLB)
    [<c003cd30>] (show_stack+0x0/0x48) from [<c004ed3c>] (show_state+0x19c/0x22c)
    [<c004eba0>] (show_state+0x0/0x22c) from [<c01e14c8>] (sysrq_handle_showstate+0x10/0x14)
    [<c01e14b8>] (sysrq_handle_showstate+0x0/0x14) from [<c01e171c>] (__handle_sysrq+0xb4/0x178)
    [<c01e1668>] (__handle_sysrq+0x0/0x178) from [<c00dc4a0>] (write_sysrq_trigger+0x40/0x48)
    [<c00dc460>] (write_sysrq_trigger+0x0/0x48) from [<c009bd1c>] (vfs_write+0xdc/0x1b8)
     r4 = C1C8BA80
    [<c009bc40>] (vfs_write+0x0/0x1b8) from [<c009c8d4>] (sys_write+0x94/0x124)
     r7 = 00000001  r6 = 00000002  r5 = C1C8BA80  r4 = 00000002
    [<c009c840>] (sys_write+0x0/0x124) from [<c0037f78>] (__sys_trace_return+0x0/0x28)
     r8 = C0037FA8  r7 = 00000004  r6 = 401A5630  r5 = 000B6408
     r4 = 00000002
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24
    copy_from_user call failed 24

  • Hello,

    Could you give more detail as to the application scenario just prior to the crash?

    I need more clarification for this line in the application scenario:  When the ARM9 is waiting for a DSP Notification, it sits in a small polling loop which includes a print of the last event logged by the DSP.

    [Deepali]: Are you spinning in user or kernel space?

    Deepali

  • I'm not sure how I can tell whether a context switch has occurred. The box has successfully completed Linux boot and I've telnetted into the box to run my app. Before running it, I insmod dsplinkk.ko. The program runs in a loop collecting data from the DSP. I have a short delay in the ARM9's code and I've tried changing that delay from 10 ms to 500 ms. The ARM9 successfully collects the data and prints a sampling of the received results before looping back. I can examine the stack pointers for the tasks on the DSP and they are all fine: no evidence of memory leaks. When I see the failure, it might happen within the first 20 loops or after 50000 loops. It does not seem to be dependent on the length of the delay. That is, it does not happen any faster when I run with a short delay.

    If a context switch has not occurred, then I'm in user mode. Can I identify which mode via CCS? I do not seem to see system hangs when my program isn't running. My first concern was that the target box is not in its case, and therefore is sensitive to resets caused by static electricity. We performed some modifications to the system to minimize those, and I've stopped seeing the resets.

    I've tried tracing in DSPLink, but the latency in the prints makes me think that there isn't much useful info there. At the very least, I can say that the DSPLink trace statements seem to suggest that the hang is not in the same place. That is, the DSPLink trace statements are not identical when it hangs. There are several sections of DSPLink code that might be run commonly among several commands. For example, drv_pmgr.c.

    If there was some assurance that DSPLink code did not perform polling loops forever (that is, if they timed out and returned a bad status), this might be easily blamed on Linux, but unfortunately, DSPLink is known to loop forever.

    What I'm in need of is a method of determining which code is resident at this location in memory. Can I hack Linux some way that would notify me (or Log) the processes loaded into memory to run?

  • Hello,

    Please find my comments inline:

    >>If there was some assurance that DSPLink code did not perform polling loops forever (that is, if they timed out and returned a bad status), this might be easily blamed on Linux, but >>unfortunately, DSPLink is known to loop forever.
    [Deepali]: DSPLink does not perform polling loops any where in the code.

    >>What I'm in need of is a method of determining which code is resident at this location in memory. Can I hack Linux some way that would notify me (or Log) the processes loaded >>into memory to run
    [Deepali]: After spending some time looking at the log, the following are my observations:
    1) The log does not prove that DSPLink is the problem. DSPLink is coming in the trace dump when the problem occurs but may not necessarily be the root cause.


    Consider a sample trace dump of the DSPLink functions and associated explanation below:
    [<c02ddb20>] (__schedule+0x0/0x75c) from [<c02de498>] (schedule+0xbc/0xfc)
    [<c02de3dc>] (schedule+0x0/0xfc) from [<c02df170>] (schedule_timeout+0x98/0xc4)
     r4 = 00064241
    [<c02df0d8>] (schedule_timeout+0x0/0xc4) from [<bf00287c>] (DPC_Callback+0x90/0xa8 [dsplinkk])
     r7 = C1747C98  r6 = 00000002  r5 = C11A0000  r4 = BF023DD0
    [<bf0027ec>] (DPC_Callback+0x0/0xa8 [dsplinkk]) from [<c0069c94>] (kthread+0xe8/0x128)
     r6 = BF0027EC  r5 = C11A0000  r4 = 00000000
    [<c0069bac>] (kthread+0x0/0x128) from [<c0056478>] (do_exit+0x0/0xa44)
     r7 = 00000000  r6 = 00000000  r5 = 00000000  r4 = 000000
    The above shows that the DSPLink DPC has run and then sheduled out to allow another function to run. The same function flow is seen for the other DSPLink functions which come up in the trace dump for DSPLINK_DPC_0 and DRV_Read. So when the crash actually happens, some other code seems to be running.


    2) The dmesg log does not seem to be complete. Will it possible for you to get the complete crash dump on a tera term console? This will help us isolate the actual cause.

    Deepali

  • [Deepali]: DSPLink does not perform polling loops any where in the code.

    I found polling loops in the dmXXX_hal_pci_dma.c files. There is another in ips.c. (See below.) However, my comment was targeted at the concept of no timeouts. In dsp/src/mpcs/mpcs.c, you will find a SEM_pend with a timeout of SYS_FOREVER. The timeout value is not provided by the user as a parameter.

    The kinds of "hangs" I've seen involve missed NOTIFY events. If you look in ips.c, you'll find a (necessary because of the way notify works) loop:

                    do {
                        HWI_restore (irqFlags) ;
                        irqFlags = HWI_disable () ;
                        HAL_cacheInv ((Ptr) &(gppEventChart [eventNo]),
                                      sizeof (IPS_EventEntry)) ;
                    }
                    while (gppEventChart [eventNo].flag != DOWN) ;

    This is a sticking point that I've run into in the past, and it might be associated with this case, but I can't tell for sure. Both sides are waiting, but neither side is going to get any further notification than has already been sent.

    Why didn't the flag change? I have no idea. I'd like to see this code changed to use a timeout provided by the user.

    [Deepali]: The above shows that the DSPLink DPC has run and then sheduled out to allow another function to run. The same function flow is seen for the other DSPLink functions which come up in the trace dump for DSPLINK_DPC_0 and DRV_Read. So when the crash actually happens, some other code seems to be running.

    You may be right about another process, and that is what I'm trying to find out. That is, what other process could be running when the hang occurs. Although I've also found that the Trace logs are not always complete either. That is, the trace buffer might contain additional trace info that isn't "flushed".

    [Deepali]: The dmesg log does not seem to be complete. Will it possible for you to get the complete crash dump on a tera term console? This will help us isolate the actual cause.

    When the system hangs, Linux is no longer responding to anything. That is, I can't log onto the box, kill the program, etc. All I can do is reset the board.


  • Hello,

    >>The kinds of "hangs" I've seen involve missed NOTIFY events. If you look in ips.c, you'll find a (necessary because of the way notify works) loop:

    Can you give more information here?

     

  • I'm not sure what additional information would be useful. This is something that I'm not working on right now, but I'll give you what I can remember.

    One problem (not polling): My code on the DSP responds to notify events from the ARM9. The normal behavior of notify events is to be prioritized by their number. That paradigm didn't work for me, so to serialize the events, the ARM9 sends a single request and waits for a "CommandComplete" notify event from the DSP. This paradigm seems to work fine. Occasionally, the system would hang. The ARM9 had called NOTIFY_notify, but the DSP seemed to have missed the event. I connected with CCS and the DSP code was operating, the stack pointers and high water marks were all good. Background behavior, including SPI data collection and data processing were all happening as expected. The large structure of Notify events, callback pointers, etc looked good and the event was posted to the structure! The only thing missing was the processing of the event.

    Another problem (polling): Similarly, the "CommandComplete" events are sent by the DSP. I can see breakpoints on the DSP sending them. I put breakpoints in each received request and in the code that signals the complete. I put debug prints in the ARM9 code, so I know what command it is sending and whether it has received the CommandComplete. In several cases, after working fine for many minutes, the ARM9 would say that it was waiting for the CommandComplete, but the DSP had called NOTIFY_notify to send it. When I examined the DSP further, it was stuck in the loop I mentioned in ips.c. Oddly enough, the ARM9 had successfully received every CommandComplete up until the hang. That is, there are no retries on the ARM9 (No timeouts on failure to receive the CommandComplete because I want to be able to step in the DSP debugger.). The event that the DSP is waiting to go DOWN has already been received on the ARM9, and it is waiting to receive the next identical event.

    As I understand it, the ARM9 and DSP share this structure, so my viewing of it on the DSP is equivalent to view it on the ARM9. That is, in this case, the ARM9 has received the event, but it wasn't cleared in the structure. I don't know how that can happen, but the rest of the structure is as expected (no memory corruption.)

     

  • Hello,

    I apologize for the delay in the response.

    To give some background for the loop in ips.c

                    do {
                        HWI_restore (irqFlags) ;
                        irqFlags = HWI_disable () ;
                        HAL_cacheInv ((Ptr) &(gppEventChart [eventNo]),
                                      sizeof (IPS_EventEntry)) ;
                    }
                    while (gppEventChart [eventNo].flag != DOWN) ;

    This logic gets invoked in IPS_notify function.. This loop waits for the event to be processed to ensure that notify payload is processed before the next notification call. If you see that the logic spins here for a long time, it indicates that the other core is to busy to process the interrupt. This can happen when the other core has disabled interrupts. Do you see this kind of scenario in your application?

    Another point of importance here is that this logic gets called only when application uses RingIO or NOTIFY modules. This loop is not invoked for MSGQ and CHNL protocols as their notificatiosn do not involve a payload. Which protocol does your application use?

    >>Occasionally, the system would hang. The ARM9 had called NOTIFY_notify, but the DSP seemed to have missed the event
    How did you reach the conclusion that the event has been missed? Did some DSPLink API like MSGQ_get hang forever?

    Deepali

  • My project uses Proc, Pool, Notify, and MPCS. DSPLink is built with only those. The ARM9 task does not ever intentionally disable interrupts, although Linux and its drivers might.

    Once the link is initialized, all traffic is via notify. Initialization occurs via the ARM9 sending pool pointers via notify events. All of these are received. Then the ARM9 loops requesting any available results. When it sends a request, it prints a message. The DSP receives the request, processes it, and then sends a "reply" notification saying that it is complete. The ARM9 receives the completion notification and prints a message. It does not continue with additional requests until it receives the notification.

    The print statements indicated that each request received a reply up until the failure. That is, each notification sent each direction was received and processed until the "hang" which occurs on the last ARM9 request for results. The DSP receives the request, processes it, and attempts to notify the ARM9 of completion. That is when the hang occurs. I've verified that the DSP code is in that loop.

    Your suggestion is that the previous notification was not received due to the interrupts being disabled, but the print statement suggests that this is not true. Instead, the flag did not get cleared even though the event was received and processed. Alternatively, some rampant process clobbered the data structure and set the flag after it was cleared. I've displayed the data structure in CCS, and in particular checked for memory corruption in the table area. What I find is that the table looks fine, although the flag is set when it shouldn't be.

    I do not know why this happens. It certainly does not happen frequently. My suggestion is that you provide a timeout for the notification (or another API with a timeout).

  • Hello,

    Coming out of the loop after a specified timeout breaks the functionality and deterministic behavior of Notify. If we come out of the loop after a specified timeout, the payload may/may not have gone across.

    Here, I would like to see why the event is not cleared as it indicates a deeper problem. Allowing it to timeout and proceed may simple hide the problem and is not the solution.

    Which DSPLink release are you on?

    In DSPLink 1.61.03, we fixed a defect:

    SDOCM00054941 DSPLINK hangs in IPS_notify () on both GPP and DSP side causing system deadlock

    This defect occurs when one core is repeatedly calling NOTIFY_unregisterNotifier while the other core is sending notifications using NOTIFY_notify repeatedly. As a fix, the IPS_notify code was updated  to enable and disable interrupts in between in while loop to  reduce the latency for other higher priority interrupts while checking for the other processors eventchart[eventFlag]. Also IPS_unregister function was updated to clear the pending events if there are no  eventhandlers registered for the event.

    Deepali

  • I'm running with dsplink-1_61_03-prebuilt. The code on both sides has finished intialization (where I register for events). Both sides are doing nothing except calls to Notify. Furthermore, they are forced by the logic of the code to have one and only one notify/notify reply active at a time. That is, without the timeout, the ARM9 makes a request via notify, the DSP "catches" it and replies to it. The DSP completes its activity by sending a "command complete" notify to the ARM9, which catches it.

    The process then repeats. (Timing on the repeat varies, but the sequences do not overlap. That is, the ARM9 does not send a request unless the previous request was handled.

    So, as I mentioned earlier, the event that the DSP is pending on will not occur because it has already occurred and been handled.

    What would happen if a higher priority notify event asserted while the code was in the loop. Would it get processed? Maybe I can "fix" the problem by resetting the notification structure (with both sides coordinating) via a high priority event that is reserved for the purpose?

    To get back to my original question, given the assembly dump of the original post, can I tell what code is running on the ARM9? Here is one way you might be able to help. Can you compare the assembly to the assembly for the notify loop on the ARM9? Or can you tell me how to get assembly from the DSPLink build?