TDA4VM: C7 stall during TIDL inference, need reboot

Christophe ALLART

Part Number: TDA4VM

Hello,

I am using TIDL from SDK ti-processor-sdk-rtos-j721e-evm-08_06_00_12 to import CNN models and run them on TDA4VM-SK board ( SD card image 8.6 ).

With some models (some other run fine) , the inference stall and C7 remains blocked. Killing the app is not sufficient, and a reboot is needed.

I have the same behaviour when using python and OSRT API (TFLITE) and using the OpenVX/TIOX API .

When using the TIOVX API , i am able to enable log traces on C7 ( view on terminal using the tool vx_app_arm_remote_log.out ) and i can see in which layer the C7 hangs.

It is a PAD layer. (between a resize and a concat ).

Below the traces on C7:

[C7x_1 ] 202194.891312 s: Alg Alloc for Layer # -    0
[C7x_1 ] 202194.891336 s: Alg Alloc for Layer # -    1
[C7x_1 ] 202194.891376 s: Alg Alloc for Layer # -    2
[C7x_1 ] 202194.891405 s: Alg Alloc for Layer # -    3
[C7x_1 ] 202194.891430 s: Alg Alloc for Layer # -    4
[C7x_1 ] 202194.891459 s: Alg Alloc for Layer # -    5
[C7x_1 ] 202194.891486 s: Alg Alloc for Layer # -    6
[C7x_1 ] 202194.891510 s: Alg Alloc for Layer # -    7
[C7x_1 ] 202194.891536 s: Alg Alloc for Layer # -    8
[C7x_1 ] 202194.891562 s: Alg Alloc for Layer # -    9
[C7x_1 ] 202194.891585 s: Alg Alloc for Layer # -   10
[C7x_1 ] 202194.891618 s: Alg Alloc for Layer # -   11
[C7x_1 ] 202194.891650 s: Alg Alloc for Layer # -   12
[C7x_1 ] 202194.891674 s: Alg Alloc for Layer # -   13
[C7x_1 ] 202194.891701 s: Alg Alloc for Layer # -   14
[C7x_1 ] 202194.891727 s: Alg Alloc for Layer # -   15
[C7x_1 ] 202194.891751 s: Alg Alloc for Layer # -   16
[C7x_1 ] 202194.891776 s: Alg Alloc for Layer # -   17
[C7x_1 ] 202194.891810 s: Alg Alloc for Layer # -   18
[C7x_1 ] 202194.891842 s: Alg Alloc for Layer # -   19
[C7x_1 ] 202194.891866 s: Alg Alloc for Layer # -   20
[C7x_1 ] 202194.891888 s: Alg Alloc for Layer # -   21
[C7x_1 ] 202194.891911 s: Alg Alloc for Layer # -   22
[C7x_1 ] 202194.891941 s: Alg Alloc for Layer # -   23
[C7x_1 ] 202194.891970 s: Alg Alloc for Layer # -   24
[C7x_1 ] 202194.891993 s: Alg Alloc for Layer # -   25
[C7x_1 ] 202194.892016 s: Alg Alloc for Layer # -   26
[C7x_1 ] 202194.892046 s: Alg Alloc for Layer # -   27
[C7x_1 ] 202194.892075 s: Alg Alloc for Layer # -   28
[C7x_1 ] 202194.892099 s: Alg Alloc for Layer # -   29
[C7x_1 ] 202194.892121 s: Alg Alloc for Layer # -   30
[C7x_1 ] 202194.892149 s: Alg Alloc for Layer # -   31
[C7x_1 ] 202194.892179 s: Alg Alloc for Layer # -   32
[C7x_1 ] 202194.892207 s: Alg Alloc for Layer # -   33
[C7x_1 ] 202194.892268 s: PREEMPTION: Requesting memory of size 8159232 for targetPriority = 512
[C7x_1 ] 202194.892321 s: min(FM):       22, max(FM): 40586134, min(WT): 40586240, max(WT): 43940480
[C7x_1 ] 202194.892340 s:
[C7x_1 ] 202194.892357 s: --------------------------------------------
[C7x_1 ] 202194.892380 s: TIDL Memory size requiement (record wise):
[C7x_1 ] 202194.892409 s: MemRecNum   , Space       , Attribute   , Size(KBytes)
[C7x_1 ] 202194.892440 s: 0           , DDR Cacheable, Persistent  , 14.84
[C7x_1 ] 202194.892471 s: 1           , DDR Cacheable, Persistent  , 0.14
[C7x_1 ] 202194.892501 s: 2           , L1D         , Scratch     , 16.00
[C7x_1 ] 202194.892529 s: 3           , L2          , Scratch     , 448.00
[C7x_1 ] 202194.892558 s: 4           , L3/MSMC     , Scratch     , 7968.00
[C7x_1 ] 202194.892587 s: 5           , DDR Cacheable, Persistent  , 3848.24
[C7x_1 ] 202194.892623 s: 6           , DDR Cacheable, Scratch     , 0.25
[C7x_1 ] 202194.892653 s: 7           , DDR Cacheable, Persistent  , 39634.90
[C7x_1 ] 202194.892682 s: 8           , DDR Cacheable, Scratch     , 0.13
[C7x_1 ] 202194.892711 s: 9           , DDR Cacheable, Scratch     , 3.13
[C7x_1 ] 202194.892740 s: 10          , DDR Cacheable, Scratch     , 0.13
[C7x_1 ] 202194.892769 s: 11          , DDR Cacheable, Persistent  , 460.13
[C7x_1 ] 202194.892798 s: 12          , DDR Cacheable, Scratch     , 512.25
[C7x_1 ] 202194.892827 s: 13          , DDR Cacheable, Persistent  , 7968.00
[C7x_1 ] 202194.892856 s: 14          , DDR Cacheable, Persistent  , 742.44
[C7x_1 ] 202194.892881 s: --------------------------------------------
[C7x_1 ] 202194.892905 s: Total memory size requirement (space wise):
[C7x_1 ] 202194.892923 s: Mem Space , Size(KBytes)
[C7x_1 ] 202194.892940 s: L1D       , 16.00
[C7x_1 ] 202194.892956 s: L2        , 448.00
[C7x_1 ] 202194.892971 s: L3/MSMC   , 7968.00
[C7x_1 ] 202194.892988 s: DDR Cacheable, 53184.55
[C7x_1 ] 202194.893010 s: --------------------------------------------
[C7x_1 ] 202194.893044 s: NOTE: Memory requirement in host emulation can be different from the same on EVM
[C7x_1 ] 202194.893082 s:       To get the actual TIDL memory requirement make sure to run on EVM with
[C7x_1 ] 202194.893104 s:       debugTraceLevel = 2
[C7x_1 ] 202194.893112 s:
[C7x_1 ] 202194.893131 s: --------------------------------------------
[C7x_1 ] 202194.894698 s: Alg Init for Layer # -    0 out of   33
[C7x_1 ] 202194.894725 s: Alg Init for Layer # -    1 out of   33
[C7x_1 ] 202194.895231 s: Alg Init for Layer # -    2 out of   33
[C7x_1 ] 202194.895728 s: Alg Init for Layer # -    3 out of   33
[C7x_1 ] 202194.895799 s: Alg Init for Layer # -    4 out of   33
[C7x_1 ] 202194.896115 s: Alg Init for Layer # -    5 out of   33
[C7x_1 ] 202194.896456 s: Alg Init for Layer # -    6 out of   33
[C7x_1 ] 202194.896518 s: Alg Init for Layer # -    7 out of   33
[C7x_1 ] 202194.896763 s: Alg Init for Layer # -    8 out of   33
[C7x_1 ] 202194.897109 s: Alg Init for Layer # -    9 out of   33
[C7x_1 ] 202194.897173 s: Alg Init for Layer # -   10 out of   33
[C7x_1 ] 202194.897779 s: Alg Init for Layer # -   11 out of   33
[C7x_1 ] 202194.898818 s: Alg Init for Layer # -   12 out of   33
[C7x_1 ] 202194.898882 s: Alg Init for Layer # -   13 out of   33
[C7x_1 ] 202194.899820 s: Alg Init for Layer # -   14 out of   33
[C7x_1 ] 202194.900739 s: Alg Init for Layer # -   15 out of   33
[C7x_1 ] 202194.900807 s: Alg Init for Layer # -   16 out of   33
[C7x_1 ] 202194.900868 s: Alg Init for Layer # -   17 out of   33
[C7x_1 ] 202194.901917 s: Alg Init for Layer # -   18 out of   33
[C7x_1 ] 202194.902283 s: Alg Init for Layer # -   19 out of   33
[C7x_1 ] 202194.902350 s: Alg Init for Layer # -   20 out of   33
[C7x_1 ] 202194.902390 s: Alg Init for Layer # -   21 out of   33
[C7x_1 ] 202194.902447 s: Alg Init for Layer # -   22 out of   33
[C7x_1 ] 202194.902845 s: Alg Init for Layer # -   23 out of   33
[C7x_1 ] 202194.903090 s: Alg Init for Layer # -   24 out of   33
[C7x_1 ] 202194.903157 s: Alg Init for Layer # -   25 out of   33
[C7x_1 ] 202194.903217 s: Alg Init for Layer # -   26 out of   33
[C7x_1 ] 202194.903562 s: Alg Init for Layer # -   27 out of   33
[C7x_1 ] 202194.903875 s: Alg Init for Layer # -   28 out of   33
[C7x_1 ] 202194.903939 s: Alg Init for Layer # -   29 out of   33
[C7x_1 ] 202194.904003 s: Alg Init for Layer # -   30 out of   33
[C7x_1 ] 202194.904521 s: Alg Init for Layer # -   31 out of   33
[C7x_1 ] 202194.905037 s: Alg Init for Layer # -   32 out of   33
[C7x_1 ] 202194.905537 s: Alg Init for Layer # -   33 out of   33
[C7x_1 ] 202194.905594 s: PREEMPTION: Adding a new priority object for targetPriority = 512, handle = 1173fc900
[C7x_1 ] 202194.905660 s: PREEMPTION: Now total number of priority objects = 1 at priorityId = 512,    with new memRec of base = 119eea900 and size = 8159232
[C7x_1 ] 202194.905716 s: PREEMPTION: Requesting context memory addr for handle 1173fc900, return Addr = 119eea900
[C7x_1 ] 202194.905791 s: TIDL_initializeHandleForPreemption is completed
[C7x_1 ] 202194.981953 s: TIDL_process is started with handle : 173fc900
[C7x_1 ] 202194.981987 s: TIDL_activate is called with handle : 173fc900
[C7x_1 ] 202194.982099 s: Starting Layer # -    1
[C7x_1 ] 202194.982123 s: Processing Layer # -    1
[C7x_1 ] 202194.983217 s: End of Layer # -    1 with outPtrs[0] = 11816de1a
[C7x_1 ] 202194.983244 s: Starting Layer # -    2
[C7x_1 ] 202194.983266 s: Processing Layer # -    2
[C7x_1 ] 202194.985309 s: End of Layer # -    2 with outPtrs[0] = 1177c2a1a
[C7x_1 ] 202194.985333 s: Starting Layer # -    3
[C7x_1 ] 202194.985353 s: Processing Layer # -    3
[C7x_1 ] 202194.986102 s: End of Layer # -    3 with outPtrs[0] = 7002004c
[C7x_1 ] 202194.986125 s: Starting Layer # -    4
[C7x_1 ] 202194.986145 s: Processing Layer # -    4
[C7x_1 ] 202194.986631 s: End of Layer # -    4 with outPtrs[0] = 7028b84c
[C7x_1 ] 202194.986655 s: Starting Layer # -    5
[C7x_1 ] 202194.986675 s: Processing Layer # -    5
[C7x_1 ] 202194.987602 s: End of Layer # -    5 with outPtrs[0] = 7002004c
[C7x_1 ] 202194.987626 s: Starting Layer # -    6
[C7x_1 ] 202194.987645 s: Processing Layer # -    6
[C7x_1 ] 202194.987798 s: End of Layer # -    6 with outPtrs[0] = 704f70e5
[C7x_1 ] 202194.987822 s: Starting Layer # -    7
[C7x_1 ] 202194.987842 s: Processing Layer # -    7
[C7x_1 ] 202194.988323 s: End of Layer # -    7 with outPtrs[0] = 704f70e5
[C7x_1 ] 202194.988347 s: Starting Layer # -    8
[C7x_1 ] 202194.988366 s: Processing Layer # -    8
[C7x_1 ] 202194.989282 s: End of Layer # -    8 with outPtrs[0] = 11816de65
[C7x_1 ] 202194.989306 s: Starting Layer # -    9
[C7x_1 ] 202194.989325 s: Processing Layer # -    9
[C7x_1 ] 202194.989551 s: End of Layer # -    9 with outPtrs[0] = 704f70b2
[C7x_1 ] 202194.989574 s: Starting Layer # -   10
[C7x_1 ] 202194.989596 s: Processing Layer # -   10
[C7x_1 ] 202194.990054 s: End of Layer # -   10 with outPtrs[0] = 70599132
[C7x_1 ] 202194.990077 s: Starting Layer # -   11
[C7x_1 ] 202194.990097 s: Processing Layer # -   11
[C7x_1 ] 202194.991047 s: End of Layer # -   11 with outPtrs[0] = 11840e232
[C7x_1 ] 202194.991071 s: Starting Layer # -   12
[C7x_1 ] 202194.991091 s: Processing Layer # -   12
[C7x_1 ] 202194.991231 s: End of Layer # -   12 with outPtrs[0] = 704f70d8
[C7x_1 ] 202194.991255 s: Starting Layer # -   13
[C7x_1 ] 202194.991275 s: Processing Layer # -   13
[C7x_1 ] 202194.991584 s: End of Layer # -   13 with outPtrs[0] = 7054b158
[C7x_1 ] 202194.991607 s: Starting Layer # -   14
[C7x_1 ] 202194.991628 s: Processing Layer # -   14
[C7x_1 ] 202194.991937 s: End of Layer # -   14 with outPtrs[0] = 7063b158
[C7x_1 ] 202194.991960 s: Starting Layer # -   15
[C7x_1 ] 202194.991981 s: Processing Layer # -   15
[C7x_1 ] 202194.992064 s: End of Layer # -   15 with outPtrs[0] = 704f70b2
[C7x_1 ] 202194.992087 s: Starting Layer # -   16
[C7x_1 ] 202194.992107 s: Processing Layer # -   16
[C7x_1 ] 202194.993048 s: End of Layer # -   16 with outPtrs[0] = 11855e632
[C7x_1 ] 202194.993074 s: Starting Layer # -   17
[C7x_1 ] 202194.993095 s: Processing Layer # -   17
[C7x_1 ] 202194.995573 s: End of Layer # -   17 with outPtrs[0] = 704f70b2
[C7x_1 ] 202194.995597 s: Starting Layer # -   18
[C7x_1 ] 202194.995617 s: Processing Layer # -   18
[C7x_1 ] 202194.995869 s: End of Layer # -   18 with outPtrs[0] = 704f70b2
[C7x_1 ] 202194.995894 s: Starting Layer # -   19
[C7x_1 ] 202194.995914 s: Processing Layer # -   19
[C7x_1 ] 202194.996145 s: End of Layer # -   19 with outPtrs[0] = 11840e266
[C7x_1 ] 202194.996169 s: Starting Layer # -   20
[C7x_1 ] 202194.996188 s: Processing Layer # -   20

Following are the details for this PAD layer , extracted from the .SVG file produced during import:

weightsElementSizeInBits=8
 strideOffsetMethod=TIDL_StrideOffsetCenter
 actParams:
 actType=TIDL_NoAct
 slopeScale=0.000000 clipMin/Max=(0.000000,0.000000)
 Data Flow Info:
 Space: 1, Base Mem: 5075072(5.075) - 7606528(7.607)
 Size: 2531456(2.531), Ch Pitch: 19776
 PadR/C: [1,1] -> [0,0], PadR/CZeros=0,0 PadR/CFillZeros=0,0
 Inputs:
 [19] 
 Outputs:
 [20] numDim=4 dims=[1,128,128,154] elementType=TIDL_UnsignedChar
 padH/W=[0,0] batchPadH/W=[0,0] numBatchH/W=[1,1] pitch=[2531456,19776,154]
 dataQ=0 roundBits=0
 min/maxValue=(0,0) min/maxTensorValue=(0.000000,0.378313)
 tensorScale=372.707245

Do you have any idea of the problem.

Alternatively, are you aware of a C7 stall problem that may be fixed in a more recent TIDL version ?

Thank you

Christophe

over 3 years ago

0 Anshu Jain over 3 years ago

TI__Guru 56820 points

Hi Christophe,

Can you try putting pad layer in deny_list and see if there is no hang?

Regards,

Anshu

0 Christophe ALLART over 3 years ago in reply to Anshu Jain

Prodigy 135 points

Hi Anshu,

I made the same test with deny_list = " 34 " ( i.e PAD ) .

Now, it runs fine on the TDA4VM.

But the perf is horrible ( 6 sec inference time ), and it's not usable.

Anyway, it proves there is some bug in TIDL runtime in C7 on PAD layer (TIDL emulation on X86 runs fine ).

Regards,

Christophe

0 Anshu Jain over 3 years ago in reply to Christophe ALLART

TI__Guru 56820 points

Hi Christophe,

Thanks for experimenting and confirming the issue. Can you share a small model ( even toy model is fine ) where the issue is reproducible?

Regards,

Anshu

0 Christophe ALLART over 3 years ago in reply to Anshu Jain

Prodigy 135 points

Hi Anshu,

Sorry, but the model is not public, and i don't have the right to send it.

I can only share the following:

this pad has input vector float32[1,128,152,128] ( it's tflite NHWC )

output tensor float32[1,128,153,128]

and paddings :

[
    [
        0,
        0
    ],
    [
        0,
        0
    ],
    [
        0,
        1
    ],
    [
        0,
        0
    ]
]



Is the fact one dimension of output tensor is odd , may explain the problem. I don'y know.

This is not a critical problem for us as we have switched on another similar model without this pad.

Regards

0 Anand Pathak over 3 years ago in reply to Christophe ALLART

TI__Genius 9065 points

Hi Christophe,

Thanks for sharing the pad configuration and also for clarifying the criticality of the problem. We don't have any restrictions as such on pad layer for spatial padding, so we need to investigate the cause of failure.

Regards,

Anand

Processors

Processors forum

TDA4VM: C7 stall during TIDL inference, need reboot