This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: Trying to run basic tfl_classification demo with acceleration

Part Number: TDA4VM

I'm trying to run a basic tfl_classification demo from the edgeai-tidl-tools repo (checked out to commit-ID 4f870c5, tagged release 08.00.01.10) on a Jacinto development board target with acceleration (through TFL delegate) enabled.

I am at the point where I can compile models on PC using the Python examples, copy them over to the board, and run the same demo successfully with acceleration disabled, as below:

root@j7-evm:/opt/edgeai-tidl-tools# ./bin/Debug/tfl_clasification -m models/public/tflite/mobilenet_v1_1.0_224.tflite -l test_data/labels.txt -i test_data/airshow.jpg -f model-artifacts/tfl/mobilenet_v1_1.0_224/ -c 1
Loaded model models/public/tflite/mobilenet_v1_1.0_224.tflite
resolved reporter
interpreter->Invoke - Started
interpreter->Invoke - Done
invoked
average time: 166.615 ms
0.43202: 896 warplane, military plane
0.354699: 658 missile
0.198914: 745 projectile, missile
0.0127183: 404 aircraft carrier, carrier, flattop, attack aircraft carrier

... but when I enable acceleration, I get poor classification results and a whole bunch of error printouts:

root@j7-evm:/opt/edgeai-tidl-tools# ./bin/Debug/tfl_clasification -m models/public/tflite/mobilenet_v1_1.0_224.tflite -l test_data/labels.txt -i test_data/airshow.jpg -f model-artifacts/tfl/mobilenet_v1_1.0_224/ -a 1 -d 1 -c 1
Loaded model models/public/tflite/mobilenet_v1_1.0_224.tflite
resolved reporter
ModifyGraphWithDelegate - Done

Number of subgraphs:1 , 31 nodes delegated out of 31 nodes

Calling appInit() in TIDL-RT!
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
25925.715089 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
25925.715444 s: VX_ZONE_INIT:Enabled
25925.715541 s: VX_ZONE_ERROR:Enabled
25925.715686 s: VX_ZONE_WARNING:Enabled
25925.716381 s: VX_ZONE_INIT:[tivxInitLocal:111] Initialization Done !!!
25925.716720 s: VX_ZONE_INIT:[tivxHostInitLocal:86] Initialization Done for HOST !!!
25925.731497 s: VX_ZONE_ERROR:[ownContextSendCmd:815] Command ack message returned failure cmd_status: -1
25925.731524 s: VX_ZONE_ERROR:[ownContextSendCmd:851] tivxEventWait() failed.
25925.731537 s: VX_ZONE_ERROR:[ownNodeKernelInit:538] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
25925.731546 s: VX_ZONE_ERROR:[ownNodeKernelInit:539] Please be sure the target callbacks have been registered for this core
25925.731554 s: VX_ZONE_ERROR:[ownNodeKernelInit:540] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
25925.731564 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl ... failed !!!
25925.731718 s: VX_ZONE_ERROR:[vxVerifyGraph:2044] Node kernel init failed
25925.731727 s: VX_ZONE_ERROR:[vxVerifyGraph:2098] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
interpreter->Invoke - Started
25925.747306 s: VX_ZONE_ERROR:[ownContextSendCmd:815] Command ack message returned failure cmd_status: -1
25925.747331 s: VX_ZONE_ERROR:[ownContextSendCmd:851] tivxEventWait() failed.
25925.747343 s: VX_ZONE_ERROR:[ownNodeKernelInit:538] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
25925.747351 s: VX_ZONE_ERROR:[ownNodeKernelInit:539] Please be sure the target callbacks have been registered for this core
25925.747360 s: VX_ZONE_ERROR:[ownNodeKernelInit:540] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
25925.747370 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl ... failed !!!
25925.747381 s: VX_ZONE_ERROR:[vxVerifyGraph:2044] Node kernel init failed
25925.747389 s: VX_ZONE_ERROR:[vxVerifyGraph:2098] Graph verify failed
25925.747505 s: VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:820] graph is not in a state required to be scheduled
25925.747515 s: VX_ZONE_ERROR:[vxProcessGraph:755] schedule graph failed
25925.747528 s: VX_ZONE_ERROR:[vxProcessGraph:760] wait graph failed
interpreter->Invoke - Done
invoked
average time: 9.424 ms
-nan: 944 cucumber, cuke
-nan: 941 spaghetti squash
-nan: 903 whistle
-nan: 893 wall clock
-nan: 886 velvet
Calling appDeInit() in TIDL-RT!
25925.748854 s: VX_ZONE_INIT:[tivxHostDeInitLocal:100] De-Initialization Done for HOST !!!
25925.753212 s: VX_ZONE_INIT:[tivxDeInitLocal:155] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
MEM: Alloc's: 9 alloc's of 5894221 bytes
MEM: Free's : 9 free's of 5894221 bytes
MEM: Open's : 0 allocs of 0 bytes
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

Have I messed up my setup somehow? Can anyone advise how I might debug this issue further?

  • Following up on this, I see very similar results out of the tidlrt_classification sample application:

    root@j7-evm:/opt/edgeai-tidl-tools# ./bin/Debug/tidlrt_clasification -l test_data/labels.txt -i test_data/airshow.jpg -f model-artifacts/tfl/mobilenet_v1_1.0_224/ -d 1
    Model Files names : model-artifacts/tfl/mobilenet_v1_1.0_224//86_tidl_net.bin,model-artifacts/tfl/mobilenet_v1_1.0_224//86_tidl_io_1.bin
    Calling appInit() in TIDL-RT!
    APP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=4) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
    26672.336499 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
    26672.336559 s: VX_ZONE_INIT:Enabled
    26672.336569 s: VX_ZONE_ERROR:Enabled
    26672.336577 s: VX_ZONE_WARNING:Enabled
    26672.337070 s: VX_ZONE_INIT:[tivxInitLocal:111] Initialization Done !!!
    26672.337264 s: VX_ZONE_INIT:[tivxHostInitLocal:86] Initialization Done for HOST !!!
    26672.349805 s: VX_ZONE_ERROR:[ownContextSendCmd:815] Command ack message returned failure cmd_status: -1
    26672.349831 s: VX_ZONE_ERROR:[ownContextSendCmd:851] tivxEventWait() failed.
    26672.349856 s: VX_ZONE_ERROR:[ownNodeKernelInit:538] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
    26672.349875 s: VX_ZONE_ERROR:[ownNodeKernelInit:539] Please be sure the target callbacks have been registered for this core
    26672.349892 s: VX_ZONE_ERROR:[ownNodeKernelInit:540] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
    26672.349911 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl ... failed !!!
    26672.349936 s: VX_ZONE_ERROR:[vxVerifyGraph:2044] Node kernel init failed
    26672.349953 s: VX_ZONE_ERROR:[vxVerifyGraph:2098] Graph verify failed
    TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
    TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
    invoked
    26672.365283 s: VX_ZONE_ERROR:[ownContextSendCmd:815] Command ack message returned failure cmd_status: -1
    26672.365308 s: VX_ZONE_ERROR:[ownContextSendCmd:851] tivxEventWait() failed.
    26672.365319 s: VX_ZONE_ERROR:[ownNodeKernelInit:538] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
    26672.365328 s: VX_ZONE_ERROR:[ownNodeKernelInit:539] Please be sure the target callbacks have been registered for this core
    26672.365336 s: VX_ZONE_ERROR:[ownNodeKernelInit:540] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
    26672.365345 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl ... failed !!!
    26672.365356 s: VX_ZONE_ERROR:[vxVerifyGraph:2044] Node kernel init failed
    26672.365364 s: VX_ZONE_ERROR:[vxVerifyGraph:2098] Graph verify failed
    26672.365458 s: VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:820] graph is not in a state required to be scheduled
    26672.365467 s: VX_ZONE_ERROR:[vxProcessGraph:755] schedule graph failed
    26672.365475 s: VX_ZONE_ERROR:[vxProcessGraph:760] wait graph failed
    average time: 9.143 ms
    -nan: 944 cucumber, cuke
    -nan: 941 spaghetti squash
    -nan: 903 whistle
    -nan: 893 wall clock
    -nan: 886 velvet
    Calling appDeInit() in TIDL-RT!
    26672.366571 s: VX_ZONE_INIT:[tivxHostDeInitLocal:100] De-Initialization Done for HOST !!!
    26672.370932 s: VX_ZONE_INIT:[tivxDeInitLocal:155] De-Initialization Done !!!
    APP: Deinit ... !!!
    REMOTE_SERVICE: Deinit ... !!!
    REMOTE_SERVICE: Deinit ... Done !!!
    IPC: Deinit ... !!!
    IPC: DeInit ... Done !!!
    MEM: Deinit ... !!!
    MEM: Alloc's: 9 alloc's of 5894221 bytes
    MEM: Free's : 9 free's of 5894221 bytes
    MEM: Open's : 0 allocs of 0 bytes
    MEM: Deinit ... Done !!!
    APP: Deinit ... Done !!!
    MEM: WARNING: Free'ing more memory than allocated
    double free or corruption (fasttop)
    Aborted (core dumped)

  • yep, I can confirm that upgrading the SD image to SDK release 8.1 and updating edgeai-tidl-tools to the top of master has solved this problem. I can compile models and run the demos with acceleration enabled now.