• State TI Thinks Resolved
  • Locked Locked
  • Replies 4 replies
  • Answers 1 answer
  • Subscribers 97 subscribers
  • Views 170 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-AM62A-LP: Unstable tidl rt initialization

Stanislav Lisetskii
Prodigy 40 points

Part Number: SK-AM62A-LP

Hi, my sdk version is 11_01_07_05.

We chose tidl_rt to run our production model because it's based on one of your yolox examples and is fully offloaded to the C7 accelerator.

Now, I've been experiencing strange initialization issues when debugging. I usually just terminate the process with ctrl+c in cli, or kill with shift+f5 in VS Code.

Now after a few runs it stops initializing, 
There's an error that I cant remember right now, because after a recent code change the app started working exactly once.

it either hangs in TIDLRT_create() call and the last log is see:
  6330.365374 s:  VX_ZONE_INFO: [tivxInitLocal:202] Initialization Done !!!
  6330.365453 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO

or it shows error:

The futex facility returned an unexpected error code and TIDLRT_create() returns a null handle.

I'll try to repro the 1st error for you. It usually starts appearing on approx 5th run of the app.

And only reboot helps to reset this. Is there maybe some background process that I can kill or, or de-initialize the tidlrt hande, or I need to do to reset the driver or something like that.

 
  • 0
    Prodigy 40 points

    upd: I rolled back todays changes and am able to repro the error. It happened on 7th run (previous runs were working well)

    7310.580709 s: VX_ZONE_INFO: [tivxInitLocal:202] Initialization Done !!!
    7310.581712 s: VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
    7310.654628 s: VX_ZONE_ERROR: [ownContextSendCmd:1001] Command ack message returned failure cmd_status: -1
    7310.654700 s: VX_ZONE_ERROR: [ownNodeKernelInit:704] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
    7310.654745 s: VX_ZONE_ERROR: [ownNodeKernelInit:705] Please be sure the target callbacks have been registered for this core
    7310.654778 s: VX_ZONE_ERROR: [ownNodeKernelInit:706] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
    7310.654813 s: VX_ZONE_ERROR: [ownGraphNodeKernelInit:793] kernel init for node 0, kernel com.ti.tidl:1:3 ... failed !!!
    7310.654969 s: VX_ZONE_ERROR: [ TIDL subgraph stride_8 ] Node kernel init failed
    7310.655041 s: VX_ZONE_ERROR: [ TIDL subgraph stride_8 ] Graph verify failed

  • 0 in reply to Stanislav Lisetskii
    TI__Expert 3980 points

    Hi Stanislav,

    Can you try resetting the C7x core itself? You can do that using remoteproc:

    echo stop > /sys/class/remoteproc/remoteproc0/state
echo start > /sys/class/remoteproc/remoteproc0/state

    If this doesn't work, run the following command in the background and then run your application. This should give more verbose logging that might help point in the right direction.

    /opt/vx_app_arm_remote_log.out &

    Regards,
    Jay

  • 0 in reply to Jay Goyal
    Prodigy 40 points

    Hello Jay!

    These are useful things to know, thank you. The logs did reveal that it's a memory allocation failure.

    [C7x_1 ] 2421.918127 s: VX_ZONE_ERROR: [tivxAlgiVisionAllocMem:194] Failed to Allocate memory record 15 @ space = 17 and size = 8964480 !!!
    [C7x_1 ] 2421.918166 s: VX_ZONE_ERROR: [tivxAlgiVisionCreate:358] tivxAlgiVisionAllocMem Failed
    [C7x_1 ] 2421.918199 s: VX_ZONE_ERROR: [tivxKernelTIDLCreate:1131] tivxAlgiVisionCreate returned NULL

    This does get resolved if I reset the C7x with the commands you gave.

    Now the question is - why do memory mappings persist when the app quits? I could run the reset automatically every time I launch the debug session, no problem. But, I just want to understand why this is happening to avoid potential issues in the production releases.

    Also the futex/hang thing was just me messing up the output tensor memory allocations, so it's unrelated.

  • 0 in reply to Stanislav Lisetskii
    TI__Expert 3980 points

    Hi Stanislav,

    The issue is with the TIDL kernel not de-initializing properly. Can you clarify what your application is? Are you using this binary: TI_DEVICE_armv8_test_dl_algo_host_rt.out or are you writing your own application?

    In the first case, the signal handlers for Ctrl+C are not present in SDK 11.1 and this is to be expected. However, it will de-initialize everything correctly once the specified number of iterations finish.

    In case you are writing your own application, then you need to call TIDLRT_delete and TIDLRT_freeSharedMem in the signal handler for Ctrl+C.

    Regards,
    Jay