CCS: Custom Application, Graph Verify getting hang on Target

Hi All, 

On debugging further, I found that the "vxVerifyGraph" is getting stuck inside this snippet.

            if(status == (vx_status)VX_SUCCESS)
            {
                /* Call target kernel init for each node
                 * This results in message coomunication with target kernels
                 * Memory gets allocated which MUST be free'ed via target kernel
                 * deinit.
                 * kernel deinit called during node release
                 */
                status = ownGraphNodeKernelInit(graph);   //stuck inside this function call
                if(status != (vx_status)VX_SUCCESS)
                {
                    VX_PRINT(VX_ZONE_ERROR,"Node kernel init failed\n");
                }
            }

On further debugging of the function "ownGraphNodeKernelInit", I am seeing that this for loop is the location, where it gets stuck.

        for(i=0; i<graph->num_nodes; i++)
        {
            node = graph->nodes[i];
            printf("In the ownGraphNodeKernelInit loop node %d\n",i);
            if(node && node->kernel)
            {
                status = ownNodeKernelInit(node);
                VX_PRINT(VX_ZONE_INFO, "kernel init for node %d, kernel %s ...\n", i, node->kernel->name);
            }
            printf("In the ownNodeKernelInit %d\n",i);
            if(status != (vx_status)VX_SUCCESS )
            {
                VX_PRINT(VX_ZONE_ERROR,"kernel init for node %d, kernel %s ... failed !!!\n", i, node->kernel->name);
                break;
            }

            if(node && node->kernel)
            {
                VX_PRINT(VX_ZONE_INFO, "kernel init for node %d, kernel %s ... done !!!\n", i, node->kernel->name);
            }
        }

UART Log for this for loop shows:

In the ownGraphNodeKernelInit loop node 0
In the ownNodeKernelInit 0
In the ownGraphNodeKernelInit loop node 1
In the ownNodeKernelInit 1
In the ownGraphNodeKernelInit loop node 2

Please give us some pointers on what to look for this issue.

Hello Anshuman,

Could you please add a few additional debug statements as listed below?

Please put a few print statements inside tiovx/source/framework/vx_node.c inside the function ownNodeKernelInit in the following locations:

1. At the very beginning of this function

2. before and after the callback function below.  This callback occurs in two places inside ownNodeKernelInit

status = node->kernel->initialize(node, node->parameters,
node->kernel->signature.num_parameters);

3. At the very end of this function

Please send me the results of this experiment.

Regards,

Lucas

Dear Lucas,

Verify graph is parsing properly, but we tried to debug more to find out the issue.

So we tried to follow your suggestions given to Sagar in the thread (https://e2e.ti.com/support/processors/f/791/t/914156)

To put print inside the "tivxAddTargetKernelDrawBoxDetections "  and  "tivxKernelDrawBoxDetectionsCreate" to know if the kernel was registered using the tivxRegister call and also if the communication between the cores is working properly.

We found that control goes inside the function call " tivxKernelDrawBoxDetectionsCreate "  but didn't goes inside the function call "tivxAddTargetKernelDrawBoxDetections".

Please find the code below with the prints added

static vx_status VX_CALLBACK tivxKernelDrawBoxDetectionsCreate
(
  tivx_target_kernel_instance kernel,
  tivx_obj_desc_t *obj_desc[],
  vx_uint16 num_params,
  void *priv_arg
)
{
  printf("I am inside tivxKernelDrawBoxDetectionsCreate\n");
  vx_status status = VX_SUCCESS;
  uint32_t i;
    //printf("I am in tivxkernel Draw Detection Create \n");
  for (i = 0U; i < num_params; i ++)
  {
      if (NULL == obj_desc[i])
      {
            printf("NUll param Create %d\n",i);
          status = VX_FAILURE;
          break;
      }
  }
  printf("I am going out from tivxKernelDrawBoxDetectionsCreate\n");
  return(status);
}

void tivxAddTargetKernelDrawBoxDetections()
{
    char target_name[TIVX_TARGET_MAX_NAME];
    vx_enum self_cpu;

    self_cpu = tivxGetSelfCpuId();

    if ((self_cpu == TIVX_CPU_ID_DSP1) || (self_cpu == TIVX_CPU_ID_DSP2) || (self_cpu == TIVX_CPU_ID_A72_0))
    {
        printf("tivxAddTargetKernelDrawBoxDetections::first if condition\n");
        if (self_cpu == TIVX_CPU_ID_DSP1)
        {
            printf("tivxAddTargetKernelDrawBoxDetections::DSP1\n");
            strncpy(target_name, TIVX_TARGET_DSP1, TIVX_TARGET_MAX_NAME);
        }
        else if (self_cpu == TIVX_CPU_ID_DSP2)
        {
            printf("tivxAddTargetKernelDrawBoxDetections::DSP2\n");
            strncpy(target_name, TIVX_TARGET_DSP2, TIVX_TARGET_MAX_NAME);
        }
        else if (self_cpu == TIVX_CPU_ID_A72_0)
        {
            printf("tivxAddTargetKernelDrawBoxDetections::A72_0\n");
            strncpy(target_name, TIVX_TARGET_A72_0, TIVX_TARGET_MAX_NAME);
        }

        vx_DrawBoxDetections_kernel = tivxAddTargetKernelByName
                                     (
                                        TIVX_KERNEL_DRAW_BOX_DETECTIONS_NAME,
                                        target_name,
                                        tivxKernelDrawBoxDetectionsProcess,
                                        tivxKernelDrawBoxDetectionsCreate,
                                        tivxKernelDrawBoxDetectionsDelete,
                                        NULL,
                                        NULL
                                     );
    }
    printf("tivxAddTargetKernelDrawBoxDetections::Outside If Condition\n");
}

Please find console output too:

root@j7-evm:/opt/vision_apps# ./vx_app_tidl_od.out --cfg app_dasnetod.cfg 
APP: Init ... !!!
APP_LOG: Mapping 0xac000000 ...
APP_LOG: Mapped 0xac000000 -> 0xffffb4fc0000 of size 262144 bytes 
MEM: Init ION ... !!!
MEM: Initialized ION (fd=4) !!!
MEM: Init ION ... Done !!!
IPC: Init ... !!!
APP_LOG: Mapping 0xac040000 ...
APP_LOG: Mapped 0xac040000 -> 0xffffb3020000 of size 33161216 bytes 
APP_LOG: Mapping 0x30e00000 ...
APP_LOG: Mapped 0x30e00000 -> 0xffffb54b0000 of size 3072 bytes 
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
APP: Init ... Done !!!
     0.009780 s:  VX_ZONE_INIT:Enabled
     0.009792 s:  VX_ZONE_ERROR:Enabled
     0.009797 s:  VX_ZONE_WARNING:Enabled
     0.010212 s:  VX_ZONE_INIT:[tivxInit:64] Initialization Done !!!
     0.010355 s:  VX_ZONE_INIT:[tivxHostInit:44] Initialization Done for HOST !!!
sithara start
Default param set! 
Parsed user params! 
Updated user params! 
Scaler Init Done! 
Computing checksum1234 at 0x0000FFFFB0711740, size = 579328
TIDL Init Done! 
Pre Proc Update Done! 
Pre Proc Init Done! 
Draw detections Update Done! 
Draw Detections Init Done! 
Img Mosaic Init Done! 
Display Init Done! 
App Init Done! 
graph Init Done! 
scalar create Done! 
Preproc create Done! 
TIDL Create Done! 
Draw detection node call
Draw Detection Set node target
replicate
draw Create Done! 
mosaic Create Done! 
Display Create Done! 
App Create Graph Done! 
[C6x_1 ]     42.756835 s: I am inside tivxKernelImgPrePRocCreate
[C6x_1 ]     42.756882 s: I am going outside tivxKernelImgPrePRocCreate
[C6x_2 ]     42.791555 s: I am inside tivxKernelDrawBoxDetectionsCreate
[C6x_2 ]     42.791589 s: I am going out from tivxKern[   41.662666] EXT4-fs error (device mmcblk1p2): ext4_validate_block_bitmap:376: comm vx_app_tidl_od.: bg 104: bad block bitmap checksum
elDrawBoxDetectionsCreate
[C7x_1 ]     42.732728 s:  VX_ZONE_WARNING:[tivxKernelTIDLCreate:603] All Interrupts DISABLED during TIDL process
App Verify Graph Done!
App Send MSC Command Done!
App Verify Graph Done! 
verify_graph_done
Running frame 0
0
im here inside pipeline
pipeline less than 0

 =========================
 TIDL Demo - Object Detection
 =========================

 p: Print performance statistics

 x: Exit

 Enter Choice: App Reading Input Done!
App Process Graph Done!
process done
pipeline
app_tidl_od: Frame ID 0 of 1 ... Done.
app_tidl_od123: Iteration 0 of 1000000000 ... Done.
doneRunning frame 0
0
im here inside pipeline
pipeline less than 0App Reading Input Done!
pipeline greatet than 0App Reading Input Done!
App Process Graph Done!
process done
pipeline
app_tidl_od: Frame ID 0 of 1 ... Done.
app_tidl_od123: Iteration 1 of 1000000000 ... Done.
doneRunning frame 0
0
im here inside pipeline
pipeline greatet than 0App Reading Input Done!
App Process Graph Done!
process done
pipeline
app_tidl_od: Frame ID 0 of 1 ... Done.
app_tidl_od123: Iteration 2 of 1000000000 ... Done.
doneRunning frame 0
0
im here inside pipeline
[C6x_1 ]     43.010890 s: Preproc Kernel 608 344 640 608
[C6x_1 ]     43.010913 s: I am in c66 kernel for YUVTORGB
[C6x_1 ]     43.013909 s: Preproc Kernel 608 344 640 608
[C6x_1 ]     43.013936 s: I am in c66 kernel for YUVTORGB
pipeline greatet than 0App Reading Input Done!
App Process Graph Done!
process done
pipeline
app_tidl_od: Frame ID 0 of 1 ... Done.
app_tidl_od123: Iteration 3 of 1000000000 ... Done.
doneRunning frame 0
0
im here inside pipeline
pipeline greatet than 0App Reading Input Done!
App Process Graph Done!
process done
pipeline
app_tidl_od: Frame ID 0 of 1 ... Done.
app_tidl_od123: Iteration 4 of 1000000000 ... Done.
doneRunning frame 0
0
im here inside pipeline
[C6x_1 ]     43.030137 s: Preproc Kernel 608 344 640 608
[C6x_1 ]     43.030161 s: I am in c66 kernel for YUVTORGB
[C6x_2 ]     42.982890 s: input image pointer read with size 926720 463360
[C6x_2 ]     42.983022 s: output image pointer read with size 926720 463360
[C6x_2 ]     42.986084 s: memcpy done
pipeline greatet than 0App Reading Input Done!
App Process Graph Done!
process done
pipeline
app_tidl_od: Frame ID 0 of 1 ... Done.
app_tidl_od123: Iteration 5 of 1000000000 ... Done.
doneRunning frame 0
0
im here inside pipeline
[  129.188377] NET: Registered protocol family 15
[  129.230683] Initializing XFRM netlink socket
[C6x_1 ]    262.265837 s: Preproc Kernel 608 344 640 608
[C6x_1 ]    262.265863 s: I am in c66 kernel for YUVTORGB
[C6x_2 ]    262.218583 s: input image pointer read with size 926720 463360
[C6x_2 ]    262.218714 s: output image pointer read with size 926720 463360
[C6x_2 ]    262.221863 s: memcpy done
pipeline greatet than 0App Reading Input Done!
App Process Graph Done!
process done
pipeline
app_tidl_od: Frame ID 0 of 1 ... Done.
app_tidl_od123: Iteration 6 of 1000000000 ... Done.
doneRunning frame 0
0
im here inside pipeline

Kindly do the needful.

Regards,

Vyom Mishra

Hello Vyom,

It looks like the issue with verify graph has been resolved.  Are there any other further issues?

The print inside the tivxAddTargetKernelDrawBoxDetections should be seen prior to running the use case.  You should see it when you run the "vision_apps_init.sh" script.  If this was not hit, the app would not work, so I am guessing that this was probably hit above.

Regards,

Lucas