Original question:
Tool/software:
Hello, I have some questions to ask you. I am currently learning the edgeai-tiovx-apps project. I hope to use tiovx and yolov5 to infer an image, but after reading the code, I did not find any code related to model inference and result post-processing. Can you give me some tips? The second question is, if I have a yolov5 model, how can I turn it into a .bin file that can be used on the board? I hope to get your reply. Thank you.
best wishes
zhuang yh
Hello,
Yes, edgeai-tiovx-apps is the right place to start for this for this. There are examples in this repo for using deep learning models and postprocessing.
For a standalone example, look in the tests/ directory for the repo
And the main apps leverages 'deep_learning_block.c' to handle the model pre/post processing and DL itself
The second question is, if I have a yolov5 model, how can I turn it into a .bin file that can be used on the board?
When you compile a network, it produces artifacts that include a _net.bin and io_1.bin pair of binaries. This is all you need to run the model with acceleration. Open source runtimes require the original network file (e.g. yolov5.onnx) but the TIOVX application use the TIDL_RT software directly, which only needs the two binaries.
BR,
Reese
Hello, Reese, I am now confused when using the tiovx tool. My idea is to detect the target in an image based on tiovx, and then obtain the location information (four coordinate points) and confidence of the detection results. However, I did not find relevant code in the edgeai-tiovx-apps project code. Can you provide me with a simple demo of target detection pre-processing, post-processing specific calls and result acquisition? Thank you very much
best wishes
zhuangyihao
Hello,
We do not have a singular demo that includes object detection within one file. The closest to a standalone object detection application is an image classification application under tests/app_tiovx_dl_pipeline_test.c.
Edgeai-tiovx-apps is organized as a framework that can build a TIOVX application based on a YAML configuration file. The deep_learning_block.c will put together the set of nodes and configuration parameters for preprocessing, TIDL inference, and postprocessing TIOVX nodes. The source for the nodes themselves will include that pre/post processing code and TIDL:
The format of data coming out of the TIDL node will always be model dependent. User is expected to know the input and output dimensions of their model. The deep_learning_block.c can be used as a reference for how this is read from most TI models compiled with edgeai-tidl-tools (typically reads a param.yaml file with pre/post proc information).
At runtime, the TIOVX framework will implicitly pass data from one node to another. At an application level, this is not exposed until they reach an output of the overall graph. In edgeai-tiovx-apps, we reference a buffer_pool attached to a node input/output, and either enqueue or dequeue buffers from it to get data out of the graph. Some of the standalone tests are the best code reference to see this, since the additional complexity / abstraction of the main tiovx-apps is not present in tests.
BR,
Reese
Hello，Reese.
Thank you for your reply. I will read the code you mentioned below again. I have another question. I need to perform target tracking processing (for example, ByteTrack) on all detected targets after target detection based on tiovx. Therefore, I need to obtain the location information (x1, y1, x2, y2) and confidence of the detection results for subsequent target tracking. However, if I follow the yaml configuration, I can only save a picture with a rectangular frame drawn. How can I get the specific information of all detection results, especially the coordinates and confidence of each target for subsequent tracking processing? I look forward to your answer.
BR,
zhuangyh
Hello,
However, if I follow the yaml configuration, I can only save a picture with a rectangular frame drawn. How can I get the specific information of all detection results, especially the coordinates and confidence of each target for subsequent tracking processing? I look forward to your answer.
The YAML-based applications are meant as a starting place. For application-specific tasks, this will require some modification.
The information (box coords, conf value) you are looking for would become available within the tiovx_dl_post_proc_module.c, but specifically within the kernel. Those kernels are programmed in edgeai-tiovx-kernels
You could handle your app-specific processing within the kernel. Tracking across frames may require static variables or otherwise to save data between frames. Alternatively, you can build the graph similar to examples under tests/ and consume output from the TIDL node directly such that the information is processed outside the TIOVX graph in your application code.
If you update the kernels, you will need to rebuild those. You rebuild on target with edgeai-gst-apps/scripts/install_tiovx_kernels.sh or cross compile with edgeai-app-stack
BR,
Reese
Hello, Reese，
you said " Alternatively, you can build the graph similar to examples under tests/ and consume output from the TIDL node directly such that the information is processed outside the TIOVX graph in your application code" , this method you mentioned may be more suitable for my needs, how should I modify the process in the demo to consume output from the TIDL node directly? For example, in the demo edgeai—tiovx-apps/tests/app_tiovx_capture_dl_display_pipeline.c, how should I modify the code to directly obtain the output of TIDL?
Hello,
If you want the output from TIDL directly, then you should remove any nodes after the tidl_node. The first node after TIDL is the TIOVXDLPostProc. Make sure that these two TIOVX modules are not 'linked' (function tiovx_modules_link_pads). If you want output directly from TIDL, then make sure the TIDL node/module does not link its output to another node/module -- otherwise, the outputs from TIDL will automatically be passed to the next node in the graph.
To retrieve the output from the TIDL node, you will need to dequeue a buffer from the src(s) of the TIDL node. Note that if your model has multiple outputs, then there should be an equivalent number of outputs.
To use an node's outputs this way, you must:
There are several examples of this behavior; in the app_tiovx_capture_dl_display_pipeline.c file, see the bottom of the main function where this strategy is applied to the inputs to the graph -- this should be applied to the output instead.
As a note on naming conventions:
BR,
Reese
Hello Reese, thanks for you guidance
Thank you for your guidance. Following the steps you mentioned, I made some modifications to app_tiovx_capture_dl_display_pipeline.c, including commenting out the nodes after the tidl node and directly obtaining the output of the tidl stage. The attached is the demo I modified. In line 318, I obtained the output of the TILD node and saved it in output_pad. Then in line 435, I obtained the result of output_pad. Is this correct? If it is correct, how should I process this Buf type data in the future to obtain the information of the detection result?
/*
*
* Copyright (c) 2024 Texas Instruments Incorporated
*
* All rights reserved not granted herein.
*
* Limited License.
*
* Texas Instruments Incorporated grants a world-wide, royalty-free, non-exclusive
* license under copyrights and patents it now or hereafter owns or controls to make,
* have made, use, import, offer to sell and sell ("Utilize") this software subject to the
* terms herein. With respect to the foregoing patent license, such license is granted
* solely to the extent that any such patent is necessary to Utilize the software alone.
* The patent license shall not apply to any combinations which include this software,
* other than combinations with devices manufactured by or for TI ("TI Devices").
* No hardware patent is licensed hereunder.
*
* Redistributions must preserve existing copyright notices and reproduce this license
* (including the above copyright notice and the disclaimer and (if applicable) source
* code license limitations below) in the documentation and/or other materials provided
* with the distribution
*
* Redistribution and use in binary form, without modification, are permitted provided
* that the following conditions are met:
*
* * No reverse engineering, decompilation, or disassembly of this software is
* permitted with respect to any software provided in binary form.
*
* * any redistribution and use are licensed by TI for use only with TI Devices.
*
* * Nothing shall obligate TI to provide you with source code for the software
* licensed and provided to you in object code.
*
* If software source code is provided to you, modification and redistribution of the
* source code are permitted provided that the following conditions are met:
*
* * any redistribution and use of the source code, including any resulting derivative
* works, are licensed by TI for use only with TI Devices.
*
* * any redistribution and use of any object code compiled from the source code
* and any resulting derivative works, are licensed by TI for use only with TI Devices.
*
* Neither the name of Texas Instruments Incorporated nor the names of its suppliers
*
* may be used to endorse or promote products derived from this software without
* specific prior written permission.
*
* DISCLAIMER.
*
* THIS SOFTWARE IS PROVIDED BY TI AND TI'S LICENSORS "AS IS" AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
* IN NO EVENT SHALL TI AND TI'S LICENSORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
* OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
/*
* Pipeline:
*
* Capture(IMX390 2MP) --> VISS --> LDC --> MSC0 --> MSC1 --> DL_PRE_PROC --> TIDL
* | |
* | --> DL_POST_PROC --> Display
* | |
* ---------------------------------->
*/
#include <tiovx_modules.h>
#include <tiovx_utils.h>
#include <stdio.h>
#include <unistd.h>
#include <v4l2_capture_module.h>
#define NUM_ITERATIONS (300)
#define APP_BUFQ_DEPTH (4)
#define APP_NUM_CH (1)
#define VISS_INPUT_WIDTH 1920
#define VISS_INPUT_HEIGHT 1536
#define VISS_OUTPUT_WIDTH (VISS_INPUT_WIDTH)
#define VISS_OUTPUT_HEIGHT (VISS_INPUT_HEIGHT)
#define LDC_INPUT_WIDTH VISS_OUTPUT_WIDTH
#define LDC_INPUT_HEIGHT VISS_OUTPUT_HEIGHT
#define LDC_OUTPUT_WIDTH 1920
#define LDC_OUTPUT_HEIGHT 1080
#define LDC_TABLE_WIDTH (1920)
#define LDC_TABLE_HEIGHT (1080)
#define LDC_DS_FACTOR (2)
#define LDC_BLOCK_WIDTH (64)
#define LDC_BLOCK_HEIGHT (32)
#define LDC_PIXEL_PAD (1)
#define SENSOR_NAME "X3F"
#define DCC_VISS "/opt/imaging/ox03f/dcc_viss.bin"
#define DCC_LDC "/opt/imaging/ox03f/dcc_ldc.bin"
#define POST_PROC_OUT_WIDTH (1280)
#define POST_PROC_OUT_HEIGHT (720)
#define TIDL_IO_CONFIG_FILE_PATH "/opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416/artifacts/detslabels_tidl_io_1.bin"
#define TIDL_NETWORK_FILE_PATH "/opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416/artifacts/detslabels_tidl_net.bin"
static char imgnet_labels3[TIVX_DL_POST_PROC_MAX_NUM_CLASSNAMES][TIVX_DL_POST_PROC_MAX_SIZE_CLASSNAME] =
{
"Not Available"
};
typedef struct
{
kmsDisplayHandle * kms_handle;
v4l2CaptureHandle* capture_handle;
Pad* pad;
bool enable;
}threadParams;
static int32_t label_offset3[TIVX_DL_POST_PROC_MAX_NUM_CLASSNAMES] = {0};
vx_status app_modules_capture_dl_display_test(int argc, char* argv[])
{
vx_status status = VX_FAILURE;
GraphObj graph;
Pad *input_pad = NULL, *output_pad[4] = {NULL, NULL, NULL, NULL}, *post_proc_in_img_pad = NULL;
Buf *inbuf = NULL;
status = tiovx_modules_initialize_graph(&graph);
graph.schedule_mode = VX_GRAPH_SCHEDULE_MODE_QUEUE_AUTO;
/* Capture */ //改为 v4l2 capture
// TIOVXCaptureNodeCfg capture_cfg;
// NodeObj *capture_node;
// tiovx_capture_init_cfg(&capture_cfg);
// capture_cfg.ch_mask = 1;
// capture_node = tiovx_modules_add_node(&graph, TIOVX_CAPTURE, (void *)&capture_cfg);
// input_pad = &capture_node->srcs[0];
v4l2CaptureHandle* v4l2_capture_handle;
v4l2CaptureCfg v4l2_capture_cfg;
v4l2_capture_init_cfg(&v4l2_capture_cfg);
v4l2_capture_cfg.width = 1920;
v4l2_capture_cfg.height = 1536;
v4l2_capture_cfg.pix_format = V4L2_PIX_FMT_SBGGR12;
v4l2_capture_cfg.bufq_depth = 4;
sprintf(v4l2_capture_cfg.device, "/dev/video3");
v4l2_capture_handle = v4l2_capture_create_handle(&v4l2_capture_cfg);
status = tiovx_modules_initialize_graph(&graph);
graph.schedule_mode = VX_GRAPH_SCHEDULE_MODE_QUEUE_AUTO;
/* TEE */
{
// TIOVXTeeNodeCfg tee_cfg;
// NodeObj *tee_node;
// tiovx_tee_init_cfg(&tee_cfg);
// tee_cfg.peer_pad = input_pad;
// tee_cfg.num_outputs = 2;
// tee_node = tiovx_modules_add_node(&graph, TIOVX_TEE, (void *)&tee_cfg);
// tee_node->srcs[0].bufq_depth = 4; /* This must be greater than 3 */
// input_pad = &tee_node->srcs[0]; // This is for enqueing
// output_pad[0] = &tee_node->srcs[1]; // This will go to viss
}
/* VISS */
TIOVXVissNodeCfg viss_cfg;
NodeObj *viss_node;
tiovx_viss_init_cfg(&viss_cfg);
sprintf(viss_cfg.sensor_name, SENSOR_NAME);
snprintf(viss_cfg.dcc_config_file, TIVX_FILEIO_FILE_PATH_LENGTH, "%s", DCC_VISS);
viss_cfg.width = VISS_INPUT_WIDTH;
viss_cfg.height = VISS_INPUT_HEIGHT;
sprintf(viss_cfg.target_string, TIVX_TARGET_VPAC_VISS1);
// viss_cfg.enable_h3a_pad = vx_true_e;
viss_cfg.input_cfg.params.format[0].pixel_container = TIVX_RAW_IMAGE_16_BIT;
viss_cfg.input_cfg.params.format[0].msb = 11;
viss_node = tiovx_modules_add_node(&graph, TIOVX_VISS, (void *)&viss_cfg);
/* Link TEE0 to VISS1 */
// tiovx_modules_link_pads(output_pad[0], &viss_node->sinks[0]);
viss_node->sinks[0].bufq_depth = APP_BUFQ_DEPTH;
/* AEWB */
{
// TIOVXAewbNodeCfg aewb_cfg;
// NodeObj *aewb_node;
// tiovx_aewb_init_cfg(&aewb_cfg);
// sprintf(aewb_cfg.sensor_name, SENSOR_NAME);
// aewb_cfg.ch_mask = 1;
// aewb_cfg.awb_mode = ALGORITHMS_ISS_AWB_AUTO;
// aewb_cfg.awb_num_skip_frames = 9;
// aewb_cfg.ae_num_skip_frames = 9;
// aewb_node = tiovx_modules_add_node(&graph, TIOVX_AEWB, (void *)&aewb_cfg);
// /* Link VISS to AEWB */
// tiovx_modules_link_pads(output_pad[1], &aewb_node->sinks[0]);
}
/* LDC */
TIOVXLdcNodeCfg ldc_cfg;
NodeObj *ldc_node;
tiovx_ldc_init_cfg(&ldc_cfg);
sprintf(ldc_cfg.sensor_name, SENSOR_NAME);
snprintf(ldc_cfg.dcc_config_file, TIVX_FILEIO_FILE_PATH_LENGTH, "%s", DCC_LDC);
// ldc_cfg.ldc_mode = TIOVX_MODULE_LDC_OP_MODE_DCC_DATA;
ldc_cfg.input_cfg.color_format = VX_DF_IMAGE_NV12;
ldc_cfg.input_cfg.width = LDC_INPUT_WIDTH;
ldc_cfg.input_cfg.height = LDC_INPUT_HEIGHT;
ldc_cfg.output_cfgs[0].color_format = VX_DF_IMAGE_NV12;
ldc_cfg.output_cfgs[0].width = LDC_OUTPUT_WIDTH;
ldc_cfg.output_cfgs[0].height = LDC_OUTPUT_HEIGHT;
// ldc_cfg.init_x = 0;
// ldc_cfg.init_y = 0;
// ldc_cfg.table_width = LDC_TABLE_WIDTH;
// ldc_cfg.table_height = LDC_TABLE_HEIGHT;
// ldc_cfg.ds_factor = LDC_DS_FACTOR;
// ldc_cfg.out_block_width = LDC_BLOCK_WIDTH;
// ldc_cfg.out_block_height = LDC_BLOCK_HEIGHT;
// ldc_cfg.pixel_pad = LDC_PIXEL_PAD;
sprintf(ldc_cfg.target_string, TIVX_TARGET_VPAC_LDC1);
ldc_node = tiovx_modules_add_node(&graph, TIOVX_LDC, (void *)&ldc_cfg);
ldc_node->sinks[0].bufq_depth = APP_BUFQ_DEPTH;
/* Link VISS0 to LDC */
tiovx_modules_link_pads(&viss_node->srcs[0], &ldc_node->sinks[0]);
/* MSC0 */
TIOVXMultiScalerNodeCfg msc_cfg;
NodeObj *msc_node;
tiovx_multi_scaler_init_cfg(&msc_cfg);
msc_cfg.color_format = VX_DF_IMAGE_NV12;
msc_cfg.num_outputs = 1;
msc_cfg.input_cfg.width = LDC_OUTPUT_WIDTH;
msc_cfg.input_cfg.height = LDC_OUTPUT_HEIGHT;
// This is for pre proc
msc_cfg.output_cfgs[0].width = 900;
msc_cfg.output_cfgs[0].height = 720;
sprintf(msc_cfg.target_string, TIVX_TARGET_VPAC_MSC1);
tiovx_multi_scaler_module_crop_params_init(&msc_cfg);
msc_node = tiovx_modules_add_node(&graph, TIOVX_MULTI_SCALER, (void *)&msc_cfg);
/* Link LDC to MSC0 */
tiovx_modules_link_pads(&ldc_node->sinks[0], &msc_node->sinks[0]);
/* MSC1 */
TIOVXMultiScalerNodeCfg msc_cfg1;
NodeObj *msc_node1;
tiovx_multi_scaler_init_cfg(&msc_cfg1);
msc_cfg1.color_format = VX_DF_IMAGE_NV12;
msc_cfg1.num_outputs = 1;
msc_cfg1.input_cfg.width = 900;
msc_cfg1.input_cfg.height = 720;
msc_cfg1.output_cfgs[0].width = 416;
msc_cfg1.output_cfgs[0].height = 416;
sprintf(msc_cfg1.target_string, TIVX_TARGET_VPAC_MSC1);
tiovx_multi_scaler_module_crop_params_init(&msc_cfg1);
msc_node1 = tiovx_modules_add_node(&graph, TIOVX_MULTI_SCALER, (void *)&msc_cfg1);
/* Link MSC0 to MSC1 */
tiovx_modules_link_pads(&msc_node->sinks[0], &msc_node1->sinks[0]);
/* DL Pre Proc */
TIOVXDLPreProcNodeCfg dl_pre_proc_cfg;
NodeObj *dl_pre_proc_node;
tiovx_dl_pre_proc_init_cfg(&dl_pre_proc_cfg);
dl_pre_proc_cfg.io_config_path = TIDL_IO_CONFIG_FILE_PATH;
dl_pre_proc_cfg.params.tensor_format = 1; //BGR
dl_pre_proc_node = tiovx_modules_add_node(&graph, TIOVX_DL_PRE_PROC, (void *)&dl_pre_proc_cfg);
/* Link MSC1 to DLPreProc */
tiovx_modules_link_pads(&msc_node1->sinks[0], &dl_pre_proc_node->sinks[0]);
/* TIDL */
TIOVXTIDLNodeCfg tidl_cfg;
NodeObj *tidl_node;
tiovx_tidl_init_cfg(&tidl_cfg);
tidl_cfg.io_config_path = TIDL_IO_CONFIG_FILE_PATH;
tidl_cfg.network_path = TIDL_NETWORK_FILE_PATH;
tidl_node = tiovx_modules_add_node(&graph, TIOVX_TIDL, (void *)&tidl_cfg);
/* Link DLPreProc to TIDL */
tiovx_modules_link_pads(&dl_pre_proc_node->sinks[0], &tidl_node->sinks[0]);
printf("tidl_node->num_outputs is %d \n",tidl_node->num_outputs);
for (int32_t i = 0; i < tidl_node->num_outputs; i++)
{
output_pad[i] = &tidl_node->srcs[i];
}
/*注释掉POST Proc和 Display*/
// /* DL Post Proc */
// {
// TIOVXDLPostProcNodeCfg dl_post_proc_cfg;
// NodeObj *dl_post_proc_node;
// tiovx_dl_post_proc_init_cfg(&dl_post_proc_cfg);
// dl_post_proc_cfg.width = POST_PROC_OUT_WIDTH;
// dl_post_proc_cfg.height = POST_PROC_OUT_HEIGHT;
// dl_post_proc_cfg.params.task_type = TIVX_DL_POST_PROC_DETECTION_TASK_TYPE;
// dl_post_proc_cfg.params.od_prms.viz_th = 0.5;
// dl_post_proc_cfg.params.od_prms.formatter[0] = 0;
// dl_post_proc_cfg.params.od_prms.formatter[1] = 1;
// dl_post_proc_cfg.params.od_prms.formatter[2] = 2;
// dl_post_proc_cfg.params.od_prms.formatter[3] = 3;
// dl_post_proc_cfg.params.od_prms.formatter[4] = 5;
// dl_post_proc_cfg.params.od_prms.formatter[5] = 4;
// dl_post_proc_cfg.params.od_prms.scaleX = POST_PROC_OUT_WIDTH / 416.0;
// dl_post_proc_cfg.params.od_prms.scaleY = POST_PROC_OUT_HEIGHT / 416.0;
// dl_post_proc_cfg.params.od_prms.labelIndexOffset = 0;
// dl_post_proc_cfg.params.od_prms.labelOffset = label_offset3;
// dl_post_proc_cfg.params.od_prms.classnames = imgnet_labels3;
// dl_post_proc_cfg.io_config_path = TIDL_IO_CONFIG_FILE_PATH;
// dl_post_proc_node = tiovx_modules_add_node(&graph, TIOVX_DL_POST_PROC, (void *)&dl_post_proc_cfg);
// /* Link MSC0 and TIDL to DL Post Proc */
// for (int32_t i = 0; i < dl_post_proc_node->num_inputs - 1; i++)
// {
// tiovx_modules_link_pads(output_pad[i], &dl_post_proc_node->sinks[i]);
// }
// tiovx_modules_link_pads(post_proc_in_img_pad, &dl_post_proc_node->sinks[dl_post_proc_node->num_inputs - 1]);
// output_pad[0] = &dl_post_proc_node->srcs[0];
// }
// /* Display */
// {
// TIOVXDisplayNodeCfg display_cfg;
// NodeObj *display_node;
// tiovx_display_init_cfg(&display_cfg);
// display_cfg.width = POST_PROC_OUT_WIDTH;
// display_cfg.height = POST_PROC_OUT_HEIGHT;
// display_cfg.params.outWidth = POST_PROC_OUT_WIDTH;
// display_cfg.params.outHeight = POST_PROC_OUT_HEIGHT;
// display_cfg.params.posX = (1920 - POST_PROC_OUT_WIDTH)/2;
// display_cfg.params.posY = (1080 - POST_PROC_OUT_HEIGHT)/2;
// display_node = tiovx_modules_add_node(&graph, TIOVX_DISPLAY, (void *)&display_cfg);
// /* Link DL Post Proc to Display */
// tiovx_modules_link_pads(output_pad[0], &display_node->sinks[0]);
// }
status = tiovx_modules_verify_graph(&graph);
BufPool *in_buf_pool = NULL, *out_buf_pool = {NULL};
in_buf_pool = viss_node->sinks[0].buf_pool;
for (int i = 0; i < APP_BUFQ_DEPTH; i++) {
inbuf = tiovx_modules_acquire_buf(in_buf_pool);
v4l2_capture_enqueue_buf(v4l2_capture_handle, inbuf);
}
//v4l2 start
v4l2_capture_start(v4l2_capture_handle);
// threadParams mosaic_param, ldc_param;
// ldc_param.capture_handle = v4l2_capture_handle;
// ldc_param.pad = &viss_node->sinks[0];
// ldc_param.enable = true;
// pthread_create(&id, NULL, ldc_enqueue, (void *)&ldc_param);
// pthread_create(&id, NULL, ldc_dequeue, (void *)&ldc_param);
// for (int32_t i = 0; i < input_pad->buf_pool->bufq_depth; i++)
// {
// inbuf = tiovx_modules_acquire_buf(input_pad->buf_pool);
// tiovx_modules_enqueue_buf(inbuf);
// }
/*
修改 不需要投递buf
*/
// for(int32_t i = 0; i < tidl_node->num_outputs; i++){
// for (int32_t j = 0; j < output_pad[i]->buf_pool->bufq_depth; j++)
// {
// Buf* inbuftmp = tiovx_modules_acquire_buf(output_pad[i]->buf_pool);
// tiovx_modules_enqueue_buf(inbuftmp);
// }
// }
int frame_count = 0;
while (frame_count++ < NUM_ITERATIONS)
{
//不断投递buf,类似于更新帧
inbuf = v4l2_capture_dqueue_buf(v4l2_capture_handle);
tiovx_modules_enqueue_buf(inbuf);
// 获取tidl的输出，对输出进行操作
for(int32_t i = 0; i < tidl_node->num_outputs; i++){
Buf* output = tiovx_modules_dequeue_buf(output_pad[i].buf_pool);
/*
process(output); // 模仿对tidl输出的后处理
*/
printf("index %d, buf_index %d, num_channels %d", i,output->buf_index,output->num_channels);
//将buf重新投递
tiovx_modules_enqueue_buf(output);
}
v4l2_capture_enqueue_buf(v4l2_capture_handle, inbuf);
}
for (int32_t i = 0; i < 2; i++)
{
inbuf = tiovx_modules_dequeue_buf(input_pad->buf_pool);
tiovx_modules_release_buf(inbuf);
}
tiovx_modules_clean_graph(&graph);
return status;
}
Hello,
Yes, this looks correct to me. Can you confirm you are getting buffers out at this stage in the application? It would be informative to look at the size of the data object in this buffer, and compare it to what you expect from the model output.
To get the tensor from the buf, please consider the code from the app_tiovx_tidl_module_test.c -- using the outbuf->handle cast to vx_tensor, we should be able to retrieve the data within the buffer (in your case, a tensor object). The writeTensor function may be a good reference here on how to use this tensor, because it will still need to be mapped to a physical address (otherwise, it is a virtual object that enables memory-optimizations like passing between cores without going to DDR).
You can also use the edgeai-tiovx-kernels repo as a reference for how some of the post processing is done, specifically the files related to vx_dl_post_proc (: https://git.ti.com/cgit/edgeai/edgeai-tiovx-kernels/tree/target/armv8/vx_dl_post_proc_target.c?h=develop
Hello Reese,
I have two questions about tiovx that I would like to ask you.
The first question is that I used the edgeai_tidl_tools tool to compile the yolov5s6_640_ti_lite_37p4_56p0-640 model of edgeai_yolov5. The compilation went well and subgraph_0_tidl_io_1.bin and subgraph_0_tidl_net.bin were generated. However, I put the compiled model on the board and used the demo of edgeai_tiovx_apps to run the algorithm, but it did not take effect. Why? After asking the fae, he said that some processing was required to draw the results. How do I do it?
The second question is that I modified edgeai_tiovx_apps/test/app_tiovx_dl_pipeline_test.c and wanted to directly obtain the output of tild_node, but an error was reported during runtime, and the log said Graph verify failed. What is the reason?
I submitted all the files used for the two questions（model.zip .c error_log.png） to you. Please help me answer them. Thank you.file1111.zip
Hello,
owever, I put the compiled model on the board and used the demo of edgeai_tiovx_apps to run the algorithm, but it did not take effect. Why? After asking the fae, he said that some processing was required to draw the results. How do I do it?
Are you certain the model is giving good, usable results? If not, then you may not see outputs drawn.
When testing this, did you use an application provided by TI? And when you say didn't take effect, nothing was drawn on the screen? If you are using an application from us like /opt/edgeai-tiovx-apps/bin/Release/...-main (replace ... with name of main execbutable), then I expect it should run the model as part of the pipeline and draw results, if any were produced by the model. That application already handles postprocessing -- it uses a TIOVX TIDL PostProcessing module to draw model outputs to the frame.
The second question is that I modified edgeai_tiovx_apps/test/app_tiovx_dl_pipeline_test.c and wanted to directly obtain the output of tild_node, but an error was reported during runtime, and the log said Graph verify failed. What is the reason?
The graph verification error is indicating that it could not initialize a multiscaler node -- something was probably setup incorrectly when configuring nodes and their links to one another
Okay, error is noted first in function tiovx_modules_compare_image. This is being called from tiovx_modules_link_pads. The error occurred because the output of a node doesn't have the same format as the input to the next, linked node. This usually indicates something incorrectly specified in the node-configuration objects/structs passed when creating the node, e.g. dimensions of an input/output, data type, etc.
I'd recommend adding some prints to find which nodes were being linked when this failed, and then investigate what configurations were being supplied to both.
Looks like the multiscaler node is complaining here. I have noted in past work that multiscaler can fail if crop parameters are not setup/initialized, even if you do not intend to crop the images (the struct must be initialized to default values). If it's this node that is the problem, take a look at some of the other files under tests/ that use multiscaler and see how they initialize configurations.
BR,
Reese
Hello,Reese, Can you help me look at the demo I wrote? When the program executes outbuf[i] = tiovx_modules_acquire_buf(out_buf_pool[i]);
, the program will die and no error log will appear
vx_status status = VX_FAILURE;
GraphObj graph;
NodeObj *node = NULL;
NodeObj *node2 = NULL;
NodeObj *dlPreProcNode = NULL, *tidl_node = NULL;
TIOVXMultiScalerNodeCfg cfg;
TIOVXMultiScalerNodeCfg cfg2;
BufPool *in_buf_pool = NULL, *out_buf_pool[APP_NUM_OUTPUTS];
Buf *inbuf = NULL, *outbuf[APP_NUM_OUTPUTS];
char input_filename[100];
char output0_filename[100];
char output1_filename[100];
const char* inputimg = "/opt/edgeai-tiovx-apps/input/a.yuv";
sprintf(input_filename, inputimg,strlen(inputimg));
printf("%s\n",input_filename);
sprintf(output0_filename, "%s/output/416_416.yuv", EDGEAI_DATA_PATH);
sprintf(output1_filename, "%s/output/baboon_160x120_nv12_msc_out1.yuv", EDGEAI_DATA_PATH);
graph.schedule_mode = VX_GRAPH_SCHEDULE_MODE_QUEUE_AUTO;
status = tiovx_modules_initialize_graph(&graph);
tiovx_multi_scaler_init_cfg(&cfg);
cfg.color_format = VX_DF_IMAGE_NV12;
cfg.num_outputs = APP_NUM_OUTPUTS;
cfg.input_cfg.width = INPUT_WIDTH;
cfg.input_cfg.height = INPUT_HEIGHT;
cfg.output_cfgs[0].width = 480;
cfg.output_cfgs[0].height = 480;
sprintf(cfg.target_string, TIVX_TARGET_VPAC_MSC1);
tiovx_multi_scaler_module_crop_params_init(&cfg);
node = tiovx_modules_add_node(&graph, TIOVX_MULTI_SCALER, (void *)&cfg);
//scale2
tiovx_multi_scaler_init_cfg(&cfg2);
cfg2.color_format = VX_DF_IMAGE_NV12;
cfg2.num_outputs = APP_NUM_OUTPUTS;
cfg2.input_cfg.width = 480;
cfg2.input_cfg.height = 480;
cfg2.output_cfgs[0].width = 416;
cfg2.output_cfgs[0].height = 416;
sprintf(cfg2.target_string, TIVX_TARGET_VPAC_MSC1);
tiovx_multi_scaler_module_crop_params_init(&cfg2);
node2 = tiovx_modules_add_node(&graph, TIOVX_MULTI_SCALER, (void *)&cfg2);
tiovx_modules_link_pads(&node->srcs[0], &node2->sinks[0]);
// pre proc
TIOVXDLPreProcNodeCfg dlPreProcCfg;
tiovx_dl_pre_proc_init_cfg(&dlPreProcCfg);
dlPreProcCfg.io_config_path = TIDL_IO_CONFIG_FILE_PATH;
dlPreProcCfg.params.tensor_format = 1; //BGR
dlPreProcNode = tiovx_modules_add_node(&graph, TIOVX_DL_PRE_PROC, (void *)&dlPreProcCfg);
tiovx_modules_link_pads(&node2->srcs[0], &dlPreProcNode->sinks[0]);
// tidl
TIOVXTIDLNodeCfg tidlCfg;
tiovx_tidl_init_cfg(&tidlCfg);
tidlCfg.io_config_path = TIDL_IO_CONFIG_FILE_PATH;
tidlCfg.network_path = TIDL_NETWORK_FILE_PATH;
tidl_node = tiovx_modules_add_node(&graph, TIOVX_TIDL, (void *)&tidlCfg);
tiovx_modules_link_pads(&dlPreProcNode->srcs[0], &tidl_node->sinks[0]);
status = tiovx_modules_verify_graph(&graph);
in_buf_pool = node->sinks[0].buf_pool;
for (int32_t i = 0; i < tidl_node->num_outputs; i++)
{
out_buf_pool[i] = &tidl_node->srcs[i].buf_pool;
}
// out_buf_pool[0] = dlPreProcNode->srcs[0].buf_pool;
for(int i = 0;i < 2;i++){
printf("i: %d\n",i);
inbuf = tiovx_modules_acquire_buf(in_buf_pool);
readImage(input_filename, (vx_image)inbuf->handle);
tiovx_modules_enqueue_buf(inbuf);
for(int32_t i = 0; i < tidl_node->num_outputs; i++){
outbuf[i] = tiovx_modules_acquire_buf(out_buf_pool[i]);
tiovx_modules_enqueue_buf(outbuf[i]);
}
// tiovx_modules_schedule_graph(&graph);
// tiovx_modules_wait_graph(&graph);
inbuf = tiovx_modules_dequeue_buf(in_buf_pool);
for(int32_t i = 0; i < tidl_node->num_outputs; i++){
outbuf[i] = tiovx_modules_dequeue_buf(out_buf_pool[i]);
tiovx_modules_release_buf(outbuf[i]);
}
// outbuf[0] = tiovx_modules_dequeue_buf(out_buf_pool[0]);
// writeImage(output0_filename, (vx_image)outbuf[0]->handle);
tiovx_modules_release_buf(inbuf);
// for(uint8_t i = 0; i < tidl_node->num_outputs; i++) {
// }
}
tiovx_modules_clean_graph(&graph);
printf("done111\n");
return status;
Hello,
Is this the same file code that you were trying to run or has it been edited? Line 86 has an open brace that doesn't seem to have been closed anywhere. The indentation for loops is a bit hard to follow.
Does it cleanly exit or throw something like a segfault? I assume this is compiled without error and running on AM62A. I would recommend running /opt/vx_app_arm_remote_log.out in the background or in a separate terminal. Some TIOVX logs will show there instead of stdout.
The code probably needs to be instrumented with a few more prints to determine where it's failing. You have many lines commented out, including schedule_graph and wait_graph. However, those calls may not be necessary if you did not change the graph parameter to be manually scheduled. Otherwise, there is no app-iterations being applied, yet maybe you only want to have single input from file and exit.
When the program executes outbuf[i] = tiovx_modules_acquire_buf(out_buf_pool[i]);
, the program will die and no error log will appear
This is line 95, right?
Lines 94-96 are suspect. You have have tidl_node->num_outputs buffer pools (let's say 2 for a yolo model). Now, each of these buffer pools can have buf_pool->bufq_depth number of buffers within (could be 1, 2, 3, 4, etc. but usually 2 or 4 is typical value). In your code, it looks like you are using a single buffer per pool. That may work technically, but wanted to mention it.
Otherwise, I'm not sure why this program would exit on you. It shouldn't exit in the middle of the program without throwing some error or signal. Probably worth running in gdb.
BR,
Reese
Hello,Reese
I seem to be able to get the output of the tidl node, then convert the data of output->handle to vx_tensor and call writeTensor. Finally, I get two bin files, one is 5X200.bin and the other is 200x1.bin. Are these two tensors output by yolox? Then I want to ask, how should I process the tensor data after getting it? Should I call tivxMapTensorPatch and get a void* pointer? How can I decode the unknown information and knowledge information of all targets in the yolox detection results?
My second question is the function tivxKernelDLPostProcProcess in vx_dl_post_proc_target.c of edgeai_tiovx_kernels. I don’t know if this function can be called at the application layer. It needs to pass in tivx_target_kernel_instance type and tivx_obj_desc_t * type variables. Is this variable the type obtained or converted in the demo of edgeai_tiovx_apps/tests?
Hello,
Yes, that sounds like the correct sizes. That is important progress.
I would recommend looking at the writeTensor function for guidelines on how to retrieve the actual tensor data. I assume the 5x200 is in float32 format and the 200x1 is in int64, if I recall correctly the yolox output style. You can look at the output layers in the SVG files within the artifacts/tempDir directory (hover over the output TIDL-Data-Layer nodes with SVG loaded in a browser like Chrome) to see what the data type is, but I should think the vx_tensor would also carry this information.
My second question is the function tivxKernelDLPostProcProcess
That function is intended to be run as part of the kernel itself, so it may not be trivial to run it directly in your code. I do not expect that it can be easily called since it is reliant on data structures that are produced as part of the TIOVX graph.
In the example under edgeai-tiovx-apps/tests, it is called within a TIOVX graph, so the direct interface to that tivx...Process call is abstracted from the user point of view.
Please note that this very long function is complicated by the fact that it needs to handle many different input types and model types, while the actual code that yolox would need is just a subset of this for drawing boxes onto the input image. This kernel will have one input for the image to visualize on, and the other inputs are the outputs of a TIDL node
Hello Reese,
When I was simulating the model using onnxrt_ep. py, the effect was good when I used - d-m, and the size of the tensor output was (21, 6). However, when I used - d, I found that the output of the model was (300, 6), without filtering for nms. But when I executed export-oriented, I added the option of export_nms. Then I uploaded the tensors of these two results in the attachments, which were saved in OCD2_608pc-d - m. txt and OCD2_608pc-m. txt files respectively. Finally, I also uploaded pictures of onnx, ti simulation, and board end tiovx execution results, three of which are py_ovx. ut_oCD2_608pc-d - m. jpg, py_out_oCD2_608pc-m. jpg, and board.jpg, it can be observed that the accuracy and effectiveness are gradually decreasing. Can you help analyze the reasons for this
best wish
zhuangyh1205 experiment.zip
Hello,
I think I understand your issue here, but let's take a step back first.
Is your TIOVX-related query resolved?
I see in the latest response you are discussing model accuracy issues. This will be an independent topic to the TIOVX issues we worked through previous.
My suggestion is the following: In this thread, let's focus on the TIOVX topic. For the model accuracy, please create a separate thread. This is helpful for others reading the forum posts -- it is difficult to understand when many topics are being worked on at the same time.
I will also be the right engineer to help with your model accuracy challenges, but let us work through this in a separate thread. We can always reopen this thread later.
BR,
Reese
Hello Reese,
I hope to remove the truncation points in the post-processing of YOLOv5, and then I will implement the post-processing of YOLOv5 myself. That is, the output of onnx is three feature maps. I also did this by changing the modelotype from 'OD' to classification, and then compiling the bin file with edgeai_tidl_tool. However, when I loaded the bin file on the board, there was an error message as follows:
12049.270725 s: VX_ZONE_ERROR:[ownContextSendCmd:885] Command ack message returned failure cmd_status: -1
12049.270769 s: VX_ZONE_ERROR:[ownNodeKernelInit:592] Target kernel, TIVX_CMD_NODE_CREATE failed for node tidl_node
12049.270787 s: VX_ZONE_ERROR:[ownNodeKernelInit:593] Please be sure the target callbacks have been registered for this core
12049.270799 s: VX_ZONE_ERROR:[ownNodeKernelInit:594] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
12049.270814 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:620] kernel init for node 2, kernel com.ti.tidl:1:3 ... failed !!!
12049.270847 s: VX_ZONE_ERROR:[vxVerifyGraph:2254] Node kernel init failed
12049.270859 s: VX_ZONE_ERROR:[vxVerifyGraph:2311] Graph verify failed
[TIOVX_MODULES][ERROR] 795: tiovx_modules_verify_graph: Graph Verify failedtidl_node->num_outputs is 3
where the problem lies
best wishes
zhuangyihao
Hello,
Understood -- you are removing the part of the OD head that handles NMS. As a result, you have 3 outputs to the network
[TIOVX_MODULES][ERROR] 795: tiovx_modules_verify_graph: Graph Verify failedtidl_node->num_outputs is 3
The TIDL binaries should know that there are 3 outputs to the network, and the configuration passed when adding the tidl_node to the TIOVX graph must reflect this.
Did you modify the configuration (struct TIOVXTIDLNodeCfg) of the tidl_node after you changed your model?
BR,
Reese