PROCESSOR-SDK-AM62A: Environment issue

Hello Machida-san,

For an end-to-end AI application, I would recommend edgeai-gst-apps instead of edgeai-benchmark. The benchmark tools are intended for testing a models' performance on a dataset of saved images.

If you still want to move forward with edgeai-benchmark, would you please provide a log for the setup_pc.sh? It should have made this tool direcotry, along with subdirectories for individual processors' tool sets. This would be TDA4VM, AM68A, AM69A, and AM62A. By default, it should be downloading the 9.1 tools unless a version was specified otherwise.

BR,
Reese

Hello Reese-san,

Thank you for your reply.
>For an end-to-end AI application, I would recommend edgeai-gst-apps instead of edgeai-benchmark. The >benchmark tools are intended for testing a models' performance on a dataset of saved images.

According to below repo, I think that edgeai-gst-apps is better to create AI app, however, this description is written in "edgeai-benchmark" as shown below.
https://github.com/TexasInstruments/edgeai


So, I thouhgt that we need to clone "edgeai-benchmark" and set environment (assume compile for model ?)
after that clone "edgeai-gst-apps" for application.
In case of "edgeai-gst-apps", is it NOT necessory "edgeai-benchmark" repo ?
(Is it OK only clone "edgeai-gst-apps" ?)

BR,

Hello,

There are two primary stages for running a model. Compilation on x86 PC and inference on TI SoC.

The Edgeai-benchmark and edgeai-tidl-tools repos are useful for compiling models, which must happen on x86 PC. These two repos can be used for inference as well, but this is restricted to static image files:

• Edgeai-benchmark can run an entire dataset of image files to collect accuracy information
• edgeai-tidl-tools can run on a small set of images, and is better used as a source of examples for inference.

Now for an end-to-end application, I would not recommend those two repositories. They both operate on static images rather than video files or a camera input stream. This is where edgeai-gst-apps is useful.

Edgeai-gst-apps is only for inference, not compilation. Once you have a compiled model, I would recommend trying it with edgeai-gst-apps.

I understand why the description in the shown document may have been confusing. I will deliver this as feedback. The sentence "for deployment on SDK with camera, inference and display (using edgeai-gst-apps)" is meant to say that edgeai-benchmark can produce artifacts that deploy using edgeai-gst-apps.

I hope this provides more clarification!
-Reese

Hello Reese-san,

Thank you for your clarification.
I almost understand how I can shift End-t-End application.

Here is my position.

1. We created custom model based on "edgeai-yolox" repo. (Download checkpoint and training in our environment.)
2. After that we converted pytorch character to onnx by using following script.
"export_onnx.py"
3. I compiled onnx and prototxt data by using "edgeai-tidl-tools" to create "artifact" for AM62A.
4. I could run this model under AM62A EVM. And confirmed this model work correctly as shown below. (inference time is also expected result.)
 


However, I noticed there is following issue.

Issue 1 : "dataset.yaml" is NOT created.
Issue 2 : It seems poor "param.yaml" to operate under "edgeai-gst-app". (Several parameters are lack.)

Q, To move "edgeai-gst-apps", should I re-compile model under "edgeai-benchmark" ?
     Or is there any way to create "dataset.yaml" and enough "param.yaml" under "edgeai-tidl-tools" ?

BR,

Hello,

I understand. It looks like the model you have is working well on the static images -- 40ms for the yolox-s model is well within the expected range of performance. Good progress!

I understand your two issues with the dataset.yaml and param.yaml

1. dataset.yaml: correct, it will not be generated by edgeai-tidl-tools since that repository does not take into account the dataset the model was trained for. edgeai-benchmark is capable of generating this, but it will need information about the dataset. This is generally handled by a dataset python script like this one for COCO_seg. This is worth generating for your model if the dataset tested several times / automated as part of a model optimization process. Otherwise, it may be less effort to use another model's dataset.yaml as a template and write your own.
2. param.yaml: Yes, this is a known bug within edgeai-tidl-tools. Edgeai-benchmark will produce a valid param.yaml for use with edgeai-gst-apps. Another option (just like creating dataset.yaml) is to view another model from /opt/model_zoo's param.yaml as a template, and modify a few fields to match this. Not all fields are necessary -- it is mostly parameters related to input preprocessing and output postprocessing (e.g., input mean, input scale, output dimensions).

Please let me know if I can be of more assistance!

Best,
Reese

Hello,

Thank you for your reply.

For 1
I see. 
>Otherwise, it may be less effort to use another model's dataset.yaml as a template and write your own.
Yes, at first I also think about this.
However also try to create dataset.yaml by using our original dataset.

For 2
I understood.

When I use edgeai-tidl-tools with param.yaml and dataset.yaml by modifing template files, is it possible to connect "edgeai-gst-apps" like "edgeai-benchmark" ?
It seems that you said I can create End-to-End application with "edgeai-tidl-tools" instead of "edgeai-benchmark"...

BR,

 

Hello Reese-san,

I could run my custom model by using edgeai-benchmark and edgeai-gst-apps.
(input is camera and output is display environment.)
However, I have one issue.(I try to run "object detection".)

* I could confirm inference result under terminal as shown below.

+--------------------------------------------------------------------------+
| dl-inference : 66.16 ms from 7 samples |
| total time : 92.71 ms from 6 samples |
| framerate : 10.79 fps from 6 samples |
+--------------------------------------------------------------------------+

However, I could not confirm bounding box on displayed image.
At first, I think that this issue is caused by rate of recognization of model.
So, I tested model by using image file (.jpg) which is used validation at compling model on edgeai-benchmark.
Actually, I also confirmed result as shown below.

+--------------------------------------------------------------------------+
| dl-inference : 55.74 ms from 2 samples |
| total time : 954.03 ms from 1 samples |
| framerate : 1.05 fps from 1 samples |
+--------------------------------------------------------------------------+

However I could NOT confirm bounding box on image file as well.
Do you have any idea about how I should proceed this debug ?

BR,
 

Hello Ryuuichi,

Is the issue that the bounding boxes are not drawn on the object? Or are they drawn but without their associated labels? 

For now, I suggest to check the follwoing:

1. Check the task_type parameter in the param.yaml of the model you are calling. This parameter should be set to "detection" in order to perform the object detection post process (i.e draw bounding boxes around the detected objects). 

2. Check the dataset.yaml file in the model folder you are calling. This file should contain the ids and name of all classes. 

Please, let me now what happens. If these steps did not solve the issue, we can go to a more involved debug process by checking the results out of the model at run time.

Bests,

Qutaiba

Hello,

Thank you for your reply.

>Is the issue that the bounding boxes are not drawn on the object?
Yes, this is issue.

For above "1" and "2", I confirmed that both were fine.
So, I try to implement "print" to application python code. Then I noticed that I could not satisfy following branch which is described "post_process.py".(When I confirm, b[5] show "0" any time.)

            if b[5] > self.model.viz_threshold:

So, I confirmed python code, and change value for below(setting for dataset_loaders).

From :  split='train2017',       
To : split='val2017',

Then I got following error when I run "app_edgeai.py"

Error :
sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidProtobuf: [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from ~.onnx failed:Protobuf parsing failed.

Have you ever seen such error ?

Best Regards,

Hello Ryuuichi,

Ryuuichi machida said:

For above "1" and "2", I confirmed that both were fine.
So, I try to implement "print" to application python code. Then I noticed that I could not satisfy following branch which is described "post_process.py".(When I confirm, b[5] show "0" any time.)

            if b[5] > self.model.viz_threshold:

This sounds that the model is not sending the correct results. From the questions in this thread, I am not sure if you already have a trained and compile model. Would you please confirm if the model is already trained and compile?

If the model is trained and compiled, I suggest the following as a next debug step:
1. In edgai-gst-apps, use a picture(s) which you are sure that the model is able to process even if it is from the training dataset.

2. Try to print the whole bbox array. Insert "print(bbox)" at this line https://github.com/TexasInstruments/edgeai-gst-apps/blob/43c36d1e1381a718869c0f6242da0648d4ee85e6/apps_python/post_process.py#L232. 

3. See if any of the threshold values at bbox[:,5] is more than the default threshold value the model is trained with. Also check the bounding box parameters at bbox[:,0:3] if they make sense. 

From the above steps we can confirm if the issue is the the post process code or if the model itself is not working properly and not providing the correct results. 

-------

Ryuuichi machida said:

So, I confirmed python code, and change value for below(setting for dataset_loaders).

From :  split='train2017',       
To : split='val2017',

Then I got following error when I run "app_edgeai.py"

This is related to training the model and not using the model. The code app_edgeai.py expects a full trained and compiled model. 

Best regards,

Qutaiba

Hello,

At first, here is result when I implement print(bbox)

---
[-0.0015625 -0.0015625 -0.0015625 -0.0015625 -1.         0. ]
---
I only got above value whole log.
I'm not sure meaning of each value at this time, however if you have comment about this, please let me know.

BR,

Hello Machida-san,

This looks like a null detection. The first 4 values should be the box coordinates. The 5th seems to be the class index (-1 meaning null), and the 6th is the confidence, i.e. 0

Do you receive realistic boxes when running inference without acceleration, i.e. on the CPU? The results for CPU inference should be identical whether it is on PC or the SoC - this is useful as a baseline to check if the model is behaving correctly or not, independent of any quantization effects.

Best,
Reese

Hello,

>Do you receive realistic boxes when running inference without acceleration, i.e. on the CPU? 
How can I confirm it on PC ?
To disable acceralator, I believe that I just disable setting on yaml file, however according to run_custom_pc.sh on edgeai_benchmark, I can confirm whether inference work correctly or not by referring "result.yaml" file. But this file do not contain image result such as "output_image" of "edgeai-tidl-tool" repo. So, I ask this question.

BR,

Hi,

By the way, I wonder which function should I choose for postprocessing of "edgeai-yolox" model on pipeline config.
According to _init__.py at edgeai_benchmark/postprocess, you define several functions under following comment.

###############################################################
# post process transforms for detection
###############################################################

Now, I choose "get_transform_detection_yolov5_onnx" however I'm not sure this is correct in my yolox model..
(It seems that my issue is related to postprocess so I also check my pipeline setting.)

BR,

For this, I would recommend finding the place where "interpreter.run" is called (assuming this is an ONNX model, otherwise a .invoke() call for tflite) and adding some prints to show the output results (note that many will be null, generally denoted by class value = -1 or confidence = 0.

Ryuuichi machida said:

Now, I choose "get_transform_detection_yolov5_onnx" however I'm not sure this is correct in my yolox model..

I believe there are differences in the format of the output data for yolov5 vs. yolox, so it is true that you may need a different postprocessing function. I believe this one should work for yolox: get_transform_detection_mmdet_onnx

Hello,

At first, I tried to perform below.

>I believe there are differences in the format of the output data for yolov5 vs. yolox, so it is true that you may need a different postprocessing function. I >believe this one should work for yolox: get_transform_detection_mmdet_onnx
When I changed postprocessing operation from "get_transform_detection_yolov5_onnx" to "get_transform_detection_mmdet_onnx", I got following result.

--
91.921446 s: VX_ZONE_ERROR:[ownContextSendCmd:868] Command ack message returned failure cmd_status: -1
91.921552 s: VX_ZONE_ERROR:[ownNodeKernelInit:584] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
91.921573 s: VX_ZONE_ERROR:[ownNodeKernelInit:585] Please be sure the target callbacks have been registered for this core
91.921591 s: VX_ZONE_ERROR:[ownNodeKernelInit:586] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
91.921617 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
91.921647 s: VX_ZONE_ERROR:[vxVerifyGraph:2055] Node kernel init failed
91.921682 s: VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed
91.921864 s: VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:812] graph is not in a state required to be scheduled
91.921881 s: VX_ZONE_ERROR:[vxProcessGraph:747] schedule graph failed
91.921894 s: VX_ZONE_ERROR:[vxProcessGraph:752] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
viz_threshold is [0. 0. 0. 0. 0. 0.]
viz_threshold is [0. 0. 0. 0. 0. 0.]
viz_threshold is [0. 0. 0. 0. 0. 0.]
viz_threshold is [0. 0. 0. 0. 0. 0.]
viz_threshold is [0. 0. 0. 0. 0. 0.]
viz_threshold is [0. 0. 0. 0. 0. 0.]
viz_threshold is [0. 0. 0. 0. 0. 0.]
viz_threshold is [0. 0. 0. 0. 0. 0.]
・
・
--

All "bbox" values show "0", so it seems that this did not work correctly..
I'm trying to put "print" arround interpreter.run and recomplile later.

BR, 

Hello,

For this, I would recommend finding the place where "interpreter.run" is called (assuming this is an ONNX >model, otherwise a .invoke() call for tflite) and adding some prints to show the output results (note that many will be null, generally denoted by class value = -1 or confidence = 0.

I also performed above by recompiling model.(Preprocessing is "get_transform_detection_yolov5_onnx")
I added "print" to show "outputs" variable under following code on onnxrt_sesssion.py.

*  line71          outputs = self.interpreter.run(output_keys, calib_dict) 
*  line106        outputs = self.interpreter.run(output_keys, input_dict)

For line 71, here is result.

Import Output result [array([[[[[[-1., -1., -1., -1., 0., -1.],
[-1., -1., -1., -1., 0., -1.],
[-1., -1., -1., -1., 0., -1.],
...,
[-1., -1., -1., -1., 0., -1.],
[-1., -1., -1., -1., 0., -1.],
[-1., -1., -1., -1., 0., -1.]]]]]], dtype=float32)]


For line106, here is result.

Import Output result [array([[[[[[-1., -1., -1., -1., 0., -1.],
[-1., -1., -1., -1., 0., -1.],
[-1., -1., -1., -1., 0., -1.],
...,
[-1., -1., -1., -1., 0., -1.],
[-1., -1., -1., -1., 0., -1.],
[-1., -1., -1., -1., 0., -1.]]]]]], dtype=float32)]

I'm not sure this is expected result...
Could you please your comment ?

BR,

Hello,

This output indicates that the model did not find any objects of interest in the image. There can be several reasons, like improper preprocessing (typically mean subtraction and scale multiplication -- must be the same values as used during training, which a param.yaml would include if TI tools did the training) or using an image that simply doesn't have visible objects of interest.

Ryuuichi machida said:

All "bbox" values show "0", so it seems that this did not work correctly..

Correct, that doesn't look quite right. Was this running on the target or the PC? Some of the messages look like it's on the target, and the OpenVX stack has gotten into the wrong state (usually after many starts and stops of an application). I would recommend rebooting the SoC if you are seeing VX_ZONE_ERROR's like this. The message afterward with outputs of the form "[-1., -1., -1., -1., 0., -1.]," is functionally correct for an input without objects the model can detect.

Best,
Reese

Hello,

Thank you for your reply.
Understood about my current situation what should I check.

By the way, after posting this thread, I have additional following two question.

Q1, When I use model which is created by edgeai-tidl-tool and use configuration file which is created by edgeai-benchmark, I confirmed bounding box by changing viz_threadhold to low value.
Do you think that this difference should be caused by preprocessing operation on edgeai-benchmark ?

Q2, When I use jpeg image which use validation on compiling stage of edgeai-benchmark as input of gstreamer app, I saw strange image on display.
On the other hand, when I use jpeg image under opt/edegai-test-data/xxx.jpg, I saw correct image on display.
The difference each image is size.
My image is "1920x1080" and "edgeai-test-data" image was "640x425".
Here is my configuration file which is related to input.

*
input0:
source: /xxxx/0000.jpg
width: 1280
height: 720
framerate: 1
index: 0

Note : When I change input size from 1280x720 to 1920x1080, my phenomenon was not solved..

Is there any restriction for input size of jpeg file on edgeai-gst-app ?

Best Regards,

Ryuuichi machida said:

Do you think that this difference should be caused by preprocessing operation on edgeai-benchmark ?

This is possible, yes. They may be using different preprocessing features like mean and scale. In edgeai-tidl-tools, I know these values come from model_configs.py file. For edgeai-benchmark, I am less certain. Preprocessing does make a large difference.

Ryuuichi machida said:

When I use jpeg image which use validation on compiling stage

Are you saying you are using the same image that it used for compilation / calibration for inference?

I think there are some restrictions on file sizes, particularly that the image should be larger in width and height than the model will be using. It is probably good if all the input images are the same dimensions, but I don't think this is required.

What does the 'strange display' look like? Images are helpful of course :)

Please also check the console window for any indication that a file was not read correctly -- use the '-n' tag when calling the script since otherwise ncurses can write over any error messages. I know I've had nonexistant files referenced before because I missed changing ".jpg" and ".jpeg"

Best,
Reese

Hello,

>Are you saying you are using the same image that it used for compilation / calibration for inference?
Yes. But same image file was used validation stage on our model training as well. So I used this "validation" word. Sorry for confusing.

>What does the 'strange display' look like? Images are helpful of course :)
Sorry, it is difficult to share image on public. If you can accept, I will send one image on private.

Best Regards,

Hello,

Thank you for accepting my request.
I will close this thread and discuss privately.

BR,