TDA4VM: TIDL: Facing the trouble at inference after quantization.

Hi HaTa,

     I believe that the issue is not related to quantization. As per the error, network compiler resulted into an assertion. Will it be possible for you to share the mode ( if not original some representative model) where the issue can be reprouced?

Regards,

Anshu

Hi Anshu

I sent the file via BOX just few minutes before.

Please let me know if you cannot download it properly.

Thanks and Best regards,

HaTa

Hi HaTa,
   I am able to download the model shared. Give us sometime and we will update if we could reproduce the issue. Can you also share the import config file which you have used? Also share if you  have used any different device_config file.

Regards,
Anshu

Hi HaTa,

    Can you share the import config file which you have used?

Regards,

Anshu

Hi Anshu

I send you the Box link which contains the files.

Thanks!

HaTa

Hi Anshu

Please share your latest investigation status to here?

Q#1: Could you please provide the advice to solve above?

In addition to my first input, I'd like to add more information related to this issue which customer confirmed.

They found that the error is not occurred if they minimize the resolution of the input image data.

And also, the error is not occurred if they they try to split into multiple Subgraphs by adding the layers that are not supported at TIDL.

------------------ Network Compiler Traces -----------------------------

ti_cnnperfsim.out: src/netmemmanger.c:366: int32_t updateInstanceMem(sMemorySizeinfo*, sBufProperty_t*, int32_t, sDataFlowInfo_t*): Assertion `0' failed.

Aborted (core dumped)

substitute string tidl_net_ not found

Could not open /home/xxxx/Ti/ti-processor-sdk-rtos-j721e-evm-07_02_00_06/tidl_j7_01_04_00_08/ti_dl/test/tvm-dlr/torch_ocr_torch-split--conv-with-contiguous-simple/tempDir/subgraph0_net/perfSimInfo.bin

INFORMATION: [TIDL_ResizeLayer] 538 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.

INFORMATION: [TIDL_ResizeLayer] 603 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.

INFORMATION: [TIDL_ResizeLayer] 615 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.

INFORMATION: [TIDL_ResizeLayer] 613 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.

INFORMATION: [TIDL_ResizeLayer] 672 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.

WARNING: [TIDL_E_DATAFLOW_INFO_NULL] ti_cnnperfsim.out fails to allocate memory in MSMC. Please look into perfsim log. This model can only be used on PC emulation, it will get fault on target.

【Case1】The entire network fits into one Subgraph and the input image resolution is 480 * 640

　　　⇒ Compile is failed with above error

【Case2】The entire network fits into one Subgraph and the input image resolution is 128 * 128

　　　⇒Compile is succeeded

【Case3】 divided into multiple Subgraphs by inserting a layer (reshape) that TIDL does not support.

                   The resolution of the input image is 480 * 640

　　　⇒Compile is succeeded

 Q#2: I would like to know the cause and solution of the Case 1 error.

 Q"3: What is the reason why Case3 works and Case1 fails? As you can see by looking at the onnx file, the only difference is that Case 3 has a reshape layer that does nothing.

Q#4:  Are there any networks that cannot be used due to MSMC size restrictions?

           If so, I would like to know how to judge whether it can be used or not.

           Also, is there a way to find out which part of the network is running out of memory (= the perfsim tool is failing)?

QA#5: What kind of information will be stored in MSMC? Layer output? Weight?

           If MSMC is used only to put the output and weight, the difference between Case 1 and Case 3 cannot be explained,

           therefore I guess if other data (e.g. entire graph) also use MSMC.

Thanks and Best regards,

HaTa

Hi HaTa,

 Q#1: Could you please provide the advice to solve above?

We are able to reproduce this issue and we are looking into the figuring out the reason for the same. First level it looks like the buffers which are  held in MSMC at the same time are becoming more than MSMC memory available but this shouldn't have happened because in this case buffer will be moved to DDR. But we still haven't concluded this and need some more time to root cause this issue. We will keep you posted here.

Its difficult to answer the rest of the questions before we find the reason for the first one. We are looking this at priority and will provide you an update soon.

Regards,

Anshu

Hi Anshu

Thanks for sharing your current verification status to this.

Please continue to share information.

Thanks and Best regards,

HaTa

Hi Anshu

Thanks for sharing your current verification status to this.

Please continue to share information.

Thanks and Best regards,

HaTa

Hi Anshu

Please let me know your current status for this topic

Hi Hata,

     This issue seems more involved than expected and we will need more time. Can you tell us what is the priority of this issue?

Regards,

Anshu

Hi Anshu

What they'd like to do is, modifying the network to run the unsupported layers on MMA instead of ARM as much as possible.

However, while doing that, they faced the trouble as my 1st post.

Therefore, the question on my 1st post is highest priority.

Would like to know the reason/solution to the fail.

------------------ Network Compiler Traces -----------------------------

ti_cnnperfsim.out: src/netmemmanger.c:366: int32_t updateInstanceMem(sMemorySizeinfo*, sBufProperty_t*, int32_t, sDataFlowInfo_t*): Assertion `0' failed.

Aborted (core dumped)

substitute string tidl_net_ not found

Could not open /home/xxxx/Ti/ti-processor-sdk-rtos-j721e-evm-07_02_00_06/tidl_j7_01_04_00_08/ti_dl/test/tvm-dlr/torch_ocr_torch-split--conv-with-contiguous-simple/tempDir/subgraph0_net/perfSimInfo.bin

INFORMATION: [TIDL_ResizeLayer] 538 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.

INFORMATION: [TIDL_ResizeLayer] 603 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.

INFORMATION: [TIDL_ResizeLayer] 615 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.

INFORMATION: [TIDL_ResizeLayer] 613 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.

INFORMATION: [TIDL_ResizeLayer] 672 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.

WARNING: [TIDL_E_DATAFLOW_INFO_NULL] ti_cnnperfsim.out fails to allocate memory in MSMC. Please look into perfsim log. This model can only be used on PC emulation, it will get fault on target.

Thanks and Best regards,

HaTa

Hi Hata,

Regarding the following statement :

"What they'd like to do is, modifying the network to run the unsupported layers on MMA instead of ARM as much as possible"

    Here what do you mean by modifying the network? Does it mean that they are replacing the un-supported layers by layers which are supported? Or they are implementing them as custom layer?

Regards,
Anshu

<<Procedures &  Environment>>

- Pytorch is used to define and train the model

- The network we use are based on CenternNet=HarDNet（https://github.com/PingoLH/CenterNet-HarDNet）, and modified by ourself.

- Convert Pytorch trained weights to Onnx files using torch.onnx built into pytorch

- Simplify onnx files using onnx simplifer (github.com/.../onnx-simplifier)

- From the simplified Onnx file, Generate the artifacts required for inference by following the procedure of TVM / Neo-AI-DLR + TIDL Heterogeneous Execution (The following error is occurred here)

  NOTE: For compilation, tvm-compilation-onnx-example.py is modified and used (only the path to be compiled and the image size are changed).

Hi HaTa,

   We will look into this after the other higher priority items.

Regards,

Anshu

Unlocking the thread to provide some latest updates.

Hi, This is filed as defect TIDL-1533 and will be fixed in 8.1 release ( Nov end 2021)

Hi Pramod

I'd like to confirm TIDL-1533 also.

So, could you send me the like for it, please?

Thanks and Best regards,

Tsurumoto.