• TI Thinks Resolved

AM5749: TIDL custom SSD isssue

Mastermind 9350 points

Replies: 10

Views: 115

Part Number: AM5749

Hi,

Our custom SSD model does not work.
1.In the host pc environment, the created SSD model can be recognized normally.
2.It succeeded in tidl_import, and it seems that the ssd_multibox execution result of example does not have any error.
DEFAULT_XXX of ssd_multibox/main.cpp is changed.

In the default one, the following log is output, but in the custom created one, the log is not output and it seems that it has completed with a timeout.
Even if the created multibox_0.png is displayed, the recognition BOX was not displayed.

Saving frame 0 to: frame_0.png
 0: (204, 57)-> (568, 300): horse, score = 0.525879
 1: (274, -4)-> (380, 173): person, score = 0.338013
Saving frame 0 with SSD multiboxes to: multibox_0.png
Loop total time (including read / write / opencv / print / etc): 601.4ms

I don't know how to debug.
Please give me advice on countermeasures.

The result of executing ssd_multibox is as follows.

cd /usr/share/ti/tidl/examples/ssd_multibox2
./ssd_multibox -i /home/root/cm_dl/cm_caffemodel/input_img/000000.jpg -v
[12878.281723] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
[12878.320418] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
[12878.326973] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
[12878.334406] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[12878.340291] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
Input: /home/root/cm_dl/cm_lego_caffemodel/input_img/000000.jpg
-> Executor::Executor()
OCL Device: EVE created
Creating kernel ocl_tidl_setup
OCL Create B:0x55a4c
Arg[0]: 0x55a4c
OCL Create B:0x55adc
Arg[1]: 0x55adc
OCL Create B:0xbff14
Arg[2]: 0xbff14
OCL Create B:0xbffa4
Arg[3]: 0xbffa4
Kernel: EVE device 0 executing ocl_tidl_setup, context 0
Kernel: waiting context 0...
[eve 0] TIDL Device Trace: -> ocl_tidl_setup()
[eve 0] TIDL Device Trace: PARAM heap: Size 9437184, Free 6340248, Total requested 3096936
[eve 0] TIDL Device Trace: <- ocl_tidl_setup()
Kernel: finished execution
OCL Release B:0x55a4c
OCL Release B:0x55adc
OCL Release B:0xbff14
OCL Release B:0xbffa4
-> ExecutionObject::ExecutionObject()
Creating kernel ocl_tidl_cleanup
Creating kernel ocl_tidl_initialize
OCL Create B:0xbffa4
Arg[0]: 0xbffa4
OCL Create B:0xbff14
Arg[1]: 0xbff14
OCL Create B:0x55adc
Arg[2]: 0x55adc
OCL Create B:0x55a4c
Arg[3]: 0x55a4c
Arg[4]: local, 4
Creating kernel ocl_tidl_process
OCL Create B:0xc06e4
Arg[0]: 0xc06e4
OCL Create B:0xc079c
Arg[1]: 0xc079c
OCL Create B:(nil)
Arg[2]: (nil)
Arg[3]: 0xbed3e548
<- ExecutionObject::ExecutionObject()
Kernel: EVE device 0 executing ocl_tidl_initialize, context 0
Kernel: waiting context 0...
[eve 0] TIDL Device Trace: -> ocl_tidl_initialize()
[eve 0] TIDL Device Trace: -> SetupMemoryRecords()
[eve 0] TIDL Device Trace: <- SetupMemoryRecords()
[eve 0] TIDL Device Trace: -> InitializeInputOutputBuffers()
[eve 0] TIDL Device Trace: <- InitializeInputOutputBuffers()
[eve 0] TIDL Device Trace: <- ocl_tidl_initialize()
[eve 0] TIDL Device Trace: NETWORK heap: Size 67108864, Free 30793344, T otal requested 36315520
Kernel: finished execution
<- Executor::Executor()
-> Executor::Executor()
OCL Device: DSP created
Creating kernel ocl_tidl_setup
OCL Create B:0xc2a1c
Arg[0]: 0xc2a1c
OCL Create B:0xc2cb4
Arg[1]: 0xc2cb4
OCL Create B:0xc2f4c
Arg[2]: 0xc2f4c
OCL Create B:0xc31e4
Arg[3]: 0xc31e4
Kernel: DSP device 0 executing ocl_tidl_setup, context 0
Kernel: waiting context 0...
[core 0] TIDL Device Trace: -> ocl_tidl_setup()
[core 0] TIDL Device Trace: PARAM heap: Size 9437184, Free 6340248, Total requested 3096936
[core 0] TIDL Device Trace: <- ocl_tidl_setup()
Kernel: finished execution
OCL Release B:0xc2a1c
OCL Release B:0xc2cb4
OCL Release B:0xc2f4c
OCL Release B:0xc31e4
-> ExecutionObject::ExecutionObject()
Creating kernel ocl_tidl_cleanup
Creating kernel ocl_tidl_initialize
OCL Create B:0xc31e4
Arg[0]: 0xc31e4
OCL Create B:0xc2f4c
Arg[1]: 0xc2f4c
OCL Create B:0xc2cb4
Arg[2]: 0xc2cb4
OCL Create B:0xc2a1c
Arg[3]: 0xc2a1c
Arg[4]: local, 4
Creating kernel ocl_tidl_process
OCL Create B:0xc3c6c
Arg[0]: 0xc3c6c
OCL Create B:0xc3f64
Arg[1]: 0xc3f64
OCL Create B:(nil)
Arg[2]: (nil)
Arg[3]: 0xbed3e548
<- ExecutionObject::ExecutionObject()
Kernel: DSP device 0 executing ocl_tidl_initialize, context 0
Kernel: waiting context 0...
[core 0] TIDL Device Trace: -> ocl_tidl_initialize()
[core 0] TIDL Device Trace: -> SetupMemoryRecords()
[core 0] TIDL Device Trace: <- SetupMemoryRecords()
[core 0] TIDL Device Trace: -> InitializeInputOutputBuffers()
[core 0] TIDL Device Trace: <- InitializeInputOutputBuffers()
[core 0] TIDL Device Trace: <- ocl_tidl_initialize()
[core 0] TIDL Device Trace: NETWORK heap: Size 67108864, Free 66256000, T otal requested 852864
Kernel: finished execution
<- Executor::Executor()
Kernel: EVE device 0 executing ocl_tidl_process, context 0
[eve 0] TIDL Device Trace: -> ocl_tidl_process(frame 0, context 0)
[eve 0] TIDL Device Trace: <- ocl_tidl_process(frame 0, context 0)
-> [12905.564542] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
ExecutionObject::WaitAndReleaseContext(0)
Kernel: waiting cont[12905.576771] omap-iommu 40d02000.mmu: 40d02000.mmu : version 3.0
ext 0...
Kernel: finished execution
Kernel: DSP device 0 executing ocl_tidl_process, context 0
[core 0] TIDL Device Trace: -> ocl_tidl_process(frame 0, context 0)
[core 0] TIDL Device Trace: <- ocl_tidl_process(frame 0, context 0)
-> ExecutionObject::WaitAndReleaseContext(0)
Kernel: waiting context 0...
Ke[12905.612666] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
rnel: finished execution
Saving frame 0 with SSD multiboxes to:[12905.621229] omap-iommu 41502000.mmu: 41 502000.mmu: version 3.0
multibox_0.png
Loop total time (including read/write/opencv/print/etc): 2.421e+04ms
Kernel: EVE device 0 executing ocl_tidl_cleanup, context 0
Kernel: waiting context 0...
Kernel: finished execution
OCL Device: deleted
OCL Release B:0xc06e4
OCL Release B:0xc079c
OCL Release B:0xbffa4
OCL Release B:0xbff14
OCL Release B:0x55adc
OCL Release B:0x55a4c
Kernel: DSP device 0 executing ocl_tidl_cleanup, context 0
Kernel: waiting context 0...
Kernel: finished execution
OCL Device: deleted
OCL Release B:0xc3c6c
OCL Release B:0xc3f64
OCL Release B:0xc31e4
OCL Release B:0xc2f4c
OCL Release B:0xc2cb4
OCL Release B:0xc2a1c
ssd_multibox PASSED

Best Regards,
Shigehiro Tsuda



  • Hi,

    It was not recognized normally in the following environments as well as the custom model.
    model
    github.com/.../sparse

    github.com/.../sparse

    import
    \usr\share\ti\tidl\utils\test\testvecs\config\import\caffe\tidl_import_JDetNet_768x320.txt
    config
    \usr\share\ti\tidl\examples\test\testvecs\config\infer\tidl_config_jdetnet_voc.txt

    Are there any conditions?
    Could you provide an import file and a config file for ssd to work properly?

    How can I create * .y files in the following directories?
    If you have a * .y file viewer, please let me know.

    \usr\share\ti\tidl\examples\test\testvecs\input

    What is the difference between these files?
    preproc_0_768x320.y
    preproc_0_768x320_multi.y

    preproc_0_224x224.y
    preproc_2_224x224.y

    Best Regards,
    Shigehiro Tsuda

  • In reply to shigehiro tsuda:

    Hi,

    Is there update information?
    TIDL evaluation is not progressing with our customer.

    Best Regards,
    Shigehiro Tsuda

  • In reply to shigehiro tsuda:

    Hi Shigehiro-san,

    Apologies for the delay, several folks were out over the holiday week.

    We are taking a look and will get back to you this week.

    Regards,
    Mike

  • In reply to shigehiro tsuda:

    Tsuda-san, let me ask you few questions, to see if I understand the problem.

    1) Out-of-box demo SSD works OK, but custome SSD model import and runs OK, but there are not boxes displayed? Is my understanding correct?

    2) If so, did you get SSD model from github caffe-jacinto-models SSD? if not any other public place? In case I need it in order to reproduce the issue.

    3) *.y input image, should be the luminance component of an YUV image

    4) Which Linux PSDK and TIDL versions are you using? Linux PSDK 06_01_00_08?

    5) Which AM57x9 board are you using? (just to confirm number of EVE cores)

    6) When running the demo what is your configuration? want to check if running on DSP, EVE or both..

    From http://downloads.ti.com/mctools/esd/docs/tidl-api/example.html#ssd we have:

    The ssd network used in both categories has 43 layers. Input to the network is RGB image of size 768x320. Output is a list of boxes (up to 20), each box has information about the box coordinates, and which pre-trained category that the object inside the box belongs to. The example will take the network output, draw boxes accordingly, and create an output image. The network can be run entirely on either EVE or C66x. However, the best performance comes with running the first 30 layers as a group on EVE and the next 13 layers as another group on C66x. Our end-to-end example shows how easy it is to assign a Layer Group id to an Executor and how easy it is to construct an ExecutionObjectPipeline to connect the output of one Executor’s ExecutionObject to the input of another Executor’s ExecutionObject

    Thank you,

    Paula

  • In reply to Michael Erdahl:

    Hi Mike,

    Thank you for your reply.

    I could understand that several staffs were on holiday last week.
    Please continue to follow me.

    Best Regards,
    Shigehiro Tsuda

  • In reply to Paula Carrillo:

    Hi Paula,

    Thank you for the reply and your support.

    I answer your question below

    1) Out-of-box demo SSD works OK, but custome SSD model import and runs OK, but there are not boxes displayed? Is my understanding correct?

    [answer]
    Yes,it is correct with your understanding.

    2) If so, did you get SSD model from github caffe-jacinto-models SSD? if not any other public place? In case I need it in order to reproduce the issue.

    [answer]
    Yes,I get  SSD model from github caffe-jacinto-models SSD.

    3) *.y input image, should be the luminance component of an YUV image

    [answer]
    Thank you for your answer.
    Can you tell me how to create it?
    I use the raw image which created by this command.
    This is BGR image.

    convert sample.jpg resize 768x320 gravity center extent
    768x320 ./sample_rgb_768x320.png
    convert ./sample_rgb_768x320.png separate +channel swap 0,2 combine colorspace
    sRGB ./sample_bgr_768x320.png
    convert ./sample_bgr_768x320.png interlace plane BGR:sample_bgr_768x320.raw


    4) Which Linux PSDK and TIDL versions are you using? Linux PSDK 06_01_00_08?

    [answer]
    Yes,I use Linux PSDK 06_01_00_08

    5) Which AM57x9 board are you using? (just to confirm number of EVE cores)

    [answer]
    I use tmdsidkAM574 board.
    EVE core have two.

    6) When running the demo what is your configuration? want to check if running on DSP, EVE or both..

    [answer]
    I use both DSP and EVE.

    tidl import file and result file are the attached file.
    Is there problem?

    tidl_import_result2.txt

    Best Regards,
    Shigehiro Tsuda

  • In reply to shigehiro tsuda:

    Tsuda-san, I was able to import  SSD 512x512 (voc0712_ssdJacintoNetV2_iter_104000.caffemodel) from https://github.com/tidsp/caffe-jacinto-models and run it with camera as input. Let me share some configuration files and steps with you, in case it helps.

    1) TIDL import config file for JDetNet: 

    tidl_import_jdetNet_modif.txt 

    I believe this is similar to what you are using.. as your results (attached in above post) looks similar to mine.

    2) Then I copied output "tidl_param_jdetNet_512x512.bin" and "tidl_net_jdetNet_512x512.bin" inside /usr/share/ti/tidl/examples/test/testvecs/config/tidl_models/ folder

    3) Next, I modified /usr/share/ti/tidl/examples/test/testvecs/config/infer/tidl_config_jdetnet.txt to reflect new Width/Height dimensions.. I keep it inData as  "preproc_0_768x320.y" but you can find a better *512x512.y or cropped it..

    1882.tidl_config_jdetnet.txt

    4) For running the demo I used below commands:

    root@am57xx-evm:/usr/share/ti/tidl/examples/ssd_multibox# /etc/init.d/matrix-gui-2.0 stop
    root@am57xx-evm:/usr/share/ti/tidl/examples/ssd_multibox# ./ssd_multibox -i camera -f 500

    I noticed some latency but in general boxes shows OK, at least it detects me, and detected a toy car I had handy.

    Thank you,

    Paula

  • In reply to Paula Carrillo:

    Hi Paula,

    Thank you for your support.
    I will try like your answer.

    There are two questions.
    1.
    The default model of ssd_multibox sample is jdetnet_voc.
    I think you should modify tidl_config_jdetnet_voc.
    Is it wrong?

    2.
    What should be defined "network" of jdetnet_voc_objects.json?
    The default is "jdetnet_voc".
    Result of tidl import is the following network name.

    caffe Network File : ./deploy.prototxt
    Caffe Model File : ./cm-custom_ssdJacintoNetV2_iter_1000.caffemodel
    TIDL Network File : ./tidl_net_cm-custom_ssdJacintoNetV2.bin
    TIDL Model File : ./tidl_param_cm-custom_ssdJacintoNetV2.bin
    Name of the Network : ssdJacintoNetV2_deploy

    Does it need to change?

    Best Regards,
    Shigehiro Tsuda

  • In reply to shigehiro tsuda:

    Tsuda-san, my bad, if I only use "./ssd_multibox -i camera -f 500" in command line for executing ssd_multibox application, then by default "jdetnet_voc" was used. Then, I was evaluating the wrong NN model binary.

    After trying the run the new NN model binary "tidl_param_jdetNet_512x512.bin" and "tidl_net_jdetNet_512x512.bin" it actually didn't work.

    In order to understand what it could be the issue, I used "tidl_viewer" and noticed the division of layers between group 0/1 and group 2 were wrong. I knew it was wrong after creating a similar *.svg graph for OOB "tidl_net_jdetNet_ssd.bin" and comparing them.

    So, I proceed to modify my "tidl_import_jdetNet_modif.txt" "layersGroupId". I didn't used "conv2dKernelType", but you can give a try to this.

    Also, using ffmpeg in Ubuntu, I scaled and crop 1920x1080 *.mp4 video to a 512x512 *.mp4 (attached).

    With above changes, now new 512x512 SSD runs OK. I used: ./ssd_multibox -c jdetnet -l jdetnet_objects.json clips/pexels_2_512x512.mp4

    Answering your questions:

    Q1) yes you need to modify tidl_config_jdetnet_voc if using default..

    Q2) you just need to be careful to call the correct network according with the class list of objects you want to use in *.json

    I am attaching my NN model binaries, just in case they help. 

    jdetNet_512x512.dot.svg

    0131.tidl_import_jdetNet_modif.txt

    tidl_net_jdetNet_512x512.bin

    tidl_param_jdetNet_512x512.bin

  • In reply to Paula Carrillo:

    Hi Paula,

    Thank you for quick reply and your support.
    I would like to try your answer procedure.

    Your import file is the following parameter setting of sampleInData.
    sampleInData = "/home/root/TIDL_utils_bin/testvecs/input/pexels_2_515x512_frame.y"

    Please tell me how to create "pexels_2_515x512_frame.y"

    Best Regards,
    Shigehiro Tsuda