Caffe-Jacinto LeNet on MNIST not working with TIDL

Mladen Stupar

Prodigy 115 points

Hi TI team,

I am trying to convert LeNet trained with Caffe-Jacinto v0.16 using TIDL model import tool.

Training:

- I pulled Caffe-Jacinto v0.16 from GitHub on Ubuntu 16.04 (x64)

- I modified Makefile.config in order to use CPU_ONLY and to use ATLAS instead of BLAS

- Then i built caffe-jacinto with "make all"

- After that

./data/mnist/get_mnist.sh
./examples/mnist/create_mnist.sh
./examples/mnist/train_lenet.sh

to train the net.

My final model (after 10000 iterations) had 99.2% accuracy on test set.

This are files used for training:

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/LeNet.7z

I also tried without

transform_param {
   scale: 0.00390625
}

Converting:

I used both TIDL.00.08.00.00 and TIDL.01.00.00.00 model import tool for converting the trained Caffe-Jacinto model to TIDL model. Those are files used for converting:

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/LeNetConverting.7z

When i run the TIDL model import tool like this

tidl_model_import.out.exe ..\..\test\testvecs\config\import\tidl_import_mnist_caffe_jacinto.txt

I get weird output of the network. As you can see from the import file, i am using MNIST test data and i am running the tool on 10 samples.

The output I get is this:

Fullscreen 0871.output.txt Download

Caffe Network File : ..\..\test\testvecs\config\caffe_jacinto_models\mnist\lenet.prototxt
Caffe Model File   : ..\..\test\testvecs\config\caffe_jacinto_models\mnist\lenet_iter_10000.caffemodel
TIDL Network File  : ..\..\test\testvecs\config\tidl_models\mnist\tidl_mnist_caffe_jacinto_016_net.bin
TIDL Model File    : ..\..\test\testvecs\config\tidl_models\mnist\tidl_mnist_caffe_jacinto_016_param.bin
Name of the Network :           LeNet
Num Inputs :               1
 Num of Layer Detected :   6
  0, TIDL_DataLayer                , data                            0,  -1 ,  1 ,   x ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  0 ,    0 ,    0 ,    0 ,    0 ,    1 ,    2 ,   28 ,
28 ,         0 ,
  1, TIDL_ConvolutionLayer         , conv1                           1,   1 ,  1 ,   0 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  1 ,    1 ,    2 ,   28 ,   28 ,    1 ,   20 ,   12 ,
12 ,    288000 ,
  2, TIDL_ConvolutionLayer         , conv2                           1,   1 ,  1 ,   1 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  2 ,    1 ,   20 ,   12 ,   12 ,    1 ,   50 ,    4 ,
 4 ,   1600000 ,
  3, TIDL_InnerProductLayer        , ip1                             1,   1 ,  1 ,   2 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  3 ,    1 ,   50 ,    4 ,    4 ,    1 ,    1 ,    1 ,  5
00 ,    400000 ,
  4, TIDL_InnerProductLayer        , ip2                             1,   1 ,  1 ,   3 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  4 ,    1 ,    1 ,    1 ,  500 ,    1 ,    1 ,    1 ,
10 ,      5000 ,
  5, TIDL_SoftMaxLayer             , prob                            1,   1 ,  1 ,   4 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  5 ,    1 ,    1 ,    1 ,   10 ,    1 ,    1 ,    1 ,
10 ,        10 ,
Total Giga Macs : 0.0023
Total Giga Macs : 0.0344 @15 fps
Total Giga Macs : 0.0688 @30 fps
        1 file(s) copied.

Processing config file .\tempDir\qunat_stats_config.txt !
  0, TIDL_DataLayer                ,  0,  -1 ,  1 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  0 ,    0 ,    0 ,    0 ,    0 ,    1 ,    2 ,   28 ,   28 ,
  1, TIDL_ConvolutionLayer         ,  1,   1 ,  1 ,  0 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  1 ,    1 ,    2 ,   28 ,   28 ,    1 ,   20 ,   12 ,   12 ,
  2, TIDL_ConvolutionLayer         ,  1,   1 ,  1 ,  1 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  2 ,    1 ,   20 ,   12 ,   12 ,    1 ,   50 ,    4 ,    4 ,
  3, TIDL_InnerProductLayer        ,  1,   1 ,  1 ,  2 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  3 ,    1 ,   50 ,    4 ,    4 ,    1 ,    1 ,    1 ,  500 ,
  4, TIDL_InnerProductLayer        ,  1,   1 ,  1 ,  3 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  4 ,    1 ,    1 ,    1 ,  500 ,    1 ,    1 ,    1 ,   10 ,
  5, TIDL_SoftMaxLayer             ,  1,   1 ,  1 ,  4 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  5 ,    1 ,    1 ,    1 ,   10 ,    1 ,    1 ,    1 ,   10 ,
  6, TIDL_DataLayer                ,  0,   1 , -1 ,  5 ,  x ,  x ,  x ,  x ,  x ,  x ,  x ,  0 ,    1 ,    1 ,    1 ,   10 ,    0 ,    0 ,    0 ,    0 ,
Layer ID    ,inBlkWidth  ,inBlkHeight ,inBlkPitch  ,outBlkWidth ,outBlkHeight,outBlkPitch ,numInChs    ,numOutChs   ,numProcInChs,numLclInChs ,numLclOutChs,numProcItrs ,numAccItrs
 ,numHorBlock ,numVerBlock ,inBlkChPitch,outBlkChPitc,alignOrNot
      1           40           28           40           32           24           32            2           20            2            1            8            1            2
        1            1         1120          768            1
      2           24           12           24           16            8           16           20           50           20            8            8            1            3
        1            1          288          128            1

Processing Frame Number : 0

 Layer    1 : Max PASS : -2147483648 :   219761 Out Q :       39 ,   442983, TIDL_ConvolutionLayer, PASSED  #MMACs =     0.58,     0.40,     0.44, Sparsity :  23.20,  30.40
 Layer    2 : Max PASS : -2147483648 :   507641 Out Q :        8 ,  1023276, TIDL_ConvolutionLayer, PASSED  #MMACs =     1.60,     1.78,     1.79, Sparsity : -11.81, -11.01
 Layer    3 : Max PASS : -2147483648 :    10798 Out Q :      267 ,    10840, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.03,     0.00,     0.03, Sparsity :   0.00, 100.00
 Layer    4 : Max PASS : -2147483648 :    47034 Out Q :       93 ,   373464, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00
 Layer    5 :-------Max Index    0 : 255 ------- #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00

Processing Frame Number : 1

 Layer    1 : Max PASS :   219761 :   195796 Out Q :       44 ,   394675, TIDL_ConvolutionLayer, PASSED  #MMACs =     0.58,     0.40,     0.44, Sparsity :  23.20,  30.40
 Layer    2 : Max PASS :   507641 :   373450 Out Q :        9 ,   927974, TIDL_ConvolutionLayer, PASSED  #MMACs =     1.60,     1.78,     1.79, Sparsity : -11.81, -11.01
 Layer    3 : Max PASS :    10798 :    34689 Out Q :       93 ,    34825, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.03,     0.00,     0.03, Sparsity :   0.00, 100.00
 Layer    4 : Max PASS :    47034 :    46066 Out Q :       32 ,   379049, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00
 Layer    5 :-------Max Index    0 : 255 ------- #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00

Processing Frame Number : 2

 Layer    1 : Max PASS :   217365 :   217155 Out Q :       40 ,   437730, TIDL_ConvolutionLayer, PASSED  #MMACs =     0.58,     0.40,     0.44, Sparsity :  23.20,  30.40
 Layer    2 : Max PASS :   494222 :   487886 Out Q :        8 ,  1043297, TIDL_ConvolutionLayer, PASSED  #MMACs =     1.60,     1.78,     1.79, Sparsity : -11.81, -11.01
 Layer    3 : Max PASS :    13187 :    31407 Out Q :       92 ,    31530, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.03,     0.00,     0.03, Sparsity :   0.00, 100.00
 Layer    4 : Max PASS :    46938 :   -12483 Out Q :       27 ,   445281, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00
 Layer    5 :-------Max Index    0 : 254 ------- #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00

Processing Frame Number : 3

 Layer    1 : Max PASS :   217344 :   212397 Out Q :       40 ,   428139, TIDL_ConvolutionLayer, PASSED  #MMACs =     0.58,     0.40,     0.44, Sparsity :  23.20,  30.40
 Layer    2 : Max PASS :   493589 :   534392 Out Q :        7 ,  1077200, TIDL_ConvolutionLayer, PASSED  #MMACs =     1.60,     1.78,     1.79, Sparsity : -11.81, -11.01
 Layer    3 : Max PASS :    15009 :    55597 Out Q :       45 ,    55815, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.03,     0.00,     0.03, Sparsity :   0.00, 100.00
 Layer    4 : Max PASS :    40996 :    -1438 Out Q :       18 ,   328924, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00
 Layer    5 :-------Max Index    3 : 255 ------- #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00

Processing Frame Number : 4

 Layer    1 : Max PASS :   216850 :   223228 Out Q :       38 ,   449971, TIDL_ConvolutionLayer, PASSED  #MMACs =     0.58,     0.40,     0.44, Sparsity :  23.20,  30.40
 Layer    2 : Max PASS :   497669 :   522722 Out Q :        6 ,  1203510, TIDL_ConvolutionLayer, PASSED  #MMACs =     1.60,     1.78,     1.79, Sparsity : -11.81, -11.01
 Layer    3 : Max PASS :    19067 :    33610 Out Q :       64 ,    33742, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.03,     0.00,     0.03, Sparsity :   0.00, 100.00
 Layer    4 : Max PASS :    36753 :    45671 Out Q :       23 ,   361823, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00
 Layer    5 :-------Max Index    0 : 254 ------- #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00

Processing Frame Number : 5

 Layer    1 : Max PASS :   217487 :   236129 Out Q :       36 ,   475977, TIDL_ConvolutionLayer, PASSED  #MMACs =     0.58,     0.40,     0.44, Sparsity :  23.20,  30.40
 Layer    2 : Max PASS :   500174 :   468077 Out Q :        7 ,  1028292, TIDL_ConvolutionLayer, PASSED  #MMACs =     1.60,     1.78,     1.79, Sparsity : -11.81, -11.01
 Layer    3 : Max PASS :    20521 :    42438 Out Q :       59 ,    42604, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.03,     0.00,     0.03, Sparsity :   0.00, 100.00
 Layer    4 : Max PASS :    37644 :      272 Out Q :       20 ,   378015, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00
 Layer    5 :-------Max Index    3 : 255 ------- #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00

Processing Frame Number : 6

 Layer    1 : Max PASS :   219351 :   228737 Out Q :       38 ,   461076, TIDL_ConvolutionLayer, PASSED  #MMACs =     0.58,     0.40,     0.44, Sparsity :  23.20,  30.40
 Layer    2 : Max PASS :   496965 :   483652 Out Q :        7 ,  1132697, TIDL_ConvolutionLayer, PASSED  #MMACs =     1.60,     1.78,     1.79, Sparsity : -11.81, -11.01
 Layer    3 : Max PASS :    22712 :    19603 Out Q :      129 ,    19680, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.03,     0.00,     0.03, Sparsity :   0.00, 100.00
 Layer    4 : Max PASS :    33907 :   -26145 Out Q :       29 ,   581989, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00
 Layer    5 :-------Max Index    6 : 255 ------- #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00

Processing Frame Number : 7

 Layer    1 : Max PASS :   220289 :   230042 Out Q :       37 ,   463707, TIDL_ConvolutionLayer, PASSED  #MMACs =     0.58,     0.40,     0.44, Sparsity :  23.20,  30.40
 Layer    2 : Max PASS :   495634 :   345923 Out Q :        7 ,  1019630, TIDL_ConvolutionLayer, PASSED  #MMACs =     1.60,     1.78,     1.79, Sparsity : -11.81, -11.01
 Layer    3 : Max PASS :    22402 :    34403 Out Q :       73 ,    34538, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.03,     0.00,     0.03, Sparsity :   0.00, 100.00
 Layer    4 : Max PASS :    27902 :    46251 Out Q :       25 ,   378308, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00
 Layer    5 :-------Max Index    0 : 255 ------- #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00

Processing Frame Number : 8

 Layer    1 : Max PASS :   221264 :   236664 Out Q :       36 ,   477055, TIDL_ConvolutionLayer, PASSED  #MMACs =     0.58,     0.40,     0.44, Sparsity :  23.20,  30.40
 Layer    2 : Max PASS :   480663 :   476167 Out Q :        7 ,   972635, TIDL_ConvolutionLayer, PASSED  #MMACs =     1.60,     1.78,     1.79, Sparsity : -11.81, -11.01
 Layer    3 : Max PASS :    23602 :    29096 Out Q :       87 ,    29210, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.03,     0.00,     0.03, Sparsity :   0.00, 100.00
 Layer    4 : Max PASS :    29736 :   -40317 Out Q :       23 ,   494382, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00
 Layer    5 :-------Max Index    3 : 255 ------- #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00

Processing Frame Number : 9

 Layer    1 : Max PASS :   222804 :   238222 Out Q :       36 ,   480196, TIDL_ConvolutionLayer, PASSED  #MMACs =     0.58,     0.40,     0.44, Sparsity :  23.20,  30.40
 Layer    2 : Max PASS :   480214 :   581398 Out Q :        6 ,  1186512, TIDL_ConvolutionLayer, PASSED  #MMACs =     1.60,     1.78,     1.79, Sparsity : -11.81, -11.01
 Layer    3 : Max PASS :    24151 :    30285 Out Q :       71 ,    30404, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.03,     0.00,     0.03, Sparsity :   0.00, 100.00
 Layer    4 : Max PASS :    22731 :    37500 Out Q :       23 ,   405700, TIDL_InnerProductLayer,     PASSED  #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00
 Layer    5 :-------Max Index    0 : 254 ------- #MMACs =     0.00,     0.00,     0.00, Sparsity :   0.00, 100.00

The network is giving the same Max Index-es always (randomly 0, 3 and 6). Also as you can notice from the output file the tool is not recognizing Pooling layers (size: 2, stride: 2), can that be a problem?

What am I doing wrong here? When i convert trained cifar-10 from https://github.com/tidsp/caffe-jacinto-models everything work as expected (it classifies test data well) but it's not same with the LeNet MNIST.

Best regards

Mladen Stupar

RT-RK

over 7 years ago

0 kumar.desappan over 7 years ago

TI__Mastermind 22145 points

Mladen Stupar,

TIDL_InnerProductLayer in TIDL expects the input data to be flattened ( N=1 and H=1). Please add Flatten layer (refer below) before first Inner product layer (FC layer) in you deploy prototext (No need to re-train) and try importing again.

layer {
name: "flatten"
type: "Flatten"
bottom: "conv1b"
top: "flatten"
flatten_param {
axis: 1
}
}

0 Mladen Stupar over 7 years ago in reply to kumar.desappan

Prodigy 115 points

Like this?

...
layer {
name: "pool2"
type: "Pooling"
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "flatten"
type: "Flatten"
bottom: "pool2"
top: "flatten"
flatten_param {
axis: 1
}
}
layer {
name: "ip1"
type: "InnerProduct"
bottom: "flatten"
top: "ip1"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
inner_product_param {
num_output: 500
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
...

0 kumar.desappan over 7 years ago in reply to Mladen Stupar

TI__Mastermind 22145 points

Yes you are right.

Thanks and regards

Kumar.D

0 Mladen Stupar over 7 years ago in reply to kumar.desappan

Prodigy 115 points

I tried with the Flatten layer, and it still doesn't work as expected. Can you please check if my lenet_solver.txt and lenet_train_test.txt files are good or the issue is somewhere else? Can you reproduce the same output with provided .caffemodel and .prototxt files? Is it OK to have transform_param in data layer in training phase?

Thanks in advance
Mladen Stupar

0 kumar.desappan over 7 years ago in reply to Mladen Stupar

TI__Mastermind 22145 points

Using transform in training shall be ok. I hope you have validated caffe interfere using deploy prototxt. Caffe inference input and TIDL inference input shall match.

For debugging this, you may need dump layer level features from caffe and match with TIDL import traces .

TDIL trace is in fixed point and can be converted to floating point ( using OutQ printed in the import log)

Due to quantisation the results will not exactly match, but will be similar.
Please make sure the input matches and share the layer where it is mismatching.

(OutQ/256) shall be used for deviding TIDL trace to get float value.
Thanks and regards
Kumar.D

0 kumar.desappan over 7 years ago in reply to kumar.desappan

TI__Mastermind 22145 points

Please check the format of input used during import

sampleInData = "..\..\test\testvecs\input\t10k-images.idx3-ubyte" // This Shall be sample binary data which is passed a input to first layer of your net. (basically your input tensor of size 28x28 byes)

Rgerads,

Kumar.D

0 Mladen Stupar over 7 years ago in reply to kumar.desappan

Prodigy 115 points

The input format is good. It's raw 28x28 bytes.
Now I changed the NET, I trained MNIST dataset on jacintonet just to ensure it is possible and it works with TIDL Convert Tool (it classifies digits well).
Then I changed tidl usecase in order to support gray inputs (1 channel). It looks like this:
NullSrc -> TIDLProcess(EVE1) -> Null
I removed TIDLPreProc AlgPlugin because I have only 1 channel in input image, so no need for TIDLPreProc (at least I think), but I can not get similar output as with the TIDL convert tool (it always gives the same output class).
Is this possible to do with the current version of TIDL? Because every network I've tried with 3 channels (RGB) and TIDLPreProc worked well, but this one, with one color channel, doesn't work.
Can you tell me what format TIDLProcess expects? From NullSrc link, AlgPlugin TIDL receives one channel RAW08 data of size 28*28 and the output is wrong. Do I need to preprocess the data somehow even when they are already in format which the network expects?

For the record, I am using PROCESSOR_SDK_VISION_03.01.00.00

Regards,

Mladen Stupar

0 kumar.desappan over 7 years ago in reply to Mladen Stupar

TI__Mastermind 22145 points

Please refer the TIDL standalone test bench code ti_tb.c to prepare input for TIDL process.

Please make changes to your null source in the vision SDK use case based on this test bench code.

Thanks and Regards

Kumar.D

0 Mladen Stupar over 7 years ago in reply to kumar.desappan

Prodigy 115 points

Thanks, this resolved problem.

For the others having the same problem, TIDL AlgPlugin expects input format per channel like on image below:

#define TIDL_MAX_PAD_SIZE (4)
at the moment

0 kumar.desappan over 7 years ago in reply to Mladen Stupar

TI__Mastermind 22145 points

Hi Mladen Stupar,
Thanks for updating this info here. We will add this detail in the user guide.

Thanks and Regards,
Kumar.D

Processors

Processors forum

Caffe-Jacinto LeNet on MNIST not working with TIDL