Part Number: TDA2SX
Dear Champs,
When my customer tried to make using Jacinto-caffe, they faced error while cuda run test as below attached log.
Could you please let me know if there is any issue in their setup or what is the issue?
.build_release/tools/caffe
I0212 16:33:57.105792 3278 caffe.cpp:902] This is NVCaffe 0.17.0 started at Wed Feb 12 16:33:57 2020
I0212 16:33:57.257547 3278 caffe.cpp:904] CuDNN version: 7605
I0212 16:33:57.257551 3278 caffe.cpp:905] CuBLAS version: 9010
I0212 16:33:57.257553 3278 caffe.cpp:906] CUDA version: 10020
I0212 16:33:57.257555 3278 caffe.cpp:907] CUDA driver version: 10020
I0212 16:33:57.257556 3278 caffe.cpp:908] Arguments:
[0]: .build_release/tools/caffe
caffe: command line brew
usage: caffe <command> <args>
commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time
Flags from tools/caffe.cpp:
-ap_version (Average Precision type for object detection) type: string
default: "11point"
-display_sparsity (Display the amount of sparsity) type: bool
default: false
-gpu (Optional; run in GPU mode on given device IDs separated by ', '.Use
'-gpu all' to run on all available GPUs. The effective training batch
size is multiplied by the number of devices.) type: string default: ""
-iterations (The number of iterations to run.) type: int32 default: 50
-level (Optional; network level.) type: int32 default: 0
-model (The model definition protocol buffer text file.) type: string
default: ""
-optimize_net (Optimize the Net (Merge BN to Conv) before test) type: bool
default: false
-output_model (Prefix for output prototxt and caffemodel) type: string
default: ""
-phase (Optional; network phase (TRAIN or TEST). Only used for 'time'.)
type: string default: ""
-show_per_class_result (Show per class result for object detection)
type: bool default: true
-sighup_effect (Optional; action to take when a SIGHUP signal is received:
snapshot, stop or none.) type: string default: "snapshot"
-sigint_effect (Optional; action to take when a SIGINT signal is received:
snapshot, stop or none.) type: string default: "stop"
-snapshot (Optional; the snapshot solver state to resume training.)
type: string default: ""
-solver (The solver definition protocol buffer text file.) type: string
default: ""
-stage (Optional; network stages (not to be confused with phase), separated
by ','.) type: string default: ""
-weights (Optional; the pretrained weights to initialize finetuning,
separated by ', '. Cannot be set simultaneously with snapshot.)
type: string default: ""
.build_release/test/test_all.testbin 0 --gtest_shuffle
Cuda number of devices: 1
Setting to use device 0
Current device id: 0
Current device name: GeForce RTX 2080 Ti
Note: Randomizing tests' orders with a seed of 82163 .
[==========] Running 2101 tests from 283 test cases.
[----------] Global test environment set-up.
[----------] 8 tests from SliceLayerTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN ] SliceLayerTest/1.TestSetupNum
[ OK ] SliceLayerTest/1.TestSetupNum (0 ms)
[ RUN ] SliceLayerTest/1.TestTrivialSlice
[ OK ] SliceLayerTest/1.TestTrivialSlice (0 ms)
[ RUN ] SliceLayerTest/1.TestSliceAcrossChannels
[ OK ] SliceLayerTest/1.TestSliceAcrossChannels (0 ms)
[ RUN ] SliceLayerTest/1.TestGradientAcrossChannels
[ OK ] SliceLayerTest/1.TestGradientAcrossChannels (18 ms)
[ RUN ] SliceLayerTest/1.TestGradientAcrossNum
[ OK ] SliceLayerTest/1.TestGradientAcrossNum (17 ms)
[ RUN ] SliceLayerTest/1.TestSetupChannels
[ OK ] SliceLayerTest/1.TestSetupChannels (0 ms)
[ RUN ] SliceLayerTest/1.TestSliceAcrossNum
[ OK ] SliceLayerTest/1.TestSliceAcrossNum (0 ms)
[ RUN ] SliceLayerTest/1.TestGradientTrivial
[ OK ] SliceLayerTest/1.TestGradientTrivial (0 ms)
[----------] 8 tests from SliceLayerTest/1 (35 ms total)
[----------] 5 tests from DeconvolutionLayerTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN ] DeconvolutionLayerTest/1.TestSetup
[ OK ] DeconvolutionLayerTest/1.TestSetup (0 ms)
[ RUN ] DeconvolutionLayerTest/1.TestSimpleDeconvolution
[ OK ] DeconvolutionLayerTest/1.TestSimpleDeconvolution (7 ms)
[ RUN ] DeconvolutionLayerTest/1.TestGradient3D
[ OK ] DeconvolutionLayerTest/1.TestGradient3D (84 ms)
[ RUN ] DeconvolutionLayerTest/1.TestNDAgainst2D
[ OK ] DeconvolutionLayerTest/1.TestNDAgainst2D (499 ms)
[ RUN ] DeconvolutionLayerTest/1.TestGradient
[ OK ] DeconvolutionLayerTest/1.TestGradient (355 ms)
[----------] 5 tests from DeconvolutionLayerTest/1 (945 ms total)
[----------] 1 test from SolverFactoryTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN ] SolverFactoryTest/0.TestCreateSolver
[ OK ] SolverFactoryTest/0.TestCreateSolver (4 ms)
[----------] 1 test from SolverFactoryTest/0 (4 ms total)
[----------] 3 tests from XavierFillerTest/1, where TypeParam = double
[ RUN ] XavierFillerTest/1.TestFillFanIn
[ OK ] XavierFillerTest/1.TestFillFanIn (1 ms)
[ RUN ] XavierFillerTest/1.TestFillFanOut
[ OK ] XavierFillerTest/1.TestFillFanOut (0 ms)
[ RUN ] XavierFillerTest/1.TestFillAverage
[ OK ] XavierFillerTest/1.TestFillAverage (1 ms)
[----------] 3 tests from XavierFillerTest/1 (2 ms total)
[----------] 10 tests from DetectNetTransformationLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN ] DetectNetTransformationLayerTest/0.TestRotation
[ OK ] DetectNetTransformationLayerTest/0.TestRotation (58 ms)
[ RUN ] DetectNetTransformationLayerTest/0.TestScaleUp
[ OK ] DetectNetTransformationLayerTest/0.TestScaleUp (0 ms)
[ RUN ] DetectNetTransformationLayerTest/0.TestScaleDown
[ OK ] DetectNetTransformationLayerTest/0.TestScaleDown (0 ms)
[ RUN ] DetectNetTransformationLayerTest/0.TestHueRotation
[ OK ] DetectNetTransformationLayerTest/0.TestHueRotation (0 ms)
[ RUN ] DetectNetTransformationLayerTest/0.TestNoAugmentation
[ OK ] DetectNetTransformationLayerTest/0.TestNoAugmentation (0 ms)
[ RUN ] DetectNetTransformationLayerTest/0.TestDesaturation
[ OK ] DetectNetTransformationLayerTest/0.TestDesaturation (0 ms)
[ RUN ] DetectNetTransformationLayerTest/0.TestAllAugmentation
[ OK ] DetectNetTransformationLayerTest/0.TestAllAugmentation (0 ms)
[ RUN ] DetectNetTransformationLayerTest/0.TestFlip
[ OK ] DetectNetTransformationLayerTest/0.TestFlip (0 ms)
[ RUN ] DetectNetTransformationLayerTest/0.TestSetup
[ OK ] DetectNetTransformationLayerTest/0.TestSetup (0 ms)
[ RUN ] DetectNetTransformationLayerTest/0.TestCrop
[ OK ] DetectNetTransformationLayerTest/0.TestCrop (0 ms)
[----------] 10 tests from DetectNetTransformationLayerTest/0 (58 ms total)
[----------] 6 tests from FlattenLayerTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN ] FlattenLayerTest/3.TestSetupWithEndAxis
[ OK ] FlattenLayerTest/3.TestSetupWithEndAxis (5 ms)
[ RUN ] FlattenLayerTest/3.TestSetupWithAxis
[ OK ] FlattenLayerTest/3.TestSetupWithAxis (1 ms)
[ RUN ] FlattenLayerTest/3.TestSetup
[ OK ] FlattenLayerTest/3.TestSetup (0 ms)
[ RUN ] FlattenLayerTest/3.TestSetupWithStartAndEndAxis
[ OK ] FlattenLayerTest/3.TestSetupWithStartAndEndAxis (1 ms)
[ RUN ] FlattenLayerTest/3.TestForward
[ OK ] FlattenLayerTest/3.TestForward (0 ms)
[ RUN ] FlattenLayerTest/3.TestGradient
[ OK ] FlattenLayerTest/3.TestGradient (1 ms)
[----------] 6 tests from FlattenLayerTest/3 (8 ms total)
[----------] 3 tests from VarSzTransformsTest/0, where TypeParam = float
[ RUN ] VarSzTransformsTest/0.TestVarSzImgRandomResize
[ OK ] VarSzTransformsTest/0.TestVarSzImgRandomResize (0 ms)
[ RUN ] VarSzTransformsTest/0.TestVarSzImgRandomCrop
[ OK ] VarSzTransformsTest/0.TestVarSzImgRandomCrop (0 ms)
[ RUN ] VarSzTransformsTest/0.TestVarSzImgCenterCrop
[ OK ] VarSzTransformsTest/0.TestVarSzImgCenterCrop (0 ms)
[----------] 3 tests from VarSzTransformsTest/0 (0 ms total)
[----------] 3 tests from XavierStaticFillerTest/1, where TypeParam = double
[ RUN ] XavierStaticFillerTest/1.TestFillAverage
[ OK ] XavierStaticFillerTest/1.TestFillAverage (1 ms)
[ RUN ] XavierStaticFillerTest/1.TestFillFanOut
[ OK ] XavierStaticFillerTest/1.TestFillFanOut (1 ms)
[ RUN ] XavierStaticFillerTest/1.TestFillFanIn
[ OK ] XavierStaticFillerTest/1.TestFillFanIn (1 ms)
[----------] 3 tests from XavierStaticFillerTest/1 (3 ms total)
[----------] 1 test from LayerFactoryTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN ] LayerFactoryTest/3.TestCreateLayer
src/caffe/test/test_layer_factory.cpp:45: Failure
Value of: layer->type()
Actual: "Accuracy"
Expected: iter->first
Which is: "SegmentationAccuracy"
[ FAILED ] LayerFactoryTest/3.TestCreateLayer, where TypeParam = caffe::GPUDevice<double> (16 ms)
[----------] 1 test from LayerFactoryTest/3 (16 ms total)
[----------] 12 tests from ReshapeLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN ] ReshapeLayerTest/0.TestFlattenOutputSizes
[ OK ] ReshapeLayerTest/0.TestFlattenOutputSizes (0 ms)
[ RUN ] ReshapeLayerTest/0.TestForwardAfterReshape
[ OK ] ReshapeLayerTest/0.TestForwardAfterReshape (0 ms)
[ RUN ] ReshapeLayerTest/0.TestFlattenValues
[ OK ] ReshapeLayerTest/0.TestFlattenValues (0 ms)
[ RUN ] ReshapeLayerTest/0.TestForward
[ OK ] ReshapeLayerTest/0.TestForward (0 ms)
[ RUN ] ReshapeLayerTest/0.TestCopyDimensions
[ OK ] ReshapeLayerTest/0.TestCopyDimensions (0 ms)
[ RUN ] ReshapeLayerTest/0.TestInsertSingletonAxesStart
[ OK ] ReshapeLayerTest/0.TestInsertSingletonAxesStart (0 ms)
[ RUN ] ReshapeLayerTest/0.TestInferenceOfUnspecifiedWithStartAxis
[ OK ] ReshapeLayerTest/0.TestInferenceOfUnspecifiedWithStartAxis (0 ms)
[ RUN ] ReshapeLayerTest/0.TestInferenceOfUnspecified
[ OK ] ReshapeLayerTest/0.TestInferenceOfUnspecified (0 ms)
[ RUN ] ReshapeLayerTest/0.TestFlattenMiddle
[ OK ] ReshapeLayerTest/0.TestFlattenMiddle (0 ms)
[ RUN ] ReshapeLayerTest/0.TestInsertSingletonAxesEnd
[ OK ] ReshapeLayerTest/0.TestInsertSingletonAxesEnd (0 ms)
[ RUN ] ReshapeLayerTest/0.TestInsertSingletonAxesMiddle
[ OK ] ReshapeLayerTest/0.TestInsertSingletonAxesMiddle (0 ms)
[ RUN ] ReshapeLayerTest/0.TestGradient
[ OK ] ReshapeLayerTest/0.TestGradient (0 ms)
[----------] 12 tests from ReshapeLayerTest/0 (0 ms total)
[----------] 6 tests from FlattenLayerTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN ] FlattenLayerTest/1.TestSetupWithEndAxis
[ OK ] FlattenLayerTest/1.TestSetupWithEndAxis (0 ms)
[ RUN ] FlattenLayerTest/1.TestSetupWithStartAndEndAxis
[ OK ] FlattenLayerTest/1.TestSetupWithStartAndEndAxis (0 ms)
[ RUN ] FlattenLayerTest/1.TestSetup
[ OK ] FlattenLayerTest/1.TestSetup (0 ms)
[ RUN ] FlattenLayerTest/1.TestSetupWithAxis
[ OK ] FlattenLayerTest/1.TestSetupWithAxis (0 ms)
[ RUN ] FlattenLayerTest/1.TestForward
[ OK ] FlattenLayerTest/1.TestForward (0 ms)
[ RUN ] FlattenLayerTest/1.TestGradient
[ OK ] FlattenLayerTest/1.TestGradient (0 ms)
[----------] 6 tests from FlattenLayerTest/1 (0 ms total)
[----------] 12 tests from NesterovSolverTest/0, where TypeParam = caffe::GPUDevice<float>
[ RUN ] NesterovSolverTest/0.TestNesterovLeastSquaresUpdateLROneHundredth
F0212 16:33:58.809607 3309 math_functions.cu:24] Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED
*** Check failure stack trace: ***
@ 0x7fe510f840cd google::LogMessage::Fail()
@ 0x7fe510f85f33 google::LogMessage::SendToLog()
@ 0x7fe510f83c28 google::LogMessage::Flush()
@ 0x7fe510f86999 google::LogMessageFatal::~LogMessageFatal()
@ 0x7fe50df6208e caffe::caffe_gpu_gemm<>()
@ 0x7fe50e05af8c caffe::InnerProductLayer<>::Forward_gpu()
@ 0x564439d16156 caffe::Layer<>::Forward()
@ 0x7fe50db7f3d3 caffe::Net::ForwardFromTo()
@ 0x7fe50db7f567 caffe::Net::Forward()
@ 0x564439ee77cf caffe::GradientBasedSolverTest<>::ComputeLeastSquaresUpdate()
@ 0x564439eea25b caffe::GradientBasedSolverTest<>::TestLeastSquaresUpdate()
@ 0x56443a184c7a testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x56443a17defa testing::Test::Run()
@ 0x56443a17dfdc testing::TestInfo::Run()
@ 0x56443a17e115 testing::TestCase::Run()
@ 0x56443a17e5f0 testing::internal::UnitTestImpl::RunAllTests()
@ 0x56443a17e737 testing::UnitTest::Run()
@ 0x564439cfa394 main
@ 0x7fe50c9a0b97 __libc_start_main
@ 0x564439d05c0a _start
Makefile:560: recipe for target 'runtest' failed
make: *** [runtest] 중지됨 (메모리 덤프됨)
There was no issue with Caffe, and their Host PC is Ubuntu18.04 and graphic card is 2080 TI.
Thanks and Best Regards,
SI.
