This thread has been locked.
If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.
Dear Champs,
When my customer tried to make using Jacinto-caffe, they faced error while cuda run test as below attached log.
Could you please let me know if there is any issue in their setup or what is the issue?
.build_release/tools/caffe I0212 16:33:57.105792 3278 caffe.cpp:902] This is NVCaffe 0.17.0 started at Wed Feb 12 16:33:57 2020 I0212 16:33:57.257547 3278 caffe.cpp:904] CuDNN version: 7605 I0212 16:33:57.257551 3278 caffe.cpp:905] CuBLAS version: 9010 I0212 16:33:57.257553 3278 caffe.cpp:906] CUDA version: 10020 I0212 16:33:57.257555 3278 caffe.cpp:907] CUDA driver version: 10020 I0212 16:33:57.257556 3278 caffe.cpp:908] Arguments: [0]: .build_release/tools/caffe caffe: command line brew usage: caffe <command> <args> commands: train train or finetune a model test score a model device_query show GPU diagnostic information time benchmark model execution time Flags from tools/caffe.cpp: -ap_version (Average Precision type for object detection) type: string default: "11point" -display_sparsity (Display the amount of sparsity) type: bool default: false -gpu (Optional; run in GPU mode on given device IDs separated by ', '.Use '-gpu all' to run on all available GPUs. The effective training batch size is multiplied by the number of devices.) type: string default: "" -iterations (The number of iterations to run.) type: int32 default: 50 -level (Optional; network level.) type: int32 default: 0 -model (The model definition protocol buffer text file.) type: string default: "" -optimize_net (Optimize the Net (Merge BN to Conv) before test) type: bool default: false -output_model (Prefix for output prototxt and caffemodel) type: string default: "" -phase (Optional; network phase (TRAIN or TEST). Only used for 'time'.) type: string default: "" -show_per_class_result (Show per class result for object detection) type: bool default: true -sighup_effect (Optional; action to take when a SIGHUP signal is received: snapshot, stop or none.) type: string default: "snapshot" -sigint_effect (Optional; action to take when a SIGINT signal is received: snapshot, stop or none.) type: string default: "stop" -snapshot (Optional; the snapshot solver state to resume training.) type: string default: "" -solver (The solver definition protocol buffer text file.) type: string default: "" -stage (Optional; network stages (not to be confused with phase), separated by ','.) type: string default: "" -weights (Optional; the pretrained weights to initialize finetuning, separated by ', '. Cannot be set simultaneously with snapshot.) type: string default: "" .build_release/test/test_all.testbin 0 --gtest_shuffle Cuda number of devices: 1 Setting to use device 0 Current device id: 0 Current device name: GeForce RTX 2080 Ti Note: Randomizing tests' orders with a seed of 82163 . [==========] Running 2101 tests from 283 test cases. [----------] Global test environment set-up. [----------] 8 tests from SliceLayerTest/1, where TypeParam = caffe::CPUDevice<double> [ RUN ] SliceLayerTest/1.TestSetupNum [ OK ] SliceLayerTest/1.TestSetupNum (0 ms) [ RUN ] SliceLayerTest/1.TestTrivialSlice [ OK ] SliceLayerTest/1.TestTrivialSlice (0 ms) [ RUN ] SliceLayerTest/1.TestSliceAcrossChannels [ OK ] SliceLayerTest/1.TestSliceAcrossChannels (0 ms) [ RUN ] SliceLayerTest/1.TestGradientAcrossChannels [ OK ] SliceLayerTest/1.TestGradientAcrossChannels (18 ms) [ RUN ] SliceLayerTest/1.TestGradientAcrossNum [ OK ] SliceLayerTest/1.TestGradientAcrossNum (17 ms) [ RUN ] SliceLayerTest/1.TestSetupChannels [ OK ] SliceLayerTest/1.TestSetupChannels (0 ms) [ RUN ] SliceLayerTest/1.TestSliceAcrossNum [ OK ] SliceLayerTest/1.TestSliceAcrossNum (0 ms) [ RUN ] SliceLayerTest/1.TestGradientTrivial [ OK ] SliceLayerTest/1.TestGradientTrivial (0 ms) [----------] 8 tests from SliceLayerTest/1 (35 ms total) [----------] 5 tests from DeconvolutionLayerTest/1, where TypeParam = caffe::CPUDevice<double> [ RUN ] DeconvolutionLayerTest/1.TestSetup [ OK ] DeconvolutionLayerTest/1.TestSetup (0 ms) [ RUN ] DeconvolutionLayerTest/1.TestSimpleDeconvolution [ OK ] DeconvolutionLayerTest/1.TestSimpleDeconvolution (7 ms) [ RUN ] DeconvolutionLayerTest/1.TestGradient3D [ OK ] DeconvolutionLayerTest/1.TestGradient3D (84 ms) [ RUN ] DeconvolutionLayerTest/1.TestNDAgainst2D [ OK ] DeconvolutionLayerTest/1.TestNDAgainst2D (499 ms) [ RUN ] DeconvolutionLayerTest/1.TestGradient [ OK ] DeconvolutionLayerTest/1.TestGradient (355 ms) [----------] 5 tests from DeconvolutionLayerTest/1 (945 ms total) [----------] 1 test from SolverFactoryTest/0, where TypeParam = caffe::CPUDevice<float> [ RUN ] SolverFactoryTest/0.TestCreateSolver [ OK ] SolverFactoryTest/0.TestCreateSolver (4 ms) [----------] 1 test from SolverFactoryTest/0 (4 ms total) [----------] 3 tests from XavierFillerTest/1, where TypeParam = double [ RUN ] XavierFillerTest/1.TestFillFanIn [ OK ] XavierFillerTest/1.TestFillFanIn (1 ms) [ RUN ] XavierFillerTest/1.TestFillFanOut [ OK ] XavierFillerTest/1.TestFillFanOut (0 ms) [ RUN ] XavierFillerTest/1.TestFillAverage [ OK ] XavierFillerTest/1.TestFillAverage (1 ms) [----------] 3 tests from XavierFillerTest/1 (2 ms total) [----------] 10 tests from DetectNetTransformationLayerTest/0, where TypeParam = caffe::CPUDevice<float> [ RUN ] DetectNetTransformationLayerTest/0.TestRotation [ OK ] DetectNetTransformationLayerTest/0.TestRotation (58 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestScaleUp [ OK ] DetectNetTransformationLayerTest/0.TestScaleUp (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestScaleDown [ OK ] DetectNetTransformationLayerTest/0.TestScaleDown (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestHueRotation [ OK ] DetectNetTransformationLayerTest/0.TestHueRotation (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestNoAugmentation [ OK ] DetectNetTransformationLayerTest/0.TestNoAugmentation (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestDesaturation [ OK ] DetectNetTransformationLayerTest/0.TestDesaturation (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestAllAugmentation [ OK ] DetectNetTransformationLayerTest/0.TestAllAugmentation (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestFlip [ OK ] DetectNetTransformationLayerTest/0.TestFlip (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestSetup [ OK ] DetectNetTransformationLayerTest/0.TestSetup (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestCrop [ OK ] DetectNetTransformationLayerTest/0.TestCrop (0 ms) [----------] 10 tests from DetectNetTransformationLayerTest/0 (58 ms total) [----------] 6 tests from FlattenLayerTest/3, where TypeParam = caffe::GPUDevice<double> [ RUN ] FlattenLayerTest/3.TestSetupWithEndAxis [ OK ] FlattenLayerTest/3.TestSetupWithEndAxis (5 ms) [ RUN ] FlattenLayerTest/3.TestSetupWithAxis [ OK ] FlattenLayerTest/3.TestSetupWithAxis (1 ms) [ RUN ] FlattenLayerTest/3.TestSetup [ OK ] FlattenLayerTest/3.TestSetup (0 ms) [ RUN ] FlattenLayerTest/3.TestSetupWithStartAndEndAxis [ OK ] FlattenLayerTest/3.TestSetupWithStartAndEndAxis (1 ms) [ RUN ] FlattenLayerTest/3.TestForward [ OK ] FlattenLayerTest/3.TestForward (0 ms) [ RUN ] FlattenLayerTest/3.TestGradient [ OK ] FlattenLayerTest/3.TestGradient (1 ms) [----------] 6 tests from FlattenLayerTest/3 (8 ms total) [----------] 3 tests from VarSzTransformsTest/0, where TypeParam = float [ RUN ] VarSzTransformsTest/0.TestVarSzImgRandomResize [ OK ] VarSzTransformsTest/0.TestVarSzImgRandomResize (0 ms) [ RUN ] VarSzTransformsTest/0.TestVarSzImgRandomCrop [ OK ] VarSzTransformsTest/0.TestVarSzImgRandomCrop (0 ms) [ RUN ] VarSzTransformsTest/0.TestVarSzImgCenterCrop [ OK ] VarSzTransformsTest/0.TestVarSzImgCenterCrop (0 ms) [----------] 3 tests from VarSzTransformsTest/0 (0 ms total) [----------] 3 tests from XavierStaticFillerTest/1, where TypeParam = double [ RUN ] XavierStaticFillerTest/1.TestFillAverage [ OK ] XavierStaticFillerTest/1.TestFillAverage (1 ms) [ RUN ] XavierStaticFillerTest/1.TestFillFanOut [ OK ] XavierStaticFillerTest/1.TestFillFanOut (1 ms) [ RUN ] XavierStaticFillerTest/1.TestFillFanIn [ OK ] XavierStaticFillerTest/1.TestFillFanIn (1 ms) [----------] 3 tests from XavierStaticFillerTest/1 (3 ms total) [----------] 1 test from LayerFactoryTest/3, where TypeParam = caffe::GPUDevice<double> [ RUN ] LayerFactoryTest/3.TestCreateLayer src/caffe/test/test_layer_factory.cpp:45: Failure Value of: layer->type() Actual: "Accuracy" Expected: iter->first Which is: "SegmentationAccuracy" [ FAILED ] LayerFactoryTest/3.TestCreateLayer, where TypeParam = caffe::GPUDevice<double> (16 ms) [----------] 1 test from LayerFactoryTest/3 (16 ms total) [----------] 12 tests from ReshapeLayerTest/0, where TypeParam = caffe::CPUDevice<float> [ RUN ] ReshapeLayerTest/0.TestFlattenOutputSizes [ OK ] ReshapeLayerTest/0.TestFlattenOutputSizes (0 ms) [ RUN ] ReshapeLayerTest/0.TestForwardAfterReshape [ OK ] ReshapeLayerTest/0.TestForwardAfterReshape (0 ms) [ RUN ] ReshapeLayerTest/0.TestFlattenValues [ OK ] ReshapeLayerTest/0.TestFlattenValues (0 ms) [ RUN ] ReshapeLayerTest/0.TestForward [ OK ] ReshapeLayerTest/0.TestForward (0 ms) [ RUN ] ReshapeLayerTest/0.TestCopyDimensions [ OK ] ReshapeLayerTest/0.TestCopyDimensions (0 ms) [ RUN ] ReshapeLayerTest/0.TestInsertSingletonAxesStart [ OK ] ReshapeLayerTest/0.TestInsertSingletonAxesStart (0 ms) [ RUN ] ReshapeLayerTest/0.TestInferenceOfUnspecifiedWithStartAxis [ OK ] ReshapeLayerTest/0.TestInferenceOfUnspecifiedWithStartAxis (0 ms) [ RUN ] ReshapeLayerTest/0.TestInferenceOfUnspecified [ OK ] ReshapeLayerTest/0.TestInferenceOfUnspecified (0 ms) [ RUN ] ReshapeLayerTest/0.TestFlattenMiddle [ OK ] ReshapeLayerTest/0.TestFlattenMiddle (0 ms) [ RUN ] ReshapeLayerTest/0.TestInsertSingletonAxesEnd [ OK ] ReshapeLayerTest/0.TestInsertSingletonAxesEnd (0 ms) [ RUN ] ReshapeLayerTest/0.TestInsertSingletonAxesMiddle [ OK ] ReshapeLayerTest/0.TestInsertSingletonAxesMiddle (0 ms) [ RUN ] ReshapeLayerTest/0.TestGradient [ OK ] ReshapeLayerTest/0.TestGradient (0 ms) [----------] 12 tests from ReshapeLayerTest/0 (0 ms total) [----------] 6 tests from FlattenLayerTest/1, where TypeParam = caffe::CPUDevice<double> [ RUN ] FlattenLayerTest/1.TestSetupWithEndAxis [ OK ] FlattenLayerTest/1.TestSetupWithEndAxis (0 ms) [ RUN ] FlattenLayerTest/1.TestSetupWithStartAndEndAxis [ OK ] FlattenLayerTest/1.TestSetupWithStartAndEndAxis (0 ms) [ RUN ] FlattenLayerTest/1.TestSetup [ OK ] FlattenLayerTest/1.TestSetup (0 ms) [ RUN ] FlattenLayerTest/1.TestSetupWithAxis [ OK ] FlattenLayerTest/1.TestSetupWithAxis (0 ms) [ RUN ] FlattenLayerTest/1.TestForward [ OK ] FlattenLayerTest/1.TestForward (0 ms) [ RUN ] FlattenLayerTest/1.TestGradient [ OK ] FlattenLayerTest/1.TestGradient (0 ms) [----------] 6 tests from FlattenLayerTest/1 (0 ms total) [----------] 12 tests from NesterovSolverTest/0, where TypeParam = caffe::GPUDevice<float> [ RUN ] NesterovSolverTest/0.TestNesterovLeastSquaresUpdateLROneHundredth F0212 16:33:58.809607 3309 math_functions.cu:24] Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED *** Check failure stack trace: *** @ 0x7fe510f840cd google::LogMessage::Fail() @ 0x7fe510f85f33 google::LogMessage::SendToLog() @ 0x7fe510f83c28 google::LogMessage::Flush() @ 0x7fe510f86999 google::LogMessageFatal::~LogMessageFatal() @ 0x7fe50df6208e caffe::caffe_gpu_gemm<>() @ 0x7fe50e05af8c caffe::InnerProductLayer<>::Forward_gpu() @ 0x564439d16156 caffe::Layer<>::Forward() @ 0x7fe50db7f3d3 caffe::Net::ForwardFromTo() @ 0x7fe50db7f567 caffe::Net::Forward() @ 0x564439ee77cf caffe::GradientBasedSolverTest<>::ComputeLeastSquaresUpdate() @ 0x564439eea25b caffe::GradientBasedSolverTest<>::TestLeastSquaresUpdate() @ 0x56443a184c7a testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x56443a17defa testing::Test::Run() @ 0x56443a17dfdc testing::TestInfo::Run() @ 0x56443a17e115 testing::TestCase::Run() @ 0x56443a17e5f0 testing::internal::UnitTestImpl::RunAllTests() @ 0x56443a17e737 testing::UnitTest::Run() @ 0x564439cfa394 main @ 0x7fe50c9a0b97 __libc_start_main @ 0x564439d05c0a _start Makefile:560: recipe for target 'runtest' failed make: *** [runtest] 중지됨 (메모리 덤프됨)
There was no issue with Caffe, and their Host PC is Ubuntu18.04 and graphic card is 2080 TI.
Thanks and Best Regards,
SI.
Dear Sung-IL,
The customer can use Caffe-Jacinto eventhough the run test fails. The failure is for features that we do not use.
Also, we would like to bring to your attention a newer repository:
https://github.com/TexasInstruments/jacinto-ai-devkit
In this place you can find pytorch-jacinto-ai-devkit which is a repository that can be used for Deep Learning training for Jacinto7 / TDA4x.
Best regards,
Manu.