.build_release/tools/caffe I0212 16:33:57.105792 3278 caffe.cpp:902] This is NVCaffe 0.17.0 started at Wed Feb 12 16:33:57 2020 I0212 16:33:57.257547 3278 caffe.cpp:904] CuDNN version: 7605 I0212 16:33:57.257551 3278 caffe.cpp:905] CuBLAS version: 9010 I0212 16:33:57.257553 3278 caffe.cpp:906] CUDA version: 10020 I0212 16:33:57.257555 3278 caffe.cpp:907] CUDA driver version: 10020 I0212 16:33:57.257556 3278 caffe.cpp:908] Arguments: [0]: .build_release/tools/caffe caffe: command line brew usage: caffe commands: train train or finetune a model test score a model device_query show GPU diagnostic information time benchmark model execution time Flags from tools/caffe.cpp: -ap_version (Average Precision type for object detection) type: string default: "11point" -display_sparsity (Display the amount of sparsity) type: bool default: false -gpu (Optional; run in GPU mode on given device IDs separated by ', '.Use '-gpu all' to run on all available GPUs. The effective training batch size is multiplied by the number of devices.) type: string default: "" -iterations (The number of iterations to run.) type: int32 default: 50 -level (Optional; network level.) type: int32 default: 0 -model (The model definition protocol buffer text file.) type: string default: "" -optimize_net (Optimize the Net (Merge BN to Conv) before test) type: bool default: false -output_model (Prefix for output prototxt and caffemodel) type: string default: "" -phase (Optional; network phase (TRAIN or TEST). Only used for 'time'.) type: string default: "" -show_per_class_result (Show per class result for object detection) type: bool default: true -sighup_effect (Optional; action to take when a SIGHUP signal is received: snapshot, stop or none.) type: string default: "snapshot" -sigint_effect (Optional; action to take when a SIGINT signal is received: snapshot, stop or none.) type: string default: "stop" -snapshot (Optional; the snapshot solver state to resume training.) type: string default: "" -solver (The solver definition protocol buffer text file.) type: string default: "" -stage (Optional; network stages (not to be confused with phase), separated by ','.) type: string default: "" -weights (Optional; the pretrained weights to initialize finetuning, separated by ', '. Cannot be set simultaneously with snapshot.) type: string default: "" .build_release/test/test_all.testbin 0 --gtest_shuffle Cuda number of devices: 1 Setting to use device 0 Current device id: 0 Current device name: GeForce RTX 2080 Ti Note: Randomizing tests' orders with a seed of 82163 . [==========] Running 2101 tests from 283 test cases. [----------] Global test environment set-up. [----------] 8 tests from SliceLayerTest/1, where TypeParam = caffe::CPUDevice [ RUN ] SliceLayerTest/1.TestSetupNum [ OK ] SliceLayerTest/1.TestSetupNum (0 ms) [ RUN ] SliceLayerTest/1.TestTrivialSlice [ OK ] SliceLayerTest/1.TestTrivialSlice (0 ms) [ RUN ] SliceLayerTest/1.TestSliceAcrossChannels [ OK ] SliceLayerTest/1.TestSliceAcrossChannels (0 ms) [ RUN ] SliceLayerTest/1.TestGradientAcrossChannels [ OK ] SliceLayerTest/1.TestGradientAcrossChannels (18 ms) [ RUN ] SliceLayerTest/1.TestGradientAcrossNum [ OK ] SliceLayerTest/1.TestGradientAcrossNum (17 ms) [ RUN ] SliceLayerTest/1.TestSetupChannels [ OK ] SliceLayerTest/1.TestSetupChannels (0 ms) [ RUN ] SliceLayerTest/1.TestSliceAcrossNum [ OK ] SliceLayerTest/1.TestSliceAcrossNum (0 ms) [ RUN ] SliceLayerTest/1.TestGradientTrivial [ OK ] SliceLayerTest/1.TestGradientTrivial (0 ms) [----------] 8 tests from SliceLayerTest/1 (35 ms total) [----------] 5 tests from DeconvolutionLayerTest/1, where TypeParam = caffe::CPUDevice [ RUN ] DeconvolutionLayerTest/1.TestSetup [ OK ] DeconvolutionLayerTest/1.TestSetup (0 ms) [ RUN ] DeconvolutionLayerTest/1.TestSimpleDeconvolution [ OK ] DeconvolutionLayerTest/1.TestSimpleDeconvolution (7 ms) [ RUN ] DeconvolutionLayerTest/1.TestGradient3D [ OK ] DeconvolutionLayerTest/1.TestGradient3D (84 ms) [ RUN ] DeconvolutionLayerTest/1.TestNDAgainst2D [ OK ] DeconvolutionLayerTest/1.TestNDAgainst2D (499 ms) [ RUN ] DeconvolutionLayerTest/1.TestGradient [ OK ] DeconvolutionLayerTest/1.TestGradient (355 ms) [----------] 5 tests from DeconvolutionLayerTest/1 (945 ms total) [----------] 1 test from SolverFactoryTest/0, where TypeParam = caffe::CPUDevice [ RUN ] SolverFactoryTest/0.TestCreateSolver [ OK ] SolverFactoryTest/0.TestCreateSolver (4 ms) [----------] 1 test from SolverFactoryTest/0 (4 ms total) [----------] 3 tests from XavierFillerTest/1, where TypeParam = double [ RUN ] XavierFillerTest/1.TestFillFanIn [ OK ] XavierFillerTest/1.TestFillFanIn (1 ms) [ RUN ] XavierFillerTest/1.TestFillFanOut [ OK ] XavierFillerTest/1.TestFillFanOut (0 ms) [ RUN ] XavierFillerTest/1.TestFillAverage [ OK ] XavierFillerTest/1.TestFillAverage (1 ms) [----------] 3 tests from XavierFillerTest/1 (2 ms total) [----------] 10 tests from DetectNetTransformationLayerTest/0, where TypeParam = caffe::CPUDevice [ RUN ] DetectNetTransformationLayerTest/0.TestRotation [ OK ] DetectNetTransformationLayerTest/0.TestRotation (58 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestScaleUp [ OK ] DetectNetTransformationLayerTest/0.TestScaleUp (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestScaleDown [ OK ] DetectNetTransformationLayerTest/0.TestScaleDown (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestHueRotation [ OK ] DetectNetTransformationLayerTest/0.TestHueRotation (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestNoAugmentation [ OK ] DetectNetTransformationLayerTest/0.TestNoAugmentation (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestDesaturation [ OK ] DetectNetTransformationLayerTest/0.TestDesaturation (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestAllAugmentation [ OK ] DetectNetTransformationLayerTest/0.TestAllAugmentation (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestFlip [ OK ] DetectNetTransformationLayerTest/0.TestFlip (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestSetup [ OK ] DetectNetTransformationLayerTest/0.TestSetup (0 ms) [ RUN ] DetectNetTransformationLayerTest/0.TestCrop [ OK ] DetectNetTransformationLayerTest/0.TestCrop (0 ms) [----------] 10 tests from DetectNetTransformationLayerTest/0 (58 ms total) [----------] 6 tests from FlattenLayerTest/3, where TypeParam = caffe::GPUDevice [ RUN ] FlattenLayerTest/3.TestSetupWithEndAxis [ OK ] FlattenLayerTest/3.TestSetupWithEndAxis (5 ms) [ RUN ] FlattenLayerTest/3.TestSetupWithAxis [ OK ] FlattenLayerTest/3.TestSetupWithAxis (1 ms) [ RUN ] FlattenLayerTest/3.TestSetup [ OK ] FlattenLayerTest/3.TestSetup (0 ms) [ RUN ] FlattenLayerTest/3.TestSetupWithStartAndEndAxis [ OK ] FlattenLayerTest/3.TestSetupWithStartAndEndAxis (1 ms) [ RUN ] FlattenLayerTest/3.TestForward [ OK ] FlattenLayerTest/3.TestForward (0 ms) [ RUN ] FlattenLayerTest/3.TestGradient [ OK ] FlattenLayerTest/3.TestGradient (1 ms) [----------] 6 tests from FlattenLayerTest/3 (8 ms total) [----------] 3 tests from VarSzTransformsTest/0, where TypeParam = float [ RUN ] VarSzTransformsTest/0.TestVarSzImgRandomResize [ OK ] VarSzTransformsTest/0.TestVarSzImgRandomResize (0 ms) [ RUN ] VarSzTransformsTest/0.TestVarSzImgRandomCrop [ OK ] VarSzTransformsTest/0.TestVarSzImgRandomCrop (0 ms) [ RUN ] VarSzTransformsTest/0.TestVarSzImgCenterCrop [ OK ] VarSzTransformsTest/0.TestVarSzImgCenterCrop (0 ms) [----------] 3 tests from VarSzTransformsTest/0 (0 ms total) [----------] 3 tests from XavierStaticFillerTest/1, where TypeParam = double [ RUN ] XavierStaticFillerTest/1.TestFillAverage [ OK ] XavierStaticFillerTest/1.TestFillAverage (1 ms) [ RUN ] XavierStaticFillerTest/1.TestFillFanOut [ OK ] XavierStaticFillerTest/1.TestFillFanOut (1 ms) [ RUN ] XavierStaticFillerTest/1.TestFillFanIn [ OK ] XavierStaticFillerTest/1.TestFillFanIn (1 ms) [----------] 3 tests from XavierStaticFillerTest/1 (3 ms total) [----------] 1 test from LayerFactoryTest/3, where TypeParam = caffe::GPUDevice [ RUN ] LayerFactoryTest/3.TestCreateLayer src/caffe/test/test_layer_factory.cpp:45: Failure Value of: layer->type() Actual: "Accuracy" Expected: iter->first Which is: "SegmentationAccuracy" [ FAILED ] LayerFactoryTest/3.TestCreateLayer, where TypeParam = caffe::GPUDevice (16 ms) [----------] 1 test from LayerFactoryTest/3 (16 ms total) [----------] 12 tests from ReshapeLayerTest/0, where TypeParam = caffe::CPUDevice [ RUN ] ReshapeLayerTest/0.TestFlattenOutputSizes [ OK ] ReshapeLayerTest/0.TestFlattenOutputSizes (0 ms) [ RUN ] ReshapeLayerTest/0.TestForwardAfterReshape [ OK ] ReshapeLayerTest/0.TestForwardAfterReshape (0 ms) [ RUN ] ReshapeLayerTest/0.TestFlattenValues [ OK ] ReshapeLayerTest/0.TestFlattenValues (0 ms) [ RUN ] ReshapeLayerTest/0.TestForward [ OK ] ReshapeLayerTest/0.TestForward (0 ms) [ RUN ] ReshapeLayerTest/0.TestCopyDimensions [ OK ] ReshapeLayerTest/0.TestCopyDimensions (0 ms) [ RUN ] ReshapeLayerTest/0.TestInsertSingletonAxesStart [ OK ] ReshapeLayerTest/0.TestInsertSingletonAxesStart (0 ms) [ RUN ] ReshapeLayerTest/0.TestInferenceOfUnspecifiedWithStartAxis [ OK ] ReshapeLayerTest/0.TestInferenceOfUnspecifiedWithStartAxis (0 ms) [ RUN ] ReshapeLayerTest/0.TestInferenceOfUnspecified [ OK ] ReshapeLayerTest/0.TestInferenceOfUnspecified (0 ms) [ RUN ] ReshapeLayerTest/0.TestFlattenMiddle [ OK ] ReshapeLayerTest/0.TestFlattenMiddle (0 ms) [ RUN ] ReshapeLayerTest/0.TestInsertSingletonAxesEnd [ OK ] ReshapeLayerTest/0.TestInsertSingletonAxesEnd (0 ms) [ RUN ] ReshapeLayerTest/0.TestInsertSingletonAxesMiddle [ OK ] ReshapeLayerTest/0.TestInsertSingletonAxesMiddle (0 ms) [ RUN ] ReshapeLayerTest/0.TestGradient [ OK ] ReshapeLayerTest/0.TestGradient (0 ms) [----------] 12 tests from ReshapeLayerTest/0 (0 ms total) [----------] 6 tests from FlattenLayerTest/1, where TypeParam = caffe::CPUDevice [ RUN ] FlattenLayerTest/1.TestSetupWithEndAxis [ OK ] FlattenLayerTest/1.TestSetupWithEndAxis (0 ms) [ RUN ] FlattenLayerTest/1.TestSetupWithStartAndEndAxis [ OK ] FlattenLayerTest/1.TestSetupWithStartAndEndAxis (0 ms) [ RUN ] FlattenLayerTest/1.TestSetup [ OK ] FlattenLayerTest/1.TestSetup (0 ms) [ RUN ] FlattenLayerTest/1.TestSetupWithAxis [ OK ] FlattenLayerTest/1.TestSetupWithAxis (0 ms) [ RUN ] FlattenLayerTest/1.TestForward [ OK ] FlattenLayerTest/1.TestForward (0 ms) [ RUN ] FlattenLayerTest/1.TestGradient [ OK ] FlattenLayerTest/1.TestGradient (0 ms) [----------] 6 tests from FlattenLayerTest/1 (0 ms total) [----------] 12 tests from NesterovSolverTest/0, where TypeParam = caffe::GPUDevice [ RUN ] NesterovSolverTest/0.TestNesterovLeastSquaresUpdateLROneHundredth F0212 16:33:58.809607 3309 math_functions.cu:24] Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED *** Check failure stack trace: *** @ 0x7fe510f840cd google::LogMessage::Fail() @ 0x7fe510f85f33 google::LogMessage::SendToLog() @ 0x7fe510f83c28 google::LogMessage::Flush() @ 0x7fe510f86999 google::LogMessageFatal::~LogMessageFatal() @ 0x7fe50df6208e caffe::caffe_gpu_gemm<>() @ 0x7fe50e05af8c caffe::InnerProductLayer<>::Forward_gpu() @ 0x564439d16156 caffe::Layer<>::Forward() @ 0x7fe50db7f3d3 caffe::Net::ForwardFromTo() @ 0x7fe50db7f567 caffe::Net::Forward() @ 0x564439ee77cf caffe::GradientBasedSolverTest<>::ComputeLeastSquaresUpdate() @ 0x564439eea25b caffe::GradientBasedSolverTest<>::TestLeastSquaresUpdate() @ 0x56443a184c7a testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x56443a17defa testing::Test::Run() @ 0x56443a17dfdc testing::TestInfo::Run() @ 0x56443a17e115 testing::TestCase::Run() @ 0x56443a17e5f0 testing::internal::UnitTestImpl::RunAllTests() @ 0x56443a17e737 testing::UnitTest::Run() @ 0x564439cfa394 main @ 0x7fe50c9a0b97 __libc_start_main @ 0x564439d05c0a _start Makefile:560: recipe for target 'runtest' failed make: *** [runtest] 중지됨 (메모리 덤프됨)