TDA4VM: RTOS Decoder graph verify failed after run multiple times

Ming Zhong10

Intellectual 2235 points

Part Number: TDA4VM

Dear experts,

We ran into a issue that rtos decoder graph verify failed.

Reproduce steps are:

(PSDKRA 6.2 version on TDA4VMXEVM)

1. run "VX_TEST_DATA_PATH=/opt/vision_apps/test_data ./vx_app_conformance.out --filter=tivxHwaVideoDecoder.MultiStreamProcessing" for at least 3~5 times.

2. you will see "[ FAILED ] 1 test(s)"

Here is my log: from TIOVX

202004142237-multi-decode-error-tiovx.log

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
   118.037524 s:  VX_ZONE_INFO:[ownGraphScheduleGraph:763] Scheduling Graph (graph=13, pipe=0)
   118.037578 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=8, pipe=0)
   118.037607 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=7, pipe=0)
   118.072115 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:676] Graph Completed (graph=13, pipe=0)
   118.072144 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:701] All Graphs Completed
   118.717528 s:  VX_ZONE_INFO:[ownGraphScheduleGraph:763] Scheduling Graph (graph=13, pipe=0)
   118.717550 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=8, pipe=0)
   118.717595 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=7, pipe=0)
   118.752087 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:676] Graph Completed (graph=13, pipe=0)
   118.752116 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:701] All Graphs Completed
   119.485541 s:  VX_ZONE_INFO:[ownGraphScheduleGraph:763] Scheduling Graph (graph=13, pipe=0)
   119.485561 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=8, pipe=0)
   119.485608 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=7, pipe=0)
   119.520099 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:676] Graph Completed (graph=13, pipe=0)
   119.520127 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:701] All Graphs Completed
   120.309507 s:  VX_ZONE_INFO:[ownGraphScheduleGraph:763] Scheduling Graph (graph=13, pipe=0)
   120.309529 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=8, pipe=0)
   120.309574 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=7, pipe=0)
   120.343900 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:676] Graph Completed (graph=13, pipe=0)
   120.343928 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:701] All Graphs Completed
   121.125495 s:  VX_ZONE_INFO:[ownGraphScheduleGraph:763] Scheduling Graph (graph=13, pipe=0)
   121.125516 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=8, pipe=0)
   121.125559 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=7, pipe=0)
   121.160043 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:676] Graph Completed (graph=13, pipe=0)
   121.160072 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:701] All Graphs Completed
   122.094723 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.095337 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.095371 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
[     DONE ] tivxHwaVideoDecoder.MultiStreamProcessing
[ -------- ] 1 tests from test case tivxHwaVideoDecoder
[ ======== ]
[ ALL DONE ] 1 test(s) from 1 test case(s) ran
[ PASSED   ] 1 test(s)
[ FAILED   ] 0 test(s)
[ DISABLED ] 0 test(s)
=================================
OpenVX Conformance report summary
=================================
To be conformant to the OpenVX baseline, 1 required test(s) must pass. 1 tests passed, 0 tests failed. PASSED.
To be conformant to the User Data Object extension, 0 required test(s) must pass. 0 tests passed, 0 tests failed. PASSED.
Note: The 0 disabled tests are optional and are not considered for conformance.
#REPORT: YYYYMMDDHHMMSS FILTERED 21932 0 1 1 1 0 (version 1.1-20170301)
   122.105486 s:  VX_ZONE_INIT:[tivxHostDeInit:49] De-Initialization Done for HOST !!!
   122.106599 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.107671 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.108738 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.109804 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.109849 s:  VX_ZONE_INIT:[tivxDeInit:100] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ION ... !!!
MEM: Alloc's: 11 alloc's of 8986376 bytes 
MEM: Free's : 11 free's  of 8986376 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ION ... Done !!!
APP: Deinit ... Done !!!
root@j7-evm:/opt/vision_apps# VX_TEST_DATA_PATH=/opt/vision_apps/test_data ./vx_app_conformance.out --filter=tivxHwaVideoDecoder.MultiStreamProcessing
APP: Init ... !!!
APP_LOG: Mapping 0xac000000 ...
APP_LOG: Mapped 0xac000000 -> 0xffff8b0f0000 of size 262144 bytes 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

   118.037524 s:  VX_ZONE_INFO:[ownGraphScheduleGraph:763] Scheduling Graph (graph=13, pipe=0)
   118.037578 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=8, pipe=0)
   118.037607 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=7, pipe=0)
   118.072115 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:676] Graph Completed (graph=13, pipe=0)
   118.072144 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:701] All Graphs Completed
   118.717528 s:  VX_ZONE_INFO:[ownGraphScheduleGraph:763] Scheduling Graph (graph=13, pipe=0)
   118.717550 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=8, pipe=0)
   118.717595 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=7, pipe=0)
   118.752087 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:676] Graph Completed (graph=13, pipe=0)
   118.752116 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:701] All Graphs Completed
   119.485541 s:  VX_ZONE_INFO:[ownGraphScheduleGraph:763] Scheduling Graph (graph=13, pipe=0)
   119.485561 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=8, pipe=0)
   119.485608 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=7, pipe=0)
   119.520099 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:676] Graph Completed (graph=13, pipe=0)
   119.520127 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:701] All Graphs Completed
   120.309507 s:  VX_ZONE_INFO:[ownGraphScheduleGraph:763] Scheduling Graph (graph=13, pipe=0)
   120.309529 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=8, pipe=0)
   120.309574 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=7, pipe=0)
   120.343900 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:676] Graph Completed (graph=13, pipe=0)
   120.343928 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:701] All Graphs Completed
   121.125495 s:  VX_ZONE_INFO:[ownGraphScheduleGraph:763] Scheduling Graph (graph=13, pipe=0)
   121.125516 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=8, pipe=0)
   121.125559 s:  VX_ZONE_INFO:[ownNodeKernelSchedule:570] Scheduling Node (node=7, pipe=0)
   121.160043 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:676] Graph Completed (graph=13, pipe=0)
   121.160072 s:  VX_ZONE_INFO:[ownCheckGraphCompleted:701] All Graphs Completed
   122.094723 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.095337 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.095371 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
[     DONE ] tivxHwaVideoDecoder.MultiStreamProcessing
[ -------- ] 1 tests from test case tivxHwaVideoDecoder

[ ======== ]
[ ALL DONE ] 1 test(s) from 1 test case(s) ran
[ PASSED   ] 1 test(s)
[ FAILED   ] 0 test(s)
[ DISABLED ] 0 test(s)

=================================
OpenVX Conformance report summary
=================================

To be conformant to the OpenVX baseline, 1 required test(s) must pass. 1 tests passed, 0 tests failed. PASSED.
To be conformant to the User Data Object extension, 0 required test(s) must pass. 0 tests passed, 0 tests failed. PASSED.
Note: The 0 disabled tests are optional and are not considered for conformance.

#REPORT: YYYYMMDDHHMMSS FILTERED 21932 0 1 1 1 0 (version 1.1-20170301)
   122.105486 s:  VX_ZONE_INIT:[tivxHostDeInit:49] De-Initialization Done for HOST !!!
   122.106599 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.107671 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.108738 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.109804 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
   122.109849 s:  VX_ZONE_INIT:[tivxDeInit:100] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ION ... !!!
MEM: Alloc's: 11 alloc's of 8986376 bytes 
MEM: Free's : 11 free's  of 8986376 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ION ... Done !!!
APP: Deinit ... Done !!!
root@j7-evm:/opt/vision_apps# VX_TEST_DATA_PATH=/opt/vision_apps/test_data ./vx_app_conformance.out --filter=tivxHwaVideoDecoder.MultiStreamProcessing
APP: Init ... !!!
APP_LOG: Mapping 0xac000000 ...
APP_LOG: Mapped 0xac000000 -> 0xffff8b0f0000 of size 262144 bytes 
MEM: Init ION ... !!!
MEM: Initialized ION (fd=4) !!!
MEM: Init ION ... Done !!!
IPC: Init ... !!!
APP_LOG: Mapping 0xac040000 ...
APP_LOG: Mapped 0xac040000 -> 0xffff89150000 of size 33161216 bytes 
APP_LOG: Mapping 0x30e00000 ...
APP_LOG: Mapped 0x30e00000 -> 0xffff8b5e0000 of size 3072 bytes 
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
APP: Init ... Done !!!
     0.000913 s:  VX_ZONE_INIT:Enabled
     0.000921 s:  VX_ZONE_ERROR:Enabled
     0.000925 s:  VX_ZONE_WARNING:Enabled
     0.000929 s:  VX_ZONE_INFO:Disabled
     0.001806 s:  VX_ZONE_INIT:[tivxInit:64] Initialization Done !!!
     0.002183 s:  VX_ZONE_INIT:[tivxHostInit:44] Initialization Done for HOST !!!
VxTests version: 1.1-20170301
VCS version: unknown
Build config: Release

[ ======== ] Total 21932 tests from 144 test cases
Use test filter: tivxHwaVideoDecoder.MultiStreamProcessing

Use global OpenVX context: FALSE


[ -------- ] tests from tivxHwaVideoDecoder
[ RUN 0001 ] tivxHwaVideoDecoder.MultiStreamProcessing ...
     0.015868 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:203] Attempting to set parameter[0] on com.ti.hwa.video_decoder (enum:-2097101) to 0xb075c0
     0.015891 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:236] Query returned type 00000816 for ref 0xb075c0
     0.015899 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:302] Assigned Node[0] 0xadafc0 type:00000816 ref=0xb075c0
     0.015904 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:203] Attempting to set parameter[1] on com.ti.hwa.video_decoder (enum:-2097101) to 0xb07a00
     0.015910 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:236] Query returned type 00000816 for ref 0xb07a00
     0.016055 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:302] Assigned Node[1] 0xadafc0 type:00000816 ref=0xb07a00
     0.016066 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:203] Attempting to set parameter[2] on com.ti.hwa.video_decoder (enum:-2097101) to 0xb396c0
     0.016072 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:236] Query returned type 0000080f for ref 0xb396c0
     0.016077 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:302] Assigned Node[2] 0xadafc0 type:0000080f ref=0xb396c0
     0.016112 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:203] Attempting to set parameter[0] on com.ti.hwa.video_decoder (enum:-2097101) to 0xb077e0
     0.016118 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:236] Query returned type 00000816 for ref 0xb077e0
     0.016123 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:302] Assigned Node[0] 0xadb290 type:00000816 ref=0xb077e0
     0.016128 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:203] Attempting to set parameter[1] on com.ti.hwa.video_decoder (enum:-2097101) to 0xb07c20
     0.016133 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:236] Query returned type 00000816 for ref 0xb07c20
     0.016139 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:302] Assigned Node[1] 0xadb290 type:00000816 ref=0xb07c20
     0.016144 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:203] Attempting to set parameter[2] on com.ti.hwa.video_decoder (enum:-2097101) to 0xb39978
     0.016149 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:236] Query returned type 0000080f for ref 0xb39978
     0.016154 s:  VX_ZONE_PARAMETER:[vxSetParameterByIndex:302] Assigned Node[2] 0xadb290 type:0000080f ref=0xb39978
     0.086957 s:  VX_ZONE_INFO:[ownGraphNodeKernelInit:578] kernel init for node 0, kernel com.ti.hwa.video_decoder ...
     0.086975 s:  VX_ZONE_INFO:[ownGraphNodeKernelInit:589] kernel init for node 0, kernel com.ti.hwa.video_decoder ... done !!!
     0.146141 s:  VX_ZONE_ERROR:[ownContextSendCmd:553] Command ack message returned failure cmd_status: -1
     0.146148 s:  VX_ZONE_ERROR:[ownNodeKernelInit:486] Target kernel, TIVX_CMD_NODE_CREATE failed
     0.146153 s:  VX_ZONE_INFO:[ownGraphNodeKernelInit:578] kernel init for node 1, kernel com.ti.hwa.video_decoder ...
     0.146158 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 1, kernel com.ti.hwa.video_decoder ... failed !!!
     0.146164 s:  VX_ZONE_ERROR:[vxVerifyGraph:2010] Node kernel init failed
     0.146169 s:  VX_ZONE_ERROR:[vxVerifyGraph:2064] Graph verify failed

FAILED at /work/j7/6.2/psdk_rtos_auto_j7_06_02_00_21/tiovx/kernels_j7/hwa/test/test_video_decoder.c:549
        Expected: VX_SUCCESS == vxVerifyGraph(graph)
        Actual: VX_SUCCESS != VX_FAILURE

     0.183689 s:  VX_ZONE_WARNING:[vxReleaseContext:743] Stale reference 0xb075c0 of type 00000816 at external count 1, internal count 1
     0.183697 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased reference name = user_data_object_84
     0.183702 s:  VX_ZONE_WARNING:[vxReleaseContext:743] Stale reference 0xb077e0 of type 00000816 at external count 1, internal count 1
     0.183707 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased reference name = user_data_object_85
     0.183712 s:  VX_ZONE_WARNING:[vxReleaseContext:743] Stale reference 0xb07a00 of type 00000816 at external count 1, internal count 1
     0.183716 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased reference name = user_data_object_86
     0.183721 s:  VX_ZONE_WARNING:[vxReleaseContext:743] Stale reference 0xb07c20 of type 00000816 at external count 1, internal count 1
     0.183725 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased reference name = user_data_object_87
     0.183730 s:  VX_ZONE_WARNING:[vxReleaseContext:743] Stale reference 0xb396c0 of type 0000080f at external count 1, internal count 1
     0.183735 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased reference name = image_88
     0.183740 s:  VX_ZONE_WARNING:[vxReleaseContext:743] Stale reference 0xb39978 of type 0000080f at external count 1, internal count 1
     0.183744 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased reference name = image_89
     0.183749 s:  VX_ZONE_WARNING:[vxReleaseContext:743] Stale reference 0xa9af20 of type 00000802 at external count 1, internal count 0
     0.183753 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased reference name = graph_90
     0.183995 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
     0.184038 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
     0.184048 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
[ !FAILED! ] tivxHwaVideoDecoder.MultiStreamProcessing
[ -------- ] 1 tests from test case tivxHwaVideoDecoder

[ ======== ]
[ ALL DONE ] 1 test(s) from 1 test case(s) ran
[ PASSED   ] 0 test(s)
[ FAILED   ] 1 test(s), listed below:
[ FAILED   ] tivxHwaVideoDecoder.MultiStreamProcessing
[ DISABLED ] 0 test(s)

=================================
OpenVX Conformance report summary
=================================

To be conformant to the OpenVX baseline, 1 required test(s) must pass. 0 tests passed, 1 tests failed. FAILED.
To be conformant to the User Data Object extension, 0 required test(s) must pass. 0 tests passed, 0 tests failed. PASSED.
Note: The 0 disabled tests are optional and are not considered for conformance.

#REPORT: YYYYMMDDHHMMSS FILTERED 21932 0 1 1 0 1 (version 1.1-20170301)
     0.193843 s:  VX_ZONE_INIT:[tivxHostDeInit:49] De-Initialization Done for HOST !!!
     0.194946 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
     0.196018 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
     0.197085 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
     0.198151 s:  VX_ZONE_INFO:[tivxQueueDelete:182] tivxQueueDelete: if this hangs, please ensure all application threads have been destroyed
     0.198194 s:  VX_ZONE_INIT:[tivxDeInit:100] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ION ... !!!
MEM: Alloc's: 12 alloc's of 8986632 bytes 
MEM: Free's : 12 free's  of 8986632 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ION ... Done !!!
APP: Deinit ... Done !!!
root@j7-evm:/opt/vision_apps#

Another log from my own app, FYI:

202004142219-multi-decode-error.log

Thanks & Best Regards!

over 5 years ago

0 David Huang over 5 years ago

TI__Intellectual 2440 points

Hi ZM,

Can you provide a full log from boot to the end of the test? Also, this issue starts occurring only after a few runs, is that correct?

Thanks,

David

0 Keerthy J over 5 years ago in reply to David Huang

TI__Guru**** 155100 points

Hi David,

I am attaching the logs.
I am able to reproduce the issue. 4th time i ran the command:
VX_TEST_DATA_PATH=/opt/vision_apps/test_data ./vx_app_conformance.out --filter=tivxHwaVidDecoder.MultiStreamProcessing

Regards,
Keerthy

decoder-issue-log.txt

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
root@j7-evm:/opt/vision_apps# VX_TEST_DATA_PATH=/opt/vision_apps/test_data ./vx_app_conformance.out --filter=tivxHwaVideoDecoder.MultiStreamProcessing
APP: Init ... !!!
APP_LOG: Mapping 0xac000000 ...
APP_LOG: Mped 0xac000000 -> 0xffffa4440000 of size 262144 bytes 
MEM: Iit ION ... !!!
MEM: Initialized ION (fd=4) !!!
MEM: Init ION.. Done !!!
IPC: Init ... !!!
APP_LOG: Mapping 0xac040000 ..
                              APP_LOG: Mapped 0xac040000 -> 0xffffa24a0000 of size 33161216 bytes 
APP_LOG: Mapping 0x30e00000 ...
APP_LOG: Mappe0x30e00000 -> 0xffffa4930000 of size 3072 bytes 
IPC: Init ..Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
APP: Init ... Done !!!
     0.015201 s:  VZONE_INIT:Enabled
     0.015214 s:  VX_ZONE_ERROR:Enabled
   0.015218 s:  VX_ZONE_WARNING:Enabled
     0.017453 s:  VX_ZONE_INIT:[tivxInit:64] Initialization Done !!!
     0.0179 s:  VX_ZONE_INIT:[tivxHostInit:44] Initialization Done for HT !!!
VxTests version: 1.1-20170301
VCS version: unknown
Build config: Release
[ ======== ] Total 21932 tests from 144 test cases
Use test filter: tivxHwaVideoDecoder.MtiStreamProcessing
Use global OpenVX context: FALSE
[ -------- ] tests from tivxHwaVideoDecoder
[ RUN 0001 ] tivxaVideoDecoder.MultiStreamProcessing ...
[  133.333456] ion_buffer_destroy: buffer still mapped in the rnel
[     DONE ] tivxHwaVideoDecoder.MultiStreamProcessing
-------- ] 1 tests from test case tivxHwaVideoDecoder
[ ======== ]
[ ALL DE ] 1 test(s) from 1 test case(s) ran
[ PASSED   ] 1 test(s)
[ ILED   ] 0 test(s)
[ DISABLED ] 0 test(s)
===============================
OpenVX Conformance report summary
=================================
To be conformant to the OpenVbaseline, 1 required test(s) must pass. 1 tests passed, 0 testfailed. PASSED.
To be conformant to the User Data Object extension, 0 required test(s) must pass. 0 tests passed, 0 tts failed. PASSED.
Note: The 0 disabled tests are optional anare not considered for conformance.
#REPORT: YYYYMMDDHHMMSS FILTERED 21932 0 1 1 1 0 (version 1.1-20170301)
    48.5891 s:  VX_ZONE_INIT:[tivxHostDeInit:49] De-Initialization Do for HOST !!!
    48.190236 s:  VX_ZONE_INIT:[tivxDeInit:100] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTEERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
I: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ION ... !!!
MEM: Alloc's: 11 alloc's of 8986376 bytes 
MEM: Fe's : 11 free's  of 8986376 bytes 
MEM: Open's : 0 allocs  of bytes 
MEM: Deinit ION ... Done !!!
APP: Deinit ... Done !!!
root@j7-evm:/opt/vision_apps# 
root@j7-evm:/opt/vision_apps# 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

root@j7-evm:/opt/vision_apps# VX_TEST_DATA_PATH=/opt/vision_apps/test_data ./vx_app_conformance.out --filter=tivxHwaVideoDecoder.MultiStreamProcessing
APP: Init ... !!!
APP_LOG: Mapping 0xac000000 ...
APP_LOG: Mped 0xac000000 -> 0xffffa4440000 of size 262144 bytes 
MEM: Iit ION ... !!!
MEM: Initialized ION (fd=4) !!!
MEM: Init ION.. Done !!!
IPC: Init ... !!!
APP_LOG: Mapping 0xac040000 ..
                              APP_LOG: Mapped 0xac040000 -> 0xffffa24a0000 of size 33161216 bytes 
APP_LOG: Mapping 0x30e00000 ...
APP_LOG: Mappe0x30e00000 -> 0xffffa4930000 of size 3072 bytes 
IPC: Init ..Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
APP: Init ... Done !!!
     0.015201 s:  VZONE_INIT:Enabled
     0.015214 s:  VX_ZONE_ERROR:Enabled
   0.015218 s:  VX_ZONE_WARNING:Enabled
     0.017453 s:  VX_ZONE_INIT:[tivxInit:64] Initialization Done !!!
     0.0179 s:  VX_ZONE_INIT:[tivxHostInit:44] Initialization Done for HT !!!
VxTests version: 1.1-20170301
VCS version: unknown
Build config: Release

[ ======== ] Total 21932 tests from 144 test cases
Use test filter: tivxHwaVideoDecoder.MtiStreamProcessing

Use global OpenVX context: FALSE


[ -------- ] tests from tivxHwaVideoDecoder
[ RUN 0001 ] tivxaVideoDecoder.MultiStreamProcessing ...
[  133.333456] ion_buffer_destroy: buffer still mapped in the rnel
[     DONE ] tivxHwaVideoDecoder.MultiStreamProcessing
-------- ] 1 tests from test case tivxHwaVideoDecoder

[ ======== ]
[ ALL DE ] 1 test(s) from 1 test case(s) ran
[ PASSED   ] 1 test(s)
[ ILED   ] 0 test(s)
[ DISABLED ] 0 test(s)

===============================
OpenVX Conformance report summary
=================================

To be conformant to the OpenVbaseline, 1 required test(s) must pass. 1 tests passed, 0 testfailed. PASSED.
To be conformant to the User Data Object extension, 0 required test(s) must pass. 0 tests passed, 0 tts failed. PASSED.
Note: The 0 disabled tests are optional anare not considered for conformance.

#REPORT: YYYYMMDDHHMMSS FILTERED 21932 0 1 1 1 0 (version 1.1-20170301)
    48.5891 s:  VX_ZONE_INIT:[tivxHostDeInit:49] De-Initialization Do for HOST !!!
    48.190236 s:  VX_ZONE_INIT:[tivxDeInit:100] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTEERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
I: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ION ... !!!
MEM: Alloc's: 11 alloc's of 8986376 bytes 
MEM: Fe's : 11 free's  of 8986376 bytes 
MEM: Open's : 0 allocs  of bytes 
MEM: Deinit ION ... Done !!!
APP: Deinit ... Done !!!
root@j7-evm:/opt/vision_apps# 
root@j7-evm:/opt/vision_apps# 
root@j7-evm:/opt/vision_apps# VX_TEST_DATA_PATH=/opt/vision_apps/test_data ./vx_app_conformae.out --filter=tivxHwaVideoDecoder.MultiStreamProcessing
APP: Init ... !!!
APP_LOG: Mapping 0xac000000 ...
APP_LOG: Mped 0xac000000 -> 0xffff848b0000 of size 262144 bytes 
MEM: Init ION ... !!!
MEM: InitializeION (fd=4) !!!
MEM: Init ION ... Done !!!
PP_LOG: Mapping 0xac040000 ...
APP_LOG: Mapped 0xac040000 -> 0xffff82910000 of size 33161216 bytes 
APP_LOG: Mappingx30e00000 ...
APP_LOG: Mapped 0x30e00000 -> 0xffff84da0000 ofize 3072 bytes 
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
APP: Init ..Done !!!
     0.000638 s:  VX_ZONE_INIT:Enabled
     0.00064s:  VX_ZONE_ERROR:Enabled
     0.000652 s:  VX_ZONE_WARNING:Enabled
     0.001203 s:  VX_ZONE_INIT:[tivxInit:64] Initlization Done !!!
     0.001456 s:  VX_ZONE_INIT:[tivxHostIni44] Initialization Done for HOST !!!
VxTests version: 1.1-20170301
VCS version: unknown
Build config: Release

[ ====== ] Total 21932 tests from 144 test cases
Use test filte tivxHwaVideoDecoder.MultiStreamProcessing

Use global OpenVX context: FALSE


[ -------- ] tests from tivxHwaVideecoder
[ RUN 0001 ] tivxHwaVideoDecoder.MultiStreamProcessing..
[     DONE ] tivxHwaVideoDecoder.MultiStreamProcessing
[ ------ ] 1 tests from test case tivxHwaVideoDecoder

[ ======== ]
[ ALL DONE 1 test(s) from 1 test case(s) ran
[ PASSED   ] 1 test(s)
[ FLED   ] 0 test(s)
[ DISABLED ] 0 test(s)

=================================
OpenVX Conformance report summary
===============================

To be conformant to the OpenVX seline, 1 required test(s) must pass. 1 tests passed, 0 tests failed. PASSED.
To be conformant to the User Data Objecextension, 0 required test(s) must pass. 0 tests passed, 0 tes failed. PASSED.
Note: The 0 disabled tests are optional and are not considered for conformance.

#REPORT: YYYYMMDDMMSS FILTERED 21932 0 1 1 1 0 (version 1.1-20170301)
    13.6064 s:  VX_ZONE_INIT:[tivxHostDeInit:49] De-Initialization Done for HOST !!!
    13.656385 s:  VX_ZONE_INIT:[tivxDeIni100] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit N ... !!!
MEM: Alloc's: 11 alloc's of 8986376 bytes 
MEM: Fr's : 11 free's  of 8986376 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ION ... Done !!!
APP: Deinit ... De !!!
root@j7-evm:/opt/vision_apps# VX_TEST_DATA_PATH=/opt/vision_apps/test_data ./vx_app_conformae.out --filter=tivxHwaVideoDecoder.MultiStreamProcessing
APP: Init ... !!!
APP_LOG: Mapping 0xac000000 ...
APP_LOG: Mped 0xac000000 -> 0xffff905a0000 of size 262144 bytes 
MEM: It ION ... !!!
MEM: Initialized ION (fd=4) !!!
MEM: Init ION ... Done !!!
IPC: Init ... !!!
APP_LOG: Mappi 0xac040000 ...
APP_LOG: Mapped 0xac040000 -> 0xffff8e600000  size 33161216 bytes 
APP_LOG: Mapping 0x30e00000 ...
APP_LOG: Mapped 0x30e00000 -> 0xffff90a90000 of size 3072 byte
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTEERVICE: Init ... Done !!!
APP: Init ... Done !!!
     0.000330 s:  VX_ZONE_INIT:Enabled
     0.000345 s:  VX_ZONE_EOR:Enabled
     0.000349 s:  VX_ZONE_WARNING:Enabled
     0.0769 s:  VX_ZONE_INIT:[tivxInit:64] Initialization Done !!!
     0.000978 s:  VX_ZONE_INIT:[tivxHostInit:44] Initializion Done for HOST !!!
VxTests version: 1.1-20170301
VCS versn: unknown
Build config: Release

[ ======== ] Total 21932 tests from 144 test cases
Use test filter: tivxHwaVideocoder.MultiStreamProcessing

Use global OpenVX context: FALS


[ -------- ] tests from tivxHwaVideoDecoder
[ RUN 0001 ] tivxHwaVideoDecoder.MultiStreamProcessing ...
[     DONE ] tivxHwaVideoDecoder.MultiStreamProcessing
[ -------- ] 1 tests from test case tiHwaVideoDecoder

[ ======== ]
[ ALL DONE ] 1 test(s) from 1est case(s) ran
[ PASSED   ] 1 test(s)
[ FAILED   ] 0 test(s)
[ DISABLED ] 0 test(s)

===============================
OpenVX Conformance report summary
===============================

To be conformant to the OpenVX baseline, 1 required test(s) must pass. 1 tests passed, 0 tests failed. PAED.
To be conformant to the User Data Object extension, 0 reqred test(s) must pass. 0 tests passed, 0 tests failed. PASSED.
Note: The 0 disabled tests are optional and are not coidered for conformance.

#REPORT: YYYYMMDDHHMMSS FILTERED 212 0 1 1 1 0 (version 1.1-20170301)
    45.180468 s:  VX_ZONE_INIT:[tivxHostDeInit:49] De-Initialization Done for HOST !
    45.184812 s:  VX_ZONE_INIT:[tivxDeInit:100] De-Initialition Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit . !!!
IPC: DeInit ... Done !!!
MEM: Deinit ION ... !!!
MEM: loc's: 11 alloc's of 8986376 bytes 
MEM: Free's : 11 free's  of 8986376 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
M: Deinit ION ... Done !!!
APP: Deinit ... Done !!!
root@j7-e:/opt/vision_apps# VX_TEST_DATA_PATH=/opt/vision_apps/test_data ./vx_app_conformance.out --filter=tivxHwaVidDecoder.MultiStreamProcessing
APP: Init ... !!!
APP_LOG: Mapping 0xac000000 ...
APP_LOG: Mapped 0xac000000 -> 0xffff854e0000 of size 262144 bytes 
MEM:nit ION ... !!!
MEM: Initialized ION (fd=4) !!!
MEM: Init IO... Done !!!
IPC: Init ... !!!
APP_LOG: Mapping 0xac040000 ...
APP_LOG: Mapped 0xac040000 -> 0xffff83540000 of size3161216 bytes 
APP_LOG: Mapping 0x30e00000 ...
APP_LOG: Mapp 0x30e00000 -> 0xffff859d0000 of size 3072 bytes 
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVIC Init ... Done !!!
APP: Init ... Done !!!
     0.000404 s:  _ZONE_INIT:Enabled
     0.000427 s:  VX_ZONE_ERROR:Enabled
     0.000432 s:  VX_ZONE_WARNING:Enabled
     0.001092 s VX_ZONE_INIT:[tivxInit:64] Initialization Done !!!
     0.0053 s:  VX_ZONE_INIT:[tivxHostInit:44] Initialization Done for HOST !!!
VxTests version: 1.1-20170301
VCS version: unown
Build config: Release

[ ======== ] Total 21932 tests fm 144 test cases
Use test filter: tivxHwaVideoDecoder.MultiStreamProcessing

Use global OpenVX context: FALSE


-------- ] tests from tivxHwaVideoDecoder
[ RUN 0001 ] tivxHwideoDecoder.MultiStreamProcessing ...
     0.138337 s:  VX_ZONE_ERROR:[ownContextSendCmd:553] Command ack message returned failure cmd_status: -1
     138357 s:  VX_ZONE_ERROR:[ownNodeKernelInit:486] Target kernelTIVX_CMD_NODE_CREATE failed
     0.138377 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 1, kernecom.ti.hwa.video_decoder ... failed !!!
     0.138383 s:  VX_NE_ERROR:[vxVerifyGraph:2010] Node kernel init failed
     0.138388 s:  VX_ZONE_ERROR:[vxVerifyGraph:2064] Graph veri failed
[MCU2_1]    254.766752 s: REMOTE_SERVICE: Unable to cate tx semaphore
[MCU2_1]    254.766841 s: BSPP stream create failed 12
[MCU2_1]    254.767066 s:  VX_ZONE_ERROR:[txVideoDecoderCreate:562] tivxVideoDecoderCreate: MM_DEC Createailed

FAILED at /ti/j7/releases/06_02_00/06_02_00_21/psdk_rtos_auto_j7_06_02_00_21/tiovx/kernels_j7/hwa/test/test_deo_decoder.c:549
        Expected: VX_SUCCESS == vxVerifyGraph(grap
        Actual: VX_SUCCESS != VX_FAILURE

     0.175411 s:  VX_ZONE_WARNING:[vxReleaseContext:743] Stale reference 0xb075 of type 00000816 at external count 1, internal count 1
     175417 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased reference name = user_data_object_84
     0.175423 s:  VXONE_WARNING:[vxReleaseContext:743] Stale reference 0xb077e0 ofype 00000816 at external count 1, internal count 1
     0.175428 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleasereference name = user_data_object_85
     0.175434 s:  VX_ZONWARNING:[vxReleaseContext:743] Stale reference 0xb07a00 of type 00000816 at external count 1, internal count 1
     0.5438 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased refence name = user_data_object_86
     0.175444 s:  VX_ZONE_WARNING:[vxReleaseContext:743] Stale reference 0xb07c20 of pe 00000816 at external count 1, internal count 1
     0.1754 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased reference name = user_data_object_87
     0.175453 s:  VX_ZONE_RNING:[vxReleaseContext:743] Stale reference 0xb396c0 of type 00080f at external count 1, internal count 1
     0.175458 s:  VX_ZONE_WARNING:[vxReleaseContext:746] Unreleased refence name = image_88
     0.175463 s:  VX_ZONE_WARNING:[vxReleeContext:743] Stale reference 0xb39978 of type 0000080f at external count 1, internal count 1
     0.175467 s:  VX_ZO_WARNING:[vxReleaseContext:746] Unreleased reference name = ime_89
     0.175473 s:  VX_ZONE_WARNING:[vxReleaseContext:743] Stale reference 0xa9af20 of type 00000802 at external cot 1, internal count 0
     0.175477 s:  VX_ZONE_WARNING:[vxReaseContext:746] Unreleased reference name = graph_90
[ !FAILED! ] tivxHwaVideoDecoder.MultiStreamProcessing
[ ------ ] 1 tests from test case tivxHwaVideoDecoder

 ALL DONE ] 1 test(s) from 1 test case(s) ran
[ PASSED   ] 0 test(s)
[ FAILED   ] 1 test(s), listed below:
[ FAILED ] tivxHwaVideoDecoder.MultiStreamProcessing
[ DISABLED ] 0 te(s)

=================================
OpenVX Conformance report summary
=================================

To bconformant to the OpenVX baseline, 1 required test(s) must pas 0 tests passed, 1 tests failed. FAILED.
To be conformant to the User Data Object extension, 0 required test(s) must ps. 0 tests passed, 0 tests failed. PASSED.
Note: The 0 disabl tests are optional and are not considered for conformance.

#REPORT: YYYYMMDDHHMMSS FILTERED 21932 0 1 1 0 1 (versi 1.1-20170301)
     0.186380 s:  VX_ZONE_INIT:[tivxHostDeInit9] De-Initialization Done for HOST !!!
     0.190703 s:  VX_ZONE_INIT:[tivxDeInit:100] De-Initialization Done !!!
APPDeinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICEDeinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ION ... !!!
MEM: Alloc's: 12 alloc's  8986632 bytes 
MEM: Free's : 12 free's  of 8986632 bytes 
M: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ION ... Done !!!

0 Keerthy J over 5 years ago in reply to Keerthy J

TI__Guru**** 155100 points

decoder-robustness.7zHi Zhong Ming,

David has fixed the issue and myself and David tested the test case 100 & 500 times respectively.

2 Things:

1) Patch: 0001-osal-tirtos-osa_queue-Fix-semaphore-cleanup

To fix the actual issue with destroying the acquired semaphores. This needs the following build commands:

video_codec: rm –rf out
video_codec: make codec_apps
vision_apps: make

to install the changes to SD card i am doing:
make linux_fs_install_sd

2) The above will get us running mltiple iterations of decoder. That means by default output gets continuously written to SD
Card and we observed that card gets full by 8-12 iteration depending on the memory size of the card.

So we suggest the following change in the file \psdk_rtos_auto_j7_06_02_00_21\tiovx\kernels_j7\hwa\test\test_video_decoder.c:

Comment out the below line:

#define DUMP_DECODED_VIDEO_TO_FILE.

The above ensures that output is not written to SD or else one will have manually clear the output of decoder every iteration
to avoid SD card memory getting full.

This needs the following build commands:

tiovx directory: make tiovx
vision_apps: make

to install the changes to SD card i am doing:
make linux_fs_install_sd

I am posting this on behalf of David.

Thanks,
Keerthy

0 Leon Liao over 5 years ago in reply to Keerthy J

Prodigy 240 points

Hi Keerthy,

The patch does not work in my environment. And once the patch is used, it causes other stuck problem when decode multi H.264 streams. And if we run multi times, it will end up with decoder's crash just as before.

Leon

0 Keerthy J over 5 years ago in reply to Leon Liao

TI__Guru**** 155100 points

Hi Leon,

Can you please share the logs?

Also please make sure that 2 is taken into account.
That is needed to run multiple times successfully as the SD Card full can also result in failures.

Regards,
Keerthy

Processors

Processors forum

TDA4VM: RTOS Decoder graph verify failed after run multiple times