• State TI Thinks Resolved
  • Locked Locked
  • Replies 15 replies
  • Answers 2 answers
  • Subscribers 100 subscribers
  • Views 552 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Original question:

SK-TDA4VM: Gstreamer object detection latency issues

SK-TDA4VM: TDA4VM restart issue, after changing display to weston and using glimagesink

Gokul Soman
Prodigy 60 points
Part Number: SK-TDA4VM
Other Parts Discussed in Thread: TDA4VM

As I have not been able to achieve high FPS (greater than 30 FPS) using kmssink, I had changed the display element in gstreamer to glimagesink. This was based on a previous query I had raised (Link).

Although the achievable FPS is higher, there is a restart issue that is occurring as of now. I am not able to diagnose the error as the restart of the board happens fairly quickly. The terminal output is as shown

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
root@tda4vm-sk:/opt/edge_ai_apps/apps_python/working_code/detection_480# ./board_inference.py
libtidl_onnxrt_EP loaded 0x39e92c70
Final number of subgraphs created are : 1, - Offloaded Nodes - 276, Total Nodes - 276
APP: Init ... !!!·
MEM: Init ... !!!·
MEM: Initialized DMA HEAP (fd=4) !!!·
MEM: Init ... Done !!!·
IPC: Init ... !!!·
IPC: Init ... Done !!!·
REMOTE_SERVICE: Init ... !!!·
REMOTE_SERVICE: Init ... Done !!!·
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The final line is the error that is displayed.

Kindly help resolve this issue.

I would also like to get more information on these 2 topics

1. How to optimize the targeting of DSPs so as to avoid bottlenecks, or overloading of any DSP

2. The significance of the parameter out-pool-size in tiovxdlpreproc gstreamer element, and how to determine the best value for this parameter

Thanks,

Gokul Soman

  • 0
    TI__Mastermind 40623 points

    Hi Gokul,

    As for the initial issue, could you try running the attached script before running the pipeline:

    tda4vm_gpu_fix.sh

    As for 1.), I would recommend using the perfstat tool to see which cores are being overloaded (at 100% usage): https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-sk-tda4vm/08_04_00/exports/docs/performance_visualizer.html#generating-performance-logs and move the plugins between the cores such that the DSPs are not overloaded.

    2.) I will need to check on this one, but our examples usually specify out-pool-size=4: https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-sk-tda4vm/08_04_00/exports/docs/data_flows.html

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    TI__Mastermind 40623 points

    Hi Gokul,

    Just got some more clarifications on 2). The out-pool-size should depend on where the bottleneck is, that is, if there is a GStreamer plugin that often waits for a buffer to be free due to all the buffers in the buffer pool being used, increasing the out-pool-size should improve the performance.

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    Prodigy 60 points

    Hi,

    I have tried reducing the frame rate below 20 to run this. This happens even when the DSPs are not overloaded (20 %).

    Specifically, the error that keeps on coming is this

    (4253) PVR:(Error): ScheduleTA: Skipping render from different gc/thread! [ :5864 ]

    Can you explain when this error pops up in the TDA4VM platform? And also any method to solve this?

    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    As of now, the current pipeline is such

    v4l2src device=/dev/video2 io-mode=2 ! image/jpeg, width=640, height=480 ! jpegdec ! tiovxdlcolorconvert target=1 ! video/x-raw, format=NV12 ! tiovxmultiscaler name=split_01
    split_01. ! queue max-size-buffers=2 leaky=2 ! video/x-raw, width=640, height=480 ! videoscale ! video/x-raw, width=640, height=640 ! tiovxdlpreproc data-type=3 target=0 channel-order=0 tensor-format=bgr out-pool-size=4 ! application/x-tensor-tiovx ! appsink name=pre_0 max-buffers=1 drop=true
    split_01. ! queue ! video/x-raw, width=640, height=480 ! tiovxdlcolorconvert target=1 out-pool-size=4 ! video/x-raw, format=RGB ! appsink name=sen_0 max-buffers=2 drop=true
    appsrc format=GST_FORMAT_TIME is-live=true block=true do-timestamp=true name=post_0 ! tiovxdlcolorconvert target=1 ! video/x-raw,format=NV12, width=640, height=480 ! fpsdisplaysink video-sink="glimagesink"

    As a further input, I am using threading in my code --> different appsink pipelines outputs are taken in parallel threads in the python code. Could this be causing issue?

    Thanks,

    Gokul

  • 0 in reply to Gokul Soman
    TI__Mastermind 40623 points

    Hi Gokul,

    The PVR error is GPU related. Could you try running the script tda4vm_gpu_fix.sh that I sent two responses back?

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    Prodigy 60 points

    Hi,

    Based on this input

    1. I first changed the display to wayland

    2. Ran the gpu fix script

    3. Ran the rest of the code.

    The board does not restart, but becomes unusable, after which I have to manually restart (by pressing the reset button). The terminal also stops working.

    A section of the logs that is show in minicom is as such

     93.818919] PVR_K:(Error):  2086- 2163: PVRSRVWaitForValueKM() failed (PVRSR]
    [   94.696452] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   95.211078] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   95.725696] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   96.240314] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   96.754944] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   97.269557] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   97.785207] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   98.300009] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   98.814794] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   99.329486] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   99.844641] PVR_K:(Error):     7: RGXPollForGPCommandCompletion: Failed! Err]
    [   99.859388] PVR_K:(Error):     7: RGXForcedIdleRequest: Idle request failed.]
    [   99.871292] PVR_K:(Error):     7: PVRSRVSetDeviceSystemPowerState: Forced id]
    [   99.882657] PVR_K:(Error):     7: PVRSRVSetDeviceSystemPowerState: Transitio]
    [   99.896898] PVR_K:  7: ------------[ PVR DBG: START (High) ]------------     
    [   99.903700] PVR_K:  7: OS kernel info: Linux 5.10.100-g7a7a3af903 #1 SMP PRE4
    [   99.914495] PVR_K:  7: DDK info: Rogue_DDK_Linux_WS rogueddk 1.13@5776728 (rx
    [   99.923349] PVR_K:  7: Time now: 99923342us                                  
    [   99.927620] PVR_K:  7: Services State: OK                                    
    [   99.931808] PVR_K:  7: Connections: P1970-V1970-T1970-weston, P2086-V2086-T2t
    [   99.940392] PVR_K:  7: ------[ Driver Info ]------                           
    [   99.945338] PVR_K:  7: Comparison of UM/KM components: MATCHING              
    [   99.951445] PVR_K:  7: KM Arch: 64 Bit                                       
    [   99.955294] PVR_K:  7: UM Connected Clients: 64 Bit                          
    [   99.960236] PVR_K:  7: UM info: 1.13 @  5776728 (release) build options: 0x80
    [   99.968007] PVR_K:  7: KM info: 1.13 @  5776728 (release) build options: 0x00
    [   99.975806] PVR_K:  7: Window system: wayland                                
    [   99.980365] PVR_K:  7: FW info: 1.13 @  5776728 (release) build options: 0x80
    [   99.988179] PVR_K:  7: ------[ RGX Device: Start ]------                     
    [   99.993580] PVR_K:  7: ------[ RGX Info ]------                              
    [   99.998264] PVR_K:  7: RGX BVNC: 22.104.208.318 (rogue)                      
    [  100.003679] PVR_K:  7: RGX Device State: Active                              
    [  100.008311] PVR_K:  7: RGX Power State: ON                                   
    [  100.012442] PVR_K:  7: BIF0 - OK                                             
    [  100.015704] PVR_K:  7: RGX firmware connection state: UP (Fw=active; OS=acti)
    [  100.023163] PVR_K:  7: RGX FW State: OK (HWRState 0x000000c1: HWR OK; FW fau)
    [  100.031793] PVR_K:  7: RGX FW Power State: RGXFWIF_POW_IDLE (APM disabled: 0)
    [  100.045135] PVR_K:  7: RGX DVFS: 0 frequency changes. Current frequency: 749.
    [  100.057960] PVR_K:  7: RGX FW OS 0 - State: active; Freelists: Ok; Priority:
    [  100.065508] PVR_K:  7: FW Fault 14: "Unknown Client CCB command" (firmware/r)
    [  100.075189] PVR_K:  7:             Data = 0xc00c08e0, CRTimer = 0x00000eda0c4
    [  100.084650] PVR_K:  7: FW Fault 15: "Unknown Client CCB command" (firmware/r)
    [  100.094595] PVR_K:  7:             Data = 0xc00c0c00, CRTimer = 0x00000eda0d3
    [  100.104094] PVR_K:  7: FW Fault 16: "Unknown Client CCB command" (firmware/r)
    [  100.113665] PVR_K:  7:             Data = 0xc00c0840, CRTimer = 0x000010034f2
    [  100.123139] PVR_K:  7: FW Fault 17: "Unknown Client CCB command" (firmware/r)
    [  100.133370] PVR_K:  7:             Data = 0xc00c08e0, CRTimer = 0x000010034f7
    [  100.142808] PVR_K:  7: FW Fault 18: "Unknown Client CCB command" (firmware/r)
    [  100.152418] PVR_K:  7:             Data = 0xc00c0c00, CRTimer = 0x000010034f1
    [  100.161714] PVR_K:(Error):   249: DevicesWatchdogThread: Device status not O]
    [  100.169615] PVR_K:  7: FW Fault 19: "Unknown Client CCB command" (firmware/r)
    [  100.179083] PVR_K:  249: ------------[ PVR DBG: START (High) ]------------   
    [  100.185944] PVR_K:  7:             Data = 0xc00c0840, CRTimer = 0x00001004595
    [  100.195238] PVR_K:  249: OS kernel info: Linux 5.10.100-g7a7a3af903 #1 SMP P4
    [  100.206001] PVR_K:  7: FW Fault 20: "Unknown Client CCB command" (firmware/r)
    [  100.215471] PVR_K:  249: DDK info: Rogue_DDK_Linux_WS rogueddk 1.13@5776728 x
    [  100.224331] PVR_K:  7:             Data = 0xc00c08e0, CRTimer = 0x00001004596
    [  100.233627] PVR_K:  249: Time now: 100227367us                               
    [  100.238064] PVR_K:  7: FW Fault 21: "Unknown Client CCB command" (firmware/r)
    [  100.247534] PVR_K:  249: Services State: OK                                  
    [  100.251714] PVR_K:  7:             Data = 0xc00c0c00, CRTimer = 0x000010045a6
    [  100.261006] PVR_K:  249: Connections: P1970-V1970-T1970-weston, P2086-V2086-t
    [  100.269602] PVR_K:  7: RGX Kernel CCB WO:0x36 RO:0x36                        
    [  100.274644] PVR_K:  249: ------[ Driver Info ]------                         
    [  100.279597] PVR_K:  7: RGX Firmware CCB WO:0x1 RO:0x0                        
    [  100.284953] PVR_K:  249: Comparison of UM/KM components: MATCHING            
    [  100.291034] PVR_K:  7: RGX Kernel CCB commands executed = 566                
    [  100.296765] PVR_K:  249: KM Arch: 64 Bit                                     
    [  100.300674] PVR_K:  7: RGX SLR: Forced UFO updates requested = 0             
    [  100.306664] PVR_K:  249: UM Connected Clients: 64 Bit                        
    [  100.311701] PVR_K:  7: Thread0: FW IRQ count = 1049                          
    [  100.316985] PVR_K:  7: Last sampled IRQ count in LISR = 1049                 
    [  100.322635] PVR_K:  249: UM info: 1.13 @  5776728 (release) build options: 00
    [  100.330448] PVR_K:  7: FW System config flags = 0x00020020 (Ctx switch optio)
    [  100.342854] PVR_K:  249: KM info: 1.13 @  5776728 (release) build options: 00
    [  100.350667] PVR_K:  7: FW OS config flags = 0x00000007 (Ctx switch: TA; 3D; )
    [  100.358145] PVR_K:  249: Window system: wayland                              
    [  100.362668] PVR_K:  7: ------[ RGX registers ]------                         
    [  100.367869] PVR_K:  7: RGX Register Base Address (Linear):   0x00000000cc48d1
    [  100.375338] PVR_K:  7: RGX Register Base Address (Physical): 0x4E20000000    
    [  100.382329] PVR_K:  7: CORE_ID                       : 0x0000000008470000    
    [  100.389213] PVR_K:  7: CORE_REVISION                 : 0x00D0013E            
    [  100.395424] PVR_K:  7: DESIGNER_REV_FIELD1           : 0x00000000            
    [  100.401703] PVR_K:  7: DESIGNER_REV_FIELD2           : 0x00000000            
    [  100.407989] PVR_K:  7: CHANGESET_NUMBER              : 0x0000000000000000    
    [  100.414861] PVR_K:  7: CLK_CTRL                      : 0x0aaaaa002a2aaaaa    
    [  100.421851] PVR_K:  7: CLK_STATUS                    : 0x0000000000600000    
    [  100.428741] PVR_K:  7: CLK_CTRL2                     : 0x0000000000000000    
    [  100.435701] PVR_K:  7: CLK_STATUS2                   : 0x0000000000000000    
    [  100.442627] PVR_K:  7: EVENT_STATUS                  : 0x00004410       

    ----------------------------------------------

    Hence the code does not work at all If I run the gpu_fix script, but works to some extent if I run without it.

    Thanks for your constant support.

    Regards,

    Gokul

  • 0 in reply to Gokul Soman
    TI__Mastermind 40623 points

    Hi Gokul,

    Could you also post the output from the gpu_fix script? I would like to make sure the registers are actually getting updated, since on my setup this script was able to fix the initial PVR issue.

    Also, please note that the gpu_fix script needs to run after each reboot, since the register changes are not persistent over reboot.

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    Prodigy 60 points

    Hi,

    Attaching the output of the gpu_fix script here

    -------------------------------------------

    I tried some more things to locate the error. All of the output is taken through minicom.

    If I run the gpu fix script first, and then initialize weston, I'm getting the error as such.

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    /dev/mem opened.
    Memory mapped at address 0xffff9ed80000.
    Read at address 0x45DC5100 (0xffff9ed85100): 0x00003070
    Write at address 0x45DC5100 (0xffff9ed85100): 0x30000000, readback 0x30000000
    /dev/mem opened.
    Memory mapped at address 0xffff87410000.
    Read at address 0x45DC5104 (0xffff87415104): 0x00007070
    Write at address 0x45DC5104 (0xffff87415104): 0x30000000, readback 0x30000000
    /dev/mem opened.
    Memory mapped at address 0xffff84150000.
    Read at address 0x45DC5108 (0xffff84155108): 0x00007070
    Write at address 0x45DC5108 (0xffff84155108): 0x30000000, readback 0x30000000
    /dev/mem opened.
    Memory mapped at address 0xffff95350000.
    Read at address 0x45DC510C (0xffff9535510c): 0x00007070
    Write at address 0x45DC510C (0xffff9535510c): 0x30000000, readback 0x30000000
    /dev/mem opened.
    Memory mapped at address 0xffff87b40000.
    Read at address 0x45DC5110 (0xffff87b45110): 0x00007070
    Write at address 0x45DC5110 (0xffff87b45110): 0x30000000, readback 0x30000000
    /dev/mem opened.
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    Hence, I am initializing weston first and then running the gpu fix script. Which does not cause any errors.

    After this, even if I try a simple pipeline, as such

    gst-launch-1.0 v4l2src device=/dev/video2 ! image/jpeg, width=640, height=480, framerate=60/1 ! jpegdec ! glimagesink

    , I am getting errors as such

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    Setting pipeline to PAUSED ...
    Pipeline is live and does not need PREROLL ...
    Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(Gs;
    Setting pipeline to PLAYING ...
    New clock: GstSystemClock
    [ 180.706668] PVR_K:(Error): 66: RGXPollForGPCommandCompletion: Failed! Err]
    [ 181.721455] PVR_K:(Error): 66: RGXPollForGPCommandCompletion: Failed! Err]
    [ 181.832363] PVR_K:(Error): 267: DevicesWatchdogThread: Device status not O]
    [ 181.840389] PVR_K: 267: ------------[ PVR DBG: START (High) ]------------
    [ 181.847342] PVR_K: 267: OS kernel info: Linux 5.10.100-g7a7a3af903 #1 SMP P4
    [ 181.858151] PVR_K: 267: DDK info: Rogue_DDK_Linux_WS rogueddk 1.13@5776728 x
    [ 181.867053] PVR_K: 267: Time now: 181867050us
    [ 181.871526] PVR_K: 267: Services State: OK
    [ 181.875744] PVR_K: 267: Connections: P1437-V1437-T1437-weston, P1464-V1464-t
    [ 181.884387] PVR_K: 267: ------[ Driver Info ]------
    [ 181.889379] PVR_K: 267: Comparison of UM/KM components: MATCHING
    [ 181.895497] PVR_K: 267: KM Arch: 64 Bit
    [ 181.899453] PVR_K: 267: UM Connected Clients: 64 Bit
    [ 181.904534] PVR_K: 267: UM info: 1.13 @ 5776728 (release) build options: 00
    [ 181.912391] PVR_K: 267: KM info: 1.13 @ 5776728 (release) build options: 00
    [ 181.920243] PVR_K: 267: Window system: wayland
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    If I only initialize the weston, and skip the gpu fix script, atleast the pipeline mentioned above will run. Although the PVR error will be displayed in this case

    Fullscreen
    1
    2
    3
    4
    5
    6
    Setting pipeline to PAUSED ...
    Pipeline is live and does not need PREROLL ...
    Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayWayland\)\ gldisplaywayland0";
    Setting pipeline to PLAYING ...
    New clock: GstSystemClock
    (1461) PVR:(Error): ScheduleTA: Skipping render from different gc/thread! [ :5864 ]
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    The main problem for me in executing my script is the restart issue. Would like any points that may help regarding this.

    Thanks and Regards,

    Gokul

  • 0 in reply to Gokul Soman
    TI__Mastermind 40623 points

    Hi Gokul,

    It is probably due to the version of GPU driver. If you have the bandwidth, can you see if you can follow this guide using PSDK 8.4 instead of 8.2 to update GPU PVR driver to 1.15: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1094811/faq-tda4vm-how-do-i-update-the-gpu-driver-to-a-newer-version

    If not, I will need around 1 to 2 days to generate/test the binaries that would work with the Edge AI SDK 8.4 image.

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    Prodigy 60 points

    Hi,

    I tried to do this part. But in the end, the image I built does not have many of the required functions that were available in the balena-etcher image.

    This includes

    - onnxruntime (import error)

    - tiovx gstreamer elements

    If these are also present, then I maybe able to port the code into this.

    ---------------------------------------------

    Were you asking me to flash the 8.4 balena-etcher image first to sd card, and then use the generated .ko file (step 8) in that image?

    Regards,

    Gokul

  • 0 in reply to Gokul Soman
    TI__Mastermind 40623 points

    Hi Gokul,

    Gokul Soman said:
    Were you asking me to flash the 8.4 balena-etcher image first to sd card, and then use the generated .ko file (step 8) in that image?

    Yes, exactly. Could you try copying in the following into the rootfs partition of the SD card:

    8_4_sdk_new_gpu_driver.zip

    These are the latest GPU drivers available, compiled for the latest 8.4 SDK. I have tested this combination out and so far have not seen the PVR error messages.

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    Prodigy 60 points

    Hi,

    Tried the method you told. Still facing errors --> no restart, but inference is not happening

    - In minicom, the error is as such

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    [ 207.074895] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 207.824590] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 207.979414] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 208.600068] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 209.319356] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 209.762895] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 213.895349] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 214.829283] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 215.136581] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 216.528537] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 216.585591] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 217.579463] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 219.508908] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 221.654535] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 221.826509] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 222.936417] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 223.410014] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 224.101742] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 224.262895] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 224.733143] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    [ 225.769607] tidss 4a00000.dss: CRTC1 SYNC LOST: (irq 800)
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    In SSH, when I run the code, the error is coming as such

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    ./board_inference.py
    libtidl_onnxrt_EP loaded 0xba962c0
    Final number of subgraphs created are : 1, - Offloaded Nodes - 276, Total Nodes - 276
    APP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=4) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
    527.530898 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
    527.530949 s: VX_ZONE_INIT:Enabled
    527.530958 s: VX_ZONE_ERROR:Enabled
    527.530966 s: VX_ZONE_WARNING:Enabled
    527.532497 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
    527.535444 s: VX_ZONE_INIT:[tivxHostInitLocal:86] Initialization Done for HOST !!!
    527.565732 s: VX_ZONE_ERROR:[ownContextSendCmd:814] Command ack message returned failure cmd_status: -1
    527.565764 s: VX_ZONE_ERROR:[ownContextSendCmd:850] tivxEventWait() failed.
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    Again, thanks for the constant support from your side.

    Thanks,

    Gokul

  • 0 in reply to Gokul Soman
    TI__Mastermind 40623 points

    Hi Gokul,

    If this is on the 8.2 SDK version, could you try running the following script:

    sk_pmic_reboot_fix.sh

    We had a bug that was fixed in 8.4 SDK version where the board will reboot itself in certain circumstances.

    As for the logs:

    For the first set of logs shared with the tidss (TI display sub system) issues, it looks like a display/GPU issue. It may be the drivers I shared were not happy with the system - I compiled the binaries for 8.4, but if you are on 8.2, I can recompile and test them on that version as well before sharing.

    As for the second set of logs, they are related to TIDL (TI Deep Learning) node, which is called by the runtimes like ONNX or TFLite runtimes. These seem separate from the issues that we were having, since the TIDL error messages should be unrelated to GPU or display. These error messages may come up if the application shuts itself off in a weird state, and usually fixed by a reboot of the board. Do these error messages come up after rebooting?

    Lastly, thank you for your patience with this debug. We will continue to debug on our end as well, and hopefully we can work out a fix together.

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    Prodigy 60 points

    Hi Takuma,

    As for the logs:

    For the first set of logs shared with the tidss (TI display sub system) issues, it looks like a display/GPU issue. It may be the drivers I shared were not happy with the system - I compiled the binaries for 8.4, but if you are on 8.2, I can recompile and test them on that version as well before sharing.

    As for the second set of logs, they are related to TIDL (TI Deep Learning) node, which is called by the runtimes like ONNX or TFLite runtimes. These seem separate from the issues that we were having, since the TIDL error messages should be unrelated to GPU or display. These error messages may come up if the application shuts itself off in a weird state, and usually fixed by a reboot of the board. Do these error messages come up after rebooting?

    The two sets of logs are coming together. I just kept two terminals open, one in minicom, and one in SSH. I was taking the outputs that were coming in each terminal in this case. I ran the code on the 8.4 version of TI SDK, so dont really know what happened there..

    It would be great if you can compile the binaries for 8.2. In the 8.4 one, I am facing the previous error logs, constant glitches on screen, and no bounding boxes being displayed. I will test it on that as well from my side.

    I will try the script provided by you as well, and update you if any issues are obtained on that side.

    Regards

    Gokul

  • 0 in reply to Gokul Soman
    TI__Mastermind 40623 points

    Hi Gokul,

    I posted on our previous thread the solution to the original issue where kmssink was capping at 30 FPS: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1159489/sk-tda4vm-gstreamer-object-detection-latency-issues

    We will continue to look into the issue with glimagesink/GPU, but let us know if the changes to the GStreamer pipeline I posted in our previous thread will unblock you.

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    TI__Mastermind 40623 points

    Hi Gokul,

    I have not heard back in a while, but were you able to work around the issue being faced? Or is this still an open issue?

    Regards,

    Takuma