TDA4VM: Using tivxOpenglMosaicNode causes screen stuck, PVR_K: 1350: HW operation timeout, dump server info

Hello,

Let me check with the team on the status of this application here. Meanwhile, did you see this in any other SDK version between 7.3 and 8.4?

Regards,

Erick

Also, did you make any modifications to the SDK? Our SDK team has tested this application and it ran correctly. Was there any change to the filesystem, drivers, etc.? FYI the graphics drivers are not compatible across SDK releases.

- Erick

Hi Erick

I only use versions 7.3 and 8.4.

Regards,

gj

Hello Erick

The reason why such errors occur in operation is that it needs to run continuously for a certain time. Is the time for using our program so fixed? Sometimes it takes more than 10 minutes, and sometimes it takes 9 hours to run.

I only modified the mcu2-0 sensor driver.

I use ti-processor-sdk-rtos-j721e-evm-08_ 04_00_02\vision_ apps\apps\basic_demos\app_linux_arm_opengl_Mosaic app

It is only modified as follows, so that it can be processed repeatedly,

app_linux_arm_opengl_Mosaic.out  After running the Mosaic app for 60 hours, I went to check and found that the HW operation timeout program was not running.

vision_apps/apps/basic_demos/app_linux_arm_opengl_mosaic/main_linux_arm.c
@@ -387,7 +387,8 @@ int main(int argc, char *argv[])
{
status = vxVerifyGraph(graph);
}
- for (i=0; i<1; i++)
+ //for (i=0; i<1; i++)
+ while (1)
{
if(status == VX_SUCCESS)
{
@@ -397,6 +398,23 @@ int main(int argc, char *argv[])
{
status = vxWaitGraph(graph);
}
+
+ if(status == VX_SUCCESS)
+ {
+ #include <unistd.h>
+ if (access(output_file_path, F_OK)==0)
+ {
+ char new_file_path[APP_MAX_FILE_PATH];
+ static int index = 0;
+ //remove(output_file_path);
+ snprintf(new_file_path, APP_MAX_FILE_PATH, "%s/output/mosaic_output_file%d.bin", test_data_path, index);
+ rename(output_file_path, new_file_path);
+ index++;
+ if (index >= 10)
+ index = 0;
+ }
+ status = app_save_vximage_rgbx_to_bin_file(output_file_path, (vx_image)output);
+ }
}

Because there are too many logs. Some logs have been overwritten.

But there are also logs such as     HW operation timeout,

log:

4x_2023-01-16_09_22_22.log

Regards,

gj

This time, the exception occurred after only 60 seconds

The time of sending exception is not fixed each time.

log：

4x_2023-01-16_13_35_50_openGl_test.log

Hello,

Does the same error occur without your patch? We want to see if this patch is causing this or if it is due to the original application as our team does not see the error in the original app.

Regards,

Erick

Hello Erick

    The original program is executed once. No problem. This patch can keep him running repeatedly. This problem occurs only when it is always repeated. Our program can be executed once, but it is meaningless to execute it only once. In fact, it takes a long time to run.

Regards,

gj

Hello Erick

    Is there any new progress? If this problem is not found, how can I find it? What are the relevant modules that cause this problem?

Regards,

gj

Hello gj,

The issue might stem from a difference in GPU driver, its been updated since the SDK 7.3 release. If this is the case, we will need to identify the root cause. The logs currently are not sufficient to debug the GPU driver complaints. Let us collect more logs with the following procedure:

After GPU driver initialization, please run:

pvrdebug -loggroups main,mts,hwr

After issue has occurred, please run:

pvrlogdump

We can use these logs to get a better idea on what the issue can be.

Regards,

Erick

Hello Erick

run   app_linux_arm_opengl_mosaic.out pvr issue.  

pvr_error-dump.log

pvrlogdump file:

pvrlogdump_2208301921.txt.gz

run my app  opengl_display.out  pvr issue   pvrlogdump file：

pvrlogdump_2208301106.txt.gz

pvrlogdump_2208301128.txt.gz

Regards,

gj

Hello Erick

   I look forward to receiving your reply.

Regards,

gj

Hello,

Still waiting for analysis of these logs, they are actively being studied, I will get back to you soon.

Regards,

Erick

Hello,

The initial analysis of the logs shows an abnormality in the GPU register header initialization.

Are there any clocking/power changes that you are aware of? Also, have you reproduced this issue on any other EVM, or only 1 EVM is displaying this issue?

I've built this test on my side and will try to reproduce the issue as well.

Thanks,

Erick

Hello,

All our boards have issues,not just one.

Regards,

gj

Hello,  Erick

New log information.

[11:08:31:489][ 9.015986] PVR_K: 1023: RGX Firmware image 'rgx.fw.22.104.208.318' loaded
[11:08:31:489][ 9.026597] PVR_K: 1023: Shader binary image 'rgx.sh.22.104.208.318' loaded
[11:08:31:489][ 15.955389] PVR_K: 255: ------------[ PVR DBG: START (High) ]------------
[11:08:31:490][ 15.955400] PVR_K: 255: OS kernel info: Linux 5.10.120 #1 SMP PREEMPT Thu Feb 2 04:55:33 PST 2023 aarch64
[11:08:31:490][ 15.955404] PVR_K: 255: DDK info: Rogue_DDK_Linux_WS rogueddk 1.15@6133109 (release) j721e_linux
[11:08:31:490][ 15.955407] PVR_K: 255: Time now: 15955405us
[11:08:31:491][ 15.955410] PVR_K: 255: Services State: OK
[11:08:31:491][ 15.955413] PVR_K: 255: Server Errors: 0
[11:08:31:492][ 15.955419] PVR_K: 255: Connections Device ID:0(128) P1212-V1212-T1214-vx_app_
[11:08:31:492][ 15.955421] PVR_K: 255: ------[ Driver Info ]------
[11:08:31:492][ 15.955430] PVR_K: 255: Comparison of UM/KM components: MATCHING
[11:08:31:492][ 15.955434] PVR_K: 255: KM Arch: 64 Bit
[11:08:31:492][ 15.955437] PVR_K: 255: UM Connected Clients: 64 Bit
[11:08:31:492][ 15.955442] PVR_K: 255: UM info: 1.15 @ 6133109 (release) build options: 0x80000810
[11:08:31:493][ 15.955445] PVR_K: 255: KM info: 1.15 @ 6133109 (release) build options: 0x00000810
[11:08:31:493][ 15.955448] PVR_K: 255: Window system: wayland
[11:08:31:827][ 15.955453] PVR_K: 255: ------[ RGX Device ID:0 Start ]------
[11:08:31:827][ 15.955455] PVR_K: 255: ------[ RGX Info ]------
[11:08:31:827][ 15.955460] PVR_K: 255: Device Node (Info): 000000001c0ce814 (00000000d93b1683)
[11:08:31:827][ 15.955463] PVR_K: 255: RGX BVNC: 22.104.208.318 (rogue)
[11:08:31:827][ 15.955466] PVR_K: 255: RGX Device State: Active
[11:08:31:827][ 15.955469] PVR_K: 255: RGX Power State: ON
[11:08:31:828][ 15.955474] PVR_K: 255: FW info: 1.15 @ 6133109 (release) build options: 0x80000810
[11:08:31:828][ 15.955479] PVR_K: 255: BIF0 - OK
[11:08:31:828][ 15.955484] PVR_K: 255: RGX FW State: OK (HWRState 0x000000c1: HWR OK; FW fault; Restarting;)
[11:08:31:829][ 15.955489] PVR_K: 255: RGX FW Power State: RGXFWIF_POW_ON (APM disabled: 0 ok, 0 denied, 0 non-idle, 0 retry, 0 other, 0 total. Latency: 100 ms)
[11:08:31:830][ 15.955497] PVR_K: 255: RGX DVFS: 0 frequency changes. Current frequency: 750.000 MHz (sampled at 15954880330 ns). FW frequency: 100.000 MHz.
[11:08:31:830][ 15.955501] PVR_K: 255: RGX FW OS 0 - State: active; Freelists: Ok; Priority: 0; MTS on;
[11:08:31:830][ 15.955514] PVR_K: 255: FW Fault 1: "Unknown Client CCB command" (firmware/rgxfw_client_ccb.c:499)
[11:08:31:831][ 15.955518] PVR_K: 255: Data = 0xc0028640, CRTimer = 0x0000021c964f, OSTimer = 15.954532322
[11:08:31:831][ 15.955530] PVR_K: 255: FW Fault 2: "Unknown Client CCB command" (firmware/rgxfw_client_ccb.c:499)
[11:08:31:831][ 15.955534] PVR_K: 255: Data = 0xc0028640, CRTimer = 0x0000021c96f8, OSTimer = 15.954590007
[11:08:31:832][ 15.955546] PVR_K: 255: FW Fault 3: "Unknown Client CCB command" (firmware/rgxfw_client_ccb.c:499)
[11:08:31:832][ 15.955550] PVR_K: 255: Data = 0xc0028640, CRTimer = 0x0000021c9794, OSTimer = 15.954643255
[11:08:31:833][ 15.955561] PVR_K: 255: FW Fault 4: "Unknown Client CCB command" (firmware/rgxfw_client_ccb.c:499)
[11:08:31:833][ 15.955565] PVR_K: 255: Data = 0xc0028640, CRTimer = 0x0000021c9812, OSTimer = 15.954686263
[11:08:31:833][ 15.955579] PVR_K: 255: FW Fault 5: "Unknown Client CCB command" (firmware/rgxfw_client_ccb.c:499)
[11:08:31:834][ 15.955582] PVR_K: 255: Data = 0xc0028640, CRTimer = 0x0000021c98ad, OSTimer = 15.954739170
[11:08:31:834][ 15.955596] PVR_K: 255: FW Fault 6: "Unknown Client CCB command" (firmware/rgxfw_client_ccb.c:499)
[11:08:31:834][ 15.955599] PVR_K: 255: Data = 0xc0028640, CRTimer = 0x0000021c9931, OSTimer = 15.954784226
[11:08:31:835][ 15.955611] PVR_K: 255: FW Fault 7: "Unknown Client CCB command" (firmware/rgxfw_client_ccb.c:499)
[11:08:31:835][ 15.955614] PVR_K: 255: Data = 0xc0028640, CRTimer = 0x0000021c9d9b, OSTimer = 15.955170121
[11:08:31:835][ 15.955625] PVR_K: 255: FW Fault 8: "Unknown Client CCB command" (firmware/rgxfw_client_ccb.c:499)
[11:08:31:835][ 15.955629] PVR_K: 255: Data = 0xc0028640, CRTimer = 0x0000021c9ec3, OSTimer = 15.955271156
[11:08:31:837][ 15.955643] PVR_K: 255: Number of HWR: GP(0/0+0), 2D(0/0+0), TA(0/0+0), 3D(1/1+0), CDM(0/0+0), RAY(0/0+0), FALSE(0,0,0,0,0,0)
[11:08:31:837][ 15.955648] PVR_K: 255: DM 0 (GP)
[11:08:31:837][ 15.955656] PVR_K: 255: DM 1 (HWRflags 0x00000000: working;)
[11:08:31:837][ 15.955662] PVR_K: 255: DM 2 (HWRflags 0x00000000: working;)
[11:08:31:837][ 15.955668] PVR_K: 255: DM 3 (HWRflags 0x00000000: working;)
[11:08:31:837][ 15.955673] PVR_K: 255: Recovery 1: PID = 1212, frame = 0, HWRTData = 0x00000000, EventStatus = 0x00000400, Guilty Lockup
[11:08:31:845][ 15.955678] PVR_K: 255: CRTimer = 0x0000021c9c44, OSTimer = 15.955053044, CyclesElapsed = 27904
[11:08:32:167][ 15.955683] PVR_K: 255: PreResetTimeInCycles = 20224, HWResetTimeInCycles = 20224, TotalResetTimeInCycles = 40448
[11:08:32:170][ 15.955687] PVR_K: 255: BIF0 - FAULT:
[11:08:32:170][ 15.955690] PVR_K: 255: * MMU status (0x0000000000001041): PC = 1, Page Size = 0 (Page Catalog).
[11:08:32:170][ 15.955696] PVR_K: 255: * Request (0x00041e0000209000): MCU (128bit word within the Lower 256bits, TPUA_USC, Banks 4-7), Reading from 0x0000209000.
[11:08:32:171][ 15.955704] PVR_K: 255: PC index (0) out of bounds (0)
[11:08:32:172][ 15.955710] PVR_K: 255: DM 4 (HWRflags 0x00000000: working;)
[11:08:32:172][ 15.955716] PVR_K: 255: DM 5 (HWRflags 0x00000000: working;)
[11:08:32:172][ 15.955721] PVR_K: 255: RGX Kernel CCB WO:0x8 RO:0x8
[11:08:32:172][ 15.955725] PVR_K: 255: RGX Firmware CCB WO:0x4 RO:0x3
[11:08:32:172][ 15.955727] PVR_K: 255: RGX Kernel CCB commands executed = 8
[11:08:32:173][ 15.955730] PVR_K: 255: RGX SLR: Forced UFO updates requested = 0
[11:08:32:173][ 15.955733] PVR_K: 255: RGX Errors: WGP:0, TRP:0
[11:08:32:173][ 15.955736] PVR_K: 255: Thread0: FW IRQ count = 5
[11:08:32:173][ 15.955738] PVR_K: 255: Last sampled IRQ count in LISR = 5
[11:08:32:173][ 15.955744] PVR_K: 255: FW System config flags = 0x00020000 (Ctx switch options: Medium CSW profile; VDM CS INDEX mode;)
[11:08:32:174][ 15.955748] PVR_K: 255: FW OS config flags = 0x0000000F (Ctx switch: TDM; TA; 3D; CDM;)
[11:08:32:174][ 15.955752] PVR_K: 255: ------[ RGX registers ]------
[11:08:32:174][ 15.955768] PVR_K: 255: RGX Register Base Address (Linear): 0x000000001064c1d8
[11:08:32:174][ 15.955771] PVR_K: 255: RGX Register Base Address (Physical): 0x4E20000000
[11:08:32:174][ 15.955775] PVR_K: 255: CORE_ID : 0x0000000008470000
[11:08:32:175][ 15.955779] PVR_K: 255: CORE_REVISION : 0x00D0013E
[11:08:32:176][ 15.955782] PVR_K: 255: DESIGNER_REV_FIELD1 : 0x00000000
[11:08:32:176][ 15.955785] PVR_K: 255: DESIGNER_REV_FIELD2 : 0x00000000
[11:08:32:176][ 15.955788] PVR_K: 255: CHANGESET_NUMBER : 0x0000000000000000
[11:08:32:176][ 15.955792] PVR_K: 255: CLK_CTRL : 0x0aaaaa002a2aaaaa
[11:08:32:176][ 15.955796] PVR_K: 255: CLK_STATUS : 0x0000000000600000
[11:08:32:176][ 15.955800] PVR_K: 255: CLK_CTRL2 : 0x0000000000000000
[11:08:32:177][ 15.955803] PVR_K: 255: CLK_STATUS2 : 0x0000000000000000
[11:08:32:177][ 15.955806] PVR_K: 255: EVENT_STATUS : 0x00000400
[11:08:32:177][ 15.955810] PVR_K: 255: TIMER : 0x00000000021ca273
[11:08:32:177][ 15.955813] PVR_K: 255: BIF_FAULT_BANK0_MMU_STATUS : 0x00000000
[11:08:32:177][ 15.955817] PVR_K: 255: BIF_FAULT_BANK0_REQ_STATUS : 0x0000000000000000
[11:08:32:178][ 15.955820] PVR_K: 255: BIF_FAULT_BANK1_MMU_STATUS : 0x00000000
[11:08:32:178][ 15.955824] PVR_K: 255: BIF_FAULT_BANK1_REQ_STATUS : 0x0000000000000000
[11:08:32:179][ 15.955827] PVR_K: 255: BIF_MMU_STATUS : 0x00000000
[11:08:32:188][ 15.955830] PVR_K: 255: BIF_MMU_ENTRY : 0x00000000
[11:08:32:188][ 15.955833] PVR_K: 255: BIF_MMU_ENTRY_STATUS : 0x0000000000000000
[11:08:32:188][ 15.955837] PVR_K: 255: BIF_STATUS_MMU : 0x00000000
[11:08:32:188][ 15.955840] PVR_K: 255: BIF_READS_EXT_STATUS : 0x00000000
[11:08:32:189][ 15.955842] PVR_K: 255: BIF_READS_INT_STATUS : 0x00000000
[11:08:32:189][ 15.955846] PVR_K: 255: BIFPM_STATUS_MMU : 0x00000000
[11:08:32:189][ 15.955849] PVR_K: 255: BIFPM_READS_EXT_STATUS : 0x00000000
[11:08:32:318][ 15.955852] PVR_K: 255: BIFPM_READS_INT_STATUS : 0x00000000
[11:08:32:318][ 15.955857] PVR_K: 255: BIF_CAT_BASE_INDEX : 0x0000000000000100
[11:08:32:318][ 15.955860] PVR_K: 255: BIF_CAT_BASE0 : 0x0000000000000000
[11:08:32:318][ 15.955863] PVR_K: 255: BIF_CAT_BASE1 : 0x00000008ad5ed000
[11:08:32:319][ 15.955867] PVR_K: 255: BIF_CAT_BASE2 : 0x0000000000000000
[11:08:32:319][ 15.955870] PVR_K: 255: BIF_CAT_BASE3 : 0x0000000000000000
[11:08:32:319][ 15.955874] PVR_K: 255: BIF_CAT_BASE4 : 0x0000000000000000
[11:08:32:319][ 15.955877] PVR_K: 255: BIF_CAT_BASE5 : 0x0000000000000000
[11:08:32:319][ 15.955880] PVR_K: 255: BIF_CAT_BASE6 : 0x0000000000000000
[11:08:32:320][ 15.955884] PVR_K: 255: BIF_CAT_BASE7 : 0x0000000000000000
[11:08:32:320][ 15.955887] PVR_K: 255: BIF_CTRL_INVAL : 0x00000000
[11:08:32:320][ 15.955890] PVR_K: 255: BIF_CTRL : 0x000000C0
[11:08:32:320][ 15.955893] PVR_K: 255: BIF_PM_CAT_BASE_VCE0 : 0x0000000000000000
[11:08:32:321][ 15.955896] PVR_K: 255: BIF_PM_CAT_BASE_TE0 : 0x0000000000000000
[11:08:32:321][ 15.955900] PVR_K: 255: BIF_PM_CAT_BASE_ALIST0 : 0x0000000000000000
[11:08:32:321][ 15.955903] PVR_K: 255: BIF_PM_CAT_BASE_VCE1 : 0x0000000000000000
[11:08:32:323][ 15.955906] PVR_K: 255: BIF_PM_CAT_BASE_TE1 : 0x0000000000000000

Hello,

I've tried this application myself, but I do not see this crash after 24 hours of running. Are the EVMs you are using the TI EVMs or your own custom ECU?

We've had our GPU vendor analyze the logs you provided and they do not show any abnormalities. We need some reproduce-ability to try and debug this.

Are you still seeing this issue?

Regards,

Erick

Hello,

Is this still an issue? Can I close the ticket?

Regards,

Erick