• State
  • Replies 18 replies
  • Subscribers 32 subscribers
  • Views 345 views
  • Users 0 members are here
Support feedback
Options
Options
Related

AM625: AM62x: PowerVR "Guilty Lockup" under sustained WebGL — GPU resets, client never regains a context

Siddhesh Hande
Prodigy 65 points

Part Number: AM625

On AM62x, sustained WebGL in Chromium intermittently causes the PowerVR firmware to
time out a render job, declare `Guilty Lockup`, and reset the GPU. The reset succeeds
— `RGX Health Status: OK` afterwards — but the client never gets a working rendering
context back. The page goes blank permanently and only recovers if the browser is
restarted.

**Reproduced on an unmodified `tisdk-default-image-am62xx-evm-12.01.00.05.03`, on
HDMI at 1600x900, with DDK 26.1.6967606 and kernel 6.18.38-ti** — no custom image,
kernel, DTS or panel involved. Also reproduced on DDK 23.3.6512818 (SDK 9.2) and
24.2.6643903 (SDK 11).

Every variable differs across the failing set except one: **firmware
`rgx.fw.33.15.11.3` is identical on SDK 9.2, 11 and 12.1.**

Timing is random and spans a wide range — observed at 1.25 h, 3.89 h, 5.42 h,
7.62 h, 8.06 h and 49.28 h of continuous rendering. **No run has survived
indefinitely.** The board previously recorded as clean past 20 h went on to fail at
49.28 h with an identical signature, so a long clean run indicates a slow board, not
an immune one.

Full `PVR DBG` dump from the SDK 12.1 reproduction is attached
(`pvr-dump-sdk12.1-guilty-lockup.txt`).AM62x PowerVR Guilty Lockup under.txt 

  • 0
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    Thank you for providing steps for reproducing the issue, I will let you know of an update as soon as progress has been made. 

    Regards,

    Shriya

  • 0 in reply to Shriya Surti
    Prodigy 65 points

    Hi,

    able to reproduce issue on your side ?

    Or anyone facing this kind of issue, and resolution given by TI team ?

  • 0 in reply to Siddhesh Hande
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    We have not had the chance to reproduce just yet, I will get back to you sometime this week. Thanks for your patience. 

    Regards,

    Shriya

  • 0 in reply to Shriya Surti
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    I am currently running the chromium test on my end, so far, no issues.

    To move along the debugging process, I have a couple of questions:

    1. Would you be able to see if you can reproduce on the AM62 evm?
    2. You mentioned in the text file provided that custom hardware was used, we would want to see if this is also the case on the evm to see if it is a hardware or software issue.
    3. It was indicated that the board was slow, can this be replicated on multiple boards?
    4. Have you tried additional display panels at different resolutions like 1024x1024?
    5. Is this only an issue in full screen mode?
    6. Were additional flags added when launching chromium and are you running as the root user or weston user? 

    Additionally, the full pvr dbg dumps would be appreciated.

    Regards,

    Shriya

  • 0 in reply to Shriya Surti
    Prodigy 65 points

    Hi Team,

    Yes it got reproduced on AM62x evm as well, below SDK12 ti-default-os i tried on EVM.  I already attached log of that as well (pvr-dump-sdk12.1-guilty-lockup.txt)

    https://dr-download.ti.com/software-development/software-development-kit-(sdk)/MD-PvdSyIiioq/12.01.00.05.03/tisdk-default-image-am62xx-evm-12.01.00.05.03.rootfs.wic.xz

    below web graphics website used for this, number of objects keep more e.g above 10k or 20K.
    Webgl website : https://webglsamples.org/aquarium/aquarium.html.

    3. It was indicated that the board was slow, can this be replicated on multiple boards?

    ---> Yes it reproduced on multiple boards. our custon board we used exact replica and reference from AM62 evm.

    4. Have you tried additional display panels at different resolutions like 1024x1024?
    --> with AM62x  EVM I used desktop monitor. But with our custom board we have resultion of 1024x600 panel.

    5. Is this only an issue in full screen mode?

    ---> browser runs in kiosk mode, screen is default FULL

    6. Were additional flags added when launching chromium and are you running as the root user or Weston user?

    ---> Weston.

    Fullscreen pvr-dump-sdk12.1-guilty-lockup.txt Download 
    # PVR DBG dump — Guilty Lockup on stock Processor SDK 12.1

Captured from an unmodified `tisdk-default-image-am62xx-evm-12.01.00.05.03`
running `chromium-bin … https://webglsamples.org` on HDMI-A-1 at 1600x900.
Lockup at t=14004s (3.89 h). Attach this to the TI E2E post.

    OS       Arago 2026.04, kernel 6.18.38-ti-00934-g8933766be737
    weston   15.0.0
    DDK      26.1.6967606
    firmware rgx.fw.33.15.11.3

```
[14004.078885] PVR_K:  521: ------------[ PVR DBG: START (High) ]------------
[14004.085941] PVR_K:  521: OS kernel info: Linux 6.18.38-ti-00934-g8933766be737-dirty #1 SMP PREEMPT Thu Jul  9 22:16:35 UTC 2026 aarch64
[14004.098239] PVR_K:  521: DDK info: Rogue_DDK_Linux+Linux_WS rogueddk 26.1@6967606 (release) am62_linux
[14004.109447] PVR_K:  521: Time now: 14004109398us
[14004.114281] PVR_K:  521: Services State: OK
[14004.118578] PVR_K:  521: Server Errors: 0
[14004.122730] PVR_K:  521: Connections Device ID:0(128) P690-V690-T690-weston, P1152-V1152-T1152-chromium-bin
[14004.132608] PVR_K:  521: ------[ Driver Info ]------
[14004.137684] PVR_K:  521: Comparison of UM/KM components: MATCHING
[14004.143899] PVR_K:  521: KM Arch: 64 Bit
[14004.147943] PVR_K:  521: Driver Mode: Native
[14004.152350] PVR_K:  521: UM Connected Clients: 64 Bit
[14004.157523] PVR_K:  521: UM info: 26.1 @  6967606 (release) build options: 0x80020810
[14004.165507] PVR_K:  521: KM info: 26.1 @  6967606 (release) build options: 0x00020810
[14004.173480] PVR_K:  521: Window system: lws-generic
[14004.178457] PVR_K:  521: Power lock status: Free
[14004.184486] PVR_K:  521: ------[ Server Thread Summary ]------
[14004.190703] PVR_K:  521:   pvr_defer_free : Running
[14004.195897] PVR_K:  521:     Number of deferred cleanup items: QUEUED: 00001  CONNECTION : 00001 MMU : 00000 OSMEM : 00000 PMR : 00000 RAMEM : 00000
[14004.209314] PVR_K:  521:     Number of deferred cleanup items dropped after retry limit reached : 0
[14004.218517] PVR_K:  521:   pvr_device_wdg : Running
[14004.224938] PVR_K:  521: Active Threads (UM/KM): 0 / 2
[14004.230338] PVR_K:  521: Suspended Threads: 0
[14004.234861] PVR_K:  521: Power lock status: Locked (Reader bias)
[14004.241035] PVR_K:  521: ------[ RGX Device ID:0 Start ]------
[14004.246981] PVR_K:  521: ------[ RGX Info ]------
[14004.251809] PVR_K:  521: Device Node (Info): 00000000324aef6e (000000007913a4ab)
[14004.259334] PVR_K:  521:     DevmemHistoryRecordStats - None
[14004.265147] PVR_K:  521: RGX BVNC: 33.15.11.3 (rogue)
[14004.270334] PVR_K:  521: RGX Device State: ACTIVE
[14004.275160] PVR_K:  521: RGX Power State: ON
[14004.279533] PVR_K:  521: RGX Health Status: OK, Reason: NONE
[14004.285281] PVR_K:  521: FW info: 26.1 @  6967606 (release) build options: 0x80020810
[14004.293218] PVR_K:  521: TRP: HW support - No
[14004.297697] PVR_K:  521: WGP: HW support - No
[14004.302181] PVR_K:  521: BIF0 - OK
[14004.305665] PVR_K:  521: BIF1 - OK
[14004.309140] PVR_K:  521: RGX HWR State 0x00000001: HWR OK;
[14004.314742] PVR_K:  521: RGX FW Power State: RGXFWIF_POW_ON (APM enabled: 272 ok, 0 denied, 0 non-idle, 11 retry, 0 other, 283 total. Latency: 100 ms)
[14004.328296] PVR_K:  521: RGX FW Forced Idle Timeout Count: 0
[14004.334019] PVR_K:  521: RGX DVFS: 0 frequency changes. Current frequency: 500.000 MHz (sampled at 14004234698290 ns). FW frequency: 500.000 MHz.
[14004.347121] PVR_K:  521: RGX FW OS 0 - State: active; Freelists: Ok; Priority: 0; Isolation group: 0;  Time Slice*: 0% (0ms); MTS on;
[14004.359273] PVR_K:  521: Number of HWR: GP(0/0+0), 2D(0/0+0), GEOM(1/1+0), 3D(0/0+0), CDM(0/0+0), FALSE(0,0,0,0,0)
[14004.369712] PVR_K:  521: DM 0 (GP)
[14004.373196] PVR_K:  521: DM 1 (HWRflags 0x00000000: working;)
[14004.379051] PVR_K:  521: DM 2 (HWRflags 0x00000000: working;)
[14004.384872] PVR_K:  521:   Recovery 1: PID = 1152 / chromium-bin, frame = 0, HWRTData = 0xD018EB80, EventStatus = 0x00000000, Guilty Lockup
[14004.397477] PVR_K:  521:               CRTimer = 0x000229D1651F, OSTimer = 14004.078640825, CyclesElapsed = 47705600
[14004.408081] PVR_K:  521:               PreResetTimeInCycles = 40448, HWResetTimeInCycles = 18944, FreelistReconTimeInCycles = 936448, TotalRecoveryTimeInCycles = 995840
[14004.423163] PVR_K:  521: DM 3 (HWRflags 0x00000000: working;)
[14004.428977] PVR_K:  521: DM 4 (HWRflags 0x00000000: working;)
[14004.434844] PVR_K:  521: RGX Kernel CCB WO:0x385 RO:0x385
[14004.440346] PVR_K:  521: RGX Firmware CCB WO:0xE RO:0xE
[14004.445625] PVR_K:  521: RGX Kernel CCB commands executed = 838533
[14004.451852] PVR_K:  521: RGX SLR: Forced UFO updates requested = 0
[14004.458070] PVR_K:  521: RGX Errors: WGP:0, TRP:0
[14004.462830] PVR_K:  521: Thread0: FW IRQ count = 1507852
[14004.468201] PVR_K:  521: Last sampled IRQ count in LISR = 1507852
[14004.474397] PVR_K:  521: FW System config flags = 0x000A0000 (Ctx switch options: Medium CSW profile; DM Overlap Off;)
[14004.485157] PVR_K:  521: FW OS config flags = 0x0000000F (Ctx switch: TDM; GEOM; 3D; CDM;)
[14004.493458] PVR_K:  521: ------[ RGX registers ]------
[14004.498607] PVR_K:  521: RGX Register Base Address (Linear):   0x00000000a330c90c
[14004.506089] PVR_K:  521: RGX Register Base Address (Physical): 0x0FD00000
[14004.512875] PVR_K:  521: CORE_ID__PBVNC                : 0x0021000F000B0003
[14004.519845] PVR_K:  521: DESIGNER_REV_FIELD1           : 0x00000000
[14004.526113] PVR_K:  521: DESIGNER_REV_FIELD2           : 0x00000000
[14004.532381] PVR_K:  521: CHANGESET_NUMBER              : 0x0000000000000000
[14004.539337] PVR_K:  521: CLK_CTRL                      : 0x002AAA002A22AAAA
[14004.546298] PVR_K:  521: CLK_STATUS                    : 0x0000000000600000
[14004.553256] PVR_K:  521: CLK_CTRL2                     : 0x0000000000000000
[14004.560214] PVR_K:  521: CLK_STATUS2                   : 0x0000000000000000
[14004.567194] PVR_K:  521: EVENT_STATUS                  : 0x00000000
[14004.573465] PVR_K:  521: TIMER                         : 0x0000000229E0244F
[14004.580425] PVR_K:  521: BIF_FAULT_BANK0_MMU_STATUS    : 0x00000000
[14004.586686] PVR_K:  521: BIF_FAULT_BANK0_REQ_STATUS    : 0x0000000000000000
[14004.593641] PVR_K:  521: BIF_FAULT_BANK1_MMU_STATUS    : 0x00000000
[14004.599910] PVR_K:  521: BIF_FAULT_BANK1_REQ_STATUS    : 0x0000000000000000
[14004.606865] PVR_K:  521: BIF_MMU_STATUS                : 0x00000000
[14004.613164] PVR_K:  521: BIF_MMU_ENTRY                 : 0x00000000
[14004.619441] PVR_K:  521: BIF_MMU_ENTRY_STATUS          : 0x0000000000000000
[14004.626399] PVR_K:  521: BIF_STATUS_MMU                : 0x00000000
[14004.632679] PVR_K:  521: BIF_READS_EXT_STATUS          : 0x00000000
[14004.638966] PVR_K:  521: BIF_READS_INT_STATUS          : 0x00000000
[14004.645248] PVR_K:  521: BIFPM_STATUS_MMU              : 0x00000000
[14004.651556] PVR_K:  521: BIFPM_READS_EXT_STATUS        : 0x00000000
[14004.657853] PVR_K:  521: BIFPM_READS_INT_STATUS        : 0x00000000
[14004.664131] PVR_K:  521: BIF_CAT_BASE_INDEX            : 0x0000000000000101
[14004.671088] PVR_K:  521: BIF_CAT_BASE0                 : 0x0000000000000000
[14004.678045] PVR_K:  521: BIF_CAT_BASE1                 : 0x00000000BB86D000
[14004.685008] PVR_K:  521: BIF_CAT_BASE2                 : 0x00000000A3AE2000
[14004.691964] PVR_K:  521: BIF_CAT_BASE3                 : 0x0000000000000000
[14004.698929] PVR_K:  521: BIF_CAT_BASE4                 : 0x0000000000000000
[14004.705890] PVR_K:  521: BIF_CAT_BASE5                 : 0x0000000000000000
[14004.712853] PVR_K:  521: BIF_CAT_BASE6                 : 0x0000000000000000
[14004.719815] PVR_K:  521: BIF_CAT_BASE7                 : 0x0000000000000000
[14004.726778] PVR_K:  521: BIF_CTRL_INVAL                : 0x00000000
[14004.733039] PVR_K:  521: BIF_CTRL                      : 0x00000000
[14004.739318] PVR_K:  521: BIF_PM_CAT_BASE_VCE0          : 0x00000000AC38D001
[14004.746278] PVR_K:  521: BIF_PM_CAT_BASE_TE0           : 0x00000000AC391001
[14004.753235] PVR_K:  521: BIF_PM_CAT_BASE_ALIST0        : 0x00000000AC384001
[14004.760204] PVR_K:  521: BIF_PM_CAT_BASE_VCE1          : 0x00000000C7F32001
[14004.767183] PVR_K:  521: BIF_PM_CAT_BASE_TE1           : 0x00000000C7F3D001
[14004.774157] PVR_K:  521: BIF_PM_CAT_BASE_ALIST1        : 0x00000000C7F42001
[14004.781120] PVR_K:  521: PERF_TA_PHASE                 : 0x0000000A
[14004.787386] PVR_K:  521: PERF_TA_CYCLE                 : 0x00ED5A9A
[14004.793656] PVR_K:  521: PERF_3D_PHASE                 : 0x00000008
[14004.799925] PVR_K:  521: PERF_3D_CYCLE                 : 0x03E91FEC
[14004.806195] PVR_K:  521: PERF_TA_OR_3D_CYCLE           : 0x04D67A86
[14004.812458] PVR_K:  521: PERF_TA_AND_3D_CYCLE          : 0x00000000
[14004.818721] PVR_K:  521: PERF_COMPUTE_PHASE            : 0x00000000
[14004.824984] PVR_K:  521: PERF_COMPUTE_CYCLE            : 0x00000000
[14004.831244] PVR_K:  521: PM_PARTIAL_RENDER_ENABLE      : 0x00000000
[14004.837505] PVR_K:  521: ISP_RENDER                    : 0x00000000
[14004.843769] PVR_K:  521: TLA_STATUS                    : 0x0000000000000000
[14004.850732] PVR_K:  521: MCU_FENCE                     : 0x0000018000000000
[14004.857688] PVR_K:  521: VDM_CONTEXT_STORE_STATUS      : 0x00000001
[14004.863967] PVR_K:  521: VDM_CONTEXT_STORE_TASK0       : 0x000142C522001004
[14004.870928] PVR_K:  521: VDM_CONTEXT_STORE_TASK1       : 0x0000000000014300
[14004.877906] PVR_K:  521: VDM_CONTEXT_STORE_TASK2       : 0x0001008000000420
[14004.884872] PVR_K:  521: VDM_CONTEXT_RESUME_TASK0      : 0x000143C520041004
[14004.891836] PVR_K:  521: VDM_CONTEXT_RESUME_TASK1      : 0x0000000000014400
[14004.898805] PVR_K:  521: VDM_CONTEXT_RESUME_TASK2      : 0x0001018000000420
[14004.905770] PVR_K:  521: ISP_CTL                       : 0x0003000F
[14004.912044] PVR_K:  521: ISP_STATUS                    : 0x00000003
[14004.918308] PVR_K:  521: CDM_CONTEXT_STORE_STATUS      : 0x00000000
[14004.924570] PVR_K:  521: CDM_CONTEXT_PDS0              : 0x0000000000000000
[14004.931528] PVR_K:  521: CDM_CONTEXT_PDS1              : 0x0000000000000000
[14004.938482] PVR_K:  521: CDM_TERMINATE_PDS             : 0x0000000000000000
[14004.945437] PVR_K:  521: CDM_TERMINATE_PDS1            : 0x0000000000000000
[14004.952397] PVR_K:  521: CDM_CONTEXT_LOAD_PDS0         : 0x0000000000000000
[14004.959357] PVR_K:  521: CDM_CONTEXT_LOAD_PDS1         : 0x0000000000000000
[14004.966312] PVR_K:  521: MTS_INTCTX                    : 0x00000000
[14004.972580] PVR_K:  521: MTS_BGCTX                     : 0x00000000
[14004.978843] PVR_K:  521: MTS_BGCTX_COUNTED_SCHEDULE    : 0x00000000
[14004.985120] PVR_K:  521: MTS_SCHEDULE                  : 0x00000000
[14004.991387] PVR_K:  521: MTS_GPU_INT_STATUS            : 0x00004110
[14004.997666] PVR_K:  521: SLC_IDLE                      : 0x000003FF
[14005.003934] PVR_K:  521: SLC_STATUS0                   : 0x00000000
[14005.010200] PVR_K:  521: SLC_STATUS1                   : 0x0000000000000000
[14005.017159] PVR_K:  521: SLC_STATUS2                   : 0x0000000000000000
[14005.024115] PVR_K:  521: SLC_CTRL_BYPASS               : 0x00003D1F01000000
[14005.031070] PVR_K:  521: SLC_CTRL_MISC                 : 0x0000000001200003
[14005.038025] PVR_K:  521: SIDEKICK_IDLE                 : 0x0000007E
[14005.044288] PVR_K:  521: MIPS_ADDR_REMAP1_CONFIG1      : 0x1FC00001
[14005.050573] PVR_K:  521: MIPS_ADDR_REMAP1_CONFIG2      : 0x0000000098E5100C
[14005.057549] PVR_K:  521: MIPS_ADDR_REMAP2_CONFIG1      : 0x1FC01001
[14005.063816] PVR_K:  521: MIPS_ADDR_REMAP2_CONFIG2      : 0x0000000098EB700C
[14005.070772] PVR_K:  521: MIPS_ADDR_REMAP3_CONFIG1      : 0x1FC02001
[14005.077062] PVR_K:  521: MIPS_ADDR_REMAP3_CONFIG2      : 0x0000000098E5000C
[14005.084028] PVR_K:  521: MIPS_ADDR_REMAP4_CONFIG1      : 0x1FC00000
[14005.090296] PVR_K:  521: MIPS_ADDR_REMAP4_CONFIG2      : 0x000000000000000C
[14005.097252] PVR_K:  521: MIPS_ADDR_REMAP5_CONFIG1      : 0x00000001
[14005.103517] PVR_K:  521: MIPS_ADDR_REMAP5_CONFIG2      : 0x0000000098E5100C
[14005.110513] PVR_K:  521: MIPS_WRAPPER_CONFIG           : 0x000000000001CF80
[14005.117480] PVR_K:  521: MIPS_EXCEPTION_STATUS         : 0x00000020
[14005.124860] PVR_K:  521: ---- [ MIPS internal state ] ----
[14005.130401] PVR_K:  521: PC                            : 0xFFFFFFFF
[14005.136695] PVR_K:  521: STATUS_REGISTER               : 0x00000000
[14005.142974] PVR_K:  521: CAUSE_REGISTER                : 0x00000000
[14005.149242] PVR_K:  521:     Interrupt uses general interrupt vector
[14005.155592] PVR_K:  521: BAD_REGISTER                  : 0x00000000
[14005.161879] PVR_K:  521: EPC                           : 0xFFFFFFFF
[14005.168153] PVR_K:  521: SP                            : 0x00000000
[14005.174418] PVR_K:  521: BAD_INSTRUCTION               : 0x00000000
[14005.180699] PVR_K:  521: TLB                           :
[14005.186013] PVR_K:  521:  0) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.195265] PVR_K:  521:  1) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.204510] PVR_K:  521:  2) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.213750] PVR_K:  521:  3) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.222984] PVR_K:  521:  4) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.232201] PVR_K:  521:  5) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.241422] PVR_K:  521:  6) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.250651] PVR_K:  521:  7) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.259872] PVR_K:  521:  8) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.269086] PVR_K:  521:  9) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.278301] PVR_K:  521: 10) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.287516] PVR_K:  521: 11) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.296729] PVR_K:  521: 12) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.305941] PVR_K:  521: 13) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.315151] PVR_K:  521: 14) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.324362] PVR_K:  521: 15) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[14005.333601] PVR_K:  521: --------------------------------
[14005.339006] PVR_K:  521: ------[ RGX FW Trace Info ]------
[14005.344502] PVR_K:  521: Debug log type: none
[14005.348863] PVR_K:  521: RGX FW thread 0: Trace buffer not yet allocated
[14005.355563] PVR_K:  521: ------[ Full CCB Status ]------
[14005.360877] PVR_K:  521: FWCtx 0xD00E3380 (TQ_3D-P690-T690-weston)
[14005.367060] PVR_K:  521:   `--<Empty>
[14005.370732] PVR_K:  521: FWCtx 0xD00E3B40 (TQ_3D-P1152-T1152-chromium-bi)
[14005.377519] PVR_K:  521:   `--<Empty>
[14005.381201] PVR_K:  521: FWCtx 0xD00E3040 (TA-P690-T690-weston)
[14005.387118] PVR_K:  521:   `--<Empty>
[14005.390782] PVR_K:  521: FWCtx 0xD00E3190 (3D-P690-T690-weston)
[14005.396700] PVR_K:  521:   `--<Empty>
[14005.400358] PVR_K:  521: FWCtx 0xD00E3800 (TA-P1152-T1152-chromium-bin)
[14005.406964] PVR_K:  521:   `--<Empty>
[14005.410634] PVR_K:  521: FWCtx 0xD00E3950 (3D-P1152-T1152-chromium-bin)
[14005.417242] PVR_K:  521:   `--<Empty>
[14005.420904] PVR_K:  521: ------[ RGX Device ID:0 End ]------
[14005.426603] PVR_K:  521: ------[ Device ID: 128 - Phys Heaps ]------
[14005.433046] PVR_K:  521: 0x0000000037ab8504 -> PdMs: SYSMEM, Type: UMA, default, Usage Flags: 0x00000004 (GPU_LOCAL), Refs: 12, Free Size: 146919424B, Total Size: 2023616512B, Spas Region: 0000000000000000
[14005.451380] PVR_K:  521: PMR Zombie Count: 0, PMR Zombie Count In Cleanup: 318
[14005.458691] PVR_K:  521: PMR Live Count: 250
[14005.463050] PVR_K:  521: ------[ System Summary Device ID:0 ]------
[14005.469372] PVR_K:  521: Device System Power State: ON
[14005.474573] PVR_K:  521: MaxHWTOut: 500000us, WtTryCt: 10000, WDGTOut(on,off): (10000ms,3600000ms)
[14005.483565] PVR_K:  521: ------[ AppHint Settings ]------
[14005.488976] PVR_K:  521:   Build Vars
[14005.492637] PVR_K:  521:     EnableTrustedDeviceAceConfig: N
[14005.498312] PVR_K:  521:     CleanupThreadPriority: 0x00000005
[14005.504158] PVR_K:  521:     WatchdogThreadPriority: 0x00000000
[14005.510082] PVR_K:  521:     HWPerfClientBufferSize: 0x000c0000
[14005.516003] PVR_K:  521:     DevmemHistoryBufSizeLog2: 0x0000000b
[14005.522092] PVR_K:  521:     DevmemHistoryMaxEntries: 0x00002710
[14005.528100] PVR_K:  521:   Module Params
[14005.532035] PVR_K:  521:     DisableDMOverlap: Y
[14005.536669] PVR_K:  521:   Debug Info Params
[14005.540945] PVR_K:  521:     CacheOpConfig: 0x0000000c
[14005.546085] PVR_K:  521:     CacheOpUMKMThresholdSize: 0xffffffff
[14005.552225] PVR_K:  521:   Debug Info Params Device ID: 0
[14005.557685] PVR_K:  521:     none
[14005.561036] PVR_K:  521: ------[ Active Sync Checkpoints ]------
[14005.567054] PVR_K:  521: (SyncCP Counts: InUse:10 Max:103)
[14005.572528] PVR_K:  521: (SyncCP Mirroring Counts: InUse:0 Max:0)
[14005.578607] PVR_K:  521: (SyncCP Defer Free List Count: Current:0 Max:2)
[14005.585315] sw: RM_SWTimeline-weston-690 @0 cur=0
[14005.590024] ------[ Native Fence Sync: timelines ]------
[14005.595334] foreign_sync: @0 ctx=2 refs=1
[14005.599342] rogue-ta3d: @167849 ctx=4 refs=4
[14005.603606]  @1882: (++) refs=1 fwaddr=0xc00df288 enqueue=1 status=Signalled 1882-update fence
[14005.612206]  @167847: (++) refs=1 fwaddr=0xc00df024 enqueue=1 status=Signalled 167847-update fence
[14005.621151]  @167848: (++) refs=2 fwaddr=0xc00df3d8 enqueue=1 status=Signalled 167848-update fence
[14005.630132] rogue-tq3d: @0 ctx=5 refs=1
[14005.634016] QE-weston-690: @72 ctx=6 refs=1
[14005.638266] V3-weston-690: @167849 ctx=7 refs=3
[14005.642809]  @167847: (++) refs=1 fwaddr=0xc00df270 enqueue=1 status=Signalled 167847-es3_DoKickTA_167920
[14005.652362]  @167848: (++) refs=3 fwaddr=0xc00df294 enqueue=1 status=Signalled 167848-es3_DoKickTA_167921
[14005.661935] P3-weston-690: @167849 ctx=8 refs=3
[14005.666466]  @167847: (++) refs=1 fwaddr=0xc00df1d4 enqueue=1 status=Signalled 167847-es3_DoKick3D_167920
[14005.676013]  @167848: (++) refs=3 fwaddr=0xc00df1c8 enqueue=1 status=Signalled 167848-es3_DoKick3D_167921
[14005.685580] rogue-ta3d: @331850 ctx=1129 refs=2
[14005.690115]  @331848: (++) refs=1 fwaddr=0xc00df27c enqueue=3 status=Signalled 331848-update fence
[14005.699055]  @331849: (++) refs=1 fwaddr=0xc00df4f8 enqueue=1 status=Signalled 331849-update fence
[14005.708010] rogue-tq3d: @0 ctx=1130 refs=0
[14005.712116] V3-AAAAAAAAAI-chromium-bin-1152: @502212 ctx=1131 refs=0
[14005.718467] P3-AAAAAAAAAI-chromium-bin-1152: @502109 ctx=1132 refs=1
[14005.724808]  @502106: (++) refs=1 fwaddr=0xc00df5ac enqueue=2 status=Signalled 502106-es3_DoKick3D_502641
[14005.734366] QE-AAAAAAAAAI-chromium-bin-1152: @178 ctx=1133 refs=0
[14005.740480] PVR_K:  521: ------------[ PVR DBG: END ]------------

--- kernel WARNING and stack trace that follows the dump ---

[14005.740480] PVR_K:  521: ------------[ PVR DBG: END ]------------
[14005.746806] ------------[ cut here ]------------
[14005.751446] WARNING: CPU: 2 PID: 521 at services/server/common/pvr_notifier.c:650 PVRSRVDebugRequest+0x5f4/0x700 [pvrsrvkm]
[14005.762933] Modules linked in: xt_conntrack xt_MASQUERADE bridge stp llc ip6table_nat ip6table_filter ip6_tables iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype iptable_filter ip_tables x_tables nf_tables overlay cfg80211 bluetooth 8250_pruss rpmsg_ctrl rpmsg_char snd_soc_hdmi_codec pru_rproc irq_pruss_intc ecdh_generic ecc rfkill pvrsrvkm(O) powervr snd_soc_simple_card snd_soc_simple_card_utils pwrseq_core drm_shmem_helper drm_gpuvm drm_exec gpu_sched display_connector rtc_ti_k3 rti_wdt ti_k3_r5_remoteproc k3_j72xx_bandgap tidss drm_client_lib snd_soc_davinci_mcasp snd_soc_ti_udma snd_soc_tlv320aic3x_i2c drm_dma_helper snd_soc_ti_edma drm_display_helper snd_soc_tlv320aic3x snd_soc_ti_sdma snd_soc_core ti_k3_m4_remoteproc snd_pcm_dmaengine ti_k3_common snd_pcm cec snd_timer sa2ul tps6598x snd sii902x sha1 ramoops mux_ti_k3_event soundcore typec pruss at24 reed_solomon drm_kms_helper omap_mailbox pwm_tiehrpwm pwm_tiecap omap_hwspinlock cryptodev(O) drm fuse drm_panel_orientation_quirks
[14005.763252]  backlight nfnetlink ipv6
[14005.856210] CPU: 2 UID: 0 PID: 521 Comm: pvr_device_wdg Tainted: G           O        6.18.38-ti-00934-g8933766be737-dirty #1 PREEMPT 
[14005.868281] Tainted: [O]=OOT_MODULE
[14005.871757] Hardware name: Texas Instruments AM625 SK (DT)
[14005.877236] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[14005.884183] pc : PVRSRVDebugRequest+0x5f4/0x700 [pvrsrvkm]
[14005.889974] lr : PVRSRVDebugRequest+0x5f4/0x700 [pvrsrvkm]
[14005.895604] sp : ffff800083c23c80
[14005.898907] x29: ffff800083c23c80 x28: ffff800079a690e0 x27: ffff00000649b908
[14005.906040] x26: ffff00000649b808 x25: ffff00000c87f608 x24: ffff00000c87f720
[14005.913165] x23: ffff00000649b920 x22: ffff00000c87f720 x21: 0000000000000002
[14005.920290] x20: 0000000000000000 x19: 0000000000000000 x18: 000000000000000a
[14005.927414] x17: 6c6c616e6769533d x16: 7375746174732032 x15: 0000000000000000
[14005.934540] x14: 0000000000000000 x13: 2d2d2d2d2d2d2d2d x12: 2d2d2d2d5d20444e
[14005.941664] x11: 0000000000000058 x10: 0000000000000018 x9 : ffff800081381f10
[14005.948788] x8 : 0000000000057fa8 x7 : 00000000000002df x6 : 0000000000000001
[14005.955914] x5 : 0000000000000000 x4 : 0000000000000001 x3 : 0000000000000000
[14005.963038] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000004ffe600
[14005.970166] Call trace:
[14005.972604]  PVRSRVDebugRequest+0x5f4/0x700 [pvrsrvkm] (P)
[14005.978308]  DevicesWatchdogThread_ForEachVaCb+0x130/0x188 [pvrsrvkm]
[14005.984887]  List_PVRSRV_DEVICE_NODE_ForEach_va+0x88/0xcc [pvrsrvkm]
[14005.991394]  DevicesWatchdogThread+0xbc/0x220 [pvrsrvkm]
[14005.996846]  OSThreadRun+0x24/0x60 [pvrsrvkm]
[14006.001337]  kthread+0x12c/0x204
[14006.004567]  ret_from_fork+0x10/0x20
[14006.008142] ---[ end trace 0000000000000000 ]---
```

  • 0 in reply to Siddhesh Hande
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    Thank you for the logs. I was able to replicate the issue on my end as well and have filed a ticket with our GPU provider, IMG, for further investigation. I will ping this thread when further progress has been made in finding the cause of this issue. 

    Regards,

    Shriya

  • 0 in reply to Shriya Surti
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    We are still waiting for IMG to replicate issue on their end. To help move along the debugging efforts, would you be able to replicate the issue again and enable the following for the pvr dbg dump as well? 

    #!/bin/sh

pvrdebug -loggroups main,mts,cleanup,csw,bif,pm,rtd,spm,pow,hwr
echo "Y" > /sys/kernel/debug/pvr/gpu00/apphint/AssertOnHWRTrigger

    This will bump up logging and lock GPU when a recovery occurs. 

    Regards,

    Shriya

  • 0 in reply to Shriya Surti
    Prodigy 65 points

    Sure, 

    I will add this file on both EVM and our custom HW as well and run

  • 0 in reply to Siddhesh Hande
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    Thank you, please keep us updated. Was there anything you have recently tried that can help speed up the replication process?

    Regards,

    Shriya

  • 0 in reply to Shriya Surti
    Prodigy 65 points

    Hi Shriya,

    The issue has been reproduced three times so far on the EVM.

    Starting yesterday, I began collecting PVR dumps. I have one PVR log dump available, which I am attaching here for further analysis.

    Setup

    • Platform: AM62x EVM

    • Display: HDMI connected to a standard desktop monitor

    • Resolution: 1920 × 1080

    • SDK: As mentioned below

    • Website: As mentioned below



    https://dr-download.ti.com/software-development/software-development-kit-(sdk)/MD-PvdSyIiioq/12.01.00.05.03/tisdk-default-image-am62xx-evm-12.01.00.05.03.rootfs.wic.xz

    below web graphics website used for this, number of objects keep more e.g above 10k or 20K.
    Webgl website : https://webglsamples.org/aquarium/aquarium.html.


    Fullscreen pvr_log_dump.txt Download 
    [Sep29 17:50] PVR_K:(Error):  1363: PVRSRVWaitForValueKM() failed (PVRSRV_ERROR_TIMEOUT) in RGXWaitForKCCBSlotUpdate() [3995]
[  +0.011337] PVR_K:(Error):  1363: GetGpuUtilStats() failed (PVRSRV_ERROR_TIMEOUT) in _UtilStatsDIShow() [982]
[  +0.010016] PVR_K:(Error):  1363: _Read: failed to read from file, pfnRead() returned -5 [280]
[  +0.487194] PVR_K:(Error):   604: PVRSRVWaitForValueKM() failed (PVRSRV_ERROR_TIMEOUT) in RGXWaitForKCCBSlotUpdate() [3995]
[  +0.012995] PVR_K:  604: RGXDestroyHWRTDataSet: Unexpected error from RGXFWRequestHWRTDataCleanUp(PVRSRV_ERROR_INVALID_PARAMS)
[  +0.490284] PVR_K:(Error):  1363: PVRSRVWaitForValueKM() failed (PVRSRV_ERROR_TIMEOUT) in RGXWaitForKCCBSlotUpdate() [3995]
[  +0.011260] PVR_K:(Error):  1363: GetGpuUtilStats() failed (PVRSRV_ERROR_TIMEOUT) in _UtilStatsDIShow() [982]
[  +0.010083] PVR_K:(Error):  1363: _Read: failed to read from file, pfnRead() returned -5 [280]
[  +0.122689] PVR_K:(Error):   604: PVRSRVWaitForValueKM() failed (PVRSRV_ERROR_TIMEOUT) in RGXWaitForKCCBSlotUpdate() [3995]
[  +0.011275] PVR_K:  604: RGXDestroyHWRTDataSet: Unexpected error from RGXFWRequestHWRTDataCleanUp(PVRSRV_ERROR_INVALID_PARAMS)
[  +0.214975] PVR_K:(Error):  1363: Device in bad state in _UtilStatsDIShow() [970]
[  +0.007872] PVR_K:(Error):  1363: _Read: failed to read from file, pfnRead() returned -22 [280]
[  +0.065833] PVR_K:(Error):   605: DevicesWatchdogThread: Device status not OK!!! [1096]
[  +0.008086] PVR_K:  605: ------------[ PVR DBG: START (High) ]------------
[  +0.006922] PVR_K:  605: OS kernel info: Linux 6.18.38-ti-00934-g8933766be737-dirty #1 SMP PREEMPT Thu Jul  9 22:16:35 UTC 2026 aarch64
[  +0.012181] PVR_K:  605: DDK info: Rogue_DDK_Linux+Linux_WS rogueddk 26.1@6967606 (release) am62_linux
[  +0.009326] PVR_K:  605: Time now: 15084100570us
[  +0.004619] PVR_K:  605: Services State: OK
[  +0.004200] PVR_K:  605: Server Errors: 11
[  +0.004118] PVR_K:  605: Connections Device ID:0(128) P743-V743-T743-weston, P1363-V1363-T1363-ti-apps-launche, P16773-V16773-T16773-chromium-bin
[  +0.013056] PVR_K:  605: ------[ Driver Info ]------
[  +0.004973] PVR_K:  605: Comparison of UM/KM components: MATCHING
[  +0.006102] PVR_K:  605: KM Arch: 64 Bit
[  +0.003940] PVR_K:  605: Driver Mode: Native
[  +0.004278] PVR_K:  605: UM Connected Clients: 64 Bit
[  +0.005060] PVR_K:  605: UM info: 26.1 @  6967606 (release) build options: 0x80020810
[  +0.007829] PVR_K:  605: KM info: 26.1 @  6967606 (release) build options: 0x00020810
[  +0.007861] PVR_K:  605: Window system: lws-generic
[  +0.004915] PVR_K:  605: Power lock status: Free
[  +0.004632] PVR_K:  605: ------[ Server Thread Summary ]------
[  +0.005846] PVR_K:  605:   pvr_defer_free : Running
[  +0.004950] PVR_K:  605:     Number of deferred cleanup items: QUEUED: 00007  CONNECTION : 00002 MMU : 00002 OSMEM : 00000 PMR : 00003 RAMEM : 00000
[  +0.013313] PVR_K:  605:     Number of deferred cleanup items dropped after retry limit reached : 0
[  +0.009048] PVR_K:  605:   pvr_device_wdg : Running
[  +0.004915] PVR_K:  605: Active Threads (UM/KM): 2 / 2
[  +0.005161] PVR_K:  605: Suspended Threads: 0
[  +0.004369] PVR_K:  605: Power lock status: Locked (Reader bias)
[  +0.006025] PVR_K:  605: ------[ RGX Device ID:0 Start ]------
[  +0.005841] PVR_K:  605: ------[ RGX Info ]------
[  +0.004733] PVR_K:  605: Device Node (Info): 0000000012f58a4f (0000000072eddc18)
[  +0.007409] PVR_K:  605:     DevmemHistoryRecordStats - None
[  +0.005668] PVR_K:  605: RGX BVNC: 33.15.11.3 (rogue)
[  +0.005064] PVR_K:  605: RGX Device State: ACTIVE
[  +0.004712] PVR_K:  605: RGX Power State: ON
[  +0.004280] PVR_K:  605: RGX Health Status: DEAD, Reason: ASSERTED
[  +0.006193] PVR_K:  605: FW info: 26.1 @  6967606 (release) build options: 0x80020810
[  +0.007840] PVR_K:  605: TRP: HW support - No
[  +0.004378] PVR_K:  605: WGP: HW support - No
[  +0.004375] PVR_K:  605: BIF0 - OK
[  +0.003413] PVR_K:  605: BIF1 - OK
[  +0.003430] PVR_K:  605: RGX HWR State 0x00000020: DM stalling;
[  +0.005939] PVR_K:  605: RGX FW Power State: RGXFWIF_POW_ON (APM enabled: 160 ok, 0 denied, 0 non-idle, 1 retry, 0 other, 161 total. Latency: 100 ms)
[  +0.013422] PVR_K:  605: RGX FW Forced Idle Timeout Count: 0
[  +0.005679] PVR_K:  605: RGX DVFS: 0 frequency changes. Current frequency: 500.000 MHz (sampled at 15084064034555 ns). FW frequency: 500.000 MHz.
[  +0.013042] PVR_K:  605: RGX FW OS 0 - State: active; Freelists: Ok; Priority: 0; Isolation group: 0;  Time Slice*: 0% (0ms); MTS on;
[  +0.012044] PVR_K:  605: FW-T0 Assert: !BITMASK_HAS(gsRGXFWCtl.ui32FWConfigFlags, RGXFWIF_INICFG_ASSERT_ON_HWR_TRIGGER) (firmware/rogue/rgxfw_hwr.c:3038)
[  +0.013784] PVR_K:  605: Number of HWR: GP(0/0+0), 2D(0/0+0), GEOM(0/1+0), 3D(0/0+0), CDM(0/0+0), FALSE(0,0,0,0,0)
[  +0.010363] PVR_K:  605: DM 0 (GP)
[  +0.003427] PVR_K:  605: DM 1 (HWRflags 0x00000000: working;)
[  +0.005773] PVR_K:  605: DM 2 (HWRflags 0x00000020: guilty lockup;)
[  +0.006288] PVR_K:  605:   Recovery 1: PID = 1618 / chromium-bin, frame = 0, HWRTData = 0xD0219980, EventStatus = 0x00000000, Guilty Lockup
[  +0.012528] PVR_K:  605:               CRTimer = 0x0006C3D3C276, OSTimer = 15081.481295328, CyclesElapsed = 55485440
[  +0.010555] PVR_K:  605: DM 3 (HWRflags 0x00000000: working;)
[  +0.005753] PVR_K:  605: DM 4 (HWRflags 0x00000000: working;)
[  +0.005749] PVR_K:(Error):   604: PVRSRVWaitForValueKM() failed (PVRSRV_ERROR_TIMEOUT) in RGXWaitForKCCBSlotUpdate() [3995]
[  +0.011119] PVR_K:  605: RGX Kernel CCB WO:0x367 RO:0x317
[  +0.005394] PVR_K:  604: RGXDestroyHWRTDataSet: Unexpected error from RGXFWRequestHWRTDataCleanUp(PVRSRV_ERROR_INVALID_PARAMS)
[  +0.011392] PVR_K:  605: RGX Firmware CCB WO:0x1E RO:0x1E
[  +0.005452] PVR_K:  605: RGX Kernel CCB commands executed = 813847
[  +0.006203] PVR_K:  605: RGX SLR: Forced UFO updates requested = 0
[  +0.006191] PVR_K:  605: RGX Errors: WGP:0, TRP:0
[  +0.004717] PVR_K:  605: Thread0: FW IRQ count = 1448391
[  +0.005319] PVR_K:  605: Last sampled IRQ count in LISR = 1448391
[  +0.006100] PVR_K:  605: FW System config flags = 0x001A0000 (Ctx switch options: Medium CSW profile; DM Overlap Off; Assert on HWR;)
[  +0.012011] PVR_K:  605: FW OS config flags = 0x0000000F (Ctx switch: TDM; GEOM; 3D; CDM;)
[  +0.008283] PVR_K:  605: ------[ RGX registers ]------
[  +0.005160] PVR_K:  605: RGX Register Base Address (Linear):   0x00000000a85dc308
[  +0.007494] PVR_K:  605: RGX Register Base Address (Physical): 0x0FD00000
[  +0.006802] PVR_K:  605: CORE_ID__PBVNC                : 0x0021000F000B0003
[  +0.006964] PVR_K:  605: DESIGNER_REV_FIELD1           : 0x00000000
[  +0.006276] PVR_K:  605: DESIGNER_REV_FIELD2           : 0x00000000
[  +0.006278] PVR_K:  605: CHANGESET_NUMBER              : 0x0000000000000000
[  +0.006981] PVR_K:  605: CLK_CTRL                      : 0x002AAA002A22AAAA
[  +0.006977] PVR_K:  605: CLK_STATUS                    : 0x000000000170206C
[  +0.007126] PVR_K:  605: CLK_CTRL2                     : 0x0000000000000000
[  +0.006985] PVR_K:  605: CLK_STATUS2                   : 0x0000000000000000
[  +0.006999] PVR_K:  605: EVENT_STATUS                  : 0x00000000
[  +0.006279] PVR_K:  605: TIMER                         : 0x80000006C43039F5
[  +0.006974] PVR_K:  605: BIF_FAULT_BANK0_MMU_STATUS    : 0x00000000
[  +0.006275] PVR_K:  605: BIF_FAULT_BANK0_REQ_STATUS    : 0x0000000000000000
[  +0.007161] PVR_K:  605: BIF_FAULT_BANK1_MMU_STATUS    : 0x00000000
[  +0.006278] PVR_K:  605: BIF_FAULT_BANK1_REQ_STATUS    : 0x0000000000000000
[  +0.006970] PVR_K:  605: BIF_MMU_STATUS                : 0x00000000
[  +0.006268] PVR_K:  605: BIF_MMU_ENTRY                 : 0x00000000
[  +0.006269] PVR_K:  605: BIF_MMU_ENTRY_STATUS          : 0x0000000000000000
[  +0.006969] PVR_K:  605: BIF_STATUS_MMU                : 0x00000000
[  +0.006294] PVR_K:  605: BIF_READS_EXT_STATUS          : 0x00000000
[  +0.006275] PVR_K:  605: BIF_READS_INT_STATUS          : 0x00000000
[  +0.006291] PVR_K:  605: BIFPM_STATUS_MMU              : 0x00000000
[  +0.006295] PVR_K:  605: BIFPM_READS_EXT_STATUS        : 0x00000000
[  +0.006397] PVR_K:  605: BIFPM_READS_INT_STATUS        : 0x00000000
[  +0.006283] PVR_K:  605: BIF_CAT_BASE_INDEX            : 0x0000000000000707
[  +0.006981] PVR_K:  605: BIF_CAT_BASE0                 : 0x0000000000000000
[  +0.006965] PVR_K:  605: BIF_CAT_BASE1                 : 0x0000000000000000
[  +0.006970] PVR_K:  605: BIF_CAT_BASE2                 : 0x0000000000000000
[  +0.006982] PVR_K:  605: BIF_CAT_BASE3                 : 0x0000000000000000
[  +0.006990] PVR_K:  605: BIF_CAT_BASE4                 : 0x0000000000000000
[  +0.006969] PVR_K:  605: BIF_CAT_BASE5                 : 0x00000000BABE5000
[  +0.006970] PVR_K:  605: BIF_CAT_BASE6                 : 0x00000000A99F1000
[  +0.006968] PVR_K:  605: BIF_CAT_BASE7                 : 0x00000000C7BDC000
[  +0.006980] PVR_K:  605: BIF_CTRL_INVAL                : 0x00000000
[  +0.006279] PVR_K:  605: BIF_CTRL                      : 0x00000000
[  +0.006280] PVR_K:  605: BIF_PM_CAT_BASE_VCE0          : 0x00000000B1553001
[  +0.007150] PVR_K:  605: BIF_PM_CAT_BASE_TE0           : 0x00000000B14A0001
[  +0.006995] PVR_K:  605: BIF_PM_CAT_BASE_ALIST0        : 0x00000000A9A09001
[  +0.006974] PVR_K:  605: BIF_PM_CAT_BASE_VCE1          : 0x00000000E15AB001
[  +0.006981] PVR_K:  605: BIF_PM_CAT_BASE_TE1           : 0x00000000E121A001
[  +0.006981] PVR_K:  605: BIF_PM_CAT_BASE_ALIST1        : 0x00000000E121F001
[  +0.006992] PVR_K:  605: PERF_TA_PHASE                 : 0x000C0588
[  +0.006269] PVR_K:  605: PERF_TA_CYCLE                 : 0xA2F12E46
[  +0.006277] PVR_K:  605: PERF_3D_PHASE                 : 0x0009D112
[  +0.006271] PVR_K:  605: PERF_3D_CYCLE                 : 0x0A1CC0D2
[  +0.006273] PVR_K:  605: PERF_TA_OR_3D_CYCLE           : 0xAD9D8614
[  +0.006640] PVR_K:  605: PERF_TA_AND_3D_CYCLE          : 0x00000000
[  +0.006335] PVR_K:  605: PERF_COMPUTE_PHASE            : 0x00000000
[  +0.006348] PVR_K:  605: PERF_COMPUTE_CYCLE            : 0x00000000
[  +0.006578] PVR_K:  605: PM_PARTIAL_RENDER_ENABLE      : 0x00000000
[  +0.006327] PVR_K:  605: ISP_RENDER                    : 0x00000000
[  +0.006312] PVR_K:  605: TLA_STATUS                    : 0x0000000000000000
[  +0.006965] PVR_K:  605: MCU_FENCE                     : 0x0000008000000000
[  +0.006985] PVR_K:  605: VDM_CONTEXT_STORE_STATUS      : 0x00000001
[  +0.006404] PVR_K:  605: VDM_CONTEXT_STORE_TASK0       : 0x000142C522001004
[  +0.006986] PVR_K:  605: VDM_CONTEXT_STORE_TASK1       : 0x0000000000014300
[  +0.006974] PVR_K:  605: VDM_CONTEXT_STORE_TASK2       : 0x0001008000000420
[  +0.006980] PVR_K:  605: VDM_CONTEXT_RESUME_TASK0      : 0x000143C520041004
[  +0.006990] PVR_K:  605: VDM_CONTEXT_RESUME_TASK1      : 0x0000000000014400
[  +0.006995] PVR_K:  605: VDM_CONTEXT_RESUME_TASK2      : 0x0001018000000420
[  +0.006980] PVR_K:  605: ISP_CTL                       : 0x0001000F
[  +0.006290] PVR_K:  605: ISP_STATUS                    : 0x00000003
[  +0.006280] PVR_K:  605: CDM_CONTEXT_STORE_STATUS      : 0x00000000
[  +0.006278] PVR_K:  605: CDM_CONTEXT_PDS0              : 0x0000000000000000
[  +0.006973] PVR_K:  605: CDM_CONTEXT_PDS1              : 0x0000000000000000
[  +0.006982] PVR_K:  605: CDM_TERMINATE_PDS             : 0x0000000000000000
[  +0.006979] PVR_K:  605: CDM_TERMINATE_PDS1            : 0x0000000000000000
[  +0.006981] PVR_K:  605: CDM_CONTEXT_LOAD_PDS0         : 0x0000000000000000
[  +0.006973] PVR_K:  605: CDM_CONTEXT_LOAD_PDS1         : 0x0000000000000000
[  +0.007030] PVR_K:  605: MTS_INTCTX                    : 0x00000000
[  +0.006379] PVR_K:  605: MTS_BGCTX                     : 0x00000001
[  +0.006287] PVR_K:(Error):  1363: Device in bad state in _UtilStatsDIShow() [970]
[  +0.007497] PVR_K:  605: MTS_BGCTX_COUNTED_SCHEDULE    : 0x00000000
[  +0.006265] PVR_K:(Error):  1363: _Read: failed to read from file, pfnRead() returned -22 [280]
[  +0.008711] PVR_K:  605: MTS_SCHEDULE                  : 0x00000000
[  +0.006547] PVR_K:  605: MTS_GPU_INT_STATUS            : 0x00004150
[  +0.006363] PVR_K:  605: SLC_IDLE                      : 0x000003FF
[  +0.006315] PVR_K:  605: SLC_STATUS0                   : 0x00000000
[  +0.006278] PVR_K:  605: SLC_STATUS1                   : 0x0000000000000000
[  +0.006977] PVR_K:  605: SLC_STATUS2                   : 0x0000000000000000
[  +0.006971] PVR_K:  605: SLC_CTRL_BYPASS               : 0x00003D1F01000000
[  +0.006976] PVR_K:  605: SLC_CTRL_MISC                 : 0x0000000001200003
[  +0.006985] PVR_K:  605: SIDEKICK_IDLE                 : 0x0000007E
[  +0.006320] PVR_K:  605: MIPS_ADDR_REMAP1_CONFIG1      : 0x1FC00001
[  +0.006297] PVR_K:  605: MIPS_ADDR_REMAP1_CONFIG2      : 0x00000000A991A00C
[  +0.007125] PVR_K:  605: MIPS_ADDR_REMAP2_CONFIG1      : 0x1FC01001
[  +0.006305] PVR_K:  605: MIPS_ADDR_REMAP2_CONFIG2      : 0x00000000A997E00C
[  +0.007020] PVR_K:  605: MIPS_ADDR_REMAP3_CONFIG1      : 0x1FC02001
[  +0.006294] PVR_K:  605: MIPS_ADDR_REMAP3_CONFIG2      : 0x00000000A991900C
[  +0.006977] PVR_K:  605: MIPS_ADDR_REMAP4_CONFIG1      : 0x1FC00000
[  +0.006271] PVR_K:  605: MIPS_ADDR_REMAP4_CONFIG2      : 0x000000000000000C
[  +0.006971] PVR_K:  605: MIPS_ADDR_REMAP5_CONFIG1      : 0x00000001
[  +0.006272] PVR_K:  605: MIPS_ADDR_REMAP5_CONFIG2      : 0x00000000A991A00C
[  +0.006975] PVR_K:  605: MIPS_WRAPPER_CONFIG           : 0x000000000001CF80
[  +0.007111] PVR_K:  605: MIPS_EXCEPTION_STATUS         : 0x00000022
[  +0.007389] PVR_K:  605: ---- [ MIPS internal state ] ----
[  +0.005503] PVR_K:  605: PC                            : 0xFFFFFFFF
[  +0.006307] PVR_K:  605: STATUS_REGISTER               : 0x00000000
[  +0.006297] PVR_K:  605: CAUSE_REGISTER                : 0x00000000
[  +0.006287] PVR_K:  605:     Interrupt uses general interrupt vector
[  +0.006372] PVR_K:  605: BAD_REGISTER                  : 0x00000000
[  +0.006281] PVR_K:  605: EPC                           : 0xFFFFFFFF
[  +0.006275] PVR_K:  605: SP                            : 0x00000000
[  +0.006295] PVR_K:  605: BAD_INSTRUCTION               : 0x00000000
[  +0.006315] PVR_K:  605: TLB                           :
[  +0.005366] PVR_K:  605:  0) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009242] PVR_K:  605:  1) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009254] PVR_K:  605:  2) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009241] PVR_K:  605:  3) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009236] PVR_K:  605:  4) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009219] PVR_K:  605:  5) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009232] PVR_K:  605:  6) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009234] PVR_K:  605:  7) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009255] PVR_K:  605:  8) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009238] PVR_K:  605:  9) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009235] PVR_K:  605: 10) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009223] PVR_K:  605: 11) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009237] PVR_K:  605: 12) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009237] PVR_K:  605: 13) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009237] PVR_K:  605: 14) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009269] PVR_K:  605: 15) VA 0x00000000 (  4k) -> PA0 0x00000000        C, PA1 0x00000000        C
[  +0.009247] PVR_K:  605: --------------------------------
[  +0.005418] PVR_K:  605: ------[ RGX FW Trace Info ]------
[  +0.005502] PVR_K:  605: Debug log type: trace ( main mts cleanup csw bif pm rtd spm pow hwr )
[  +0.008611] PVR_K:  605: ------[ RGX FW thread 0 trace START ]------
[  +0.006368] PVR_K:  605: FWT[traceptr]: 22B0
[  +0.004279] PVR_K:  605: FWT[tracebufsize]: 2EE0
[  +0.004652] PVR_K:  605: FWT[00000000]: 000024e8 00000004 7001105c 00000006 c3c845fe 000000a2 7005a02e 00000006 c3c8464b 00000002 00000001 00000000 001395d6 001395d7 7000700e 00000006 c3c84691 70021010 00000006 c3c846e9 d0198450 000013c8 70031013 00000006 c3c8470c c0215114 00009a89 00009a89 70021010 00000006
[  +0.027274] PVR_K:  605: FWT[00000078]: c3c8474a d0198450 00001548 70031013 00000006 c3c84767 c021509c 000130ca 000130cb 70017006 00000006 c3c847b3 00000000 70017014 00000006 c3c847e8 00000000 70041056 00000006 c3c84843 00000002 d0198300 000024e8 00000004 70041056 00000006 c3c84880 00000002 d0198300 000024e8
[  +0.027306] PVR_K:  605: FWT[000000f0]: 00000004 7005a02e 00000006 c3c848dd 00000002 00000001 00000000 001395d6 001395d7 7000700e 00000006 c3c8490b 7001700d 00000006 c3c84938 00000001 70017006 00000006 c3c84946 00000000 70017014 00000006 c3c84953 00000000 70041056 00000006 c3c849a0 00000002 d0198300 000024e8
[  +0.027288] PVR_K:  605: FWT[00000168]: 00000004 70041056 00000006 c3c849e4 00000002 d0198300 000024e8 00000004 7005a02e 00000006 c3c84a0e 00000002 00000001 00000000 001395d6 001395d7 7000700e 00000006 c3c84a3d 70017012 00000006 c3c84a58 00000000 70021010 00000006 c3c84ae8 d0198300 00002558 70031011 00000006
[  +0.027284] PVR_K:  605: FWT[000001e0]: c3c84afe c00df480 00000ac1 00000519 70021010 00000006 c3c84b42 d0198450 00001548 70031013 00000006 c3c84b5a c021509c 000130ca 000130cb 7001700d 00000006 c3c84b7d 00000000 70017014 00000006 c3c88e91 00000018 70021002 00000006 c3c88ebd 00000004 00000003 70074010 00000006
[  +0.027459] PVR_K:  605: FWT[00000258]: c3c88ed6 00000000 00000001 00000000 d00e36c0 d00e3000 00000001 00000002 700a4011 00000006 c3c88ee4 000000fa 00008900 000001bf 00000000 00000000 000000fa 00009010 00000019 00000000 00000000 700a4001 00000006 c3c88ef0 00000000 00000011 00000000 00000009 00000000 a99d9001
[  +0.027298] PVR_K:  605: FWT[000002d0]: 00000000 b14a0001 00000000 b1429001 70025001 00000006 c3c88ef8 d00e36c0 00000000 70021016 00000006 c3c88f28 d00e3190 00004ae0 70021017 00000006 c3c88f37 c0136004 00026222 70021017 00000006 c3c88f3f c00df204 00000519 70021017 00000006 c3c88f48 c00df33c 00000519 70038019
[  +0.027297] PVR_K:  605: FWT[00000348]: 00000006 c3c88f73 d00e3000 00000005 00000000 7003700a 00000006 c3c88fa2 00000000 00000002 d0198300 70041058 00000006 c3c88fbb 00000002 d0198300 000024e8 00000000 70027016 00000006 c3c88fc8 00000002 d01e74e8 7003800e 00000006 c3c88fe8 d01982c0 00000002 00000000 70068013
[  +0.027459] PVR_K:  605: FWT[000003c0]: 00000006 c3c88ff6 00000006 00000002 00000000 c7bdc000 00000000 00000000 70028004 00000006 c3c89014 00000006 00000001 7003500a 00000006 c3c89025 d0218900 d00e36c0 00000000 700c501e 00000006 c3c89049 d01359c0 00000000 00000000 0000001a 00000020 00000000 000000fa 0006f010
[  +0.027302] PVR_K:  605: FWT[00000438]: 00000000 00000001 0000000e 0000000a 700c501f 00000006 c3c89071 d0135600 00000000 00000000 0000001a 00000020 00000000 000000fa 00042010 00000000 00000019 00000000 00000000 7003501d 00000006 c3c89080 00000003 00000001 00000001 7002500f 00000006 c3c89096 d0218900 00000000
[  +0.027407] PVR_K:  605: FWT[000004b0]: 70075021 00000006 c3c890a1 d0198300 d0218900 00000080 0253d600 00000080 0254a000 d01982c0 70035018 00000006 c3c890c9 00000080 025462c0 00000001 70074010 00000006 c3c89112 00000001 00000000 00000000 d0218900 d01982c0 00000001 00000002 700a4012 00000006 c3c8911d 000000fa
[  +0.027293] PVR_K:  605: FWT[00000528]: 00000000 000021d5 0000022a 00000000 000000fa 00042010 00000019 00000000 00000000 700a4001 00000006 c3c89129 00000000 00000000 00000000 00000000 00000000 a99d9001 00000000 b14a0001 00000000 b1429001 700b1077 00000006 c3c89137 d0198300 000024e8 d0218900 00000001 00000001
[  +0.027300] PVR_K:  605: FWT[000005a0]: 00000000 00000652 00000001 00000000 00072e3c 0009c94a 70041056 00000006 c3c8917f 00000003 d0198450 000013e0 00000008 7005a02e 00000006 c3c891a9 00000002 00000001 00000000 001395d7 001395d8 7000700e 00000006 c3c891dc 70017012 00000006 c3c891f3 00000000 7001700d 00000006
[  +0.027311] PVR_K:  605: FWT[00000618]: c3c89295 00000000 70017012 00000006 c3c892ac 00000000 7001700d 00000006 c3c892f3 00000000 70017014 00000006 c3c89376 00000020 70001008 00000006 c3c8938b 70074010 00000006 c3c8939c 00000001 00000001 00000000 d0218900 d01982c0 00000001 00000002 700a4012 00000006 c3c893a8
[  +0.027512] PVR_K:  605: FWT[00000690]: 000000fa 00000000 000021bb 00000244 00000000 000000fa 00042010 00000008 00000008 00000009 700a4001 00000006 c3c893b6 00000000 00000010 00000000 00000009 00000000 e07b7001 00000000 e07a6001 00000000 e07a3001 7002500e 00000006 c3c893cc d0218900 00000000 70001049 00000006
[  +0.027313] PVR_K:  605: FWT[00000708]: c3c893d6 70021016 00000006 c3c893fd d0198300 00002538 70021017 00000006 c3c89407 c021509c 000130cb 70021017 00000006 c3c89417 c00df5d0 00000519 70038019 00000006 c3c8943f d01982c0 00000006 00000000 7003700a 00000006 c3c89479 00000000 00000003 d0198450 70041058 00000006
[  +0.027286] PVR_K:  605: FWT[00000780]: c3c89485 00000003 d0198450 000013e0 00000000 70027016 00000006 c3c89490 00000003 d01c63e0 7003800e 00000006 c3c894b3 d01982c0 00000003 00000000 70068013 00000006 c3c894cd 00000006 00000003 00000000 c7bdc000 00000000 00000000 70028004 00000006 c3c894d8 00000006 00000001
[  +0.027464] PVR_K:  605: FWT[000007f8]: 70025008 00000006 c3c894ea d021ae80 00000001 700c501e 00000006 c3c8950b d0135040 00000000 00000001 0000001a 00000020 00000000 000000fa 00009010 00000000 00000019 00000000 00000000 700c501f 00000006 c3c89531 d01359c0 00000000 00000001 0000001a 00000020 00000000 000000fa
[  +0.027406] PVR_K:  605: FWT[00000870]: 0006f010 00000000 00000001 0000000e 0000000a 700c501e 00000006 c3c89540 d0135000 00000001 00000001 000001c0 00002400 00000100 000000fa 00008900 00000000 000001bf 00000000 00000000 700c501e 00000006 c3c8954e d0135100 00000001 00000000 00002400 00002400 00000100 000000fa
[  +0.027311] PVR_K:  605: FWT[000008e8]: 00000000 00000000 000021bb 00000244 00000000 700c501f 00000006 c3c8955b d0135100 00000001 00000001 00002400 00002400 00000100 000000fa 00000000 00000000 000021bb 00000244 00000000 7003501d 00000006 c3c89567 00000002 00000001 00000000 70045022 00000006 c3c89587 d0198450
[  +0.027289] PVR_K:  605: FWT[00000960]: d021ae80 00000001 d01982c0 70025002 00000006 c3c89596 d021ae80 00000001 7002110a 00000006 c3c895be d0198450 00072e41 70011107 00000006 c3c895d6 00072e41 70011108 00000006 c3c895df 00072e41 700310c6 00000006 c3c895eb 00000001 00000000 0003000f 70021087 00000006 c3c895f4
[  +0.027308] PVR_K:  605: FWT[000009d8]: 00000000 00000001 70074010 00000006 c3c89605 00000000 00000000 00000001 d021ae80 d01982c0 00000000 00000002 700a4011 00000006 c3c89612 000000fa 00000000 000021bb 00000244 00000000 000000fa 0006f010 00000001 0000000e 0000000a 700a4001 00000006 c3c8962c 00000001 0000011b
[  +0.027260] PVR_K:(Error):  1363: Device in bad state in _UtilStatsDIShow() [970]
[  +0.007506] PVR_K:  605: FWT[00000a50]: 00000001 0000000a 00000000 f0e75001 00000000 f0e79001 00000000 f0e7b001 700a1078 00000006 c3c89639 d0198450 000013e0 d021ae80 00000000 00000000 00000652 00000001 00000000 00072e41 0009c948 7005a02e 00000006 c3c896a0 00000002 00000001 00000000 001395d8 001395d9 7000700e
[  +0.027261] PVR_K:(Error):  1363: _Read: failed to read from file, pfnRead() returned -22 [280]
[  +0.008696] PVR_K:  605: FWT[00000ac8]: 00000006 c3c896c1 70017012 00000006 c3c896e3 00000000 70021010 00000006 c3c89745 d0198450 00001548 70031013 00000006 c3c89759 c021509c 000130cb 000130cb 70021010 00000006 c3c89777 d0198450 000016d0 70031013 00000006 c3c89791 c02150e4 00013058 00013059 7001700d 00000006
[  +0.027451] PVR_K:  605: FWT[00000b40]: c3c897a5 00000000 70017012 00000006 c3c897b2 00000000 7001700d 00000006 c3c897f3 00000000 70017012 00000006 c3c8c4a1 00000000 7001700d 00000006 c3c8c506 00000000 70017012 00000006 c3c91113 00000000 7003b049 00000006 c3c91148 00000003 00007afc 04a8a080 7003b054 00000006
[  +0.027300] PVR_K:  605: FWT[00000bb8]: c3c9115e 00000000 00000003 00000000 7003b030 00000006 c3c91163 00000003 00000000 00000005 7001700d 00000006 c3c911c3 00000000 70017012 00000006 c3c911e7 00000000 7001700d 00000006 c3c9123e 00000000 70017012 00000006 c3c937c6 00000000 70037013 00000006 c3c937ed 00000308
[  +0.027305] PVR_K:  605: FWT[00000c30]: 2abc0071 00000000 7003100e 00000006 c3c9380b 000038a8 00000002 d00e3040 70021010 00000006 c3c93839 d00e3040 00003818 70031011 00000006 c3c9384b c00df480 00000ac1 00000519 7003100e 00000006 c3c93874 00004c90 00000003 d00e3190 70021010 00000006 c3c938a5 d00e3190 00004b08

  • 0 in reply to Siddhesh Hande
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    Thank you for the logs. IMG has just been able to reproduce the issue as well and are now working on developing and testing patches. I will send an update once completed. 

    Thanks,

    Shriya

  • 0 in reply to Shriya Surti
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    I received a patch from IMG earlier last week but after testing it over the course of the last 3 days, we still are running into the issue. I have reported the new logs to IMG and will keep you posted when further progress is made. 

    Regards,

    Shriya

  • 0 in reply to Shriya Surti
    Prodigy 65 points

    Hi Shriya,

    Thank you for updates,

  • 0 in reply to Siddhesh Hande
    Prodigy 65 points

    Hi Shriya,

    Could you please share an update on this issue? It has been a long time since we received any updates, and we are currently blocked and unable to proceed further due to this issue.

  • 0 in reply to Siddhesh Hande
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    Apologies for the delay. Currently no new patches have been provided as a potential fix, however, we are still testing internally to see if anything can help isolate this issue. Will have a response for you by Tuesday of next week with further information. 

    Regards,

    Shriya

  • 0 in reply to Shriya Surti
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    Currently IMG has been running a build since October 14th with modifications. I will be testing the patches they provided to me on the release build for DDK 26.1 and 23.3 for the next couple of days and get back to you with an update on if this continues to crash or run without issues. 

    Regards,

    Shriya

  • 0 in reply to Shriya Surti
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    IMG and I have tested a debug build with some modifications that ran for over 50 hours. We are now in the process of running the release build with the same configurations. We will update you early next week on if this build succeeds as well.

    Regards,

    Shriya

  • 0 in reply to Shriya Surti
    TI__Intellectual 2690 points

    Hi Siddhesh, 

    To provide an update, the release build has failed after running the past couple of days. I have a meeting with IMG in place for Monday to go over additional debugging steps and will update you accordingly once given further information. 

    Regards,

    Shriya