# AM62x: PowerVR "Guilty Lockup" under sustained WebGL — GPU resets, client never regains a context Draft for TI E2E. Check for an existing ticket on `Guilty Lockup` / `HWR` / am62 GPU reset before posting. --- ## Summary On AM62x, sustained WebGL in Chromium intermittently causes the PowerVR firmware to time out a render job, declare `Guilty Lockup`, and reset the GPU. The reset succeeds — `RGX Health Status: OK` afterwards — but the client never gets a working rendering context back. The page goes blank permanently and only recovers if the browser is restarted. **Reproduced on an unmodified `tisdk-default-image-am62xx-evm-12.01.00.05.03`, on HDMI at 1600x900, with DDK 26.1.6967606 and kernel 6.18.38-ti** — no custom image, kernel, DTS or panel involved. Also reproduced on DDK 23.3.6512818 (SDK 9.2) and 24.2.6643903 (SDK 11). Every variable differs across the failing set except one: **firmware `rgx.fw.33.15.11.3` is identical on SDK 9.2, 11 and 12.1.** Timing is random and spans a wide range — observed at 1.25 h, 3.89 h, 5.42 h, 7.62 h, 8.06 h and 49.28 h of continuous rendering. **No run has survived indefinitely.** The board previously recorded as clean past 20 h went on to fail at 49.28 h with an identical signature, so a long clean run indicates a slow board, not an immune one. Full `PVR DBG` dump from the SDK 12.1 reproduction is attached (`pvr-dump-sdk12.1-guilty-lockup.txt`). ## Hardware and software | | | |---|---| | SoC / GPU | AM62x, PowerVR A-Series AXE-1-16M (`GL_RENDERER: PowerVR A-Series AXE-1-16M`) | | Boards | custom AM62x with 1024x600 and 480x272 DPI panels (`card0-DPI-1`), plus HDMI at 1600x900 on the stock SDK 12.1 image | | SDK 11 arm | Processor SDK 11.01, kernel 6.12.35, weston 13.0.1, Chromium 132.0.6834.83, DDK **24.2.6643903** | | SDK 12.1 arm | **stock** `tisdk-default-image-am62xx-evm-12.01.00.05.03`, Arago 2026.04, kernel 6.18.38-ti, weston 15.0.0, DDK **26.1.6967606**, HDMI-A-1 @ 1600x900 | | SDK 9.2 arm | Processor SDK 9.02, kernel 6.1.80, weston 10.0.2, DDK **23.3.6512818** | | Firmware | `rgx.fw.33.15.11.3` — **identical on all three SDKs** | | RAM | 2 GB (and a 1 GB unit in production) | ## Reproduction 1. Boot to a weston kiosk with Chromium fullscreen. 2. Load `https://webglsamples.org/aquarium/aquarium.html` (default sample count). 3. Leave it rendering. 4. Somewhere between ~1 h and ~50 h, the canvas goes blank. The HTML page is still there; only the WebGL content stops. Externally visible signature — the scanned-out framebuffer id stops changing: ```sh for i in $(seq 1 8); do cat /sys/kernel/debug/dri/*/state | sed -n 's/.*fb=\([0-9]*\).*/\1/p' | head -1; sleep 2 done # healthy : 50 51 50 51 50 51 ... # failed : 50 50 50 50 50 50 ... ``` ## Kernel log at the failure On stock SDK 12.1 / DDK 26.1 (the cleanest reproduction — nothing custom involved): ``` PVR_K: DDK info: Rogue_DDK_Linux+Linux_WS rogueddk 26.1@6967606 (release) am62_linux PVR_K: Comparison of UM/KM components: MATCHING PVR_K: Recovery 1: PID = 1152 / chromium-bin, frame = 0, HWRTData = 0xD018EB80, EventStatus = 0x00000000, Guilty Lockup PVR_K: PreResetTimeInCycles = 40448, HWResetTimeInCycles = 18944, FreelistReconTimeInCycles = 936448, TotalRecoveryTimeInCycles = 995840 PVR_K: RGX Health Status: OK, Reason: NONE PVR_K: RGX FW OS 0 - State: active; Freelists: Ok; ... PVR_K: Number of HWR: GP(0/0+0), 2D(0/0+0), GEOM(1/1+0), 3D(0/0+0), CDM(0/0+0) ``` Earlier, on SDK 11 / DDK 24.2 — note `Freelists: Not Ok` and no reconstruction step: ``` PVR_K: Recovery 1: PID = 321 / chromium-bin, frame = 0, HWRTData = 0xC00E3400, EventStatus = 0x00000000, Guilty Lockup PVR_K: PreResetTimeInCycles = 36096, HWResetTimeInCycles = 14336, TotalResetTimeInCycles = 50432 PVR_K: RGX Health Status: OK, Reason: NONE PVR_K: RGX FW OS 0 - State: active; Freelists: Not Ok; Priority: 0; ... PVR_K: Number of HWR: GP(0/0+0), 2D(0/0+0), GEOM(1/1+0), 3D(0/0+0), CDM(0/0+0) PVR_K: DM 1..4 (HWRflags 0x00000000: working;) WARNING: CPU: 1 PID: 254 at .../services/server/common/pvr_notifier.c:663 PVRSRVDebugRequest+0x530/0x6ec [pvrsrvkm] ``` Timeout configuration in the same dump: `MaxHWTOut: 500000us, WtTryCt: 10000, WDGTOut(on,off): (10000ms,3600000ms)`. ## After the reset Chromium detects the context loss and restarts its GPU process — a **new** pid appears and reacquires 10 `/dev/dri` fds and the PowerVR libraries. The service itself never restarts (`NRestarts: 0`). Despite that, no further frames are ever presented. On the DDK 24.2 board that failed at 49.28 h, the GPU process was replaced twice after the fault (`298 → 60069 → 60455`), each replacement holding a full complement of 10 `/dev/dri` fds, while the scanned-out framebuffer stayed frozen indefinitely. On one DDK 23.3 board the page resumed rendering for several hours after the reset and then stopped anyway, having cycled through five GPU process ids (922 → 981 → 11030 → 12632 → 31716) with only the one `Guilty Lockup` logged. So context re-creation appears to succeed at the process level while still not yielding a usable context. **Restarting the whole browser process always recovers, with hardware acceleration intact.** This is the most useful diagnostic we have. After a real lockup on the stock SDK 12.1 image, a relaunched Chromium's GPU process holds 10 `/dev/dri/render` fds and maps the hardware driver, not a software fallback: ``` libGLESv2_PVR_MESA.so.26.1.6967606 libpvr_dri_support.so.26.1.6967606 libglslcompiler.so.26.1.6967606 libEGL.so.1.0.0 ``` Same result on SDK 11 / DDK 24.2. So after the HWR the GPU, the kernel driver and the firmware are all demonstrably healthy and able to serve a brand-new client immediately. The only thing that cannot be rebuilt is the rendering context inside the browser process that was live at the moment of the reset — even though that process successfully reopens the device. ## What we ruled out **Not a DDK version issue, and not anything of ours.** The fault reproduces across three DDK majors (23.3, 24.2, 26.1), three kernels (6.1.80, 6.12.35, 6.18.38), three weston versions (10, 13, 15) and three Chromium versions — including on TI's own unmodified SDK 12.1 image, on HDMI, with TI's kernel and device tree. The only component common to every failure is the firmware, `rgx.fw.33.15.11.3`, unchanged across all three SDK releases. **Recovery has measurably improved without changing the outcome.** DDK 24.2 left `Freelists: Not Ok` after the reset; 26.1 reports `Freelists: Ok` and performs an explicit freelist reconstruction — `FreelistReconTimeInCycles = 936448, TotalRecoveryTimeInCycles = 995840`. The client still never regains a usable context. **Not memory.** 286 MiB of 1914 MiB system RAM in use at failure, zero OOM events. GPU heap reported `Free Size: 1063018496B` of `Total Size: 2007990272B`. **Not GPU load in general.** `glmark2-es2-wayland` looped on-screen for 4.2 h — 43 passes, scores flat at 151–154 — produced no lockup at all on DDK 24.2. **Not proportional to load — if anything, the opposite.** A production unit on SDK 9.2 ran the same site at 30k samples (heavy enough that FPS dropped noticeably) for **15 days** with no failure, while the lighter default page has failed in as little as 1.25 h. This suggests exposure tracks the *rate* of job submission rather than the work per job. Worth stating plainly, though: the observed times to failure span 1.25 h to 49.28 h on comparable hardware, so we cannot claim to have isolated what sets the rate. **Not the network.** The page renders entirely from memory; disconnecting Ethernet does not affect it. **Not the compositor's VNC mirror.** The SDK 9.2 board runs no screen mirroring and still fails. ## Possibly relevant Every dump shows a very high retry rate on GPU power transitions: ``` SDK 11 / DDK 24.2: RGX FW Power State: RGXFWIF_POW_ON (APM enabled: 14 ok, 0 denied, 0 non-idle, 2195 retry, 0 other, 2209 total. Latency: 100 ms) ``` 2195 retries out of 2209 attempts on SDK 11. However the SDK 12.1 reproduction shows `APM enabled: 272 ok, 0 denied, 0 non-idle, 11 retry, 0 other, 283 total` — about 4%. So the retry storm is NOT required for the fault, and this lead is much weaker than it first appeared. Included for completeness. `RGX DVFS: 0 frequency changes. Current frequency: 500.000 MHz` — GPU frequency is static throughout, so DVFS is not a variable. ## Observations across runs | Board | DDK | Panel | Outcome | |---|---|---|---| | A | 24.2 | 1024x600 | lockup at 1.25 h | | B | 24.2 | 480x272 | lockup at 5.42 h, and 8.06 h on a later boot | | C | 23.3 | 1024x600 | lockup at 7.62 h; rendered again for hours, then stopped | | A (rerun) | 24.2 | 1024x600 | clean through 20.7 h, then **lockup at 49.28 h** — longest run observed | | A | 24.2 | 1024x600 | glmark2 loop, 4.2 h, no lockup | | **D (stock SDK 12.1)** | **26.1** | **HDMI 1600x900** | **lockup at 3.89 h** | ## Questions 1. Is a `Guilty Lockup` followed by a successful HWR expected to leave the *existing* client able to create a working context again? A newly started client always gets full hardware acceleration immediately afterwards, so the device, driver and firmware are healthy — yet the process that was live during the reset never presents another frame, even after it reopens `/dev/dri` and remaps the PowerVR libraries. Is there a post-HWR recovery step the user-mode driver or the client is expected to perform, and is failing to? 2. Is `Freelists: Not Ok` a cause or a consequence of the reset, and is there a recovery step the user-mode driver is expected to perform after HWR? 3. On SDK 11 the dump showed `APM … 2195 retry of 2209 total` (~99%), while SDK 12.1 shows ~4%. Is that difference expected between DDK versions, and is the high figure a problem in its own right even though it is clearly not required for this fault? 4. Given the firmware `rgx.fw.33.15.11.3` is unchanged across SDK 9.2, 11 and 12.1 while the DDK has moved three majors, is the watchdog/reset path in the firmware the right place to look? Full `PVR DBG` dumps can be provided.