[FAQ] How can I analyze a GPU (Graphics Process Unit) driver or hardware issue?

TI platforms that contain a graphics accelerator (GPU) can observe a crash or an abnormality that manifests in many different ways including:

• A Linux kernel panic/crash

• Display output rendering incorrectly

  • Random artifacts in the output

  • Tearing, freezing or general corruption

• Recoverable or unrecoverable failures in other modules

• A failure in the graphics driver

Furthermore, due to the many other modules/accelerators on the SoC, the system can manifest issues outside the GPU context.

The focus of this post are the issues relating to GPU driver or hardware failures. GPU application failures, like incorrect usage of OpenGL APIs, can be debugged in a different manner.

Below is the debugging procedure in chronological order. When posting a ticket about a possible GPU issue, please include at least the Basic Debug. Preferably also the PVR Debug Dump.

Basic Debug

Symptoms

The first step to analyze the issue is listing the symptoms seen, for example:

• The display is frozen/went blank
• There is corruption in the output, such as artifacts or tearing
• The application "resets" periodically

Replication

Also, provide the following details about the issue:

• How easy is it to replicate? Statistics such as 1 in 1000 EVMs are sufficient
• Is this reproducible on TI EVM? Or only on customer hardware?
• Is it possible to share the application that is causing the issue?

The easier it is for TI Engineers to reproduce the issue, the quicker we can root-cause.

Console Logs

It is important to save the console logs when the issue is present. This will give an idea of the application running on the GPU and other useful information from the trace, like the GPU driver version. But it usually is not sufficient and requires further logs.

PVR Debug Dump

To narrow-down on the issue, more debug information is usually needed to get firmware logs and other GPU driver information. This information can be gathered for the GPU-related incidents in the following manner:

GPU Firmware Trace

The GPU contains a separate CPU which runs its firmware. This firmware can provide logs, but are not enabled by default. To enable and capture these logs, please follow the steps below:

1. Before running the application, run the following command:
   `pvrdebug -loggroups main,mts,hwr`
2. After experiencing an error in execution, run the following command: `pvrlogdump`. This will generate a file that can be shared on the E2E ticket for further analysis.

More logs might be required, and more flags will need to be enabled. Further guidance will be provided by the TI Engineer.

Disabling auto-loading of the GPU driver

PVR Carbon

PVRCarbon is another powerful tool that can be used to record and play-back an application execution. You can find the details and the tool on Imagination Technologies' website:
developer.imaginationtech.com/.../

This tool can be used to record the OpenGLES API calls to the driver and "play" them back as if you were running the application. The instructions are as follows:

1. Install PVRCarbon onto your Windows or Linux PC with the appropriate installer
2. Copy the contents of the below folder into your target device's filesystem. Use the appropriate architecture, i.e TDA4x devices use Linux_armv8_64/
   PowerVR_Graphics/PowerVR_Tools/PVRCarbon/Recorder/GLES/Linux_armv8_64
   - Note: if you have an ssh server running on the target, you can copy the files over like this:
   scp -r Linux_armv8_64/ root@<ip-addr>:/home/root
3. Use the libraries in the Recorder folder for running the GPU app: export LD_LIBRARY_PATH=<Copied Carbon recorder Location>/Linux_armv8_64
4. Run your application, you will see PVRCarbon log messages in the console
   PVRCarbon Recorder v0.7 (21.1@5966430)
   PVRCarbon|Note: By default to support multi-platform playback we override api alignment values to our recommended values. To disable this set the align recommended option to false
   PVRCarbon|Writing intermediate data to '/run/media/mmcblk1p3/gfxbench5-bins/testfw_app.0001.parts'
5. Your PVRCarbon file will be created in the directory you ran the test from. Log examples below:
   - PVRCarbon|Creating file '/run/media/mmcblk1p3/gfxbench5-bins/testfw_app.pvrcbn'
   - PVRCarbon|Successfully created '/run/media/mmcblk1p3/gfxbench5-bins/testfw_app.pvrcbn'

To play back the PVRCarbon file created, please use the following steps.

1. Copy the contents of the below folder into your target device's filesystem:
   PowerVR_Graphics/PowerVR_Tools/PVRCarbon/Player/GLES/Linux_armv8_64
2. Make sure weston is running. If it is not running, please use the command: /etc/init.d/weston start
3. Copy the PVRCarbon file you recorded into the Player/GLES/Linux_armv8_64 directory
4. Change directory into the Player/GLES/Linux_armv8_64 on the target device
5. Run: ./PVRCarbonPlayer --ws=wayland *.pvrcbn