SK-TDA4VM: Rogue PowerVR Display Driver stability

Hello,

Your in the right place! Please let us know the issue you are facing, the logs and what commands you used to collect them, and the SDK version you are on.

Thanks,

Erick

I'm running the latest Arago release (tisdk-core-image with a few additional changes) on the SK-TDA4VM.  I'm seeing quite a few PVR error outputs happening.  My app (homescreen) is what's running.  It's running a GLES/EGL backend.  The most recent pvrlogdump below is showing it's powered off, and I have a frozen image on the screen.  Seems the display output block is still being clocked.  The app call stack for crash:

(gdb) info threads

  Id   Target Id                                          Frame 
* 1    Thread 0xffffb9a42020 (LWP 2413) "homescreen"      0x0000ffffb957c96c in ?? () from /lib/libc.so.6
  2    Thread 0xffffb92ff0a0 (LWP 2414) "homescreen"      0x0000ffffb957c96c in ?? () from /lib/libc.so.6
  3    Thread 0xffffb57580a0 (LWP 2415) "homescreen"      0x0000ffffb957c96c in ?? () from /lib/libc.so.6
  4    Thread 0xffffae9ee0a0 (LWP 2416) "io.flutter.ui"   0x0000ffffb95e7dc4 in epoll_pwait () from /lib/libc.so.6
  5    Thread 0xffffae1de0a0 (LWP 2417) "homescreen"      0x0000ffffb95de1a0 in poll () from /lib/libc.so.6
  6    Thread 0xffffad9ce0a0 (LWP 2418) "io.flutter.io"   0x0000ffffb95e7dc4 in epoll_pwait () from /lib/libc.so.6
  7    Thread 0xffff9ffff0a0 (LWP 2419) "io.worker.1"     0x0000ffffb957c96c in ?? () from /lib/libc.so.6
  8    Thread 0xffff9f7ef0a0 (LWP 2420) "io.worker.2"     0x0000ffffb957c96c in ?? () from /lib/libc.so.6
  9    Thread 0xffffad1be0a0 (LWP 2421) "dart:io EventHa" 0x0000ffffb95e7dc4 in epoll_pwait () from /lib/libc.so.6
(gdb) bt 10
#0  0x0000ffffb957c96c in ?? () from /lib/libc.so.6
#1  0x0000ffffb957f698 in pthread_cond_wait () from /lib/libc.so.6
#2  0x0000ffffb9986c84 in wl_display_read_events () from /usr/lib/libwayland-client.so.0
#3  0x0000aaaac8aefea8 in ?? ()
#4  0x0000ffffb952b230 in ?? () from /lib/libc.so.6
#5  0x0000ffffb952b30c in __libc_start_main () from /lib/libc.so.6
#6  0x0000aaaac8af3670 in ?? ()

#
# Localization
#
DEFAULT_TIMEZONE = "America/Los_Angeles"
ENABLE_BINARY_LOCALE_GENERATION = "1"
IMAGE_LINGUAS:append = " en-us en-gb es-us"
GLIBC_GENERATE_LOCALES:append = " en_US.UTF-8 es_US.UTF-8 en_GB.UTF-8"
IMAGE_INSTALL:append = "\
    tzdata-core \
    tzdata-americas \
    "

DISTRO_FEATURES:append = " opengl opencl wayland bluetooth"
DISTRO_FEATURES:remove = " x11 vulkan zeroconf wireless"
PACKAGECONFIG:append:pn-weston = " remoting"

IMAGE_INSTALL:append = " \
    k3conf \
    \
    ivi-homescreen \
    flutter-gallery \
    \
    opencl-icd-loader \
    libopencl-rogue \
    clinfo \
    libopencl-rogue-tools \
    ti-img-rogue-umlibs-tools \
    \
    optee-client \
    optee-test \
    optee-examples \
    \
    adwaita-icon-theme-cursors \
    liberation-fonts \
    geoclue \
    gsettings-desktop-schemas \
    \
    libcamera \
    libgpiod \
    libserialport \
    os-release \
    upower \
    xdg-user-dirs \
    xkeyboard-config \
    \
    libgpiod-tools \
    i2c-tools \
    spidev-test \
    can-utils \
    \
    bluez5 \
    bluez5-noinst-tools \
    bluez5-obex \
    bluez5-testtools \
    \
    weston \
    weston-init \
    \
    elfutils \
    "

IMAGE_FEATURES:remove = "splash"

#
# Selinux
#
DISTRO_FEATURES:append = " acl xattr pam selinux audit"
PREFERRED_PROVIDER_virtual/refpolicy = "refpolicy-mls"

#
# Security
#
DISTRO_FEATURES:append = " security seccomp"

#
# Virtualization
#
DISTRO_FEATURES:append = " virtualization kvm"
IMAGE_INSTALL:append = " \
    kvmtool dmidecode \
    firecracker jailer curl \
    "

DISTRO_FEATURES:append = " systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"

meta-arago-distro    
meta-arago-extras    
meta-arago-demos     = "HEAD:4d7741adb72c626d05b845867c72f157b64d5944"
meta-qt5             = "HEAD:bff5bd937f0776166e81a63f3dd39ede348ef758"
meta-virtualization  = "HEAD:b3b3dbc67504e8cd498d6db202ddcf5a9dd26a9d"
meta-networking      
meta-python          
meta-oe              
meta-gnome           
meta-filesystems     = "HEAD:346753705e49a2486867dc150181a1c7f4d69377"
meta-ti-extras       
meta-ti-bsp          = "HEAD:474af844b0e808f5401ab0b89e59d1ade39f3d16"
meta-arm             
meta-arm-toolchain   = "HEAD:c39bb4ce3b60b73d35c5fb06af012432e70d6b38"
meta                 = "HEAD:200c2783b3f8546f561382fff6bd5268680d403a"
meta-aws             = "HEAD:b72cb364f79aa055242e703e284900b23e61facd"
meta-psdkla          = "HEAD:4b85d0888c8e9fb9d708072cc6e9db33941ea1b2"
meta-perl            = "HEAD:346753705e49a2486867dc150181a1c7f4d69377"
meta-selinux         = "kirkstone:a401f4b2816a0b41ce8d9351542658c721935bcd"
meta-security        = "kirkstone:1a3e42cedbd94ca73be45800d0e902fec35d0f0f"
meta-flutter         = "kirkstone:7d8cf538802a12be9f60673b6e0210266dffcd07"

I'm running homescreen with the following command from weston terminal:

homescreen --b=/usr/share/flutter/gallery/3.13.2/release/ --p=.5

The --p=.5 gets passed to the Flutter Engine to set the pixel ratio.  Not including it uses pixel ratio of 1.0.

Joel,

Do you only get these PVR Errors when running your homescreen aapp? Or when running other graphics tests?

It looks like you've enabled flutter, wondering if you still see this without flutter.

Either way, let's take a look at this issue. I'll forward your debug dumps to start analyzing.

Regards,

Erick

I haven't ran any other graphics tests, yet.  Homescreen (which is flutter-auto in AGL) runs on numerous other platforms without any problems, so my current impression is that it's isolated to the Rogue driver.  Repro took >2 days to hit, as you can see in the log timestamps.

Joel,

Ok, thanks for the info. Not trying to narrow down if it's a flutter issue, rather if any other graphics test shows the issue, or if I have to use flutter to reproduce this.

I'll update you on our progress.

Thanks,

Erick

Cool thanks

No problem!

Joel,

Same as the other thread, I am trying to replicate your setup to make the debug faster as we will probably need more logs. But in the meanwhile my update is that the gpu vendor is taking a look at the logs you provided to give us the next steps in the debug. I'll be waiting for the feedback and in the meanwhile setting up your environment.

Thanks,

Erick

Joel,

Meanwhile I replicate your setup, I've got some instructions of collecting more verbose logs. Could you please run this command before you start any GPU application:

pvrdebug -loggroups main,hwr,bif,rtd,csw
echo Y > /sys/kernel/debug/pvr/apphint/0/AssertOnHWRTrigger

This will 1) Enable more verbose firmware logs and 2) halt the GPU once an issue has occurred so we can get a snapshot of the firmware trace.

Once the GPU has observed an error, can you collect a pvrlogdump and send it over?

Thanks,

Erick

Cool.  I will start stability run in the next couple of hours with these settings.

Joel,

Running it myself now, seeing if it replicates. I'll run other graphics tests as well.

Regards,

Erick

Hi Joel, 

I just built the same environment and tested it on my AM62xx-EVK.

With the flutter auto, this is what I see on the screen:

https://youtu.be/3v6xKIM4pmk

with homescreen, it just worked well for me.

Raul,

Are you getting any log dumps while this happens? The issue Joel reported looks related to a GPU issue we have in the current SDK 9.0 release. We currently have known GPU driver issues that can be triggered during long runs of a variety of GPU applications. The team is working towards stabilizing the latest release, but at the moment we still expect some instability in SDK 9.0 on TDA4x platforms (AM68, AM69).

Regards,

Erick

Hi Erick, I'm, sorry but I didn't get the logs during that test. I know you all are working on the GPU,  I have been in touch and following this work.

I'm using also SDK 9.0. 

If I find more things I will keep you posted. 

Raul,

Understood. AM62x problems might or might not be related, we have noticed the TDA4x platforms have issues relating to L3 cache, and the GPU cores are different between the systems. If you do get more logs, please provide them on a separate ticket so we can keep better track of the two issues.

Regards,

Erick