PROCESSOR-SDK-AM335X: Font Rendering Issue

Hi,

Have you referred to the following threads: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/877088/am3354-opengl-qt-font-rendering-problem---corrupt-glyphs/3365678#3365678 and https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1031265/am3354-am3354-opengl-qt-font-rendering-problem---corrupt-glyphs---solution-committed/3812776#3812776 

Regards,

Krunal

Do they work with our kernel version 5.4.52? 

It depends on the Yocto version you are using. We already addressed the issue on Dunfell but the libraries I shared were for Thud. 

Regards,

Krunal

Do you have a patch for buildroot by any chance? 

No, it's only for Linux.

Regards,

Krunal

Could you please list the specific commits needed for this glphy fix so that we can cherry pick them onto our kernel 5.4.52? We would like to avoid kernel update from 5.4.52 to 5.4.106 which has over 5000 commits. 

Hi,

The GPU driver is split into two parts: UM and KM

UM: https://git.ti.com/cgit/graphics/omap5-sgx-ddk-um-linux/?h=ti-img-sgx%2Fdunfell%2F1.17.4948957 

KM: https://git.ti.com/cgit/graphics/omap5-sgx-ddk-linux/?h=ti-img-sgx%2F1.17.4948957%2Fk5.4

The fixes we introduced was in UM and we only release the binaries. The binaries were updated in the above UM branch.

Regards,
Krunal

Hi Krunal,

Sha and I have been trying to tackle this glyph rendering issue however we're getting coredumps when we switch over to these new commits whereas previously our system was fine.

Here is my current configuration:

1. linux kernel: v5.4.52

•  from branch 5.4.52-ti-r17 branch of https://github.com/beagleboard/linux.git

2. ti-sgx-um is set to: 742cf38aba13e1ba1a910cf1f036a1a212c263b6

• from branch: origin/ti-img-sgx/dunfell/1.17.4948957 of http://git.ti.com/git/graphics/omap5-sgx-ddk-um-linux.git
• previous commit: 909e237baf47d0bde006ff25552f5403fd7e359d

3. ti-sgx-km is set to: bfe83bbabb3849c24b03d5172cf678e7c5915e04

• from branch: origin/ti-img-sgx/1.17.4948957/k5.4 of https://git.ti.com/git/graphics/omap5-sgx-ddk-linux.git
• previously set to: 50c1ec2308b9f64488d252ac55d65b51a0dfe287

Unfortunately, with this current configuration, we're getting coredumps when using QT5's default hellowindow sample application. The failure occurs at "QEglFSIntegration::initialize()"

Any suggestions?

# export QT_QPA_PLATFORM=eglfs
# export QT_QPA_EGLFS_INTEGRATION=none
# /usr/lib/qt/examples/opengl/hellowindow/hellowindow 
Could not initialize egl display
Aborted (core dumped)
# coredumpctl debug /usr/lib/qt/examples/opengl/hellowindow/hellowindow                                                                                                                     
           PID: 3176 (hellowindow)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Mon 2024-06-17 14:29:43 UTC (16s ago)
  Command Line: /usr/lib/qt/examples/opengl/hellowindow/hellowindow
    Executable: /usr/lib/qt/examples/opengl/hellowindow/hellowindow
 Control Group: /system.slice/system-serial\x2dgetty.slice/serial-getty@ttyO0.service
          Unit: serial-getty@ttyO0.service
         Slice: system-serial\x2dgetty.slice
       Boot ID: 6c33eef9bac74582aec1a0ffdd7ce1a6
    Machine ID: db252b17d27c4059aa2c59a9e994de83
      Hostname: rosebud
       Storage: /var/lib/systemd/coredump/core.hellowindow.0.6c33eef9bac74582aec1a0ffdd7ce1a6.3176.1718634583000000000000.xz
       Message: Process 3176 (hellowindow) of user 0 dumped core.
                
                Stack trace of thread 3176:
                #0  0x00000000b63d1208 raise (libc.so.6 + 0x2d208)

GNU gdb (GDB) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-buildroot-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/qt/examples/opengl/hellowindow/hellowindow...done.
[New LWP 3176]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `/usr/lib/qt/examples/opengl/hellowindow/hellowindow'.
Program terminated with signal SIGABRT, Aborted.
#0  0xb63d1208 in raise () from /lib/libc.so.6
(gdb) bt
#0  0xb63d1208 in raise () from /lib/libc.so.6
#1  0xb63bb2f4 in abort () from /lib/libc.so.6
#2  0xb67b1d04 in QMessageLogger::fatal(char const*, ...) const () from /lib/libQt5Core.so.5
#3  0xb5fbb280 in QEglFSIntegration::initialize() () from /lib/libQt5EglFSDeviceIntegration.so.5
#4  0xb6c9454c in QGuiApplicationPrivate::eventDispatcherReady() () from /lib/libQt5Gui.so.5
#5  0xb690ce20 in QCoreApplicationPrivate::init() () from /lib/libQt5Core.so.5
#6  0xb6c959c8 in QGuiApplicationPrivate::init() () from /lib/libQt5Gui.so.5
#7  0xb6c964a0 in QGuiApplication::QGuiApplication(int&, char**, int) () from /lib/libQt5Gui.so.5
#8  0x004a7a28 in main (argc=<optimized out>, argv=<optimized out>) at main.cpp:63
(gdb)

Hi Davis,

The expert on this matter is currently out of office. Please, expect some delays in the response. 

Bests,

Qutaiba

Hello Qutaiba, do you know when the subject matter expert will be back? 

Hi,

Please help me understand your setup here. Are you using Debian or Yocto?

I am not too familiar with Debian and their infrastructure but for Yocto we need 3 things in order for the GPU to function correctly:

1. Yocto Project Version: The glyph issue on AM335x was fixed in Yocto project Dunfell. This is important because there is a mesa component and GPU libraries are built against a certain mesa version.

2. GPU userspace libraries: https://git.ti.com/cgit/graphics/omap5-sgx-ddk-um-linux/tree/targetfs?h=ti-img-sgx/dunfell/1.17.4948957

3. GPU Kernelspace libraries: https://git.ti.com/cgit/graphics/omap5-sgx-ddk-um-linux/tree/?h=ti-img-sgx/dunfell/1.17.4948957

All three components needs to match otherwise GPU will not function. In Yocto, we take care of all 3 components but I don't know how Debian works and you may need to check with the Debian community. 

Regards,
Krunal

Hi Krunal,

We are using buildroot and have no ability to switch to Yocto at this stage of the project.

a) GPU user space libraries - I am already pointing to the tip of https://git.ti.com/cgit/graphics/omap5-sgx-ddk-um-linux/tree/targetfs?h=ti-img-sgx/dunfell/1.17.4948957 which corresponds to commit: 742cf38aba13e1ba1a910cf1f036a1a212c263b6

b) GPU kernel space libraries - I am already pointing to the tip of https://git.ti.com/git/graphics/omap5-sgx-ddk-linux.git which corresponds to commit: bfe83bbabb3849c24b03d5172cf678e7c5915e04

If you clone the above two responds, you can confirm on your end as well.

In regards to point (1) that you made in your previous message, please be more specific regarding which mesa package you're referring to. I understand that you keep saying the fix is in yocto dunfell but it really doesn't matter what buildsystem is used. We just need to identify the exact needed packages and their specific commits. 

Please let me know how to proceed.

Thank you,

Davis  

Hi Davis,

Refer to the following package config file: https://git.ti.com/cgit/graphics/omap5-sgx-ddk-um-linux/tree/targetfs/ti335x/lib/pkgconfig/egl.pc?h=ti-img-sgx/dunfell/1.17.4948957. Is your buildroot mesa version 19.1.6? 

Also, try running glmark2-drm test and if that runs fine we know what Qt using using different mesa compared to GPU libs.

Regards,
Krunal

Hi Krunal,

We are currently using SDK-06.01.00.08 therefore we are not using mesa in our build. This puts us in a difficult situation because if we need to do a BSP upgrade to your latest revision then this brings a lot of risk. Is it possible to get a patch for our specific BSP version?

Additionally, the reason we're on an older bsp is because of issues that our developers found in the past. I attempted to attach a patch to this thread which explains in more detail the issue that we found but am unable to do so so instead I've email the patch to Brandon Elliot, our TI FAE, and hopefully he'll forward to you.

Thank you,

Davis

Hi Davis,

Could you try the attached UM libraries? All you need to do is extract the folder and copy the files to your filesystem. On my side, I tested on the AM335 SK EVM and GPU was stable. I ran various Qt tests, gles2test 1 and weston based examples were all passing.

With regards to Yocto, I made the following change for SDK 6.01 build: Change the git://arago-project.org/git/meta-arago.git to https://git.ti.com/git/arago-project/meta-arago.git in the txt file.

Regards,
Krunal

gpu_libs.tar.gz

Hi Krunal,

Thank you for providing the tarball unfortunately I see many files and this is a bit of a problem.

Based on our conversation, you were clear that the gpu glyph fix resides in the gpu user module therefore would you agree that it is sufficient to just copy over libsrv_um.so?

I just want to verify that the fix does in fact only reside in the user mode.

Please let me know what you think.

Thank you,

Davis

Hi Davis,

Unfortunately, the user mode consists of the all the libraries under the /usr/lib folder. At minimum, you need to update all the shared libraries. The bin, include folder an dinit folders probably don't impact anything.

Regards,
Krunal

Hi Krunal,

Since we do not have source code for these user mode binaries that were just provided we have no way to track what patch or patches were applied. Our concern is that if we have another issue in the future, we need to be able to convey clearly what was already done so we need some metadata that we can store on our end to describe the patch that was applied. Do you have a git hash or something that you think would be useful for us to keep on our end for traceability?

Thank you,

Davis

Hi Davis,

Before we look into the git/metadata request, could you please confirm if the libraries worked with your buildroot system and the glyph issue was resolved?

Regards,
Krunal

Hi Krunal,

I have integrated the binaries that you provided into buildroot and have yet to see the issue. As a result, I provided a build to our testing teams in Torrance and they're currently running further testing along with regression testing and so far it looks good.

Thank you,

Davis 

Thanks for the update Davis! With regards to the git request, could you please give me 1-2 days to discuss internally? I need to check with our dev team around the feasibility and logistics. 

Regards,
Krunal

Hi Davis,

Apologies for the delay. Based on my internal discussions, we will host the updated binaries on the following repo: https://git.ti.com/cgit/graphics/omap5-sgx-ddk-um-linux/. Unfortunately, our SW dev team is in the middle of a release cycle and their bandwidth should free up in the next 1-2 days. We plan to meet internally again on Friday to discuss the timelines for hosting the new binaries. I will keep update you as soon as I hear back from SW dev team. 

Regards,
Krunal

Hi Krunal,

Our team is reporting that they're experiencing a side effect of the new patched binaries that were provided. Here are their comments:

Comment 1: "...doing some testing, and the letters and numbers on the latest nightly aren’t rendering very well. "

Comment 2: "Looks pretty much like nice squiggly writing and confusing even if it is intended. HOWEVER, this won’t be acceptable by UX."

I asked our team to confirm that this is in fact a side effect and not a previously existing issue. After checking prior builds, they're saying that they are confident that this is a side effect of the new binaries that were provided as this issue was not present before.

Please let us know how to proceed.

Thank you,

Davis

Hi Davis,

Would it be possible for you to test your UI on TI EVM with SDK 7.x and 9.x Linux images? I am trying to understand if the regression is introduced by my changes or it is present in the GPU driver by default.

Regards,

Krunal

Hi Krunal, 

Sorry, perhaps I wasn't clear but we tested prior builds (before your new binaries were introduced containing the glyph fix) and our testing shows that this recent issue is NOT present. These old builds contain ti-sgx-um commit 909e237baf47d0bde006ff25552f5403fd7e359d (http://git.ti.com/git/graphics/omap5-sgx-ddk-um-linux.git) and we've been on this commit since the start of the project. 

Additionally, as far as testing the latest build on the TI EVM, this would require a big porting effort so no, this is not a feasible path forward.

Lastly, this is also why I'm very concerned about the use of opaque binary files as it leaves us in the dark without any traceability. We've been on commit 909e237baf47d0bde006ff25552f5403fd7e359d since the start of the project and I have no way to confirm that your patches were made onto of this commit or some other commit. Can we please confirm that the patch was applied on this commit?

Thank you,

Davis

Hi Davis,

Okay, I found the old archives and was able to map the UM repo(omap5-sgx-ddk-um-linux) to our internal UM source code repo. Next, I only applied the additional patch that IMG gave us to address the glyph issue. I have attached the new GPU libs in the ticket and in summary, the attached libs correspond to the commit 909e237baf47d0bde006ff25552f5403fd7e359d plus the one additional IMG patch.

If the second issue still persist then I believe it could be the side effect of the IMG patch. In order to confirm, it would be useful if there was a standalone QT UI app that we can test on TI EVM. We don't need your entire system ported on our EVM and just a simple Qt GUI app that shows the issue. 

Regards,
Krunal

gpu_libs_v2.tar.gz