Linux/AM3354: AM3354 3D Graphic Issue

+ files.

0842.booting_log1.txt

8750.booting_log2.txt

1830.Issue_log.txt

4375.dmesg-log1.txt

7455.dmesg_log2.txt

Hi Team,
Together with customer, we checked the AM335x configuration. It was running @ 800MHz, DDR3@303MHz, CORE@OPP100. There seems nothing went wrong. However, we also can easily reproduce this issue. Any idea to debug this issue?
[ 1461.396508] PVR_K: HWRecoveryResetSGX: SGX Hardware Recovery triggered
[ 1461.403442] PVR_K: SGX debug (SGX_DDK_Linux_CustomerTI sgxddk 19 1.9@2188537)
[ 1461.410970] PVR_K: (P0) EUR_CR_CORE_ID: 01120000
[ 1461.416663] PVR_K: (P0) EUR_CR_CORE_REVISION: 00010205
[ 1461.422349] PVR_K: (P0) EUR_CR_EVENT_STATUS: 20000000
[ 1461.428049] PVR_K: (P0) EUR_CR_EVENT_STATUS2: 00000010
[ 1461.433743] PVR_K: (P0) EUR_CR_BIF_CTRL: 00000000
[ 1461.439429] PVR_K: (P0) EUR_CR_BIF_INT_STAT: 00000000
[ 1461.445124] PVR_K: (P0) EUR_CR_BIF_FAULT: 00000000
[ 1461.450811] PVR_K: (P0) EUR_CR_BIF_MEM_REQ_STAT: 00000000
[ 1461.456495] PVR_K: (P0) EUR_CR_CLKGATECTL: 00222220
[ 1461.462174] PVR_K: (P0) EUR_CR_PDS_PC_BASE: 0010004C
[ 1461.467870] PVR_K: Flip Command Complete Data 0 for display device 1:
[ 1461.474654] PVR_K: SRC 0: (Not in use)
[ 1461.478683] PVR_K: SRC 1: (Not in use)
[ 1461.482719] PVR_K: SGX Host control:
[ 1461.486498] PVR_K: (HC-0) 0x00000001 0x00000000 0x00000000 0x00000001
[ 1461.493377] PVR_K: (HC-10) 0x00000001 0x00000000 0x0000000A 0x00030D40
[ 1461.500334] PVR_K: (HC-20) 0x00000000 0x00000001 0x00000001 0x00000000
[ 1461.507501] PVR_K: (HC-30) 0x00000000 0x00163BC5 0xEE5FD2D0 0x00000000
[ 1461.514482] PVR_K: (HC-40) 0x00000000 0x00000000 0x00000000 0x00000000
[ 1461.521444] PVR_K: SGX TA/3D control:
[ 1461.525312] PVR_K: (T3C-0) 0x0F003000 0x0F003120 0x0F002000 0x00000000
[ 1461.532267] PVR_K: (T3C-10) 0x00000000 0x00000000 0x00000002 0x00000000
[ 1461.539331] PVR_K: (T3C-20) 0x00000000 0x00000000 0x00000000 0x00000000
[ 1461.546391] PVR_K: (T3C-30) 0x00000000 0x00000000 0x00000000 0x00000000
[ 1461.553454] PVR_K: (T3C-40) 0x00000000 0x00000000 0x00000000 0x00000000
[ 1461.560558] PVR_K: (T3C-50) 0x00000000 0x00000000 0x00000000 0x00000000
[ 1461.567622] PVR_K: (T3C-60) 0x00000000 0x00000000 0x00000000 0x00000000
[ 1461.574690] PVR_K: (T3C-70) 0x00000000 0x00000000 0x00000000 0x00000000
[ 1461.581739] PVR_K: (T3C-80) 0x00000000 0x00000000 0x0F0AD55C 0x0F000000
[ 1461.588803] PVR_K: (T3C-90) 0x9E01D000 0x0F091480 0x0F0AD55C 0x0F08B920
[ 1461.595913] PVR_K: (T3C-A0) 0x0F00AEA0 0x0F0AD518 0x0F08B920 0x00000000
[ 1461.602964] PVR_K: (T3C-B0) 0x00000000 0x00000000 0x00000000 0x00000000
[ 1461.610031] PVR_K: (T3C-C0) 0x00000000 0x00000000 0x00007118 0x00007117
[ 1461.617088] PVR_K: (T3C-D0) 0x0F000000 0x8000B000 0x8004B000 0x0F004000
[ 1461.624150] PVR_K: (T3C-E0) 0x0F00A420 0x0F00A740 0x0F08B000 0x0F08B000
[ 1461.631201] PVR_K: (T3C-F0) 0x00000000 0x000003DC 0x000003DC 0x00000000
[ 1461.638280] PVR_K: (T3C-100) 0x00000002 0x00000000 0x1BE6B1F6 0x00000001
[ 1461.645436] PVR_K: (T3C-110) 0x00000000 0x00000000 0x00000000 0x00000000
[ 1461.652575] PVR_K: SGX Kernel CCB WO:0xE2 RO:0xE2
[ 4993.343989] PHY: 0:00 - Link is Down
[ 5051.313109] PVR_K: HWRecoveryResetSGX: SGX Hardware Recovery triggered
[ 5051.320019] PVR_K: SGX debug (SGX_DDK_Linux_CustomerTI sgxddk 19 1.9@2188537)
[ 5051.327553] PVR_K: (P0) EUR_CR_CORE_ID: 01120000
[ 5051.333248] PVR_K: (P0) EUR_CR_CORE_REVISION: 00010205
[ 5051.338927] PVR_K: (P0) EUR_CR_EVENT_STATUS: 20000000
[ 5051.344632] PVR_K: (P0) EUR_CR_EVENT_STATUS2: 00000010
[ 5051.350312] PVR_K: (P0) EUR_CR_BIF_CTRL: 00000000
[ 5051.356001] PVR_K: (P0) EUR_CR_BIF_INT_STAT: 00004002
[ 5051.361684] PVR_K: (P0) EUR_CR_BIF_FAULT: 006E8000
[ 5051.367376] PVR_K: (P0) EUR_CR_BIF_MEM_REQ_STAT: 00000002
[ 5051.373063] PVR_K: (P0) EUR_CR_CLKGATECTL: 00222220
[ 5051.378748] PVR_K: (P0) EUR_CR_PDS_PC_BASE: 00000000
[ 5051.384439] PVR_K: Found MMU context for page fault 0x006e8000
[ 5051.390578] PVR_K: GPU memory context is for PID=988 (MM)
[ 5051.396266] PVR_K: PDE valid: PTE = 0x00000000 (PhysAddr = 0x00000000, Invalid)
[ 5051.403969] PVR_K: Flip Command Complete Data 0 for display device 1:
[ 5051.410750] PVR_K: SRC 0: (Not in use)
[ 5051.414797] PVR_K: SRC 1: (Not in use)
[ 5051.418830] PVR_K: SGX Host control:
[ 5051.422601] PVR_K: (HC-0) 0x00000001 0x00000000 0x00000000 0x00000001
[ 5051.429478] PVR_K: (HC-10) 0x00000001 0x00000001 0x0000000A 0x00030D40
[ 5051.436451] PVR_K: (HC-20) 0x00000000 0x00000000 0x00000003 0x00000000
[ 5051.443420] PVR_K: (HC-30) 0x00000000 0x004D010F 0x453D5F50 0x00000000
[ 5051.450392] PVR_K: (HC-40) 0x00000000 0x00000000 0x00000000 0x00000000
[ 5051.457354] PVR_K: SGX TA/3D control:
[ 5051.461209] PVR_K: (T3C-0) 0x0F003000 0x0F003120 0x0F002000 0x00000000
[ 5051.468180] PVR_K: (T3C-10) 0x00000000 0x00000000 0x00000002 0x00000000
[ 5051.475242] PVR_K: (T3C-20) 0x00000000 0x00000000 0x00000000 0x00000000
[ 5051.482289] PVR_K: (T3C-30) 0x00000000 0x00000000 0x00000000 0x00000000
[ 5051.489350] PVR_K: (T3C-40) 0x00000000 0x00000000 0x00000000 0x00000000
[ 5051.496408] PVR_K: (T3C-50) 0x00000000 0x00000000 0x00000000 0x00000000
[ 5051.503465] PVR_K: (T3C-60) 0x00000000 0x00000000 0x00000000 0x00000000
[ 5051.510520] PVR_K: (T3C-70) 0x00000000 0x00000000 0x00000000 0x00000000
[ 5051.517577] PVR_K: (T3C-80) 0x00000000 0x0F0AD55C 0x00000000 0x0F000000
[ 5051.524637] PVR_K: (T3C-90) 0x9E01D000 0x0F092400 0x0F0AD55C 0x0F08B920
[ 5051.531685] PVR_K: (T3C-A0) 0x0F00AEA0 0x0F0AD518 0x0F08B920 0x00000000
[ 5051.538751] PVR_K: (T3C-B0) 0x00000000 0x00000000 0x00000000 0x00000000
[ 5051.545816] PVR_K: (T3C-C0) 0x00000000 0x00000000 0x0001897C 0x0001897B
[ 5051.552864] PVR_K: (T3C-D0) 0x0F000000 0x8000B000 0x8004B000 0x0F004000
[ 5051.559926] PVR_K: (T3C-E0) 0x0F00A420 0x0F00A740 0x0F08B000 0x0F08B000
[ 5051.566981] PVR_K: (T3C-F0) 0x00000000 0x000003DC 0x000003DC 0x00000000
[ 5051.574049] PVR_K: (T3C-100) 0x00000003 0x00000000 0x1BE6B1F6 0x00000004
[ 5051.581192] PVR_K: (T3C-110) 0x00000000 0x00000000 0x00000000 0x00000000
[ 5051.588344] PVR_K: SGX Kernel CCB WO:0x82 RO:0x81

BTW, we did a test to show a static picture without using OPENGL/3D. The result is that it could work well for a long time without issues.

You are using very old SDK and corresponding driver. AM335x LCDC has vulnerability to race condition under certain circumstances. Older LCDC driver was more prone to such race condition. Newer kernel driver was architect again to reduce race condition issue. I wonder if you are facing that problem in the system, having it's impact on SGX, thereby misleading it as SGX issue. Check these e2e post and wiki post - 

https://e2e.ti.com/support/arm/sitara_arm/f/791/t/198769

http://e2e.ti.com/support/arm/sitara_arm/f/791/p/446385/1848230#pi316653=6

http://processors.wiki.ti.com/index.php/DA8xx_LCDC_Linux_FB_FAQs#Flicker_due_to_underflow_error

Hi Steven,

Could you also let us know what kind of example application is able to reproduce the issue? Is it SGX-based example application from Imagination? If yes, which example is it? Also, which window system is being used - X11 or fullscreen mode??

Hi Manisha,
Many thanks for your reply!
According to the feedback from customer, they already used this patch in their application. All these issue description and reports are based on the software which used patch already.

//if ((stat & LCD_SYNC_LOST) && (stat & LCD_FIFO_UNDERFLOW)) {
if ((stat & LCD_SYNC_LOST) || (stat & LCD_FIFO_UNDERFLOW)) {
//printk(KERN_ERR "LCDC sync lost or underflow error occured\n");
lcd_disable_raster(NO_WAIT_FOR_FRAME_DONE);
lcdc_write(stat, LCD_MASKED_STAT_REG);
lcd_enable_raster();

Hi Manisha,
The example application is ti-sdk-am335x/Graphics_SDK_4_09_00_01/gfxsdkdemos/ogles2/OGLES2Water.
It was running in fullscreen mode.
We are pending here without any further debug approach, could you help? Many thanks!

Hi Steven,

We tried to run this demo on our EVM board using latest Processor Linux SDK (3.3)  but couldn't see the issue you reported. Could you provide input to below questions?

1. Is this the only ogles2 application showing the problem?
2. Does this application fails only on customer board or AM335x GP EVM/BBB?
3. What is the reproducible rate of the failure?
4. Can this be reproduced using a PVRTrace of the application?

Regards,

Manisha

Hi Steven,

Is  this issue resolved?? Shall we close the ticket?

Regards,

Manisha

Hi Manisha,
Thanks for your following up!
I'm sorry that this issue is not solved yet, customer is still trying to find the root cause by quantities of tests. If there's further update, I'll let you know! Many thanks!