AM625: Unexpected TIDSS interrupt result high loading during boot up occasionally.

Hi Tony

Krunal is out of office till end of next week. Responses will be delayed. 

Regards

Mukul 

Hi Tony,

I need more details on how to replicate the above issue on a AM625 SK EVM. Based on your pictures, I see a ton of TIDSS interrupts (not GPU) but you need to provide more details on how the customer collected the above data, when they collected they above data and the impact on their system. 

Regards,
Krunal

Krunal,

it is hard to reproduce, need to  power cycle over several hundred times to produce once. customer log the interrupt every 500ms. when the problem reproduced, system loading became very high, can't launch application smoothly.

How customer find this issue:

#1. the project is under test stage.

#2. some board sometimes system loading very high after boot up.

#3. there is not abnormal in booting message, no abnormal Kernel log.

#4. cat /proc/interrupts, see toms of interupts.

#5. The issue happens on booting up, not during running, so use power cycle test to reproduce.

Hi Tony,

I have yet to reproduce the interrupt issue on my AM625 SK EVM. Is it possible for the customer to use the latest release " 08.05.00.21" and see if the issue is still replicable? Also, does the issue happen on their custom board with custom Kernel or they see the issue on AM62 SK EVM?

Regards,
Krunal

Krunal,

I won't expect BU can reproduce it as it is hard to reproduce on customer side also.

Customer hopes to get help:

#1. what trigger this kind of interrupt.

#2. what register to check for further information、

like if you reproduced, what will you do?

Hi Tony,

It seems like the number of interrupt increases every time a buffer is read by the DSS. For example, I have two terminals running the following contents:

T1 -> kmscube --device=/dev/dri/card0 and T2 -> watch -n1 --no-title cat /proc/interrupts. I can see the tidss (DSS) and pvrsrvkm(GPU) interrupt increase frequently as new content is being rendered every frame.

 When the customer is able to replicate the issue and the performance is low, it would be interesting to know if anything is running in the background that is sending continuous data to the DSS.

Regards,
Krunal

Krunal,

Krunal Bhargav34 said:

 When the customer is able to replicate the issue and the performance is low, it would be interesting to know if anything is running in the background that is sending continuous data to the DSS.

No, as it happened during system booting up, did no run application yet, and even DSS enabled and running, it should generate interrupt based on frame rate. how can user trigger so frequency interrupt?

Hi Tony,

I thought the splash screen gets enabled during the booting up process.

Regards,
Krunal

Krunal,

TI SDK booting stage also enabled splash, but it doesn't generate so frequency interrupt. And with the same software, need about 500 times power cycle to reproduce once.

Hi Tony,

My only recommendation would be to rebuild the filesystem with the following changes:

1. Yocto: Add the following "IMAGE_FEATURES_remove = "splash" " to disable splash
2. Linux Kernel: Disable the following:

# CONFIG_FRAMEBUFFER_CONSOLE is not set

# CONFIG_LOGO is not set

Based on the above settings, nothing should be writing to the DSS framebuffer and the interrupt should not be firing. 

Regards,
Krunal

Krunal Bhargav34 said:

Yocto: Add the following "IMAGE_FEATURES_remove = "splash" " to disable splash

If point out where to add it would help a lot.

Hello,

Are you familiar with Yocto? If yes, there is a local.conf file where you add the above statement.

Regards,
Krunal

Krunal,

if disable TIDSS interrupt, the issue won't happen, and display still work.

but in future with high resolution panel, customer will use double frame buffer, TIDSS still need interrupt feature

But my suggestion does not disable TIDSS interrupt and I am trying to figure out the root cause of the random interrupts. The only application that runs at startup is the splash screen and I am trying to understand if disabling the splash helps. 

Also, does your customer do a hard or soft reset after every test run? 

Regards,
Krunal

Hi Tony,

Any updates? Is there a way to consistently report the above issue on SK EVM? Also, did disabling the splash have any impact?

Regards,
Krunal

Krunal,

Customer use their own filesystem, did no use splash. the DSS interrupt counter increase after driver initialized. usually the counter stop increase at 600+. but ocassionally keep increase, doesn't stop, which lead to high system loading.

Recently customer add below code to avoid the issue temporally. as when in normal condition, the DSS interrupt counter won't increase over 700, so if over 700, disable the DSS interrupt as current application doesn't use interrupt mode to switch frame buffer.

0753.dss-interrupt.txt

Recently we find a bug on CPSW of similar interrupt issue and get resolved, hope it give you guys some clue to analysis this issue.

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1176683/am625-unexpected-ethernet-interrupt-result-in-system-loading

https://jira.itg.ti.com/browse/LCPD-32470

Hi Tony,

On my setup, I made tidss a loadable module by changing the ".config" to "CONFIG_DRM_TIDSS=m". Next, I manually tried to insert and remove the module but I don't see the interrupts increasing.

Next, I tried running the a display based test and I observed the following:
1. Launch test and interrupt increases by 1
2. End the test and interrupt increases by 350.

I am trying to verify if that's an expected behavior but could you please verify if your customer is doing same thing as me with their custom filesystem. They have to be running a display based tests otherwise interrupt should not increase.

Regards,
Krunal

Krunal,

on AM62x-SK, disconnect HDMI cable to power the board, there is no tidss interrupt, but insert HDMI cable, tidss interrupt will be 400+, and stop increasing,  the counter is not same every time.

 6153.tidss-interrupt.txt

I connected a mouse on AM62x-SK, Then I run weston start to display:

root@am62xx-evm:/# /etc/init.d/weston start

the tidss interrupt increased to several thousands, then stop increasing.
Then I use mouse to click display GUI ON/OFF, the tidss interrupt increase again.
8228.interrupt with weston.txt

Question.

Seems the tidss interrupt source is not the DMA completion of frame buffer reading, as although the display content is not updated, DMA activity should always be same, then tidss  interrupt frequency should be same. but actually when display static content, the tidss interrupt count does increase sometimes for a while. When display dynamical connect, for example run the 3D demo, tidss interrupt counter keep increase, which looks more normally.

Even after weston stop, actually on screen still display static Please waiting,.. but tidss interrupt doesn't increase.

So what is the interrupt source of tidss?

Hi Tony,

I am also noticing the same behavior and I am working with our developer to better understand the phenomena. I will keep you posted and please give me 24-48 hours.

Regards,
Krunal 

Hi Tony,

Update: Unfortunately, our DSS developer is sick so he has not looked into the above issue. I will be out of office next week and I will follow up with the developer once I am back during the Feb 27th week.

Regards,
Krunal 

Hi Tony,

I just wanted to update you that we are still looking into the issue. The developer is actively debugging the bug and we are trying to understand why the DSS interrupts increases after frame completion.

Regards,
Krunal

Krunal,

Is there progress?

Hi Tony,

At the moment, no updates and we are still debugging the issue. 

Regards,
Krunal

Hi Tony,

After going through the DRM framework, we found a property called "vblankoffdelay". We are still trying to investigate more about the property but on your setup you can try the following "echo -1 > /sys/module/drm/parameters/vblankoffdelay". The above command will prevent you from observing the extra interrupts after closing an application. Also, the above property is not introduced by our driver and it comes from the DRM driver that the Linux community has developed. 

Regards,
Krunal

Krunal,

How to set it in code to rebuild image, then we can do overnight power cycle test.

Stop at the Uboot prompt and run the following:

=> editenv args_mmc
edit: run finduuid;setenv bootargs console=${console} ${optargs} root=PARTUUID=${uuid} rw rootfstype=${mmcrootfstype} drm.vblankoffdelay=-1
=> saveenv
Saving Environment to MMC... Writing to MMC(0)... OK
=> boot

Regards,
Krunal

Hi Tony,

Any updates?

Regards,
Krunal

Hi Tony,

I am wondering if there are any updates.

Regards,
Krunal

Krunal,

it is still under test, it is not reproduced so far over 1 week continue test,  as it is not easy to reproduce, so customer won't confirm it is solved or not.

before usually can reproduce over one or two nights.

Thanks Tony and please keep us updated.

Regards,

Krunal

Krunal Bhargav34 said:

After going through the DRM framework, we found a property called "vblankoffdelay". We are still trying to investigate more about the property but on your setup you can try the following "echo -1 > /sys/module/drm/parameters/vblankoffdelay". The above command will prevent you from observing the extra interrupts after closing an application. Also, the above property is not introduced by our driver and it comes from the DRM driver that the Linux community has developed. 

Is there further explanation of this?

Hi,
There have been a significant number of updates with respect to this issue. Can you please test with latest SDKs (10.1 or 11.0(preferred)).

Hi Divyansh,

I agree there were a lot of changes, but customer need to know if TI figured out the root cause, it yes, should have record. 

Hi,
The following patch resolved the issue: https://lore.kernel.org/all/20241021-tidss-irq-fix-v1-0-82ddaec94e4a@ideasonboard.com/ 
Issue was diagnosed through another issue mentioned at https://sir.ext.ti.com/jira/browse/EXT_EP-12058 which most likely solves your issue too.

Hi,

It is very like this description, but it is "unclear" , if the root cause clear now?

https://lore.kernel.org/all/20241021-tidss-irq-fix-v1-1-82ddaec94e4a@ideasonboard.com/

It has been observed that sometimes DSS will trigger an interrupt and the top level interrupt (DISPC_IRQSTATUS) is not zero, but the VP and VID level interrupt-statuses are zero.

As the top level irqstatus is supposed to tell whether we have VP/VID interrupts, the thinking of the driver authors was that this particular case could never happen. Thus the driver only clears the DISPC_IRQSTATUS bits which has corresponding interrupts in VP/VID status. So when this issue happens, the driver will not clear DISPC_IRQSTATUS, and we get an interrupt flood.

It is unclear why the issue happens. It could be a race issue in the driver, but no such race has been found. It could also be an issue with the HW. However a similar case can be easily triggered by manually writing to DISPC_IRQSTATUS_RAW. This will forcibly set a bit in the DISPC_IRQSTATUS and trigger an interrupt, and as the driver never clears the bit, we get an interrupt flood.

To fix the issue, always clear DISPC_IRQSTATUS. The concern with this solution is that if the top level irqstatus is the one that triggers the interrupt, always clearing DISPC_IRQSTATUS might leave some interrupts unhandled if VP/VID interrupt statuses have bits set. However, testing shows that if any of the irqstatuses is set (i.e. even if DISPC_IRQSTATUS == 0, but a VID irqstatus has a bit set), we will get an interrupt.

If description in this patch is the root cause?

https://lore.kernel.org/all/20241021-tidss-irq-fix-v1-6-82ddaec94e4a@ideasonboard.com/

Hi,

Tony Tang said:

If description in this patch is the root cause?

Yes, the description of the patch describes the root cause of your issue as well.

There are multiple patches in the same series, some of which also fix the race conditions.

Divyansh Mittal said:

the description of the patch describes the root cause of your issue as well

Which description is the root cause?

The probable race issue was addressed with these patches: https://lore.kernel.org/all/20241021-tidss-irq-fix-v1-6-82ddaec94e4a@ideasonboard.com/ and https://lore.kernel.org/all/20241021-tidss-irq-fix-v1-5-82ddaec94e4a@ideasonboard.com/ 

These 2 descriptions hopefully answer your questions.