AM625: kernel errors "rcu stall" running memory test

Jhon ,

Thanks for your Query, I have assigned the thread to our Kernel expert. Please allow some time to get back.

Regards

Anshu

From the log, it doesn't appear the kernel is crashing, just that you are getting the rcu stall errors.  I would say that if the test continues to run, it is probably not a DDR issue.  You are not getting any memtester errors, just kernel errors.  I'll let our software team comment on the kernel errors.  

Regards,

James

Hi James,

Thanks for your reply, we will wait for the software team's response.

Also, are you able to check our DDR configuration (Attached above) against the DDR part we use (K4AAG165WA-BIWE) and verify settings are OK?

Thank you,

Jhon

JJD, Anshu Choudhary,

We have discovered the RCU stalls are reproducible on the TI AM62 EVK. Here are the steps:

Reproduce steps on TI AM62 EVK:

1. Plug in a HDMI display to the EVK

2. Boot https://dr-download.ti.com/software-development/software-development-kit-sdk/MD-PvdSyIiioq/08.06.00.42/tisdk-default-image-am62xx-evm.wic.xz

3. Start 23 instances of memtester (it may happen with more or less, our scripts have been using 23 to leave a certain amount of memory free):

for i in {1..23}; do memtester 50m | grep FAIL >> /tmp/memtest.log & done && \
   sleep 1 && 
   echo $(expr $(free | awk '/^Mem:/{print $4}') / 1024)MB Free

4. Usually the RCU errors will print before 24 hours of running memtester.

We discovered it only happens if the display is enabled. If the HDMI cable is not attached (and therefore fb0 not initialized) or if dss and dss_ports device tree nodes are disabled, the RCU errors will not occur.

After reviewing the reference manual section in the screenshot below, we are wondering if this is related to the QoS configuration. From "3.1.12 QoS Programming Guide":

> AM62 has two real-time peripherals DSS and CSI_RX, both of them have real time deadline.

...

> In addition, GPU can burst out a long back to back burst of transactions to completely exhaust the DDR bandwidth. In order to mitigate the risk of missing real-time peripheral deadline, bandwidth limiters are added to AM62 to control the transactions from both GPU and A53. User can configure and tune the bandwidth limiter parameters based on the use cases.

Can you please confirm this behavior on your side, and advise if 1) the QoS parameters need to be adjusted or 2) some other configuration should be changed?

Thanks,

Nate

Hi Nate,

I will test the above on my setup and get back to you. In general, could you please provide more background on your test case? I am curious why you are running 20+ instances of memtester and attaching a HDMI monitor. Also, is GPU activated and rendering any objects on the screen. 

Regards,
Krunal

Hi Krunal,

> I am curious why you are running 20+ instances of memtester

We are running 20+ instances of memtester to verify our DDR configuration.

> and attaching a HDMI monitor.

On our custom hardware, an LVDS display is always enabled. We discovered that the RCU errors did not occur on our hardware if the dss and dss_ports device tree nodes were disabled. We also discovered that the RCU errors do not occur on the TI AM62 EVK unless the HDMI monitor is attached, since, without an HDMI monitor the framebuffer is not initalized.

> Also, is GPU activated and rendering any objects on the screen.

During our normal memtester testing, the display is showing the weston desktop.

I did try running /usr/bin/SGX/demos/Wayland/OpenGLESBinaryShaders while also running memtester, it did not seem to have any impact.

Thanks! I am trying the above on my setup and will let you know if I observe the same behavior. In general, if HDMI is not connected, I am assuming no stalls are observed even if we increase the number of memtester instances?

Regards,
Krunal

> In general, if HDMI is not connected, I am assuming no stalls are observed even if we increase the number of memtester instances?

I don't know if adding more threads will make it worse. If I recall correctly, I was even able to reproduce this on a 1GB system with ~5 threads.

Thanks for your help investigating this!

Nate

Hi Nate,

Just wanted to update you that I am observing the issue on my side as well and still trying to debug what's causing the stall.

Regards,
Krunal

Hi Nate,

With that many # of memtesters running, the only way to avoid stall is to add swap memory. I added a 1GB of swap memory and did not see the issue.

Regards,
Krunal

Krunal Bhargav34 said:

With that many # of memtesters running, the only way to avoid stall is to add swap memory. I added a 1GB of swap memory and did not see the issue.

Hi Krunal,

I don't think it's related to swap, or running out of memory. During the test, we make sure to leave 500MB of memory free.

Regards,

Nate

Hi Nate,

I came across this document and it might be useful for your usecase: https://docs.kernel.org/RCU/stallwarn.html#fine-tuning-the-rcu-cpu-stall-detector

Regards,
Krunal