AM437x ondemand DVFS causing random crashes

Robert Woerle

Other Parts Discussed in Thread: TPS65218

Dear List

We are facing a reproducible issue with the am437x on a ti kernel 4.1.18 when enabling cpufreq "ondemand" governor.

If we keep the default Opp Modes which include frequency and voltage scaling on vdd_mpu ( we do exclude OPP 50 - 300Mhz from the list) we still can cause the kernel to panic or oops on various points.

The only way to "fix this issue" is changing the freq/volt table and fix all the voltages to 1.325v ( which is the voltage for 1Ghz mode). like this at arch/arm/mach-omap2/opp43xx_data.c:

static struct omap_opp_def am43xx_es1_0_opp_list[] __initdata = {
   /* MPU OPP1 - OPP50 */
   OPP_INITIALIZER("mpu", false, 300000000, 1325000),
   /* MPU OPP2 - OPP100 */
   OPP_INITIALIZER("mpu", false, 600000000, 1325000),
   /* MPU OPP3 - OPP120 */
   OPP_INITIALIZER("mpu", false, 720000000, 1325000),
   /* MPU OPP3 - OPPTurbo */
   OPP_INITIALIZER("mpu", false, 800000000, 1325000),
   /* MPU OPP4 - OPPNitro */
   OPP_INITIALIZER("mpu", false, 1000000000, 1325000),
};

So it seems any voltage switching causes an unstable behaviour.

I did notice that u-boot does use also a different voltage table for the IDK kits.

Any hints or things i miss ?

Regards Robert

over 9 years ago

0 Biser Gatchev-XID over 9 years ago

TI__Guru**** 393215 points

Hi,

Is this a custom board?

0 Robert Woerle over 9 years ago in reply to Biser Gatchev-XID

Intellectual 380 points

Hi Bizer
Yes, this is a custom board. It is pretty close on the am437x-gp-evm and also uses tps65218.
If time permits , i try to test that on our am437x-gp-evm board. But maybe you do have some other hints .
Feel free to request any other informations which you might need to debug this !
We would love to get this working , as our target environments are also temperature critical and having this stable, could save us some degrees .

0 Biser Gatchev-XID over 9 years ago in reply to Robert Woerle

TI__Guru**** 393215 points

I will notify the PM experts about this issue. They will post directly here.

0 Biser Gatchev-XID over 9 years ago in reply to Biser Gatchev-XID

TI__Guru**** 393215 points

Offhand I can say that this could signal hardware problems in the power rails on your board (routing and/or decoupling), as we haven't seen a similar complaint on the forum so far.

0 Robert Woerle over 9 years ago in reply to Biser Gatchev-XID

Intellectual 380 points

Hi Bizer

Your hint already brought some light into possible issue.

Hardware Department did add some extra decoupling into the vdd_lines ( just in case) , so we will investigate the actual timing influence of those , which might lead into disturbed ordering in actual vdd switch vs freq switch.

Will report back on that.

0 Brad Griffis over 9 years ago in reply to Biser Gatchev-XID

TI__Guru*** 125430 points

Have you checked the slew rate of your power supplies during OPP changes? If you have a lot more capacitance on your board the software might not be waiting long enough for the rail to stabilize.

0 Robert Woerle over 9 years ago in reply to Brad Griffis

Intellectual 380 points

that is what will will look into as we do have extra capacitance inside (which i just learned from our hardware department)

I will report back , Thank you

0 kshin over 9 years ago in reply to Robert Woerle

TI__Genius 16690 points

You can try userspace governor and see if each OPP works with manual control from userspace governor. Please check voltage level at both VDD_MPU and VDD_CORE power supplies. The voltage level should be only changed for VDD_MPU and VDD_CORE should stay within 1.1V +/- 4% when VDD_MPU OPP is changed from userspace govorner. It's easier to measure accurate core voltage level by measuring voltage across the largest value de-cap for each core power supplies.

0 Michael Erdahl over 9 years ago

TI__Genius 13505 points

Hi Robert,

What is PMIC I2C clock frequency in your device tree? If you're using 400KHz, can you try 100KHz?

Regards,
Mike

0 Brad Griffis over 9 years ago in reply to Michael Erdahl

TI__Guru*** 125430 points

I haven't seen any updates on this thread. Can you please share the latest info from your end?

0 Robert Woerle over 9 years ago in reply to Brad Griffis

Intellectual 380 points

No direkt update yet, we are working on it in between projects.

What i tryed is to follow back the actual driver codes. I found that cpufreq is now using the voltdm subsystem, but our kernel fails to register a proper voltdm device.

Still the procedure do work but i suspect due to the failing voltdm the "voltdm_get_latency() in drivers/power/voltdm/core.c" is not properly detecting the voltage latency.

I tryied to fall back to older setups without voltdm with no success.

I will report , as promised on the outcomes.

0 Dave Gerlach over 9 years ago in reply to Robert Woerle

TI__Intellectual 1830 points

We're you able to validate you i2c bus frequency? We have seen failures exactly as you described in your original post when running the i2c bus at 400kHz as documented here: patchwork.kernel.org/.../

0 Robert Woerle over 9 years ago in reply to Dave Gerlach

Intellectual 380 points

We do run 400khz i2c .. so tomorrow i will retest with a lower speed and validate. I will report back on that.

Thanks for the hint.

Rob

0 Robert Woerle over 9 years ago in reply to Robert Woerle

Intellectual 380 points

Dear List

reducing the i2c speed from 400khz to 100khz does have a significant effect. Before, the hang/crash occurs seconds after enabling the ondemand profile.

Now the systems ( one in a heat chamber) are running stable for over 1 hour. That leads me to the conclusion that this was the case and i will assume this fixed until further notice.

Thanks all for your hints and your support.

Processors

Processors forum

AM437x ondemand DVFS causing random crashes