Linux/AM3352: C1 state issue

Akira Kobayashi

Part Number: AM3352

Tool/software: Linux

Hi Experts,

Regarding a system using AM3352, we have a symptom that CPU usage sys for the whole system jumps to 60-70% at a certain point.

Though such heavy CPU load remains once the symptom occurs, it can temporarily be resolved as changing CPU frequency by DVFS.

As looking into it, we have found that bit0-2 of CM_CLKMODE_DPLL_MPU register (0x44e0_0488) is changed from 7(Lock mode) to 5 (Idle Bypass Low Power) while having this symptom.

The register is changed by M3 Firmware for PM and seems like the symptom occurs when adopting C1(Bypass MPU PLL) for idle state.

While we have been able to improve the problem by giving the following modification, we are not certain if this is the right way.

Could you please verify it for us?

Note that we used the Firmware version 0x192 to confirm.

diff --git a/src/foundation/ext_intr_handlers.c b/src/foundation/ext_intr_handlers.c

index 31f3728..be991a1 100644

--- a/src/foundation/ext_intr_handlers.c

+++ b/src/foundation/ext_intr_handlers.c

@@ -70,7 +70,8 @@ void extint34_handler(void)

for (i = 0; i < CM3_NUM_EXT_INTERRUPTS; i++)

{

nvic_disable_irq(i);

- nvic_clear_irq(i);

+ if (i != CM3_IRQ_MPU_WAKE)

+ nvic_clear_irq(i);

}

msg_cmd_dispatcher();

over 6 years ago

0 Biser Gatchev-XID over 6 years ago

TI__Guru**** 393215 points

Which Linux version is this?

0 Pavel Botev over 6 years ago

TI__Guru**** 170625 points

Hi Akira,

Akira Kobayashi said:
we have a symptom that CPU usage sys for the whole system jumps to 60-70% at a certain point.

Akira Kobayashi said:

As looking into it, we have found that bit0-2 of CM_CLKMODE_DPLL_MPU register (0x44e0_0488) is changed from 7(Lock mode) to 5 (Idle Bypass Low Power) while having this symptom.

The register is changed by M3 Firmware for PM and seems like the symptom occurs when adopting C1(Bypass MPU PLL) for idle state.

CPUIdle state should be entered only when the Linux scheduler has no thread to run. Which means that CPUIdle is entered when there is no CPU usage. Looks like you somehow enter CPUIdle state while CPU is still working, which is NOT correct. Please refer to the below wiki pages for details of the proper CPUIdle usage:

http://processors.wiki.ti.com/index.php/Linux_Core_Power_Management_User's_Guide_(v4.4)#CPUIdle

Regards,
Pavel

0 Akira Kobayashi over 6 years ago in reply to Biser Gatchev-XID

Prodigy 70 points

Hi Biser,

Linux kernel Version is 3.12.10 with 'ARM: AM33XX: Add CPU idle support' patch.
PM: CM3 Firmware Version = 0x185
By the way, we tried up to 0x193 of CM3 Firmware Version, but we could not confirm the improvement.

Best Regards,
Akira

0 Akira Kobayashi over 6 years ago in reply to Pavel Botev

Prodigy 70 points

Hi Pavel,

I found that it goes into idle state depending on status of each process even when CPU is relatively busy.
At that time, with the kernel mentioned above, CMD_ID_CPUIDLE_V2(0xd) command of M3 is used to move into idle state.
In such case, the following two in M3 firmware must always be called together (I mean to be paired), but there are times that the latter isn’t called.
　a8_cpuidle_v2_handler() of
　a8_wake_cpuidle_v2_handlerr()
I guess that’s why the value of CM_CLKMODE_DPLL_MPU register isn’t restituted.

Best Regards,
Akira

0 Pavel Botev over 6 years ago in reply to Akira Kobayashi

TI__Guru**** 170625 points

Akira,

Akira Kobayashi said:
I found that it goes into idle state depending on status of each process even when CPU is relatively busy.

That is NOT correct. The AM335x Cortex-A8 MPU should automatically enter into low power Idle state ONLY when there is no more processes to run. Seems that you are using very old AM335x TI SDK7 (kernel 3.12.10) where CPUIdle is not implemented by default, and you implement CPUIdle by yourself, but there is something wrong with your own implementation.

Please try to reproduce this issue with the latest AM335x TI PSDK4.02 (kernel 4.9.59) where CPUIdle is implemented by default. If you are not able to reproduce the issue with latest PSDK, then you should double check your custom CPUIdle implementation in SDK7.

Regards,
Pavel

0 Akira Kobayashi over 6 years ago in reply to Pavel Botev

Prodigy 70 points

Hi Pavel,

Thank you for your comments.

Kernel upgrade must take a while for me since I’m using a dedicated hardware.

Well, I believe 4.x kernel uses CMD_ID_CPUIDLE(0x10) command of M3 to move into idle state.

drivers/soc/ti/wkup_m3_ipc.c
#define IPC_CMD_IDLE			0x10

Could you let me know if CMD_ID_CPUIDLE_V2(0xd) would work instead?

Regards,
Akira

0 Pavel Botev over 6 years ago

TI__Guru**** 170625 points

Akira Kobayashi said:
As looking into it, we have found that bit0-2 of CM_CLKMODE_DPLL_MPU register (0x44e0_0488) is changed from 7(Lock mode) to 5 (Idle Bypass Low Power) while having this symptom.

CPUIdle lets the system enter low power states during periods of no activity. The idle loop is executed when the Linux scheduler has no thread to run. When the idle loop is executed, current 'governor' is called to decide the target C-state. A C-state is used to identify the power state supported through the cpu idle loop.

AM335x device support two C-states:

1. MPU WFI

2. MPU WFI + Bypass MPU DPLL

Looks like you enter the second C-state when enter the CPUIdle low power state.

Regards,
Pavel

0 Pavel Botev over 6 years ago in reply to Akira Kobayashi

TI__Guru**** 170625 points

Akira Kobayashi said:

Kernel upgrade must take a while for me since I’m using a dedicated hardware.

Well, I believe 4.x kernel uses CMD_ID_CPUIDLE(0x10) command of M3 to move into idle state.

drivers/soc/ti/wkup_m3_ipc.c
#define IPC_CMD_IDLE			0x10

Could you let me know if CMD_ID_CPUIDLE_V2(0xd) would work instead?

We have some info in the below e2e threads, please have a look:

I was also able to find the below info:

Problem:

Sometimes, when exiting by this idle state, it seems that the MPU PLL is still in bypass, and thus they have very poor ARM core performance.
The situation come back to normal when entering again C1 idle state, because sometimes the MPU PLL is functional again. The main drawback of this is that, under certain load, when MPU PLL is bypass the CPU stuck at 100% usage and, thus, never goes again in C1 idle.

The ONLY proposed solution is a workaround that disable C1 state completely. This has, of course, some drawbacks on power consumption.

Another workaround, that keep C1 state enabled but force MPU PLL lock if you find it in bypass. As soon as the Cortex-A8 comes out of WFI we check that the MPU DPLL is always locked. However, if we read the AM33XX_CM_CLKMODE_DPLL_MPU register a few times (usually > 50) sometimes we saw that it changes from 0x07 value (which means that the DPLL is configured as locked) and, in those cases, we experienced the problem. We try a dirty hack by locking the DPLL is we found that it's unlocked, and we never experience the issue again.

AFAIK this means that something else (Cortex-M3) for some reason (synchronization problem?) in some situation tell the DPLL to go in bypass mode while the Cortex-A8 already resumed from WFI.

Proposed solution:

He needs to change

#define IPC_CMD_IDLE 0xd

in arch/arm/mach-omap2/pm33xx.h

to

#define IPC_CMD_IDLE 0x10

This was eventual fix in the v3.14 kernel, he will need to do the same in v3.12.

This causes the MPU Clock Domain to go to sleep but prevents BYPASS of the MPU PLL which did not save a significant amount of power anyway.

The clock is gated in both cases and that is where the majority of the power reduction comes from, the bypass of the MPU PLL is what causes the issue, and it's not worth using it because the amount of power it saves is minimal compared to the clock gating itself.

Regards,
Pavel

0 Akira Kobayashi over 6 years ago in reply to Pavel Botev

Prodigy 70 points

Hi Pavel,

Thank you so much for your sincere correspondence to my question.

Disabling C1 state completely seems to be a good workaround for my case.
It may be due to 3.12 kernel, but there are times that a8_cpuidle_handler() and a8_wake_cpuidle_handler() of M3 firmware are not called together.
It won't be as big a trouble as the case of CMD_ID_CPUIDLE_V2(0xd) though since fewer registers are modified.
I would appreciate it if you could have a chance to check and confirm the M3 firmware patch I previously posted.

diff --git a/src/foundation/ext_intr_handlers.c b/src/foundation/ext_intr_handlers.c
index 31f3728..be991a1 100644
--- a/src/foundation/ext_intr_handlers.c
+++ b/src/foundation/ext_intr_handlers.c
@@ -70,7 +70,8 @@ void extint34_handler(void)
for (i = 0; i < CM3_NUM_EXT_INTERRUPTS; i++)
{
nvic_disable_irq(i);
- nvic_clear_irq(i);
+ if (i != CM3_IRQ_MPU_WAKE)
+ nvic_clear_irq(i);
}

msg_cmd_dispatcher();

Regards,
Akira

0 Pavel Botev over 6 years ago in reply to Akira Kobayashi

TI__Guru**** 170625 points

Akira,

Akira Kobayashi said:
I would appreciate it if you could have a chance to check and confirm the M3 firmware patch I previously posted.

diff --git a/src/foundation/ext_intr_handlers.c b/src/foundation/ext_intr_handlers.c
index 31f3728..be991a1 100644
--- a/src/foundation/ext_intr_handlers.c
+++ b/src/foundation/ext_intr_handlers.c
@@ -70,7 +70,8 @@ void extint34_handler(void)
for (i = 0; i < CM3_NUM_EXT_INTERRUPTS; i++)
{
nvic_disable_irq(i);
- nvic_clear_irq(i);
+ if (i != CM3_IRQ_MPU_WAKE)
+ nvic_clear_irq(i);
}

msg_cmd_dispatcher();

Do you mean you want to disable CPUIdle C1 state (MPU WFI and MPU PLL bypass) with applying that patch only and nothing more/else?

Regards,
Pavel

0 Akira Kobayashi over 6 years ago in reply to Pavel Botev

Prodigy 70 points

Hi Pavel-san,

When I apply this patch, I don't meet this issue if entering CPU idle C1 state.
I'd like to ask your team to review this patch.

Regars,
Akira

0 Pavel Botev over 6 years ago in reply to Akira Kobayashi

TI__Guru**** 170625 points

Akira,

I understand.

You apply that patch on top of CM3 firmware version 192 (link below), is that correct?

git.ti.com/.../ti-v4.1.y

I will check with the CM3 firmware owner regarding patch review.

Regards,
Pavel

0 Akira Kobayashi over 6 years ago in reply to Pavel Botev

Prodigy 70 points

Hi Pavel,

Thank you for your understanding.
The previous patch is for v05.00.00.02 (0x185).
I believe that this patch can also be applied to ti-v4.1.Y (0x192), but the patch for ti-v4.1.Y (0x192) is as follows.

--- ext_intr_handlers.c.orig    2016-09-07 05:29:38.000000000 +0900
+++ ext_intr_handlers.c    2018-03-03 08:17:18.381015900 +0900
@@ -79,7 +79,8 @@
     for (i = 0; i < CM3_NUM_EXT_INTERRUPTS; i++)
     {
         nvic_disable_irq(i);
-        nvic_clear_irq(i);
+        if (i != CM3_IRQ_MPU_WAKE)
+            nvic_clear_irq(i);
     }
     if (soc_id == AM43XX_SOC_ID) {
         nvic_disable_irq(CM3_IRQ_TPM_WAKE);

Regards,
Akira

0 Pavel Botev over 6 years ago in reply to Akira Kobayashi

TI__Guru**** 170625 points

Akira,

This is the feedback from CM3 firmware owner:

From looking at the e2e thread you appear to be on 3.12 kernel. You do not want to modify the firmware, this issue was addressed in 3.14 kernel with the patch below. Have you been able to measure power consumption with your firmware patch? Just from a quick glance I would bet it is high. From what I can tell you are effectively preventing anytime spent in c-state by making the system wake up right away. I included the full patch below because the commit message explains why switching cpu idle modes is a better solution. It has a negligible effect on power consumption.

commit dd1af75f2773ffb2060aaad51e04d23fce0cddd1
Author: Dave Gerlach <d-gerlach@ti.com>
Date:   Thu Mar 26 13:41:47 2015 -0500

    ARM: OMAP2+: cpuidle33xx: Change wkup_m3 idle state to avoid MPU PLL bypass

    To get to C1-state in cpuidle am33xx currently uses the wkup_m3 to gate
    the MPU clock domain and bypass the MPU PLL. Because we are not shutting
    off the MPU power domain in this C-state it is possible for both to be
    awake and executing at the same time, and we do not have the ability
    to synchronize execution between the two in the cpuidle path due to
    no interrupt context.

    This leads to racy behavior between the two and in certain instances
    where c1-state is entered often during periods of high cpu activity it
    is possible for the MPU to interfere with the wake path on the wkup_m3,
    which prevents the MPU PLL from being re-locked and causing extreme
    system slowdown.

    Because of this, we must now use a different idle state available on the
    current wkup_m3 firmware (0x190) which does nothing more than put the
    MPU clockdomain to sleep, giving us the same power savings seen
    previously on the MPU power rail but slightly higher power on the
    MPU_PLL voltage rail.

    Signed-off-by: Dave Gerlach <d-gerlach@ti.com>

diff --git a/arch/arm/mach-omap2/pm33xx.h b/arch/arm/mach-omap2/pm33xx.h
index 8c4f45995b63..7b6a6f7164ba 100644
--- a/arch/arm/mach-omap2/pm33xx.h
+++ b/arch/arm/mach-omap2/pm33xx.h
@@ -65,7 +65,7 @@ void __iomem *omap_rtc_get_base_addr(void);
#define IPC_CMD_RTC_ONLY               0x1
#define        IPC_CMD_DS0                     0x4
#define        IPC_CMD_STANDBY                 0xc
-#define        IPC_CMD_IDLE                    0xd
+#define        IPC_CMD_IDLE                    0x10
#define IPC_CMD_RESET                  0xe
#define DS_IPC_DEFAULT                 0xffffffff
#define M3_VERSION_UNKNOWN             0x0000ffff

Processors

Processors forum

Linux/AM3352: C1 state issue