Linux LPM driver failing in Codec Engine 2.23.01 sanity_test/app.out

I assume you are using an OMAP3530 device? What silicon revision are you using? I tested on OMAP3530-GP Rev 2. I noticed some difference in behavior going from Rev 1 to Rev 2. Maybe you are on a Rev 3 device and the behavior has changed again? I agree that the board should not make any difference, but my hardware knowledge is very limited. Maybe one some of the pins are not integrated into the board correctly with the proper number of capacitors and/or resistors?

Your analysis looks correct. At the point of failure, LPM is requesting an IVA2 power off state transition and then waiting for the transition to complete. It gives up after a while. My understanding is that the transition will *not* complete if there are any active components within the power domain. However, we know the IVA2 is idle from the loop just above, so it must be something else.

You probably noticed that the LPM_off() function does not simply turn the power off, it performs the following sequence of transitions:

1. power ON + clock active
2. power OFF + clock inactive
3. power ON + clock active
4. setup autonomous power off trigger
5. boot DSP to ROM, mode = 1 (execute the idle instruction)
6. wait for IVA2 to power OFF
7. turn off clocks

I think you are seeing a failure at step 2. The rational for this power off sequence is to ensure the IVA2 will power on again. The LPM_off() function cannot make any assumptions on the current state of the IVA2; it might be already off, it might be stuck with a memory bus error in the L3 interconnect, it might be halted with a cpu exception trap, etc. My experience is that it might not power on again unless the IVA2 performed a "clean" autonomous power off transition, meaning that any exceptions or memory bus hangs have been cleared. Hence all the extra steps in the power off sequence to execute a clean autonomous power off transition.

It might be that in your silicon revision this power off sequence could be minimized or maybe it needs more steps. I would suggest making the sequence as short as possible and then building from there. Start by removing steps 1 - 2 above. That is, comment out all the code from the beginning of the function down to around line 502. You want the function to start at the *second* instance of the comment "MPU controlled ON + ACTIVE'. Leave the rest of the function. Then try the lpmOFF test again. One note: I found that sometimes getting lpmOFF to work ended up breaking lpmON. If you can run these two  program multiple times (and the same one successively multiple times) then you are in good shape.

~ Ramsey

Thanks, Ramsey.

Yes, the device is OMAP3530, not sure what silicon revision.  (I'll work on this in parallel with our discussion.)

I'm not sure I can match up the LPM_off() code (in modules/omap3530/lpm_omap.c) exactly with your 7-step sequence, so instead here are the major comments (/* */) and error messages for each of the spin waits:  (I am intending these references to be complete enough so that you can browse your own copy of lpm_omap.c; let me know if you want a copy of mine from Codec Engine 2.23.01.)

1. /* MPU controlled ON + ACTIVE */
2. TRACE("lpm.ko: Error: IVA2 power domain not ON in LPM_off\n");
3. TRACE("lpm.ko: Error: IVA2_RST2 signal was not released\n");
4. TRACE("lpm.ko: Error: IVA2 did not become active\n");
5. /* MPU controlled OFF */
6. TRACE("lpm.ko: Error: IVA2 did not become idle\n");
7. TRACE("lpm.ko: Error: IVA2 power domain not OFF (1)\n");
8. /* MPU controlled ON + ACTIVE */
9. TRACE("lpm.ko: Error: IVA2 power domain not ON in LPM_off\n");
10. TRACE("lpm.ko: Error: IVA2_RST2 signal was not released\n");
11. TRACE("lpm.ko: Error: IVA2 did not become active\n");
12. // set DSP boot address and boot mode
13. TRACE("lpm.ko: Error: IVA2_RST1 signal was not released\n");
14. TRACE("lpm.ko: Error: IVA2 power domain not OFF (2)\n");

I am encountering a failure in my step 7 (which I think corresponds to your step 2).

Regarding a debug approach:  I do not understand the chip hardware well enough to conceive of viable/valid alternative power strategies (not sure there's enough information in the public domain to know power/reset/clock sequencing constraints and requirements to deterministically and sanely change power state).  I will try your specific recommendation (of temporarily removing your steps 1 and 2), but I'm unclear what I would learn and/or how to gain confidence in this being a final/working solution.

Perhaps a more promising approach is to see if I can plug in a new OMAP3530 module (rev 3 or newer) and see if it changes the behavior.

If silicon revision ends up being important, do you have any way of recovering the old code for old silicon?  (Perhaps I could run different code based on the silicon version (read from a register) so that chips of any age would work.)  I don't know how many "old" chips this code might hit, but it might be easier to work around the problem than to try to quantify it.

CONTROL.CONTROL_IDCODE reads back a value of 0x2B7AE02F, so the silicon revision is ES2.1.

Commenting out your sections 1 and 2 (which correspond to my sections 1 to 7) still fails.  (Looks nearly identical once I added code to print the IVA2 power control/status registers after failure.)

Here is the output from running lpmOFF.xv5T:

==== LPM OFF Test ====
lpm.ko: --> LPM_off
app: LPM_open
app: LPM_set(LPM_CTRL_REFCOUNTOVR)
app: LPM_off
lpm.ko: Error: IVA2 power domain not OFF (2)
lpm.ko: IVA2 power control = 0x00FF0F04
lpm.ko: IVA2 power status  = 0x00000FF7
lpm.ko: <-- LPM_off: stat = 0x1
Error: LPM_off failed
app: LPM_close
==== All Done ====

I was also able to borrow a neighbor's ES3.1 on a Zoom dev kit, and got exactly the same result.

The problem does not appear to be related to the silicon version. I did all my testing on ES 2.1,  the same as you. I think you understood my suggestion correctly. Since it is still failing, it seems we cannot get it to do an autonomous power-off sequence. The next step would be to attempt a force mode power-off from the MPU side. I've attached a modified version of lpm_omap.c with this change to the LPM_off() function. Essentially, it will just execute the following code block.

    // set next IVA2 power state to OFF
    REG(prm + PM_PWSTCTRL_IVA2) &= ~(0x3);
    REG(prm + PM_PWSTCTRL_IVA2);

    // start a software supervised sleep transition
    REG(cm + CM_CLKSTCTRL_IVA2) = 0x1;
    REG(cm + CM_CLKSTCTRL_IVA2);

    /* BEGIN NEW */
    // Remove all power domain dependencies
    REG(prm + PM_WKDEP_IVA2) = 0;
    REG(prm + PM_WKDEP_IVA2);

    // turn off IVA2 domain functional clock
    REG(cm + CM_FCLKEN_IVA2) = 0;
    REG(cm + CM_FCLKEN_IVA2);
    /* END NEW */

    // wait for IVA2 power domain to switch OFF
    for (i = 0; i < MAX_WAIT_COUNT; i++) {
        if ((REG(prm + PM_PWSTST_IVA2) & 0x3) == 0) {
            break;
        }
    }
    if (i == MAX_WAIT_COUNT) {
        TRACE("lpm.ko: Error: IVA2 power domain not OFF (1)\n");
        status = LPM_EFAIL;
        goto fail;
    }

    // clear power state change request
    REG(cm + CM_CLKSTCTRL_IVA2) = 0;
    REG(cm + CM_CLKSTCTRL_IVA2);

I've moved the clock off statement to the end. I also turned off the wakeup dependency just in case there is some module issuing wake-up events (but this seems unlikely). Give this a try.

Another basic item to check is to make sure the code is accessing the hardware through non-cachable memory. This should be taken care of by the call to ioremap_nocache() in lpm_driver.c.

What version of Linux are you using? Do you get any compiler warnings when building the LPM driver?

Do you have access to CCS 3.3 for debugging? If so, we could setup a meeting and do a joint debug session. Let me know if you would like to give this a try.

Thanks again!

Unfortunately, the behavior is the same (added register print to your code, then ran lpmOFF.xv5T):

==== LPM OFF Test ====
lpm.ko: --> LPM_off
app: LPM_open
app: LPM_set(LPM_CTRL_REFCOUNTOVR)
app: LPM_off
lpm.ko: Error: IVA2 power domain not OFF (1)
lpm.ko: PM_PWSTCTRL_IVA2 = 0x00FF0F04
lpm.ko: PM_PWSTST_IVA2   = 0x00000FF7
lpm.ko: <-- LPM_off: stat = 0x1
Error: LPM_off failed
app: LPM_close
==== All Done ====

Do let me know what other register values might be of interest (I'll make some guesses in the meantime).

Doing a forced power-off would seem to be the preferred way of doing things (else misbehavior prevents power termination -- sort of like a light bulb not being able to be turned off if it's burned out).

Linux = 2.6.28, gcc = 4.1.2 (instead of gcc 4.2.1)

Now that you mention it, there is one compiler warning being reported, but it looks entirely benign:

lpm_omap.c: In function 'LPM_off':
lpm_omap.c:373: warning: unused variable 'control'

Checked lpm_driver.c : lpm_os_remap() -- it is indeed calling ioremap_nocache().  Tracing lpm_init() shows that addresses are being remapped:

--> LPM_init
LPM_init: os_remap pa = 0x48002000, va = 0xD8002000
LPM_init: os_remap pa = 0x48004000, va = 0xD8004000
LPM_init: os_remap pa = 0x48306000, va = 0xD8306000
LPM_init: os_remap pa = 0x5D000000, va = 0xC58A2000
Control Revision: 0x10
<-- LPM_init: stat = 0x0

I can get Code Composer Studio (it's free!), but I don't have a debug pod handy.  (printf never lets me down!  ;-)  In the meantime, I'll add code to print more register values.  (Do you have any recommendations for a command line utility to read/write physical registers?)  And do let me know what registers would be most interesting (and when).

I temporarily reverted to the original lpm_omap.c behavior of Codec Engine 2.23.01, then dumped the values of the registers that LPM touches throughout the execution of LPM_off() (basically before and after each "for" loop).  Then I manually appended an asterisk when the value changed from the previous dump.  The textual dump (and a copy of the hacked lpm_omap.c file that generated it) are attached.

The board affects the result, but does not fix it.  Specifically, I ran the same Linux image on a Mistral EVM3530 and a Logic Zoom OMAP3530 Dev Kit, and got different results.  The test procedure is to run the Codec Engine loadmodules.sh script (which loads the lpm_omap3530.ko driver), then run the lpmOFF.xv5T test program (which exercises the heavily instrumented LPM_off() function in the driver).

The EVM3530 failed after the second "for" loop ("Error: IVA2_RST2 signal was not released"), whereas the Zoom failed after the fifth "for" loop ("Error: IVA2 power domain not OFF (1)").

I have attached a side-by-side log of the register values during LPM_off().  I will start analyzing the differences.

Would you compare the IDCODE register on both boards to see if the OMAP3530 silicon revision is the same? Thanks.

Good catch; they're not the same:  The CONTROL_IDCODE register (0x4830A204) reads back 0x3B7AE02F (ES3.0) on EVM3530, and 0x2B7AE02F (ES2.1) on Zoom.

Recent (real-time!) observations:

• Thinking about the differences between the boards:  The EVM3530 board initially booted with x-loader (and then Logic LoLo to u-boot to Linux), but Zoom initially booted with Logic LoLo (to u-boot to Linux).  Perhaps x-loader is helpfully writing a register value that LoLo and u-boot and Linux don't touch.  (If LoLo or u-boot or Linux wrote to the register, then the behavior should be the same across boards (other than differences due to electronic differences).)
  
  I'll see if I can find anything in the x-loader tar ball in Linux PSP 2.0.0.1.
  
  
• In modules/omap3530/lpm/lpm_omap.c, comments indicate the intent to clear specific bits of RM_RSTST_IVA2, but the C operator "|=" is used.  Note that the RM_RSTST_IVA2 register follows the convention of "write 1 to clear", so using the C operator even with a value of zero (i.e., "REG(RM_RSTST_IVA2) |= 0;") would result in completely clearing the register (by clearing all the bits that are currently set).  The comments suggest that the intention was to only clear specific bits, so changing this to simple assignment (i.e., "REG(RM_RSTST_IVA2) = (1 << bit);") would change the behavior to only clear the bit of interest, leaving the other bits unmodified.
  
  At a glance, I can't say that this is hurting anything (clearing all bits instead of just one), but it may help to see what's going on (since the other bits won't be destroyed).  For example, looking back at the EVM3530-vs-Zoom differences, I can see at the top of LPM_off() that Zoom was booted from cold reset (bit 0 is still set), but EVM3530 has already completely cleared the register by that time.

Overview:  Using u-boot (prior to Linux), I was able to run the register sequence of LPM_off(), and it worked fine on both EVM3530 and Zoom.

Details:

I interrupted u-boot autoboot on each board to look for differences, but didn't see any:

• md 48004000 14
  48004000: 00000000 00000037 00000000 00000000    ....7...........
  48004010: 00000000 00000000 00000000 00000000    ................
  48004020: 00000001 00000001 00000000 00000000    ................
  48004030: 00000000 00000000 00000000 00000000    ................
  48004040: 0009680c 00000001 00000000 00000000    .h..............
• md 48306050 28
  48306050: 00000007 00000000 00000000 00000000    ................
  48306060: 00000000 00000000 00000000 00000000    ................
  48306070: 00000000 00000000 00000000 00000000    ................
  48306080: 00000000 00000000 00000000 00000000    ................
  48306090: 00000000 00000000 00000000 00000000    ................
  483060a0: 00000000 00000000 00000000 00000000    ................
  483060b0: 00000000 00000000 00000000 00000000    ................
  483060c0: 00000000 00000000 000000b3 00000000    ................
  483060d0: 00000000 00000000 00000000 00000000    ................
  483060e0: 00ff0f07 00000ff7 00000000 00000000    ................

I then converted lpm_omap.c:LPM_off() into a sequence of u-boot commands ("mw" is memory write; "md" is memory display (read)).  From a cold boot, both boards accept the sequence:

• Hit any key to stop autoboot:  0
  mw 48004048 00000000
  mw 48004000 00000000
  mw 48306050 00000005
  mw 48306050 00000007
  mw 483060E0 00FF0F07
  mw 48004048 00000002
  md 483060E4  1
  483060e4: 00000ff7    ....
  mw 48004048 00000000
  mw 48004000 00000001
  mw 48306058 00000200
  mw 48306050 00000005
  md 48306058  1
  48306058: 00000200    ....
  mw 48306058 00000200
  md 48004020  1
  48004020: 00000000    ....
  mw 48004000 00000000
  mw 48306050 00000007
  md 48004020  1
  48004020: 00000001    ....
  mw 483060E0 00FF0F04
  mw 48004048 00000001
  md 483060E4  1
  483060e4: 00000000    ....
  mw 48004048 00000000
  mw 483060E0 00FF0F07
  mw 48004048 00000002
  md 483060E4  1
  483060e4: 00000ff7    ....
  mw 48004048 00000000
  mw 48004000 00000001
  mw 48306058 00000200
  mw 48306050 00000005
  md 48306058  1
  48306058: 00000200    ....
  mw 48306058 00000200
  md 48004020  1
  48004020: 00000000    ....
  mw 48002400 00000000
  mw 48002404 00000001
  mw 48004048 00000003
  mw 483060E0 00FF0F04
  mw 48306058 00000100
  mw 48306050 00000004
  md 48306058  1
  48306058: 00000105    ....
  mw 48306058 00000100
  md 483060E4  1
  483060e4: 00000000    ....
  mw 48306050 00000005
  mw 48306050 00000007
  mw 48004048 00000000
  mw 48004000 00000000

The reset at the very end of LPM_off() is a little suspicious:  TRM sections 4.5.9.3 "IVA 2.2 Subsystem Power-Up Sequence" and 4.5.9.4 "IVA2 Software Reset Sequence" both show IVA2_RST3 being asserted prior to IVA2_RST1 and IVA2_RST2, but the Codec Engine 2.23.01 lpm_omap.c code asserts 3 and 1, and then asserts 2.

Observation:  The register values upon entry into LPM_off() suggest that on my Linux boot, IVA2 has never yet been powered up -- all three RM_RSTCTRL_IVA2 bits are set, but none of the reset completion indicators in RM_RSTST_IVA2 are set (IVA2_SW_RST1..3).  Fortunately, the u-boot version of LPM_off() handles this just fine.

My current thoughts lie in the direction that the power/reset/clock hardware sequencing of IVA2 has somehow gotten "stuck" by my Linux boot sequence.

Please let me know if any of these thoughts revive a memory or inspire an idea for an experiment.  In the meantime, I'll look into Linux (both TI Linux PSP 2.0.0.1 and my Linux) to see what IVA2 registers (if any) are tweaked during boot.

lpmON.xv5T and lpmOFF.xv5T are failing on my Zoom using the same kernel and LPM versions that you described.  I wonder if I'm seeing the same problem that you described.  Can you post the source for your final fix?

Thanks!!

I inserted the following code into cetools/packages/ti/bios/power/modules/omap3530/lpm/lpm_omap.c:LPM_off() after TRACE("lpm.ko: --> LPM_off\n") and before /* MPU controlled ON + ACTIVE */:

    if ((REG(cm + CM_CLKEN_PLL_IVA2) & 0x7) == 0x1) {
        TRACE("lpm.ko: Found IVA2 DPLL in low power stop mode; changing to bypass\n");
        REG(cm + CM_CLKEN_PLL_IVA2) = (REG(cm + CM_CLKEN_PLL_IVA2) & ~0x7) | 0x5;
        REG(cm + CM_CLKEN_PLL_IVA2);
    }

Note that this fix is not board-specific, but rather is addressing an implicit dependency on boot code.  (Presumably TI's Linux PSP sets this, but my Linux version doesn't...)  This tweak (hack!) to LPM gets me by, but I look forward to getting this into boot code (especially in case any other code makes the same assumption).

Good luck!  -Dirk

I applied this change to local_power_manager_1_23_01/packages/ti/bios/power/modules/omap3530/lpm/lpm_omap.c from recently downloaded local_power_manager_1_23_01.tar.gz

lpmOFF.xv5T and lpmON.xv5T from local_power_manager_1_23_01/packages/ti/bios/power/test/bin/ti_platforms_evm3530/linux/release now function correctly on a Zoom running TimeSys 2.6.28 basic kernel!

Thanks for the tweak! 

~CN

FYI:  Just found that a defensive variation of this code has been added to Codec Engine 2.24:

    if ((REG(cm + CM_CLKEN_PLL_IVA2) & 0x7) == 0x1) {
        hndl->os_trace("Error: IVA2 PLL is not enabled (needed by LPM)");
        status = LPM_EFAIL;
        goto fail;
    }

Nice.  So you better get your boot code to enable DPLL2!  -Dirk