This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM3358 Suspend resume support

Other Parts Discussed in Thread: AM3358

Hi,

    I am using AM3358 EVM board able to boot with TI SDK 3.2 kerne and suspend/resume feature works fine.But we have ported the AM3358 machine to Custom kernel version 2.6.35 which is also booting and working fine.

Currently we tried to test the suspend/resume support in the custom kernel which is not resuming after suspend.We are able to suspend/resume through GPIO0 when TI emulator is connected to arm-cortex-A8 core.Without connecting the emulator target is not resuming from the suspend state.

We further debug by connecting M3_wakupss core through emulator.We observed below error.

1.After bootup,once we give suspend (echo mem > /sys/power/state) and we stopped the M3 core ,checked the register info list in CCS.Found there is a exception number in the register xPSR.Since R0,R1 registers shows the address 0x44E000E0 we suspects the there is  issue with CM_PER_L3_CLKCTRL  module.We added following lines to pm33xx.c at am33xx_pm_suspend function.


        gpio1_oh        = omap_hwmod_lookup("gpio1");   /* WKUP domain GPIO */
        tptc0_oh        = omap_hwmod_lookup("tptc0");
        tptc1_oh        = omap_hwmod_lookup("tptc1");
        tptc2_oh        = omap_hwmod_lookup("tptc2");
        cpsw_oh         = omap_hwmod_lookup("cpgmac0");
        lcdc_oh         = omap_hwmod_lookup("lcdc");
        tpcc_oh         = omap_hwmod_lookup("tpcc");
        mmc_oh          = omap_hwmod_lookup("mmc3");

        omap_hwmod_enable(usb_oh);
        omap_hwmod_enable(gpmc_oh);
        omap_hwmod_enable(gpio1_oh);

        omap_hwmod_enable(tptc0_oh);
        omap_hwmod_enable(tptc1_oh);
        omap_hwmod_enable(tptc2_oh);
        omap_hwmod_enable(cpsw_oh);
        omap_hwmod_enable(lcdc_oh);
        omap_hwmod_enable(tpcc_oh);
        omap_hwmod_enable(mmc_oh);

        omap_hwmod_idle(usb_oh);
        omap_hwmod_idle(gpmc_oh);

        omap_hwmod_idle(tptc0_oh);
        omap_hwmod_idle(tptc1_oh);
        omap_hwmod_idle(tptc2_oh);
        omap_hwmod_idle(cpsw_oh);
        omap_hwmod_idle(lcdc_oh);
        omap_hwmod_idle(tpcc_oh);
        omap_hwmod_idle(mmc_oh);

Please find the register details below.

Core Registers    
    PC    0x00000CA6    Core Register: Program Counter    
    SP    0x00080F60    Core Register: General Purpose Register 13 - Stack Pointer    
    LR    0x00000CD3    Core Register: General Purpose Register 14 - Link Register    
    xPSR    0x81000032    Core Register: Stores the status of interrupt enables and critical processor status signals    
        N    1    Stores bit 31 of the result of the instruction. In other words stores the sign of the number    
        Z    0    Is set to 1 if the result of the operation is zero else stays 0    
        C    0    Stores the value of the carry bit if it occurred in an addition or the borrow bit in a subtraction. In a shift stores the last bit shifted out.    
        V    0    Set to 1 if an overflow occurred     
        Q    0    Indicates whether an overflow/saturation occurred in the enhanced DSP instructions    
        ICI_IT_2    00    ICI/IT - bit26-bit25     
        T    1    Thumb State     
        RESV    00000000    Reserved     
        ICI_IT_1    000000    ICI/IT - bit15-bit10     
        RESV2    0    Reserved     
        EXCEPTION    000110010    Exception Number     
    R0    0x44E000E0    Core Register: General Purpose Register 0    
    R1    0x44E000E0    Core Register: General Purpose Register 1    
    R2    0x00000000    Core Register: General Purpose Register 2    
    R3    0x00010000    Core Register: General Purpose Register 3    
    R4    0x00000000    Core Register: General Purpose Register 4    
    R5    0x00000000    Core Register: General Purpose Register 5    
    R6    0x00000000    Core Register: General Purpose Register 6    
    R7    0x00080F60    Core Register: General Purpose Register 7    
    R8    0x00000000    Core Register: General Purpose Register 8    
    R9    0x00000000    Core Register: General Purpose Register 9    
    R10    0x00000000    Core Register: General Purpose Register 10    
    R11    0x00000000    Core Register: General Purpose Register 11    
    R12    0x00000000    Core Register: General Purpose Register 12    
    R13    0x00080F60    Core Register: General Purpose Register 13    
    R14    0x00000CD3    Core Register: General Purpose Register 14    
    MSP    0x00080F60    Core Register: MSP Register    
    PSP    0x00000000    Core Register: PSP Register    
    DSP    0x00000000    Core Register: DSP Register    
    CTRL_FAULT_BASE_PRI    0x00000000    Core Register: CM3 Special Registers    


Please let us know how to debug M3 core further?Can you share the source code of M3_Wkup firmware?
Are we missing any clock/power initialisation in linux kernel running at A8 core?

Regards

Yogavanan

  • Hi Yogavanan,

    You will have to first debug and conclude if the issue is in suspend process or resume process.

    On connecting CCS to CM3 Core , the resume you are seeing could be via abort path.

    Do you see any extra print "Kernel core reported suspend failure" ?

    Source code for CM3 firmware is available here

    http://arago-project.org/git/projects/?p=am33x-cm3.git;a=commit;h=27ca4643e422245a95723de1df0247a00eada45b

    Regards

    Sandhya

  • Hi Sandhya,

    Yes we are seeing "Kernel core reported suspend failure" message in the console while suspending.

    We tried with the latest Binary as well in the link you have provided but issue is still the same.  

    can we understand that resume is happening via abort path since, CM3 core is generated an exception?

    Why exception error arises  during suspend sequence in CM3 core on the CM_PER_L3_CLKCTRL register?  are we missing anything in the kernel (Cortex-A8) side?

    In 3.2 TI SDK kernel once MPU is suspended we are not able to connect the emulator to the CM3 core. Is this an expected behavior or we need to change any settings in the CCS debugger. Please find the CCS error message mentioned below,

    CS_DAP_M3: Error: (Error -275 @ 0xFFFFFEED) The attempt to poll a target device exceeded its timeout limit. The utility or debugger has requested that a target device be repeatedly accessed for a specific data or status value. This has failed because the built-in limit for the maximum number of attempts when polling the JTAG scan-path has been exceeded. (Emulation package 5.0.623.0)
    M3_wakeupSS_0: Error: (Error -1045 @ 0xFFFFFEED) The emulator reported an error. Confirm emulator configuration and connections, reset the emulator, and retry the operation. (Emulation package 5.0.623.0)
    CS_DAP_M3: Error: (Error -5002 @ 0xFFFFEC76) Unable to access the DAP. Reset the device, and retry the operation. If error persists, confirm configuration, power-cycle the board, and/or try more reliable JTAG settings (e.g. lower TCLK). (Emulation package 5.0.623.0)
    M3_wakeupSS_0: Trouble Halting Target CPU: (Error -1044 @ 0xFFFFEC76) The emulator reported an error. Confirm emulator configuration and connections, reset the emulator, and retry the operation. (Emulation package 5.0.623.0)

  • Hi Biser,

    We have already checked these links and followed the procedure mentioned in this link.

    It is directly working in 3.2 TI  sdk, but we are working on a custom kernel.

    Regards,

    Yogavanan M

  • Hi,

    Resume happened via abort path does not mean CM3 core generated an exception. This term is only used to say that resume did not happen the normal way and due to a last minute interrupt the suspend process was aborted and resume happened even without the control going to Cortex M3.

    CCS does not allow you to connect to A8 after a cycle of  suspend/resume. This is a known issue. But connecting to CM3 should be possible and the error you have reported, haven't come across it before.

    Which version of CCS are you using?

    You still have to find out if the issue is in suspend process or resume process.

    Regards

    Sandhya

  • Hi Sandhya,

    Regarding the CCS debugging,

    Can you please share any target Configuration xml file which can be used to debug CM3 core when A8 core is suspended. 

    Currently we are using  CCS Version 5.1.1.00033.

    Using the source code of CM3 core we were able to locate the problem during the suspend process and we temporarily blocked the code causing the exception and with that binary we are able to suspend the A8 core now. Once we suspend we suspend we are not able to debug CM3 core.

    Can you let us know how we can debug the resume process in A8? As per our understanding A8 core resumes from bootrom as per 26.1.11 Wakeup section of TRM.

    So which part of the Linux kernel identifies the boot process is a normal or an resume from a suspend? 

    Because currently in our custom kernel we are able to suspend the device successfully, however we are not able to resume from GPIO still.

    Regards

    Yogavanan M

  • Hi Yogavanan,

    What was the problem in CM3 code on removing which you were able to proceed with suspend process?

    Can you connect CCS to CM3 before entering suspend and when you are hung after suspend just check where is PC of CM3?

    Debugging CM3 is possible through IPC registers , no debug prints can be seen.

    Check the code flow for suspend process from pm33xx.c to sleep33xx.S (WFI is done here) after which control goes to CM3.

    On a wakeup event, CM3 receives the wakeup interrupt, does the initial resume steps and finally ROM code jumps the execution back to sleep33xx.S after WFI.


    Regards

    Sandhya


  • Hi Sandhya,

    In CM3 core side as per our understanding in case of custom kernel, CM3 core aborts at the below line,

    So we currently commented the below line in interconnect_modules_disable() function and the allowed the suspend  process to completed.

    // module_state_change(MODULE_DISABLE, AM335X_CM_PER_L3_CLKCTRL);

    we confirmed that suspend is completed by adding logs CM3 core.

    so what we have to do in the A8 core (Linux Kernel) to fix this issue on CM_PER_L3 module?  Do we need to need to make it idle? are we missing something in Kernel side? 

    Regards

    Yogavanan 

  • Hi,

    How different is your custom kernel from the TI-SDK kernel with which suspend - resume works correctly?

    Were you able to connect CCS to CM3?

    If you were, please take a dump of registers in L4_WKUP (0x44C0_0000 - 0x44FF_FFFF) mainly the CM_ ,PRM_ and Control modules in the working and non-working scenarios. The comparison could point you to the issue in your custom kernel.

    Regards

    Sandhya

  • Hi,

    In our custom kernel, mach-omap2 and plat-omap are "almost" same TI SDK and we have backported the drivers that are necessary according to our need.

    We are not able to connect CCS to CM3 when MPU is suspended. This is the case in case of both TI-SDK kernel and custom kernel. If you can help us to solve this problem it would greatly help us in debugging this issue.

    Also as on today, the suspend process is complete with the modified CM3 binary.  To Confirm that problem in the resume process, we gave the resume address as NULL in pm33xx.c and when wake up event is given device goes to reset vector and jumps to bootloader and hangs. This behavior is same in both TI-SDK and custom kernel. 

    am33xx_lp_ipc.resume_addr = NULL; //(DS_RESUME_BASE + am33xx_resume_offset + 4);

    As per our understanding, the address given in the above code is used by bootrom to resume the MPU which is address of "am33xx_resume_from_deep_sleep function" in sleep33xx.s

    #define AM33XX_SRAM_PA      0x40300000

    #define DS_RESUME_BASE          0x40300000

    boorom expects "am33xx_resume_from_deep_sleep function" function to be sram at the specified offset. However we didn't see "am33xx_resume_from_deep_sleep function" function getting pushed  into sram.  During sram initialization only "am33xx_do_wfi_sz" function which executes the WFI instruction is pushed. 

    void am33xx_push_sram_idle(void)
    {
    am33xx_do_wfi_sram = (void *)omap_sram_push
    (am33xx_do_wfi, am33xx_do_wfi_sz);
    }

    We are missing some understanding here. as per our understanding "am33xx_resume_from_deep_sleep" is not available for bootrom to resume MPU and this the reason why device hangs during resume process.

    which function address does the bootrom expects to be stored in IPC0 register? and which code in the A8 core stores that in SRAM?

    Can you please help us to find the missing dots? 

  • Hi,

    >We are missing some understanding here. as per our understanding "am33xx_resume_from_deep_sleep" is not available for bootrom to resume MPU and this the >reason why device hangs during resume process.

    This is not so. If you look at omap_sram_push(), the function and the size are specified. In sleep33xx.S, am33xx_do_wfi_sz is at the end of the file which ensures all the functions/data in sleep33xx.S is pushed to SRAM.

    So, your failing to resume is not due to this.

    I will still suggest taking a dump of all Control module, CM_ and PM_ registers to check what could be the issue.

    I have also seen, sometimes the delay 'wait_sdram_config' in sleep33xx.S needs to be varied for a few boards/designs.

    Regards

    Sandhya


  • Hi Sandhya,

    Thanks for your reply.  

    we took a  dump of all Control module, CM_ and PM_ registers and we found that clock was not disabled properly for few modules.  After fixing this issue device is  now able to jump to the SRAM address 0x403005F0 which is the offset to the "am33xx_resume_from_deep_sleep" from bootrom.

    Now we are seeing a crash in the DDR2 resume path, device crashes while configuring the EMIF timings value in sleep33xx.x. Can you give  us any pointers as to what can cause this problem?

    timing1:
    ldr r4, emif_timing1_val
    str r4, [r3, #EMIF4_0_SDRAM_TIM_1]
    str r4, [r3, #EMIF4_0_SDRAM_TIM_1_SHADOW]

  • Hi,

    Ok great, so now you are on the resume path.

    Are you sure the crash is while writing emif timing val only? Please check if  r3 and r4 have correct/expected values.

    Was DDR held in self-refresh properly throughout the suspend process?

    Regards

    Sandhya

  • Hi,

    We are now able to finish the resume process with minimal work around for DDR2 initialization.

    We can fix other issues from here and  thanks for your support. 

    We will get back to you if in case we need any other help and hence i will close this thread.