Linux/TPS65910: Kernel panic at shutdown

Harshvardhan Ruchandani

Part Number: TPS65910
Other Parts Discussed in Thread: TPS65217

Tool/software: Linux

We use a custom PCB design based on the beaglebone Black.

After issuing /sbin/halt we get a kernel panic during shutdown. Below is the serial output form Am335x once the halt command is executed.

[ 67.247026] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[ 67.284712] systemd-journald[131]: Received SIGTERM from PID 1 (systemd-shutdow).
[ 67.345959] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 67.377139] systemd-shutdown[1]: Unmounting file systems.
[ 67.384882] systemd-shutdown[1]: Unmounting /sys/fs/fuse/connections.
[ 67.413705] systemd-shutdown[1]: Unmounting /sys/kernel/config.
[ 67.443548] systemd-shutdown[1]: Unmounting /sys/kernel/debug.
[ 67.473490] systemd-shutdown[1]: Unmounting /dev/mqueue.
[ 68.620047] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 68.656456] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 68.662182] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 68.668229] systemd-shutdown[1]: All filesystems unmounted.
[ 68.674701] systemd-shutdown[1]: Deactivating swaps.
[ 68.680958] systemd-shutdown[1]: All swaps deactivated.
[ 68.686839] systemd-shutdown[1]: Detaching loop devices.
[ 68.700775] systemd-shutdown[1]: All loop devices detached.
[ 68.707199] systemd-shutdown[1]: Detaching DM devices.
[ 68.714447] systemd-shutdown[1]: All DM devices detached.
[ 68.762750] systemd-shutdown[1]: Powering off.
[ 68.792038] musb-dsps 47401c00.usb: otg: usb_otg_unregister_hcd: device not registered to otg core
[ 68.801615] musb-hdrc musb-hdrc.1.auto: remove, state 4
[ 68.807343] usb usb1: USB disconnect, device number 1
[ 68.815470] musb-hdrc musb-hdrc.1.auto: USB bus 1 deregistered
[ 68.855730] reboot: Power down
[ 68.858956] System will go to power_off state in approx. 1 second
[ 71.343720] rtc_power_off failed, bailing out.
[ 71.356128] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[ 71.356128]
[ 71.365545] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 4.1.8-ti-r18 #1
[ 71.372633] Hardware name: Generic AM33XX (Flattened Device Tree)
[ 71.378928] [<c001a088>] (unwind_backtrace) from [<c0014aa4>] (show_stack+0x20/0x24)
[ 71.386873] [<c0014aa4>] (show_stack) from [<c09af280>] (dump_stack+0x8c/0xcc)
[ 71.394282] [<c09af280>] (dump_stack) from [<c09acb0c>] (panic+0xb0/0x230)
[ 71.401320] [<c09acb0c>] (panic) from [<c004aaf4>] (complete_and_exit+0x0/0x2c)
[ 71.408829] [<c004aaf4>] (complete_and_exit) from [<c00686e4>] (SyS_reboot+0x1c8/0x238)
[ 71.417042] [<c00686e4>] (SyS_reboot) from [<c0010920>] (ret_fast_syscall+0x0/0x3c)
[ 71.424894] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[ 71.424894]

We also had some problems inn keeping the sitara powered on using TPS65910 so, we worked around the issue by editing the uboot include/configs/am335x_evm.h by adding the following in bold.

#define CONFIG_BOOTCOMMAND \
"setenv umsmedia 0; "\
"run findfdt; " \
"setenv mmcdev 0; " \
"setenv bootpart 0:1; " \
"run mmcboot;" \
"setenv mmcdev 1; " \
"setenv bootpart 1:1; " \
"gpio clear 64; " \
"gpio clear 67; " \
"gpio clear 68; " \
"gpio clear 12; " \
"gpio clear 13; " \
"gpio clear 14; " \
"gpio clear 15; " \
"gpio clear 22; " \
"i2c mw 2d 3f 34; " \
"i2c mw 69 c0 3c; " \
"gpio clear 53; " \
"gpio set 54; " \
"gpio set 55; " \
"gpio set 56; " \
"gpio set 57; " \
"gpio set 58; " \
"gpio set 22; " \
"run mmcboot;" \
"run failumsboot;"

We are not sure if these two problems are related but they as well might be. Any help would be appreciated.

Regards,

Harshvardhan.

over 8 years ago

0 Nick Jakse49 over 8 years ago

TI__Expert 4155 points

Harshvardhan,

I am moving this post to the AM335x team as they will be able to help more with the software side of this question. We will continue to monitor the post so we can help wherever possible.

Thank you,
Nick

0 Biser Gatchev-XID over 8 years ago

TI__Guru**** 393215 points

You state that this is a Beaglebone Black based design. However Beaglebone Black does not use the TPS65910 PMIC. Please post the PMIC - AM335X related portion of your schematic.

0 Harshvardhan Ruchandani over 8 years ago in reply to Biser Gatchev-XID

Prodigy 170 points

0486.swcu093c.pdf

Please find the document which we used to attach the PMIC and Sitara Processor. Also find the PMIC part of the schematic. The connections made are identical to the pdf document.

Thanks,

Harsh

0 Michael Erdahl over 8 years ago

TI__Genius 13495 points

Harshvardhan,

Can you please probe your PMIC_PWR_EN signal attached to PWRHOLD on the PMIC?

This message indicates the AM335x RTC module failed to turnoff the PMIC.

[ 71.343720] rtc_power_off failed, bailing out.

I am curious if the PMIC_PWR_EN signal is deasserted, but something else is keeping the PMIC active, or, the PMIC_PWR_EN signal is never de-asserted.

Regards,
Mike

0 Harshvardhan Ruchandani over 8 years ago in reply to Michael Erdahl

Prodigy 170 points

Michael,

I anticipated that there was some problem related to PMIC_PWR_EN so I have collected the data for PMIC_PWR_EN(sitara) and POWERON(pmic). Please find the data in the attached document. Yes, PMIC_PWR_EN signal never goes low. It always shows high voltage of 1.8 volts(even when the device is supposedly off ).

Note: There is no kernel panic with the 4.1.2-bone12 kernel. This panic only occurs on 4.1.8-ti-r18 kernel.

Thanks,

Harshvardhan

pmic_data_collected.xlsx

0 Michael Erdahl over 8 years ago in reply to Harshvardhan Ruchandani

TI__Genius 13495 points

Harshvardhan,

Thanks for the data.

Can you share your tps and rtc nodes from your device tree? I am curious how these were modified going from the TPS65217 to the TPS65910.

Regards,
Mike

0 Harshvardhan Ruchandani over 8 years ago in reply to Michael Erdahl

Prodigy 170 points

Michael,

Please find the tps & rtc nodes of our device tree in the attached document.

Thanks,

Harshvardhan

rtc_tps_dtsi.txt

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
//tps node of our device tree 
&tps {
    compatible = "ti,tps65910";
    regulators {
        #address-cells = <1>;
        #size-cells = <0>;
        vrtc_reg: regulator@0 {
            reg = <0>;
            regulator-compatible = "vrtc";
        };
        vio_reg: regulator@1 {
            reg = <1>;
            regulator-compatible = "vio";
        };
        vdd1_reg: regulator@2 {
            reg = <2>;
            regulator-compatible = "vdd1";
        };
        vdd2_reg: regulator@3 {
            reg = <3>;
            regulator-compatible = "vdd2";
        };
        vdd3_reg: regulator@4 {
            reg = <4>;
            regulator-compatible = "vdd3";
        };
        vdig1_reg: regulator@5 {
            reg = <5>;
            regulator-compatible = "vdig1";
        };
        
        
// rtc node for our device tree.        
        rtc: rtc@44e3e000 {
            compatible = "ti,am3352-rtc", "ti,da830-rtc";
            reg = <0x44e3e000 0x1000>;
            interrupts = <75
                      76>;
            ti,hwmods = "rtc";
        };
        vdig2_reg: regulator@6 {
            reg = <6>;
            regulator-compatible = "vdig2";
        };
        vpll_reg: regulator@7 {
            reg = <7>;
            regulator-compatible = "vpll";
        };
        vdac_reg: regulator@8 {
            reg = <8>;
            regulator-compatible = "vdac";
        };
        vaux1_reg: regulator@9 {
            reg = <9>;
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

//tps node of our device tree 

&tps {
	compatible = "ti,tps65910";

	regulators {
		#address-cells = <1>;
		#size-cells = <0>;

		vrtc_reg: regulator@0 {
			reg = <0>;
			regulator-compatible = "vrtc";
		};

		vio_reg: regulator@1 {
			reg = <1>;
			regulator-compatible = "vio";
		};

		vdd1_reg: regulator@2 {
			reg = <2>;
			regulator-compatible = "vdd1";
		};

		vdd2_reg: regulator@3 {
			reg = <3>;
			regulator-compatible = "vdd2";
		};

		vdd3_reg: regulator@4 {
			reg = <4>;
			regulator-compatible = "vdd3";
		};

		vdig1_reg: regulator@5 {
			reg = <5>;
			regulator-compatible = "vdig1";
		};
		
		
// rtc node for our device tree.		
		rtc: rtc@44e3e000 {
			compatible = "ti,am3352-rtc", "ti,da830-rtc";
			reg = <0x44e3e000 0x1000>;
			interrupts = <75
				      76>;
			ti,hwmods = "rtc";
		};


		vdig2_reg: regulator@6 {
			reg = <6>;
			regulator-compatible = "vdig2";
		};

		vpll_reg: regulator@7 {
			reg = <7>;
			regulator-compatible = "vpll";
		};

		vdac_reg: regulator@8 {
			reg = <8>;
			regulator-compatible = "vdac";
		};

		vaux1_reg: regulator@9 {
			reg = <9>;
			regulator-compatible = "vaux1";
		};

		vaux2_reg: regulator@10 {
			reg = <10>;
			regulator-compatible = "vaux2";
		};

		vaux33_reg: regulator@11 {
			reg = <11>;
			regulator-compatible = "vaux33";
		};

		vmmc_reg: regulator@12 {
			reg = <12>;
			regulator-compatible = "vmmc";
		};

		vbb_reg: regulator@13 {
			reg = <13>;
			regulator-compatible = "vbb";
		};
	};
};

0 Michael Erdahl over 8 years ago in reply to Harshvardhan Ruchandani

TI__Genius 13495 points

Hi Harshvardhan,

Is that the only rtc node you have defined?

We have two rtc nodes. Typically we have one in the base dtsi (am33xx.dtsi) that defines the base address, interrupts, etc, as you have, and another in the top-level board dts file (check am335x-evm.dts).

Try adding the system-power-controller property to your rtc node, like this:

&rtc {
	clocks = <&clk_32768_ck>, <&clkdiv32k_ick>;
	clock-names = "ext-clk", "int-clk";
	system-power-controller;
};

You can find more information about rtc properties in Documentation/devicetree/bindings/rtc/rtc-omap.txt

Regards,
Mike

0 Harshvardhan Ruchandani over 8 years ago in reply to Michael Erdahl

Prodigy 170 points

Hi Mike,

Instead of am335x-evm.dts we have our top-level board dts file and this file contains another rtc node(apart from the one in am33xx.dtsi ).

&rtc {
system-power-controller;
};

So I guess the system-power-controller property is already in the rtc node. Although, I will try adding the lines below in my rtc node and see if that makes any difference.

clocks = <&clk_32768_ck>, <&clkdiv32k_ick>;
clock-names = "ext-clk", "int-clk";

Thanks,
Harshvardhan

0 Harshvardhan Ruchandani over 8 years ago in reply to Harshvardhan Ruchandani

Prodigy 170 points

Hi Mike,

I tried changing the rtc node in device tree as you suggested. I still get the same kernel panic related to rtc. I did try some other settings after reading Documentation/devicetree/bindings/rtc/rtc-omap.txt but still no success.

Thanks,
Harshvardhan

0 Michael Erdahl over 8 years ago in reply to Harshvardhan Ruchandani

TI__Genius 13495 points

Hi Harshvardhan,

Sorry this has not gone smoothly - please see the attached patch. This will add some debugging information to the RTC kernel module that will help diagnose whether or not the correct code paths are getting called.

The rtc-omap kernel driver is a built as a loadable module by default, so you will need to build and install modules to see any changes.

Once your modules have been built and installed, boot your kernel, and grep for "rtc", and verify the rtc is the pmic controller. Next, run the "poweroff" command, and verify the PMIC_POWER_EN is deasserted message is printed.

On my AM335x-evm, I see the following:

root@am335x-evm:~# dmesg | grep rtc                                             
[    1.865252] vrtc: supplied by vbat                                           
[    2.016292] hctosys: unable to open rtc device (rtc0)                        
[   11.596033] omap_rtc 44e3e000.rtc: omap_rtc_probe: rtc is_pmic_controller    
[   11.679800] rtc rtc0: 44e3e000.rtc: dev (253:0)                              
[   11.679841] omap_rtc 44e3e000.rtc: rtc core: registered 44e3e000.rtc as rtc0 
[   18.035852] PM: bootloader does not support rtc-only!

root@am335x-evm:~# poweroff                                                     
         Stopping Load/Save Random Seed...                                      
         Unmounting /run/media/mmcblk1p2... 
...
[  411.777818] reboot: Power down                                               
[  411.781026] System will go to power_off state in approx. 1 second            
[  411.787188] omap_rtc_power_off: deasserting OMAP_RTC_PMIC_POWER_EN

Regards,
Mike

rtc-omap-pmic-power-en-debug.patch

0 Michael Erdahl over 8 years ago in reply to Michael Erdahl

TI__Genius 13495 points

Harshvardhan,

I also verified "halt" is not the same as "poweroff". On my system, halt does not turn off the pmic, but poweroff does.

Regards,
Mike

0 Harshvardhan Ruchandani over 8 years ago in reply to Michael Erdahl

Prodigy 170 points

Hi Mike,

Yes, that's true but poweroff also gives the same kernel panic in our system. Although, I wil try the patch and get back to you with the results. I hope some more debugging info will help.

Thanks,
Harshvardhan

0 Harshvardhan Ruchandani over 8 years ago in reply to Michael Erdahl

Prodigy 170 points

Hi Mike,

I did patch the kernel and ran power off command. I still got the kernel panic but, I did get the debug message that we added in the rtc driver. I have attached the log in the file below.

for the driver message command I was able to see the rtc driver loaded and running.

dmesg | grep rtc

[ 0.000000] Kernel command line: console=tty0 console=ttyO0,115200n8 capemgrc

[ 4.903600] omap_rtc 44e3e000.rtc: already running

[ 4.908770] omap_rtc 44e3e000.rtc: omap_rtc_probe: rtc is_pmic_controller

[ 4.917036] omap_rtc 44e3e000.rtc: rtc core: registered 44e3e000.rtc as rtc0

[ 5.558228] omap_rtc 44e3e000.rtc: setting system clock to 2000-01-01 00:20:)

Thanks,

Harshvardhan

poweroff_log.txt

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
root@arm:~# poweroff                                                            
         Starting Synchronise Hardware Clock to System Clock...                 
[  OK  ] Stopped target RPC Port Mapper.                                        
         Stopping LSB: The Kamailio SIP proxy server...                         
[  OK  ] Stopped target Graphical Interface.                                    
[  OK  ] Stopped target Multi-User System.                                      
         Stopping Regular background program processing daemon...               
         Stopping OpenBSD Secure Shell server...                                
         Stopping Login Service...                                              
[  OK  ] Stopped target Login Prompts.                                          
         Stopping Getty on tty1...                                              
         Stopping Serial Getty on ttyS0...                                      
         Stopping LSB: Start daemon at boot time...                             
         Stopping LSB: Start daemon at boot time...                             
         Stopping LSB: Start daemon at boot time...                             
         Stopping LSB: set CPUFreq kernel parameters...                         
         Stopping LSB: Start daemon at boot time...                             
         Stopping D-Bus System Message Bus...                                   
         Stopping System Logging Service...                                     
[  OK  ] Stopped Regular background program processing daemon.                  
[  OK  ] Stopped OpenBSD Secure Shell server.                                   
[  OK  ] Stopped Login Service.                                                 
[  OK  ] Stopped D-Bus System Message Bus.                                      
[  OK  ] Stopped System Logging Service.                                        
[  OK  ] Stopped Getty on tty1.                                                 
[  OK  ] Stopped Serial Getty on ttyS0.                                         
[  OK  ] Stopped LSB: The Kamailio SIP proxy server.                            
[  OK  ] Stopped LSB: Start daemon at boot time.                                
[  OK  ] Stopped LSB: Start daemon at boot time.                                
[  OK  ] Stopped LSB: Start daemon at boot time.                                
[   94.842661] usb 1-1: new high-speed USB device number 2 using musb-hdrc      
[  OK  ] Stopped LSB: set CPUFreq kernel parameters.                            
[  OK  ] Stopped LSB: Start daemon at boot time.                                
[   95.672361] IPv6: ADDRCONF(NETDEV_CHANGE): umtsm1: link becomes ready        
[  OK  ] Started Synchronise Hardware Clock to System Clock.                    
         Stopping LSB: Load kernel modules needed to enable cpufreq scaling...  
         Stopping LSB: Start and stop the mysql database server daemon...       
[   98.117648] musb-hdrc musb-hdrc.1.auto: Babble                               
[   98.122689] musb_stage0_irq 856: unhandled DISCONNECT transition (a_wait_bco)
[  OK  ] Removed slice system-serial\x2dgetty.slice.                            
[  OK  ] Removed slice system-getty.slice.                                      
         Stopping /etc/rc.local Compatibility...                                
         Stopping Permit User Sessions...                                       
[  OK  ] Stopped LSB: Load kernel modules needed to enable cpufreq scaling.     
[   98.622852] usb 1-1: new full-speed USB device number 3 using musb-hdrc      
[  OK  ] Stopped Permit User Sessions.                                          
[   98.786541] usb 1-1: New USB device found, idVendor=0925, idProduct=1236     
[   98.804304] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   98.811610] usb 1-1: Product: Globalstar Zeus Remote NDIS                    
[   98.851468] usb 1-1: Manufacturer: Hughes                                    
[   98.864225] usb 1-1: SerialNumber: 0200dedcdce0                              
[   98.907087] PM: am33xx_prepare_push_sram_idle: EMIF function copy failed     
[   98.928812] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[   99.007047] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[   99.072881] cpu cpu0: of_init_opp_table: Failed to add OPP 800000000         
[   99.079429] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[   99.174536] cpu cpu0: of_init_opp_table: Failed to add OPP 600000000         
[   99.181088] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[   99.276675] cpu cpu0: of_init_opp_table: Failed to add OPP 300000000         
[   99.313544] cpu cpu0: of_pm_voltdm_notifier_register: Failed to get cpu0 reg7
[   99.365898] cpu cpu0: cpu0 clock notifier not ready, retry                   
[  OK  ] Stopped /etc/rc.local Compatibility.                                   
[   99.544274] cfg80211: Calling CRDA to update world regulatory domain         
[  100.228262] usbcore: registered new interface driver cdc_ether               
[  100.324117] PM: am33xx_prepare_push_sram_idle: EMIF function copy failed     
[  100.335940] rndis_host 1-1:1.0: RNDIS_MSG_QUERY(0x00010202) failed, -47      
[  100.344152] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

root@arm:~# poweroff                                                            
         Starting Synchronise Hardware Clock to System Clock...                 
[  OK  ] Stopped target RPC Port Mapper.                                        
         Stopping LSB: The Kamailio SIP proxy server...                         
[  OK  ] Stopped target Graphical Interface.                                    
[  OK  ] Stopped target Multi-User System.                                      
         Stopping Regular background program processing daemon...               
         Stopping OpenBSD Secure Shell server...                                
         Stopping Login Service...                                              
[  OK  ] Stopped target Login Prompts.                                          
         Stopping Getty on tty1...                                              
         Stopping Serial Getty on ttyS0...                                      
         Stopping LSB: Start daemon at boot time...                             
         Stopping LSB: Start daemon at boot time...                             
         Stopping LSB: Start daemon at boot time...                             
         Stopping LSB: set CPUFreq kernel parameters...                         
         Stopping LSB: Start daemon at boot time...                             
         Stopping D-Bus System Message Bus...                                   
         Stopping System Logging Service...                                     
[  OK  ] Stopped Regular background program processing daemon.                  
[  OK  ] Stopped OpenBSD Secure Shell server.                                   
[  OK  ] Stopped Login Service.                                                 
[  OK  ] Stopped D-Bus System Message Bus.                                      
[  OK  ] Stopped System Logging Service.                                        
[  OK  ] Stopped Getty on tty1.                                                 
[  OK  ] Stopped Serial Getty on ttyS0.                                         
[  OK  ] Stopped LSB: The Kamailio SIP proxy server.                            
[  OK  ] Stopped LSB: Start daemon at boot time.                                
[  OK  ] Stopped LSB: Start daemon at boot time.                                
[  OK  ] Stopped LSB: Start daemon at boot time.                                
[   94.842661] usb 1-1: new high-speed USB device number 2 using musb-hdrc      
[  OK  ] Stopped LSB: set CPUFreq kernel parameters.                            
[  OK  ] Stopped LSB: Start daemon at boot time.                                
[   95.672361] IPv6: ADDRCONF(NETDEV_CHANGE): umtsm1: link becomes ready        
[  OK  ] Started Synchronise Hardware Clock to System Clock.                    
         Stopping LSB: Load kernel modules needed to enable cpufreq scaling...  
         Stopping LSB: Start and stop the mysql database server daemon...       
[   98.117648] musb-hdrc musb-hdrc.1.auto: Babble                               
[   98.122689] musb_stage0_irq 856: unhandled DISCONNECT transition (a_wait_bco)
[  OK  ] Removed slice system-serial\x2dgetty.slice.                            
[  OK  ] Removed slice system-getty.slice.                                      
         Stopping /etc/rc.local Compatibility...                                
         Stopping Permit User Sessions...                                       
[  OK  ] Stopped LSB: Load kernel modules needed to enable cpufreq scaling.     
[   98.622852] usb 1-1: new full-speed USB device number 3 using musb-hdrc      
[  OK  ] Stopped Permit User Sessions.                                          
[   98.786541] usb 1-1: New USB device found, idVendor=0925, idProduct=1236     
[   98.804304] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   98.811610] usb 1-1: Product: Globalstar Zeus Remote NDIS                    
[   98.851468] usb 1-1: Manufacturer: Hughes                                    
[   98.864225] usb 1-1: SerialNumber: 0200dedcdce0                              
[   98.907087] PM: am33xx_prepare_push_sram_idle: EMIF function copy failed     
[   98.928812] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[   99.007047] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[   99.072881] cpu cpu0: of_init_opp_table: Failed to add OPP 800000000         
[   99.079429] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[   99.174536] cpu cpu0: of_init_opp_table: Failed to add OPP 600000000         
[   99.181088] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[   99.276675] cpu cpu0: of_init_opp_table: Failed to add OPP 300000000         
[   99.313544] cpu cpu0: of_pm_voltdm_notifier_register: Failed to get cpu0 reg7
[   99.365898] cpu cpu0: cpu0 clock notifier not ready, retry                   
[  OK  ] Stopped /etc/rc.local Compatibility.                                   
[   99.544274] cfg80211: Calling CRDA to update world regulatory domain         
[  100.228262] usbcore: registered new interface driver cdc_ether               
[  100.324117] PM: am33xx_prepare_push_sram_idle: EMIF function copy failed     
[  100.335940] rndis_host 1-1:1.0: RNDIS_MSG_QUERY(0x00010202) failed, -47      
[  100.344152] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[  100.380628] rndis_host 1-1:1.0 eth0: register 'rndis_host' at usb-musb-hdrc.d
[  100.392010] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[  100.417257] cpu cpu0: of_init_opp_table: Failed to add OPP 800000000         
[  100.435859] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[  100.451526] usbcore: registered new interface driver rndis_host              
[  100.472147] cpu cpu0: of_init_opp_table: Failed to add OPP 600000000         
[  100.490175] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[  100.528929] cfg80211: World regulatory domain updated:                       
[  100.534817] cpu cpu0: of_init_opp_table: Failed to add OPP 300000000         
[  100.541483] cpu cpu0: of_pm_voltdm_notifier_register: Failed to get cpu0 reg7
[  100.567147] usbcore: registered new interface driver rndis_wlan              
[  100.573860] cpu cpu0: cpu0 clock notifier not ready, retry                   
[  100.580029] PM: am33xx_prepare_push_sram_idle: EMIF function copy failed     
[  100.592975] cfg80211:  DFS Master region: unset                              
[  100.597463] cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_ga)
[  100.608981] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[  100.642036] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[  100.657894] cfg80211:   (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 )
[  100.672850] cfg80211:   (2457000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 )
[  100.681740] cpu cpu0: of_init_opp_table: Failed to add OPP 800000000         
[  100.694656] cfg80211:   (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 )
[  100.703429] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[  100.724726] cfg80211:   (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz A)
[  100.735183] cpu cpu0: of_init_opp_table: Failed to add OPP 600000000         
[  100.751335] cfg80211:   (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz A)
[  100.761525] cpu cpu0: _opp_add_dynamic: duplicate OPPs detected. Existing: f1
[  100.780033] cfg80211:   (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000)
[  100.789839] cpu cpu0: of_init_opp_table: Failed to add OPP 300000000         
[  100.799028] cfg80211:   (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 )
[  100.808554] cpu cpu0: of_pm_voltdm_notifier_register: Failed to get cpu0 reg7
[  100.820006] cfg80211:   (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0)
[  100.829435] cpu cpu0: cpu0 clock notifier not ready, retry                   
[  OK  ] Stopped LSB: Start and stop the mysql database server daemon.          
[  OK  ] Stopped target Host and Network Name Lookups.                          
         Stopping dnsmasq - A lightweight DHCP and caching DNS server...        
[  OK  ] Stopped target Remote File Systems.                                    
[  OK  ] Stopped target Remote File Systems (Pre).                              
[  OK  ] Stopped dnsmasq - A lightweight DHCP and caching DNS server.           
[  OK  ] Stopped target Basic System.                                           
[  OK  ] Stopped target Slices.                                                 
[  OK  ] Removed slice User and Session Slice.                                  
[  OK  ] Stopped target Paths.                                                  
[  OK  ] Stopped target Timers.                                                 
[  OK  ] Stopped target Sockets.                                                
[  OK  ] Closed D-Bus System Message Bus Socket.                                
[  OK  ] Closed Syslog Socket.                                                  
[  OK  ] Stopped target System Initialization.                                  
         Stopping Apply Kernel Variables...                                     
[  OK  ] Stopped Apply Kernel Variables.                                        
         Stopping Load Kernel Modules...                                        
[  OK  ] Stopped Load Kernel Modules.                                           
         Stopping Update UTMP about System Boot/Shutdown...                     
[  OK  ] Stopped target Encrypted Volumes.                                      
         Stopping LSB: RPC portmapper replacement...                            
[  OK  ] Stopped target Swap.                                                   
[  OK  ] Stopped Update UTMP about System Boot/Shutdown.                        
[  OK  ] Stopped LSB: RPC portmapper replacement.                               
[  OK  ] Stopped target Network is Online.                                      
[  OK  ] Stopped target Network.                                                
         Stopping LSB: Raise network interfaces....                             
         Stopping Create Volatile Files and Directories...                      
[  OK  ] Stopped Create Volatile Files and Directories.                         
[  OK  ] Stopped LSB: Raise network interfaces..                                
         Stopping Load/Save Random Seed...                                      
[  OK  ] Stopped target Local File Systems.                                     
         Unmounting /ram0...                                                    
[  OK  ] Stopped Load/Save Random Seed.                                         
[  OK  ] Unmounted /ram0.                                                       
[  OK  ] Reached target Unmount All Filesystems.                                
[  OK  ] Stopped target Local File Systems (Pre).                               
         Stopping Create Static Device Nodes in /dev...                         
[  OK  ] Stopped Create Static Device Nodes in /dev.                            
         Stopping Remount Root and Kernel File Systems...                       
[  OK  ] Stopped Remount Root and Kernel File Systems.                          
[  OK  ] Reached target Shutdown.                                               
[  105.454831] systemd-shutdown[1]: Sending SIGTERM to remaining processes...   
[  105.520859] systemd-journald[128]: Received SIGTERM from PID 1 (systemd-shut.
[  105.544414] systemd-shutdown[1]: Sending SIGKILL to remaining processes...   
[  105.603969] systemd-shutdown[1]: Unmounting file systems.                    
[  105.612014] systemd-shutdown[1]: Unmounting /sys/fs/fuse/connections.        
[  105.643323] systemd-shutdown[1]: Unmounting /sys/kernel/config.              
[  105.672863] systemd-shutdown[1]: Unmounting /dev/mqueue.                     
[  105.732705] systemd-shutdown[1]: Unmounting /sys/kernel/debug.               
[  106.001986] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)                    
[  106.039273] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)                    
[  106.045333] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)                    
[  106.051021] systemd-shutdown[1]: All filesystems unmounted.                  
[  106.057653] systemd-shutdown[1]: Deactivating swaps.                         
[  106.064047] systemd-shutdown[1]: All swaps deactivated.                      
[  106.069474] systemd-shutdown[1]: Detaching loop devices.                     
[  106.084263] systemd-shutdown[1]: All loop devices detached.                  
[  106.090090] systemd-shutdown[1]: Detaching DM devices.                       
[  106.097066] systemd-shutdown[1]: All DM devices detached.                    
[  106.128737] systemd-shutdown[1]: Powering off.                               
[  106.136122] musb-dsps 47401c00.usb: otg: usb_otg_unregister_hcd: device not e
[  106.145597] musb-hdrc musb-hdrc.1.auto: remove, state 1                      
[  106.151033] usb usb1: USB disconnect, device number 1                        
[  106.156509] usb 1-1: USB disconnect, device number 3                         
[  106.162243] rndis_host 1-1:1.0 eth0: unregister 'rndis_host' usb-musb-hdrc.1e
[  106.207287] musb-hdrc musb-hdrc.1.auto: USB bus 1 deregistered               
[  106.244810] reboot: Power down                                               
[  106.248056] System will go to power_off state in approx. 1 second            
[  106.254303] omap_rtc_power_off: deasserting OMAP_RTC_PMIC_POWER_EN           
[  108.739167] rtc_power_off failed, bailing out.                               
[  108.750372] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00
[  108.750372]                                                                  
[  108.759807] CPU: 0 PID: 1 Comm: systemd-shutdow Tainted: G           O    4.1
[  108.768158] Hardware name: Generic AM33XX (Flattened Device Tree)            
[  108.774454] [<c001a088>] (unwind_backtrace) from [<c0014aa4>] (show_stack+0x)
[  108.782401] [<c0014aa4>] (show_stack) from [<c09af2b8>] (dump_stack+0x8c/0xc)
[  108.789810] [<c09af2b8>] (dump_stack) from [<c09acb44>] (panic+0xb0/0x230)   
[  108.796849] [<c09acb44>] (panic) from [<c004aaf4>] (complete_and_exit+0x0/0x)
[  108.804359] [<c004aaf4>] (complete_and_exit) from [<c00686e4>] (SyS_reboot+0)
[  108.812572] [<c00686e4>] (SyS_reboot) from [<c0010920>] (ret_fast_syscall+0x)
[  108.820425] ---[ end Kernel panic - not syncing: Attempted to kill init! exi0
[  108.820425]

0 Michael Erdahl over 8 years ago in reply to Harshvardhan Ruchandani

TI__Genius 13495 points

Harshvardhan,

Can you try a kernel from the TI Processor SDK? I noticed you're using versions that are not provided by the SDK. We only do full regression tests on the kernels provided by the Processor SDK.

Also, have you made any modifications to the kernel that could be affecting the suspend path? Do you have a TI EVM you can use to verify an unmodified version of the kernel against your custom board?

Regards,
Mike

0 Michael Erdahl over 8 years ago in reply to Michael Erdahl

TI__Genius 13495 points

Hi Harshvardhan,

Have you been able to solve this issue, or is it still outstanding? As mentioned, would like to understand if you still experience issues with the TI kernel.

Regards,
Mike

0 Harshvardhan Ruchandani over 8 years ago in reply to Michael Erdahl

Prodigy 170 points

Hi Mike,

Thanks for reaching out. We are currently porting our stuff to the latest TI sdk uboot and kernel. As of now we can get to the uboot prompt and access MMC and we are working towards porting the latest version of sdk linux to our custom board. We have some question,

how does the the TI SDK linux rtc code sets PMIC_PWR_EN low?

Does it set ALARM2 using the RTC, is there any other way?

Thanks,

-Harshvardhan

0 Michael Erdahl over 8 years ago in reply to Harshvardhan Ruchandani

TI__Genius 13495 points

Harshvardhan,

Please take a look at the function omap_rtc_power_off in the file Linux/drivers/rtc/rtc-omap.c

If you have the "system-power-controller" property in your rtc device tree node, this function should be called when you power down.

Alternatively, fi you have the "ti,system-power-controller" property in your pmic device tree node, the tps65910_power_off function in Linux/drivers/mfd/tps65910.c will be called, which programs the PMIC off via an I2C command.

It seems as though your ALARM2 event is not triggering. Can you put a busy loop at the end of omap_rtc_power_off function, and verify if the ALARM2 interrupt was triggered - check the value of RTC_STATUS_REG - ALARM2 bit should be set, and I would think the RUN bit should be set too.

Regards,
Mike

0 Harshvardhan Ruchandani over 8 years ago in reply to Michael Erdahl

Prodigy 170 points

Mike,

Thanks for the info. As suggested we successfully ported the TI supported kernel to our custom board. After the port we still see the same kernel panic of rtc bailing out. We did apply the patch you provided earlier for debugging to the new TI SDK kernel, this time we were able to see the deasserting OMAP_RTC_PMIC_POWER_EN.

here is the log

root@am335x-evm:~# poweroff
INIT: Switching to runlevel: 0
INIT: Sending
Broadcast message from root@am335x-evm (ttyS0) (Sat Feb 27 01:25:19 2016):
The system is going down for system halt NOW!
Stopping syslogd/klogd: stopped syslogd (pid 575) stopped klogd (pid 578) done Deconfiguring network interfaces... ifdown: interface eth0 not configured
ifdown: interface usb0 not configured
done.
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Unmounting remote filesystems...
Deactivating swap...
Unmounting local filesystems...
[ 41.555965] reboot: Power down
[ 41.559144] System will go to power_off state in approx. 1 second
[ 41.565302] omap_rtc_power_off: deasserting OMAP_RTC_PMIC_POWER_EN
[ 44.062145] rtc_power_off failed, bailing out.

Apart from this we have not changed anything in the ti sdk kernel yet. Also, when compared to the schematic of the TI starter kit we don't hold the RTC_PORTZ low for 1ms could this be the source of problem?

Thanks,
Harshvardhan

0 Harshvardhan Ruchandani over 8 years ago in reply to Harshvardhan Ruchandani

Prodigy 170 points

Mike,

We were able to poweroff the board without a kernel panic by making changes in the omap rtc driver. The changes we made and the poweroff log is attached below.

Here is the section I changed (original and the new code in the #else case)
#if 0
val |= OMAP_RTC_PMIC_POWER_EN_EN | OMAP_RTC_PMIC_EXT_WKUP_POL(0) |
OMAP_RTC_PMIC_EXT_WKUP_EN(0);
#else
val |= OMAP_RTC_PMIC_POWER_EN_EN; /* allow turn off by ALARM2 */
val &= ~OMAP_RTC_PMIC_EXT_WKUP_POL(0); /* active high - ext wakeup tied to GND */
val &= ~OMAP_RTC_PMIC_EXT_WKUP_EN(0); /* disable ext wakeup (redundant since GND) */
#endif

Here is the log

root@am335x-evm:~# poweroff

Broadcast message from root@am335x-evm (ttyS0) (Sun Feb 28 20:03:48 2016):
The system is going down Stopping syslogd/klogd: stopped syslogd (pid 557)
stopped klogd (pid 560)
done
Deconfiguring network interfaces... ifdown: interface eth0 not configured
ifdown: interface usb0 not configured
done.
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Unmounting remote filesystems...
Deactivating swap...
Unmounting local filesystems...
[ 446.916430] reboot: Power down
[ 446.919611] System will go to power_off state in approx. 1 second
[ 446.925768] omap_rtc_power_off: deasserting OMAP_RTC_PMIC_POWER_EN
[ 446.931988] value to be written to OMAP_RTC_PMIC_REG = 0x71000
[ 446.937858] read back value OMAP_RTC_PMIC_REG = 0x10000
[ 446.943113] waiting for power off...

Now the board will not wake up unless I am holding the power button but, it shuts down if I let go or after about 10 seconds because of the PMIC long-press turn-off. We are able to boot board with battery disconnected and directly supplying the power. Before supplying the power the board needs to be disconnected from the USB as well.

Thanks,

Harshvardhan

0 Michael Erdahl over 8 years ago in reply to Harshvardhan Ruchandani

TI__Genius 13495 points

Harshvardhan,

Do you want to disable the external wakeup? I understand inverting the polarity to match your connection, however you would still want the external wakeup enabled.

Try restoring the OMAP_RTC_PMIC_EXT_WKUP_EN(0), and see if that enables wake from the power button.

Regards,
Mike

0 Harshvardhan Ruchandani over 8 years ago in reply to Michael Erdahl

Prodigy 170 points

Mike,

We tried restoring the OMAP_RTC_PMIC_EXT_WKUP_EN(0) but, it behaved the same as before.

What could be causing a need to hold power button for 1-2 seconds to keep power from shutting off after a powerdown. Does it take that long to load MLO from eMMC?

Maybe there is some delay caused by the connections.

I see 5 pins that are recommended on page 1210 of the TRM:

"If your application uses the RTC functionality and requires RTC-only mode, the internal LDO is required to enable proper wakeup signaling from the RTC domain. The proper wakeup signaling requires the following

connections:

• RTC_KALDO_ENn is grounded

• CAP_VDD_RTC is connected to 1uF decoupling capacitor to ground • RTC_PWRONRSTn is connected to 1.8V RTC power on reset • PMIC_POWER_EN is connected to power input of PMIC • EXT_WAKEUP0 is connected to a wakeup source See the device datasheet for more information on these signals."

Attached are the parts of schematic could you please have a look if this is the cause for additional hold time.

Thanks,

Harshvardhan

0 Michael Erdahl over 8 years ago in reply to Harshvardhan Ruchandani

TI__Genius 13495 points

Harshvardhan,

You connections look fine.

When you press the power button, do you see a transition on nRESPWRON immediately, or after the long press? nRESPWRON should take the processor out of reset once the PMIC reaches the ACTIVE state, so if you see a delay on this signal that corresponds to your long press, the PMIC state machine is not programmed properly.

Regards,
Mike

0 Harshvardhan Ruchandani over 8 years ago in reply to Michael Erdahl

Prodigy 170 points

Mike,

It comes out of reset OK, but then shuts off when we release the button unless we hold it for a few seconds so this is happening before the long press time.

The system is battery powered so when we press the button, the PMIC_POWER_EN is still low because of ALARM2 from the last poweroff command in Linux. The RTC Module is not reset by the power button in this design. It looks like we need to hold the power button long enough for u-boot to bring PMIC_POWER_EN high (which we added to am33xx_spl_board_init in board.c). The PMIC will give 1 second to do this
(tps65910 tdOINT ) but it must take longer than a second to load u-boot and set PWRHOLD high or we wouldn't have to hold the power button.

We think we need to move the code to bring PMIC_PWR_EN high earlier in u-boot so it executes in within 1 second. Is that possible? How long should it take to load u-boot from eMMC and execute the code to clear ALARM2?

Thanks,
Harshvardhan

0 Kevin LaRosa over 8 years ago in reply to Harshvardhan Ruchandani

TI__Genius 17380 points

Hi Harshvardhan,

Which TPS65910 part number are you using? (TPS65910A31A1 for example)

How long is it taking the processor to assert PWRHOLD? The t<sub>dOINT1</sub> (aka t<sub>dOINT</sub>) time delay is 1 s, so the processor would need to assert it within 1 s or else the power on condition would no longer be available if PWRON is released. See "Device power-on enable conditions" on page 47 for more information.

0 David McRell over 8 years ago in reply to Kevin LaRosa

Intellectual 445 points

Hello Harshvardhan,

Maybe I have simple solution, and it will take only a couple minutes to check with oscope.

I stumbled across this thread while searching for a similar symptom related to beagle bone: kernel panic associated with "rtc_power_off failed"

Referencing Y1, the 32.768kHz crystal, please check with an oscope that it is indeed oscillating. Our crystal was not oscillating, and was the sole cause of our "rtc_power_off failed".

If not, best of luck!

Kind Regards,

David McRell

0 Tyler Hopaluk over 7 years ago in reply to David McRell

Prodigy 90 points

David,

You have saved me many hours. I was just about to debug this same type of issue on our board:

[ 178.547622] reboot: Power down
[ 178.552183] System will go to power_off state in approx. 1 second
[ 181.035662] rtc_power_off failed, bailing out.
[ 181.051112] systemd-shutdow: 11 output lines suppressed due to ratelimiting
[ 181.058378] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000

The 5 seconds it took to probe the watch crystal revealed that it was indeed not oscillating. I never would have looked at that.

Something to fix in the morning.

Tyler

0 Harshvardhan Ruchandani over 7 years ago in reply to Michael Erdahl

Prodigy 170 points

Mike,

Thank you for your help throughout the process. We have successfully moved to the TI Processor SDK supported kernel and it did resolve our issues with pmic and kernel panic.

Thanks,
Harshvardhan

Processors

Processors forum

Linux/TPS65910: Kernel panic at shutdown