[AM335x] Kernel Crashes When on Soft Reset

Note that I think the crash below is when trying write the time to the hwclock, that’s the process that is getting hung and triggering the watchdog.

Are there any known problems with the hwclock and Linux?

Hi,

It does seem like the hwclock task is hanging (or taking really long time to complete) and thus triggering a soft lockup warning. Is there a chance the /dev/rtc symlink has been unregistered before hwclock has written the system time to it? I'm not a hundred percent sure, but I think hwclock expects /dev/rtc to exist in order to access the RTC through the symlink. (EDIT: Judging by what's written in the man page, it seems like the hwclock should fall back to another method if /dev/rtc is not found. However I don't know if the embedded version of hwclock has the same functionality as the desktop version.)

Reading the man page for hwclock I found this:

How hwclock Accesses the Hardware Clock
       hwclock uses many different ways to get and set Hardware Clock values.  The most normal way is to do I/O to the device special file /dev/rtc, which is presumed to be driven by the rtc device driver. However,  this  method  is  not  always  available.  For one thing, the rtc driver is a relatively recent addition to Linux.  Older systems don't have it.  Also, though there are versions of the rtc driver that work on DEC Alphas, there appear to be plenty of Alphas on which the rtc driver does not work (a common symptom is hwclock hanging).  Moreover, recent Linux  systems  have  more  generic support for RTCs, even systems that have more than one, so you might need to override the default by specifying /dev/rtc0 or /dev/rtc1 instead.

       On older systems, the method of accessing the Hardware Clock depends on the system hardware.

       On  an ISA system, hwclock can directly access the "CMOS memory" registers that constitute the clock, by doing I/O to Ports 0x70 and 0x71.  It does this with actual I/O instructions and consequently can only do it if running with superuser effective userid.  (In the case of a Jensen Alpha, there is no way for hwclock to execute those I/O instructions, and so it uses instead the /dev/port device special file, which provides almost as low-level an interface to the I/O subsystem).

       This  is a really poor method of accessing the clock, for all the reasons that user space programs are generally not supposed to do direct I/O and disable interrupts.  Hwclock provides it because it is the only method available on ISA and Alpha systems which don't have working rtc device drivers available.

       On an m68k system, hwclock can access the clock via the console driver, via the device special file /dev/tty1.

       hwclock tries to use /dev/rtc.  If it is compiled for a kernel that doesn't have that function or it is unable to open /dev/rtc (or the alternative special file you've defined on the  command  line) hwclock will fall back to another method, if available.  On an ISA or Alpha machine, you can force hwclock to use the direct manipulation of the CMOS registers without even trying /dev/rtc by specifying the --directisa option.

I know this is mentioned as related to Alpha systems, but I guess if there is any problem with the RTC driver, hwclock would hang on any other system as well.

You may try and run a script that constantly reads or updates the RTC through hwclock and see if any hanging would occur.

Also reading the hwclock man page (man hwclock) may give you an idea about what could be going wrong. I know the embedded version of hwclock included in the busybox binary is much smaller and trimmed as opposed to the hwclock version on your host PC, but still...

This is all I can think of.

Best regards,
Miroslav

Hi Miroslav,

I'm the customer with the problem.  To add some more data, our system has only /dev/rtc0 and we are using Debian, so are running the full version of hwclock as opposed to a BusyBox version.

I've copied the Debian RTC init script to http://pastebin.com/Z4qjbAQD

I ran a test for about 30 mins where I was calling /etc/init.d/hwclock stop repeatedly, but this did not expose the error unfortunately.

Hi Sean,

Just a quick morning idea - can you try substituting the full version binary of hwclock with a link to "busybox hwclock". I have no idea if it would work without hanging (probably not if the problem is with the driver), but I guess it's worth a try.

Best regards,
Miroslav

I think it would be better to try and figure out the root cause from the register dump.  Can TI assist with that?

Sean,

Let's test Miroslav theory by removing the symlink to the RTC and then call hwclock either nativly or by busybox.  This should trigger the lock up condition (hopefully).

Note that there is no symlink to any rtc, there is just /dev/rtc0.

Miroslav,

Another were going to try is to increase priority of the hwclock.sh in the shutdown sequence.  We are still having a hard time reliably reproducing the issue, so this "fix" will be hard to test.  Any additional insight is always appreciated.

Tear Down Sequence: