AM4378: l4-wkup-clkctrl:0118:0: failed to enable

Part Number: AM4378

Tool/software:

Hi,

We are using a custom AM4378 board with realtime kernel 5.10.168 (ti-rt-linux-5.10.y 101d44260b8dfbab27763581e4bbd534dd36ae33).
Occasionally, we get the following kernel message:

Jun 19 09:45:24 5555 kernel: l4-wkup-clkctrl:0118:0: failed to enable

From /sys/kernel/debug/clk/clk_summary:
 clk_rc32k_ck                         1        2        0       32768          0     0  50000
 ...
    wdt1_fck                          1        1        0       32768          0     0  50000
       l4-wkup-clkctrl:0118:0         1        1        0       32768          0     0  50000

I suspect that this is connected to the watchdog (which we are using).

What is the significance and severity of this message?

Let me know if you need more information.

Kind regards, Daniel

  • Hello Daniel,

    Support expectations

    Just to set expectations, SDK 8.2 is now too old for us to support on the forums. I will still do what I can to assist you, but I will not be running any tests on hardware.

    Off the top of my head, this is not a terminal output that I am familiar with.

    Please tell us more about what you are observing 

    How often are you seeing this output?

    After you see the output, does the watchdog still work as expected? for example,

    1) can you still read and write to the watchdog registers (e.g., with devmem2)

    2) can you still configure the watchdog and see it reboot the system if it does not get pet

    Other thoughts

    This clock is used by multiple peripherals. If you are using any of the other peripherals sourced by the clock, you could do a quick sanity check to make sure the clock itself was still there by testing those peripherals:

    linux-5.10.100+gitAUTOINC+7a7a3af903-g7a7a3af903/arch/arm/boot/dts$ grep -r --include=am43* l4_wkup_clkctrl
    am43xx-clocks.dtsi:             l4_wkup_clkctrl: l4-wkup-clkctrl@220 {
    am437x-l4.dtsi:                 clocks = <&l4_wkup_clkctrl AM4_L4_WKUP_GPIO1_CLKCTRL 0>,
    am437x-l4.dtsi:                          <&l4_wkup_clkctrl AM4_L4_WKUP_GPIO1_CLKCTRL 8>;
    am437x-l4.dtsi:                 clocks = <&l4_wkup_clkctrl AM4_L4_WKUP_UART1_CLKCTRL 0>;
    am437x-l4.dtsi:                 clocks = <&l4_wkup_clkctrl AM4_L4_WKUP_I2C1_CLKCTRL 0>;
    am437x-l4.dtsi:                 clocks = <&l4_wkup_clkctrl AM4_L4_WKUP_TIMER1_CLKCTRL 0>;
    am437x-l4.dtsi:                 clocks = <&l4_wkup_clkctrl AM4_L4_WKUP_WD_TIMER2_CLKCTRL 0>;
    
    Where AM4_L4_WKUP_WD_TIMER2_CLKCTRL is the watchdog timer

    Regards,

    Nick

  • Hello Nick

    Thanks for your answer.

    - It happens rarely, about 5 times the last couple of months on the two systems I have logs for.
    - I can still configure the watchdog and it reboots the system if not pet.
    - I checked a GPIO pin (on bank GPIO1) and it still worked as expected, e.g. I can toggle the pin.

    Kind regards, Daniel

  • Hello Daniel,

    And to confirm, you are able to configure the watchdog and the watchdog works on a boot where you see that warning output? Or have you only been able to test on a "normal" boot?

    There are a bunch of drivers that print "failed to enable", but it is actually pretty rare for that to be the entire output - most output tells you a bit more about what failed to enable. It looks like the print you are seeing is probably coming from the clkctrl driver:

    drivers/clk/ti/clkctrl.c
    
    static int _omap4_clkctrl_clk_enable(struct clk_hw *hw)
    {
        ...
            /* Wait until module is enabled */
            while (!_omap4_is_ready(ti_clk_ll_ops->clk_readl(&clk->enable_reg))) {
                    if (_omap4_is_timeout(&timeout, OMAP4_MAX_MODULE_READY_TIME)) {
                            pr_err("%s: failed to enable\n", clk_hw_get_name(hw));
                            return -EBUSY;
                    }
            }
    

    The next step would be to verify that this is actually where the print is coming from

    Assuming this is the cause of the print statement:

    So this is a timeout error. It looks like the error happens if the timer is not ready within 2msec (2000 us).

    Depending on what else is going on during boot, I have seen non-RT Linux drivers like I2C get paused for tens to hundreds of milliseconds on a more recent device (AM62x). I am not sure how much control you have during boot time to raise the relative priority of this driver, but I would hope that RT Linux would help to limit the worst-case timing.

    Without looking into the potential follow-on effects, the easiest "fix" would probably be to just increase the timeout value. As long as your entire boot is not freezing or crashing, I would expect that the ARM core would eventually finish up the other tasks that it was executing, and come back to the clkctrl driver to finish setting up the clocks.

    Regards,

    Nick

  • Hello Nick

    Yes, after the warning message was displayed, the watchdog and a gpio still operated as expected.

    I can confirm that the error is coming from the location you pointed out, I also added a message on success (see below).

    I managed to reproduce it: I am repeatedly enabling and disabling the watchdog (by writing V and then abc to /dev/watchdog).
    Then I introduce heavy load on the cpu by busy looping. Eventually, it will fail.

    [  210.382703] l4-wkup-clkctrl:0118:0: enabled successfully
    [  210.431105] watchdog: watchdog0: watchdog did not stop!
    [  211.013113] l4-wkup-clkctrl:0118:0: enabled successfully
    [  211.089211] watchdog: watchdog0: watchdog did not stop!
    [  211.620067] l4-wkup-clkctrl:0118:0: enabled successfully
    [  211.668160] watchdog: watchdog0: watchdog did not stop!
    [  212.247453] l4-wkup-clkctrl:0118:0: enabled successfully
    [  212.289189] watchdog: watchdog0: watchdog did not stop!
    [  212.918972] l4-wkup-clkctrl:0118:0: enabled successfully
    [  213.001608] watchdog: watchdog0: watchdog did not stop!
    [  213.541859] l4-wkup-clkctrl:0118:0: failed to enable       <- failure
    [  213.597160] watchdog: watchdog0: watchdog did not stop!
    [  214.193884] watchdog: watchdog0: watchdog did not stop!
    [  214.757848] watchdog: watchdog0: watchdog did not stop!
    [  215.312557] watchdog: watchdog0: watchdog did not stop!

    It is now also clear that at least for this timer it is not related to system boot since the timer is enabled/disabled when the watchdog itself is enabled/disabled.

    It seems to me that the error condition returned by `_omap4_clkctrl_clk_enable` is not evaluated and acted upon.

    Kind regards, Daniel