This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/AM4378: Kernel Hangs During Boot

Part Number: AM4378
Other Parts Discussed in Thread: TPS65218, TPS74801, AM4372

Tool/software: Linux

I have a custom board based on the AM4378 EVM, and I am trying to get it to boot, but it hangs when it tries to load the kernel. U-Boot runs all the way through and has the following output:

U-Boot SPL 2018.01-00558-g8617e02-dirty (Apr 01 2019 - 17:00:43)
Trying to boot from MMC1
SPL: Please implement spl_start_uboot() for your board
SPL: Direct Linux boot not active!


U-Boot 2018.01-00558-g8617e02-dirty (Apr 01 2019 - 17:00:43 -0500)

CPU : AM437X-GP rev 1.2
Model: TI AM437x UTI BOARD
DRAM: 512 MiB
Can't find PMIC:TPS65218_PMIC
NAND: 0 MiB
MMC: OMAP SD/MMC: 0
Net: <ethaddr> not set. Validating first E-fuse MAC
Could not get PHY for cpsw: addr 0
cpsw, usb_ether
Hit any key to stop autobo 0
=>setenv ip_method none
=>setenv bootfile zImage
=>setenv devtype mmc
=>setenv getuenv 'setenv devnum ${mmcdev}; if mmc rescan; then if run loadbootenv; then run importbootenv; fi; fi;'
=>setenv bootcmd 'mmc rescan; run findfdt; run getuenv; setenv devtype mmc; run loadimage; run loadfdt; run args_mmc; bootz ${loadaddr} - ${fdtaddr}'
=>saveenv
Saving Environment to FAT...
writing uboot.env
done
=>boot
709 bytes read in 3 ms (230.5 KiB/s)
Importing environment from mmc0 ...
3639496 bytes read in 203 ms (17.1 MiB/s)
41750 bytes read in 28 ms (1.4 MiB/s)
## Flattened Device Tree blob at 88000000
Booting using the fdt blob at 0x88000000
Loading Device Tree to 8fff2000, end 8ffff315 ... OK

Starting kernel ...

I can see when stepping through the code in CCS that execution gets to the function "rest_init", and stops there - it seems like it's going into some idle loop at that point. I am wondering if it has something to do with the fact that I am not using a PMIC to boot my board currently. I am using TPS74801 LDOs to boot the board, but it seems like U-Boot is expecting a PMIC to be used. Do I have to change some things in U-Boot to let it know that I am not using a PMIC chip, and if so, how do I do that?

  • Hi Tanner,

    I looked into the linux-kernel/init/main.c rest_init() function, and it does not seems to me that it is related to PMIC. Can you check with JTAG breakpoint at which rest_init() function your boot flow exactly hang?

    Can you please clarify if you are using AM437x PSDK Linux? If yes, which version?

    I would suggest you to enable kernel debug messages and early printk, thus you might have some messages printed by the kernel boot process.

    See below pointers for details:

    e2e.ti.com/.../2231166

    processors.wiki.ti.com/.../Kernel_-_Common_Problems_Booting_Linux

    training.ti.com/.../Kernel-Debug-Series-Part3-printk.pdf

    elinux.org/Debugging_by_printing

    e2e.ti.com/.../428227
    e2e.ti.com/.../349423
    e2e.ti.com/.../598561
    e2e.ti.com/.../658579


    Regards,
    Pavel
  • Hi Pavel,

    I am using SDK version 04.03.00.05. What all do I need to do to enable early debug messages and early printk? I tried adding CONFIG_DEBUG_LL=y and CONFIG_EARLY_PRINTK=y to my U-Boot config file, but I still am not getting any printouts. Is there something else I need to do to enable it?

    Also, in the function rest_int(), it looks like it's failing at the call to the function schedule_preempt_disabled().
  • Sorry, I meant Kernel config file. Right now I have the following set:

    CONFIG_DEBUG_LL=y
    CONFIG_DEBUG_AM33XXUART1=y
    CONFIG_DEBUG_LL_INCLUDE="mach/debug-macro.S"
    CONFIG_DEBUG_UART_8250=y
    CONFIG_EARLY_PRINTK=y

    But I am still not getting anything to print after "Starting kernel ...". From the other posts you pointed me to it seems like I might need to set bootargs to something, but the different posts all say to set bootargs to something different. Can you tell me exactly the steps I should to do to get the early printouts?
  • Hi,
    The kernel can hang before it gets to being able to do any early printouts. Here are a couple of questions:

    - Is the DTS file that you are using the exact same as the EVM or did you modify it? I would recommend falling back to a very minimal DTS, processor node, UART node with it's pin mux and the rootfs device and it's pin mux everything else turned off. Just to see the kernel start booting, let it crash when it tried to mount the root FS.

    - Try the GP-EVM dtb from the SDK you are using, most likely won't match the custom board but perhaps at least you will start to see output on the console and be able to move on from there.

    - What are the differences between the custom board in the EVM in terms of crystals? Are all the one that are implemented in the custom board the same as the EVM? I am interested to see if the RTC crystal is installed on the EVM.

    Best Regards,
    Schuyler
  • Hey Schuyler,

    To answer your questions:

    - I am using a custom DTS file right now. I have tried using the EVM DTS file, and that had the same result. I have been using a stripped down version of the DTS file that only includes UART and MMC nodes and their pin muxes. What exactly do you mean by the rootfs device?

    - I answered this in the first answer but I tried using the EVM DTB and had the same result.

    - I believe that the differences between the crystals on our custom board and the EVM is we are using a 19.2 MHz crystal for the main clock instead of 24 MHz, and that we do not have an RTC crystal.
  • Tanner,

    Can you attach your custom minimal kernel DTS file?

    AM437x EVM is using UART0 for console and MMC0 (mmc1 in kernel DTS) for SD card, what about your custom board?

    Regarding 19.2MHz crystal, you can check if you have the correct value 0x0 on sysboot[15:14] pins and CTRL_STS[23:22] SYSBOOT15_14 bits at the time of he kernel hang. Sysboot[15:14] pins can be checked with scope, SYSBOOT15_14 bits with CCS JTAG.

    "Also, in the function rest_int(), it looks like it's failing at the call to the function schedule_preempt_disabled()."

    Can you continue to investigate at which line exactly the boot flow hang? That might give us better view of the root cause.

    Regards,
    Pavel
  • Tanner,

    I have also checked rest_init() function on the AM437x EVM. I made the below update of the default kernel source code:

    linux-kernel/init/main.c

    asmlinkage __visible void __init start_kernel(void)

    {

    ........

    /* Do the rest non-__init'ed, we're now alive */

           + printk("before rest_init()\n");

    rest_init();

           + printk("after rest_init()\n");

    }

    I can see that "before rest_init()" is printed at very later stage of the kernel boot flow, I have so many console messages before that message. I can not see "after rest_init()" message, which means that it is normal that the flow never goes out of rest_init(). I am using PSDK v5.01, but should be similar for v4 of the PSDK.

    Please find the console log I have attached as txt file:

    https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/am437x_5F00_boot_5F00_log

    So your kernel is booting, but you do not see console messages. Please check and verify your UART console.

    Regards,
    Pavel

  • Pavel,

    I have verified both on the oscilloscope and in the CTRL_STS register that the value of SYS_BOOT[15:14] is set to 0x0.

    It seems like from your second post that even during a normal boot flow does not get past the function rest_init(), so it seems like that might be a dead end...

    I have attached the stripped down dts file that I am currently using. Our custom board also uses UART0 for the console and MMC0 for the SD card. U-Boot is printing everything out, so the console has to be set up correctly, right? Or is there a different spot in the kernel where you also have to set up the UART console?

    Another thing we have done is use all of our software configuration stuff (U-Boot configuration, dts file, kernel config file), load it onto the SD card, and plug the SD card into the EVM and boot. It fails terribly obviously, but the kernel still prints out all of its messages. Our board I'm sure is failing somewhere during the kernel boot process, but I just can't understand why it isn't printing out anything at all.

    /*
     * Copyright (C) 2013 Texas Instruments Incorporated - http://www.ti.com/
     *
     * This program is free software; you can redistribute it and/or modify
     * it under the terms of the GNU General Public License version 2 as
     * published by the Free Software Foundation.
     */
    
    /* AM437x GP EVM */
    
    /dts-v1/;
    
    #include "am4372.dtsi"
    #include <dt-bindings/pinctrl/am43xx.h>
    #include <dt-bindings/pwm/pwm.h>
    #include <dt-bindings/gpio/gpio.h>
    
    / {
    	model = "TI AM437x UTI BOARD";
    	compatible = "ti,am437x-UTI", "ti,am4372","ti,am43";
    
    	chosen {
    		stdout-path = &uart0;
    	};
    };
    
    &am43xx_pinmux {
    	uart_0_pins_default: uart_0_pins_default {
    		pinctrl-single,pins = <
    			AM4372_IOPAD(0x970, PIN_INPUT | MUX_MODE0)            // (K25) uart0_rxd.uart0_rxd
    			AM4372_IOPAD(0x974, PIN_OUTPUT_PULLDOWN | MUX_MODE0)  // (J24) uart0_txd.uart0_txd
    			/*AM4372_IOPAD(0x968, PIN_INPUT_PULLUP | MUX_MODE0)     // (L25) uart0_ctsn.uart0_ctsn
    			AM4372_IOPAD(0x96c, PIN_OUTPUT_PULLDOWN | MUX_MODE0)  // (J25) uart0_rtsn.uart0_rtsn*/
    		>;
    	};
    
    	// Optional sleep pin settings. Must manually enter values in the below skeleton.
    	uart_0_pins_sleep: uart_0_pins_sleep {
    		pinctrl-single,pins = <
    			AM4372_IOPAD(0x970, PIN_INPUT | MUX_MODE0)            // (K25) uart0_rxd.uart0_rxd
    			AM4372_IOPAD(0x974, PIN_OUTPUT_PULLDOWN | MUX_MODE0)  // (J24) uart0_txd.uart0_txd
    			/*AM4372_IOPAD(0x968, PIN_INPUT_PULLUP | MUX_MODE0)     // (L25) uart0_ctsn.uart0_ctsn
    			AM4372_IOPAD(0x96c, PIN_OUTPUT_PULLDOWN | MUX_MODE0)  // (J25) uart0_rtsn.uart0_rtsn*/
    		>;
    	};
    
    	mmc_0_pins_default: mmc_0_pins_default {
    		pinctrl-single,pins = <
    			AM4372_IOPAD(0x960, PIN_INPUT | MUX_MODE7) // (R25) spi0_cs1.mmc0_sdcd
    		>;
    	};
    };
    
    &mmc1 {
    	status = "okay";
    	bus-width = <4>;
    	pinctrl-names = "default";
    	pinctrl-0 = <&mmc_0_pins_default>;
    	broken-cd;
    };
    
    &uart0 {
    	status = "okay";
    	pinctrl-names = "default", "sleep";
    	pinctrl-0 = <&uart_0_pins_default>;
    	pinctrl-1 = <&uart_0_pins_sleep>;
    };
    
    
    

  • Tanner,

    First message after "Starting kernel" is:

    Starting kernel ...

    [ 0.000000] Booting Linux on physical CPU 0x0


    And this is printed in the above file and line:

    linux-kernel/arch/arm/kernel/setup.c

    void __init smp_setup_processor_id(void)
    {
    .......
    pr_info("Booting Linux on physical CPU 0x%x\n", mpidr);
    }

    You might put your breakpoint in this line to verify you have the same flow as AM437x EVM.


    In AM437x EVM DTS file, the UART0 pinmux is:

    uart0_pins_default: uart0_pins_default {
    pinctrl-single,pins = <
    AM4372_IOPAD(0x968, DS0_PULL_UP_DOWN_EN | INPUT_EN | MUX_MODE0) /* uart0_ctsn.uart0_ctsn */
    AM4372_IOPAD(0x96C, DS0_PULL_UP_DOWN_EN | INPUT_EN | MUX_MODE0) /* uart0_rtsn.uart0_rtsn */
    AM4372_IOPAD(0x970, PIN_INPUT_PULLUP | SLEWCTRL_FAST | DS0_PULL_UP_DOWN_EN | MUX_MODE0) /* uart0_rxd.uart0_rxd */
    AM4372_IOPAD(0x974, PIN_INPUT | PULL_DISABLE | SLEWCTRL_FAST | DS0_PULL_UP_DOWN_EN | MUX_MODE0) /* uart0_txd.uart0_txd */
    >;
    };


    I see you have some differences in your custom DTS file. You can try to align with kernel DTS file and see if that will give any improvement.

    Make sure you have setup "console=ttyO0,115200n8" in your bootargs for the kernel command line.

    You can also dump UART0 pinmux and module registers in u-boot stage and in kernel, and compare if you have any differences in the registers settings. At u-boot stage, this can be done with "md" command.

    Regards,
    Pavel
  • I tried making the pin settings exactly as you have listed above, but still nothing. When I run the md command in U-Boot, I got the following:

    => md 0x44e10970
    44e10970: 000e0000 000b0000 08060007 08060007

    I think those settings look right? Based on what the register values are set to in mux.c, although mux.c doesn't set the ctsn or rtsn registers, so I'm not sure what those values should be.

    When you say "Make sure you have setup 'console=ttyO0,115200n8' in your bootargs for the kernel command line", does that mean just having console=ttyO0,115200n8 in the U-Boot environment? Because I do have that in there.

    Also, I can see that execution gets into the smp_setup_processor_id() function, and to the printk line, but nothing prints. So it seems like what you said in your previous response, that the kernel is booting but not printing anything, is correct. It seems like printk functions are being called, but not printing anything.
  • Hi,
    I would like to add a suggestion of something to try since the RTC crystal is not in the board design that is in the TI EVM design.

    Could set a breakpoint on this function in arch/arm/mach-omap2/omap_hwmod_43xx_data.c and step through it to see if the marked lines below are executed? You might also try commenting them out.

    Also please comment out the MMC node for the time being, this is the device used by the root file system. This mmc definition has an incorrect pin mux array, only one pin is defined for a 4 pin interface.

    Regards,
    Schuyler

    int __init am43xx_hwmod_init(void)
    {
    int ret;

    omap_hwmod_am43xx_reg();
    omap_hwmod_init();
    ret = omap_hwmod_register_links(am43xx_hwmod_ocp_ifs);

    -> if (!ret && of_machine_is_compatible("ti,am4372"))
    -> ret = omap_hwmod_register_links(am43xx_rtc_hwmod_ocp_ifs);

    return ret;
    }
  • what is the proper setup for a system without an RTC crystal installed? is that something you set up in u-boot for the kernel (changing a clock MUX) or does the kernel set it up and you have to change a config setting? or do you have to do what you suggested and directly edit the kernel code? i see in the menuconfig there is an option in > System Type > TI OMAP Common Features > Use 32KHz timer. does that mean the RTC clock or is this option unrelated?

    i see from the clock tree tool there are 5 things that use the RTC crystal besides the RTC module. Timer0, GPIO0, USB phy 0/1 always on, and sync timer. which one of these could cause this behavior?
  • Tanner,

    Tanner Hunsucker73 said:
    I tried making the pin settings exactly as you have listed above, but still nothing. When I run the md command in U-Boot, I got the following:

    => md 0x44e10970
    44e10970: 000e0000 000b0000 08060007 08060007

    I think those settings look right? Based on what the register values are set to in mux.c, although mux.c doesn't set the ctsn or rtsn registers, so I'm not sure what those values should be.

    Please align your kernel DTS UART0 pinmux settings, with thse from u-boot, as these are known working settings.

    Tanner Hunsucker73 said:
    When you say "Make sure you have setup 'console=ttyO0,115200n8' in your bootargs for the kernel command line", does that mean just having console=ttyO0,115200n8 in the U-Boot environment? Because I do have that in there.

    Yes, u-boot environment.

    Tanner Hunsucker73 said:
    Also, I can see that execution gets into the smp_setup_processor_id() function, and to the printk line, but nothing prints. So it seems like what you said in your previous response, that the kernel is booting but not printing anything, is correct. It seems like printk functions are being called, but not printing anything.

    Have you mage any significant changes in your defconfig file? Can you attach your.config file, may be you have some UART misconfiguration there.

    What else I can suggest you is to:

    - probe the UART0 signals with scope and compare the signals with u-boot stage

    - dump UART0 module registers settings in u-boot and in kernel and compare for differences

    Also make sure you have remove any reference to RTC, see below e2e thread for details:

    Regards,
    Pavel

  • Schuyler and Pavel,

    Commenting out the two lines that Schuyler pointed me to allowed me to see the kernel messages. I can now boot all the way and get to the command line. Mike and I are wondering, do you have to comment out these two lines and directly edit the kernel code to get past this, or is there some config option in U-Boot or the Kernel you can set to let it know that we do not have a RTC crystal on our board?
  • Hi,

    For the moment the code will need to be commented out in the kernel to allow you continued development. I will need to research this a little further with the driver development team to understand if there is a config option or if commenting out this code is the best solution.

    The other question asked by a user on this thread about the configurability of this RTC-less use case is one that I will also research where the DTS board process happens in relation to the low-level init process called hwmod. 

    Best Regards,

    Schuyler

  • Tanner,

    I made a search and have found below e2e thread:

    e2e.ti.com/.../656735

    So you can revert back the patch that enables RTC in omap_hwmod_43xx_data.c or a little logic needs to be added to use the "compatible" field to check to see what board is booting and call this code appropriately. The current omap_hwmod_43xx_data.c is using the generic "ti,am4372", while you can add a check for your custom board string and skip RTC enable if that string match.

    You can create your custom board string similar to strings used for GP EVM, Starter Kit, IDK, Compulab boards:

    ti,am437x-gp-evm
    ti,am437x-sk-evm
    ti,am437x-idk-evm
    compulab,am437x-cm-t43
    compulab,am437x-sbc-t43

    Regards,
    Pavel