Boot Fail @ "PM: Adding info for platform:arm-pmu.0"

Hi Michael,

Michael Podel23 said:

We respun the boards with minor changes unrelated to the DMP and cannot get any boards to fully boot.

Can you share what are these changes? What is DMP? Do you mean DPM (Dynamic Power Management)?

Can you explain from where you get the linux kernel source? Do you make any changes in the linux kernel code base or you are using the default (ti8148_evm_defconfig)?

Michael Podel23 said:

Kernel command line: console=ttyO0,115200n8 root=/dev/ram0 rw initrd=0x82000000,40MB mem=176M notifyk.vpssm3_sva=0xBF900000 earlyprintk debug ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}::eth0:on

If you remove "debug" from the environment settings, at which point exactly the kernel will hang?

Regards,
Pavel

Hi Pavel,

Thank you for your quick response!

Changes that were made were for issues unrelated to the TMS320DM8148 which we call our Digital Media Processor (DMP) as far as we can tell and it would be hard to describe them.

I am one of the electrical engineer's on this issue and am not as familiar with the software specifics.  I will have our software engineer respond to this when he returns on Monday.

Thank You!

M. Podel

Hello Pavel

The Linux kernel source is from the EZSDK 05_05_02_00. We used the ti8148_evm_defconfig but made changes to the kernel config for our needs. The two versions of the board are loaded with the exact same software loads. One board version boots completely and we can successfully utilize the hardware encoders. The other board version stops before it finishes loading the kernel. I've pasted the console output after removing the "debug" from the environment settings. 

Environment size: 512/8188 bytes
TI8148_EVM#boot
16384 KiB N25Q128A at 0:0 is now current device
## Booting kernel from Legacy Image at 81000000 ...
   Image Name:   Linux-2.6.37
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2692160 Bytes = 2.6 MiB
   Load Address: 80008000
   Entry Point:  80008000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
Linux version 2.6.37 (tidev@ubuntu) (gcc version 4.3.3 (Sourcery G++ Lite 2009q1-203) ) #14 Tue Jan 12 13:36:25 EST 2016
CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine: ti8148evm
bootconsole [earlycon0] enabled
reserved size = 5242880 at 0x0
FB: Reserving 5242880 bytes SDRAM for VRAM
Memory policy: ECC disabled, Data cache writeback
OMAP chip is TI8148 3.0
SRAM: Mapped pa 0x402f1000 to va 0xfe400000 size: 0xf000
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 43424
Kernel command line: console=ttyO0,115200n8 root=/dev/ram0 rw initrd=0x82000000,40MB mem=176M notifyk.vpssm3_sva=0xBF900000 earlyprintk ip=192.168.0.251:192.168.0.150:192.168.0.150:255.255.255.0::eth0:on
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 171MB = 171MB total
Memory: 126780k/126780k available, 53444k reserved, 0K highmem
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
    DMA     : 0xffc00000 - 0xffe00000   (   2 MB)
    vmalloc : 0xcb800000 - 0xf8000000   ( 712 MB)
    lowmem  : 0xc0000000 - 0xcb000000   ( 176 MB)
    pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
    modules : 0xbf000000 - 0xbfe00000   (  14 MB)
      .init : 0xc0008000 - 0xc003c000   ( 208 kB)
      .text : 0xc003c000 - 0xc0515000   (4964 kB)
      .data : 0xc0516000 - 0xc0557980   ( 263 kB)
SLUB: Genslabs=11, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:407
omap_hwmod: gpio2: softreset failed (waited 10000 usec)
omap_hwmod: gpio3: softreset failed (waited 10000 usec)
omap_hwmod: gpio4: softreset failed (waited 10000 usec)
IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts
Total of 128 interrupts on 1 active controller
GPMC revision 6.0
Trying to install interrupt handler for IRQ400
Trying to install interrupt handler for IRQ401
Trying to install interrupt handler for IRQ402
Trying to install interrupt handler for IRQ403
Trying to install interrupt handler for IRQ404
Trying to install interrupt handler for IRQ405
Trying to install interrupt handler for IRQ406
Trying to install type control for IRQ407
Trying to set irq flags for IRQ407
OMAP clockevent source: GPTIMER1 at 20000000 Hz
Console: colour dummy device 80x30
Calibrating delay loop... 598.01 BogoMIPS (lpj=2990080)
pid_max: default: 32768 minimum: 301
Security Framework initialized
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
devtmpfs: initialized
omap_voltage_early_init: voltage driver support not added
regulator: core version 0.5
regulator: dummy:
NET: Registered protocol family 16
omap_voltage_domain_lookup: Voltage driver init not yet happened.Faulting!
omap_voltage_add_dev: VDD specified does not exist!
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
omap_mux_init: Add partition: #1: core, flags: 4
SPI Init
hw-breakpoint: debug architecture 0x4 unsupported.
hw perfevents: enabled with ARMv7 Cortex-A8 PMU driver, 5 counters available

On the good boards, the next printout is

Debugfs: Only enabling/disabling deep sleep and wakeup timer is supported now

I tried another kernel with the DebugFS options disabled, Kernel hacking -> Debug Filesystem. It stopped in the exact same spot whether the "debug" environment setting was there or not.

Thanks,

Shaun

Hi Shaun,

Shaun said:

OMAP chip is TI8148 3.0

I see your non-booting board is silicon revision 3.0

What about the booting board, is it also 3.0 chip?

Regards,
Pavel

Shaun said:

OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
omap_mux_init: Add partition: #1: core, flags: 4
SPI Init
hw-breakpoint: debug architecture 0x4 unsupported.
hw perfevents: enabled with ARMv7 Cortex-A8 PMU driver, 5 counters available

On the DM814x TI EVM I have:

OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
omap_mux_init: Add partition: #1: core, flags: 4
NOR: Can't request GPMC CS
Cannot clk_get ck_32
Debugfs: Only enabling/disabling deep sleep and wakeup timer is supported now

Have you add something regarding SPI in linux kernel? Note that the kernel handles SPI at later stage, see if you are aligned with the below wiki page:

Regards,
Pavel

Can you put one printk right after perf_pmu_register(&pmu) line in /arch/arm/kernel/perf_event.c init_hw_perf_events() function. I want to see if the kernel stuck inside this perf_pmu_register(&pmu) function:

if (armpmu) {
pr_info("enabled with %s PMU driver, %d counters available\n",
arm_pmu_names[armpmu->id], armpmu->num_events);
} else {
pr_info("no hardware support available\n");
}

perf_pmu_register(&pmu);
printk("after perf_pmu_register\n"); -----> put printk
return 0;
}
arch_initcall(init_hw_perf_events);

I see that ARM PMU is not enabled by default, can you provide me the steps used to enable it, I will try it on my side. Refer also to the below pointers regarding ARM PMU usage:

e2e.ti.com/.../308981
e2e.ti.com/.../439073
e2e.ti.com/.../1351193
e2e.ti.com/.../184885

Regards,
Pavel

Pavel Botev said:

Hi Shaun,

Shaun

OMAP chip is TI8148 3.0

I see your non-booting board is silicon revision 3.0

What about the booting board, is it also 3.0 chip?

Regards,
Pavel

Yeah both boards are using the 3.0 chip

Pavel Botev said:

Shaun

OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
omap_mux_init: Add partition: #1: core, flags: 4
SPI Init
hw-breakpoint: debug architecture 0x4 unsupported.
hw perfevents: enabled with ARMv7 Cortex-A8 PMU driver, 5 counters available

On the DM814x TI EVM I have:

OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
omap_mux_init: Add partition: #1: core, flags: 4
NOR: Can't request GPMC CS
Cannot clk_get ck_32
Debugfs: Only enabling/disabling deep sleep and wakeup timer is supported now

Have you add something regarding SPI in linux kernel? Note that the kernel handles SPI at later stage, see if you are aligned with the below wiki page:

User guide Try this guide here.

Regards,
Pavel

We did make changes to the SPI but they are aligned with the wiki page you provided. Both boards use the same SPI chip in the same exact way.

Pavel Botev said:

Can you put one printk right after perf_pmu_register(&pmu) line in /arch/arm/kernel/perf_event.c init_hw_perf_events() function. I want to see if the kernel stuck inside this perf_pmu_register(&pmu) function:

if (armpmu) {
pr_info("enabled with %s PMU driver, %d counters available\n",
arm_pmu_names[armpmu->id], armpmu->num_events);
} else {
pr_info("no hardware support available\n");
}

perf_pmu_register(&pmu);
printk("after perf_pmu_register\n"); -----> put printk
return 0;
}
arch_initcall(init_hw_perf_events);

I see that ARM PMU is not enabled by default, can you provide me the steps used to enable it, I will try it on my side. Refer also to the below pointers regarding ARM PMU usage:

e2e.ti.com/.../308981
e2e.ti.com/.../439073
e2e.ti.com/.../1351193
e2e.ti.com/.../184885

Regards,
Pavel

After adding the printk, the console output was the following:

Net:   cpsw_pad_config: get_cpu_rev() != PG1_0, 4
Detected MACID:0:11:22:33:44:55
cpsw
Hit any key to stop autoboot:  0
16384 KiB N25Q128A at 0:0 is now current device
## Booting kernel from Legacy Image at 81000000 ...
   Image Name:   Linux-2.6.37
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2692164 Bytes = 2.6 MiB
   Load Address: 80008000
   Entry Point:  80008000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
Linux version 2.6.37 (tidev@ubuntu) (gcc version 4.3.3 (Sourcery G++ Lite 2009q1-203) ) #17 Tue Feb 2 11:38:41 EST 2016
CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine: ti8148evm
bootconsole [earlycon0] enabled
reserved size = 5242880 at 0x0
FB: Reserving 5242880 bytes SDRAM for VRAM
Memory policy: ECC disabled, Data cache writeback
OMAP chip is TI8148 3.0
SRAM: Mapped pa 0x402f1000 to va 0xfe400000 size: 0xf000
On node 0 totalpages: 43776
free_area_init_node: node 0, pgdat c0555f78, node_mem_map c0594000
  Normal zone: 352 pages used for memmap
  Normal zone: 0 pages reserved
  Normal zone: 43424 pages, LIFO batch:7
pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
pcpu-alloc: [0] 0
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 43424
Kernel command line: console=ttyO0,115200n8 root=/dev/ram0 rw initrd=0x82000000,40MB mem=176M notifyk.vpssm3_sva=0xBF900000 earlyprintk debug ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}::eth0:on
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 171MB = 171MB total
Memory: 126780k/126780k available, 53444k reserved, 0K highmem
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
    DMA     : 0xffc00000 - 0xffe00000   (   2 MB)
    vmalloc : 0xcb800000 - 0xf8000000   ( 712 MB)
    lowmem  : 0xc0000000 - 0xcb000000   ( 176 MB)
    pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
    modules : 0xbf000000 - 0xbfe00000   (  14 MB)
      .init : 0xc0008000 - 0xc003c000   ( 208 kB)
      .text : 0xc003c000 - 0xc0515000   (4964 kB)
      .data : 0xc0516000 - 0xc0557980   ( 263 kB)
SLUB: Genslabs=11, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:407
omap_hwmod: gpio2: softreset failed (waited 10000 usec)
omap_hwmod: gpio3: softreset failed (waited 10000 usec)
omap_hwmod: gpio4: softreset failed (waited 10000 usec)
IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts
Total of 128 interrupts on 1 active controller
GPMC revision 6.0
Trying to install interrupt handler for IRQ400
Trying to install interrupt handler for IRQ401
Trying to install interrupt handler for IRQ402
Trying to install interrupt handler for IRQ403
Trying to install interrupt handler for IRQ404
Trying to install interrupt handler for IRQ405
Trying to install interrupt handler for IRQ406
Trying to install type control for IRQ407
Trying to set irq flags for IRQ407
OMAP clockevent source: GPTIMER1 at 20000000 Hz
Console: colour dummy device 80x30
Calibrating delay loop... 598.01 BogoMIPS (lpj=2990080)
pid_max: default: 32768 minimum: 301
Security Framework initialized
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
devtmpfs: initialized
PM: Adding info for No Bus:platform
omap_voltage_early_init: voltage driver support not added
PM: Adding info for No Bus:omap
regulator: core version 0.5
PM: Adding info for platform:reg-dummy
PM: Adding info for No Bus:regulator.0
regulator: dummy:
NET: Registered protocol family 16
PM: Adding info for platform:mpu.0
omap_voltage_domain_lookup: Voltage driver init not yet happened.Faulting!
omap_voltage_add_dev: VDD specified does not exist!
PM: Adding info for platform:l3_slow.0
PM: Adding info for platform:omap_gpio.0
PM: Adding info for platform:omap_gpio.1
PM: Adding info for platform:omap_gpio.2
PM: Adding info for platform:omap_gpio.3
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
OMAP GPIO hardware version 0.1
PM: Adding info for No Bus:gpiochip0
PM: Adding info for No Bus:gpiochip32
PM: Adding info for No Bus:gpiochip64
PM: Adding info for No Bus:gpiochip96
PM: Adding info for No Bus:vtcon0
omap_mux_init: Add partition: #1: core, flags: 4
PM: Adding info for platform:omap_uart.0
PM: Adding info for platform:omap_uart.1
PM: Adding info for platform:omap_uart.2
PM: Adding info for platform:omap_uart.3
PM: Adding info for platform:omap_uart.4
PM: Adding info for platform:omap_uart.5
PM: Adding info for No Bus:gpio31
PM: Adding info for platform:omap_i2c.1
PM: Adding info for platform:omap_i2c.3
PM: Adding info for platform:davinci-mcasp.2
PM: Adding info for platform:ti81xx-usbss
SPI Init
PM: Adding info for No Bus:gpio56
PM: Adding info for platform:physmap-flash.0
hw-breakpoint: debug architecture 0x4 unsupported.
hw perfevents: enabled with ARMv7 Cortex-A8 PMU driver, 5 counters available
************************** After perf_pmu_register
PM: Adding info for platform:omap-mailbox
PM: Adding info for platform:omap2_mcspi.1
PM: Adding info for platform:omap2_mcspi.2
PM: Adding info for platform:omap2_mcspi.3
PM: Adding info for platform:omap2_mcspi.4
PM: Adding info for platform:omap2_elm.1
PM: Adding info for platform:arm-pmu.0

To enable the ARM PMU, I believe you have to set:
General Setup -> Kernel Performance Events and Counters -> Kernel performance events and counters[*]

Regards,
Shaun

Hi Shaun,

Sorry for the late reply, but there is some issue with this specific e2e thread. Our E2E team is looking at it.

Let us focus in the linux-kernel/arch/arm/mach-omap2/devices.c file. We have the below function there:

static int __init omap2_init_devices(void)

Can you put printk() after the omap_init_pmu() function like below:

    omap_init_camera();
    omap_init_mbox();
    omap_init_mcspi();
    omap_init_elm();
    omap_init_pmu();  -----> This function invokes the message "PM: Adding info for platform: arm-pmu.0"
    printk("after omap_init_pmu()\n");  ----> let we check if the flow stuck inside the omap_init_pmu() or afterwards

If you get this printk() then the flow might stuck on the next stage, which is the message "PM: Adding info for platform: davinci_mdio.0", this message is invoked from the ti81xx_ethernet_init() function.

printk("before ti81xx_ethernet_init()\n");    
ti81xx_ethernet_init();

I what to check if your flow stuck in the Ethernet init function. Have you made any changes in the Ethernet between the boards?

Regards,
Pavel

Pavel,

Sorry for the late response.

Using the information you provided, we were able to pinpoint the difference that was causing one set of boards to not boot.

Thanks for your help!