Linux/66AK2G12: Linux boot failure: PMMC Probe fails

The bootlog is as follows, after "done... booting the kernel":

** 2132 printk messages dropped ** [    0.580269] Serial: 8250/16550 driver, 10 ports, IRQ sharing disabled
[    0.580465] platform serial8250: set dma_pfn_offset00780000
[    0.587142] console [ttyS0] disabled
[    0.587244] 2530c00.serial: ttyS0 at MMIO 0x2530c00 (irq = 23, base_baud = 12500000) is a TI DA8xx/66AK2x
[    0.617483] console [ttyS0] enabled
[    0.623943] [drm] Initialized
[    0.960181] brd: module loaded
[    1.122652] loop: module loaded
[    1.132007] platform Fixed MDIO bus.0: set dma_pfn_offset00780000
[    1.138792] libphy: Fixed MDIO Bus: probed
[    1.144799] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.151678] ehci-platform: EHCI generic platform driver
[    1.157631] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.164166] ohci-platform: OHCI generic platform driver
[    1.170176] usbcore: registered new interface driver cdc_acm
[    1.176140] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[    1.184846] i2c /dev entries driver
[    1.196350] NET: Registered protocol family 10
[    1.222172] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    1.229672] NET: Registered protocol family 17
[    1.234843] Key type dns_resolver registered
[    1.239565] Registering SWP/SWPB emulation handler
[    2.321372] ti-sci 2921c00.pmmc: Mbox timedout in resp(caller: platform_drv_probe+0x58/0xa4)
[    2.330224] ti-sci 2921c00.pmmc: Mbox send fail -110
[    2.335468] ti-sci 2921c00.pmmc: Unable to communicate with TISCI(-110)
[    2.342554] ti-sci: probe of 2921c00.pmmc failed with error -110
[  471.361647] VFS: Unable to mount root fs via NFS, trying floppy.
[  471.368109] List of all partitions:
[  471.372216] 0100           65536 ram0 [  471.375962]  (driver?)
[  471.378449] 0101           65536 ram1 [  471.382284]  (driver?)
[  471.384772] 0102           65536 ram2 [  471.388513]  (driver?)
[  471.390996] 0103           65536 ram3 [  471.394816]  (driver?)
[  471.397302] 0104           65536 ram4 [  471.401042]  (driver?)
[  471.403597] 0105           65536 ram5 [  471.407339]  (driver?)
[  471.409821] 0106           65536 ram6 [  471.413593]  (driver?)
[  471.416076] 0107           65536 ram7 [  471.419817]  (driver?)
[  471.422330] 0108           65536 ram8 [  471.426071]  (driver?)
[  471.428550] 0109           65536 ram9 [  471.432319]  (driver?)
[  471.434804] 010a           65536 ram10 [  471.438634]  (driver?)
[  471.441114] 010b           65536 ram11 [  471.444973]  (driver?)
[  471.447457] 010c           65536 ram12 [  471.451287]  (driver?)
[  471.453798] 010d           65536 ram13 [  471.457631]  (driver?)
[  471.460112] 010e           65536 ram14 [  471.463971]  (driver?)
[  471.466453] 010f           65536 ram15 [  471.470285]  (driver?)
[  471.472802] No filesystem could mount root, tried: [  471.477726]  nfs
[  471.479655] 
[  471.481227] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
[  471.489900] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)

I am using Linux SDK 04.02.00.09 Downloaded on Feb 15. I am also using CCS 7.3.0.00019 . The U-Boot version is 2017.01.

On an unrelated note, has any progress been made on support for NAND boot on this platform?  I just stumbled upon this post (https://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/p/615259/2269676) and it seems we've been pinned in a corner by this.  No support for NAND boot would mean a very costly board revision for us.

Jeff

I was able to locate a firmware image for the PMMC, ti-sci-firmware-k2g.bin.  Using rproc to load this firmware as indicated in the environment variable "run_pmmc" results in what appears to be successful loading of the pmmc.  After loading the pmmc, however, the ethernet phy goes unreposnsive.  So, depending on the order in which I do things, I get different results.

For each scenario, I run the following:

setenv serverip 192.168.0.14
setenv nfs_root /home/jeff/ti/targetNFS
run init_nfs

This sets the bootargs and acquires an IP Address for the board.  After this, the procedures vary.

Scenario 1: Load all required files into DDR first

tftpboot 0x82000000 zImage
tftpboot 0x83000000 keystone-k2g-evm.dtb
tftpboot 0x0c040000 skern-k2g.bin
tftpboot 0x88000000 ti-sci-firmware-k2g.bin
mon_install 0x0c040000
rproc init; rproc load 0 0x88000000 0x8354; rproc start 0;
bootz 0x82000000 - 0x83000000

Each of these independent steps appear successful. 

The rproc command output looks something like this:

=> rproc init; rproc load 0 0x88000000 0x8354 
uclass_find_device_by_seq: 0 -1
uclass_find_device_by_seq: 0 0
   - -1 -1
   - not found
rproc_pre_probe: 'pmmc@2900000': using fdt
fdtdec_get_bool: remoteproc-internal-memory-mapped
fdtdec_get_addr_size_fixed: reg: addr=02900000
fdtdec_get_int: ti,lpsc_module: 0x1 (1)
ti_powerproc_probe: pmmc@2900000 probed with slave_addr=0x02900000 module=1(0)
uclass_find_device_by_seq: 0 0
   - -1 0
   - found
rproc_load: Loading to 'pmmc@2900000' from address 0x88000000 size of 33620 bytes
ti_powerproc_load: pmmc@2900000: Loading binary from 0x88000000, size 0x00008354 to 0x02900000
ti_powerproc_load: pmmc@2900000: Complete!
Load Remote Processor 0 with data@addr=0x88000000 33620 bytes: Success!
=> rproc start 0
uclass_find_device_by_seq: 0 0
   - -1 0
   - found
_rproc_ops_wrapper: Starting pmmc@2900000...

Running bootz produces the following output:

Uncompressing Linux... done, booting the kernel.
** 2133 printk messages dropped ** [    0.580244] Serial: 8250/16550 driver, 10 ports, IRQ sharing disabled
[    0.580444] platform serial8250: set dma_pfn_offset00780000
[    0.587128] console [ttyS0] disabled
[    0.587230] 2530c00.serial: ttyS0 at MMIO 0x2530c00 (irq = 23, base_baud = 12500000) is a TI DA8xx/66AK2x
[    0.617471] console [ttyS0] enabled
[    0.623900] [drm] Initialized
[    0.960106] brd: module loaded
[    1.122646] loop: module loaded
[    1.131990] platform Fixed MDIO bus.0: set dma_pfn_offset00780000
[    1.138771] libphy: Fixed MDIO Bus: probed
[    1.144776] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.151654] ehci-platform: EHCI generic platform driver
[    1.157588] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.164122] ohci-platform: OHCI generic platform driver
[    1.170122] usbcore: registered new interface driver cdc_acm
[    1.176086] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[    1.184781] i2c /dev entries driver
[    1.196263] NET: Registered protocol family 10
[    1.222178] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    1.229672] NET: Registered protocol family 17
[    1.234843] Key type dns_resolver registered
[    1.239569] Registering SWP/SWPB emulation handler
[    1.254429] ti-sci 2921c00.pmmc: ABI: 2.4 (firmware rev 0x0011 '0.1.1-50-g129ef (Eagle)')
[    1.265518] omap-gpmc 21818000.gpmc: Failed to get GPMC fck
[    1.307114] edma 2700000.edma: TI EDMA DMA engine driver
[    1.313656] edma 2728000.edma: memcpy is disabled
[    1.331254] edma 2728000.edma: TI EDMA DMA engine driver
[    1.338631] ecap 21d1800.ecap: failed to get clock
[    1.344368] keystone-navigator-qmss soc:qmss@4020000: qmgr start queue 0, number of queues 128
[    1.353656] keystone-navigator-qmss soc:qmss@4020000: added qmgr start queue 0, num of queues 128, reg_peek f09fc000, reg_status   (null), reg_config f09f0
[    1.375973] keystone-navigator-dma soc:knav_dmas@0: DMA dma_gbe registered 53 logical channels, flows 32, tx chans: 21, rx chans: 32
[    1.390845] davinci_mdio 4200f00.mdio: failed to get device clock
[    1.398098] netcp-1.0 2620110.netcp: ALE enabled
[    1.403307] netcp-1.0 2620110.netcp: device node lookup for pps timer failed
[    1.410716] netcp-1.0 2620110.netcp: Failed to get cpts refclk
[    1.416946] netcp-1.0 2620110.netcp: Probe of module(netcp-gbe) failed with -517
[    1.424790] netcp-1.0 2620110.netcp: module(netcp-xgbe) not used for device
[    1.432169] netcp-1.0 2620110.netcp: module(netcp-pa) not used for device
[    1.439290] netcp-1.0 2620110.netcp: module(netcp-pa2) not used for device
[    1.446533] netcp-1.0 2620110.netcp: module(netcp-sa) not used for device
[    1.454534] omap-gpmc 21818000.gpmc: Failed to get GPMC fck
[    1.462289] ecap 21d1800.ecap: failed to get clock
[    1.468007] davinci_mdio 4200f00.mdio: failed to get device clock
[  470.721698] VFS: Unable to mount root fs via NFS, trying floppy.
[  470.728156] List of all partitions:
[  470.732041] 0100           65536 ram0 [  470.735786]  (driver?)
[  470.738273] 0101           65536 ram1 [  470.742257]  (driver?)
[  470.744744] 0102           65536 ram2 [  470.748485]  (driver?)
[  470.750968] 0103           65536 ram3 [  470.754783]  (driver?)
[  470.757267] 0104           65536 ram4 [  470.761008]  (driver?)
[  470.763523] 0105           65536 ram5 [  470.767264]  (driver?)
[  470.769746] 0106           65536 ram6 [  470.773516]  (driver?)
[  470.776001] 0107           65536 ram7 [  470.779741]  (driver?)
[  470.782251] 0108           65536 ram8 [  470.785993]  (driver?)
[  470.788474] 0109           65536 ram9 [  470.792243]  (driver?)
[  470.794727] 010a           65536 ram10 [  470.798558]  (driver?)
[  470.801039] 010b           65536 ram11 [  470.804898]  (driver?)
[  470.807381] 010c           65536 ram12 [  470.811212]  (driver?)
[  470.813724] 010d           65536 ram13 [  470.817555]  (driver?)
[  470.820036] 010e           65536 ram14 [  470.823896]  (driver?)
[  470.826379] 010f           65536 ram15 [  470.830209]  (driver?)
[  470.832727] No filesystem could mount root, tried: [  470.837651]  nfs
[  470.839580] 
[  470.841149] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
[  470.849821] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)

The part of this that sticks out to me is the clock errors.  There are a number of "failed to get clock" errors in the above output.  This tells me that the PMMC firmware is changing the clock configurations set by U-Boot.  I have not found any indication of source files or modifiable code for the PMMC.  It seems to me that this is code which is not supposed to be modified and that settings elsewhere in the code (device tree, board files, etc) communicate with the PMMC to make sure things are turned on.

Scenario 2: Load and install as I go

tftpboot 0x0c040000 skern-k2g.bin
mon_install 0x0c040000
tftpboot 0x88000000 ti-sci-firmware-k2g.bin
rproc init; rproc load 0 0x88000000 0x8354; rproc start 0;
tftpboot 0x82000000 zImage
 -- Fails due to PHY Autoneg timeout

All attempts to access the network AFTER starting the PMMC result in autonegotiation timeouts.  This is no doubt  linked to the message produced in the above scenario indicating that the MDIO/RGMII interfaces do not have any clock signals.

I have also attempted to use UBIFS for the filesystem instead of NFS.  This simply results in a much faster kernel panic (~1 second instead of ~417).  It's possible that I am not configuring the UBIFS properly, or that the NAND is nonfunctional for boot, as seemed to be indicated in the post I linked to in my previous post.

With regards to the PMMC, I am having some trouble locating much documentation on how this is supposed to work.  I did find a wiki article (http://processors.wiki.ti.com/index.php/TISCI#Set_Clock_State_Message) that has some information, and there is some device tree docs about it.  This is where I will direct my search for now.