NFS mount failure during boot

Your boot log ends before any NFS error message. Looks more or less normal except for the MMC errors. Next usual thing is to try adding "rootdelay=5" to your boot arguments. Sometimes NFS comes up faster than the network.

Here is the boot sequence with "rootdelay=5" and the NFS mount failures (note I removed all the mmc0 errors for clarity):

=============================================

Linux version 2.6.32.17-davinci1 (sdk@nemo) (gcc version 4.3.3 (Sourcery G++ Lite 2009q1-203) ) #1 P
REEMPT Thu Mar 24 16:31:36 CDT 2011
CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00053177
CPU: VIVT data cache, VIVT instruction cache
Machine: DaVinci DM36x EVM
Memory policy: ECC disabled, Data cache writeback
DaVinci dm36x_rev1.2 variant 0x8
Built 1 zonelists in Zone order, mobility grouping off.  Total pages: 12192
Kernel command line: console=ttyS0,115200n8 rw m365_imp.oper_mode=0 video=davincifb:vid0=OFF:vid1=OF
F:osd0=480x272x16,4050K mem=48MB davinci_capture.device_type=4 vpfe_capture.cont_bufsize=12582912 da
vinci_enc_mngr.ch0_output=LCD davinci_enc_mngr.ch0_mode=480x272 rootdelay=5 root=/dev/nfs nfsroot=15
9.62.14.108:/home/baldwinc/targetfs ip=159.62.14.129:159.62.14.108:159.62.14.1:255.255.255.0
PID hash table entries: 256 (order: -2, 1024 bytes)
Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
Memory: 48MB = 48MB total
Memory: 44100KB available (4024K code, 393K data, 144K init, 0K highmem)
SLUB: Genslabs=11, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Hierarchical RCU implementation.
NR_IRQS:245
Console: colour dummy device 80x30
Calibrating delay loop... 215.44 BogoMIPS (lpj=1077248)
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
DaVinci: 8 gpio irqs
NET: Registered protocol family 16
davinci_serial_init:97: failed to get UART2 clock
EVM: HD imager video input
bio: create slab <bio-0> at 0
DM365 IPIPE initialized in Single Shot mode
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
pca9543a_probe
vpss vpss: dm365_vpss vpss probed
vpss vpss: dm365_vpss vpss probe success
dm365_afew_hw_init
ch0 default output "LCD", mode "480x272"
VPBE Encoder Initialized
cfg80211: Using static regulatory domain info
cfg80211: Regulatory domain: US
        (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
        (2402000 KHz - 2472000 KHz @ 40000 KHz), (600 mBi, 2700 mBm)
        (5170000 KHz - 5190000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
        (5190000 KHz - 5210000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
        (5210000 KHz - 5230000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
        (5230000 KHz - 5330000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
        (5735000 KHz - 5835000 KHz @ 40000 KHz), (600 mBi, 3000 mBm)
cfg80211: Calling CRDA for country: US
LogicPD encoder initialized
Switching to clocksource timer0_1
musb_hdrc: version 6.0, cppi-dma, host, debug=0
musb_hdrc: USB Host mode controller at fec64000 using DMA, IRQ 12
musb_hdrc musb_hdrc: MUSB HDRC host driver
musb_hdrc musb_hdrc: new USB bus registered, assigned bus number 1
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 2048 (order: 2, 16384 bytes)
TCP bind hash table entries: 2048 (order: 1, 8192 bytes)
TCP: Hash tables configured (established 2048 bind 2048)
TCP reno registered
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
msgmni has been set to 86
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler anticipatory registered (default)
Console: switching to colour frame buffer device 60x34
davincifb davincifb.0: dm_osd0_fb: 480x272x16@0,0 with framebuffer size 4050KB
davincifb davincifb.0: dm_vid0_fb: 0x0x16@0,0 with framebuffer size 382KB
davincifb davincifb.0: dm_osd1_fb: 480x272x4@0,0 with framebuffer size 255KB
davincifb davincifb.0: dm_vid1_fb: 0x0x16@0,0 with framebuffer size 382KB
DM365 IPIPEIF probed
imp serializer initialized
davinci_previewer initialized
davinci_resizer initialized
Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
serial8250.0: ttyS0 at MMIO 0x1c20000 (irq = 40) is a 16550A
console [ttyS0] enabled
brd: module loaded
at24 1-0050: 32768 byte 24c256 EEPROM (writable)
spi_davinci spi_davinci.0: DaVinci SPI driver in EDMA mode
Using RX channel = 17 , TX channel = 16 and event queue = 3
spi_davinci spi_davinci.0: Controller at 0xfec66000
console [netcon0] enabled
netconsole: network logging started
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver usbtest
mice: PS/2 mouse device common for all mice
tsc2004: probe of 1-0049 failed with error -121
input: DM365 EVM Controls as /devices/platform/i2c_davinci.1/i2c-1/1-0025/input/input0
i2c /dev entries driver
Linux video capture interface: v2.00
ths7303 1-002c: chip found @ 0x58 (DaVinci I2C adapter)
vpfe_init
vpfe-capture: vpss clock vpss_master enabled
vpfe-capture vpfe-capture: v4l2 device registered
vpfe-capture vpfe-capture: video device registered
EVM: switch to tvp5146 SD video input
tvp514x 1-005d: tvp514x 1-005d decoder driver registered !!
vpfe-capture vpfe-capture: v4l2 sub device tvp5146 registered
EVM: switch to tvp7002 HD video input
tvp7002 1-005c: tvp7002 1-005c decoder driver registered !!
vpfe-capture vpfe-capture: v4l2 sub device tvp7002 registered
EVM: switch to tvp7002 HD video input
ths7353 1-002e: chip found @ 0x5c (DaVinci I2C adapter)
ths7353 1-002e: No platform data!!
ths7353 1-002e: ths7353 write failed
ths7353: probe of 1-002e failed with error -121
vpfe-capture vpfe-capture: v4l2 sub device ths7353 register fails
vpfe_register_ccdc_device: DM365 ISIF
DM365 ISIF is registered with vpfe.
af major#: 252, minor# 0
AF Driver initialized
aew major#: 251, minor# 0
AEW Driver initialized
Trying to register davinci display video device.
layer=c10cf400,layer->video_dev=c10cf570
Trying to register davinci display video device.
layer=c10cf800,layer->video_dev=c10cf970
davinci_init:DaVinci V4L2 Display Driver V1.0 loaded
watchdog watchdog: heartbeat 60 sec
davinci_mmc davinci_mmc.0: Using DMA, 4-bit mode
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
Advanced Linux Sound Architecture Driver Version 1.0.21.
No device for DAI tlv320aic3x
No device for DAI davinci-i2s
asoc: tlv320aic3x <-> davinci-i2s mapping ok
mmc0: host doesn't support card's voltages
mmc0: error -22 whilst initialising SDIO card
ALSA device list:
  #0: DaVinci EVM (tlv320aic3x)
TCP cubic registered
NET: Registered protocol family 17
lib80211: common routines for IEEE802.11 drivers
Clocks: disable unused mmcsd1
Clocks: disable unused spi1
Clocks: disable unused spi2
Clocks: disable unused spi3
Clocks: disable unused spi4
Clocks: disable unused pwm0
Clocks: disable unused pwm1
Clocks: disable unused pwm2
Clocks: disable unused pwm3
Clocks: disable unused timer1
Clocks: disable unused timer3
Clocks: disable unused emac
Clocks: disable unused voice_codec
Clocks: disable unused rto
Clocks: disable unused mjcp
davinci_emac_probe: using random MAC addr: ee:c7:c3:57:95:41
emac-mii: probed
eth0: attached PHY driver [Generic PHY] (mii_bus:phy_addr=1:01, id=221430)
mmc0: host doesn't support card's voltages
mmc0: error -22 whilst initialising SDIO card
IP-Config: Complete:
     device=eth0, addr=159.62.14.129, mask=255.255.255.0, gw=159.62.14.1,
     host=159.62.14.129, domain=, nis-domain=(none),
     bootserver=159.62.14.108, rootserver=159.62.14.108, rootpath=
Waiting 5sec before mounting root device...
Looking up port of RPC 100003/2 on 159.62.14.108
rpcbind: server 159.62.14.108 not responding, timed out
Root-NFS: Unable to get nfsd port number from server, using default
Looking up port of RPC 100005/1 on 159.62.14.108
rpcbind: server 159.62.14.108 not responding, timed out
Root-NFS: Unable to get mountd port number from server, using default

Those kind of errors usually mean the NFS server is not quite setup correctly. Since it mounts on another machine, the NFS server must be working in some form. Perhaps as a non-root fs. Maybe your target IP address is not listed in the /etc/exports file. How did you create 159.62.14.108:/home/baldwinc/targetfs? That directory will need to be created by root and have all those special device files and symbolic links created properly.

I have also noticed that the DHCP service does not appear to working in the Linux Kernel.  The u-boot is able to use DHCP prior to tftpboot, but Linux fails.  Is this connected?  I read that the default Linux Kernel is not always configured to have DHCP working?  Is this the case?

I've never seen a default kernel config without a DHCP client. Possible I guess where there is no network HW. I assume you tried "ip=dhcp" in the boot argument. In any case, NFS should work with static or dynamic IP addresses. Another thing you should double check is that the firewall is disabled on your NFS server. Sometimes settings don't survive a power cycle. Double check your NFS server after a power cycle.

Hmm, here are the firewall settings:

$ sudo iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I'd recommend turning the firewall off for now. I've tried configuring the thing in the past but eventually gave up. This works on Fedora
sudo service iptables stop

Check you exports file. Something like this:
/home/baldwinc/targetfs *(rw,sync,no_root_squash)

It's important that the /home/baldwinc/targetfs is created by root. How did you create that directory?

Thank you for the quick response.  Per your directions:

I "disabled" the firewall in Ubuntu by installing the graphical firewall control.  I have verified that the targetfs all belongs to root using the command:

sudo chown -R root /home/baldwinc/targetfs

Still no fix.  Is there a TI direct support line?  I'm running out of ideas.

=======

I did some reading and noticed that the MAC address is currently be assigned randomly.  I understand that some routers/switches will block traffic with an invalid MAC address.  I went into the linux kernel and overrode the routine to explicitly assign a MAC address.  I used my computer MAC minus 1:

00:02:e3:0f:83:0e

I think this is a valid MAC but and not sure.  In any case it did not appear to fix anything.

The random MAC address should be fine. Odds are good that the MAC is unique in your subnet.

Check to see if the NFS ports are up. Use "rpcinfo -p". On my machine, I get these lines:

100003 2 tcp 2049 nfs
100003 2 udp 2049 nfs
100005 1 tcp 56588 mountd
100005 1 udp 59107 mountd

The above matches the boot log messages:

Looking up port of RPC 100003/2 on 159.62.14.108
Looking up port of RPC 100005/1 on 159.62.14.108

3.01.01.39/drivers$ rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  37977  status
    100024    1   tcp  44335  status
    100021    1   udp  60840  nlockmgr
    100021    3   udp  60840  nlockmgr
    100021    4   udp  60840  nlockmgr
    100021    1   tcp  41387  nlockmgr
    100021    3   tcp  41387  nlockmgr
    100021    4   tcp  41387  nlockmgr
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    1   udp  36617  mountd
    100005    1   tcp  34250  mountd
    100005    2   udp  36617  mountd
    100005    2   tcp  34250  mountd
    100005    3   udp  36617  mountd
    100005    3   tcp  34250  mountd
baldwinc@baldwinc-desktop:~/ti-dvsdk_dm368-evm_4_02_00_06/psp/linux-2.6.32.17-psp03.01.01.39/drivers$

=========================

Is there a way to load Linux and the filesystem into the NAND using just UBOOT?  The manual for DVSDK 4.02 requires you to start the system in NFS then write the filesystem to the NAND using Linux.  Since I cant get NFS working I cant do this. 

How do I put the filesystem in NAND from UBOOT, and more importantly what are the *bootargs* to make it boot properly.  If I can figure this out, then I could boot from flash, and trouble shoot the NFS issues in isolation.

I don't know what else is blocking the NFS server. My setup has more versions (3,4) of the portmapper. You might need to install a newer version of the NFS server.

Try the TI wiki for info on various methods of booting.
http://processors.wiki.ti.com
http://processors.wiki.ti.com/index.php/UG:_DaVinci_PSP_Installation_on_DM36x_EVM##Booting_with_RAMDISK_as_a_root_filesystem

You should be able to use a tftp'ed RAMDISK file system for testing. Something like this.

tftp 0x82000000 your_file_sys.ext2.gz
setenv bootarg console=ttyS0,115200n8 m365_imp.oper_mode=0 video=davincifb:vid0=OFF:vid1=OFF:osd0=480x272x16,4050K mem=48MB davinci_capture.device_type=4 vpfe_capture.cont_bufsize=12582912 davinci_enc_mngr.ch0_output=LCD davinci_enc_mngr.ch0_mode=480x272 root=/dev/ram0 rw initrd=0x82000000,4M ip=159.62.14.129:159.62.14.108:159.62.14.1:255.255.255.0
bootm

Above assumes your gz file is 4M and 8x82000000 is free memory.

Did you use tar to create the NFS directory?

I'm currently struggling with how to find/create "my_file_system.ext2.gz" file.  It seems like there should be pre-build filesystem for NAND booting, but I do not see anything in the DVSDK 4.02. 

I have been looking at all the articles on using a NAND filesystem, but all of them start with "take your filesys.ext2.gz file and....".  They all seem to skip the first step to indicate how this fileis made or found.  Any clues?

You could try one of TI's Arago file systems here:

http://arago-project.org/files/releases/davinci-psp_03.01.01.39/images/

They won't be exactly customized to your boad but they should let you boot.

Yes, that was the ticket, I was able to build a cramfs and get past mounting the filesystem.  Unfortunately it looks like the dvdsk 4.02 expects a non-read-only filesystem.  Once the cramfs system boots you get a series of "readonly" error and the boot fails.  It looks like I will have to make the NFS work first. 

Here is the boot output:

lib80211: common routines for IEEE802.11 drivers
Clocks: disable unused mmcsd1
Clocks: disable unused spi1
Clocks: disable unused spi2
Clocks: disable unused spi3
Clocks: disable unused spi4
Clocks: disable unused pwm0
Clocks: disable unused pwm1
Clocks: disable unused pwm2
Clocks: disable unused pwm3
Clocks: disable unused timer1
Clocks: disable unused timer3
Clocks: disable unused emac
Clocks: disable unused voice_codec
Clocks: disable unused rto
Clocks: disable unused mjcp
davinci_emac_probe: using random MAC addr: 00:02:e3:0f:83:0e
emac-mii: probed
RAMDISK: cramfs filesystem found at block 0
RAMDISK: Loading 12288KiB [1 disk] into ram disk... done.
VFS: Mounted root (cramfs filesystem) readonly on device 1:0.
Freeing init memory: 144K
INIT: version 2.86 booting
Please wait: booting...
Starting udev
/etc/rcS.d/S03udev: line 50: can't create /tmp/uname: Read-only file system
touch: /tmp/uname: Read-only file system
Remounting root file system...
Caching udev devnodes
Populating dev cachetar: can't open '/etc/dev.tar': Read-only file system
mv: overwrite '/etc/udev/saved.uname'? n
mv: overwrite '/etc/udev/saved.cmdline'? n
mv: overwrite '/etc/udev/saved.devices'? n
mv: overwrite '/etc/udev/saved.atags'? n

n/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-
only file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
/etc/init.d/rc: /etc/rcS.d/S37populate-volatile.sh: line 1: can't create /etc/volatile.cache: Read-o
nly file system
ALSA: Restoring mixer settings...
Configuring network interfaces... /usr/sbin/alsactl: load_state:1610: No soundcards found...
eth0: attached PHY driver [Generic PHY] (mii_bus:phy_addr=1:01, id=221430)
udhcpc (v1.13.2) started
Sending discover...
Sending discover...
Sending discover...
No lease, forking to background
done.
Setting up IP spoofing protection: rp_filter.
rm: cannot remove '/tmp': Read-only file system
ln: /tmp/tmp: Read-only file system
hwclock: can't open '/dev/misc/rtc': No such file or directory
Thu Mar 24 19:02:00 UTC 2011
hwclock: can't open '/dev/misc/rtc': No such file or directory
Collected errors:
 * opkg_conf_load: Could not create lock file /usr/lib/opkg/lock: Read-only file system.
rm: cannot remove '/etc/rcS.d/S98configure': Read-only file system
/etc/init.d/rc: /etc/rcS.d/S99finish.sh: line 5: can't create /etc/.configured: Read-only file syste
m
NET: Registered protocol family 10
ADDRCONF(NETDEV_UP): eth0: link is not ready

Not quite understanding. You should be able to make a RAMDISK be R/W. Just that you lose any changes on power cycle. Maybe try the ext2.gz version.

Interesting result, I used the pre-build kernel for DVSK 2.10 and NFS works fine.  However using the pre-build kernel for DVSDK 4.02 causes both DHCP and NFS to fail.  I suspect a build issue, has anyone else seen this?

======================================

I agree, it seems like the cramfs filesystem should not be read-only once it is in memory.  I modified the cramfs and gave read-write-execute permission to all files using "chmod -R 777 *".  This did not seem to help.

I'll try using the tar.gz file but, I'm not sure how.  I think the filesystem need to be in a specific format to boot up this way...

I don' think initrd supports tar. The ext2 ones should work:

http://arago-project.org/files/releases/davinci-psp_03.01.01.39/images/arago/arago-base-image-arago.ext2.gz

I suspect the older kernel uses an older version of NFS, probably 2. The later kernel might use version 4. Configuring NFS is beyond me. Your rpcinfo list is missing portmapper 3 and 4 aka rpcbind. I am not sure if NFS requires it. Hopefully some Ubuntu NFS experts will speak up. No idea about DHCP.

Definitely building the kernel yourself would ensure DHCP and NFS are in the kernel. All the network problems makes me wonder if the network driver in the later kernel is working properly. If you can get it booted up via RAMDISK, you can ping out to see if the network even works.

Thanks for the quick response.  Hmm, I am looking into the NFS2 vs NFS4 issue, that seems like a good possibility. 

Another issue I have noticed is the driver being used for the eth0 my be wrong, my boot sequence identifies the driver as:

eth0: attached PHY driver [Generic PHY] (mii_bus:phy_addr=1:01, id=221430)

However, in samples from other people I see this:

eth0: attached PHY driver [RTL8201 Gigabit Ethernet] (mii_bus:phy_addr=1:1f, id=8201)

This makes me suspect Linux is loading the incorrect ethernet driver.  Any thoughts?

=====

More information:

When I assign the Linux kernel an static IP address during boot I cannot "ping" the address from another machine.  This make me suspect the network card driver.

When I have the board boot using DHCP I do not see any DHCP requests on Wireshark.

When I moved from 2.6.32 to 2.6.37, my PHY also went from something specific to generic. Network still works for me though. Obviously not for you. In my case, the OMAP-L137 EVM, U-Boot was doing the important EMAC initialization via device specific I2C commands. Perhaps you have to update your U-Boot to match the updated kernel. Shouldn't have to though.

As far as I know, embedded Linux is not plug'n'play. Maybe on the desktop but rarely on embedded systems. The specific EMAC and PHY drivers have to be enabled. I would expect a pre-built kernel to be pre-configured for your board's HW. Time to build you own kernel. Hopefully your board has a clearly defined default kernel config. TI has a bad habit of poorly naming configs or using one config for a range of boards.

Is Linux fully booted when you ping it? I don't think the ping daemon is alive until the system is booted.