Kernel Panic Using PCIe FireWire on TMS320C6A8168 EVM

Hello,

Please provide following details:

1) Output of "lspci -xvvv" (you may need to build/install this if the filesystem doesn't have this utility).

2) cat /proc/iomem

3) cat /proc/ioports

Also, I haven't looked much at the firewire driver but is the crash observed on the *first* access to the register or a few accesses work before the crash?

Thanks.

   Hemant

Hi Hermant,

Thank you for the reply.

I will do these things tomorrow when I have access to the board and post them.

As far as question about the accesses, I can't tell you off hand if this is the first access to this register or not, but it is definitely not the first access to registers in the OHCI chip as a substantial number of FW transactions have occurred by this point (those required to id the FW device, mount the disk, etc.). Since the register being accessed is the pending Interrupts register, I rather think that this is *not* the first access to this register, but I need to check the code more closely to be sure.

Best regards,

B.J.

lspci -xvvv --- you may need to build/install this if the filesystem doesn't have this utility....

What is the easiest way to get the pciutils onto the EVM?

opkg install pciutils

does not work.

I was able to build pciutils with arago-oe, but I have no idea how to get the ipkg stuff to expand on the EVM, or how to get opkg to look at my arago repository. I also don't really see how to manually do the cross compile.

Any suggestions would be appreciated!

B.J.

"opkg install pciutils" should work provided conf files inside /etc/opkg directory are set correctly. Make sure you have at least 'armv7a' entry inside /etc/opkg/arch.conf and respective config files for feeds are present (e.g., /etc/opkg/arago-armv7a-feed.conf).

If you are still not able to get ispci, then you can provide dump of entries from /sys/bus/pci. I particularly wanted dump of resources and size.

   Hemant

In addition, please also provide the dump of 0x51001728 register  *before* and *after* the crash.

   Hemant

HemantPedanekar said:

"opkg install pciutils" should work provided conf files inside /etc/opkg directory are set correctly. Make sure you have at least 'armv7a' entry inside /etc/opkg/arch.conf and respective config files for feeds are present (e.g., /etc/opkg/arago-armv7a-feed.conf).

This does not appear to work. /etc/opkg/arago-armv7a-feed.conf does not exist on the rootfs that comes with the EVM. There is a sample file for that, so I copied the sample to the .conf file, and then did opkg update. But it still does not find the package for pciutils. I finally simply built a rootfs with arago, and then copied the pciutils*.ipkg files to the root image and did the opkg install from the ipkg files. 

So, I have lspci installed, but it seems like something is still not set up correctly for using the opkg tool.

When I do lspci, I get something that does not look right (this is with a NEC USB3 card [which I used to try to eliminate the PCIe <-> PCI bridge that is on the FireWire card], but the Firewire card looks the same -- with an additional entry for the bridge)

# lspci -xvvv

00:00.0 PCI bridge: Texas Instruments Device 8888 (rev ff) (prog-if ff)

        !!! Unknown header type 7f

00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

01:00.0 USB Controller: NEC Corporation Device 0194 (rev ff) (prog-if ff)

        !!! Unknown header type 7f

00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

as far as dumping the "resource" file from /sys, here is the result:

root@c6a816x-evm:~# more /sys/devices/pci0000\:00/0000\:00\:00.0/resource

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000020000000 0x00000000200fffff 0x0000000000000200

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

root@c6a816x-evm:~# more /sys/devices/pci0000\:00/0000\:00\:00.0/0000\:01\:00.0/resource

0x0000000020000000 0x0000000020001fff 0x0000000000140204

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

0x0000000000000000 0x0000000000000000 0x0000000000000000

Hi Hemant,

HemantPedanekar said:

In addition, please also provide the dump of 0x51001728 register  *before* and *after* the crash.

I'm sorry -- I am new to the TI world and still trying to get my bearings. Can you explain exactly how to do this?

Thanks!

B.J. Buchalter

Here is the other info you requested:

root@c6a816x-evm:~# cat /proc/iomem

00000000-00000000 : omap2-nand

20000000-2fffffff : pcie-nonprefetch

47401000-474017ff : musb_hdrc.0

47401800-47401fff : musb_hdrc.1

48020000-4802001f : serial

48022000-4802201f : serial

48024000-4802401f : serial

48028000-4802803f : i2c_omap.1

  48028000-4802803f : i2c_omap

4802a000-4802a03f : i2c_omap.2

  4802a000-4802a03f : i2c_omap

48030100-480301ff : omap2_mcspi.1

  48030100-480301ff : omap2_mcspi.1

48050000-48052fff : mcasp

  48050000-48052fff : davinci-mcasp

48060100-480700ff : mmci-omap-hs.0

  48060100-480700ff : mmci-omap-hs

49000000-49007fff : edma_cc0

  49000000-49007fff : edma

49800000-498003ff : edma_tc0

49900000-499003ff : edma_tc1

49a00000-49a003ff : edma_tc2

49b00000-49b003ff : edma_tc3

4a100000-4a103fff : davinci_emac.0

  4a100000-4a103fff : eth0

4a120000-4a123fff : davinci_emac.1

  4a120000-4a123fff : 

4a140000-4a150fff : ahci

51000000-51003fff : pcie-regs

80000000-ffffffff : pcie-inbound0

  80000000-8a5fffff : System RAM

    80031000-803e4fff : Kernel text

    803e6000-80447f5b : Kernel data

root@c6a816x-evm:~# cat /proc/ioports

40000000-402fffff : PCI I/O

  40000000-402fffff : pcie-io

So, I managed to build a 2.6.37 kernel and get it booted; it does not fully work because the modules don't match -- still trying to work out how to build a complete kernel and root-filesystem for the newer kernel. 

That being said, PCIe seems to work much better with the newer kernel; if I boot with the USB3 card installed, lspci -vv actually shows information about the configuration space. If I boot with a FireWire card installed, the kernel panics on boot because it cannot mount the root filesystem. But I suspect that has something to do with not properly building the kernel. 

So, can someone provide the steps required to build a more modern kernel/root file system with arago? The one that is built from the arago checkout seems to be consistent with the current ezsdk which is 2.6.34.

Best regards,

B.J. Buchalter

Hi,

B.J. Buchalter said:

[which I used to try to eliminate the PCIe <-> PCI bridge that is on the FireWire card], but the Firewire card looks the same -- with an additional entry for the bridge)

Does the lspci dump for firewire card at least show correct values (instead of 0xff 's)?

B.J. Buchalter said:

It seems like something must be screwed up with the PCI RC as all the reads seem to be coming back as 0xffffffff.

Any clue, or anything else you want me to check?

Looks like for some reason, the PCIe link went down after successful enumeration. That's why the "/sys/devices/pci0000\:00/0000\:01\:00.0/resource" dump shows correct values but "lspci -x" doesn't.

You can cross compile and use devmem2 like utility to dump the register I mentioned earlier (0x51001728). The last 5 bits should be 0x11 to indicate the link is established.

But if the firewire card at least shows correctly in lspci, then I would suggest to use it to debug rather than the USB card.

   Hemant

Hi,

B.J. Buchalter said:

Here is the other info you requested:

root@c6a816x-evm:~# cat /proc/iomem

00000000-00000000 : omap2-nand

20000000-2fffffff : pcie-nonprefetch

I guess this dump is before the driver is initialized? I was expecting an entry for driver resource (e.g., BAR0) here.

   Hemant

Hi,

B.J. Buchalter said:

So, I managed to build a 2.6.37 kernel and get it booted; it does not fully work because the modules don't match -- still trying to work out how to build a complete kernel and root-filesystem for the newer kernel. 

Yes, the SDK with latest kernel is not yet out so you cannot use pre-built modules.

B.J. Buchalter said:

That being said, PCIe seems to work much better with the newer kernel; if I boot with the USB3 card installed, lspci -vv actually shows information about the configuration space.

Did you use "lspci -vv" or "lspci -xvv"?

B.J. Buchalter said:

If I boot with a FireWire card installed, the kernel panics on boot because it cannot mount the root filesystem. But I suspect that has something to do with not properly building the kernel. 

 

If you are able to boot the kernel with PCIe-USB card, then it should boot fine with PCIe-Firewire card too as long as you use the same rootfs (e.g., NFS). Are you using the firewire driver as module  or built into kernel? It may be better if you use it as module (if possible).

   Hemant

Does the lspci dump for firewire card at least show correct values (instead of 0xff 's)?

No. It is the same as the USB card.

Looks like for some reason, the PCIe link went down after successful enumeration.

Thats sort of what I was thinking.

 You can cross compile and use devmem2 like utility to dump the register I mentioned earlier (0x51001728). The last 5 bits should be 0x11 to indicate the link is established.

I did that; when I ran devmem2 against that physical address, I got the same message that I reported when I started this thread (about a precise exception) and the program segfaulted.

It seems like, with the 2.6.34 kernel, that the PCIe RC fails after some number of accesses; in each case I checked it had failed by the time I could log in on the console to run lspci or any other tests.

As I said in a later post, I have tried to build 2.6.37 from the linux-omap3 git repository, and it seems to get me a bit further, but I am having problems getting the system up by just replacing the uImage (for example none of the driver modules are available, and the inittab references the wrong tty); I am really unclear on the process of moving to a newer kernel and building a working boot image and root filesystem.

With the "sort-of-working" 2.6.37 kernel, if I use the USB card, I can get reasonable output from lspci. Unfortunately, if I boot with the FIreWire card installed, the kernel panics when it tries to mount the root partition from the sdcard; I am unclear on why having the FireWire card plugged in at boot would cause the sdcard to not be able to mount. 

With the 2.6.37 kernel, and no FireWire card, the relevant portion of the boot sequence is:

usb 1-1: new low speed USB device using musb-hdrc and address 2

mmc0: new SDHC card at address e699

mmcblk0: mmc0:e699 SD08G 7.40 GiB 

 mmcblk0: p1 p2 p3

ata1: SATA link down (SStatus 0 SControl 300)

ata2: SATA link down (SStatus 0 SControl 300)

EXT3-fs: barriers not enabled

usb 1-1: New USB device found, idVendor=13ee, idProduct=0003

usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0

usb 1-1: Product: Optical Mouse

usb 1-1: Manufacturer: MosArt

input: MosArt Optical Mouse as /devices/platform/musb-ti81xx/musb-hdrc.0/usb1/1-1/1-1:1.0/input/input0

generic-usb 0003:13EE:0003.0001: input: USB HID v1.10 Mouse [MosArt Optical Mouse] on usb-musb-hdrc.0-1/input0

kjournald starting.  Commit interval 5 seconds

EXT3-fs (mmcblk0p2): warning: maximal mount count reached, running e2fsck is recommended

EXT3-fs (mmcblk0p2): using internal journal

EXT3-fs (mmcblk0p2): recovery complete

EXT3-fs (mmcblk0p2): mounted filesystem with writeback data mode

VFS: Mounted root (ext3 filesystem) on device 179:2.

With the 2.6.37 kernel and the FireWire card installed, the same portion looks like:

usb 1-1: new low speed USB device using musb-hdrc and address 2

ata1: SATA link down (SStatus 0 SControl 300)

ata2: SATA link down (SStatus 0 SControl 300)

Root-NFS: no NFS server address

VFS: Unable to mount root fs via NFS, trying floppy.

VFS: Cannot open root device "mmcblk0p2" or unknown-block(2,0)

Please append a correct "root=" boot option; here are the available partitions:

1f00              64 mtdblock0  (driver?)

1f02              64 mtdblock2  (driver?)

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)

Backtrace: 

[<c0043b44>] (dump_backtrace+0x0/0x110) from [<c0360558>] (dump_stack+0x18/0x1c)

 r7:c6c13000 r6:00000000 r5:c002b87c r4:c04cbd50

[<c0360540>] (dump_stack+0x0/0x1c) from [<c03605bc>] (panic+0x60/0x17c)

[<c036055c>] (panic+0x0/0x17c) from [<c0009254>] (mount_block_root+0x1e0/0x220)

 r3:00000000 r2:00000000 r1:c6c23f58 r0:c0414b4c

[<c0009074>] (mount_block_root+0x0/0x220) from [<c0009340>] (mount_root+0xac/0xcc)

[<c0009294>] (mount_root+0x0/0xcc) from [<c00094d0>] (prepare_namespace+0x170/0x1d4)

 r4:c04cb6e4

[<c0009360>] (prepare_namespace+0x0/0x1d4) from [<c0008784>] (kernel_init+0x114/0x154)

 r5:c0008670 r4:c04cb680

[<c0008670>] (kernel_init+0x0/0x154) from [<c0063b00>] (do_exit+0x0/0x5e4)

 r5:c0008670 r4:00000000

Note that it does not seem to load the driver for the mmc.

Any ideas, or is there anywhere that really describes how to rebuild the complete ezsdk kernel and filesystem, hopefully with instructions that would support moving to the newer kernel?

Hermant, have you tried using PCIe on the kernel that ships with the ezSDK? Because it doesn't seem to work at all here. If you are working with a newer kernel, can you point to the proper bits to be using?

Thanks!

HemantPedanekar said:

If you are able to boot the kernel with PCIe-USB card, then it should boot fine with PCIe-Firewire card too as long as you use the same rootfs (e.g., NFS). Are you using the firewire driver as module  or built into kernel? It may be better if you use it as module (if possible).

Look at my previous message for details, but, at least for now, I am trying to use the rootfs that comes on the SD card. The kernel is built with the FireWire drivers built in. I did not build in the drivers for the USB3 card, so that could be the difference, but the problem is that when the FireWire card is intstalled, the linux kernel driver for the SD card is not loading and the root file system does not mount at all, so it is not even to the point that a module could load.

Perhaps doing it via NFS would alleviate the problem or change the order in such a way that the MMC driver loads, but I still don't really have a good recipe to see how to actually rebuild the root file system for NFS mount; in particular, I am not sure how to get the GPU drivers installed in to the image.

Hello,

I think we are getting into too many combination here! Can you please first try the following and provide your observations? Also, let's stick to 2.6.34 kernel for now (till we see that it is not at all working).

1) Use the 2.6.34 kernel but *do not* build firewire support into it. Then boot this kernel with firewire card plugged into the PCIe slot.

2) With the above, please provide the "lspci -xvvv" dump.

3) If you see 0xff 's, then enable the PCI debugging in kernel (make menuconfig --> Bus support --> PCI Debugging) and append "debug" to the bootargs (you may also want to turn on low level debugging). Please provide the log you will get (specifically PCI enumeration part).

4) If you see lspci in (2) fine without 0xff 's, then build the firewire drivers as modules and load them. Now check the "lspci -xvvv" again.

   Hemant

Ok -- I'll do it tomorrow in the office. 

I'll start with the evm defconf and see if it works, and if it does, I'll go step by step from there.

Best regards,

B.J. Buchalter

Hi Hermant,

Ok. I restored the boot image to the original uImage that came with the EVM; so this is the prebuilt 2.6.34 kernel that came in the ti-ezsdk_c6a816x-evm_5_00_00_56. 

Here is the boot log from that boot, including the output of lspci -xvvv

1351.2.6.34_EVM_Original_Boot.log

As you can see, the PCIe RC is still screwed up. 

So, either there is something wrong with the part on my EVM, or there is something wrong with the kernel. Since it does appear to work with the 2.6.37 kernel, I am thinking that it must be the kernel.

Again, have you actually tested PCIe with ti-ezsdk_c6a816x-evm_5_00_00_56? 

Best regards,

B.J.

And here is the boot log with PCI Debug and the debug kernel command line option enabled:

6765.2.6.34_EVM_PCI_Debug_Boot.log

Still fails.

Now, I just moved aside the kernel modules that came with the EVM, and after booting

without the kernel modules (so no access to GPU or Framebuffer), lspci works. Here is

the boot log with no modules:

7245.2.6.34_EVM_PCI_Debug_NoModules_Boot.log

This appears allow PCIe to basically work. So one of the TI driver modules is hosing

the PCIESS peripheral -- this makes sense since the initial probe works, but the lspci

fails; by the time I can log in, the Matrix app has aready run.

So I guess the question is, how do I get to a boot image/kernel/moduels where I can work with all the features of the part simultaneously.

Best regards,

B.J.

Ok. So that means the main issue here is with the modules/code being executed before the shell prompt (I guess even 2.6.37 will fail if you manage to build the modules and run them).

Looks like the PCIe module is getting reset after enumeration. I will check on my side and get back to you by tomorrow.

   Hemant

Hermant,

I agree with your diagnosis. Thanks for looking into this; I look forward to see what you find.

Best regards,

B.J.

Anything further about this?

Ok -- well, the graphics is not currently critical to my application, but I do need to access the DSP and PCIe simultaneously. It appears to me that the DSP is intended to be accessed via systemlink. Is the failure due to the systemlink component, or is more specific to the graphics elements?

If it is systemlink, then I am going to need a workaround sooner than later, or I will need some guidance on how to interact with the DSP without using systemlink.

Best regards,

B.J.

Yes, the issue is in the syslink module.

   Hemant

Thanks for the patch. 

I was guessing that it was something like this as when I probed RM_DEFAULT_RSTCTRL using devmem2 it was coming back as 0xef which would indicate that the PCIe was in reset. Interestingly to me though is that would also indicate that the USB was in reset, but the USB actually woks....

Anyway, I will apply the patch and report back; thanks for the quick work!

Best regards,

B.J.

Hi Sid,

This does appear to do the trick!

I have PCIe access and the graphics work too!

Thanks for the help on this.

Do you folks have any idea why that line in the driver did not reset the USB cores as well?

Best regards,

B.J.

B.J.

USB modules do not have reset signal connected from PRCM (no LRST), so asserting reset from PRCM doesn't affect USB.

   Hemant

B.J.,

Thanks for the documentation bug report - I will submit this to be changed in the TRM.

Regards,
Marc