• Resolved

Kernel Panic Using PCIe FireWire on TMS320C6A8168 EVM

Hi Folks,

I am trying to bring up a TI XIO2213A 1394B interface via the PCIe slot on the EVM. I am using TI's XIO2213ZAY reference board for the FireWire card.

I have rebuilt the kernel using the Linux PSP included in the ti-ezsdk_c6a816x-evm_5_00_00_56 (e.g. /psp/linux-2.6.34-psp04.00.00.07) and enabled the new FireWire stack.

The OHCI controller is recognized on the PCIe bus and the driver is loaded. 

If I connect a FW HD to the Firewire bus, the device is recognized, the SBP-2 driver is loaded and the FAT filesystem on the drive is mounted.

If I then do a ls on the filesystem, I get a kernel panic:

 

root@c6a816x-evm:~# ls /media/sda1/

Unhandled fault: Precise External Abort on non-linefetch (0x1008) at 0xca87a080

Internal error: : 1008 [#1]

last sysfs file: /sys/kernel/uevent_seqnum

Modules linked in: bufferclass_ti omaplfb pvrsrvkm TI81xx_hdmi(P) ti81xxfb vpss syslink ipv6

CPU: 0    Tainted: P            (2.6.34 #1)

PC is at at_context_transmit+0x460/0x510

LR is at ___dma_single_cpu_to_dev+0x60/0x70

pc : [<c0200ae8>]    lr : [<c00379d0>]    psr: 60000093

sp : c6bb59e0  ip : c6bb59e0  fp : c6bb5a54

r10: c7011000  r9 : 00000093  r8 : c7011000

r7 : c7011534  r6 : c6ad1014  r5 : ffc61d60  r4 : c6ad106c

r3 : ca87a000  r2 : 00000003  r1 : 86ad106c  r0 : 00000003

Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user

Control: 10c5387d  Table: 8682c019  DAC: 00000015

Process ls (pid: 1233, stack limit = 0xc6bb42e8)

The PC where the exception occurs is a register access in the Firewire-OHCI driver drivers/firewire/ohci.c line 1036:

 

    reg_read(ohci, OHCI1394_IntEventSet) & OHCI1394_busReset) {

I asked about this on the 1394linux-devel list, and the first thing recommended was that I update to a newer kernel (2.6.37). I tried to simply build a 2.6.37 kernel from a tag on the omap git tree, but that fails to boot, so obviously, there is more to moving to the newer kernel than simply building from the git tree.
It seems to me that something in the PCIe controller configuration or the kernel's implementation of the PCIe controller driver must be set up incorrectly, as one would expect the basic device register access to be succesful or re-tried. 
I know this board works as I have installed it a desktop machine and it works fine. I have also installed it in a PPC (MPC8377) embedded evm and it worked fine with linux via PCIe.
I definitely need to be able to use FW with the TMS320C6A8168, so any help in attacking this issue would be greatly appreciated, whether it is advice on how to move my system to a newer kernel that may address this issue or advice on how to debug the underlying PCIe issue.
Best regards,
B.J.

 

 

  • Hello,

    Please provide following details:

    1) Output of "lspci -xvvv" (you may need to build/install this if the filesystem doesn't have this utility).

    2) cat /proc/iomem

    3) cat /proc/ioports

    Also, I haven't looked much at the firewire driver but is the crash observed on the *first* access to the register or a few accesses work before the crash?

     

    Thanks.

       Hemant

     

    ---
       Hemant

     

  • In reply to HemantPedanekar:

    Hi Hermant,

    Thank you for the reply.

    I will do these things tomorrow when I have access to the board and post them.

    As far as question about the accesses, I can't tell you off hand if this is the first access to this register or not, but it is definitely not the first access to registers in the OHCI chip as a substantial number of FW transactions have occurred by this point (those required to id the FW device, mount the disk, etc.). Since the register being accessed is the pending Interrupts register, I rather think that this is *not* the first access to this register, but I need to check the code more closely to be sure.

    Best regards,

    B.J.

  • In reply to B.J. Buchalter:

    lspci -xvvv --- you may need to build/install this if the filesystem doesn't have this utility....

     

    What is the easiest way to get the pciutils onto the EVM?

    opkg install pciutils

    does not work.

    I was able to build pciutils with arago-oe, but I have no idea how to get the ipkg stuff to expand on the EVM, or how to get opkg to look at my arago repository. I also don't really see how to manually do the cross compile.

    Any suggestions would be appreciated!

    B.J.

  • In reply to B.J. Buchalter:

    "opkg install pciutils" should work provided conf files inside /etc/opkg directory are set correctly. Make sure you have at least 'armv7a' entry inside /etc/opkg/arch.conf and respective config files for feeds are present (e.g., /etc/opkg/arago-armv7a-feed.conf).

    If you are still not able to get ispci, then you can provide dump of entries from /sys/bus/pci. I particularly wanted dump of resources and size.

       Hemant

     

     

     

     

    ---
       Hemant

     

  • In reply to HemantPedanekar:

    In addition, please also provide the dump of 0x51001728 register  *before* and *after* the crash.

       Hemant

    ---
       Hemant

     

  • In reply to HemantPedanekar:

    HemantPedanekar

    "opkg install pciutils" should work provided conf files inside /etc/opkg directory are set correctly. Make sure you have at least 'armv7a' entry inside /etc/opkg/arch.conf and respective config files for feeds are present (e.g., /etc/opkg/arago-armv7a-feed.conf).

    This does not appear to work. /etc/opkg/arago-armv7a-feed.conf does not exist on the rootfs that comes with the EVM. There is a sample file for that, so I copied the sample to the .conf file, and then did opkg update. But it still does not find the package for pciutils. I finally simply built a rootfs with arago, and then copied the pciutils*.ipkg files to the root image and did the opkg install from the ipkg files. 

    So, I have lspci installed, but it seems like something is still not set up correctly for using the opkg tool.

    When I do lspci, I get something that does not look right (this is with a NEC USB3 card [which I used to try to eliminate the PCIe <-> PCI bridge that is on the FireWire card], but the Firewire card looks the same -- with an additional entry for the bridge)

    # lspci -xvvv

    00:00.0 PCI bridge: Texas Instruments Device 8888 (rev ff) (prog-if ff)

            !!! Unknown header type 7f

    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

    10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

    20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

    30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

     

    01:00.0 USB Controller: NEC Corporation Device 0194 (rev ff) (prog-if ff)

            !!! Unknown header type 7f

    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

    10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

    20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

    30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

     

    as far as dumping the "resource" file from /sys, here is the result:

     

    root@c6a816x-evm:~# more /sys/devices/pci0000\:00/0000\:00\:00.0/resource

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000020000000 0x00000000200fffff 0x0000000000000200

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

     

    root@c6a816x-evm:~# more /sys/devices/pci0000\:00/0000\:00\:00.0/0000\:01\:00.0/resource

    0x0000000020000000 0x0000000020001fff 0x0000000000140204

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    0x0000000000000000 0x0000000000000000 0x0000000000000000

    It seems like something must be screwed up with the PCI RC as all the reads seem to be coming back as 0xffffffff.
    Any clue, or anything else you want me to check?

  • In reply to HemantPedanekar:

    Hi Hemant,

    HemantPedanekar

    In addition, please also provide the dump of 0x51001728 register  *before* and *after* the crash.

    I'm sorry -- I am new to the TI world and still trying to get my bearings. Can you explain exactly how to do this?

    Thanks!

    B.J. Buchalter

  • In reply to HemantPedanekar:

    Here is the other info you requested:

     

    root@c6a816x-evm:~# cat /proc/iomem

    00000000-00000000 : omap2-nand

    20000000-2fffffff : pcie-nonprefetch

    47401000-474017ff : musb_hdrc.0

    47401800-47401fff : musb_hdrc.1

    48020000-4802001f : serial

    48022000-4802201f : serial

    48024000-4802401f : serial

    48028000-4802803f : i2c_omap.1

      48028000-4802803f : i2c_omap

    4802a000-4802a03f : i2c_omap.2

      4802a000-4802a03f : i2c_omap

    48030100-480301ff : omap2_mcspi.1

      48030100-480301ff : omap2_mcspi.1

    48050000-48052fff : mcasp

      48050000-48052fff : davinci-mcasp

    48060100-480700ff : mmci-omap-hs.0

      48060100-480700ff : mmci-omap-hs

    49000000-49007fff : edma_cc0

      49000000-49007fff : edma

    49800000-498003ff : edma_tc0

    49900000-499003ff : edma_tc1

    49a00000-49a003ff : edma_tc2

    49b00000-49b003ff : edma_tc3

    4a100000-4a103fff : davinci_emac.0

      4a100000-4a103fff : eth0

    4a120000-4a123fff : davinci_emac.1

      4a120000-4a123fff : 

    4a140000-4a150fff : ahci

    51000000-51003fff : pcie-regs

    80000000-ffffffff : pcie-inbound0

      80000000-8a5fffff : System RAM

        80031000-803e4fff : Kernel text

        803e6000-80447f5b : Kernel data

    root@c6a816x-evm:~# cat /proc/ioports

    40000000-402fffff : PCI I/O

      40000000-402fffff : pcie-io

  • In reply to B.J. Buchalter:

    So, I managed to build a 2.6.37 kernel and get it booted; it does not fully work because the modules don't match -- still trying to work out how to build a complete kernel and root-filesystem for the newer kernel. 

    That being said, PCIe seems to work much better with the newer kernel; if I boot with the USB3 card installed, lspci -vv actually shows information about the configuration space. If I boot with a FireWire card installed, the kernel panics on boot because it cannot mount the root filesystem. But I suspect that has something to do with not properly building the kernel. 

    So, can someone provide the steps required to build a more modern kernel/root file system with arago? The one that is built from the arago checkout seems to be consistent with the current ezsdk which is 2.6.34.

    Best regards,

    B.J. Buchalter

  • In reply to B.J. Buchalter:

    Hi,

    B.J. Buchalter

    [which I used to try to eliminate the PCIe <-> PCI bridge that is on the FireWire card], but the Firewire card looks the same -- with an additional entry for the bridge)

    Does the lspci dump for firewire card at least show correct values (instead of 0xff 's)?

    B.J. Buchalter

    It seems like something must be screwed up with the PCI RC as all the reads seem to be coming back as 0xffffffff.

    Any clue, or anything else you want me to check?

    Looks like for some reason, the PCIe link went down after successful enumeration. That's why the "/sys/devices/pci0000\:00/0000\:01\:00.0/resource" dump shows correct values but "lspci -x" doesn't.

    You can cross compile and use devmem2 like utility to dump the register I mentioned earlier (0x51001728). The last 5 bits should be 0x11 to indicate the link is established.

    But if the firewire card at least shows correctly in lspci, then I would suggest to use it to debug rather than the USB card.

       Hemant

    ---
       Hemant