Linux/TMS320C6678: Unable to use EP mode chip in "supervisory mode"

Part Number: TMS320C6678

Tool/software: Linux

We've been using the TMS320C6678 Reset Controller to trigger Soft Reset of a device in legacy EP mode behind a PEX8748 (can provide a more detailed block diagram if useful). This procedure works smoothly for us in Linux 3.12 (SLES 12 SP1) but fails us in Linux 4.4 (SLES 12 SP3).

We've been configuring "master transaction mode" via PRIORITY[MST_PRIV] before trying to write any register in the PLL Controller.  We are able to read and write RSTCTRL in Linux 3.12 in "master mode" but read all 1's in Linux 4.4.

Anyone else having mode switch problems in Linux 4.x? It is certainly necessary to be in "supervisory mode" in order to access the PLL Controller but our driver might be missing an event of some kind triggered by the mode switch.

We can provide lots more context including driver source code. Wanted to start with the high-level question in case it rings a bell.

11 Replies

  • Ok, can you elaborate one thing (I wasn't able to understand from your description).

    The TMS320C6678 device is part of a board that has an ARM device and runs linux OS, right? Because TI does not support linux on TMS320C6678 devices... And if that is the case what is the processor running the linux OS?

    Best Regards,
    Yordan

     


     Please make sure you read the forum guidelines first.

  • In reply to Yordan Kovachev:

    We in fact have eight 6678 chips clustered on a full length PCIe capture card. The card is installed in a SuperMicro server. The server is running Linux (SUSE Linux Enterprise to be exact).

    Each 6678 device on the card enumerates as a distinct PCIe endpoint behind a (transparent) PEX8748 switch. The server software is able to control each 6678 in supervisory mode when running any Linux prior to 4.1.

  • In reply to Chad Colgur:

    Hi,

    I've notified the PCIE experts.

    Best Regards,
    Yordan

     


     Please make sure you read the forum guidelines first.

  • In reply to Yordan Kovachev:

    Hi,

    In your system, you have a host running Linux as PCIE Root complex, and 6678 card as the PCIE EP. There should be some code on the C6678 side to initialize the PCIE, link training and BAR, inbound/outbound translation, I assume this code is not changed, correct?

    On the root complex side, you have a Linux PCIE driver. This driver should have the function like enumeration, access C6678's application register and read/write 6678 memory space, etc. For the old Linux 3.12 and New 4.4, can you let me know:

    - Did enumeration work?
    - Are you able to access the 6678 application register, like RIORITY[MST_PRIV], or inbound/outbound translation?
    - Are you able to access the 6678 configuration register, like device id, vendor id?
    - Are you able to read/write other DSP memory, like any of DDR, MSMC, L2?
    - The only issue is to read/write PLL registers like RSTCTRL?

    If the DSP side code is not changed, I thought this is Linux OS issue. What is your capability to use a JTAG to access DSP card to debug?

    Regards, Eric
  • In reply to lding:

    Hi Eric,

    Thanks for your questions. I will repeat and answer them below.

    Q: Did enumeration work?

    A: Yes, all 6678 PCI configuration and capability registers are visible through generally available utilities (e.g. lspci) as well as custom application code.

    Q: Are you able to access 6678 application registers?

    A: Yes, we have read/write access to all PRIORITY as well as translation registers via custom application code. We have not been using generally available utilities for reading and writing (e.g. setpci) nor does our application code depend directly on pcilib. We could experiment with both though as both are at our disposal.

    Q: Are you able to read/write other DSP memory?

    A: Yes, we have read/write access to all of DDR, MSMC and L2 until we switch to privileged mode via set of PRIORITY[MST_PRIV] at which point all reads from those same offsets return all 1's. We haven't done this yet but suspect that if we clear PRIORITY[MST_PRIV] then we will once again have read/write access to those offsets.

    Q: The only issue is to read/write PLL registers like RSTCTRL?

    A: Not exactly. We can read RSTCTRL without being in privileged mode but writes have no effect. According to section 7.6.2.7 of the latest data sheet (rev E), for example, we must write 0x5A69 to RSTCTRL[KEY] before we can set RSTCTRL[SWRST] to trigger "software reset". We confirm successful write of RSTCTRL[KEY] by reading 0x000C. We only receive confirmation when in privileged mode (and then only in Linux 3.x).

    Q: Have you made any changes to the DSP side code?

    A: No changes to the DSP code. I have no doubt this is a Linux OS issue. I was hoping someone here would be able to explain the side-effects of setting PRIORITY[MST_PRIV]. Does the host need to trigger a configuration cycle of some kind having set this bit? Does this bit generate some kind of interrupt that might have been masked in Linux 3.x but unmasked (and mishandled) in Linux 4.x? We are not looking for speculation about what might be happening in the Linux OS. We are trying to interpret the impact of setting this bit from a PCIe standard perspective i.e. What is physically happening on the bus when this bit is set?

    We definitely have JTAG access to the card. We also have a PCIe analyzer at our disposal. We can hook up one or both and trigger on the write to PRIORITY but would ideally have some context before launching that investigation.

  • In reply to Chad Colgur:

    Hi,

    The PRIORITY[MST_PRIV] is to give PCIe transaction the supervisor mode to access the device registers. I am not aware of any impact to the host Linux system. In the past we tested the PCIE Linux driver with change of this bit on Ubuntu 12.04, we didn't see any issue. Also, we didn't see customer reporting that this caused issue in different Linux releases.

    I wanted to you to look at the DSP side with JTAG:
    In working case,
    - the BAR region, starting from 0x21801010,
    - the Inbound translation region, starting from 0x21800300
    - PRIORITY register 0x2180_003C
    - DEBUG0 0X21801728
    After you changed PRIORITY[MST_PRIV] , failure happened, what are those registers? Are they corrupted or still the same?

    Regards, Eric
  • In reply to lding:

    Hi Eric,
    Good to know about the testing you have done with 12.04 and customer reporting.

    I wasn't able to connect JTAG today and haven't mapped the entire 32K PCIe config into host memory. We have mapped the first 4K though so can provide Inbound translation and PRIORITY.

    Inbound translation regions (0, 1, 2, and 3) prefixed by offset:
    300H: 0x00000002
    304H: 0xD0000000
    308H: 0x0000027F
    30CH: 0x00000000
    310H: 0x00000002
    314H: 0xD0080000
    318H: 0x0000027F
    31CH: 0x0C000000
    320H: 0x00000002
    324H: 0xD0480000
    328H: 0x0000027F
    32CH: 0x10800000
    330H: 0x00000004
    334H: 0xC0000000
    338H: 0x0000027F
    33CH: 0x80000000

    PRIORITY register is offset 03CH: 0x00000000

    We do notice an inconsistency suggesting BAR0 region at least is corrupt:

    # lspci -vn -s 07:00
    07:00.0 0480: 104c:b005 (rev 01)
    Subsystem: 13fe:8682
    Flags: bus master, fast devsel, latency 0, IRQ 276
    Memory at c4700000 (64-bit, non-prefetchable) [size=4K]
    Memory at 27ff0000000 (64-bit, prefetchable) [size=16M]
    Memory at 27fe0000000 (64-bit, prefetchable) [size=256M]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [70] Express Endpoint, MSI 00
    Capabilities: [100] Advanced Error Reporting
    Kernel driver in use: dyapcie

    # lspci -vn -s 08:00
    08:00.0 0480: 104c:b005 (rev 01)
    Subsystem: 13fe:8682
    Flags: bus master, fast devsel, latency 0, IRQ 277
    Memory at c4600000 (64-bit, non-prefetchable) [size=4K]
    Memory at 27fd0000000 (64-bit, prefetchable) [size=16M]
    Memory at 27fc0000000 (64-bit, prefetchable) [size=256M]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Capabilities: [70] Express Endpoint, MSI 00
    Capabilities: [100] Advanced Error Reporting
    Kernel driver in use: dyapcie

    # setpci -s 07:00 BASE_ADDRESS_0
    00000004
    # setpci -s 08:00 BASE_ADDRESS_0
    c4600004

    We would expect to see 'c4700004' for device 07:00. Reading zeros there instead would explain invalid transaction even if host were able to present the same address.

    We will try to read the entire 32K of PCIe config via JTAG next week. Host has at least lost the correct BAR0 of a device in supervisor mode.
  • In reply to Chad Colgur:

    One question, offset 0x300, 0x310, 0x320 are different inbound BARs, why they all set to the same "2"?

    Regards, Eric
  • In reply to lding:

    Hi Eric,

    Our host software defines three regions for BAR2 using guidance from "PCIe Use Cases for Keystone devices" (docid sprabk8) section 3.2.3. This isn't new for us in Linux 4.x but can be reverted if it seems problematic.

    Notice also that we use the chip in 64b addressing mode.

    Thanks, Chad

  • In reply to Chad Colgur:

    Hi,

    Have tried to use a CCS/JTAG to dump and compare the PCIE registers why different Linux releases broke PCIE?

    Regards, Eric