Linux/TMS320C6678: Unable to use EP mode chip in "supervisory mode"

Ok, can you elaborate one thing (I wasn't able to understand from your description).

The TMS320C6678 device is part of a board that has an ARM device and runs linux OS, right? Because TI does not support linux on TMS320C6678 devices... And if that is the case what is the processor running the linux OS?

Best Regards,
Yordan

We in fact have eight 6678 chips clustered on a full length PCIe capture card. The card is installed in a SuperMicro server. The server is running Linux (SUSE Linux Enterprise to be exact).

Each 6678 device on the card enumerates as a distinct PCIe endpoint behind a (transparent) PEX8748 switch. The server software is able to control each 6678 in supervisory mode when running any Linux prior to 4.1.

Hi,

I've notified the PCIE experts.

Best Regards,
Yordan

Hi,

In your system, you have a host running Linux as PCIE Root complex, and 6678 card as the PCIE EP. There should be some code on the C6678 side to initialize the PCIE, link training and BAR, inbound/outbound translation, I assume this code is not changed, correct?

On the root complex side, you have a Linux PCIE driver. This driver should have the function like enumeration, access C6678's application register and read/write 6678 memory space, etc. For the old Linux 3.12 and New 4.4, can you let me know:

- Did enumeration work?
- Are you able to access the 6678 application register, like RIORITY[MST_PRIV], or inbound/outbound translation?
- Are you able to access the 6678 configuration register, like device id, vendor id?
- Are you able to read/write other DSP memory, like any of DDR, MSMC, L2?
- The only issue is to read/write PLL registers like RSTCTRL?

If the DSP side code is not changed, I thought this is Linux OS issue. What is your capability to use a JTAG to access DSP card to debug?

Regards, Eric

Hi Eric,

Thanks for your questions. I will repeat and answer them below.

Q: Did enumeration work?

A: Yes, all 6678 PCI configuration and capability registers are visible through generally available utilities (e.g. lspci) as well as custom application code.

Q: Are you able to access 6678 application registers?

A: Yes, we have read/write access to all PRIORITY as well as translation registers via custom application code. We have not been using generally available utilities for reading and writing (e.g. setpci) nor does our application code depend directly on pcilib. We could experiment with both though as both are at our disposal.

Q: Are you able to read/write other DSP memory?

A: Yes, we have read/write access to all of DDR, MSMC and L2 until we switch to privileged mode via set of PRIORITY[MST_PRIV] at which point all reads from those same offsets return all 1's. We haven't done this yet but suspect that if we clear PRIORITY[MST_PRIV] then we will once again have read/write access to those offsets.

Q: The only issue is to read/write PLL registers like RSTCTRL?

A: Not exactly. We can read RSTCTRL without being in privileged mode but writes have no effect. According to section 7.6.2.7 of the latest data sheet (rev E), for example, we must write 0x5A69 to RSTCTRL[KEY] before we can set RSTCTRL[SWRST] to trigger "software reset". We confirm successful write of RSTCTRL[KEY] by reading 0x000C. We only receive confirmation when in privileged mode (and then only in Linux 3.x).

Q: Have you made any changes to the DSP side code?

A: No changes to the DSP code. I have no doubt this is a Linux OS issue. I was hoping someone here would be able to explain the side-effects of setting PRIORITY[MST_PRIV]. Does the host need to trigger a configuration cycle of some kind having set this bit? Does this bit generate some kind of interrupt that might have been masked in Linux 3.x but unmasked (and mishandled) in Linux 4.x? We are not looking for speculation about what might be happening in the Linux OS. We are trying to interpret the impact of setting this bit from a PCIe standard perspective i.e. What is physically happening on the bus when this bit is set?

We definitely have JTAG access to the card. We also have a PCIe analyzer at our disposal. We can hook up one or both and trigger on the write to PRIORITY but would ideally have some context before launching that investigation.

Hi,

The PRIORITY[MST_PRIV] is to give PCIe transaction the supervisor mode to access the device registers. I am not aware of any impact to the host Linux system. In the past we tested the PCIE Linux driver with change of this bit on Ubuntu 12.04, we didn't see any issue. Also, we didn't see customer reporting that this caused issue in different Linux releases.

I wanted to you to look at the DSP side with JTAG:
In working case,
- the BAR region, starting from 0x21801010,
- the Inbound translation region, starting from 0x21800300
- PRIORITY register 0x2180_003C
- DEBUG0 0X21801728
After you changed PRIORITY[MST_PRIV] , failure happened, what are those registers? Are they corrupted or still the same?

Regards, Eric

Hi Eric,
Good to know about the testing you have done with 12.04 and customer reporting.

I wasn't able to connect JTAG today and haven't mapped the entire 32K PCIe config into host memory. We have mapped the first 4K though so can provide Inbound translation and PRIORITY.

Inbound translation regions (0, 1, 2, and 3) prefixed by offset:
300H: 0x00000002
304H: 0xD0000000
308H: 0x0000027F
30CH: 0x00000000
310H: 0x00000002
314H: 0xD0080000
318H: 0x0000027F
31CH: 0x0C000000
320H: 0x00000002
324H: 0xD0480000
328H: 0x0000027F
32CH: 0x10800000
330H: 0x00000004
334H: 0xC0000000
338H: 0x0000027F
33CH: 0x80000000

PRIORITY register is offset 03CH: 0x00000000

We do notice an inconsistency suggesting BAR0 region at least is corrupt:

# lspci -vn -s 07:00
07:00.0 0480: 104c:b005 (rev 01)
Subsystem: 13fe:8682
Flags: bus master, fast devsel, latency 0, IRQ 276
Memory at c4700000 (64-bit, non-prefetchable) [size=4K]
Memory at 27ff0000000 (64-bit, prefetchable) [size=16M]
Memory at 27fe0000000 (64-bit, prefetchable) [size=256M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: dyapcie

# lspci -vn -s 08:00
08:00.0 0480: 104c:b005 (rev 01)
Subsystem: 13fe:8682
Flags: bus master, fast devsel, latency 0, IRQ 277
Memory at c4600000 (64-bit, non-prefetchable) [size=4K]
Memory at 27fd0000000 (64-bit, prefetchable) [size=16M]
Memory at 27fc0000000 (64-bit, prefetchable) [size=256M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: dyapcie

# setpci -s 07:00 BASE_ADDRESS_0
00000004
# setpci -s 08:00 BASE_ADDRESS_0
c4600004

We would expect to see 'c4700004' for device 07:00. Reading zeros there instead would explain invalid transaction even if host were able to present the same address.

We will try to read the entire 32K of PCIe config via JTAG next week. Host has at least lost the correct BAR0 of a device in supervisor mode.

One question, offset 0x300, 0x310, 0x320 are different inbound BARs, why they all set to the same "2"?

Regards, Eric

Hi Eric,

Our host software defines three regions for BAR2 using guidance from "PCIe Use Cases for Keystone devices" (docid sprabk8) section 3.2.3. This isn't new for us in Linux 4.x but can be reverted if it seems problematic.

Notice also that we use the chip in 64b addressing mode.

Thanks, Chad

Hi,

Have tried to use a CCS/JTAG to dump and compare the PCIE registers why different Linux releases broke PCIE?

Regards, Eric

Hi Eric,

Thanks for checking back with us. We've been pulled away on a different task but will return to this one asap.

We did try setting MSI_PRIV from generic sysfs entries for PCI (e.g. /sys/devices/pci0000:00/*) instead of using memory mapped by our driver. Initial testing was promising but we didn't follow-up with a full soft-reset cycle. That will be our next experiment. If we find positive correlation with our driver then that will help narrow the problem to something in our direct control. We don't have that confirmation yet but will keep you posted on both CCS/JTAG and this experiment with our driver.

Best, Chad