• State Resolved
  • Locked Locked
  • Replies 7 replies
  • Answers 1 answer
  • Subscribers 98 subscribers
  • Views 429 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

PROCESSOR-SDK-J721E: Does TDA4x Supports SR-IOV?

Hans Acosta
Prodigy 40 points
Part Number: PROCESSOR-SDK-J721E


I have one TDA4-VH and one TDA4-VM (https://www.ti.com/tool/J784S4XEVM) that I'm connecting to an Intel Platform through PCIe that has a couple of VMs running, they are independent systems (VH connected to one Intel Platform, VM connected to another Intel Platform).

I want to use the SR-IOV technology, does this SoC supports SR-IOV?

Where can I find the documentation for this?

If it does not support SR-IOV, how can I request this implementation to TI?

Thanks. 

  • +1
    TI__Guru 55893 points

    Hi Hans,

    Yes, the SoC supports SR-IOV. I would recommend looking through the PCIe End Point documentation: https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-jacinto7/09_01_00_06/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/PCIe/PCIe_End_Point.html

    Specifically, the section "Starting the EP device" outlines the steps to enable virtual function.

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    Prodigy 40 points

    Hi Takuma,

    I followed the instructions from the above link and ran this in the TDA4:

    # Creating virtual fn1

    mkdir functions/pci_epf_test/vf1

    echo 0x104c > functions/pci_epf_test/vf1/vendorid

    echo 0xb00d > functions/pci_epf_test/vf1/deviceid

    echo 4 > functions/pci_epf_test/vf1/msi_interrupts

    echo 8 > functions/pci_epf_test/vf1/msix_interrupts

     

    # Creating virtual fn2

    mkdir functions/pci_epf_test/vf2

    echo 0x104c > functions/pci_epf_test/vf2/vendorid

    echo 0xb00d > functions/pci_epf_test/vf2/deviceid

    echo 4 > functions/pci_epf_test/vf2/msi_interrupts

    echo 8 > functions/pci_epf_test/vf2/msix_interrupts

     

    # Creating virtual fn3

    mkdir functions/pci_epf_test/vf3

    echo 0x104c > functions/pci_epf_test/vf3/vendorid

    echo 0xb00d > functions/pci_epf_test/vf3/deviceid

    echo 4 > functions/pci_epf_test/vf3/msi_interrupts

    echo 8 > functions/pci_epf_test/vf3/msix_interrupts

     

    # Creating virtual fn4

    mkdir functions/pci_epf_test/vf4

    echo 0x104c > functions/pci_epf_test/vf4/vendorid

    echo 0xb00d > functions/pci_epf_test/vf4/deviceid

    echo 4 > functions/pci_epf_test/vf4/msi_interrupts

    echo 8 > functions/pci_epf_test/vf4/msix_interrupts

     

    mkdir functions/pci_epf_test/func1

    echo 0x104c > functions/pci_epf_test/func1/vendorid

    echo 0xb00d > functions/pci_epf_test/func1/deviceid

    echo 11 > functions/pci_epf_test/func1/msi_interrupts

    echo 11 > functions/pci_epf_test/func1/msix_interrupts

     

    mkdir functions/pci_epf_test/func2

    echo 0x104c > functions/pci_epf_test/func2/vendorid

    echo 0xb00d > functions/pci_epf_test/func2/deviceid

    echo 12 > functions/pci_epf_test/func2/msi_interrupts

    echo 12 > functions/pci_epf_test/func2/msix_interrupts

     

    # Binding the two virtual functions to pf1

    ln -s functions/pci_epf_test/vf1 functions/pci_epf_test/func1

    ln -s functions/pci_epf_test/vf2 functions/pci_epf_test/func1

    ln -s functions/pci_epf_test/vf3 functions/pci_epf_test/func2

    ln -s functions/pci_epf_test/vf4 functions/pci_epf_test/func2

     

    ln -s functions/pci_epf_test/func1/ controllers/2900000.pcie-ep/

    ln -s functions/pci_epf_test/func2/ controllers/2900000.pcie-ep/

     

    echo 1 > controllers/2900000.pcie-ep/start

    And in the Host I see the following using lspci:

    root@user:/sys/kernel/config/pci_ep/functions# sudo lspci

    01:00.0 Unassigned class [ff00]: Texas Instruments Device b00d

    01:00.1 Unassigned class [ff00]: Texas Instruments Device b00d


    root@user:~/acrn-work$ lspci -s 01:00.0 -xxx -vvv
    01:00.0 Unassigned class [ff00]: Texas Instruments Device b00d
            Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
            Interrupt: pin A routed to IRQ 255
            Region 0: Memory at 92e4d000 (32-bit, non-prefetchable) [disabled] [size=512]
            Region 1: Memory at 92e4c000 (32-bit, non-prefetchable) [disabled] [size=512]
            Region 2: Memory at 92e4b000 (32-bit, non-prefetchable) [disabled] [size=1K]
            Region 3: Memory at 92e44000 (32-bit, non-prefetchable) [disabled] [size=16K]
            Region 4: Memory at 92e20000 (32-bit, non-prefetchable) [disabled] [size=128K]
            Region 5: Memory at 92d00000 (32-bit, non-prefetchable) [disabled] [size=1M]
            Capabilities: <access denied>
            Kernel driver in use: pci-stub
    00: 4c 10 0d b0 00 00 10 00 00 00 00 ff 10 00 80 00
    10: 00 d0 e4 92 00 c0 e4 92 00 b0 e4 92 00 40 e4 92
    20: 00 00 e2 92 00 00 d0 92 00 00 00 00 00 00 00 00
    30: 00 00 00 00 80 00 00 00 00 00 00 00 ff 01 00 00

     

    root@user:~/acrn-work$ lspci -s 01:00.1 -xxx -vvv
    01:00.1 Unassigned class [ff00]: Texas Instruments Device b00d
            Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
            Interrupt: pin A routed to IRQ 255
            Region 0: Memory at 92e4a000 (32-bit, non-prefetchable) [disabled] [size=512]
            Region 1: Memory at 92e49000 (32-bit, non-prefetchable) [disabled] [size=512]
            Region 2: Memory at 92e48000 (32-bit, non-prefetchable) [disabled] [size=1K]
            Region 3: Memory at 92e40000 (32-bit, non-prefetchable) [disabled] [size=16K]
            Region 4: Memory at 92e00000 (32-bit, non-prefetchable) [disabled] [size=128K]
            Region 5: Memory at 92c00000 (32-bit, non-prefetchable) [disabled] [size=1M]
            Capabilities: <access denied>
            Kernel driver in use: pci-stub
    00: 4c 10 0d b0 00 00 10 00 00 00 00 ff 10 00 80 00
    10: 00 a0 e4 92 00 90 e4 92 00 80 e4 92 00 00 e4 92
    20: 00 00 e0 92 00 00 c0 92 00 00 00 00 00 00 00 00
    30: 00 00 00 00 80 00 00 00 00 00 00 00 ff 01 00 00

    Does this mean that the TDA4 has 2 Virtual Devices that I can assign to two different VMs?

    Thanks.

  • 0 in reply to Hans Acosta
    TI__Guru 55893 points

    Hi Hans,

    Since the Capabilities are not shown under lspci I cannot be certain, but most likely the highlighted devices are the physical functions instead of the virtual functions.

    To show virtual functions using lspci, you could do something like: echo 4 > /sys/bus/pci/devices/0001\:01\:00.0/sriov_numvfs. Where 4 is the number of virtual functions, and 0001\:01\:00.0 is the physical function the virtual function should be under. I tried this command out and it enumerates the virtual functions that are not enumerated by default under lspci.

    Command referenced from kernel docs here: https://docs.kernel.org/PCI/pci-iov-howto.html.

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    Prodigy 40 points

    This is what I see on the Capabilities section:

    01:00.0 Unassigned class [ff00]: Texas Instruments Device b00d
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 255
    Region 0: Memory at 92e4d000 (32-bit, non-prefetchable) [disabled] [size=512]
    Region 1: Memory at 92e4c000 (32-bit, non-prefetchable) [disabled] [size=512]
    Region 2: Memory at 92e4b000 (32-bit, non-prefetchable) [disabled] [size=1K]
    Region 3: Memory at 92e44000 (32-bit, non-prefetchable) [disabled] [size=16K]
    Region 4: Memory at 92e20000 (32-bit, non-prefetchable) [disabled] [size=128K]
    Region 5: Memory at 92d00000 (32-bit, non-prefetchable) [disabled] [size=1M]
    Capabilities: [80] Power Management version 3
    Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
    Address: 0000000000000000 Data: 0000
    Capabilities: [b0] MSI-X: Enable- Count=16 Masked-
    Vector table: BAR=0 offset=00000080
    PBA: BAR=0 offset=00000170
    Capabilities: [c0] Express (v2) Endpoint, MSI 00
    DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <1us
    ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
    DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
    RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
    MaxPayload 256 bytes, MaxReadReq 512 bytes
    DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
    LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <8us
    ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
    LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
    ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
    LnkSta: Speed 8GT/s (ok), Width x1 (ok)
    TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
    DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
    10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
    EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
    FRS- TPHComp- ExtTPHComp-
    AtomicOpsCap: 32bit- 64bit- 128bitCAS-
    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
    AtomicOpsCtl: ReqEn-
    LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
    LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
    Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
    Compliance De-emphasis: -6dB
    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
    EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
    Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [100 v2] Advanced Error Reporting
    UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
    UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
    UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
    CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ AdvNonFatalErr+
    CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
    AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
    MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
    HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [140 v1] Alternative Routing-ID Interpretation (ARI)
    ARICap: MFVC- ACS-, Next Function: 1
    ARICtl: MFVC- ACS-, Function Group: 0
    Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
    Capabilities: [160 v1] Power Budgeting <?>
    Capabilities: [1b8 v1] Latency Tolerance Reporting
    Max snoop latency: 15728640ns
    Max no snoop latency: 15728640ns
    Capabilities: [1c0 v1] Dynamic Power Allocation <?>
    Capabilities: [200 v1] Single Root I/O Virtualization (SR-IOV)
    IOVCap: Migration-, Interrupt Message Number: 000
    IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
    IOVSta: Migration-
    Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
    VF offset: 6, stride: 1, Device ID: b00d
    Supported Page Size: 00000553, System Page Size: 00000001
    Region 4: Memory at 92e60000 (32-bit, non-prefetchable)
    VF Migration: offset: 00000000, BIR: 0
    Capabilities: [300 v1] Secondary PCI Express
    LnkCtl3: LnkEquIntrruptEn- PerformEqu-
    LaneErrStat: LaneErr at lane: 0
    Capabilities: [400 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
    Capabilities: [440 v1] Process Address Space ID (PASID)
    PASIDCap: Exec+ Priv+, Max PASID Width: 14
    PASIDCtl: Enable- Exec- Priv-
    Capabilities: [4c0 v1] Virtual Channel
    Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
    Arb: Fixed- WRR32- WRR64- WRR128-
    Ctrl: ArbSelect=Fixed
    Status: InProgress-
    VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
    Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
    Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
    Status: NegoPending- InProgress-
    VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
    Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
    Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00
    Status: NegoPending- InProgress-
    VC2: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
    Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
    Ctrl: Enable- ID=2 ArbSelect=Fixed TC/VC=00
    Status: NegoPending- InProgress-
    VC3: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
    Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
    Ctrl: Enable- ID=3 ArbSelect=Fixed TC/VC=00
    Status: NegoPending- InProgress-
    Capabilities: [5c0 v1] Address Translation Service (ATS)
    ATSCap: Invalidate Queue Depth: 01
    ATSCtl: Enable-, Smallest Translation Unit: 00
    Capabilities: [640 v1] Page Request Interface (PRI)
    PRICtl: Enable- Reset-
    PRISta: RF- UPRGI- Stopped+
    Page Request Capacity: 00000001, Page Request Allocation: 00000000
    Capabilities: [900 v1] L1 PM Substates
    L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
    PortCommonModeRestoreTime=255us PortTPowerOnTime=26us
    L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
    T_CommonMode=0us LTR1.2_Threshold=783360ns
    L1SubCtl2: T_PwrOn=500us
    Capabilities: [a20 v1] Precision Time Measurement
    PTMCap: Requester:+ Responder:- Root:-
    PTMClockGranularity: Unimplemented
    PTMControl: Enabled:+ RootSelected:-
    PTMEffectiveGranularity: 4ns
    00: 4c 10 0d b0 00 00 10 00 00 00 00 ff 10 00 80 00
    10: 00 d0 e4 92 00 c0 e4 92 00 b0 e4 92 00 40 e4 92
    20: 00 00 e2 92 00 00 d0 92 00 00 00 00 00 00 00 00
    30: 00 00 00 00 80 00 00 00 00 00 00 00 ff 01 00 00
    40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    80: 01 90 03 5a 08 00 00 00 00 00 00 00 00 00 00 00
    90: 05 b0 80 00 00 00 00 00 00 00 00 00 00 00 00 00
    a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    b0: 11 c0 0f 00 80 00 00 00 70 01 00 00 00 00 00 00
    c0: 10 00 02 00 01 81 00 10 30 28 09 00 13 a8 41 00
    d0: 00 00 13 00 00 00 00 00 00 00 00 00 00 00 00 00
    e0: 00 00 00 00 12 08 71 00 00 04 00 00 0e 0e 07 00
    f0: 03 00 1e 00 00 00 00 00 00 00 00 00 00 00 00 00

    I see both PH 0 VF, the Host has VT-D/SRIOV enabled and I see that the GPU has SRIOV working but not this platform.

    What other things should I be checking in order to see the VF working? Is this a bug in the SDK?

    Thanks,

  • 0 in reply to Hans Acosta
    TI__Guru 55893 points

    Hi Hans,

    Is this after doing the command: "echo 4 > /sys/bus/pci/devices/0001\:01\:00.0/sriov_numvfs"? Please note that 4 is the number of virtual functions, and 0001\:01\:00.0 is the physical function the virtual function should be under, and these values would change based on how many virtual functions you set up and what other devices are enumerated on the RC-side. I have tried this command out and it enumerates the virtual functions that are not enumerated by default under lspci.

    Again, the commands are referenced from kernel docs here: https://docs.kernel.org/PCI/pci-iov-howto.html.

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    Prodigy 40 points

    Hi Takuma,

    I tried this before, since for the GPU is the same way I'm doing it to assign the VF but for this platform when I try I get this error:

    echo 2 > sriov_numvfs
    bash: echo: write error: No such file or directory
  • 0 in reply to Hans Acosta
    TI__Guru 55893 points

    Hi Hans,

    When I tried this on two TI EVMs, the virtual functions got enumerated. Therefore, I believe the issue is the difference in RC-side.

    Some differences I can think of are:

    • Permission issues - To check this, could you try with sudo
    • Doing "echo 2 > sriov_numvfs" in the wrong directory - To check this, could you do an "ls" where the command was ran
    • Trying to enable sriov_numvfs for the wrong device - To check this, what are the devices under /sys/bus/pci/devices?

    Regards,

    Takuma