TDA4VM: TDA4VM PCIe structure

Team, 

updating this internal thread with the email discussions. 
jian

From: Kishon Vijay Abraham <kishon@ti.com>
Sent: Monday, September 12, 2022 7:21 AM
To: Govindarajan, Sriramakrishnan <srk@ti.com>; Zhu, Xingyu <xingyu-zhu@ti.com>; Rajaraman, Sai <sairajaraman@ti.com>; Cobb, Brad <bradc@ti.com>; Lothamer, Heather <h-lothamer@ti.com>; Shankar, Abhishek <a.shankar@ti.com>; Shurtz, Gregory <g-shurtz1@ti.com>; Wang, Jian <jian-wang@ti.com>
Cc: Han, Tao <tao-han@ti.com>; Zhang, Fredy <fredy-zhang@ti.com>; Sojitra, Ritesh <rsojitra@ti.com>; Jadav, Brijesh <brijesh.jadav@ti.com>
Subject: Re: TDA4VM PCIe structure

All,

Please find answers inline..

From: Govindarajan, Sriramakrishnan
Sent: Monday, September 12, 2022 5:36 PM
To: Zhu, Xingyu; Rajaraman, Sai; Cobb, Brad; Lothamer, Heather; Kishon Vijay Abraham; Shankar, Abhishek; Shurtz, Gregory; Wang, Jian
Cc: Han, Tao; Zhang, Fredy; Sojitra, Ritesh; Jadav, Brijesh
Subject: RE: TDA4VM PCIe structure

Hi

Answers inline below

1. Does TDA4VM PCIe support this structure? Can TDA4VM support these 3 NTB configuration?

[Sriram] Yes , TDA4 can support this configuration. We can model NTB port connections between J5#1/J5#2/PC and a direct RC to EP connection modelled as separate function “pci_epf_tda4”

2. Can EP0 controller add this function (pci_epf_TDA4)? Do we have sample code for this function?

[Sriram] We provide an EP test function that does a simple DMA from EP side to RC which can be used as starting point.

But we have not included this alongside NTB use case, but I believe this should need configuration change largely.

Kishon, can you provide pointers for this.

[Kishon]: Both EP test function and NTB can be used simultaneously. Different physical function can be used for different usecases.

3. Can J5#1 send data to pcie_epf_tda4, PC and J5#2 at the same time?

[Sriram] Yes, J5#1 should see 3 independent functions (pcie_epf, NTB#1, NTB#2) and bind software on the RC accordingly

4. If pci_epf_tda4 can be supported, what is the implementation form? Socket or lib? Can be virtual internet card?

[Sriram]  this should be based on the intended usage model from J5 side.  PCIe acts only as  underlying transport and the actual interface is modeled by the “Function driver”  on either side

[Kishon]: Right now we support simple communication form using pci_epf_test (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/endpoint/functions/pci-epf-test.c). However upstream Linux also has a way for RC<->EP communication using ethernet

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/endpoint/functions/pci-epf-vntb.c

5. Can TDA4VM meet the bandwidth and latency (<=10ms)  requirement?

[Sriram] latency is usually in microsecond range for the individual data packets carried over PCIe.

For bandwidth I assuming  the total traffic through TDA4M (including both directions) is 100+120+200 = 420MB/s or 3.360Gbps , which TDA can handle.

Review to see if the connection to J5 is Gen2 vs Gen3 as the aggregate traffic needs to flow through this link.

6. To realize this function, does J5#1 need 2 virtual internet card (one connect to J5#2 another to PC)?

[Sriram] Connection to J5#2 and PC are modelled as NTB ports. There are additional abstractions/example(ntb_netdev, ntb_tool, ntb_perf) available in Linux that show communication model/usage over NTB shared memory region(although these may not be very efficient and use CPU to copy packets).

Overall, PCIe provides a shared memory interface for remote core access(w/NTB). Depending on the nature of data to be exchanged, we should define the right SW abstractions on top of this.

For instance if you are exchanging video/sensor data, it is better to build a PCIe function for this rather than taking video data and packetizing it to be sent as network packets

Regards

Sriram

From: Zhu, Xingyu <xingyu-zhu@ti.com>
Sent: Thursday, September 8, 2022 6:10 PM
To: Rajaraman, Sai <sairajaraman@ti.com>; Govindarajan, Sriramakrishnan <srk@ti.com>; Cobb, Brad <bradc@ti.com>; Lothamer, Heather <h-lothamer@ti.com>; Kishon Vijay Abraham <kishon@ti.com>; Shankar, Abhishek <a.shankar@ti.com>; Shurtz, Gregory <g-shurtz1@ti.com>; Wang, Jian <jian-wang@ti.com>
Cc: Han, Tao <tao-han@ti.com>; Zhang, Fredy <fredy-zhang@ti.com>; Sojitra, Ritesh <rsojitra@ti.com>
Subject: RE: TDA4VM PCIe structure

+Jian

Thanks for Sai and Sriram reply.

The PCIe structure has been confirmed (question 1), could you help to give your comments about the rest questions? Thanks a lot.

Best Regards,

Xingyu Zhu

Texas Instruments Semiconductor Co.,Ltd.

+86 +021-23073194 / +86 13482717530

From: Rajaraman, Sai <sairajaraman@ti.com>
Sent: Friday, August 19, 2022 5:25 PM
To: Govindarajan, Sriramakrishnan <srk@ti.com>; Zhu, Xingyu <xingyu-zhu@ti.com>; Cobb, Brad <bradc@ti.com>; Lothamer, Heather <h-lothamer@ti.com>; Kishon Vijay Abraham <kishon@ti.com>; Shankar, Abhishek <a.shankar@ti.com>; Shurtz, Gregory <g-shurtz1@ti.com>
Cc: Han, Tao <tao-han@ti.com>; Zhang, Fredy <fredy-zhang@ti.com>; Sojitra, Ritesh <rsojitra@ti.com>
Subject: RE: TDA4VM PCIe structure

Thanks Sriram for quick confirmation.

Best Regards,

Sai

From: Govindarajan, Sriramakrishnan <srk@ti.com>
Sent: Thursday, August 18, 2022 11:46 PM
To: Rajaraman, Sai <sairajaraman@ti.com>; Zhu, Xingyu <xingyu-zhu@ti.com>; Cobb, Brad <bradc@ti.com>; Lothamer, Heather <h-lothamer@ti.com>; Kishon Vijay Abraham <kishon@ti.com>; Shankar, Abhishek <a.shankar@ti.com>; Shurtz, Gregory <g-shurtz1@ti.com>
Cc: Han, Tao <tao-han@ti.com>; Zhang, Fredy <fredy-zhang@ti.com>; Sojitra, Ritesh <rsojitra@ti.com>
Subject: RE: TDA4VM PCIe structure

Sai

Yes we should be able to leverage multi-function capability to model NTB connections to remote host and also include PCie Functions alongside

Each connection to remote host would need one EP function. So following combinations all are possible on EP0 controller

1. NTB:SOC#2 + NTB: PC + EP FN (local epf tda4)
2. NTB:SOC#2 + NTB: PC + EP FN (local ethernet switch function mapped over PCIe to SOC#1)
3. 2 above + EP FN (SSD connected to TDA4VM mapped as remote storage over PCIe to SOC#1)

Regards

Sriram

From: Rajaraman, Sai <sairajaraman@ti.com>
Sent: Thursday, August 18, 2022 7:18 PM
To: Zhu, Xingyu <xingyu-zhu@ti.com>; Cobb, Brad <bradc@ti.com>; Lothamer, Heather <h-lothamer@ti.com>; Kishon Vijay Abraham <kishon@ti.com>; Shankar, Abhishek <a.shankar@ti.com>; Shurtz, Gregory <g-shurtz1@ti.com>; Govindarajan, Sriramakrishnan <srk@ti.com>
Cc: Han, Tao <tao-han@ti.com>; Zhang, Fredy <fredy-zhang@ti.com>; Sojitra, Ritesh <rsojitra@ti.com>
Subject: RE: TDA4VM PCIe structure

Hi Xingyu,

Since IP supports up to 6 physical functions, I don’t see a concern with adding one more function (pcie_epf_tda4).  However, it would be good to review the structure with systems as well since it involves multiple PCIe and SOC level architecture. 

Abhishek/Greg/Sriram, can you please let us know if you see any issues in supporting the 3 NTB structure provided in the first email below?

Thank you,

Sai