TDA4AP-Q1: Limitations for Configuration for SERDES Lanes when using QSGMII / SGMII and SGMII/ USB3 mixed

Thomas Paula

Part Number: TDA4AP-Q1
Other Parts Discussed in Thread: SYSCONFIG

Tool/software:

Hello,

as also asked in PROCESSOR-SDK-J784S4: SerDes Configuration - Processors forum - Processors - TI E2E support forums from SW perspective.

We wanted to use the following HW configuration for the SERDES Lanes:

Serdes 0-0 PCIe1
Serdes 0-2 PCIe3
Serdes 1-0 PCIe0
Serdes 1-2 PCIe2
Serdes 2-0 SGMII - SGMII5
Serdes 2-1 SGMII - SGMII6
Serdes 2-2 QSGMII - SGMII1 to 4
Serdes 2-3 SGMII - SGMII8
Serdes 4-2 SGMII - SGMII7
Serdes 4-3 USB3

From SYSCONFIG tool this option was possible without errors.

1) From the ticket above there was the answer that for SERDES2 the config is obviously not possible due to only two PLLs for different protocols availabe. Even if we only use SGMII, QSGMII.
Is there any hint in Reference manual or datasheet on this limitation, that you would need the following orders for QSGMII, SGMII, SGMII or SGMII, SGMII, QSGMII?

2) For the SERDES4 is SGMII and USB3 this a possible configuration?

3) How to read the Assignment table for the SERDES in chapter 12.2.5. SERDES from the technical reference manual (SPRUJ52C) Table 12-198?
Are the IPx lines (marked in red used ones) the only valid configurations for the SERDESx Ports or can this be mixed up anyway?

For SERDES4 using SGMII, USB3 in parallel there would be no selectable option, when only configs per line possible, correct? In fact this works for our target.

4) So what would be the option to provide all interfaces mentioned above, is a shift of ports in HW necessary (for SERDES2,...) or is there a configuration option per register possible without HW change?

Thanks for response in advance.

over 1 year ago

0 Siddharth Vadapalli over 1 year ago

TI__Expert 3575 points

Thomas Paula said:
From SYSCONFIG tool this option was possible without errors.

SERDES Lane Muxing is different from configuring the SERDES Lanes in the desired configuration.
The Lane Muxing has to support all possible combinations, of which the one being referred to is also a valid MUXING configuration.
This however doesn't mean that the Linux SERDES driver supports configuring the SERDES in that manner.
To clarify, the following MUXING configuration:

Thomas Paula said:
Serdes 2-0 SGMII - SGMII5
Serdes 2-1 SGMII - SGMII6
Serdes 2-2 QSGMII - SGMII1 to 4
Serdes 2-3 SGMII - SGMII8

is supported to satisfy the following use-cases:
1)

Serdes 2-0 SGMII - SGMII5
Serdes 2-1 SGMII - SGMII6
Serdes 2-2 QSGMII - SGMII1 to 4
Serdes 2-3 Unused

Serdes 2-0 Unused
Serdes 2-1 Unused
Serdes 2-2 QSGMII - SGMII1 to 4
Serdes 2-3 SGMII - SGMII8

Since the Lane MUXING is Hardware, it has to support the superset of all possible use-cases. The Linux SERDES driver can enable a single use-case at a time and the superposition of use-cases is not supported if it violates the constraint that at-most 2 sub-nodes must be used to describe the desired use-case in the device-tree.

Thomas Paula said:
Is there any hint in Reference manual or datasheet on this limitation, that you would need the following orders for QSGMII, SGMII, SGMII or SGMII, SGMII, QSGMII?

The limitation is in the Linux SERDES driver which expects at-most two sub-nodes with each sub-node describing a different protocol.

Thomas Paula said:
For the SERDES4 is SGMII and USB3 this a possible configuration?

Yes, it is a valid configuration both from the Lane MUXING perspective and the Linux SERDES driver's ability to configure SERDES4 in that configuration.

Thomas Paula said:
How to read the Assignment table for the SERDES in chapter 12.2.5. SERDES from the technical reference manual (SPRUJ52C) Table 12-198?
Are the IPx lines (marked in red used ones) the only valid configurations for the SERDESx Ports or can this be mixed up anyway?

Each row corresponds to an IP -> PCIe, CPSW (Q/SGMII), USB and Hyperlink.
Different instances of an IP also take up different rows (PCIe1 vs PCIe3, PCIe0 vs PCIe2).
At the same time, it is possible to have multiple lines from a single IP be driven to the same SERDES Lane, implying mutual exclusion.
For example, in the table for SERDES2, both IP1 and IP2 refer to the same CPSW9G instance of CPSW. Lane 2 can be used by either MAC Port 1 of CPSW (XFI/USXGMII/QSGMII/SGMII) or MAC Port 7 of CPSW (Q/SGMII) at a time.

The Assignment table can be read in the following manner:
1. Each Column corresponds to a single lane of the SERDES instance (Lane 0, Lane 1, Lane 2 and Lane 3 -> 4 Columns)
2. Each Row lists the possible options available to choose from across IPs
3. A single Cell which is the intersection of the Column and the Row indicates one option of SERDES Lane to IP Lane mapping that can be chosen.
4. A valid configuration involves choosing only one Cell per Column (The chosen Cells across Columns don't have to belong to the same Row).

Thomas Paula said:
For SERDES4 using SGMII, USB3 in parallel there would be no selectable option, when only configs per line possible, correct? In fact this works for our target.

As I have clarified above, it is possible to choose Cells across Rows. So SGMII and USB3 is a valid configuration since they are using different Lanes of SERDES 4.

Thomas Paula said:
4) So what would be the option to provide all interfaces mentioned above, is a shift of ports in HW necessary (for SERDES2,...) or is there a configuration option per register possible without HW change?

Since it is a MUX, not all combinations are possible at a time. Any valid combination is supported at a given time. The definition of a "valid combination" has been described above.

Regards,
Siddharth.

0 Thomas Paula over 1 year ago in reply to Siddharth Vadapalli

Prodigy 60 points

Hello,
so is it only a SW driver problem that the described configuration cannot be supported or does the HW not support this use case?

Siddharth Vadapalli said:
The limitation is in the Linux SERDES driver which expects at-most two sub-nodes with each sub-node describing a different protocol.

Kind regards

0 Siddharth Vadapalli over 1 year ago in reply to Thomas Paula

TI__Expert 3575 points

Thomas Paula said:
so is it only a SW driver problem that the described configuration cannot be supported or does the HW not support this use case?

It is a limitation of the existing Linux SERDES driver.

Regards,
Siddharth.

0 Dennis Nörmann over 1 year ago

Prodigy 20 points

Siddharth Vadapalli Thanks for clarifying that this is only a Linux Driver issue and no Hardware issue.

When will TI Provide a patch for this Issue ?

We can not change the HW Design, cause this will have impact on the Layout, and this will have negative Impact on Project Timeline, which is not acceptable.

Thanks and Greetings

Dennis Nörmann

0 Siddharth Vadapalli over 1 year ago in reply to Dennis Nörmann

TI__Expert 3575 points

Dennis Nörmann said:
When will TI Provide a patch for this Issue ?

I have worked on the patch to support your use-case and have posted it to Upstream Linux mailing lists for it to be merged.
Link:
https://patchwork.kernel.org/project/linux-arm-kernel/patch/20240709120703.2716397-1-s-vadapalli@ti.com/
With the above patch, the limitation that I had referred to in the E2E thread at:
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1383320/processor-sdk-j784s4-serdes-configuration
is no longer applicable. For your use-case of:

Serdes 2-0 SGMII
Serdes 2-1 SGMII
Serdes 2-2 QSGMII
Serdes 2-3 SGMII

the corresponding device-tree representation of the same will be:

&serdes2 {
	status = "okay";
	#address-cells = <1>;
	#size-cells = <0>;

	serdes2_sgmii_link1: phy@0 {
		reg = <0>;
		cdns,num-lanes = <2>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_SGMII>;
		resets = <&serdes_wiz2 1>, <&serdes_wiz2 2>;
	};

	serdes2_qsgmii_link: phy@2 {
		reg = <2>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_QSGMII>;
		resets = <&serdes_wiz2 3>;
	};

	serdes2_sgmii_link2: phy@3 {
		reg = <3>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_SGMII>;
		resets = <&serdes_wiz2 4>;
	};
};

Please test the patch with the above device-tree changes and let me know.

Regards,
Siddharth.

0 Dennis Nörmann over 1 year ago in reply to Siddharth Vadapalli

Prodigy 20 points

Great, thanks a lot, we are already testing.

0 Daniel Braunwarth over 1 year ago in reply to Siddharth Vadapalli

Intellectual 360 points

Hi Siddarth

Siddharth Vadapalli said:
Please test the patch with the above device-tree changes and let me know.

Seems like this doesn't fix the problem.

I can see the following errors in the kernel log:

dmesg | grep serdes
[    0.491989] cdns-torrent-phy: probe of 5060000.serdes failed with error -22
[    0.493375] cdns-torrent-phy: probe of 5070000.serdes failed with error -22

Those are coming from the PCIe busses we are using on Serdes0 and Serdes1.

Serdes4 is not working anyways.

Regards

Daniel

0 Siddharth Vadapalli over 1 year ago in reply to Daniel Braunwarth

TI__Expert 3575 points

Daniel Braunwarth said:
Seems like this doesn't fix the problem.

Firstly, we see no error for SERDES2. Do SGMII and QSGMII work with SERDES2?
Secondly, the patch exposed the incorrect device-tree sub-nodes within serdes0 and serdes1.
Return value of -22 corresponds to -EINVAL which is returned from the following section:

/* Maximum 2 protocols are supported */
if (hweight32(cdns_phy->protocol_bitmask) != 2)
    return -EINVAL;

It is not clear to me as to why two sub-nodes are present for a single protocol (PCIe in this case) within both serdes0 and serdes1 based on the device-tree contents in the other E2E at:
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1383320/processor-sdk-j784s4-serdes-configuration
which has:

&serdes0 {
	status = "okay";
	serdes0_pcie1_link: phy@0 {
		reg = <0>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_PCIE>;
		resets = <&serdes_wiz0 1>;
	};

	serdes0_pcie3_link: phy@2 {
		reg = <2>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_PCIE>;
		resets = <&serdes_wiz0 3>;
	};
};

and:

&serdes1 {
	status = "okay";
	serdes1_pcie0_link: phy@0 {
		reg = <0>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_PCIE>;
		resets = <&serdes_wiz1 1>;
	};

	serdes1_pcie2_link: phy@2 {
		reg = <2>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_PCIE>;
		resets = <&serdes_wiz1 3>;
	};
};

It appears to me that the above configuration is being used for the PCIe Multi-Link use-case which currently isn't in Mainline Linux. In that case, you could entirely remove the following:

/* Maximum 2 protocols are supported */
if (hweight32(cdns_phy->protocol_bitmask) != 2)
    return -EINVAL;

for your use-case and check if everything works.

Also, could you clarify whether the patch I had shared applied cleanly to the driver in your Linux source?

Regards,
Siddharth.

0 Daniel Braunwarth over 1 year ago in reply to Siddharth Vadapalli

Intellectual 360 points

Hi Siddarth

Siddharth Vadapalli said:
Firstly, we see no error for SERDES2. Do SGMII and QSGMII work with SERDES2?

Serdes 2-0, 2-1 and 2-2 are working. Serdes 2-3 is still not working.

Siddharth Vadapalli said:
It is not clear to me as to why two sub-nodes are present for a single protocol (PCIe in this case) within both serdes0 and serdes1 based on the device-tree contents in the other E2E at:

What would be the way to configure 2x PCIe X1 on those two nodes?

Siddharth Vadapalli said:
It appears to me that the above configuration is being used for the PCIe Multi-Link use-case which currently isn't in Mainline Linux. In that case, you could entirely remove the following:

It worked before so I don't understand why it should not work on mainline.

Exchanging one working SGMII port with two not working PCIe interfaces is not what we want to achieve I guess.

Siddharth Vadapalli said:
Also, could you clarify whether the patch I had shared applied cleanly to the driver in your Linux source?

I double and triple checked the patch has been applied correctly.

Regards
Daniel

0 Daniel Braunwarth over 1 year ago in reply to Daniel Braunwarth

Intellectual 360 points

Hi Siddarth

I need to correct myself.

Serdes 2-3 is working now. There was a little error in the serdes_ln_ctrl configuration. Replacing J784S4_SERDES2_LANE3_QSGMII_LANE2 with J784S4_SERDES2_LANE3_QSGMII_LANE8 fixed the error.

But I would like to clarify the PCIe configuration anyways.

Thanks!

0 Siddharth Vadapalli over 1 year ago in reply to Daniel Braunwarth

TI__Expert 3575 points

Daniel Braunwarth said:
Serdes 2-3 is working now

Great! So the patch does fix the issue for you.

Daniel Braunwarth said:
It worked before so I don't understand why it should not work on mainline.

I have posted the updated v2 patch to handle your use-case as well at:

https://patchwork.kernel.org/project/linux-phy/patch/20240710115624.3232925-1-s-vadapalli@ti.com/

With this patch, all issues should be fixed.
Please retest and let me know.

Regards,
Siddharth.

0 Daniel Braunwarth over 1 year ago in reply to Siddharth Vadapalli

Intellectual 360 points

I've applied and tested v2. Works perfect!

Thanks for your help!

Regards

Daniel

0 Siddharth Vadapalli over 1 year ago in reply to Daniel Braunwarth

TI__Expert 3575 points

Daniel Braunwarth said:
I've applied and tested v2. Works perfect!

That's good to know.
Could you please reply to the v2 patch with your "Tested-by" tag so that it makes it easier for the patch to be merged to mainline Linux? That will ensure that all future releases will contain this patch by default and enable your use-case without any overhead.

Details about the "Tested-by" tag can be found at:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?h=v6.9#n539

Regards,
Siddharth.