AM623: AM623: MAC to MAC connection support

Bert

Part Number: AM623

Tool/software:

We use the LINUX SDK 10.01.10.04, and use the RMII2 interface to communicate with the ethernet switch IP175LLF, it is MAC to MAC connection, Does am623 support it ? thanks

3 months ago

+1 Daolin Qiu 3 months ago

TI__Genius 16305 points

Hello Bert,

While within TI we do not have examples of AM62x MAC-to-MAC connection with an external switch on Linux, we have seen customers implement external switches with AM62x CPSW interface with varying levels of success.

From my understanding, the main configuration from the AM62x side is to add the "fixed-link" property in the DTS on the Ethernet port you are connecting to your external switch.

&cpsw_port2 {
	...
	fixed-link {
		speed = <100>;
		full-duplex;
	};
	...
};

Based on what I've seen from past customers, the key concerns I see (from Linux OS side) are

DTS configuration for the external switch should be determined from the switch vendor. Your ethernet switch IP175LLF is a non-TI part so I would suggest finding out what the expected DTS configuration is needed for the external switch
Does your external switch have a Linux driver?
Are you using SPI or MDIO to communicate with your external switch? There may be more issues when trying to use MDIO.
Some customers have found that some changes in the DSA (Distributed Switch Architecture) Linux driver needs to be made to get the switch attached to the correct CPSW port number
1. Appears to be observed when CPSW port 2 is selected as the one connected to the switch
Are you planning on using VLANs in the external switch?
1. While I’m less familiar with VLAN usage in external switches, it appears in the past, special changes to the CPSW/DSA drivers were necessary to make the external switches functional to the customer’s requirements

Please let us know if you have follow-up questions.

-Daolin

0 Bert 3 months ago in reply to Daolin Qiu

Intellectual 570 points

hi Daolin

Thanks for your suggestion. We meet a very strange question, the ethernet frame from am623 to switch is ok and the switch is received correctly, but for the opposite direction , am623 can not receive the ethernet frame from switch . Do you have some suggestion to investigate it ?

frame data flow:

am623----> switch : normally and the switch is received correctly

switch----> am623 : am623 can not receive the frame from switch

the command "ethtool -s eth1 " print the number of frames , all is 0

0 Bert 3 months ago in reply to Bert

Intellectual 570 points

We use the oscilloscope to measure the waveform of RMII data from switch to am623 which looks good and we don't not use the switch VLAN

0 Bert 3 months ago in reply to Daolin Qiu

Intellectual 570 points

below is the command "ethtool -S eth1" result , I found the item iet_rx_smd_err number is growing, please help analysis it

root@:~# ethtool -S eth1
NIC statistics:
p0_rx_good_frames: 88
p0_rx_broadcast_frames: 77
p0_rx_multicast_frames: 11
p0_rx_crc_errors: 0
p0_rx_oversized_frames: 0
p0_rx_undersized_frames: 0
p0_ale_drop: 0
p0_ale_overrun_drop: 0
p0_rx_octets: 5838
p0_tx_good_frames: 0
p0_tx_broadcast_frames: 0
p0_tx_multicast_frames: 0
p0_tx_octets: 0
p0_tx_64B_frames: 77
p0_tx_65_to_127B_frames: 11
p0_tx_128_to_255B_frames: 0
p0_tx_256_to_511B_frames: 0
p0_tx_512_to_1023B_frames: 0
p0_tx_1024B_frames: 0
p0_net_octets: 5838
p0_rx_bottom_fifo_drop: 0
p0_rx_port_mask_drop: 0
p0_rx_top_fifo_drop: 0
p0_ale_rate_limit_drop: 0
p0_ale_vid_ingress_drop: 0
p0_ale_da_eq_sa_drop: 0
p0_ale_block_drop: 0
p0_ale_secure_drop: 0
p0_ale_auth_drop: 0
p0_ale_unknown_ucast: 0
p0_ale_unknown_ucast_bytes: 0
p0_ale_unknown_mcast: 0
p0_ale_unknown_mcast_bytes: 0
p0_ale_unknown_bcast: 0
p0_ale_unknown_bcast_bytes: 0
p0_ale_pol_match: 0
p0_ale_pol_match_red: 0
p0_ale_pol_match_yellow: 0
p0_ale_mcast_sa_drop: 0
p0_ale_dual_vlan_drop: 0
p0_ale_len_err_drop: 0
p0_ale_ip_next_hdr_drop: 0
p0_ale_ipv4_frag_drop: 0
p0_tx_mem_protect_err: 0
p0_tx_pri0: 0
p0_tx_pri1: 0
p0_tx_pri2: 0
p0_tx_pri3: 0
p0_tx_pri4: 0
p0_tx_pri5: 0
p0_tx_pri6: 0
p0_tx_pri7: 0
p0_tx_pri0_bcnt: 0
p0_tx_pri1_bcnt: 0
p0_tx_pri2_bcnt: 0
p0_tx_pri3_bcnt: 0
p0_tx_pri4_bcnt: 0
p0_tx_pri5_bcnt: 0
p0_tx_pri6_bcnt: 0
p0_tx_pri7_bcnt: 0
p0_tx_pri0_drop: 0
p0_tx_pri1_drop: 0
p0_tx_pri2_drop: 0
p0_tx_pri3_drop: 0
p0_tx_pri4_drop: 0
p0_tx_pri5_drop: 0
p0_tx_pri6_drop: 0
p0_tx_pri7_drop: 0
p0_tx_pri0_drop_bcnt: 0
p0_tx_pri1_drop_bcnt: 0
p0_tx_pri2_drop_bcnt: 0
p0_tx_pri3_drop_bcnt: 0
p0_tx_pri4_drop_bcnt: 0
p0_tx_pri5_drop_bcnt: 0
p0_tx_pri6_drop_bcnt: 0
p0_tx_pri7_drop_bcnt: 0
rx_good_frames: 0
rx_broadcast_frames: 0
rx_multicast_frames: 0
rx_pause_frames: 0
rx_crc_errors: 0
rx_align_code_errors: 0
rx_oversized_frames: 0
rx_jabber_frames: 0
rx_undersized_frames: 0
rx_fragments: 0
ale_drop: 0
ale_overrun_drop: 0
rx_octets: 0
tx_good_frames: 88
tx_broadcast_frames: 77
tx_multicast_frames: 11
tx_pause_frames: 0
tx_deferred_frames: 0
tx_collision_frames: 0
tx_single_coll_frames: 0
tx_mult_coll_frames: 0
tx_excessive_collisions: 0
tx_late_collisions: 0
rx_ipg_error: 0
tx_carrier_sense_errors: 0
tx_octets: 5838
tx_64B_frames: 77
tx_65_to_127B_frames: 11
tx_128_to_255B_frames: 0
tx_256_to_511B_frames: 0
tx_512_to_1023B_frames: 0
tx_1024B_frames: 0
net_octets: 5838
rx_bottom_fifo_drop: 0
rx_port_mask_drop: 0
rx_top_fifo_drop: 0
ale_rate_limit_drop: 0
ale_vid_ingress_drop: 0
ale_da_eq_sa_drop: 0
ale_block_drop: 0
ale_secure_drop: 0
ale_auth_drop: 0
ale_unknown_ucast: 0
ale_unknown_ucast_bytes: 0
ale_unknown_mcast: 0
ale_unknown_mcast_bytes: 0
ale_unknown_bcast: 0
ale_unknown_bcast_bytes: 0
ale_pol_match: 0
ale_pol_match_red: 0
ale_pol_match_yellow: 0
ale_mcast_sa_drop: 0
ale_dual_vlan_drop: 0
ale_len_err_drop: 0
ale_ip_next_hdr_drop: 0
ale_ipv4_frag_drop: 0
iet_rx_assembly_err: 0
iet_rx_assembly_ok: 0
iet_rx_smd_err: 77
iet_rx_frag: 0
iet_tx_hold: 0
iet_tx_frag: 0
tx_mem_protect_err: 0
tx_pri0: 88
tx_pri1: 0
tx_pri2: 0
tx_pri3: 0
tx_pri4: 0
tx_pri5: 0
tx_pri6: 0
tx_pri7: 0
tx_pri0_bcnt: 5838
tx_pri1_bcnt: 0
tx_pri2_bcnt: 0
tx_pri3_bcnt: 0
tx_pri4_bcnt: 0
tx_pri5_bcnt: 0
tx_pri6_bcnt: 0
tx_pri7_bcnt: 0
tx_pri0_drop: 0
tx_pri1_drop: 0
tx_pri2_drop: 0
tx_pri3_drop: 0
tx_pri4_drop: 0
tx_pri5_drop: 0
tx_pri6_drop: 0
tx_pri7_drop: 0
tx_pri0_drop_bcnt: 0
tx_pri1_drop_bcnt: 0
tx_pri2_drop_bcnt: 0
tx_pri3_drop_bcnt: 0
tx_pri4_drop_bcnt: 0
tx_pri5_drop_bcnt: 0
tx_pri6_drop_bcnt: 0
tx_pri7_drop_bcnt: 0

0 Daolin Qiu 3 months ago in reply to Bert

TI__Genius 16305 points

Hi Bert,

Bert said:
switch----> am623 : am623 can not receive the frame from switch

As a sanity check can you check on the switch side if the frame successfully got sent out to the AM62x?

Can you also share the information requested at the end of this app note on Ethernet debug? https://www.ti.com/document-viewer/lit/html/SPRADJ8#GUID-4CE07F0F-B621-4759-A5F1-840452A03EF8/GUID-B068299B-24ED-4BF7-8C25-6DE622E3508F

Bert said:
below is the command "ethtool -S eth1" result , I found the item iet_rx_smd_err number is growing, please help analysis it

Below is the description of the IET Receive SMD Error statisic from the AM62x TRM. From my understanding, this statistic is relevant for IET (Interspersed Express Traffic) in TSN (time-sensitive networking, more so meant for real-time traffic.

12.3.1.4.6.18.6 IET Receive SMD Error (Offset = 3A148h)

The total number of received frames rejected due to an unknown SMD value or received frames rejected with an SMD-C when no frame is in progress.

any frame received
was any size, and
was rejected because of an unknown SMD value or SMD-C with no frame in progress.

Note: If IET_ENABLE is not set, this statistic counts any received frame with any non express SMD.

0 Bert 3 months ago in reply to Daolin Qiu

Intellectual 570 points

hi Daolin

As a sanity check can you check on the switch side if the frame successfully got sent out to the AM62x?

the switch chip don't have a statistics register for successful TX frames counts, but we use the oscilloscope to measure waveform for RMII data from switch to am623, it looks good

What is the TI SDK version in use?
LINUX SDK 10.01.10.04
What is the TI processor in use?
am623
Is the DUT a TI EVM or a custom board?
custom board

What is the test topology?

DUT(ping request)<------->switch<------->PC
DUT ping the PC, and PC can received correctly and reply , switch forward the ping reply to DUT, DUT can't received it

below is the device tree 、bootlog 、ethtool result

devicetree-bootlog.zip

0 Bert 3 months ago in reply to Daolin Qiu

Intellectual 570 points

about the " IET Receive SMD Error", could you explain more about it? what condition can cause the error?

0 Daolin Qiu 3 months ago in reply to Bert

TI__Genius 16305 points

Hi Bert,

Bert said:
about the " IET Receive SMD Error", could you explain more about it? what condition can cause the error?

I haven't seen IET Receive SMD Error counters go up before but upon some quick research my guess is that when the network is using IET to allow high-priority traffic to interrupt and over-take lower-priority traffic, typically when time-sensitive networking is a requirement, the Ethernet frames should have this "SMD" field. When the expected frames are preemption frames used in IET, and SMD is incorrect, I think this counter is supposed to increment. However, my assumption is that you are not using IET correct? If so, to me it doesn't quite make sense why this counter would even increment in the first place.

Bert said:
devicetree-bootlog.zip

A couple comments

1. I assume you are setting up cpsw_port2 to communicate with your ethernet switch. Is that correct?

2. In your DTS, your &cpsw_port2 node references phy-handle &cpsw3g_phy1 but as I understand you are trying to do MAC-to-MAC on this port so I'm confused why there still is a reference to &cpsw3g_phy1?

3. For the same reason as #2, in &cpsw3g_mdio, you define cpsw3g_phy1 with TI PHY DP83867 parameters. First, my understanding is that this port should be connected to your ethernet switch and not the DP83867. Second, are you communicating with your ethernet switch via MDIO or SPI?

&cpsw_port2 {
	phy-mode = "rmii";
	phy-handle = <&cpsw3g_phy1>;
	status = "okay";
	fixed-link {
		speed = <100>;
		full-duplex;
	};
};

&cpsw3g_mdio {
	cpsw3g_phy1: ethernet-phy@1 {
		reg = <1>;
	     /* ti,rx-internal-delay = <DP83867_RGMIIDCTL_2_00_NS>;
		ti,fifo-depth = <DP83867_PHYCR_FIFO_DEPTH_4_B_NIB>;
		ti,min-output-impedance; */
	};
};

Daolin Qiu said:
DTS configuration for the external switch should be determined from the switch vendor. Your ethernet switch IP175LLF is a non-TI part so I would suggest finding out what the expected DTS configuration is needed for the external switch

4. I see in your boot log the below message related to your ethernet switch which implies you have a Linux driver for your ethernet switch built into your kernel; however, in your DTS, I don't see a configuration set for your ethernet switch? You should reach out to your ethernet switch vendor for the DTS configuration for your ethernet switch, if the configuration is not upstreamed in a yaml file in the Linux mainline source tree.

[16:47:25:054] [    1.637472] davinci_mdio 8000f00.mdio: phy[1]: device 8000f00.mdio:01, driver ICPlus IP175C/D

-Daolin

0 Bert 3 months ago in reply to Daolin Qiu

Intellectual 570 points

hi Daolin

However, my assumption is that you are not using IET correct?

I don't do any operation for IET, not any code change for it, all is TI CPSW driver

1. I assume you are setting up cpsw_port2 to communicate with your ethernet switch. Is that correct?

yes ,I use cpsw_port2

cpsw3g_phy1 is for cpsw_port2, also have a cpsw3g_phy0 in device tree file , it is for cpsw_port1

cpsw3g_phy0 with TI PHY DP83867, not cpsw3g_phy1, and communicating with ethernet switch via MDIO

cpsw3g_phy0: ethernet-phy@0 {
bootph-all;
reg = <7>;
};

0 Bert 3 months ago in reply to Daolin Qiu

Intellectual 570 points

switch driver is very sample and the driver not use DTS, actually it is not take effect, because the driver function is to configure VALN and other functions, . , we don't use them, we only use the default configuration , and the ethernet switch vendor said it is ok to use the default configuration

0 Bert 3 months ago in reply to Daolin Qiu

Intellectual 570 points

do you have any other suggestion to investigate this issue in am623 MAC side? can I disable the IET， is it a must function for TI CPSW driver?

could you tell me what is the mean for "SMD" from am623 TRM, I could not find a document said the ethernet frame have the “SMD”

0 Daolin Qiu 3 months ago in reply to Bert

TI__Genius 16305 points

Hi Bert,

Bert said:
2. In your DTS, your &cpsw_port2 node references phy-handle &cpsw3g_phy1 but as I understand you are trying to do MAC-to-MAC on this port so I'm confused why there still is a reference to &cpsw3g_phy1?

cpsw3g_phy1 is for cpsw_port2, also have a cpsw3g_phy0 in device tree file , it is for cpsw_port1

If your intention is to use MAC-to-MAC, why is there a PHY definition involved? From my understand when using MAC-to-MAC you are connecting two processors' Ethernet interfaces directly together without a PHY in the middle. If a PHY is involved it would be MAC-to-PHY mode.

Bert said:
cpsw3g_phy0 with TI PHY DP83867, not cpsw3g_phy1, and communicating with ethernet switch via MDIO

Yes, however your cpsw3g_phy1 still references properties for DP83867. If cpsw3g_phy1 is not supposed to be using TI PHY DP83867, then the properties for DP83867 should be removed.

Bert said:
do you have any other suggestion to investigate this issue in am623 MAC side? can I disable the IET， is it a must function for TI CPSW driver?

IET is only meant for Time Sensitive Networking (TSN) applications. If your application will not require Time Sensitive Networking, more specifically IET (Interspersed Express Traffic) then it should technically not be relevant for your use case. It sounds to me TSN is a requirement for your use-case. If that is the case, I don't think IET is needed in your application. My current suggestion for you is to first verify if your DTS is correct for your MAC-to-MAC configuration; hence why I was asking about your DTS configurations to make sure it is correct.

You can see more information about IET here: https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/latest/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/Network/CPSW-IET.html

Bert said:
could you tell me what is the mean for "SMD" from am623 TRM, I could not find a document said the ethernet frame have the “SMD”

SMD is not specific to AM623 processor, it is simply part of an Ethernet frame that is used in IET for Time Sensitive Networking applications. For example if you look up "SMD Ethernet frame" you could see something like the below for example. Again, SMD is not TI specific, it is part of an IET Ethernet frame.

https://iebmedia.com/technology/tsn/tsn-technology-basics-of-ethernet-frame-preemption-part-2/

Bert said:
switch driver is very sample and the driver not use DTS, actually it is not take effect, because the driver function is to configure VALN and other functions, . , we don't use them, we only use the default configuration , and the ethernet switch vendor said it is ok to use the default configuration

Does your switch need to be recognized as a PHY? If so, that is quite a bit different from other switches that we have seen other customers use when configuring MAC-to-MAC.

-Daolin

Processors

Processors forum

AM623: AM623: MAC to MAC connection support