• State
  • Replies 22 replies
  • Subscribers 100 subscribers
  • Views 269 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

DRA821U: During CPSW2g communication, if you plug and unplug the cable in CPSWng, the interrupt may not be obtained.

user4852186
Intellectual 310 points
Part Number: DRA821U
Other Parts Discussed in Thread: J7200XSOMXEVM

Tool/software:

Hi Team,

We have designed a custom board based on the J7200XSOMXEVM reference design.

We are developing the software for the custom board with PROCESSOR-SDK-LINUX-RT-J7200 (10.00.07.03).

The custom board uses three Ethernet ports: CPSW2g (MCU-RMII1) and CPSWng (RGMII2, RGMII3).

CPSW2g is used for EtherCAT communication, and CPSWng performs general-purpose IP communication.

During CPSW2g communication, if you disconnect and connect the cable in CPSWng, a frame loss may occur in EtherCAT communication.

When we used ftrace to obtain trace data when frame loss occurred and when it did not, we found the following differences.

*image(left:frame loss occurred, right:frame loss not occurred)

Frame loss does not occur all the time, but occurs once every few times.

I plugged and unplugged the cable five times and attached the trace data when a frame lost occurred on the fifth attempt.

err_sched_switch.dat

Could this be because CPSW2g and CPSWng use the same driver (am65-cpsw-nuss.c)?
Is there a way to solve this?

Regards,

mizutani

  • 0
    TI__Guru 52120 points

    Hi,

    user4852186 said:
    Could this be because CPSW2g and CPSWng use the same driver (am65-cpsw-nuss.c)?

    It should not be because of same Driver used for CPSW2G and CPSWnG.

    Can you check CPSW statistics any errors during cable unplug and plug.

    Best Regards,
    Sudheer

  • 0 in reply to Doredla Sudheer Kumar
    Intellectual 310 points

    Hi

    Further investigation revealed that CPSW2g is an EtherCAT main device, and the issue seems to be caused by the frame transmission process not running. However, the reason why the process stops running when the CPSWng cable is unplugged and plugged back in remains unclear.

    The results of tracing the preemptirq event are shown below.

    *image(left:frame loss not occurred, right:frame loss occurred)

    Do these results tell us anything? And if so, how should we proceed with further investigation?

    Regards,

    mizutani

  • 0 in reply to user4852186
    TI__Guru 52120 points

    Hi,

    Have you observed any Link down for CPSW2G? If so, it can stop the transmission.
    Other wise no meaning of stop CPSW2G transmission process.

    Can you please share the Linux log with us while above observation.

    Best Regards,
    Sudheer

  • 0 in reply to Doredla Sudheer Kumar
    Intellectual 310 points

    Hi

    I recorded the kernel logs during the phenomenon. It appears that a link down of CPSW2G did not occur.

    Also, this phenomenon seems to occur both during link up and link down of CPSWng.

    The logs for each timing are shown below.

    * link up *

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    *** Start plugging and unplugging the cable ***
    Dec 26 11:44:59 Device kernel: am65-cpsw-nuss c000000.ethernet ethernet: Link is Down
    Dec 26 11:44:59 Device systemd-networkd[639]: ethernet: Lost carrier
    Dec 26 11:44:59 Device systemd-networkd[639]: ethernet: DHCP lease lost
    Dec 26 11:44:59 Device avahi-daemon[596]: Withdrawing address record for 192.168.22.192 on ethernet.
    Dec 26 11:44:59 Device my_app1[660]: [2024/12/26 11:44:59:3668] N: rops_handle_POLLIN_netlink: DELADDR
    Dec 26 11:44:59 Device avahi-daemon[596]: Leaving mDNS multicast group on interface ethernet.IPv4 with address 192.168.22.192.
    Dec 26 11:44:59 Device avahi-daemon[596]: Interface ethernet.IPv4 no longer relevant for mDNS.
    Dec 26 11:44:59 Device my_app2[644]: [2024/12/26 11:44:59:5158] N: rops_handle_POLLIN_netlink: DELADDR
    Dec 26 11:45:03 Device kernel: am65-cpsw-nuss c000000.ethernet ethernet: Link is Up - 1Gbps/Full - flow control rx/tx
    Dec 26 11:45:03 Device systemd-networkd[639]: ethernet: Gained carrier
    Dec 26 11:45:03 Device systemd-networkd[639]: ethernet: DHCPv4 address 192.168.22.192/24, gateway 192.168.22.1 acquired from 192.168.22.1
    Dec 26 11:45:03 Device avahi-daemon[596]: Joining mDNS multicast group on interface ethernet.IPv4 with address 192.168.22.192.
    Dec 26 11:45:03 Device avahi-daemon[596]: New relevant interface ethernet.IPv4 for mDNS.
    Dec 26 11:45:03 Device avahi-daemon[596]: Registering new address record for 192.168.22.192 on ethernet.IPv4.
    Dec 26 11:45:07 Device kernel: am65-cpsw-nuss c000000.ethernet ethernet: Link is Down
    Dec 26 11:45:07 Device systemd-networkd[639]: ethernet: Lost carrier
    Dec 26 11:45:07 Device avahi-daemon[596]: Withdrawing address record for 192.168.22.192 on ethernet.
    Dec 26 11:45:07 Device my_app1[660]: [2024/12/26 11:45:07:5598] N: rops_handle_POLLIN_netlink: DELADDR
    Dec 26 11:45:07 Device systemd-networkd[639]: ethernet: DHCP lease lost
    Dec 26 11:45:07 Device avahi-daemon[596]: Leaving mDNS multicast group on interface ethernet.IPv4 with address 192.168.22.192.
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    * link down *

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    *** Start plugging and unplugging the cable ***
    Dec 26 11:48:13 Device kernel: am65-cpsw-nuss c000000.ethernet ethernet: Link is Down
    Dec 26 11:48:13 Device systemd-networkd[639]: ethernet: Lost carrier
    Dec 26 11:48:13 Device systemd-networkd[639]: ethernet: DHCP lease lost
    Dec 26 11:48:13 Device avahi-daemon[596]: Withdrawing address record for 192.168.22.192 on ethernet.
    Dec 26 11:48:13 Device my_app1[660]: [2024/12/26 11:48:13:9323] N: rops_handle_POLLIN_netlink: DELADDR
    Dec 26 11:48:13 Device avahi-daemon[596]: Leaving mDNS multicast group on interface ethernet.IPv4 with address 192.168.22.192.
    Dec 26 11:48:13 Device avahi-daemon[596]: Interface ethernet.IPv4 no longer relevant for mDNS.
    Dec 26 11:48:14 Device my_app2[644]: [2024/12/26 11:48:14:0808] N: rops_handle_POLLIN_netlink: DELADDR
    Dec 26 11:48:20 Device kernel: am65-cpsw-nuss c000000.ethernet ethernet: Link is Up - 1Gbps/Full - flow control rx/tx
    Dec 26 11:48:20 Device systemd-networkd[639]: ethernet: Gained carrier
    Dec 26 11:48:20 Device systemd-networkd[639]: ethernet: DHCPv4 address 192.168.22.192/24, gateway 192.168.22.1 acquired from 192.168.22.1
    Dec 26 11:48:20 Device avahi-daemon[596]: Joining mDNS multicast group on interface ethernet.IPv4 with address 192.168.22.192.
    Dec 26 11:48:20 Device avahi-daemon[596]: New relevant interface ethernet.IPv4 for mDNS.
    Dec 26 11:48:20 Device avahi-daemon[596]: Registering new address record for 192.168.22.192 on ethernet.IPv4.
    Dec 26 11:48:24 Device kernel: am65-cpsw-nuss c000000.ethernet ethernet: Link is Down
    Dec 26 11:48:24 Device systemd-networkd[639]: ethernet: Lost carrier
    Dec 26 11:48:24 Device systemd-networkd[639]: ethernet: DHCP lease lost
    Dec 26 11:48:24 Device avahi-daemon[596]: Withdrawing address record for 192.168.22.192 on ethernet.
    Dec 26 11:48:24 Device my_app1[660]: [2024/12/26 11:48:24:1671] N: rops_handle_POLLIN_netlink: DELADDR
    Dec 26 11:48:24 Device avahi-daemon[596]: Leaving mDNS multicast group on interface ethernet.IPv4 with address 192.168.22.192.
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    Regards,

    mizutani

  • 0 in reply to user4852186
    TI__Guru 52120 points

    HI,

    From log it look like systemd services created for ethernet interface,

    What is is ethernet here, is it CPSWnG?
    networkd-639 ?
    Dec 26 11:44:59 Device kernel: am65-cpsw-nuss c000000.ethernet ethernet: Link is Down
    Dec 26 11:44:59 Device systemd-networkd[639]: ethernet: Lost carrier
    Dec 26 11:44:59 Device systemd-networkd[639]: ethernet: DHCP lease lost

    If you have same file service file both CPSW2G & CPSWnG, can you make it different files for each of them.

    Above looks like it is not related to driver of link issue, some services might be blocking transmission on CPSW2G.

    Best Regards,
    Sudheer

  • 0 in reply to Doredla Sudheer Kumar
    Intellectual 310 points

    Hi

    Doredla Sudheer Kumar said:
    What is is ethernet here, is it CPSWnG?

    That's right.

    Doredla Sudheer Kumar said:
    If you have same file service file both CPSW2G & CPSWnG, can you make it different files for each of them.

    Does this mean "/etc/systemd/network/*.network"?
    If so, you have different interfaces defined in different files.

    Doredla Sudheer Kumar said:
    Above looks like it is not related to driver of link issue, some services might be blocking transmission on CPSW2G.

    In our system, the following kernel’s command-line parameters are set so that the EtherCAT application occupies CPU1.

    "nohz_full=1 isolcpus=1 rcu_nocbs=1 rcu_nocb_poll irqaffinity=0 nosoftlockup"
    Also CPSW2G interrupt is assigned to CPU1 by smp_affinity.
    With these configurations, we believe the EtherCAT application is isolated from potential interference from interfaces other than CPSW2G. Is our understanding accurate?

    Regards,

    mizutani

  • 0 in reply to user4852186
    TI__Guru 52120 points

    Hi,

    user4852186 said:
    Doredla Sudheer Kumar said:
    If you have same file service file both CPSW2G & CPSWnG, can you make it different files for each of them.

    Does this mean "/etc/systemd/network/*.network"?
    If so, you have different interfaces defined in different files.

    yes, I mean maintain for different for CPSW2G and CPSWnG.

    user4852186 said:
    Doredla Sudheer Kumar said:
    Above looks like it is not related to driver of link issue, some services might be blocking transmission on CPSW2G.

    In our system, the following kernel’s command-line parameters are set so that the EtherCAT application occupies CPU1.

    "nohz_full=1 isolcpus=1 rcu_nocbs=1 rcu_nocb_poll irqaffinity=0 nosoftlockup"
    Also CPSW2G interrupt is assigned to CPU1 by smp_affinity.
    With these configurations, we believe the EtherCAT application is isolated from potential interference from interfaces other than CPSW2G. Is our understanding accurate?

    Can you please share all the . network and .netdev files used by systemd and also journalctl log in issue case vs non issue case.

    Best Regards,
    Sudheer

  • 0 in reply to Doredla Sudheer Kumar
    Intellectual 310 points

    Hi

    Attached below are all the .network files that systemd uses.

    7870.src.zip

    The journal log at the time the problem occurred has already been answered.

    user4852186 said:

    I recorded the kernel logs during the phenomenon. It appears that a link down of CPSW2G did not occur.

    Also, this phenomenon seems to occur both during link up and link down of CPSWng.

    The logs for each timing are shown below.

    These journal logs represent the period from when '~# journalctl -f' was initiated to when the EtherCAT communication failure occurred after repeatedly unplugging and plugging back in the cable.

    Also, the CPSW2G & CPSWnG logs at boot time are as follows.

    * CPSW2G *

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    [ 0.895145] davinci_mdio 46000f00.mdio: Configuring MDIO in manual mode
    [ 1.039804] davinci_mdio 46000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [ 1.249449] davinci_mdio 46000f00.mdio: phy[0]: device 46000f00.mdio:00, driver TI DP83822
    [ 1.249491] am65-cpsw-nuss 46000000.ethernet: initializing am65 cpsw nuss version 0x6BA02102, cpsw version 0x6BA82102 Ports: 2 quirks:00000000
    [ 1.249676] am65-cpsw-nuss 46000000.ethernet: initialized cpsw ale version 1.4
    [ 1.249679] am65-cpsw-nuss 46000000.ethernet: ALE Table size 64
    [ 2.231653] davinci_mdio 46000f00.mdio: Configuring MDIO in manual mode
    [ 2.265800] davinci_mdio 46000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [ 2.276619] davinci_mdio 46000f00.mdio: phy[0]: device 46000f00.mdio:00, driver TI DP83822
    [ 2.284935] am65-cpsw-nuss 46000000.ethernet: initializing am65 cpsw nuss version 0x6BA02102, cpsw version 0x6BA82102 Ports: 2 quirks:00000000
    [ 2.287967] am65-cpsw-nuss 46000000.ethernet: initialized cpsw ale version 1.4
    [ 2.287972] am65-cpsw-nuss 46000000.ethernet: ALE Table size 64
    [ 2.320505] am65-cpsw-nuss 46000000.ethernet: set new flow-id-base 48
    [ 7.069612] am65-cpsw-nuss 46000000.ethernet ethercat: renamed from eth0
    [ 11.832200] am65-cpsw-nuss 46000000.ethernet ethercat: PHY [46000f00.mdio:00] driver [TI DP83822] (irq=POLL)
    [ 11.841805] am65-cpsw-nuss 46000000.ethernet ethercat: configuring for phy/rmii link mode
    [ 13.920388] am65-cpsw-nuss 46000000.ethernet ethercat: Link is Up - 100Mbps/Full - flow control off
    [ 14.883914] am65-cpsw-nuss 46000000.ethernet ethercat: entered promiscuous mode
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    * CPSWnG  *

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    [ 1.534356] davinci_mdio c000f00.mdio: Configuring MDIO in manual mode
    [ 1.695804] davinci_mdio c000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [ 1.883913] davinci_mdio c000f00.mdio: phy[0]: device c000f00.mdio:00, driver TI DP83867
    [ 1.883948] am65-cpsw-nuss c000000.ethernet: initializing am65 cpsw nuss version 0x6BA02102, cpsw version 0x6BA82102 Ports: 5 quirks:00000000
    [ 1.912011] am65-cpsw-nuss c000000.ethernet: initialized cpsw ale version 1.4
    [ 1.912016] am65-cpsw-nuss c000000.ethernet: ALE Table size 512
    [ 2.332718] davinci_mdio c000f00.mdio: Configuring MDIO in manual mode
    [ 2.368804] davinci_mdio c000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [ 2.378959] davinci_mdio c000f00.mdio: phy[0]: device c000f00.mdio:00, driver TI DP83867
    [ 2.387081] am65-cpsw-nuss c000000.ethernet: initializing am65 cpsw nuss version 0x6BA02102, cpsw version 0x6BA82102 Ports: 5 quirks:00000000
    [ 2.400189] am65-cpsw-nuss c000000.ethernet: initialized cpsw ale version 1.4
    [ 2.404802] am65-cpsw-nuss c000000.ethernet: ALE Table size 512
    [ 2.423088] am65-cpsw-nuss c000000.ethernet: set new flow-id-base 60
    [ 6.961344] am65-cpsw-nuss c000000.ethernet PC: renamed from eth2
    [ 7.025114] am65-cpsw-nuss c000000.ethernet ethernet: renamed from eth1
    [ 11.926422] am65-cpsw-nuss c000000.ethernet ethernet: PHY [c000f00.mdio:00] driver [TI DP83867] (irq=POLL)
    [ 11.939825] am65-cpsw-nuss c000000.ethernet ethernet: configuring for phy/rgmii-rxid link mode
    [ 11.974333] am65-cpsw-nuss c000000.ethernet PC: configuring for fixed/rgmii link mode
    [ 11.983472] am65-cpsw-nuss c000000.ethernet PC: Link is Up - 1Gbps/Full - flow control off
    [ 16.033311] am65-cpsw-nuss c000000.ethernet ethernet: Link is Up - 1Gbps/Full - flow control rx/tx
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    Regards,

    mizutani

  • 0 in reply to user4852186
    TI__Guru 52120 points

    HI,

    Let me check the configuration files and logs and get back to you soon.

    Best Regards,
    Sudheer

  • 0 in reply to Doredla Sudheer Kumar
    TI__Guru 52120 points

    Hi,

    Systemd files are very simple and independent for each network interface.

    Let us check on our side with Linux-RT once, and update you in early next week.

    Best Regards,
    Sudheer

  • 0 in reply to Doredla Sudheer Kumar
    Intellectual 310 points

    Hi,

    Do you have any update on this issue?

    Regards,

    mizutani

  • 0 in reply to user4852186
    TI__Guru 52120 points

    Hi, 

    Sorry for the delay, due to other high priority activities I couldn't able to check this. 

    Will check in this week and update you as early as possible.

    Best Regards, 

    Sudherr

  • 0 in reply to Doredla Sudheer Kumar
    Intellectual 310 points

    Hi,

    I apologize for the repeated follow-up, but could you please provide an update on this matter?

    Regards,

  • 0 in reply to user4852186
    TI__Guru 52120 points

    Hi,

    Sorry for the delay.

    I have tested the RT-Link SDK by enabling both CPSW2G and CPSW9G from Linux. During the testing, I did not observe any traffic interruption on CPSW2G when disconnecting and reconnecting the cable to CPSW9G.

    I am running Iperf on the CPSW2G interface, and during this process, I am unplugging and reconnecting the CPSW9G Port connection.

    Best Regards,
    Sudheer

  • 0 in reply to Doredla Sudheer Kumar
    Intellectual 310 points

    Hi

    In our evaluation environment, EtherCAT communication is performed at 1 ms, and frame loss occurs when a cable is disconnected or reconnected.

    Looking at the kernel trace, the blocking time is only 3ms to 5ms, and then communication resumes.

    Doredla Sudheer Kumar said:
    I have tested the RT-Link SDK by enabling both CPSW2G and CPSW9G from Linux. During the testing, I did not observe any traffic interruption on CPSW2G when disconnecting and reconnecting the cable to CPSW9G.

    When you say no traffic interruptions, does that mean that even this slight blockage doesn't occur?

    Regards,

  • 0 in reply to user4852186
    TI__Guru 52120 points

    Hi

    user4852186 said:
    I have tested the RT-Link SDK by enabling both CPSW2G and CPSW9G from Linux. During the testing, I did not observe any traffic interruption on CPSW2G when disconnecting and reconnecting the cable to CPSW9G.

    When you say no traffic interruptions, does that mean that even this slight blockage doesn't occur?

    I haven't observed any drop in iperf data.

    user4852186 said:
    Looking at the kernel trace, the blocking time is only 3ms to 5ms, and then communication resumes.

    As delay could be very minimal, may not catch in my iperf data.
    Can you share how tracing has done, will check on our side.

    Best Regards,
    Sudheer

  • 0 in reply to Doredla Sudheer Kumar
    Intellectual 310 points

    Hi

    After executing the following command, I unplugged and plugged the LAN cable.

    Fullscreen
    1
    ~# trace-cmd record --file-version 6 -e sched_switch -o /tmp/sched_switch.dat
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    As explained before, this phenomenon does not always occur when the cable is unplugged or plugged in. It occurs once every few times.
    We determine the occurrence of the phenomenon by an EtherCAT communication error and stop the kernel trace.

    The intervals between unplugging and plugging in the cable are as follows.
    1. Unplug the cable
    2. "kernel: am65-cpsw-nuss c000000.ethernet ethernet: Link is Down" is output
    3. Plug in the cable
    4. "kernel: am65-cpsw-nuss c000000.ethernet ethernet: Link is Up" is output

    Regards,
    mizutani

  • 0 in reply to user4852186
    TI__Guru 52120 points

    Hi,

    user4852186 said:
    As explained before, this phenomenon does not always occur when the cable is unplugged or plugged in. It occurs once every few times.
    We determine the occurrence of the phenomenon by an EtherCAT communication error and stop the kernel trace.

    You mean issue is due to error in EtherCAT communication not because of cable unplug and reconnect?

    Best Regards,
    Sudheer

  • 0 in reply to Doredla Sudheer Kumar
    Intellectual 310 points

    Hi

    The problem is that an EtherCAT communication error occurs when the cable is unplugged and plugged back in.

    Doredla Sudheer Kumar said:
    user4852186 said:
    As explained before, this phenomenon does not always occur when the cable is unplugged or plugged in. It occurs once every few times.
    We determine the occurrence of the phenomenon by an EtherCAT communication error and stop the kernel trace.

    You mean issue is due to error in EtherCAT communication not because of cable unplug and reconnect?

    This means when to stop kernel tracing with 'Ctrl+C'.

    Regards,
    mizutani

  • 0 in reply to user4852186
    TI__Guru 52120 points

    Hi,

    user4852186 said:
    The problem is that an EtherCAT communication error occurs when the cable is unplugged and plugged back in.

    Is it an application right, are you trying to connect CPSW9G interface?
    Is PHY using for CPSW2G and CPSW9G are different? are they handled via independent MDIO interface? 

    Best Regards,
    Sudheer

  • 0 in reply to Doredla Sudheer Kumar
    Intellectual 310 points

    Hi

    Doredla Sudheer Kumar said:
    Is it an application right, are you trying to connect CPSW9G interface?

    CPSW2g is used for EtherCAT communication, and CPSWng performs general-purpose IP communication.

    During CPSW2g communication, if you disconnect and connect the cable in CPSWng, a frame loss may occur in EtherCAT communication.

    The process for EtherCAT communication and the process for Ethernet communication are independent.

    Doredla Sudheer Kumar said:
    Is PHY using for CPSW2G and CPSW9G are different? are they handled via independent MDIO interface? 

    CPSW2g uses DP83822 as PHY, and CPSWng uses DP83867 as PHY.

    CPSW2g uses MCU_MDIO0, and CPSWng uses MDIO0.

    Regards,
    mizutani

  • 0 in reply to user4852186
    TI__Guru 52120 points

    Hi,

    Thank you for sharing the details, we understood both CPSW2G and CPSWnG are running independent use cases and MDIO used for PHYs also different.

    Let me check internally about this.

    Best Regards,
    Sudheer