• State TI Thinks Resolved
  • Locked Locked
  • Replies 14 replies
  • Answers 1 answer
  • Subscribers 99 subscribers
  • Views 677 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM625: unexpected Ethernet interrupt result in system loading

Tony Tang
Mastermind 29402 points
Part Number: AM625


During test found some board running slowly. There is no abnormal in system log, further investigate found there are many unexpected Ethernet interrupt as below two snapshot of cat /proc/interrupts. the Ethernet cable is not connected during booting. . 

Now test 3 pcs boards with Ethernet cable disconnected during boot, 1 board has this issue.

when issue reproduced, connect Ethernet cable, system resume to normal.

On the issue board, power cycle many times can reproduce the issue.

Below log from good board.

the log is not same as errata specified in user guide.

https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/08_04_01_09/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/Network/CPSW3g.html#errata-i2329-mdio-interface-corruption-cpsw-and-pruss

Please help analysis.

  • 0
    TI__Mastermind 40420 points

    Hi,

    Could you please provide more detail on how the system if running slowly? Are you saying that when the ethernet cable is not connected you are getting a lot of interrupts?

    What is the number that is being subtracted from to get to the 175009 number. What do the two screens captures represent with the math?

    Is the log captures from the good board when the cable is plugged in?

    Best Regards,

    Schuyler

  • 0 in reply to Schuyler Patton
    TI__Mastermind 29402 points
    Schuyler Patton said:
    Could you please provide more detail on how the system if running slowly?

    even can't launch application when so frequency interrupt triggered.

    Schuyler Patton said:
    Are you saying that when the ethernet cable is not connected you are getting a lot of interrupts?

    Yes.

    Schuyler Patton said:
    What is the number that is being subtracted from to get to the 175009 number. What do the two screens captures represent with the math?

    captured log every 500ms, calculate how many times interrupt happened every 1 second.

    Schuyler Patton said:
    Is the log captures from the good board when the cable is plugged in?

    when issue not happened, without ethernet cable connected.

  • 0 in reply to Tony Tang
    TI__Mastermind 29402 points

    Schuyler,

    We found a way to easy reproduce. after boot up, 

    root@am62xx-evm:~# ifconfig eth0 down

    root@am62xx-evm:~# ifconfig eth1 down

    root@am62xx-evm:~# ifconfig eth0 up

    root@am62xx-evm:~# cat /proc/interrupts

    root@am62xx-evm:~# cat /proc/interrupts

    observer Ethernet interrupt by key in cat /proc/interrupts, we can see interrupt number increased frequency.

    214:   33228090          0          0          0  MSI-INTA 1971731 Level     8000000.ethernet

    if not reproduce, do more times  ifconfig eth0 down/up, then cat /proc/interrupts. very easy to reproduce. 

    after reproduced, connect Ethernet Cable to Eth0,  cat /proc/interrupts again, interrupt number won't increase frequently and back to normal.

    Tried on PSDK 8.03, PSDK 8.05, and AM62x-SK E1, E2, no difference, all easy to reproduce.

  • 0 in reply to Tony Tang
    TI__Mastermind 40420 points

    Hi Tony,

    Is it possible to provide a screen capture of htop running in both the high interrupt and what is considered normal interrupt conditions?

    htop will display the core loading and top processes. 

    Best Regards,

    Schuyler

  • 0 in reply to Schuyler Patton
    TI__Mastermind 40420 points

    Hi Tony,

    Some additional requests, please also run ethtool -S eth0 also. And please attach (not cut and paste) the complete boot log. Is there a reboot happening between the ifconfig commands?

    And you mentioned this is occurring on more one EVM?

    Best Regards,

    Schuyler

  • 0 in reply to Schuyler Patton
    TI__Mastermind 29402 points

    Schuyler,

    The Steps to reproduce is very easy, if you have AM62x-sk board, you can reproduce it right away.

    boot up.

    ethe down,

    eth up,

    cat 

    if interrupt count doesn't increase frequenty, do upper again no need power cycle the board, 2 or 3 round, should be able to reproduce. 

  • 0 in reply to Tony Tang
    TI__Mastermind 40420 points

    Hi Tony,

    I performed the test using the latest SDK I am not able to see the interrupt condition you are describing. Attached is the test run that I performed. Please review and let me know if there is something I missed in performing the test. 

    Best Regards,

    Schuyler

    Fullscreen testing_for_interrupt_condition.txt Download 
    Starting after a boot and have set both ethernet interfaces down.
root@am62xx-evm:~# ifconfig eth0 down
root@am62xx-evm:~# ifconfig eth0 up
[ 8864.613564] am65-cpsw-nuss 8000000.ethernet: down msc_sl f0000000 tmo 0
[ 8864.623482] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[ 8865.749654] omap_i2c 20000000.i2c: controller timed out
[ 8866.109145] tps6598x 0-003f: tps6598x_handle_interrupt_status: failed to read events
[ 8866.112633] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=POLL)
[ 8866.126263] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
root@am62xx-evm:~# cat /proc/interrupts | grep  8000000.ethernet
134:          2          0          0          0  MSI-INTA 1970707 Level     8000000.ethernet-tx0
224:          3          0          0          0  MSI-INTA 1971731 Level     8000000.ethernet
257:          0          0          0          0     GICv3 134 Level     8000000.ethernet
root@am62xx-evm:~# ifconfig eth0 down
root@am62xx-evm:~# ifconfig eth0 up
[ 8927.589621] am65-cpsw-nuss 8000000.ethernet: down msc_sl f0000000 tmo 0
[ 8927.600855] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[ 8927.614592] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=POLL)
[ 8927.623972] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
root@am62xx-evm:~#
root@am62xx-evm:~# cat /proc/interrupts | grep  8000000.ethernet
134:          3          0          0          0  MSI-INTA 1970707 Level     8000000.ethernet-tx0
224:          4          0          0          0  MSI-INTA 1971731 Level     8000000.ethernet
257:          0          0          0          0     GICv3 134 Level     8000000.ethernet
root@am62xx-evm:~# ifconfig eth0 down
root@am62xx-evm:~# ifconfig eth0 up
[ 8946.221565] am65-cpsw-nuss 8000000.ethernet: down msc_sl f0000000 tmo 0
[ 8946.232507] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[ 8946.246200] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=POLL)
[ 8946.255538] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
root@am62xx-evm:~#
root@am62xx-evm:~# cat /proc/interrupts | grep  8000000.ethernet
134:          4          0          0          0  MSI-INTA 1970707 Level     8000000.ethernet-tx0
224:          5          0          0          0  MSI-INTA 1971731 Level     8000000.ethernet
257:          0          0          0          0     GICv3 134 Level     8000000.ethernet
root@am62xx-evm:~# ifconfig eth0 down
root@am62xx-evm:~# ifconfig eth0 up
[ 8965.893574] am65-cpsw-nuss 8000000.ethernet: down msc_sl f0000000 tmo 0
[ 8965.904742] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[ 8965.918475] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=POLL)
[ 8965.927804] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
root@am62xx-evm:~#
root@am62xx-evm:~# cat /proc/interrupts | grep  8000000.ethernet
134:          5          0          0          0  MSI-INTA 1970707 Level     8000000.ethernet-tx0
224:          6          0          0          0  MSI-INTA 1971731 Level     8000000.ethernet
257:          0          0          0          0     GICv3 134 Level     8000000.ethernet
root@am62xx-evm:~# ifconfig eth0 down
root@am62xx-evm:~# ifconfig eth0 up
[ 8978.317602] am65-cpsw-nuss 8000000.ethernet: down msc_sl f0000000 tmo 0
[ 8978.328447] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[ 8978.342184] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=POLL)
[ 8978.351508] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
root@am62xx-evm:~#
root@am62xx-evm:~# ifconfig eth0 down
root@am62xx-evm:~# ifconfig eth0 up
[ 8990.085506] am65-cpsw-nuss 8000000.ethernet: down msc_sl f0000000 tmo 0
[ 8990.095969] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[ 8990.109712] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=POLL)
[ 8990.119083] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
root@am62xx-evm:~#
root@am62xx-evm:~# cat /proc/interrupts | grep  8000000.ethernet
134:          7          0          0          0  MSI-INTA 1970707 Level     8000000.ethernet-tx0
224:          8          0          0          0  MSI-INTA 1971731 Level     8000000.ethernet
257:          0          0          0          0     GICv3 134 Level     8000000.ethernet
root@am62xx-evm:~# uname -a
Linux am62xx-evm 5.10.153-g3eee621d16 #1 SMP PREEMPT Wed Dec 14 11:54:42 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

root@am62xx-evm:~# ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500  metric 1
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:29:42:b3:1e  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500  metric 1
        ether f4:84:4c:fc:1f:94  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4098<BROADCAST,MULTICAST>  mtu 1500  metric 1
        ether 70:ff:76:1e:a3:26  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536  metric 1
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 82  bytes 6220 (6.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 82  bytes 6220 (6.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500  metric 1
        ether 20:d7:78:b7:2e:f0  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500  metric 1
        inet 192.168.43.1  netmask 255.255.255.0  broadcast 192.168.43.255
        inet6 fe80::22d7:78ff:feb7:2ef1  prefixlen 64  scopeid 0x20<link>
        ether 20:d7:78:b7:2e:f1  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 48  bytes 6402 (6.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am62xx-evm:~#

  • 0 in reply to Schuyler Patton
    TI__Expert 6565 points

    Hi Patton:

    Could you please take a picture about your EVM board. We can double confirm with our side AM62-SK EVM board hardware version.

    You are using AM62 linux version is 08.05.00.21?

    Tony has try verify it at RDK8.03 and latest 8.05 all easy reproduce it. When he come back office will contact you about reproduce method. Believe at our hands EVM two version RDK all can find the same problem.

    Best Regards!

    Han Tao

  • 0 in reply to Tao Han
    TI__Mastermind 40420 points

    Hi, 

    Since I was not able to reproduce the issue on the current SDK I tried an internal one that is currently going through testing. I am able to reproduce the issue on this version. Please allow us a few days to work on finding the cause of this condition.

    Best Regards,

    Schuyler

  • 0 in reply to Schuyler Patton
    TI__Expert 6565 points

    Hi Patton:

    Sorry do not introduce very clear about how to reproduce it.

    Reproduce the issue the key point is not input the ethernet cable. I has try at below version EVM board with default RDK. It can find this problem.

    Maybe at this state PHY is not work, it is AM62 trigger wrong state at linux.

    Thanks a lot!

    Han Tao

    Fullscreen AM62_eth_issue.txt Download 
            which

If you do not wish to distribute GPLv3 components please remove
the above packages prior to distribution.  This can be done using
the opkg remove command.  i.e.:
    opkg remove <package>
Where <package> is the name printed in the list above

NOTE: If the package is a dependency of another package you
      will be notified of the dependent packages.  You should
      use the --force-removal-of-dependent-packages option to
      also remove the dependent packages as well
***************************************************************
***************************************************************
[  OK  ] Started weston.service.
[  OK  ] Started Print notice about GPLv3 packages.
         Starting telnetd.service...
[  OK  ] Started telnetd.service.
         Starting hmi_demo.service...
[   13.644558] PVR_K:  1090: RGX Firmware image 'rgx.fw.33.15.11.3' loaded
[   13.680356] PVR_K:  1090: Shader binary image 'rgx.sh.33.15.11.3' loaded
[  OK  ] Started hmi_demo.service.
         Stopping hmi_demo.service...
[  OK  ] Stopped hmi_demo.service.

 _____                    _____           _         _
|  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
|     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
|__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
              |___|                    |___|

Arago Project http://arago-project.org am62xx-evm ttyS2

Arago 2021.09 am62xx-evm ttyS2

am62xx-evm login:
am62xx-evm login: root
root@am62xx-evm:~# cat /proc/interrupts | grep  8000000.ethernet
124:          0          0          0          0  MSI-INTA 1970707 Level     8000000.ethernet-tx0
214:          0          0          0          0  MSI-INTA 1971731 Level     8000000.ethernet
247:          0          0          0          0     GICv3 134 Level     8000000.ethernet
root@am62xx-evm:~# ifconfig eth0 down
root@am62xx-evm:~# ifconfig eth1 down
root@am62xx-evm:~# ifconfig eth0 up
[   58.873942] TI DP83867 8000f00.mdio:00: attached PHY driver [TI DP83867] (mii_bus:phy_addr=8000f00.mdio:00, irq=POLL)
[   58.894072] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
root@am62xx-evm:~# cat /proc/interrupts | grep  8000000.ethernet
124:          1          0          0          0  MSI-INTA 1970707 Level     8000000.ethernet-tx0
214:    1071912          0          0          0  MSI-INTA 1971731 Level     8000000.ethernet
247:          0          0          0          0     GICv3 134 Level     8000000.ethernet
root@am62xx-evm:~#

  • 0 in reply to Tao Han
    TI__Expert 6565 points

    HI Patton:

    I write script accelerate this problem. When you input cable 1000 circus i do not find problem.

    When you do not input cable. first run

    #ifconfig eth0 down

    #ifconfig eth1 down

    then run those script. It will be easy reproduce it.

    Best Regards!

    Han Tao 

    Fullscreen test_net.txt Download 
    #!/bin/bash
a=1000

until [ $a -le 0 ]
do
ifconfig eth0 down
sleep 1
ifconfig eth0 up
sleep 6
cat /proc/interrupts | grep  8000000.ethernet
echo "a = " $a
((a--))
done

  • 0 in reply to Tao Han
    TI__Mastermind 40420 points

    Hi,

    Thank you for the script. I will add this script to the bug report filed with development team to assist them in re-creating the issue so it can be debugged.

    Best Regards,

    Schuyler

  • 0 in reply to Schuyler Patton
    TI__Expert 6565 points

    Hi team:

    Just get update about it. Maybe can try below modify to resolve this problem.

    #####

    adding a delay to wait for the completion of rx DMA channel teardown fixes the issue.

    In the am65_cpsw_nuss_common_stop() function, adding a delay after:
    k3_udma_glue_tdown_rx_chn(common->rx_chns.rx_chn, true);
    fixes the issue.

    While there is a check for tx channel teardown completion in am65_cpsw_nuss_common_stop(),
    there is no check for rx channel teardown completion and it needs to be added.

    #####

    k3_udma_glue_tdown_rx_chn(common->rx_chns.rx_chn, true);
    i = wait_for_completion_timeout(&common->tdown_complete,
					msecs_to_jiffies(1000)); 
    after the call to:
    k3_udma_glue_tdown_rx_chn(common->rx_chns.rx_chn, true);
    similar to how it is done after:
    k3_udma_glue_tdown_tx_chn(common->tx_chns[i].tx_chn, false);
    Best regards!
    Han Tao
  • 0 in reply to Tao Han
    TI__Expert 6565 points

    Hi Team:

    Customer has verified those modify and insure it works at system.

    Best Regards!

    Han Tao