TMDS64GPEVM: Free RTOS / LwIP example dropping pings

Hi Joel,

Can you please check if HW statistics shows any dropped packets at the CPSW side?

(+) [FAQ] MCU-PLUS-SDK-AM243X: How do I get CPSW diagnostic statistics using debug gels in MCU+ SDK Enet LLD? - Arm-based microcontrollers forum - Arm-based microcontrollers - TI E2E support forums

Regards,

Prasad

Hi Jondhale,

Attempting to load cpsw_startup.gel resulted in the following error in the console window:

This is the target config I'm using:

Thanks,

Joel

Hi Joel,

Can you please remove existing gel files and then load start up gel?

Otherwise, you can just load the cpsw_statistics.gel file alone.

Regards,

Prasad

Hi Prasad,

Yes, I tried that and can now run the statistics print.  However, what software should I be running on R5_0_0? If I try loading and running enet_lwip_cpsw_am64x-evm_r5fss0-0_freertos_ti-arm-clang I get the following error printed in the console:

Please note that enet_lwip_cpsw_am64x-evm_r5fss0-0_freertos_ti-arm-clang is running as usual.

Thanks,

Joel

Joel,

Yes, CPSW example needs to be run before executing gels. Basically, you reproduce the issue of packet drops and then print the stats.

Jbec said:

I get the following error printed in the console:

You need to pause the R5 core before running the gel to avoid the error.

~Prasad

Hi Prasad,

Paused the core as suggested and get the following results:

====

MAIN_Cortex_R5_0_0: GEL Output: --->>> CPSW Gel Load In Progress... <<<---
MAIN_Cortex_R5_0_0: GEL Output: --->>> CPSW Gel Load DONE! <<<---
MAIN_Cortex_R5_0_0: GEL Output: STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: PORT0 STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: PORT1 STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------

Here is what I did:

1.  Terminate and remove all target configurations.

2.  Power cycle the AM64 EVM

3.  Start a debug session (launch the AM64EVM_XDS110.ccxml target configuration)

4.  Connect R5_0_0

5.  Remove all gel files

6.  Load cpsw_startup.gel

7.  Load enet_lwip_cpsw_am64x-evm_r5fss0-0_freertos_ti-arm-clang binary

8 Start execution (local IP reported as 192.168.0.175)

9. send 16 pings to 192.168.0.175

10. pause R5_0_0

11. Run the statistics print gel script

Results:

MAIN_Cortex_R5_0_0: GEL Output: STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: PORT0 STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: PORT1 STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------

So, what am I doing wrong??

Thanks,  Joel

Hi Joel,

If some of pings are succeeding, stats should be non-zero. Can you please let me know which option (CPSW2G, CPSW3G) you chose when running the stats print?

Regards,

Prasad

CPSW2G

Hi Prasad,

I tried your suggestion again.  Using CPSW2G stats print I still get zero stats:

MAIN_Cortex_R5_0_0: GEL Output: STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: PORT0 STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: PORT1 STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------

If I use CPSW3G I get this:

MAIN_Cortex_R5_0_0: GEL Output: STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: PORT0 STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_RXGOODFRAMES = 0x00000046
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_RXBROADCASTFRAMES = 0x0000000A
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_RXOCTETS = 0x000016E2
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_TXGOODFRAMES = 0x00000B5A
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_TXBROADCASTFRAMES = 0x000005C3
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_TXMULTICASTFRAMES = 0x0000051F
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_TXOCTETS = 0x0009387A
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_OCTETFRAMES64 = 0x000002F7
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_OCTETFRAMES65T127 = 0x000004A8
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_OCTETFRAMES128T255 = 0x00000151
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_OCTETFRAMES256T511 = 0x000001D9
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_OCTETFRAMES512T1023 = 0x0000007F
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_OCTETFRAMES1024TUP = 0x00000058
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_NETOCTETS = 0x00094F5C
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_TX_PRI_REG [0]= 0x00000B5A
MAIN_Cortex_R5_0_0: GEL Output: STAT_0_TX_PRI_BCNT_REG [0]= 0x0009387A
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: PORT1 STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_RXGOODFRAMES = 0x00000C2B
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_RXBROADCASTFRAMES = 0x00000694
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_RXMULTICASTFRAMES = 0x0000051F
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_ALE_DROP = 0x000000D1
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_RXOCTETS = 0x00096CBA
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_TXGOODFRAMES = 0x00000046
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_TXBROADCASTFRAMES = 0x0000000A
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_TXOCTETS = 0x000016E2
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_OCTETFRAMES64 = 0x000003C8
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_OCTETFRAMES65T127 = 0x000004A8
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_OCTETFRAMES128T255 = 0x00000151
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_OCTETFRAMES256T511 = 0x000001D9
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_OCTETFRAMES512T1023 = 0x0000007F
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_OCTETFRAMES1024TUP = 0x00000058
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_NETOCTETS = 0x0009839C
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_PORTMASK_DROP = 0x000000D1
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_ALE_UNKN_UNI = 0x00000012
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_ALE_UNKN_UNI_BCNT = 0x000004C6
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_ALE_UNKN_MLT = 0x00000076
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_ALE_UNKN_MLT_BCNT = 0x00005934
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_ALE_UNKN_BRD = 0x0000016F
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_ALE_UNKN_BRD_BCNT = 0x0000CF56
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_TX_PRI_REG [0]= 0x00000046
MAIN_Cortex_R5_0_0: GEL Output: STAT_1_TX_PRI_BCNT_REG [0]= 0x000016E2
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------
MAIN_Cortex_R5_0_0: GEL Output: PORT2 STATS
MAIN_Cortex_R5_0_0: GEL Output: --------------------------------

Hi Joel,

Using CPSW3G is correct.

Statistics shows there are some drops due to PORT mask and ALE drop. But these wouldn't necessarily cause ping drop. 

PORTMASK_DROP means that the packet was dropped because there was no port to forward the packet to, i.e. unicast/multicast not registered so this can be due to other stray packets PC stack keeps on generating.

With this, I believe issue can be on PC side as well. Can you please try 1. some other PC (is possible Linux) 2. try direct connection (without switch or router in between).

Regards,

Prasad

Hi Prasad,

Tried direct connect -- No change (PC is Win 10 and connection is 100GB)

Tried another PC:  Sent 512 pings with no errors.  However, this PC is WIN7 and only 100mB capable so I'm not sure if speed is the issue.

Your thoughts?

Thanks, Joel

Hi again,

Tried pinging other devices on Win10 1GB network while enet_lwip_cpsw_am64x-evm example is also executing on the same network.  Pings are successful 100% on other devices.  Here is an example:

Ping statistics for 192.168.0.120:
Packets: Sent = 1024, Received = 1024, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 1ms, Average = 0ms

Joel

Hello Joel,

Thanks for multiple tests and providing all these data points.

Seems there is packet drop on PHY side due to some timing issue in 1Gbps mode. This is confirmed by 100Mbps test where there is no packet drop.

I will check with SW team if they see any such packets drops. I assume you are using MCU+ SDK8.1 and AM64x EVM.

Regards,

Prasad

Hi Prasad,

Yes, I am using mcu_plus_sdk_am64x_08_01_00_36 and also using AM64x EVM.

Is there a way to limit connection speed in the SDK?  I tried editing test_enet.c like this:

mii->layerType = ethPort.mii.layerType;
mii->sublayerType = ethPort.mii.sublayerType;
mii->variantType = ENET_MAC_VARIANT_FORCED;
// linkCfg->speed = ENET_SPEED_AUTO; // Original
linkCfg->speed = ENET_SPEED_100MBIT; //JB
linkCfg->duplexity = ENET_DUPLEX_AUTO;

if (Enet_isCpswFamily(ENET_CPSW_3G))

This had no effect on the connection speed.  Still 1GB.

Thanks, Joel

Hi Again,

I put an ethernet switch between the PC (Win 10) and the EVM.  I then logged into the switch to gather statistics on bad packets.  I then tried running the demo software (enet_lwip_cpsw_am64x-evm) and sending pings to the EVM at the same time.  Please note that the PC is on Port 7, the EVM is on Port 1 and the rest of the network is on Port 8.

Here are the results:

Conclusions:

There were 32 pings sent, 12 pings were lost.  There were 13 CRC error packets between the switch and the EVM and 0 errors between the PC and the switch.  Also, the rest of the network on port 8 experienced zero errors.  So, PC to Switch link is good, Switch to rest of network is good therefore, both the PC and the switch are operating correctly. I believe that leaves the problem with the EVM hardware or SDK software.

Thanks, Joel

Hi Joel,

Thanks for detailed analysis. Helps us isolate the issue.

Just wondering - how is switch able to show CRC errors when packets are sent from it (Switch to EVM) as it has no way to find what happened in EVM, isn't it?

Regards,

Prasad

Hi Prasad,

I think the switch only monitors what it receives.  It would make no sense for it to generate a bad CRC and then send it or even to resend a bad packet.  So, my suspicions  are that the switch is checking the CRC of each packet it receives.  It receives no bad packets from the PC on port 7 and so the PC is not the problem.  It does see CRC errors on the packets it receives from the EVM on port 1.

Hi Joel.

SW team has found some issues in PHY tuning which we believe is causing packet drops at 1Gbps. We are debugging this with our HW apps experts.

I will get the patch to you by tomorrow for enabling 100Mbps till we fix the 1Gbps issue.

Regards,

Prasad

Thanks

Hi Joel,

Our SW team was able to root cause packet drop issue and find the delay values which fixes it.

Can you please try below change in examples/networking/lwip/enet_lwip_cpsw/am64x-evm/r5fss0-0_freertos/board.c?

Regards,

Prasad