TMDSCNCD263: AM2634 - Ethernet drops rx TCP frames while reception.

Hi Hari,

Are you working with the MCU_PLUS_SDK or the MCAL_SDK?

Regards,
Shaunak

Hi Shaunak,

I am working with MCAL.

Kind regards,

Hari

Hi Hari,

1. The TCP frames being sent are not malformed right? Can you please use the gel scripts and share the CPSW stats? Once you have your application running and verifying that TCP is not working, halt your core and go to Scripts -> CPSW Stats and run the CPSW3G nonzero stats gel scripts. Please share these stats so we can understand the issue.

2. If my understanding is correct, the 192.168.88.77 is your External PC and 192.168.88.73 is the AM263x? If yes, the MCU did send back an ACK for TCP packet. Can you please make sure the external world PC will not drop this packet due to some firewall settings?

Regards,
Shaunak

Hi Shaunak,

1. The TCP frames being sent are not malformed right? Can you please use the gel scripts and share the CPSW stats? Once you have your application running and verifying that TCP is not working, halt your core and go to Scripts -> CPSW Stats and run the CPSW3G nonzero stats gel scripts. Please share these stats so we can understand the issue.

-> No, tcp frames are not malformed.

-> Is there any specific function, from there is have to read STATS?

Getting empty STATS, when brake execution the execution

From variable:

*When i kept brake point in TcpIp_RxIndication(), i'm able to see Tcp frame reception.

Is it some timing issue in Eth or EthTrcv?  

2. If my understanding is correct, the 192.168.88.77 is your External PC and 192.168.88.73 is the AM263x? If yes, the MCU did send back an ACK for TCP packet. Can you please make sure the external world PC will not drop this packet due to some firewall settings?

->Yes, 192.168.88.77/71 is  External PC and 192.168.88.73 is the AM263x?

->Not sure about firewall block, i will update you.

->where as,  when i kept brake point in TcpIp_RxIndication(), i'm able to see Tcp frame reception.

Kind regards,

Hari

Hi Hari,

1. From the stats, I see there are packets drops (ALE_DROP). Can you please verify that the MAC address is added in the ALE table? Can you use the gel scripts and share the dump of the ALE table as well?

2. From the stats I also see Rx count incrementing. My observation is that you try to establish TCP connection, the first SYN packet is sent from the external world to the MCU, the MCU responded with the ACK (i can see in your wireshark snippet), but then the other TCP packets seem to be dropped by the ALE which should not be ideally the case. Are you also parallelly having any other data stream (confirming that the dropped packets are not some unexpected packets but rather the TCP packets)

Regards,
Shaunak

Hi Shaunak,

1. From the stats, I see there are packets drops (ALE_DROP). Can you please verify that the MAC address is added in the ALE table? Can you use the gel scripts and share the dump of the ALE table as well?

-> It returns empty

-> At what point of time i have to read it.

-> From where i can check it in source code to confirm ALE table?

2. From the stats I also see Rx count incrementing. My observation is that you try to establish TCP connection, the first SYN packet is sent from the external world to the MCU, the MCU responded with the ACK (i can see in your wireshark snippet), but then the other TCP packets seem to be dropped by the ALE which should not be ideally the case. Are you also parallelly having any other data stream (confirming that the dropped packets are not some unexpected packets but rather the TCP packets)

-> In actual use case, there is no ACK from AM2634 

-> This only to check and not actual use case

Above is ok for the following request, ncat -t 192.168.88.73 13400 -p 60020

Hi Hari,

Please make sure you are using the latest CCS version as well. It is very weird that the ALE and the CPSW stats both return empty. Seems like some setup issue.

Please halt the core once you try to establish the TCP connection and it fails. Then run the gel scripts.

Hari krishna Kaki said:

-> This only to check and not actual use case

I did not understand this. Can you please explain what exactly is "actual" case vs the working case (ncat)

Is the above screenshot attached (in which I see ACK packet) not "actual" use case?

Regards,
Shaunak

Hi Shaunak,

Actual use case:

Following sequence of requests are sent from PC to AM2634

-> ARP, ICMP, DoIP(UDP) and Tcp.

-> ARP - Acknowledged.

-> ICMP(Ping) - No Response.

-> DoIP(UDP) - Responded.

-> Tcp - No ack and No connection establishment.

Use case to check(ncat):

Only one TCP request sent ncat -t 192.168.88.73 13400 -p 60020

Hi Hari,

In the second case 

Hari krishna Kaki said:

Use case to check(ncat):

The response is seen from the MCU in this case correct?

Hari krishna Kaki said:

Actual use case:

In the actual case, can you explain how are the packets being sent? If there was an issue with the driver, the NCAT connection shouldn't have succeeded right? Do you get a NCAT terminal connection established log?

It might also be possible that the issue is with the upper TCP/IP stack. If the issue was with the driver my assumption is the UDP/ARP packets would be dropped as well. I might be wrong but I will have to study this further.

Meanwhile, can you further investigate the ALE drop frames. Can you check stats after every transmission? (First send ICMP, check stats, then send TCP and again check stats, then send ARP and check stats)

Regards,
Shaunak

Hi Shaunak,

For every received Tcp frame, AM2634 is responding correctly.

I have added a code in Eth driver to check Tcp frame reception.

There is no issue in Tcp stack, actually Tcp frame is not reached Eth driver.

I will try to get status for every transmission of TCMP, TCP.

Hi Shaunak,

I have tried as you suggested

Meanwhile, can you further investigate the ALE drop frames. Can you check stats after every transmission? (First send ICMP, check stats, then send TCP and again check stats, then send ARP and check stats)

Stats when Eth is reached TcpIp_IpV4_rxIndicationIcmp() function - ICMP frame

Stats when Eth is reached TcpIp_Tcp_rxIndication() function -TCP frame

Stats when Eth is reached TcpIp_Udp_rxIndication() function - ARP frame

Not receiving, when sending in a sequence ARP - ICMP - UDP -TCP.

ICMP and TCP are not responding.

*When i kept brake point in TcpIp_RxIndication(), i'm able to see ICMP,  TCP frame reception & response.

As per my understanding, there is some Sync/timing issue.

Could you please let me know, where i'm missing.

Kind regards,

Hari

Hi Hari,

1. From the stats shared above, I don't see any drops, so we are sure CPSW is not dropping any frames. After CPSW, the MCAL Eth driver will get the frames. If you check out the MCAL Eth demo, we have "gAppObj.stats" to show the stats. Will it be possible for you to check these stats and see if there are any overflows or other errors?

2. Please correct me if I'm wrong, the flow here is
    Packets from external world -> CPSW -> MCAL Eth driver -> Your upper layer TCP/IP stack
    Regardless of the frame type (TCP/UDP/ARP/ICMP), the CPSW and MCAL Eth driver will not process them differently but just do the basic required processing and forward it to the upper layer stack

3. Im not sure if this is a timing issue. I will let other expert comment about this. But again, my understanding is, if it was a timing issue, UDP and ARP should also have issues.

Hari krishna Kaki said:

As per my understanding, there is some Sync/timing issue.

Regards,
Shaunak

Hi Shaunak,

All types of frames are getting received, only when frames are sending individually not in sequence.

I have added a code in Eth drive to check frame reception in driver level as shown below

These brake points are hitting, only when frames are sending individually not in sequence.

*Even above brake points are not hitting for not all individually requests, some times Eth driver is dropping frames.

Packets from external world -> CPSW -> MCAL Eth driver -> Your upper layer TCP/IP stack

-> MCAL Eth driver -> EthIf -> Your upper layer TCP/IP stack

Actual required sequence

Above brake points are not hitting, for the above sequence.

Kind regards,

Hari

Hi Hari,

Hari krishna Kaki said:

All types of frames are getting received, only when frames are sending individually not in sequence.

Thanks for confirming this. Now I understand the issue better. 

I need some time to investigate this. I will get back to you soon on what we could debug further.

Regards,
Shaunak

Hi Hari,

What is the packet rate at which you are sending the packets?

Regards,

Shaunak

Hi Shaunak,

I am not sure about exact rate of transmission, Seems it is 10 - 100ms from Wireshark trace.

We are using customized tool, developed internally.

Is there any limitation on packet rate?

I have tested Eth configuration in Polling & Interrupt modes.

Still not able to receive the frames.

No interrupt triggered for ICMP/Tcp frames.

Hi Hari,

Let me run some tests on my end. Looks like the driver is choking with burst traffic maybe because the traffic bursts are reaching a situation which the driver can't handle.

Can you please share the exact packet transmission rate? Is it 1 packet every 10-100ms? or a burst of packets? And also does increasing time interval between packets help?

Regards,

Shaunak

Hi Shaunak,

There is no chance of traffic burst.

Maximum of 1sec between frames, and minimum based on reply.

If a frame is acknowledged, then next frame will be transmitted other wise next frames is at 1sec.

Hi Hari

It is a bit difficult for me to recreate since we don't have the full AUTOSAR stack. Just testing at layer-2 does not show me this issue when I send 50,000 packet intervals of 10ms. This is because the L2 driver will process all the packets in the same way.

For timing and EthTrcv configurations, you can refer the out of box Eth demo. I tested with the same. I can help with CPSW related debug pointers but im not sure about the TCP/IP debug from the AutoSAR stack point of view.

1. To check if the DMA status, you can check "0x52834024" to get the DMA status. If it is "80000000" it means it is in idle state. If any other value, it will indicate that the DMA is blocked.

2. Check CPSW stats for Drops and Overruns (after you have sent all your packets in sequence and not getting response from the board). Do this after sending all the packets in the sequence you mentioned previously.

3. Check the FIFO queue for drops: 0x5283A04C (Tx SOF overrun), 0x5283A08C(Rx bottom of FIFO drop) (both should be zero)

4. Are there any logs or debug stats in the TCP/IP implementation of the AutoSAR stack? I recommend checking that as well once.

Regards,
Shaunak

Hi Shaunak,

ICMP/TCP frames are not received by Eth.

Is it possible to have a debug session tomorrow(16.05.2024) or Friday or ?

We can check issue on a full stack.

Kind regards,

Hari

Hi Hari,

Sure, Can you communicate your free time slots over email? We can have a call on 17th (Friday) May, 2024.

Regards,

Shaunak

Hi Hari,

As per the debug call we had, we are inclined towards the PortMask and ALE configuration issue.

1. To further test, we can try bypassing the ALE.

Can you add this to your code:

Hi Shaunak,

From which function should be called CPSWAleBypassEnable(Eth_DrvObj.baseAddr, (uint32) TRUE);?

Hi,

You can call it after CpswPort_hostPortopen(); before CPSWAleInit().

This should be happening in EthCpswInstInit() function.

Regards,
Shaunak

Hi Shaunak,

Workaround CPSWAleBypassEnable(Eth_DrvObj.baseAddr, (uint32) TRUE); is working and able to receive all frames.

I think, now you got the actual issue. Please let me know Solution.

Hi Hari,

There are multiple reasons for PORTMASK_DROP != 0, but the most likely ones are:

1. DA of the packet is a unicast not in the ALE table (i.e. not host port's MAC address)
2. DA is a multicast address but the multicast address is not in ALE table
3. DA is a multicast address but host port is not part of the portMask field in multicast entry in ALE table

Also, did you try this? Mostly this should fix the issue:

Shaunak Deshpande said:

2. Broadcast mac address will be registered during Init/SetPhys address API, we should get to know the destination MAC address of packet(which is sent from outside to controller) That mac address we should register with Eth_UpdatePhysAddrFilter in the ALE table.

Can you try the above 2 and share results?

Regards,
Shaunak

Hi Shaunak,

i will check 2nd point, i have a question before it.

Individually Tcp/Icmp frame is received, how this is working if MAC is not registered in ALE table?

Hi Shaunak,

"Broadcast mac address will be registered during Init/SetPhys address API, we should get to know the destination MAC address of packet(which is sent from outside to controller) That mac address we should register with Eth_UpdatePhysAddrFilter in the ALE table."

I have tried to get the Broadcast mac address.

->As per my understanding getting external Broadcast mac address is not possible, only during ARP from external.

->Could you please let me know address/variable of ALE table.

->In which function i can call the following function Eth_UpdatePhysAddrFilter

Could you please explain little bit more detailed, from which function i can get Broadcast Mac & from which function i can able to update it in ALE table.

Hi Hari,

Apologies for a delayed response, I was Out of office yesterday.

1. We were able to run the tests with ALE being bypassed (no packet at ingress will be evaluated/dropped by the ALE). We noticed in the debug call that the ALE Port mask was causing the issues. The ALE will forward the broadcast packets and that was the reason we were able to see no drops for the DoIP (UDP) and the ARP packets. This broadcast forwarding works because during the Init, we add the broadcast MAC in the ALE. 

2. For us the unicast addresses were not present in the ALE. My suspicion is that this is the issue: DA of the packet is a unicast not in the ALE table (i.e. not host port's MAC address)

3.

Hari krishna Kaki said:

->In which function i can call the following function Eth_UpdatePhysAddrFilter

The flow is generally as follows:

Eth_Init() -> EthApp_TrcvInit() -> Eth_SetControllerMode() -> Eth_SetPhy_Addr() 0 -> Eth_UpdatePhysAddrFilter()

The Eth and Eth Trcv are initialized first. Then we set the AM263x in controller mode, set the controller address and then we set the destination address of the packets to the ALE. The function in bold is what I feel missing in this case and causing the packet drops.

You don't need to update or add the broadcast MAC in the ALE table. Refer Point-2.

Hari krishna Kaki said:

->Could you please let me know address/variable of ALE table.

It would be difficult to directly read ALE table from the memory browser. The ALE table registers can be found in section 4.18.416 of the register guide: www.ti.com/.../spruj42d.pdf

The ALE table should be read by the gel script. The ALE table start address is 0x5283E020 and there are 512 entries. Each entry will be divided into word0 (0x5283E03C) word1 (0x5283E038) and word2 (0x5283E034). Then that can be inferred further.

This is how the gel script reads the ALE table:

hotmenu cpsw_print_ale_table_3g()
{
    uint32_t word0, word1, word2;
    uint32_t type;
    int i;

    GEL_TextOut("---------------------------------------------\n");
    GEL_TextOut("-------CPSW3G ALE TABLE----------------------\n");
    GEL_TextOut("---------------------------------------------\n");

    for (i = 0; i < ALE_TABLE_DEPTH_3G; i++)
    {
        WR_MEM_32(ALE_TBLCTL_3G, i);

        word0 = RD_MEM_32(ALE_TBLW0_3G);
        word1 = RD_MEM_32(ALE_TBLW1_3G);
        word2 = RD_MEM_32(ALE_TBLW2_3G);

        if (0)
        {

            GEL_TextOut("WORD1  = %u\n",,,,, word0);
            GEL_TextOut("WORD1  = %u\n",,,,, word1);
            GEL_TextOut("WORD1  = %u\n",,,,, word2);
        }

        /* ENTRY_TYPE (bits 61:60) */
        type = (word1 >> 28) & 0x3;

        if (type == ALE_ENTRY_EMTPY) {
            //GEL_TextOut(" Entry %u - Empty\n",,,,, i);
        } else if (type == ALE_ENTRY_ADDR) {
            if ((word1 >> 8) & 0x1) {
                show_ale_entry_multicast_3g(i, word0, word1, word2);
//            } else if ((word1 >> 30) & 0x3) {
//                show_ale_entry_oui_unicast(i, word0, word1, word2);
            } else {
                show_ale_entry_unicast_3g(i, word0, word1, word2);
            }
        } else if (type == ALE_ENTRY_VLAN) {
            show_ale_entry_vlan_3g(i, word0, word1, word2);
        } else if (type == ALE_ENTRY_VLAN_ADDR) {
            if ((word1 >> 8) & 0x1) {
                show_ale_entry_vlan_multicast_3g(i, word0, word1, word2);
            } else {
                show_ale_entry_vlan_unicast_3g(i, word0, word1, word2);
            }
        }
    }

    GEL_TextOut("Completed analysis of %u ALE entries\n",,,,, i);
}

Regards,
Shaunak

Hi Shaunak,

I think below is non autosar sequence.

Eth_Init() -> EthApp_TrcvInit() -> Eth_SetControllerMode() -> Eth_SetPhy_Addr() 0 -> Eth_UpdatePhysAddrFilter()

The sequence is in my case is little different and these functions are not called at all Eth_SetPhy_Addr(), Eth_UpdatePhysAddrFilter()

Eth_Init() - EthTrcv_Init() - Eth_SetControllerMode()

As the issue is pending from a long

I am available from 1pm IST.

Is it possible to have a debug session tomorrow, to conclude &close the issue?

Kind regards,

Hari

Hi Hari,

Sure, can we have a 45 minute sync up at 4pm IST? Please reuse the meeting link shared for the last meeting.

Regards,
Shaunak

Thank you!

Could you please send meeting request again.

By mistake i declined previous meeting request.

Sure, I will reshare the meeting invite

Hi Shaunak,

As per our discussion on last Friday, i have tried below changes in EthRxProcessPacket() function, but it is not working.

Do you have any update?

Any thing needs to be check, Today's debug session at 4pm IST is required?

Hi Hari,

Can you check one more thing,

In tresos, instead of 11:22:33:44:55:66, can you have AA:BB:CC:DD:EE:FF as the Controller MAC address.

Regards,
Shaunak

Hi Shaunak,

Previously, i used 66-55-44-33-22-11 as 11-22-33-44-55-66 is not working.

Now i checked with AA-BB-CC-DD-EE-FF and still the issue is same.

Hi Hari,

To make the debugging easier, I tried to replicate this on the MCAL out of box Eth App demo. I was not able to. I asked the other expert to review it as well and they were not able to replicate either.

If possible can you try replicating it at your end with just the Eth App demo. We just need to check the stats in MCAL demo with your tool (check for ICMP and TCP drops). The driver treats all the packets same at Layer-2. Running the MCAL ETH APP demo and sending packets with your tool, if we see MCAL Eth demo showing drops as well, then I can confirm this as an issue with ALE or our MCAL offering, else this might be test or application level issue where I do not have my expertise.

One suggestion I would also have is, if you are on MCAL 09.01 SDK, please try the same with MCAL 09.02 SDK as the team made some changes.

Regards,
Shaunak

Hi Shaunak,

I tried to test same test sequence with Mcal demo, but it not able to test it.

Tool is expecting ARP response from Mcal Demo to continue remaining test sequence, that is not happening in case of Demo.

ALE & STAT script results:

In this case there is no drop, and can not confirm of frame drop.

Because full test sequence is not completed.

From above ALE image Port 0 has both HW & Tester Mac addresses.

Here there is no ARP & transmission based on MAC not on IP address.

Q) When/from which function tester Mac is added to ALE?

Q) On reception of a frame?

Q) Normally, when tester Mac will be added to ALE?

*I will try possibility, to test same sequence, but not sure.

Hi Hari,

1. I was not able to replicate the issue in our SDK either.

Hari krishna Kaki said:

Here there is no ARP & transmission based on MAC not on IP address.

Yes since we don't have a TCP/IP AutoSAR stack, we cannot test IP based transmission, but I know some customers using who successfully implement IP based communication after integration of their stack over our MCAL. 

2. 

Hari krishna Kaki said:

Q) When/from which function tester Mac is added to ALE?

You can refer the EthApp Demo, We use Eth_SetPhysAddr() to set the controller MAC and use Eth_UpdatePhysAddrFilter() to add the test MAC address (which will be the destination address of the packet being sent from the tool).

All the MAC address addition happens after Eth_Init(), Eth_TrcvInit(), Eth_SetControllerMode(), Eth_GetControllerMode().

Hari krishna Kaki said:

Q) On reception of a frame?

No, it is added before-hand

You can refer Eth_Priv.c to see how it gets added to ALE.

First when Eth_UpdatePhysAddrFilter() is called, it calls Eth_HwUpdatePhysAddrFilter().

I would suggest putting a breakpoint in this function, while tester MAC is being added and checking the value of currPort. If this value is 0, the ALE table port number will be 0, if this is 1, it will be added as port number 1.

Can you try this and make sure it is zero (for testing purposes, you can try hardcoding 0). But since the drops are not visible in SDK, I'm not sure how much effort I can put in here. Seems to be either an issue with the testing or the configuration. Because when we test TCP independently, using ncat, we are able to connect and same for the ICMP ping.

Regards,
Shaunak

Hi Hari,

Do you have any updates regarding this issue? Do we still observe the same behavior?

Regards,
Shaunak

Hi Shaunak,

Similar use case test setup is not available to test it TI MCAL eth_app demo.

We are looking for options to test it. I will update as soon as i got use case test results.

Kind regards,

Hari