This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

MCU-PLUS-SDK-AM243X: FORWARD_MODE_CUTTHROUGH

Part Number: MCU-PLUS-SDK-AM243X
Other Parts Discussed in Thread: AM2432, SYSCONFIG

hello.


My name is Jinbo Kim, and I work at AC&T Systems in Korea.

I experienced a particular issue during development using one of TI's products, the AM2432 MCU.

The development environment is as follows:

- SDK: ind_comms_sdk_am243x_09_00_00_03

- 2-port DP83822 PHY (switch port)

-LWIP Stack

The problem I'm facing is that if queueForwardMode is set to ICSSG_QUEUE_FORWARD_MODE_CUTTHROUGH in the IcssgMacPort_initCfg function, I get communication issues during the 2PORT Swap test.

The testing steps are as follows:

1. Set CUTTHROUGH_MODE and connect your PC to one of the two ports.

2. Send a PING request from your PC and check for a response.

3. In this state, try connecting to another port that was not previously connected.

4. Send a PING request from your PC.

5. No PING response. Or there is still no response after a few successful packets.

Link detection for each port is normal.

PHY mode settings are as follows.

setPortStateInArgs.macPort = macPort;

setPortStateInArgs.portState = ICSSG_PORT_STATE_FORWARD_WO_LEARNING;

ENET_IOCTL_SET_IN_ARGS(&prms, &setPortStateInArgs);

ENET_IOCTL(gEnetMp.perCtxt->handleInfo.hEnet, gEnetMp.coreId, ICSSG_PER_IOCTL_SET_PORT_STATE, &prms, status);

ENET_IOCTL(gEnetMp.perCtxt->handleInfo.hEnet, gEnetMp.coreId, ENET_PER_IOCTL_SET_VLAN_UNAWARE, NULL, status);

All settings succeed without failure.

For the same condition, if I change the queueForwardMode to ICSSG_QUEUE_FORWARD_MODE_STOREANDFWD, that symptom does not occur and all communication works fine.

When setting queueForwardMode,

ICSSG_QUEUE_FORWARD_MODE_STOREANDFWD or Should the conditions that need to be configured be different depending on ICSSG_QUEUE_FORWARD_MODE_CUTTHROUGH?

In the provided example programming, is there any example programming that can help with this content?

I look forward to your reply.

thank you.

  • Hi Kim,

    1. I see that you're using the Industrial Comms SDK v9.0 for your development. Can you share which example are you referring to from the SDK?
    2. Can you please share the network captures for your test : both the passing and the failing test run?

    Regards
    Archit Dev

  • The example Code on which the project is based is networking's icssg_layer2_switch.
    When I look at the packet flow with wireshark, there is no response as if the port is blocked (no ARP response).

    It does not working the Rx Packet Reception logic within the project.
    (As an example source, enet_layer2_icssg.c line 1605)

    The PORT where the initial communication occurred is still communicating well.
    (If communication starts from port 1 of the 2 ports, a problem occurs when swapping to port 2. Conversely, if communication starts from port 2, a problem occurs when swapping to port 1)

    Under the same conditions, there is no problem with STORE&FORWARD mode.

  • Hi Kim,

    Thank you for the reply.

    If communication starts from port 1 of the 2 ports, a problem occurs when swapping to port 2. Conversely, if communication starts from port 2, a problem occurs when swapping to port 1


    One more clarification here, what do you mean by swapping here.

    Do you physically plug out the cable from the original port (say, Port 1) and connect it to the other port (say, Port 2) and try to ping the DUT?

    Or, do you connect a cable from a different port of the same PC to the other port (say, Port 2) of the DUT while the original cable (connected to say, Port 1) is connected and never removed?

    Regards
    Archit Dev

  • Thank you for your quick response.

    Swap, as explained earlier, means changing between two ports by physically moving the connected port. (1->2 Port, vice versa)

    This is a physical port movement between the PC port and the development board.

  • Hi Kim,

    Thank you for the confirmation.

    I am able to reproduce the error on my side. I am currently investigating why this could be caused.

    Regards
    Archit Dev

  • Thank you.

    We look forward to a quick solution.

  • Hi Kim,

    We have some updates from our investigation.

    We have used the enet_lwip_icssg example for testing. (ind_comms_sdk_am243x_09_00_00_03\examples\networking\lwip\enet_lwip_icssg)

    The scenario which we reported above on our side was when the settings was
    setPortStateInArgs.portState = ICSSG_PORT_STATE_FORWARD;
    and hence we had incorrectly reported as issue reproduced since it was not matching your test configuration of setPortStateInArgs.portState = ICSSG_PORT_STATE_FORWARD_WO_LEARNING;


    However, with the settings you have described in the first post:
    setPortStateInArgs.portState = ICSSG_PORT_STATE_FORWARD_WO_LEARNING;
    We have conducted the 5 testing steps of the 2PORT Swap test as you have described above but we do not see this behaviour on our side.
    The communication works fine in both STOREANDFWD and CUTTHROUGH mode on our side.

    Test Configuration:
    - SDK: ind_comms_sdk_am243x_09_00_00_03
    - AM243x EVM
    - enet_lwip_icssg example with the following settings adapted as described in the post above-

    setPortStateInArgs.macPort = macPort;
    setPortStateInArgs.portState = ICSSG_PORT_STATE_FORWARD_WO_LEARNING;
    ENET_IOCTL_SET_IN_ARGS(&prms, &setPortStateInArgs);
    ENET_IOCTL(gEnetMp.perCtxt->handleInfo.hEnet, gEnetMp.coreId, ICSSG_PER_IOCTL_SET_PORT_STATE, &prms, status);
    ENET_IOCTL(gEnetMp.perCtxt->handleInfo.hEnet, gEnetMp.coreId, ENET_PER_IOCTL_SET_VLAN_UNAWARE, NULL, status);

    Test Sequence:
    - 2PORT Swap test (As described above)

    Could you share more details on the Test Application/configurations for us to reproduce this on our side to debug this further.

    Also to understand the scenario better, could you check if there is any ping reply packets being transmitted on the original port (say, Port1) in your test setup when cable is swapped to Port2 in the CUTTHROUGH mode.
    This can help us narrow down if there is issue in transmit of ping reply by AM243x etc.

    Thanks and Regards,
    Miriam

  • Thank you for answer.

    As mentioned above, after the device boots up, the settings are as follows:
    - Register PHY interrupt handler
    - initialize lwIP stack startup
    -Switch Setup
    The switch settings are as follows. (Briefly)
    ENET_IOCTL_SET_INOUT_ARGS(&prms, &macPort, &linked);
    ENET_IOCTL(gEnetMp.perCtxt->handleInfo.hEnet, gEnetMp.coreId, ENET_PER_IOCTL_IS_PORT_LINK_UP, &prms, status);
    setPortStateInArgs.macPort = macPort;
    setPortStateInArgs.portState = ICSSG_PORT_STATE_FORWARD_WO_LEARNING;
    ENET_IOCTL_SET_IN_ARGS(&prms, &setPortStateInArgs);
    ENET_IOCTL(gEnetMp.perCtxt->handleInfo.hEnet, gEnetMp.coreId, ICSSG_PER_IOCTL_SET_PORT_STATE, &prms, status);
    ENET_IOCTL(gEnetMp.perCtxt->handleInfo.hEnet, gEnetMp.coreId, ENET_PER_IOCTL_SET_VLAN_UNAWARE, NULL, status);

    All settings are the same, and in the IcssgMacPort_initCfg function
    The problem occurs when only the queueForwardMode setting is changed to ICSSG_QUEUE_FORWARD_MODE_CUTTHROUGH.

    If you set ICSSG_QUEUE_FORWARD_MODE_STOREANDFWD mode, the problem mentioned above will not occur.

    1. After the device boots for the first time, both ports start without connection.
    2. When communication occurs on the first connected port of the two ports and the port statistics are checked, the rx frame is confirmed as shown below.
    Port 0 statistics
    --------------------------------
    rxGoodFrames = 635
    rxBCastFrames = 372
    rxMCastFrames = 605
    rxCRCErrors = 0
    rxMIIErrors = 0
    rxOddNibbleFrame = 0
    rxMaxSizeErrFrame = 0
    rxMinSizeErrFrame = 0
    rxOverrunFrame = 0

    3. Remove the initially connected port and connect to another port. (Port Event occurs)
    Check port statistics.
    Port 1 statistics
    --------------------------------
    rxGoodFrames = 1498
    rxBCastFrames = 971
    rxMCastFrames = 1489
    rxCRCErrors = 0
    rxMIIErrors = 0
    rxOddNibbleFrame = 0
    rxMaxSizeErrFrame = 0
    rxMinSizeErrFrame = 0
    rxOverrunFrame = 0

    Count of rxGoodFrames increases but there is no response.
    (There is not even a Bcast ARP response.)

    In this state, if you reconnect to the port you first connected to, communication will proceed normally.

    This issue does not occur in ICSSG_QUEUE_FORWARD_MODE_STOREANDFWD mode.

    When setting the ICSSG_QUEUE_FORWARD_MODE_CUTTHROUGH mode, are there any additional settings that need to be set?
    I have to use ICSSG_QUEUE_FORWARD_MODE_CUTTHROUGH setting.

    Answers I'll wait.

    thank you

    * Additionally, the maximum value of rxBCastFrames / rxMCastFrames in Port Statistics does not seem to be output above 65536. Does that mean the value does not increase beyond 65535?
    The value of rxGoodFrames continues to increase.
    Port 1 statistics
    --------------------------------
    rxGoodFrames = 99643
    rxBCastFrames = 65535
    rxMCastFrames = 65535
    rxCRCErrors = 0
    rxMIIErrors = 0
    rxOddNibbleFrame = 0
    rxMaxSizeErrFrame = 0
    rxMinSizeErrFrame = 0
    rxOverrunFrame = 0

  • While looking for similar problems on the forum, I am letting you know because my issue seems similar to the one in the link below.

    e2e.ti.com/.../lp-am243-stuck-port-in- cutthrough-mode/5216295

    Please test the same with the enet_layer2_icssg example, not the enet_lwip_icssg example.

    I also started with the enet_layer2_icssg example.

  • Hi Kim,

    Thank you for the details provided.

    We are able to see the behaviour described at 10Mbps link speed in CUTTHROUGH mode, with the same procedure we had described earlier.
    Whereas the communication works fine in CUTTHROUGH mode at 1Gbps and 100Mbps. (Also communication works fine in STOREANDFWD mode at 1Gbps, 100Mbps and 10Mbps.)

    Q. Could you please clarify the link speed used for the testing on your side to confirm if the issue is indeed reproduced ?

    my issue seems similar to the one in the link below.

    e2e.ti.com/.../lp-am243-stuck-port-in- cutthrough-mode/5216295

    Please test the same with the enet_layer2_icssg example, not the enet_lwip_icssg example.

    Thank you for clarifying this, if it's confirmed that the issue is reproduced at the 10M link speed, then we can continue the investigation. If not, we can test with layer 2 example as mentioned in the other thread.

    Thanks and Regards,
    Miriam

  • Thanks for your reply.

    The link speed is 100Mbps and CUT-THR/STORE_FWD are all the same.

    When I tested it myself, several phenomena occurred.
    (For CUT-THR mode)

    1. Communication may fail during testing under the same conditions.
    During the aging test, communication may be lost after 1 to 2 hours.

    2. As mentioned earlier, if the connected port is moved, communication will be lost after 1-2 seconds.

    3. The frequency of symptoms varies depending on the connected PC.
    (My test PC is a Lenovo LEGION laptop, and the network card is Realtek. Every test I tried showed the symptom.)
    (The symptom did not appear well on HP laptops and older MSI laptops.)
    (If communication is lost, communication will fail even if you connect another PC.)

    4. This happens more often if you have a Netgear switch between your PC and test equipment. (PC - Netgear (GS308E) - AM2432 / Other switches also require testing.)

    5. If communication is lost, it cannot be recovered until rebooted.

    In any case, if you return to the port you were initially connected to (the port where communication was initially successful), communication will succeed normally.

    And I have an additional question.
    The port statistics don't seem to show rxBCastFrames/rxMCastFrames maximums above 65536. Does this mean the value won't increase above 65535?
    The value of rxGoodFrames continues to grow.
    Port 1 Statistics
    --------------------------------
    rxGoodFrames = 99643
    rxBCastFrames = 65535
    rxMCastFrames = 65535
    rxCRCErrors = 0
    rxMIIErrors = 0
    rxOddNibbleFrame = 0
    rxMaxSizeErrFrame = 0
    rxMinSizeErrFrame = 0
    rxOverrunFrame = 0

  • HI Miriam.

    I'm curious about the progress.

    Did you succeed in reproducing the symptom?

    And I have another question during the test.

    In the following situation (AM243x CUT-THR mode), I captured packets between equipment nodes.

    A: Packet Generator (PLC)
    B: AM2432
    C: AM2432
    A - B - C

    Captured section
    - Between A and B
    - Between B and C

    When the amount of packets increases, at some point, the packets captured from A-B are not visible between B-C.

    I increased tx dma and rx dma from 8 -> 16 -> 24 in sysconfig and tested, but there was no significant change.
    (All links are 100 mbps)

    Can you tell me the limit of packets that AM243x can accept?
    If the amount of packets that can be processed is exceeded, are the packets dropped?

    I'm attaching a part of the test packet.

    I think you can measure the tx speed and total packet amount to some extent in delta time.

  • Hi Kim,

    We were able to reproduce the symptom only at 10Mbps in cut-through mode. Not at 100Mbps. And we are investigating with this symptom seen at 10Mbps now.

    And I have an additional question.
    The port statistics don't seem to show rxBCastFrames/rxMCastFrames maximums above 65536. Does this mean the value won't increase above 65535?
    The value of rxGoodFrames continues to grow.

    Yes, This is because the Good Frames counter is 32-bit wide while the BCast and MCast Counter is 16-bit hence it hits the limit of 65535. One way to handle this would be to have a task in the application to periodically consolidate the statistics and then clear/reset the register values. In this way the consolidated value will give the actual count of the Broadcast and multicast frames.

    Okay we shall look into this test scenario, and get back with more details, regarding the packet drops and the limits of packets. Also, increasing the tx dma and rx dma from 8 -> 16 -> 24 in sysConfig will not affect the forwarding path where the drops are seen, rather it would affect packets sent to and from the host.

    Also is it possible to get the TX statistics along with the RX statistics on both ports in the cases of original port and the swapped port communication test? Also could you share the dumped values from address PRU_ICSSG1_PA_STAT_WRAP_PA_SLV_QSTAT 0x0300A7000 to 0x0300A7300 - if PRU_ICSSG1
    (or)
    PRU_ICSSG0_PA_STAT_WRAP_PA_SLV_QSTAT 0x030027000 to 0x030027300 - if PRU_ICSSG0

    Thanks and Regards,
    Miriam

  • HI Miriam.

    Is the symptom reproduced only in CUT-TRH mode at 10mbps?

    I set my test PC's network adapter settings to 100Mbps instead of auto-negotiation, and the symptoms are the same.

    And even if you connect an AM2432 device (both PHY and OS are the same), communication works well at first in CUT-THR mode, but one port is blocked. (About 10 AM2432 devices are connected and sending and receiving data.) Devices that cause the symptom occur randomly.
    The time at which symptoms appear varies depending on the test, but usually occurs within 5 hours.
    As explained earlier, they are all the same symptoms.
    All port links are detected properly, but blocked ports do not communicate.

    This symptom does not appear in STORE-AND-FORWARD mode under the same conditions.

    And do you mean to reset the port statistics regularly using the ENET_STATS_IOCTL_RESET_HOSTPORT_STATS / ENET_STATS_IOCTL_RESET_MACPORT_STATS settings?

    Information you request will be processed as follows:
    The test method below is for cases where communication failure occurs when exchanging port 1 -> port 2.

    Communication status of port 1 (normal)

    FreeRTOS> md 30027000
    30027000: 00000000 00000000 00000000 00000000 ................
    30027010: 00000000 00000000 00000000 00000000 ................
    30027020: 00000000 00000000 00000000 00000000 ................
    30027030: 00000000 00000000 00000000 00000000 ................
    30027040: 00000000 00000000 00000000 00000000 ................
    30027050: 00000000 00000000 00000000 00000000 ................
    30027060: 00000000 00000000 00000000 00000000 ................
    30027070: 00000000 00000000 00000000 00000000 ................
    30027080: 00000000 00000000 00000000 00000000 ................
    30027090: 00000000 00000000 00000000 00000000 ................
    300270a0: 00000000 00000000 00000000 00000000 ................
    300270b0: 00000000 00000000 00000000 00000000 ................
    300270c0: 00000000 00000000 00000000 00000000 ................
    300270d0: 00000000 00000000 00000000 00000000 ................
    300270e0: 00000000 00000000 00000000 00000000 ................
    300270f0: 00000000 00000000 00000000 00000000 ................
    FreeRTOS> md 30027100
    30027100: 00000000 00000000 00000000 00000000 ................
    30027110: 00000000 00000000 00000000 00000000 ................
    30027120: 00000000 00000000 00000000 00000000 ................
    30027130: 00000000 00000000 00000000 00000000 ................
    30027140: 00000000 00000000 00000000 00000000 ................
    30027150: 00000000 00000000 00000000 00000000 ................
    30027160: 00000000 00000000 00000000 00000000 ................
    30027170: 00000000 00000000 00000000 00000000 ................
    30027180: 00000000 00000000 00000000 00000000 ................
    30027190: 00000000 00000000 00000000 00000000 ................
    300271a0: 00000000 00000000 00000000 00000000 ................
    300271b0: 00000000 00000000 00000000 00000000 ................
    300271c0: 00000000 00000000 00000000 00000000 ................
    300271d0: 00000000 00000000 00000000 00000000 ................
    300271e0: 00000000 00000000 00000000 00000000 ................
    300271f0: 00000000 00000000 00000000 00000000 ................
    FreeRTOS> md 30027200
    30027200: 00000000 00000000 00000000 00000000 ................
    30027210: 00000000 00000000 00000000 00000000 ................
    30027220: 00000000 00000000 00000000 00000000 ................
    30027230: 00000000 00000000 00000000 00000000 ................
    30027240: 00000000 00000000 00000000 00000000 ................
    30027250: 00000000 00000000 00000000 00000000 ................
    30027260: 00000000 00000000 00000000 00000000 ................
    30027270: 00000000 00000000 00000000 00000000 ................
    30027280: 00000000 00000000 00000000 00000000 ................
    30027290: 00000000 00000000 00000000 00000000 ................
    300272a0: 00000000 00000000 00000000 00000000 ................
    300272b0: 00000000 00000000 00000000 00000000 ................
    300272c0: 00000000 00000000 00000000 00000000 ................
    300272d0: 00000000 00000000 00000000 00000000 ................
    300272e0: 00000000 00000000 00000000 00000000 ................
    300272f0: 00000000 00000000 00000000 00000000 ................

    FreeRTOS> md 300A7000
    300a7000: 0019c5b0 00000000 00013f32 00000000 ........2?......
    300a7010: 00000000 00000000 00000000 00000000 ................
    300a7020: 00000000 00000000 00000000 00000000 ................
    300a7030: 00000000 00000000 00000000 00000000 ................
    300a7040: 00000000 00000000 00000000 00000000 ................
    300a7050: 00000000 00000000 00000000 00000000 ................
    300a7060: 00000000 00000000 00000000 00000000 ................
    300a7070: 00000000 00000000 00000000 00000000 ................
    300a7080: 00004e0f 00000458 00000000 00000000 .N..X...........
    300a7090: 00000000 00000000 00000000 00000000 ................
    300a70a0: 00000000 00000000 00000000 00000000 ................
    300a70b0: 00000000 00000000 00000000 00000000 ................
    300a70c0: 00000000 00000000 00000000 00000000 ................
    300a70d0: 00000000 00000000 00000000 00000000 ................
    300a70e0: 00000000 00000000 00000000 00000000 ................
    300a70f0: 00000000 00000000 00000000 00000000 ................
    FreeRTOS> md 300A7100
    300a7100: 00000000 00000000 00000000 00000000 ................
    300a7110: 00000000 00000000 00000000 00000000 ................
    300a7120: 00000000 00000000 00000000 00000000 ................
    300a7130: 00000000 00000000 00000000 00000000 ................
    300a7140: 00000000 00000000 00000000 00000000 ................
    300a7150: 00000000 00000000 00000000 00000000 ................
    300a7160: 00000000 00000000 00000000 00000000 ................
    300a7170: 00000000 00000000 00000000 00000000 ................
    300a7180: 00000000 00000000 00000000 00000000 ................
    300a7190: 00000000 00000000 00000000 00000000 ................
    300a71a0: 00000000 00000000 00000000 00000000 ................
    300a71b0: 00000000 00000000 00000000 00000000 ................
    300a71c0: 00000000 00000000 00000000 00000000 ................
    300a71d0: 00000000 00000000 00000000 00000000 ................
    300a71e0: 00000000 00000000 00000000 00000000 ................
    300a71f0: 00000000 00000000 00000000 00000000 ................
    FreeRTOS> md 300A7200
    300a7200: 00000000 00000000 00000000 00000000 ................
    300a7210: 00000000 00000000 00000000 00000000 ................
    300a7220: 00000000 00000000 00000000 00000000 ................
    300a7230: 00000000 00000000 00000000 00000000 ................
    300a7240: 00000000 00000000 00000000 00000000 ................
    300a7250: 00000000 00000000 00000000 00000000 ................
    300a7260: 00000000 00000000 00000000 00000000 ................
    300a7270: 00000000 00000000 00000000 00000000 ................
    300a7280: 00000000 00000000 00000000 00000000 ................
    300a7290: 00000000 00000000 00000000 00000000 ................
    300a72a0: 00000000 00000000 00000000 00000000 ................
    300a72b0: 00000000 00000000 00000000 00000000 ................
    300a72c0: 00000000 00000000 00000000 00000000 ................
    300a72d0: 00000000 00000000 00000000 00000000 ................
    300a72e0: 00000000 00000000 00000000 00000000 ................
    300a72f0: 00000000 00000000 00000000 00000000 ................

    Communication status of port 2 (communication failure).

    FreeRTOS> md 30027000
    30027000: 00000000 00000000 00000000 00000000 ................
    30027010: 00000000 00000000 00000000 00000000 ................
    30027020: 00000000 00000000 00000000 00000000 ................
    30027030: 00000000 00000000 00000000 00000000 ................
    30027040: 00000000 00000000 00000000 00000000 ................
    30027050: 00000000 00000000 00000000 00000000 ................
    30027060: 00000000 00000000 00000000 00000000 ................
    30027070: 00000000 00000000 00000000 00000000 ................
    30027080: 00000000 00000000 00000000 00000000 ................
    30027090: 00000000 00000000 00000000 00000000 ................
    300270a0: 00000000 00000000 00000000 00000000 ................
    300270b0: 00000000 00000000 00000000 00000000 ................
    300270c0: 00000000 00000000 00000000 00000000 ................
    300270d0: 00000000 00000000 00000000 00000000 ................
    300270e0: 00000000 00000000 00000000 00000000 ................
    300270f0: 00000000 00000000 00000000 00000000 ................
    FreeRTOS> md 30027100
    30027100: 00000000 00000000 00000000 00000000 ................
    30027110: 00000000 00000000 00000000 00000000 ................
    30027120: 00000000 00000000 00000000 00000000 ................
    30027130: 00000000 00000000 00000000 00000000 ................
    30027140: 00000000 00000000 00000000 00000000 ................
    30027150: 00000000 00000000 00000000 00000000 ................
    30027160: 00000000 00000000 00000000 00000000 ................
    30027170: 00000000 00000000 00000000 00000000 ................
    30027180: 00000000 00000000 00000000 00000000 ................
    30027190: 00000000 00000000 00000000 00000000 ................
    300271a0: 00000000 00000000 00000000 00000000 ................
    300271b0: 00000000 00000000 00000000 00000000 ................
    300271c0: 00000000 00000000 00000000 00000000 ................
    300271d0: 00000000 00000000 00000000 00000000 ................
    300271e0: 00000000 00000000 00000000 00000000 ................
    300271f0: 00000000 00000000 00000000 00000000 ................
    FreeRTOS> md 30027200
    30027200: 00000000 00000000 00000000 00000000 ................
    30027210: 00000000 00000000 00000000 00000000 ................
    30027220: 00000000 00000000 00000000 00000000 ................
    30027230: 00000000 00000000 00000000 00000000 ................
    30027240: 00000000 00000000 00000000 00000000 ................
    30027250: 00000000 00000000 00000000 00000000 ................
    30027260: 00000000 00000000 00000000 00000000 ................
    30027270: 00000000 00000000 00000000 00000000 ................
    30027280: 00000000 00000000 00000000 00000000 ................
    30027290: 00000000 00000000 00000000 00000000 ................
    300272a0: 00000000 00000000 00000000 00000000 ................
    300272b0: 00000000 00000000 00000000 00000000 ................
    300272c0: 00000000 00000000 00000000 00000000 ................
    300272d0: 00000000 00000000 00000000 00000000 ................
    300272e0: 00000000 00000000 00000000 00000000 ................
    300272f0: 00000000 00000000 00000000 00000000 ................

    FreeRTOS> md 300A7000
    300a7000: 001b35fe 00000000 0001523c 00000000 .5......<R......
    300a7010: 00000000 00000000 00000000 00000000 ................
    300a7020: 00000000 00000000 00000000 00000000 ................
    300a7030: 00000000 00000000 00000000 00000000 ................
    300a7040: 00000000 00000000 00000000 00000000 ................
    300a7050: 00000000 00000000 00000000 00000000 ................
    300a7060: 00000000 00000000 00000000 00000000 ................
    300a7070: 00000000 00000000 00000000 00000000 ................
    300a7080: 0000526c 0000049b 00000000 00000000 lR..............
    300a7090: 00000000 00000000 00000000 00000000 ................
    300a70a0: 00000000 00000000 00000000 00000000 ................
    300a70b0: 00000cd7 00000000 00000000 00000000 ................
    300a70c0: 00000000 00000000 00000000 00000000 ................
    300a70d0: 00000000 00000000 00000000 00000000 ................
    300a70e0: 00000000 00000000 00000000 00000000 ................
    300a70f0: 00000000 00000000 00000000 00000000 ................
    FreeRTOS> md 300A7100
    300a7100: 00000000 00000000 00000000 00000000 ................
    300a7110: 00000000 00000000 00000000 00000000 ................
    300a7120: 00000000 00000000 00000000 00000000 ................
    300a7130: 00000000 00000000 00000000 00000000 ................
    300a7140: 00000000 00000000 00000000 00000000 ................
    300a7150: 00000000 00000000 00000000 00000000 ................
    300a7160: 00000000 00000000 00000000 00000000 ................
    300a7170: 00000000 00000000 00000000 00000000 ................
    300a7180: 00000000 00000000 00000000 00000000 ................
    300a7190: 00000000 00000000 00000000 00000000 ................
    300a71a0: 00000000 00000000 00000000 00000000 ................
    300a71b0: 00000000 00000000 00000000 00000000 ................
    300a71c0: 00000000 00000000 00000000 00000000 ................
    300a71d0: 00000000 00000000 00000000 00000000 ................
    300a71e0: 00000000 00000000 00000000 00000000 ................
    300a71f0: 00000000 00000000 00000000 00000000 ................
    FreeRTOS> md 300A7200
    300a7200: 00000000 00000000 00000000 00000000 ................
    300a7210: 00000000 0000003a 00000000 00000000 ....:...........
    300a7220: 00000000 00000000 00000000 00000000 ................
    300a7230: 00000000 00000000 00000000 00000000 ................
    300a7240: 00000000 00000000 00000000 00000000 ................
    300a7250: 00000000 00000000 00000000 00000000 ................
    300a7260: 00000000 00000000 00000000 00000000 ................
    300a7270: 00000000 00000000 00000000 00000000 ................
    300a7280: 00000000 00000000 00000000 00000000 ................
    300a7290: 00000000 00000000 00000000 00000000 ................
    300a72a0: 00000000 00000000 00000000 00000000 ................
    300a72b0: 00000000 00000000 00000000 00000000 ................
    300a72c0: 00000000 00000000 00000000 00000000 ................
    300a72d0: 00000000 00000000 00000000 00000000 ................
    300a72e0: 00000000 00000000 00000000 00000000 ................
    300a72f0: 00000000 00000000 00000000 00000000 ...............

    Are you testing your current packet handling process?
    I'm curious about the progress.

    Answers I'll wait.

    thank you

  • Hi Kim,

    Yes, the symptom is reproduced only in CUT-TRH mode at 10Mbps.

    And do you mean to reset the port statistics regularly using the ENET_STATS_IOCTL_RESET_HOSTPORT_STATS / ENET_STATS_IOCTL_RESET_MACPORT_STATS settings?

     

    Yes, this is right. ENET_STATS_IOCTL_RESET_HOSTPORT_STATS to reset the offset that have been dumped (PA_STATS) .
    And ENET_STATS_IOCTL_RESET_MACPORT_STATS to reset the hardware RX and TX statistics.

    Thank you for the details, We have some progress here on the issue regarding narrowing down to a corner case being hit in this test scenario which has a bug. This is a reason we believe to cause the observed behavior. We are working on the fix for this and testing. 

    We shall keep updated on further progress by end of this week.

    Thanks and Regards,
    Miriam

  • HI Miriam.

    Thank you for your quick response.

    I'm also testing various things.

    When setting the AM2432 PHY, I tried communicating by setting it to 100Mbps/FULL-DUPLEX instead of AUTO NEGO, but the symptom still appears.
    (I set PC and AM2432 to 100Mbps each and they are the same.)

    Anyway, it's good news that there's progress on this issue!

    I hope that solving your problem will solve mine too.

    I have one inquiry.
    After the device boots up, the logic that requests port statistics is called approximately every 500ms.
    (There are two ports, so requests are made for each port.)
    Sometimes I don't receive a response or it seems to be stuck in progress.
    Should there be a limit to the number of requests or a break in the middle?
    Some of the sources I implemented are as follows.

    ENET_IOCTL_SET_INOUT_ARGS(&prms, &port, &gEnetMp_icssgStats);
    ENET_IOCTL(hEnet, coreid, ENET_STATS_IOCTL_GET_MACPORT_STATS, &prms, status);
    if (status == ENET_SINPROGRESS)
    {
    /* Wait for asyc ioctl to complete */
    do
    {
    Enet_poll(hEnet, ENET_EVT_ASYNC_CMD_RESP, NULL, 0U);
    semStatus = SemaphoreP_pend(p_ayncIoctlSemObj, 1U);
    } while (semStatus != SystemP_SUCCESS);
    status = ENET_SOK;
    }

    After testing for a longer period of time, we will see how the symptom occurs.

    We are waiting for good news.

    thank you.

  • Hi Kim,

    The ENET_STATS_IOCTL_GET_MACPORT_STATS IOCTL is synchronous so we do not need to wait for async response i.e IOCTL status ENET_SINPROGRESS. Rather we need to check for status to be ENET_SOK.

    Same is the case for ENET_STATS_IOCTL_GET_HOSTPORT_STATS /ENET_STATS_IOCTL_RESET_HOSTPORT_STATS / ENET_STATS_IOCTL_RESET_MACPORT_STATS.

    For reference, you can check the usage in the function EnetMp_printStats() in the path "examples/networking/enet_layer2_icssg/enet_layer2_icssg.c" file.

    Thanks and Regards,
    Miriam

  • HI Miriam.

    I am curious about the current progress in resolving the issue.

    Is there any progress on packet processing capabilities and CUT-THR mode issues?

    Answers I'll wait.

    thank you

  • Hi Kim, 

    Yes, we have fixed a bug which has now enabled ping to work on both ports with the swapping of the cabling in Cut-through mode.
    However, when we did some regression testing we see some issue in forwarding. We are working on this currently.

    Thanks and Regards,
    Miriam

  • HI Miriam.

    Thank you for sharing the progress.

    Is there any information that can be tested by changing the config value for the packet loss phenomenon?

    Also, is there any fixed content related to the packet loss phenomenon in the Fixed Issues content of the latest SDK (09.02.00)?
    software-dl.ti.com/.../RELEASE_NOTES_09_02_00_PAGE.html

    If so, is it okay to use it by only changing the PRU binary file in the current SDK version?

    Thank you.

  • HI Miriam.

    I have an additional question.

    (* Currently in ICSSG_QUEUE_FORWARD_MODE_STOREANDFWD mode).

    While testing the packet loss phenomenon, we confirmed that overflow had increased among the PRU Statistics items as shown below.
    hostRxByteCnt = 4977053
    hostTxByteCnt = 3944
    hostRxPktCnt = 64817
    hostTxPktCnt = 54
    rtu0PktDroppedSlice0 = 0
    rtu0PktDroppedSlice1 = 0
    port1Q0Overflow = 11
    port1Q1Overflow = 0
    port1Q2Overflow = 0
    port1Q3Overflow = 0
    port1Q4Overflow = 0
    port1Q5Overflow = 0
    port1Q6Overflow = 0
    port1Q7Overflow = 0
    port2Q0Overflow = 15
    port2Q1Overflow = 0
    port2Q2Overflow = 0
    port2Q3Overflow = 0
    port2Q4Overflow = 0
    port2Q5Overflow = 0
    port2Q6Overflow = 0
    port2Q7Overflow = 0

    - port1/2 Q0 Why did Overflow increase?
    - Are packets processed as Overflow just discarded as Drop? Or is it handled some other way?
    - 0 ~7 What are the criteria by which Q is classified?
    - Is there a way to adjust that Q?

    Thanks and Regards,
    Kim.

  • Hi Kim,

    Also, is there any fixed content related to the packet loss phenomenon in the Fixed Issues content of the latest SDK (09.02.00)?

    We do not have any fixed content here. 

    In the following situation (AM243x CUT-THR mode), I captured packets between equipment nodes.

    A: Packet Generator (PLC)
    B: AM2432
    C: AM2432
    A - B - C

    I believe this is the packet loss phenomenon you have mentioned. Could you share more details here on the rate at which packets are sent, the sizes and type (such as VLAN tagged/ PCP priority). So that we shall also reproduce the same scenario on our side to verify this.

    When the amount of packets increases, at some point, the packets captured from A-B are not visible between B-C.

    Can you please specify this point of packet increase - is it with respect to duration or rate of packet transmission. 
    And when it is not visible - is it all dropped/ corrupted/ or a certain percentage is dropped?

    Regarding the second question, I shall take a look and respond soon.

    Thanks and Regards,
    Miriam

  • HI Miriam.

    As you requested, I am trying to send dumped packets.

    Can you give me your email address?

    We will send you a brief explanation of the operating status and a dumped packet file.

    Thanks and Regards,
    Kim.

  • Hi Kim,

    Since this thread is on a public forum, I have sent you a friend request on e2e forum. Once you have accepted the request,  the e-mail address etc. can be shared in the one-to-one thread.

    Thanks and Regards,
    Miriam

  • HI Miriam.

    We have accepted your request and sent you a private message.

    Be sure to enter your email address when checking the message.

    I will send the saved packet.

    And here is the test environment:

    PLC (packet generator) - AM2432 - AM2432 - AM2432 - .... (minimum 25)

    1. The PLC already knows the information of the connected AM2432.
    2. IP/MAC is different for each AM2432.
    3. PLC sends a broadcast packet to the connected AM2432. (PLC Request)
    4. Immediately send a unicast request to each AM2432. (PLC Request)
    5. Each AM2432 responds to broadcast/unicast requests.
    6. Packet loss occurs while repeating the operation.
    7. It was confirmed that portQ0 Overflow increases as packet loss occurs.

    * Currently all switch modes for AM2432 are STOREANDFORWARD mode.

    I'd like to use CUT-THR mode, but I can't do it yet due to the issues mentioned earlier.

    Answers I'll wait.

    Thanks and Regards,
    Kim.

  • Hi Kim,

    Thank you for the mail, we shall look into the shared file.

    Thank you for the description of the test environment. To clarify, is the IP address assigned for each device -static or dynamic, in the original single AM2432 test scenario as well as this 25 AM2432 test scenario?

    Thanks and Regards,
    Miriam

  • HI Miriam.

    All AM2432 devices are using static IPs. (IP band: 192.168.1.XXX).

    Packet loss does not occur well in single AM2432.

    As explained earlier, configure an environment with multiple AM2432s and proceed with testing.

    I'm also planning to test it with a network instrument.
    (network traffic in both directions)
    network Instrument - AM2432 - AM2432 - ... - network Instrument

    Thanks and Regards,
    Kim.

  • Hi Miriam.

    MCU+SDK for AM243x-09.02.00.50
    Examples/Networking/enet_layer2_icssg/icssg_layer2_switch
    I uploaded the source to the LP board and tested it as follows.

    Network equipment: Smartbits
    Test equipment: LP board

    Smartbits Port - LP Board - Smartbits Port
    Tx/Rx bi-directional 100M throughput test.
    Packet size 64 / 256 / 1024.

    | LP board |
           | |
    ----------------
    | Smartbits |
    ----------------
    Testing shows that size 64 succeeds, but sizes 256/1024 cause packet loss.

    We will send you the results file by email.

    Thanks and Regards,
    Kim.

  • Hi Kim,

    Thank you for the clarification and the test results, we shall look into this test scenario.

    Regards,
    Miriam

  • Hi Miriam.

    Thank you for your kind reply.

    Can you tell me the results of the tests conducted so far and answers to the questions I have?

    I would like to share my issues and progress to date with my teammates.

    Thanks and Regards,
    Kim.

  • Hi Kim,

    We have some positive updates with respect to the Cut-through issue.

    Yes, we have fixed a bug which has now enabled ping to work on both ports with the swapping of the cabling in Cut-through mode.
    However, when we did some regression testing we see some issue in forwarding. We are working on this currently.

    We have resolved this regression issue also, and now the Ping swap test reported in this post originally is working fine in cut-though mode. We will be able to share the firmware by next week Tuesday/Wednesday for you to test on your side.

    Thanks and Regards,
    Miriam

  • Hi Miriam.

    that is a good news.

    I hope the modified and changed firmware will solve my problem.

    Also, I would like to inquire about PORT1/2 Q0 Overflow, which I inquired about through personal message.

    The answer was that the increase in the value is as follows.
    - The Port 1/2 Qx counter will increase when the packets in the queue are not yet processed and incoming packets are not processed and dropped.

    In my opinion, I think that the increase in value is related to the packet loss tested earlier. Is my opinion correct?
    So, is a new PRU patch needed to resolve the packet loss issue?

    Thanks and Regards,
    Kim.

  • Hi Kim,

    that is a good news.

    I hope the modified and changed firmware will solve my problem.

    We have shared the firmware patch with you via e-mail. Please let us know if you have received it successfully. You can let us know the status of the tests with the latest firmware shared.

    In my opinion, I think that the increase in value is related to the packet loss tested earlier. Is my opinion correct?

    Yes the count of dropped packet (packet loss) and overflow counter value can be compared to verify this.

    We suggest to conduct the tests with the latest firmware and provide us the summary of the statistics of both the Q overflow numbers and the number of packets drops (from packet capture file) at each node/ device observed from your setup to correlate the data-points. 

    Thanks and Regards,
    Miriam

  • Hi Miriam.

    I have received a link to the patched firmware.

    We will conduct a test with the patched firmware and share the results.
    (including Aging test)

    Thanks and Regards,
    Kim

  • Hi Miriam.

    First, I tested the Smartbits in the same way, using the patched firmware and modified icssg.c you sent me.

    It seems like I'm still getting PORT Q Overflow.
    Is this part also patchable?
    - In our application, there is an event where Tx or Rx frames occur at regular intervals during the throughput test.
    - However, even if you turn off the function, the success rate of the throughput test relatively increases, but overflow still occurs.
    - The ideal feature is that no overflow occurs in all of these modes.

    It doesn't seem like port blocking is happening yet.
    I will share the results of this part after further testing.

    Thanks and Regards,
    Kim

  • Hi Miriam.

    In the patched PRU firmware, abnormal behavior is observed that is different from before.

    Before the patch, PRU communicated well without any problems.
    Communication is not possible with patched PRUs.

    All environments are the same and only changing the PRU F/W will cause problems.

    General communication is possible, but
    Our application communication has different communication results depending on the patched PRU firmware.
    I will give you the frame file. (one-to-one thread)

    Please check this part as well.

    Thanks and Regards,
    Kim

  • Hi Kim,

    When you mean general communication - did you verify the original 2 port Swap Test in Cut-through mode? Can you share the result of this original reported issue?

    Can you explain what is your application test scenario and what is the expectation vs what is actually observed, as from the pcap files alone, this information is not clear.

    Thanks and Regards,
    Miriam

  • Hi Miriam,

    The port swap communication problem, which was previously an issue, has not yet been reproduced.
    That part seems to have been patched well.
    Aging testing is currently ongoing.

    There are two versions of the frame capture file I sent.
    One is when the patched PRU firmware is in the state,
    Another one is the PRU firmware before the patch is applied.

    The outcome of the communication (success or failure) is determined by the PRU firmware.

    The test environment is as follows.
    PLC - AM2432 (1) - AM2432 (2) - ....

    The existing PRU firmware transfers the frame from PLC -> (1) -> (2), but
    Patched PRU firmware will not be carried over.
    Also, communication is not possible, either because there is no response to the request frame from PLC -> (1) HOST, or because the frame does not go up.

    If you change only the PRU firmware in the same environment, the communication results will vary.
    I think that while modifying the port block, other parts were also modified.

    In other words, frame communication that previously worked normally does not work with the patched PRU firmware.

    Please check this part.

    Thanks and Regards,
    Miriam

  • Jinbo Kim and I have been working together recently.

    It appears that there are issues with sending and receiving in the part where the ncm_mac_addr is used in the communication problem.

    The MAC address is as follows:

    static const uint8_t ncm_mac_addr[6] = {0x00, 0xe0, 0x91, 0x02, 0x05, 0x99};


    The VLAN and port settings are as follows:

    static int32_t EnetSW_PortForwardSetupRAPIEnet(EnetICSSGSW_PerObj *perobj)
    {
    Enet_IoctlPrms prms;
    Enet_MacPort macPort;
    Icssg_VlanFidEntry vlan_entry;
    Icssg_FdbEntry fdbentry;
    IcssgMacPort_SetPortStateInArgs setPortStateInArgs;
    bool linked;
    int32_t semStatus;
    uint32_t i;
    int32_t status = ENET_SOK;

    LOG_INF("enet port forward setup for RAPIEnet mode");

    for (i = 0U; i < ENET_PORT_NUM; i++)
    {
    macPort = perobj->macPort[i];
    linked = false;

    ENET_IOCTL_SET_INOUT_ARGS(&prms, &macPort, &linked);
    status = Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ENET_PER_IOCTL_IS_PORT_LINK_UP, &prms);
    if (status != ENET_SOK)
    {
    linked = false;
    }
    /* Set port to 'Forward' state */

    setPortStateInArgs.macPort = macPort;
    setPortStateInArgs.portState = ICSSG_PORT_STATE_FORWARD_WO_LEARNING;
    ENET_IOCTL_SET_IN_ARGS(&prms, &setPortStateInArgs);

    status = Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ICSSG_PER_IOCTL_SET_PORT_STATE, &prms);
    if (status == ENET_SINPROGRESS)
    {
    /* Wait for asyc ioctl to complete */
    do
    {
    Enet_poll(perobj->hEnet, ENET_EVT_ASYNC_CMD_RESP, NULL, 0U);
    semStatus = SemaphoreP_pend(&perobj->hAsyncIoctlSem, 1U);
    } while (semStatus != SystemP_SUCCESS);

    status = ENET_SOK;
    }

    if (status != ENET_SOK)
    {
    }
    }

    Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ENET_PER_IOCTL_SET_VLAN_AWARE, NULL);

    EnetPort_VlanCfg vlancfg;

    vlancfg.portPri = 0;
    vlancfg.portCfi = 0;
    vlancfg.portVID = 1;
    ENET_IOCTL_SET_IN_ARGS(&prms, &vlancfg);

    status = Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ICSSG_PER_IOCTL_VLAN_SET_HOSTPORT_DFLT_VID, &prms);
    if (status == ENET_SINPROGRESS)
    {
    /* Wait for asyc ioctl to complete */
    do
    {
    Enet_poll(perobj->hEnet, ENET_EVT_ASYNC_CMD_RESP, NULL, 0U);
    semStatus = SemaphoreP_pend(&perobj->hAsyncIoctlSem, 1U);
    } while (semStatus != SystemP_SUCCESS);

    status = ENET_SOK;
    }

    if (status != ENET_SOK)
    {
    }

    Icssg_MacPortDfltVlanCfgInArgs macvlancfg;

    macvlancfg.macPort = ENET_MAC_PORT_1;
    macvlancfg.vlanCfg.portPri = 0;
    macvlancfg.vlanCfg.portCfi = 0;
    macvlancfg.vlanCfg.portVID = 2;

    ENET_IOCTL_SET_IN_ARGS(&prms, &macvlancfg);
    status = Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ICSSG_PER_IOCTL_VLAN_SET_MACPORT_DFLT_VID, &prms);
    if (status == ENET_SINPROGRESS)
    {
    /* Wait for asyc ioctl to complete */
    do
    {
    Enet_poll(perobj->hEnet, ENET_EVT_ASYNC_CMD_RESP, NULL, 0U);
    semStatus = SemaphoreP_pend(&perobj->hAsyncIoctlSem, 1U);
    } while (semStatus != SystemP_SUCCESS);

    status = ENET_SOK;
    }

    if (status != ENET_SOK)
    {
    LOG_ERR("%s: Failed to add fbentry: %d", perobj->name, status);
    }

    macvlancfg.macPort = ENET_MAC_PORT_2;
    macvlancfg.vlanCfg.portPri = 0;
    macvlancfg.vlanCfg.portCfi = 0;
    macvlancfg.vlanCfg.portVID = 3;

    ENET_IOCTL_SET_IN_ARGS(&prms, &macvlancfg);
    status = Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ICSSG_PER_IOCTL_VLAN_SET_MACPORT_DFLT_VID, &prms);
    if (status == ENET_SINPROGRESS)
    {
    /* Wait for asyc ioctl to complete */
    do
    {
    Enet_poll(perobj->hEnet, ENET_EVT_ASYNC_CMD_RESP, NULL, 0U);
    semStatus = SemaphoreP_pend(&perobj->hAsyncIoctlSem, 1U);
    } while (semStatus != SystemP_SUCCESS);

    status = ENET_SOK;
    }

    if (status != ENET_SOK)
    {
    }

    /* VLAN1 host port control */
    memset(&vlan_entry, 0, sizeof(vlan_entry));
    vlan_entry.vlanId = 1;
    vlan_entry.vlanFidParams.fid = 1;
    vlan_entry.vlanFidParams.hostMember = 1;
    vlan_entry.vlanFidParams.p1Member = 1;
    vlan_entry.vlanFidParams.p2Member = 1;

    ENET_IOCTL_SET_IN_ARGS(&prms, &vlan_entry);
    status = Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ICSSG_PER_IOCTL_VLAN_SET_ENTRY, &prms);
    if (status != ENET_SOK)
    {
    }

    /* VLAN2 port0(p1) control, p2 disabled, disable P1->P2 forwarding */
    memset(&vlan_entry, 0, sizeof(vlan_entry));
    vlan_entry.vlanId = 2;
    vlan_entry.vlanFidParams.fid = 2;
    vlan_entry.vlanFidParams.hostMember = 1;
    vlan_entry.vlanFidParams.p1Member = 1;
    vlan_entry.vlanFidParams.p2Member = 0;
    //floodToHost
    ENET_IOCTL_SET_IN_ARGS(&prms, &vlan_entry);
    status = Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ICSSG_PER_IOCTL_VLAN_SET_ENTRY, &prms);
    if (status != ENET_SOK)
    {
    }

    /* VLAN3 port1(p2) forward control, p1 disable=>disable P2->P1 forwarding */
    memset(&vlan_entry, 0, sizeof(vlan_entry));
    vlan_entry.vlanId = 3;
    vlan_entry.vlanFidParams.fid = 3;
    vlan_entry.vlanFidParams.hostMember = 1;
    vlan_entry.vlanFidParams.p1Member = 0;
    vlan_entry.vlanFidParams.p2Member = 1;
    ENET_IOCTL_SET_IN_ARGS(&prms, &vlan_entry);
    status = Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ICSSG_PER_IOCTL_VLAN_SET_ENTRY, &prms);
    if (status != ENET_SOK)
    {
    }

    /* VLAN2 port0(p1) NCM address filter, disable forward, route to host */

    fdbentry.vlanId = 2; /* ICSSG_VLAN_UNTAGGED; */
    fdbentry.fdbEntry[0] = ICSSG_FDB_ENTRY_VALID | ICSSG_FDB_ENTRY_P0_MEMBERSHIP;
    fdbentry.fdbEntry[1] = ICSSG_FDB_ENTRY_VALID | ICSSG_FDB_ENTRY_P0_MEMBERSHIP;
    memcpy(fdbentry.macAddr, ncm_mac_addr, 6);

    ENET_IOCTL_SET_IN_ARGS(&prms, &fdbentry);

    status = Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ICSSG_FDB_IOCTL_ADD_ENTRY, &prms);
    if (status == ENET_SINPROGRESS)
    {
    /* Wait for asyc ioctl to complete */
    do
    {
    Enet_poll(perobj->hEnet, ENET_EVT_ASYNC_CMD_RESP, NULL, 0U);
    semStatus = SemaphoreP_pend(&perobj->hAsyncIoctlSem, 1U);
    } while (semStatus != SystemP_SUCCESS);

    status = ENET_SOK;
    }

    if (status != ENET_SOK)
    {
    }

    /* VLAN3 port1(p2) NCM address filter, disable forward, route to host */
    fdbentry.vlanId = 3; /* ICSSG_VLAN_UNTAGGED; */
    fdbentry.fdbEntry[0] = ICSSG_FDB_ENTRY_VALID | ICSSG_FDB_ENTRY_P0_MEMBERSHIP;
    fdbentry.fdbEntry[1] = ICSSG_FDB_ENTRY_VALID | ICSSG_FDB_ENTRY_P0_MEMBERSHIP;
    memcpy(fdbentry.macAddr, ncm_mac_addr, 6);

    ENET_IOCTL_SET_IN_ARGS(&prms, &fdbentry);

    status = Enet_ioctl(perobj->hEnet, gEnetSWObj.coreId, ICSSG_FDB_IOCTL_ADD_ENTRY, &prms);
    if (status == ENET_SINPROGRESS)
    {
    /* Wait for asyc ioctl to complete */
    do
    {
    Enet_poll(perobj->hEnet, ENET_EVT_ASYNC_CMD_RESP, NULL, 0U);
    semStatus = SemaphoreP_pend(&perobj->hAsyncIoctlSem, 1U);
    } while (semStatus != SystemP_SUCCESS);

    status = ENET_SOK;
    }

    if (status != ENET_SOK)
    {
    }

    return status;
    }

  • Hi Changjun im,

    We shall look into this and get back with any further details needed.

    Thanks and Regards,
    Miriam

  • Hi Changjun im, Kim,

    It appears that there are issues with sending and receiving in the part where the ncm_mac_addr is used in the communication problem.

    Can you please clarify the below questions to help us investigate further:

    1. Is it only with this specific MAC address {0x00, 0xe0, 0x91, 0x02, 0x05, 0x99}, that the issue is seen? 

    2. Is this issue with ncm_mac_addr occuring - with the patched_PRU_fimware only? Or was this observed with the older firmware?

    3. Without using the ncm_mac_addr mac address, how else did you perform the testing? In other words, how did you root-cause to your above statement?

    4. Kim had shared 2 pcap files before and after the PRU firmware patch, where we see that there are no VLAN-ID 2/3 tagged frames incoming to device with Host MAC 00:0e:91:02:05:99 (ncm_mac_addr). But the two FDB entry added in the above code snippet use vlanId =2/3 (ICSSG_FDB_IOCTL_ADD_ENTRY). Is this the intended behaviour?

    Also, communication is not possible, either because there is no response to the request frame from PLC -> (1) HOST, or because the frame does not go up.

    5. Please share with us the HOST_PORT and MAC_PORT statistics of the particular device during this issue.

    Thanks and Regards,
    Miriam

  • 1. Is it only with this specific MAC address {0x000xe00x910x020x050x99}, that the issue is seen? 

    A: Yes. i think so.


    2. Is this issue with ncm_mac_addr occuring - with the patched_PRU_fimware only? Or was this observed with the older firmware?

    A: It occured only patched PRU firmware.


    3. Without using the ncm_mac_addr mac address, how else did you perform the testing? In other words, how did you root-cause to your above statement?

    A: We additionally tested Ethernet/IP, which is one of the several applications, and confirmed that both the Old and New PRU firmware functioned correctly.

    However, we found that only the Old firmware works with the dedicated application that uses the ncm_mac_addr MAC address.



    4. Kim had shared 2 pcap files before and after the PRU firmware patch, where we see that there are no VLAN-ID 2/3 tagged frames incoming to device with Host MAC 00:0e:91:02:05:99 (ncm_mac_addr). But the two FDB entry added in the above code snippet use vlanId =2/3 (ICSSG_FDB_IOCTL_ADD_ENTRY). Is this the intended behaviour?

    A: I'm not sure which files Kim shared, but it can be inferred that packet information is reaching the host in the Old PRU since the request and response appear to be paired. However, in the New PRU, it seems that the Am243x only performs the request, indicating that packet information is not reaching the host. (See attached diagram)

    5. Please share with us the HOST_PORT and MAC_PORT statistics of the particular device during this issue.

    A: I will ask Kim about it.

  • Hi Miriam,

    HOST_PORT and MAC_PORT statistics information are as follows.
    First, the statistics of PRU Firmware provided by ind_comms_sdk_am243x_09_02_00_08. (Normal communication)

    Port 0 statistics
    --------------------------------
    rxGoodFrames = 2857361
    rxBCastFrames = 1098915
    rxMCastFrames = 0
    rxUCastFrames = 1758446
    rxOddNibbleFrame = 0
    rxMaxSizeFrame = 6608000
    rxMinSizeFrame = 211456

    txGoodFrame = 4114296
    txBcastFrame = 80134
    txMcastFrame = 77506
    txUcastFrame = 3956656
    txOddNibbleFrame = 0
    txMaxSizeFrame = 6634432
    txMinSizeFrame = 211456

    rxCRCErrors = 0
    rxMIIErrors = 0
    rxMaxSizeErrFrame = 0
    rxMinSizeErrFrame = 1
    rxOverrunFrame = 0
    rxSMDFragErr = 0
    txUnderFlowErr = 0
    txMaxSizeErrFrame = 0
    txMinSizeErrFrame = 0

    Port 1 statistics
    --------------------------------
    rxGoodFrames = 2476404
    rxBCastFrames = 80558
    rxMCastFrames = 77852
    rxUCastFrames = 2317994
    rxOddNibbleFrame = 0
    rxMaxSizeFrame = 6636000
    rxMinSizeFrame = 212352

    txGoodFrame = 2317679
    txBcastFrame = 1103577
    txMcastFrame = 0
    txUcastFrame = 1214102
    txOddNibbleFrame = 0
    txMaxSizeFrame = 6662544
    txMinSizeFrame = 212352

    rxCRCErrors = 0
    rxMIIErrors = 0
    rxMaxSizeErrFrame = 0
    rxMinSizeErrFrame = 0
    rxOverrunFrame = 0
    rxSMDFragErr = 0
    txUnderFlowErr = 0
    txMaxSizeErrFrame = 0
    txMinSizeErrFrame = 0

    Host Port statistics
    --------------------------------
    hostRxByteCnt = 272913234
    hostTxByteCnt = 0
    hostRxPktCnt = 1745018
    hostTxPktCnt = 1664151
    rtu0PktDroppedSlice0 = 0
    rtu0PktDroppedSlice1 = 0
    port1Q0Overflow = 0
    port1Q1Overflow = 0
    port1Q2Overflow = 0
    port1Q3Overflow = 0
    port1Q4Overflow = 0
    port1Q5Overflow = 0
    port1Q6Overflow = 0
    port1Q7Overflow = 0
    port2Q0Overflow = 0
    port2Q1Overflow = 0
    port2Q2Overflow = 0
    port2Q3Overflow = 0
    port2Q4Overflow = 0
    port2Q5Overflow = 0
    port2Q6Overflow = 0
    port2Q7Overflow = 0
    hostQ0Overflow = 0
    hostQ1Overflow = 0
    hostQ2Overflow = 0
    hostQ3Overflow = 0
    hostQ4Overflow = 0
    hostQ5Overflow = 0
    hostQ6Overflow = 0
    hostQ7Overflow = 0
    hostEgressQPreOverflow = 0
    droppedPktSlice0 = 0
    droppedPktSlice1 = 0
    rxErrorSlice0 = 0
    rxErrorSlice1 = 0
    rxEofRtuDsInvalidSlice0 = 0
    rxEofRtuDsInvalidSlice1 = 0
    txPort1DroppedPkt = 0
    txPort2DroppedPkt = 0
    txPort1TsDroppedPkt = 0
    txPort2TsDroppedPkt = 0
    infPortDisabledSlice0 = 0
    infPortDisabledSlice1 = 0
    infSavSlice0 = 0
    infSavSlice1 = 0
    infSaBlSlice0 = 0
    infSaBlSlice1 = 0
    infPortBlockedSlice0 = 0
    infPortBlockedSlice1 = 0
    infAftDropTaggedSlice0 = 0
    infAftDropTaggedSlice1 = 0
    infAftDropPrioTaggedSlice0 = 0
    infAftDropPrioTaggedSlice1 = 0
    infAftDropNoTagSlice0 = 0
    infAftDropNoTagSlice1 = 0
    infAftDropNotMemberSlice0 = 0
    infAftDropNotMemberSlice1 = 0
    fdbNoSpaceToLearn = 0
    preemptBadFragSlice0 = 0
    preemptBadFragSlice1 = 0
    preemptAsmErrSlice0 = 0
    preemptAsmErrSlice1 = 0
    preemptFragCntTxSlice0 = 0
    preemptFragCntTxSlice1 = 0
    preemptAsmOkSlice0 = 0
    preemptAsmOkSlice1 = 0
    preemptFragCntRxSlice0 = 0
    preemptFragCntRxSlice1 = 0
    rxEofShortFrameErrSlice0 = 0
    rxEofShortFrameErrSlice1 = 0

    The statistics of patch PRU Firmware. (Unable to communicate)
    Port 0 statistics
    --------------------------------
    rxGoodFrames = 475779
    rxBCastFrames = 441321
    rxMCastFrames = 0
    rxUCastFrames = 34458
    rxOddNibbleFrame = 0
    rxMaxSizeFrame = 4574000
    rxMinSizeFrame = 146368

    txGoodFrame = 542815
    txBcastFrame = 487548
    txMcastFrame = 54270
    txUcastFrame = 997
    txOddNibbleFrame = 0
    txMaxSizeFrame = 4592296
    txMinSizeFrame = 146368

    rxCRCErrors = 0
    rxMIIErrors = 0
    rxMaxSizeErrFrame = 0
    rxMinSizeErrFrame = 0
    rxOverrunFrame = 0
    rxSMDFragErr = 0
    txUnderFlowErr = 0
    txMaxSizeErrFrame = 0
    txMinSizeErrFrame = 0

    Port 1 statistics
    --------------------------------
    rxGoodFrames = 546415
    rxBCastFrames = 490855
    rxMCastFrames = 54595
    rxUCastFrames = 969
    rxOddNibbleFrame = 0
    rxMaxSizeFrame = 4610000
    rxMinSizeFrame = 147520

    txGoodFrame = 445165
    txBcastFrame = 444256
    txMcastFrame = 4294967295
    txUcastFrame = 910
    txOddNibbleFrame = 0
    txMaxSizeFrame = 4628440
    txMinSizeFrame = 147520

    rxCRCErrors = 0
    rxMIIErrors = 0
    rxMaxSizeErrFrame = 0
    rxMinSizeErrFrame = 0
    rxOverrunFrame = 0
    rxSMDFragErr = 0
    txUnderFlowErr = 0
    txMaxSizeErrFrame = 0
    txMinSizeErrFrame = 0

    Host Port statistics
    --------------------------------
    hostRxByteCnt = 234637571
    hostTxByteCnt = 0
    hostRxPktCnt = 1014000
    hostTxPktCnt = 1951
    rtu0PktDroppedSlice0 = 0
    rtu0PktDroppedSlice1 = 0
    port1Q0Overflow = 0
    port1Q1Overflow = 0
    port1Q2Overflow = 0
    port1Q3Overflow = 0
    port1Q4Overflow = 0
    port1Q5Overflow = 0
    port1Q6Overflow = 0
    port1Q7Overflow = 0
    port2Q0Overflow = 0
    port2Q1Overflow = 0
    port2Q2Overflow = 0
    port2Q3Overflow = 0
    port2Q4Overflow = 0
    port2Q5Overflow = 0
    port2Q6Overflow = 0
    port2Q7Overflow = 0
    hostQ0Overflow = 0
    hostQ1Overflow = 0
    hostQ2Overflow = 0
    hostQ3Overflow = 0
    hostQ4Overflow = 0
    hostQ5Overflow = 0
    hostQ6Overflow = 0
    hostQ7Overflow = 0
    hostEgressQPreOverflow = 0
    droppedPktSlice0 = 0
    droppedPktSlice1 = 0
    rxErrorSlice0 = 0
    rxErrorSlice1 = 0
    rxEofRtuDsInvalidSlice0 = 0
    rxEofRtuDsInvalidSlice1 = 0
    txPort1DroppedPkt = 0
    txPort2DroppedPkt = 0
    txPort1TsDroppedPkt = 0
    txPort2TsDroppedPkt = 0
    infPortDisabledSlice0 = 0
    infPortDisabledSlice1 = 0
    infSavSlice0 = 0
    infSavSlice1 = 0
    infSaBlSlice0 = 0
    infSaBlSlice1 = 0
    infPortBlockedSlice0 = 0
    infPortBlockedSlice1 = 0
    infAftDropTaggedSlice0 = 0
    infAftDropTaggedSlice1 = 0
    infAftDropPrioTaggedSlice0 = 0
    infAftDropPrioTaggedSlice1 = 0
    infAftDropNoTagSlice0 = 0
    infAftDropNoTagSlice1 = 0
    infAftDropNotMemberSlice0 = 0
    infAftDropNotMemberSlice1 = 0
    fdbNoSpaceToLearn = 0
    preemptBadFragSlice0 = 0
    preemptBadFragSlice1 = 0
    preemptAsmErrSlice0 = 0
    preemptAsmErrSlice1 = 0
    preemptFragCntTxSlice0 = 0
    preemptFragCntTxSlice1 = 0
    preemptAsmOkSlice0 = 0
    preemptAsmOkSlice1 = 0
    preemptFragCntRxSlice0 = 0
    preemptFragCntRxSlice1 = 0
    rxEofShortFrameErrSlice0 = 0
    rxEofShortFrameErrSlice1 = 0

    All test environments are the same, and these are the results when only the PRU Firmware was changed.
    (For the same environment, it is STF mode, not CUT-THR mode.)

    Thanks and Regards,
    Kim

  • Hi Changjun im, Kim,

    Thank you sharing the statistics details and clarifying the questions.

    From the image shared above, we have understood that below is the topology:
    In working case:
    PLC (LSLG:....:ae)  <---------------->    AM243-1( AC&T:..:04) < ------Packet capture--------> AM243-2 (LGElectr:....:99)
    [ Sends Request to LGElectr ]          [Sends Response toLGElectr ]                                [Receives both NCM Request and Response]

    And in the case with the patched firmware, we had discussed internally on the potential reason for NCM response not being sent from AM243-1.
    We have some recent changes in our mainline with respect to this potential behaviour. And thus want to share a patch-2 with you to test in your test environment. 
    Please expect the firmware patch-2 to be shared by tomorrow.

    Thanks and Regards,
    Miriam

  • I think there is a misunderstanding

  • Okay Thank you for clarifying.

    We have shared the firmware patch -2 via e-mail to Kim. Please test with this firmware and you can let us know the results.

    Thanks and Regards,
    Miriam

  • Thank you for your quick response.
    
    However, the firmware seems to have been modified based on a misunderstanding about ncm mac, so I'm not sure if it will work correctly.
    
    I will contact Kim and check.
    
    thank you.
  • Hi Changjun Im, Kim

    Yes, but the latest firmware shared has relevant updates from the last patch so we request you to try and let us know of the results.

    Changjun Im, please let us know if you want to be added to the e-mail thread with the firmware shared.

    Thanks and Regards,
    Miriam

  • We conducted a ring/line network switchover test with only the 'B' patch applied.

    The test started successfully (as it seems the NCM MAC address packets are being sent and received properly given that the test setup was completed).

    However, after running the test repeatedly for one hour, a network disconnection was observed.

    Considering that the Ping test and Modbus test (one of our applications) appear to function normally, it does not seem to be a switch stuck issue.

    Since crucial information in the ring/line switchover test is exchanged using packets that utilize the NCM MAC address, it seems the NCM MAC address issue is not completely resolved.

    For Port1, the NCM MAC address packets appear to be sent and received correctly, but this is not the case for Port2.

    I didn't have time to check further, but forwarding between Port1 and Port2 might have been blocked in either direction.

    We plan to conduct an aging test in a normal ring state, not during a ring/line switch switchover. However, we hope TI can resolve this issue beforehand.

    We have the final pre-production test scheduled for this Friday.

    Please add to e-mail thread.

    Thanks and Regards,
    CJ

  • Thanks for the updates.

    Please check if this is the right understanding of the behavior:

    1. The switchover test was conducted for multiple iterations for a duration of 1 hour. After say 'n' successful iterations in 1 hour, a network disconnection was observed in the subsequent iteration.
    2. In each of the successful iteration, the NCM MAC address packets are being sent and received successfully in case of both Port-1 and Port-2.
    3.The network disconnection occurred due to the NCM packets not being sent/received on Port -2. (This is on the AM243 which is adjacent to the PLC?)
    4. During each switch-over, is there any re-initialization of the application/ICSSG/ Soft-reset in all devices ?

    Please do share following details when reporting a new behaviour for us to investigate (if you want to share via e-mail feel free to do so) :

    1. The HOST and MAC port statistics on the device during the network disconnection.
    2. Description of the topology, if any difference and
    3. Expected vs actual behaviour with packet capture file. (along with any relevant console logs)
    4. Also for this issue please try to dump the FDB contents after the the issue on the device and share with us. Reference ( File: C:\ti\mcu_plus_sdk_am243x_09_02_00_50\examples\networking\enet_layer2_icssg\enet_layer2_icssg.c   EnetMp_readAllFdbSlots() )

    Thanks and Regards,
    Miriam