AM4376: Unexpected behavior when processing DLR frames in case of link loss/recovery

Part Number: AM4376

Hello experts,

we are using the EtherNet/IP adapter v1.0.3.4 with the patched firmware v5.2 on a AM4376 device. When executing the EtherNet/IP Conformance Test CT17 the DLR related tests fail due to unexpected behaviors when processing DLR frames in case of a ring break / recovery. Up to now, I observed three different weird cases:

1. In case of a link loss at port A, a "Neighbor_Check_Request" received on port B is not processed and no „Neighbor_Check_Response“ ist transmitted (on port B).
2. In case of a link recovery at port A (after a link loss), a "Sign_On" received on port A is not processed and not forwarded on port B.
3. Sometimes, a strange combination of 1. and 2. occurs: In case of a link loss on port A (and a missing „Neighbor_Check_Response“ on port B), the link is recovered and the missing „Neighbor_Check_Response“ is then sent instead of forwarding the „Sign_On“ frame.

Port A and B are placeholders for our device Ethernet ports, where A and B may be either DLR port 0 or 1. This behavior is not static and varies from always to sporadical, depending on where our device is located in the DLR ring.The test setup definition is used from the EtherNet/IP sample test report that is delivered with the CT17 conformance test setup.

In order to start analysis, I reduced the test setup: Our device (DUT), the ring supervisor, one additional ring node and two managed switches for capturing traffic (see test_setup.png). Then I picked out case 3 and made some Wireshark captures using the port mirror function of the two embedded switches (see trace_setup_3_port_x2_dlr_port_0_ERR.pcapng and trace_setup_3_port_x1_dlr_port_1_OK.pcapng). Both switches are configured with learning disabled, forward VLAN tagged frames and mirror rx traffic from all ports tot the target port. So the duplicate frames in the capture files result from rx mirror of two switch ports.

Note 1: „trace_setup_3_port_x1_dlr_port_1_OK.pcapng“ does not show any VLAN tags, because the network interface card (NIC) dismisses VLAN tags. Iverified the presence of VLAN tags using the second NIC used for capturing „trace_setup_3_port_x2_dlr_port_0_ERR.pcapng“.

Note 2: Our device has two Ethernet ports labled with X1 and X2. Due to layout considerations, X1 is connected to DLR port 1 and X2 is connected to DLR port 0. This does not affect the internal DLR operation, only the port numbers displayed in Link_Status / Neighbor_Status frames are different.

Analysis

The error case is shown in „trace_setup_3_port_x2_dlr_port_0_ERR.pcapng“, when the Ethernet cable is disconnected and re-connected on port X1 (= DLR port 0). The capture file shows 6 connect/re-connect events at X1 which show the correct behavior (see frame 2830 ff, 3179 ff, 3585 ff, …). But on the 7th event (frame 5073 ff) something goes wrong:

1. The DUT sends a „Link_Status / Neighbor_Status“ message (frame 5073) on the active port X2 (= DLR port 0). So far so good.
2. The ring supervisor then sends a „Locate_Fault“ message (frame 5092) and the DUT responds with another „Link_Status / Neighbor_Status“ message (frame 5094). This is also correct.
3. Relatively soon, the ring node connected to the active port X2 sends a „Neighbor_Check_Request“ message to port X2, which is not answered by the DUT. The ring node repeats this two more times (delay = 100 ms) without any reaction from the DUT.
4. The ring node then gives up and notifies the related port as inactive („„Link_Status / Neighbor_Status“, frame 5107).

Having a look at the DLR driver, the following actions can be observed for the disconnect event:

• EIP_DLR_port0ISR: commonEventsFlag = 0x01, portEvtsFlag = 0x0100 => Ring-normal transition (?)
• EIP_DLR_port0ISR: commonEventsFlag = 0x08, portEvtsFlag = 0x0110 => Ring-fault detected
• EIP_DLR_port0ISR: commonEventsFlag = 0x08, portEvtsFlag = 0x0101 => „Locate_Fault“ received

The driver does not seem to receive an event indicating a „Neighbor_Check_Request“. Misteriously, it gets an event showing a ring normal transition. When reconnecting the Ethernet cable to port X1, again something goes wrong on the bus:

1. The ring supervisor sends a „Sign_On“ message from its active port (1) tot he additional ring node which forwards it tot he re-connected DUT port X1.
2. The DUT forwards the „Sign_On“ to port X2 (see „trace_setup_3_port_x1_dlr_port_1_OK.pcapng“, frame 5086).
3. But additionally, the DUT sends a „Neighbor_Check_Response“ message to port X2 containing the same Sequence Id from the „Sign_On“ message received before.

Again having a look at the DLR driver, the following actions can be observed for the reconnect event:

• EIP_DLR_port1ISR: commonEventsFlag = 0x18, portEvtsFlag = 0x0008 => Beacon frame received on DLR port 0/1
• EIP_DLR_port1ISR: commonEventsFlag = 0x09, portEvtsFlag = 0x0100 => Beacon frame received on DLR port 0; Ring-normal transition
• EIP_DLR_processDLRFrame: portEvtsFlag = 0x0102, signOnRcvdFlag = 1, portNum = 0 => „Neighbor_Check_Request“ received (?), „Sign_On“ received (OK)

Remarks

• Based on my obvervations, the issue seems to be favored by the additional ring node of type Allen Bradley 5069-AENTR.
• Up to now, I see this behavior when using a different additional ring nodes. But I can not see any reason in the Wireshark capture, why the 5069-AENTR could cause this kind issue.