Part Number: AM2432

DLR Backup Manager Communication Loss — Problem Analysis Report

1. Summary

This report analyzes two independent defects, both observed sporadically only during the DLR stress test of repeatedly power-cycling ordinary ring nodes:

Issue 2 directly matches the field symptom: the Manager reports the DLR ring as healthy, yet the Backup Manager cannot be pinged and its EIP I/O connection drops and cannot reconnect.

2. Test Topology

M.port1 ——— N1 —— N2 —— N3 —— N4 ——— B.port1
M.port2 —————————— direct cable —————————— B.port2

• M: Manager (Active Ring Supervisor, precedence 1)

• B: Backup Manager (Backup Ring Supervisor, precedence 0)

• N1..N4: ordinary ring nodes (non-TI devices, ring-node functionality only)

• Normally M blocks M.port2; data flows M.port1 — node chain — B.port1; the M.port2↔B.port2 direct cable carries only Beacon/DLR frames.

3. Symptoms

1. Occurs sporadically after repeatedly power-cycling the ordinary nodes (the Manager and Backup Manager stay powered).

2. The Manager reports the DLR ring as normal, but the Backup Manager is unreachable.

3. The Backup Manager times out pinging any device (Manager, nodes); the Manager also times out pinging the Backup Manager.

4. The Backup Manager's EIP I/O communication fails at the same time (I/O data is sent via UDP/LwIP, sharing the host TX queue with ARP/ICMP).

5. Breaking and then restoring the ring restores communication, without rebooting the device.

Reproduction: two TI managers configured as Active / Backup Ring Supervisor (both blocking port 2), four ordinary nodes chained in the ring. Keep both managers powered and repeatedly power-cycle the nodes (a few seconds per round). Issue 1 hit 2 times in 12 link-down events; Issue 2 appeared multiple times in the same stress run (always after link recovery).

4. Issue 1: DLR Backup Manager Spurious Takeover on Single-Port Link Down

4.1 Relevant mechanisms in the stock code

(1) Link-down handling

The P0/P1 link-break handlers in icss_dlr.c (EIP_DLR_port0ProcessLinkBrk() / EIP_DLR_port1ProcessLinkBrk()) set the node state to NODE_FAULT, stop that port's IEP watchdog, flush the learning table, and report a ring fault. This path itself is correct.

(2) Beacon-timeout ISR unconditionally re-arms the watchdog

At the entry of EIP_DLR_beaconTimeoutISR_P0/P1 (icss_dlr.c:873 for P0, icss_dlr.c:1022 for P1):

/*Clear the interrupt*/
HW_WR_REG32(intStatusOffset, PORT0_WD_ISR_MASK);
...
/*Need to restart the Watchdog because it's not an auto load timer*/
EIP_DLR_disableIEPWatchdog(dlrHandle, PORT0_WATCH_DOG_ID);
EIP_DLR_enableIEPWatchdog(dlrHandle, PORT0_WATCH_DOG_ID);   /* no link check */

The IEP watchdog is not an auto-reload timer, so it must be re-armed after expiry — hence the unconditional disable + enable. The problem is that this happens before any state/link check: if a port is already down (and will never see a Beacon again), one timeout interrupt that was already pending at link-down (or arrives right after) still enters the ISR and re-enables the watchdog; from then on that port keeps generating timeout interrupts every beaconTimeout (currently 4000 µs) with no way to stop itself.

(3) Backup takeover decision ignores the other port

In EIP_DLR_beaconTimeoutSupProcessing() (icss_dlr_sup.c:880 onwards), the "backup supervisor in FAULT state" branch uses only isSwitchOverPeriodActive to distinguish "first timeout" from "continuous timeout":

if(isSwitchOverPeriodActive == DLR_TRUE)
{
    /* Second and subsequent timeouts: declare the active manager lost */
    EIP_DLR_disableIEPWatchdog(dlrHandle, portNum - 1);
    EIP_DLR_resetStateMachine(dlrHandle);
    EIP_DLR_configSupervisor(dlrHandle);        /* set ACTIVE, send Beacons */
    isSwitchOverPeriodActive = DLR_FALSE;
}
else
{
    /* First timeout: enter the switchover window, stop both watchdogs, enable only P0 */
    EIP_DLR_disableIEPWatchdog(dlrHandle, PORT0_WATCH_DOG_ID);
    EIP_DLR_disableIEPWatchdog(dlrHandle, PORT1_WATCH_DOG_ID);
    EIP_DLR_enableIEPWatchdog(dlrHandle, PORT0_WATCH_DOG_ID);
    isSwitchOverPeriodActive = DLR_TRUE;
}

Nothing in this decision checks whether the other port is still receiving Beacons (nor whether this port's own link is still valid).

4.2 Defect walk-through

With the two code paths above, a single-port link down is enough to trigger takeover:

1. The node-side port (P1/PHY1) goes down → state machine enters NODE_FAULT, P1 watchdog is stopped;

2. But the Beacon-timeout interrupt that was already pending at link-down (or arrives right after) still enters the ISR, which re-enables the P1 watchdog unconditionally at entry;

3. P1 is down, so no Beacon arrives and timeouts recur every 4 ms, re-entering the FAULT branch;

4. The first timeout arms the switchover window (isSwitchOverPeriodActive = TRUE); the second executes resetStateMachine + configSupervisor and takes over as Active Supervisor;

5. Meanwhile the Manager's Beacons on P2 never stopped. Per DLR semantics, the backup may take over only when neither port receives Beacons;

6. The firmware then clears the ACTIVE bit by precedence (backup 0 < manager 1), and the host runs transitionToBackupSupervisor() on the idle→fault event, demoting back to Backup within about 1 ms.

4.3 Field event trace

[55347868] P1 LINK DOWN drvLS=01
[55347868] DLR_P1_BRK link=0 node=NORMAL en:status=0x00230000
[55347868] FDB FLUSH port1 entries=33
[55347868] DLR -> FAULT  node=FAULT ring=2
[55347868] BCN_TO_P1 node=FAULT sm=0x0201
[55347872] BCN_TO_P1 node=FAULT sm=0x0201
[55347872] DLR RESET SM (previous node=FAULT)
[55347872] DLR RESET SM (previous node=IDLE)
[55347872] SUP -> ACTIVE   EN=0x27        <-- spurious takeover
[55347873] ISR_P2 evts=0x0108 ... [RING_FAULT_TRANS STOP_BOTH_TMR ...]
[55347873] SUP -> BACKUP   EN=0x23        <-- demoted about 1 ms later

• Within the same observation window: P1 LINK DOWN 12 times, BCN_TO_P1 7 times, 2 spurious takeovers (about 1 in 6; sporadic).

• During the same period the Manager side shows no role change, so the churn happens only on the backup side.

4.4 Impact

• Each hit produces a brief "dual supervisor / role flap" on the ring, and the Backup Manager sends its own Beacons;

• The takeover path calls EIP_DLR_dRAMInit(), which clears the ActiveSup MAC/IP in SMEM; the demotion path does not restore them; updateSupervisorDetails() may write addr.supMACAddress and the learning exception table as 00:00:00:00:00:00;

• CIP status, the participant table, and ring-fault counters are repeatedly reset, degrading long-term DLR network stability.

4.5 Fix (local)

Genuine manager-loss scenarios are unaffected: when both ports go silent, the other port times out too, the criterion fails, and takeover still happens; worst-case takeover time increases from about 2× to 3× beaconTimeout.

4.6 Verification

With the new firmware under the same stress test (repeated node power cycling), the event trace no longer shows SUP -> ACTIVE; on P1 link down it shows BCN_TO P1 ignored (link down); legitimate takeovers occur only when the Manager side is truly silent.

5. Issue 2: PRU Firmware Host TX Queue (Q3) Deadlock

5.1 Stock queue protocol and driver TX path

(1) Queue descriptor (PRU1 DRAM, shared by host and firmware)

In addition, each port has a collision slot at colStatusAddr+1/+2: when the target queue is busy the host moves the frame into the collision queue and writes (queue<<1)|1; the firmware clears it after consuming the collision queue.

(2) Driver TX path (MCU+ SDK icss_emac.c, ICSS_EMAC_txPacketEnqueue(), switch mode)

/* 1. Read status.busy_m */
collision_queue_selected = (temp & 0x00000100U);      /* Check the "busy_m" bit */
if(collision_queue_selected != 0U)
{
    hostStatPtr->txNumCollision++;
    /* 2. Check whether this port's collision slot is already occupied */
    col_queue_already_occupied = HW_RD_REG8(colStatusAddr + port);
    if(col_queue_already_occupied != 0)
    {
        hostStatPtr->txDroppedPackets++;
        hostStatPtr->txCollisionDroppedPackets++;
        return ICSS_EMAC_ERR_COLLISION_FAIL;           /* No space in collision queue */
    }
}
...
/* 3. If free, put the frame into the collision queue and set the port's collision slot */
collision_status = (queuePriority << 1) | 0x0001U;
HW_WR_REG8(colStatusAddr + port, collision_status);

So: once the firmware leaves a queue's busy_m set indefinitely, every transmission takes the collision path; the host sets the collision slot on the first attempt, the firmware never clears it, and every subsequent transmission is dropped at step 2 with ERR_COLLISION_FAIL, while txDroppedPackets/txCollisionDroppedPackets keep increasing.

5.2 Field evidence

(1) TX counter deltas around a ping (Backup Manager port1; driver statistics plus TX return-code counts)

That is: the link is up (no NO_LINK drops) and queues are not full, yet every transmission fails on the collision path — an exact match for the code path in 5.1. Over the same period port2 shows txBC +534 (all broadcasts leave via port 2, which faces the Manager's blocked port).

(2) Raw queue descriptor values in the fault state (read from PRU1 DRAM)

P1 Q0 wr=1088 rd=1088 used=0 busy=0 busym=0 ovf=0
P1 Q1 wr=11a4 rd=11a4 used=0 busy=0 busym=0 ovf=0
P1 Q2 wr=1328 rd=1328 used=0 busy=0 busym=0 ovf=0
P1 Q3 wr=15a4 rd=15a4 used=0 busy=0 busym=1 ovf=0
P1 colStatus=7   (= (3<<1)|1, Q3 collision slot asserted)

• The deadlocked queue is P1 Q3 (ICSS_EMAC_QUEUE4, the LwIP transmit queue);

• The queue itself is empty (wr==rd, used=0) — no data backlog; the stuck part is the handshake bits;

• status.busy_m=1 and colStatus=7 are held together while all other queues are normal.

(3) Formation time

[1918964] P1 LINK UP drvLS=101
[1919003] DLR -> NORMAL node=NORMAL ring=1
[1919919] *** TXQ P1 STUCK colStatus=7 busy_m=1 ***

The deadlock forms after link restore, DLR returning to NORMAL, and the host resuming enqueues to Q3 — i.e., the firmware fails to complete or properly abort an in-flight TX transaction across the link/ring state transition.

5.3 Ownership conclusion

Therefore the indefinitely held busy_m=1 and collision-slot assertion are a firmware state machine that failed to converge, not a driver logic error; link events, port control registers, and DLR state transitions never make the firmware release these two states.

5.4 Impact

• Q3 is the LwIP transmit queue; in this product EIP I/O data is sent over UDP/LwIP, sharing Q3 with ARP/ICMP → I/O and ping fail together (the PLC-side connection drops and data stops updating);

• DLR/Beacon (firmware-internal path) and L2 forwarding are unaffected, which is why the Manager still sees the ring as healthy;

• The receive direction is fine: the learning table still learns the peer MAC and the DLR state machine works.

5.5 Recovery validation and host-side workaround

• Experiment: in the busy_m=1 && wr==rd state, having only the host clear the busy_m bit in status, clear the colStatus collision slot, and set wr = rd restores transmission immediately (collision counters stop, ping and EIP I/O recover);

• This also proves the PHY, driver TX path, and LwIP are all fine; the only unreleased states are the two handshake states owned by the firmware;

• A host-side workaround can therefore periodically check queue descriptors and, after several consecutive samples (about 600 ms) of busy_m=1 && wr==rd, perform the same clear; the outage per event is shortened from "permanent" to under 1 s;

• Limits of the workaround: all Q3 transmissions still fail during the 600 ms window; if the PLC connection timeout is shorter than that, the connection still drops and reconnects; repeated deadlocks on the same port within a short interval require throttling the recovery action (to avoid repeatedly clearing firmware state).

Risk note: this clears firmware-owned handshake bits and is used only in the verified "busy_m=1 && used=0" case; rebooting the device recovers from any anomaly.

5.6 Recommended TI fix

The PRU firmware should complete or roll back in-flight host TX transactions on link down/up, port reset, and DLR state transitions, releasing the queue busy_m and the corresponding collision-slot assertion; at minimum it must not leave a "queue empty but busy_m=1" state after link recovery. Attach the field data and descriptor records from sections 5.1–5.3.

6. Excluded Hypotheses

7. Requests to Texas Instruments

1. Confirm Issue 1 and provide a driver fix: on a single-port link down, the backup supervisor's beacon-timeout ISR must not re-enable the watchdog of the down port, and the takeover decision must verify that neither port is receiving Beacons. Please confirm the change direction in icss_dlr.c / icss_dlr_sup.c and provide an SDK release containing the fix.

2. Fix Issue 2 in the PRU firmware: on link down/up, port reset, and DLR state transitions, complete or roll back any in-flight host TX transaction and release the queue status.busy_m plus the corresponding collision-slot assertion. Please confirm whether "queue empty (wr==rd) while busy_m=1 and colStatus remains asserted" is a known firmware defect.

3. Confirm interface semantics: is the colStatusAddr+1/+2 collision slot guaranteed to be cleared by the firmware after it consumes the collision queue? Is status.busy_m only cleared by the firmware's internal transmit-complete event?