Other Parts Discussed in Thread: SYSBIOS
We have an issue where our EtherCAT Master does not see any slaves because the PRU ICSS driver is stopping receiving frames after a ESD test until we reboot our target.
Using WireShark we see that the EtherCAT Master is sending its messages successfully, it is getting the reply from the slaves but for some reason the EtherCAT Master application does not receive the reply and then returning 0 slave.
Prior to that the Master sees all slaves. Once we do the ESD test, it seems as the Master cannot receive any more ETherCAT frames although it sends frames successfully. This is not systematic but easily repeatable.
Rebooting the system is not acceptable for our application. After analysis, here the statistics that we have found:
it seems that after the reception of a bad CRC frame or a first misaligned frame, the PRU ICSS does not recover from it always receiving misaligned frames.
-- 120sec aSt:Idle nSl:7 TL: 0 3TL: 0 STL: 0--
txBcast: 239751 txMcast: 0 txUcast: 0 txOctets: 106407584 rxBcast: 239763 rxMcast: 0 rxUcast: 0 rxOctets: 106414344--
tx64byte: 82065 tx65_127byte: 1 tx128_255byte: 830 tx256_511byte: 78446 tx512_1023byte: 0 tx1024byte: 78449--
rx64byte: 82075 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 78456 rx512_1023byte: 0 rx1024byte: 78459--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 0 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 0 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--
-- 125sec aSt:Idle nSl:7 TL: 0 3TL: 0 STL: 0--
txBcast: 254604 txMcast: 0 txUcast: 0 txOctets: 113101336 rxBcast: 254616 rxMcast: 0 rxUcast: 0 rxOctets: 113108096--
tx64byte: 87016 tx65_127byte: 1 tx128_255byte: 830 tx256_511byte: 83397 tx512_1023byte: 0 tx1024byte: 83400--
rx64byte: 87026 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 83406 rx512_1023byte: 0 rx1024byte: 83410--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 0 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 0 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--
-- 130sec aSt:Idle nSl:7 TL: 0 3TL: 0 STL: 0--
txBcast: 269463 txMcast: 0 txUcast: 0 txOctets: 119797792 rxBcast: 269475 rxMcast: 0 rxUcast: 0 rxOctets: 119804552--
tx64byte: 91969 tx65_127byte: 1 tx128_255byte: 830 tx256_511byte: 88350 tx512_1023byte: 0 tx1024byte: 88353--
rx64byte: 91979 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 88359 rx512_1023byte: 0 rx1024byte: 88363--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 0 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 0 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--
D: ECAT: master state change from 'Op' to 'Unknown' at 131655885
D: EthercatMaster: Unknown: Setting Ethercat Master State to Initialization at 131656358
D: ECATM: Req Master State From 'Unknown' to 'Init' at 131664020
-- 135sec aSt:Idle nSl:0 TL: 0 3TL: 0 STL: 0--
txBcast: 276385 txMcast: 0 txUcast: 0 txOctets: 122118766 rxBcast: 274323 rxMcast: 0 rxUcast: 0 rxOctets: 121985328--
tx64byte: 95640 tx65_127byte: 1 tx128_255byte: 833 tx256_511byte: 89957 tx512_1023byte: 0 tx1024byte: 89957--
rx64byte: 93580 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 89956 rx512_1023byte: 0 rx1024byte: 89956--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 2073 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 1 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--
E: ECATM: Req Master State From 'Unknown' to 'Init' Failed at 135721777
D: ECATM: Req Master State From 'Unknown' to 'Init' at 136768452
E: ECATM: Req Master State From 'Unknown' to 'Init' Failed at 141063808
-- 140sec aSt:Fault SF: 0x00000004 nSl:0 TL: 0 3TL: 0 STL: 0--
txBcast: 278659 txMcast: 0 txUcast: 0 txOctets: 122264238 rxBcast: 274323 rxMcast: 0 rxUcast: 0 rxOctets: 121985328--
tx64byte: 97911 tx65_127byte: 1 tx128_255byte: 833 tx256_511byte: 89957 tx512_1023byte: 0 tx1024byte: 89957--
rx64byte: 93580 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 89956 rx512_1023byte: 0 rx1024byte: 89956--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 4336 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 1 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--
-- 145sec aSt:Fault SF: 0x00000004 nSl:0 TL: 0 3TL: 0 STL: 0--
txBcast: 280616 txMcast: 0 txUcast: 0 txOctets: 122389486 rxBcast: 274323 rxMcast: 0 rxUcast: 0 rxOctets: 121985328--
tx64byte: 99868 tx65_127byte: 1 tx128_255byte: 833 tx256_511byte: 89957 tx512_1023byte: 0 tx1024byte: 89957--
rx64byte: 93580 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 89956 rx512_1023byte: 0 rx1024byte: 89956--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 6293 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 1 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--
-- 150sec aSt:Fault SF: 0x00000004 nSl:0 TL: 0 3TL: 0 STL: 0--
txBcast: 282573 txMcast: 0 txUcast: 0 txOctets: 122514734 rxBcast: 274323 rxMcast: 0 rxUcast: 0 rxOctets: 121985328--
tx64byte: 101825 tx65_127byte: 1 tx128_255byte: 833 tx256_511byte: 89957 tx512_1023byte: 0 tx1024byte: 89957--
rx64byte: 93580 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 89956 rx512_1023byte: 0 rx1024byte: 89956--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 8250 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 1 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--
So it seems that the problem is more TI related than EtherCAT related.
What do you think? Is this a known issue? How can I address it?
I did test a solution that calls PRUICSS_pruDisable(), PRUICSS_pruReset(), PRUICSS_pruEnable() and then restart the master but it did not solve the issue (I tried only disable/enable and only reset and both).
Thanks,
Sylvain