AM2432: CRC error and frame time outs

Hi Jianyu 

Can you share pcap files for both scenarios from INT to OP

Regards

Pratheesh

All nodes except 42-44 are AM2432

All #define ENABLE_MULTIPLE_SM_ACCESS_IN_SINGLE_DATAGRAM     0

26_datagram_0.zip

node 46 and 106 is #define ENABLE_MULTIPLE_SM_ACCESS_IN_SINGLE_DATAGRAM     0

24_datagram0_2_datagram1.zip

Jianyu Wang said:

node 46 and 106 is #define ENABLE_MULTIPLE_SM_ACCESS_IN_SINGLE_DATAGRAM     0

24_datagram0_2_datagram1.zip

You meant ENABLE_MULTIPLE_SM_ACCESS_IN_SINGLE_DATAGRAM set to 1 here - right? Which version of PRU firmware used from ESC register 0x2? Will check on LRD failure seen in this pcap.

PratheeshGangadhar said:

You meant ENABLE_MULTIPLE_SM_ACCESS_IN_SINGLE_DATAGRAM set to 1 here - right?

Yes. The version is ind_comms_sdk_am243x_09_01_00_03

Hi,

Jianyu Wang said:

Yes. The version is ind_comms_sdk_am243x_09_01_00_03

Based on the above thread discussion, I believe you have picked the EtherCAT firmware from ind_comms_sdk_am243x_11_00_00_13? For firmware version, you can also read the first 4 bytes in the ESC Register space

Regards,
Aaron

version is 0x532. We revert it back as it seems working with older version.

Aaron Thomas said:

Based on the above thread discussion, I believe you have picked the EtherCAT firmware from ind_comms_sdk_am243x_11_00_00_13?

We have switched back to this version and tested last night. The test setting is 86 drives, 23 of them are AM2432.

The frame timeouts is as follows.

CRC errors

When ENABLE_MULTIPLE_SM_ACCESS_IN_SINGLE_DATAGRAM  is set back to 0, frame time outs and crc errors are 0 for 3 days.

Hi Jianyu,

Thank you for the details. Do note that ENABLE_MULTIPLE_SM_ACCESS_IN_SINGLE_DATAGRAM is to be set only in case of the following access by the EtherCAT MainDevice:

• Single datagram accessing multiple FMMU mapped areas using LRD/LWR commands from a single SubDevice
• LRW access to non-interleaved input and output process data of multiple SubDevices

Because our product is general servo drive. We cannot know in advance if our customer is using Omron or H5U or any other EtherCAT Master. So we would like one setting which can support all EtherCAT Masters.

ver 0x542 is tested. Still CRC errors and Frame timeouts

Thank you Jianyu for the update.

Will discuss further on the above issue offline.

Regards,
Aaron

Hi,

Any update on the issue?

Hi Jianyu,

As discussed offline, can we get the update on testing with reduced TX_START_DELAY (0x48) along with the latest EtherCAT firmware (0x0542) against Omron and Innovance PLC.

Please provide the ICSSG0 Memory dump (0x30000000 to 0x30040000) and also if you have the wireshark logs with the above mentioned test configuration.

Regards,
Aaron

As discussed offline, please test with the patch provided which has fix for the TX_START_DELAY bug in link break scenarios.

Regards,
Aaron

Not working. It seems after the cable is reconnected, the WKC from returned frame is 0

pru544.zip

frame is captured again between node 1(XMC4800) and 2(AM2432), the WKC of LRW command is 3, OUT is 0.

And WKC of ARMW, DC time sync frame is 1 for IN and 2 for OUT.

reconnect_1_2.zip

Hi Jianyu,

Thank you for the wireshark captures and the status.

One thing confirmed from the logs is that you are not observing the malformed packets for the LRW frames. This can confirm that the delay is correctly adjusted when the link breaks and SubDevice gets back to the network.

Couple of observations based on the logs you shared:

1.  
   • I see that in reconnect.pcapng file, frame# 5622 is the last frame where the LRW returns with WKC of 3, which can suggest the second device is out of network.
   • In frame# 5624, the returning frame has WKC 6, which suggest the second device is back in network.
   • I see the WKC of returning LRW frame is 6 till frame# 5720.
2.  
   • In frame# 5716, the second device is in SAFEOP.
   • In frame# 5620, the MainDevice is changing the state of the second device to INIT, which means it won't process the LRW frame.

Couple of follow up questions:

1. Are you using AM243x as both the SubDevices in pru544.zip?
2. Is the MainDevice Innovance H5U PLC?
3. Is it possible to share the ICSSG Memory dump (0x30000000 to 0x30040000) of the reconnected device?
4. Can I get more details on why the MainDevice is setting the second device to INIT state?

I'll also review reconnect_1_2.zip and get back to you.

Regards,
Aaron 

Hi Jianyu,

Just to add on, I'm attaching the logs with which I have tested to make sure the delay is correctly compensated when link break happens: SOEM_test_run_0x544_link_break.zip

Here, from frame# 2235 to frame# 5022, the second device goes out of network, that is, link is broken at frame 2234 and link is connected and back in network at frame 5023. From frame 5023, the second device process the incoming frames and increment the WKC. 

Regards,
Aaron 

In reconnect_1_2.zip, needs to be investigated. The CRC and Alignment Error bits are set, which needs to be checked further.

The incoming frame to AM243x has LRW WKC as 3 and BRD WKC as 1 and the expectation is that the returning frame should have LRW WKC as 6 and BRD WKC as 2, but in this case, LRW WKC is getting cleared and looks like BRD is not processed. Will need to investigate this further. If possible, can you provide the complete logs, that is, from INIT of the network till this behavior.

Regards,
Aaron 

I don't think the complete log is necessary. reconnect_1_2.zip provides the log when cable is reconnected, when disconnected, there is nothing to send in the first place. pru544.zip provides the initialization process for 2 drives.

Hi Jianyu,

What I want to understand here is recovery mechanism by the MainDevice once the disconnected device is reconnected back to the network. I don't see any commands from the MainDevice for the state transition of the reconnect_1_2.zip. This is needed to understand the MainDevice behavior.

As mentioned earlier, in pru544.zip -> reconnect.pcapng, the MainDevice is changing the state of the second device to INIT at frame# 5620, post which, CRC error bits are observed:

Jianyu Wang said:

pru544.zip provides the initialization process for 2 drives.

Just to clarify the environment in which these logs were captured, are the drives same in both the zip files? Also is the MainDevice same in both the zip files? The reason why I'm asking is because in pru544.zip -> reconnect.pcapng, I only see "Crc Error" bit being set, but in reconnect_1_2.zip, I see that "Alignment Error" and Crc Error" bits are set.

Regards,
Aaron

Aaron Thomas said:

• Are you using AM243x as both the SubDevices in pru544.zip?
• Is the MainDevice Innovance H5U PLC?
• Is it possible to share the ICSSG Memory dump (0x30000000 to 0x30040000) of the reconnected device?
• Can I get more details on why the MainDevice is setting the second device to INIT state?

1. mentioned

Jianyu Wang said:

frame is captured again between node 1(XMC4800) and 2(AM2432),

2. Yes

3. it is in the zipfile with file extension dat

4. No idea. We can only find clues from the packages.

This time, H5U ->cap channel 1 -> drive 1(xmc4800) -> cap channel 2 -> drive 2(AM2432)

init_reconnect_0_1_2.zip

On quick look, #40533 is when second drive is reconnected. WKC is correctly increased to 6.

#40725 that PLC try to restart the communication, that set 0x120 back to Init. After this frame, wkc is reset to 0 by AM2432.

Hi Jianyu,

Jianyu Wang said:

This time, H5U ->cap channel 1 -> drive 1(xmc4800) -> cap channel 2 -> drive 2(AM2432)

init_reconnect_0_1_2.zip

Thank you for this log and the ICSS memory dump.

Jianyu Wang said:

On quick look, #40533 is when second drive is reconnected. WKC is correctly increased to 6.

Yes correct. That is the observation I reported earlier and here also it is visible.

Jianyu Wang said:

After this frame, wkc is reset to 0 by AM2432.

This can occur if the FMMUs are not cleared by the MainDevice after it issues INIT to the SubDevice (here, the AM243x).

After reviewing the memory dump after Reconnecting (ReConnect.dat) and the wireshark logs, I see that the FMMU and SyncManagers Registers are not cleared:

If the FMMU (0x600 - 0x67F) and SyncManager (0x800 - 0x83F) are not cleared, abnormal behavior will be observed for the PDO transmission.

Hi Jianyu,

Additionally, can you confirm if this issue is observed if you set ENABLE_MULTIPLE_SM_ACCESS_IN_SINGLE_DATAGRAM, that is, with TIESC_PORT0_TX_DELAY_200_MHZ_CLOCK set to 0x98 (760ns) ?

For better clarity, make sure TIESC_PORT0_TX_DELAY_200_MHZ_CLOCK is 0x98 in the application and then do the link break and link re-connection activity and then observe if the EtherCAT MainDevice is setting the reconnected device to INIT, followed by clearing the FMMU and SM Registers?

Regards,
Aaron 

Aaron Thomas said:

If the FMMU (0x600 - 0x67F) and SyncManager (0x800 - 0x83F) are not cleared, abnormal behavior will be observed for the PDO transmission.

I would assume this is a bug you need to fix. As you can see in init_reconnect_0_1_2.zip, #3017 set drives to Init and #3189 reset FMMU. This seems to be the way how H5U intiate  a communication between slaves.

twincat_reconnect.zip

Here is another packages captured with twincat. It clears FMMU on startup. On reconnection, It wouldn't even bother to reset the FMMU.

In twincat_reconnect.zip, WKC count is correct after twincat reset AM2432 to Init #37693. It updates FMMU again in #54901

Hi Jianyu,

We have reproduced this behavior by playing back the wireshark frames from reconnect_1_2.zip logs. We do see why the LRW WKC is getting value of 0 within the EtherCAT firmware and the reason why CRC and Alignment Error bits are set in the ESL.

We are working on an optimal fix so that such edge cases will be taken care in all kinds of scenario.

Thank you for your patience.

Regards,
Aaron