AM2434: Ethercat communication issue of AM2434

Hi Zekun,

Thank you for the details. One small clarification from the logs snippet shared above:

In Node2, I see from frame 18803 to frame 18806 forward frame (WKC = 7) and returning frame (WKC = 11) but in the red box, it's shown as lost frame. Can you explain the lost frame in this scenario? Is the customer referring to frame 53185 (from Node 1) as the lost frame?

We are reviewing the details provided and the possible parameters that can cause such issues.

Regards,
Aaron

In addition to this, can they share all LRW packets without filtering from INIT to OP transition (even from working setup is ok)? Also is this possible to log error counters 0x300 to 0x30F from TI devices and the ones behind it during error scenario?

Hi Aaron, Pratteesh

I also attach the Ethercat packages captured by wireshark.

7268.WiresharkLog.zip

Thanks

Zekun

These log files consist of reproduceable and non reproducible packages.

Hi Zekun,

Thank you for the logs. Yes we see that there is a frame loss occurring in the reproduceable package and I understand that the MainDevice is designed to stop EtherCAT communication when the frame is not returned to the MainDevice.

PratheeshGangadhar said:

Also is this possible to log error counters 0x300 to 0x30F from TI devices and the ones behind it during error scenario?

Is it possible to share the error counter details as well.

Regards,
Aaron

Hi Aaron

Aaron Thomas said:

n Node2, I see from frame 18803 to frame 18806 forward frame (WKC = 7) and returning frame (WKC = 11) but in the red box, it's shown as lost frame. Can you explain the lost frame in this scenario? Is the customer referring to frame 53185 (from Node 1) as the lost frame?

There is a mistake in the scrren shot.

The right idea is you can see in Node 2, which is normal case, shows that the downstream and upstream pair. One downstream pair one upstream.

But for Node1, you can see 6-11-6, so there is no upstream after that, which means the package that should pass to master is lost. 

Node 1 have no relationship with Node 2 because we can not test Node1 and Node 2 in parallel. 

We just want to show Node 1 and Node2 case.

Node 1, 6 -11-6, which is wrong. It should be 6 - 11 -6 -11 -6 -11

Node2, is the right case.

So there are 2 possible reason.

1. The package from Node 1 to Node 2 is lost, so Node 1 have downstream but no upstream, but Node 2 is correct.

2. The package from Node2 to Node is lost, so Node 2 is normal, but Node 1 lost the upstream.

Thanks

Zekun

Hi Aaron, Pratheesh

Please find below error counter screen shot.

The issue also occurs in the following environment.

Currently, customer only have the error counter values ​​0x300 to 0x30F are confirmed for this environment.
[Main Device] - [IO] (5 units) - [E7B_1] - [E7B_2] - [E7B_3]

Because the upper PC cannot read register values directly, so they connected TwinCAT to confirm the register after the issue occurred. However, TwinCAT may reads and clears the error counter values, so the results may not reflect the actual values ​​at the time of the issue. This is for reference only.

E7B1

E7B2

E7B3

I also advise them to add debug code to provide the register values  after they are obtained using debug code.

Thanks

Zekun

Hi Zekun,

Zekun Bai said:

Node 1, 6 -11-6, which is wrong. It should be 6 - 11 -6 -11 -6 -11

Node2, is the right case.

I understand. Thank you for the explaination.

Zekun Bai said:

So there are 2 possible reason.

Yes correct.

Zekun Bai said:

[Main Device] - [IO] (5 units) - [E7B_1] - [E7B_2] - [E7B_3]

For my understanding, the frame loss is observed in this environment and the Error counter register captures are based on this same environment?

Zekun Bai said:

Because the upper PC cannot read register values directly, so they connected TwinCAT to confirm the register after the issue occurred. However, TwinCAT may reads and clears the error counter values, so the results may not reflect the actual values ​​at the time of the issue. This is for reference only.

The customer can also enable wireshark capture while reading the error register from TwinCAT so that we can review the access to error registers from TwinCAT and observe the error counters at real-time (without anticipating if TwinCAT is clearing those registers).

Regards,
Aaron 

Hi Aaron

Aaron Thomas said:

For my understanding, the frame loss is observed in this environment and the Error counter register captures are based on this same environment?

Correct.

So based on current scenario, do you have any ideas that may cause this issue?

Thanks

Zekun

Hi Zekun,

Zekun Bai said:

So based on current scenario, do you have any ideas that may cause this issue?

We have tried to reproduce the issue from our side using the PDO values mentioned in the above table. We suspected the issue may be related to the FMMU configuration done in the custom setup. We tried combining the PDO assignments in the following environment:

SOEM (MainDevice) <> ET2000 <> AM243x-LP (RxPDO 286bytes, TxPDO 516bytes) <> AM243x-LP (RxPDO 9bytes, TxPDO 17bytes) <> AM243x-LP (RxPDO 9bytes, TxPDO 17bytes)

With this configuration, we are not seeing frame loss. Attaching the logs for reference: SOEM_Frame_Loss_Debug_v0690_0530.zip

The Error Counter Registers from customer setup will help in further analyzing the issue.

Regards,
Aaron

Hi Aaron

Do you need customer FMMU configuration files so that we can compare it with TI's?

Thanks

Zekun

Hi Zekun,

Zekun Bai said:

Do you need customer FMMU configuration files so that we can compare it with TI's?

We had replicated the TxPDO/RxPDO lengths provided in the above xls file. Yes, we can compare if the customer has details for the FMMU/SM Configurations Register programming from the MainDevice (0x600 - 0x67F and 0x800 to 0x83F).

Regards,
Aaron

Hi Aaron

Customer reproduce the issue and dump the register we need, also the normal case's version.

Couple of things:

1. Why the error count exist in OK and NG case?

2. The definition of 0X30c of AM243 is different with Beckoff， is the register one only exist in TI?

3. And I ask customer to get the main controller FMMU configuration registers as well.

4. Any ideas to debug this?

One thing, 

Regards

Zekun

Hi Aaron

I am curious that our handling logic and Beckoff logic, is there any difference of these two?

From my understanding, we program the PRU-ISCCS according to ETG rules, which is also referred by Beckoff. So it should be same.

Then why the topologic will lead to different scenario?

What is the possible reason and the schematic behind the reason?

Thanks

Zekun

Hi Aaron

SM register value (SM4~SM7 is not used)

FMMU，（FMMU2~FMMU7 is not used)

Thanks

Zekun

Update one thing about controller:

This issue can only reproduce with a certain CNC controller, if choose Beckoff or Omron, then it can not be reproducible. 

Now there are several doubts that can analyze:

(1) reg 0x303 has a very suspicious value of 255 both in normal and abnormal conditions. This is not a normal value unless it is an 8-bit register.

(2) There is no obvious difference in the register value between normal and abnormal conditions on AM2434.

(3) There are a considerable number of errors in both normal and abnormal conditions.

(4) Under what circumstances will the ESC stop forwarding data? Should the data be forwarded even if there is a CRC error?

Thanks

Zekun

Hi Zekun,

Thank you for providing the details.

Zekun Bai said:

(1) reg 0x303 has a very suspicious value of 255 both in normal and abnormal conditions. This is not a normal value unless it is an 8-bit register.

Error Counter Register details can be found here: TI EtherCAT SubDevice Controller Register List. Looks like RX Error appears to be playing a critical role in the issue.

Zekun Bai said:

(2) There is no obvious difference in the register value between normal and abnormal conditions on AM2434.

Since RX_ERR seems to play a critical role in these scenarios, obtaining the PHY Register values would significantly help us understand the PHY behavior too.

Zekun Bai said:

(3) There are a considerable number of errors in both normal and abnormal conditions.

Correct.

Zekun Bai said:

(4) Under what circumstances will the ESC stop forwarding data? Should the data be forwarded even if there is a CRC error?

I cannot think of a circumstance where the ESC will not forward a frame during run-time where the EtherCAT network is initialized successfully. Even if the ESC control gets stuck while processing any of the datagram, malformed packets will be observed in the network since the frame header is forwarded to the next node.

Regards,
Aaron

Hi Zekun,

Any response from customer here ?

Regards

Hi Ashwani

Customer in Japan are discussing via another E2E with TI. So pend this one.

Thanks

Zekun

Hi Zekun,

So do you need this thread to stay open or close it to avoid duplication?

Regards,
Aaron