MCU-PLUS-SDK-AM243X: EtherCAT Error Counters

Hi Fabian,

May I know which SDK version are you using for this multiple SubDevice testing?

Regards,
Aaron

Hi Aaron,

sure, we are currently using the Industrial Coms SDK Version 09.02.00

Regards 

Fabian

Fabian Sorkalla said:

The counters are also randomly incrementing from time to time while one of the devices is still in INIT state. 

Which specific counters are incremented? Is there a wireshark log?

the RX Error Counters. See from above in more detail the Frame Error (Register 0x0300 and 0x0302) Counters. I can try to produce a whireshark today and send it over. May we then switch to E-Mails? 

Hi Fabian,

So I tried the EtherCAT Beckhoff Demo from Industrial Coms SDK Version 09.02.00.15 and connected 2 SubDevices in line topology to TwinCAT. I tested with Free Run mode as well as DC Synchronization mode (at cycle time of 50us) and I don't see the Error registers (0x0300 : 0x0303) getting incremented when I take one of the SubDevice to INIT, keeping the MainDevice and the other SubDevice in OP state.

Attaching the wireshark logs for the same: EtherCAT_09_02_00_15_MultiDevice_Test_50us.zip

Do let me know if I'm missing on anything for reproduction of the issue.

Regards.
Aaron

Hi Aaron, we will also create some wireshark logs on our side and will provide them. 

Thanks for the clarification already!

Hi Aaron, can you do the same test with 3 subdevices and put the middle one into INIT?

Thanks! 

Sure Fabian. I'll try it out and update you by EOD.

Regards,
Aaron

Hi Aaron,
Here is the log of Wireshark. 

EtherCat_Wireshark_RxError_Counter.zip

We have configured 3 IO-link master in Line topology. Also, to observer the behaviour we used following ST code to read out RxError.

etcreadmemory1(xExecute := xRead, usiCom:=1, wSlaveAddress := .BNI_XG5_538_1B5_Z067.PhysSlaveAddr,
     xAutoIncAdr := FALSE, xBroadcast := FALSE, uiMemOffset := 16#0310,
     iSize := 2, pDest := ADR(LinkError1), udiTimeout := 500);
etcreadmemory2(xExecute := xRead, usiCom:=1, wSlaveAddress := .BNI_XG5_538_1B5_Z067.PhysSlaveAddr,
     xAutoIncAdr := FALSE, xBroadcast := FALSE, uiMemOffset := 16#0300,
     iSize := 4, pDest := ADR(RxError1), udiTimeout := 500);
	 

etcreadmemory3(xExecute := xRead, usiCom:=1, wSlaveAddress := .BNI_XG5_538_1B5_Z067_1.PhysSlaveAddr,
     xAutoIncAdr := FALSE, xBroadcast := FALSE, uiMemOffset := 16#0310,
     iSize := 2, pDest := ADR(LinkError2), udiTimeout := 500);
etcreadmemory4(xExecute := xRead, usiCom:=1, wSlaveAddress := .BNI_XG5_538_1B5_Z067_1.PhysSlaveAddr,
     xAutoIncAdr := FALSE, xBroadcast := FALSE, uiMemOffset := 16#0300,
     iSize := 4, pDest := ADR(RxError2), udiTimeout := 500);

etcreadmemory5(xExecute := xRead, usiCom:=1, wSlaveAddress := .BNI_ECT_508_105_Z067.PhysSlaveAddr,
     xAutoIncAdr := FALSE, xBroadcast := FALSE, uiMemOffset := 16#0310,
     iSize := 2, pDest := ADR(LinkError3), udiTimeout := 500);
etcreadmemory6(xExecute := xRead, usiCom:=1, wSlaveAddress := .BNI_ECT_508_105_Z067.PhysSlaveAddr,
     xAutoIncAdr := FALSE, xBroadcast := FALSE, uiMemOffset := 16#0300,
     iSize := 4, pDest := ADR(RxError3), udiTimeout := 500);

and the result is following:

I hope it will help you to reproduce the issue. 

Best regards,

Shahin

Hi Shahin,

To clarify myself, which example are you using for this?

Regards,
Aaron

Hi Aaron,

we are not using any TI examples but our own implementation but using Sitara AM24 Processor and the PRU FW from SDK 9.02.

Best Regards

Fabian

Hi Fabian,

From the logs, looks like 0x302 ESC Register is getting incremented. And from the register description, this error counter gets incremented when RX Error is detected.

Recommend to check the HW or PHY for any noise factor which can affect the EtherCAT communication.

Regards,
Aaron

Hi Aaron, we grounded our modules accordlingly. The last module is a older revision IO-Link master (with another vendors processor) which is in the same line and does not show this behavoir. We only see it with the sitara based IO-Link Gateways.
Can you reproduce it with 3 devices?

Hi Fabian,

Can you monitor RX_ERR pin from the PHY during the tests or monitor RX_ERR counter register of the PHY (0x15 offset in TI PHYs).

Fabian Sorkalla said:

Can you reproduce it with 3 devices?

• Let me see how I can reproduce this from our side.

Regards,
Aaron

Hi Fabian,

Fabian Sorkalla said:

Hi Aaron, can you do the same test with 3 subdevices and put the middle one into INIT?

• So I tried this with our default EtherCAT Beckhoff SSC Demo and the error counters weren't getting incremented while taking the middle device to INIT state.
• One thing I can try is to replicate the similar application in our side and playback the packets where you're seeing the error, using a packet player.
• For the above, can you provide the FMMU configurations and the complete wireshark logs for EtherCAT communication (from INIT to OP and till the error counter increment is observed) ?

Regards,
Aaron

Hi Aaron,
here I attached complete Wireshark log.

Aaron Thomas said:

For the above, can you provide the FMMU configurations and the complete wireshark logs for EtherCAT communication (from INIT to OP and till the error counter increment is observed) ?

EtherCat_Wireshark_RxError_Counter_2.zip

Best regards

Shahin

Thank you Shahin for the logs. I’ll try to reproduce with this.

Aaron Thomas said:

Can you monitor RX_ERR pin from the PHY during the tests or monitor RX_ERR counter register of the PHY (0x15 offset in TI PHYs).

• Meanwhile can you look into this and update us of the behaviour?

Regards,
Aaron

Hi Aaron,

thanks!

We will try to set something up where we monitor RX_ERR. 

Do you think a call we help where we show you the issue?

Regards,

Fabian

Hi Fabian,

Fabian Sorkalla said:

Do you think a call we help where we show you the issue?

• Let me try to replicate the environment with the issue in my environment. We can plan for a call if I’m not able to reproduce it here.

Meanwhile, can you also share the value in ESC registers? I’m referring to the values in the memory space 0x30090000 to 0x30090ECF.

Regards,
Aaron

Hi Fabian, Shahin,

Following the FMMU configurations from the logs you shared, we created a reproduction environment with the following topology:

PC <-> ET2000 <-> AM243x-LP (SD1) <-> AM243x-LP (SD2) <-> AM243x-LP (SD3)

SD1:
InputSize     - 32 bytes
OutputSize  - 2 bytes

SD2:
InputSize     - 32 bytes
OutputSize  - 2 bytes

SD3:
InputSize     - 16 bytes
OutputSize  - 0 bytes

After doing a playback of the packets (EtherCat_Wireshark_RxError_Counter_2), I'm not seeing the RX Error Counter registers getting incremented when SD2 is taken to INIT.

Attaching the logs for the same: https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/908/Baluff_5F00_3_5F00_Device_5F00_Issue_5F00_Reproduction_5F00_Playback.pcapng

Fabian Sorkalla said:

We will try to set something up where we monitor RX_ERR. 

• Do let me know if you have any update for this.

Regards,
Aaron

Hi Aaron,

Aaron Thomas said:

Can you monitor RX_ERR pin from the PHY during the tests or monitor RX_ERR counter register of the PHY (0x15 offset in TI PHYs).

• We monitored the RX_ERR pin as you suggested, however no signal was visible at the Oscilloscope at time of testing. 
• We have also monitored the RX_ERR counter register using MDIO_phyRegRead method located mdio_v0.c. The method as called from Mainloop located in ectappl.c. The counter value was 0, when we stop the program read the counter value. However, when place a breakpoint inside MDIO_phyRegRead method we notice some value in RX_ERR counter.  We could not figure out the meaning of the value. The value was not same with the value read at Codeysys. 

Aaron Thomas said:

Meanwhile, can you also share the value in ESC registers? I’m referring to the values in the memory space 0x30090000 to 0x30090ECF.

I have attached the memory value here ESC_registers_0x30090000_0x30090ECF.zip.

Thank you so much

Best regards

Shahin

Hi Shahin,

Shahin Pramanik said:

The counter value was 0, when we stop the program read the counter value. However, when place a breakpoint inside MDIO_phyRegRead method we notice some value in RX_ERR counter.

• Is this the case all the time, that is, each time the break point is hit, RX_ERR counter is incremented? Recommend you to log the counter value in the terminal/console for proper monitoring so that it aligns with the RX_ERR pin reading.

Shahin Pramanik said:

I have attached the memory value here ESC_registers_0x30090000_0x30090ECF.zip.

• This is not in a readable format. Could you please provide the register dump in .dat file. You can obtain this by choosing TI Data as File Type in View -> Memory Browser -> Save Memory: 
  •  
• Followed by choosing System_View as Memory Page in the next window:
  • 

Regards,
Aaron

Hi Aaron,

Aaron Thomas said:

Is this the case all the time, that is, each time the break point is hit, RX_ERR counter is incremented? Recommend you to log the counter value in the terminal/console for proper monitoring so that it aligns with the RX_ERR pin reading

Yes, it was. I will try your suggested method again and let you know the outcome. 

Unfortunately, this week I'm not the at office anymore. I will provide this data to you next week

Regards

Shahin

Hi Shahin,

Shahin Pramanik said:

Unfortunately, this week I'm at office anymore. I provide this day to you next week

Yeah sure no problem.

Regards,
Aaron

Hi Aaron,

Aaron Thomas said:

Is this the case all the time, that is, each time the break point is hit, RX_ERR counter is incremented? Recommend you to log the counter value in the terminal/console for proper monitoring so that it aligns with the RX_ERR pin reading.

The console output shows counter value 0, However in Codesys there are non-zero values.

And here is the memory register value save as TI data, 1526.ESC_registers_0x30090000_0x30090ECF.zip

The memory register value is saved from the 2nd node. 

Best regards

Shahin

Hi Shahin,

Shahin Pramanik said:

The console output shows counter value 0, However in Codesys there are non-zero values.

• Thanks. Noted. Also did you try

Shahin Pramanik said:

And here is the memory register value save as TI data, 1526.ESC_registers_0x30090000_0x30090ECF.zip

• The values doesn't look right. I don't see the EtherCAT Firmware version in the first 4 bytes and overall it looks different than the expected values. Are you using ICSSG0 or ICSSG1 instance for EtherCAT implementation?

Also could you provide the values in the following register spaces:

• PRU_ICSSG_RAM: 0x30010000 - 0x30010ECF (for ICSSG0)                or                0x30090000 - 0x30090ECF (for ICSSG1) 
• PRU_MII_RT_MII_RT: 0x30032000 to 0x3003206C (for ICSSG0)           or                0x300B2000 to 0x300B206C (for ICSSG1)
• PRU_MDIO_MDIO: 0x30032400 to 0x30032484 (for ICSSG0)                or                0x300B2400 to 0x300B2484 (for ICSSG1)

Regards,
Aaron

Hi Aaron,

Aaron Thomas said:

The values doesn't look right. I don't see the EtherCAT Firmware version in the first 4 bytes and overall it looks different than the expected values. Are you using ICSSG0 or ICSSG1 instance for EtherCAT implementation?

We are using ICSSG1 instance.

Aaron Thomas said:

Followed by choosing System_View as Memory Page in the next window

In my Code Composer studio System_View option is missing

Aaron Thomas said:

0x30090000 - 0x30090ECF (for ICSSG1) 

ESC_registers_0x30090000_0x30090ECF.dat

Aaron Thomas said:

0x300B2000 to 0x300B206C (for ICSSG1)

ESC_registers_0x300B2000_0x300B206C.dat

Aaron Thomas said:

0x300B2400 to 0x300B2484 (for ICSSG1)

ESC_registers_0x300B2400_0x300B2484.dat

Regards

Shahin

Thank you Shahin for the memory dump.

Looking at the MII_RT Registers, I see that bit2 of MII_RT_RX_ERR0 is set:

As mentioned in the description, this points to the fact that frame total byte count is less than the defined value. This looks like the MainDevice is not transmitting as expected?

Additionally, is it possible to test with a different MainDevice? And is it possible to use a tap to capture the EtherCAT communication - like ET2000 | Industrial Ethernet multi-channel probe. Since Reg 0x0302 is getting incremented, it corresponds to the reverse path (Rx at port1). So a tap can be kept in between and see if CRC is generated and if yes, which device is causing the issue.  

Regards,
Aaron

Hi Aaron,

Thank you for pointing out the possible issue. 

Aaron Thomas said:

Additionally, is it possible to test with a different MainDevice?

Sure, I will try to reproduce the issue with Backhoff PLC and I will share the result with you. 

Aaron Thomas said:

As mentioned in the description, this points to the fact that frame total byte count is less than the defined value. This looks like the MainDevice is not transmitting as expected?

In the meantime, could you also tell me where these memory description can be found, so that I can go through it and understand stand the context too?

Aaron Thomas said:

And is it possible to use a tap to capture the EtherCAT communication

I will search it in my team if they have one, then I will use and check with it. I will give you the feedback.

Best regards

Shahin

Hi Shahin,

Shahin Pramanik said:

In the meantime, could you also tell me where these memory description can be found, so that I can go through it and understand stand the context too?

• You can refer to AM64x/AM243x Technical Reference Manual Section 6.4. Section 6.4.14 provides the register space along with the register description.

Shahin Pramanik said:

I will search it in my team if they have one, then I will use and check with it. I will give you the feedback.

• Sure. Thank you.

Regards,
Aaron

Hi Aaron,

Aaron Thomas said:

And is it possible to use a tap to capture the EtherCAT communication - like ET2000 | Industrial Ethernet multi-channel probe

Sorry to reply you late. Although it took little bit of time, I able to manage package capture device https://www.hilscher.com/de/produkte/netzwerk-und-kommunikationsdiagnose/netzwerkdiagnose/nanl-b500g-re  .

I have used following line connection for Three different EtherCat master device to capture the EtherCat communication.

As EtherCat master I have used Codesys (Soft PLC), TwinCat3.1 (Soft  PLC) and Omron NJ501. 

The capture packets are here  RX_Error.zip .

The RX_ERROR counter is only increased when Codesys was used. 

In TwinCat and Omron I can't see any increment in these counters. 

Also, when I check the capture packets for Codesys, there are some non EtherCat frames available in list. Could that be a reason for counting up for the RX_ERROR counter? 

Best regards

Shahin