AM2434: Are there any restrictions on changing EtherCAT status?

Hi,

Can you share the wireshark logs for the failing scenario? When you pause the R5F core, is the application getting stuck at any particular place?

I’m trying to understand if the issue is at the EtherCAT firmware (packets are not processed correctly) or if the host (R5F) is not able to update the state transition.

Regards,
Aaron

Hi Aaron,

The application is not in the same location, but is looping in a specific section.

I will send you the wireshark logs via mail.

Best regard,

Oyama

Hello Oyama,

I have few more queries related to your test case.

Please let me know how you are making sure that the transition to a new state is complete? Are you reading the current ESM state back and then request a new state? or the task is requesting a new state every 1ms irrespective of the previous state transition request is complete or not?

Have you tested with the EtherCAT Simple Demo application from the ICSDK or with your customized application based on Simple Demo?

Did you observe these debug prints from your EtherCAT SubDevice?

State change: 0x2 -> 0x1
State change: 0x1 -> 0x2
State change: 0x2 -> 0x1
State change: 0x1 -> 0x2
State change: 0x2 -> 0x1
State change: 0x1 -> 0x2

Could you please also share more information on your test setup and CodeSys configurations so that I can try to reproduce the same issue at my end? 

Kind Regards,

Hi Oyama,

An additional query related to the test environment/SDK:

• Which version of Industrial Communications SDK are you using? If you're not using Industrial Communications SDK 11.00, then I would recommend you to use the latest firmware available in Industrial Communications SDK 11.00 in your project. Details on how to migrate to the latest EtherCAT firmware can be found here: EtherCAT Firmware Migration guide.

Prior to this migration, please do provide the wireshark logs.

Tomoya Oyama said:

I will send you the wireshark logs via mail.

We haven't received yet. Please let us know if you need any guidance here.

Regards,
Aaron

Hi,

Harsha Begihally Shanmukhaswamy said:

Please let me know how you are making sure that the transition to a new state is complete? Are you reading the current ESM state back and then request a new state? or the task is requesting a new state every 1ms irrespective of the previous state transition request is complete or not?

The state change request is made after the state transition is complete.

Harsha Begihally Shanmukhaswamy said:

Have you tested with the EtherCAT Simple Demo application from the ICSDK or with your customized application based on Simple Demo?

I tested it with the default EtherCAT simple demo application.

Harsha Begihally Shanmukhaswamy said:

Did you observe these debug prints from your EtherCAT SubDevice?

Yes. And after hung-up, this debug prints also stopped.   

Harsha Begihally Shanmukhaswamy said:

Could you please also share more information on your test setup and CodeSys configurations so that I can try to reproduce the same issue at my end? 

The state transition is performed by the EtherCAT task.
No other operations are performed.

Aaron Thomas said:

Which version of Industrial Communications SDK are you using?

I use Version: 11.00.00.08.

Aaron Thomas said:

We haven't received yet. Please let us know if you need any guidance here.

I just sent it. If you haven't received it yet, please let me know how to send it.

Best regard,

Oyama

Tomoya Oyama said:

I use Version: 11.00.00.08.

I also tried with SDK version 11.00.00.13, but the result is the same.

Hi,

What is the situation now?
Have you been able to reproduce it?

Best regard,

Oyama

Hi,

Can I have more details on the topology you are testing here? Are you trying this in a redundancy setup ? Or is the PLC connected to the EtherCAT OUT port?

Regards,
Aaron

Hi Aaron,

The Wireshark logs I sent were obtained using the following configuration.

It is not a redundant setup. Nothing is connected to the LP's Out port.

Best regard,

Oyama

Hi,

Thank you for the details. We will try to reproduce the issue from our side.

Regards,
Aaron

Hi Aaron,

Is there any progress?

It is possible that the cause is in the master, so I would like to know the situation.

Best regard,

Oyama

Hi,

One major observation from your logs is that most of the frames having INIT Write datagram to 0x120 Reg is not returned:

I'm not sure if this is pointing to lost frame. I don't see this behavior for frames having PREOP Write datagram to 0x120 Register.

To add on,

Just to be aligned, are you connecting EtherCAT IN to Port0 and EtherCAT OUT to Port1(left open in your case)?

Hi Aaron,

Yes, I've tried both ports and get the same results.

Best regard,

Oyama

Hi,

The reason we wanted to verify if the OUT and IN port are correctly assigned is due to the presence of "Circulating bit" set in the wireshark logs you shared:

We are trying to understand the consequences when "Circulating Bit" is set. I haven't observed this bit being set in normal daisy-chain setup in TwinCAT or SOEM environment.

I have installed CODESYS but haven't gotten the complete setup ready. I will try to reproduce the issue or check internally with folks who has tested with CODESYS if this behavior is observed. Please expect an update by Friday.

Regards,
Aaron 

Hi,Aaron,

I have sent the wireshark logs again, so please check it.

So it can't be reproduced with TwinCAT ?

Best regard,

Oyama

Thank you for the logs. We do see some frames not getting acknowledged by the EtherCAT SubDevice.

• For INIT to 0x120 using APWR, I only see 241 returning frames (filter: ecat.cnt == 1 && ecat.reg.alctrl == 0x0001) for a total of 16756 APWR packets (filter: ecat.cnt == 0 && ecat.reg.alctrl == 0x0001) sent from the MDevice.
  • Frame #461212 is the last frame where PREOP write to 0x120 of the AM243x (Physical address = 0x3ea) is acknowledged, that is, WKC is incremented.
  • 
    • Frame #461219 again writes PREOP to 0x120 of the DUT, but the returning frame (461220) doesn’t seem to acknowledge the FPWR frame. Looks like the application goes into bad state from here. FPWR failure may be due to incorrect value in Reg.0x10. I don’t see Reg.0x10 corruption from the MainDevice device.
    • Looking at Frame #461222, the DUT is in INIT state and never changing in the following frames. Going further below, I see MDevice setting DUT to PREOP but the corresponding WKC is 0 (write is not happening I believe).
• Couple of follow-ups:
  • Can you make sure that Reg.0x10 is not getting corrupted/over-written, so that FPWR datagram will successfully write to the ESC Register for state transition.
  • Can you provide the ICSS Memory dump. You can capture log the following Register spaces based on the ICSS instance you are using:
    1. 0x30000000 - 0x30040000 (for ICSSG0)
    2. 0x30080000 - 0x300C0000 (for ICSSG1)

Tomoya Oyama said:

So it can't be reproduced with TwinCAT ?

I've tried the following with TwinCAT and did not observe any issue:

Connect LP to the master (TwinCAT).
Once it becomes OP, return to INIT.

Once the transition to INIT is complete, change to preOP.
Once the transition to preOP is complete, return to INIT.
Repeat this state transition.

Regards,
Aaron

Hi Aaron,

Aaron Thomas said:

Can you provide the ICSS Memory dump. You can capture log the following Register spaces based on the ICSS instance you are using:

I am attaching two files.　Are there file format OK?

_working : Before hanging up

_stopped: After hanging up

Aaron Thomas said:

Can you make sure that Reg.0x10 is not getting corrupted/over-written, so that FPWR datagram will successfully write to the ESC Register for state transition.

Do you know an effective way to check this?

Best regard,

Oyama

ICSSG1_0x30080000_0x300C0000_working.datICSSG1_0x30080000_0x300C0000_stopped.dat

Tomoya Oyama said:

I am attaching two files.　Are there file format OK?

Yes, the format is fine. Thank you. I will review this.

Tomoya Oyama said:

Do you know an effective way to check this?

You can monitor (0x30090000 + 0x10) Register from the DUT. 0x30090000 is the base address for the ESC Memory space from the host (R5F) perspective. For now I am able to see the value correctly assigned (0x3EA) from the memory dump you shared:

Regards,
Aaron

Hi Aaron,

Aaron Thomas said:

Can you make sure that Reg.0x10 is not getting corrupted/over-written, so that FPWR datagram will successfully write to the ESC Register for state transition.

I checked the value of Reg.0x10 using the code below, and confirmed that it always shows the same value after the EtherCAT connection is established.

static void EC_SLV_APP_SS_applicationRun(void *pAppCtxt_p)
{
    uint32_t readdata=0;

    readdata=CSL_REG32_RD_RAW((uint32_t *)(0x30090000+0x10));

    if((readdata&0x3EA)==0x3EA){
        OSAL_printf("x");
    }
    else{
        OSAL_printf("o");
    }
}

Best regard,

Oyama

Hi Oyama,

Thank you for the confirmation.

Aaron Thomas said:

I have installed CODESYS but haven't gotten the complete setup ready. I will try to reproduce the issue or check internally with folks who has tested with CODESYS if this behavior is observed. Please expect an update by Friday.

I'm still trying to get CodeSys working on my system. Facing issue in the Gateway login.

Regards,
Aaron 

Hi Aaron,

Has there been any progress since then?
Without changing the test configuration, we continuously changed the state of the first slave (not AM243), and it worked for a long time without any problems.
However, when we changed only the master device, the problem occurred again within a few minutes.

Best regard,

Oyama

Hi Oyama,

We just got a setup ready with CODESYS internally. Now we need to test the state transition and understand the behavior. I should be able to provide more details by mid-next week.

Tomoya Oyama said:

However, when we changed only the master device, the problem occurred again within a few minutes.

Sorry I'm not clear. So with other EtherCAT MainDevices, the issue is not observed and only with CODESYS it is observed?

Regards,
Aaron

Hi Aaron,

Aaron Thomas said:

Tomoya Oyama said:

However, when we changed only the master device, the problem occurred again within a few minutes.

The master device here refers to another EtherCAT MainDevices using CODESYS .
In other words, the problem could also happen with other MainDevices using CODESYS .

Best regard,

Oyama

Hi Aaron,

How is the progress? Can you reproduce it with CODESYS?

Best regard,

Oyama

Hi Oyama,

Apologies for the delay in response. I'm trying to reproduce the issue from my side (got delayed due to couple of other high priority tasks). I'll provide an update soon.

Regards,
Aaron

Hi Oyama,

Apologies for the delay due to holidays.

So I have brought up the Codesys setup on my side with RevPi Connect + as the PLC. Following are the configuration and topology:

• SubDevice application: TI EtherCAT Simple Demo
• Cycle Time: 1ms
• SDK:  Industrial Communications SDK 11.00.00.13
• Topology:
  • CODESYS Development System (Windows PC) <> RevPi Connect+ (EtherCAT Controller) <> ET2000 <> AM243x-LP (TI Simple Demo)

So I wrote a PLC program to change the state from INIT to OP. After that, I tried toggling the state manually between INIT and PREOP, from the "Online" window of the EtherCAT SubDevice and I do not see the application hanging after toggling the states for a minute. Attaching the logs for reference: https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/908/EtherCAT_5F00_Codesys_5F00_TI_5F00_Simple_5F00_Demo_5F00_ESC_5F00_0x0536_5F00_State_5F00_Transition.pcapng

Now that I have the setup working, is it possible for you to share the PLC Program that you have written for the same toggling experiment?

Regards,
Aaron

Hi Aaron,

I will send you the project later.
The test configuration is as follows:

• Codesys: V3.5 SP21 Patch 1
• Cycle Time: 1ms
• LP version: PROC109A(001)

It took about 3 minutes to hang.

The time it takes to hang varies.
Sometimes it takes a minute, sometimes it takes 10 minutes.

Best regard,

Oyama

Hi Oyama,

Thank you for the project file. I'll replicate similar in my setup and get back on the status.

Regards,
Aaron

Hi Oyama,

I haven't been able to reproduce the issue with the project file you have shared.

Hi Aaron,

I tried changing the master device to KEBA (Intel processor and not Codesys) and AM68-sk.

When I used KEBA, the problem did not occur.

However, the problem did occur with AM68-sk (codesyscontrol_linuxarm64_4.18.0.0_arm64).

Since this does not happen with the RevPi Connect+, the problem also may be related to the master.

The master CPU that was originally causing this problem is DRA83821.

Best regard,

Oyama

Thanks for confirming. Please reach out to CodeSys/3S for support here. We will close this ticket.

Aaron Thomas said:

So I have brought up the Codesys setup on my side with RevPi Connect + as the PLC. Following are the configuration and topology:

• SubDevice application: TI EtherCAT Simple Demo
• Cycle Time: 1ms
• SDK:  Industrial Communications SDK 11.00.00.13
• Topology:
  • CODESYS Development System (Windows PC) <> RevPi Connect+ (EtherCAT Controller) <> ET2000 <> AM243x-LP (TI Simple Demo)

So I wrote a PLC program to change the state from INIT to OP. After that, I tried toggling the state manually between INIT and PREOP, from the "Online" window of the EtherCAT SubDevice and I do not see the application hanging after toggling the states for a minute.

Aaron said it wasn't a Codesys problem.

This issue is occurring among TI evaluation boards.

In the first place, the fact that the problem can be reproduced on the AM68 only suggests the possibility that the phenomenon can be reproduced in your environment.

Ultimately, what I would like to know is why EtherCAT communication caused the AM243 to become inoperable, and what measures can be taken to address this.

Currently, there is always a risk of it hanging depending on the number of attempts.

Hi Oyama,

Tomoya Oyama said:

Ultimately, what I would like to know is why EtherCAT communication caused the AM243 to become inoperable

I believe if the MainDevice is not configured correctly, then due to some error frames, the SubDevice can become unresponsive. 

Hi Aaron,

If there are such error frames (I think you have checked Wireshark...), I think the problem is that it is stuck in an infinite loop, when in fact it should return an error.

Best regard,

Oyama

Hi Aaron,

Aaron Thomas said:

I believe there are no infinite loops observed in the host side (R5F) when this issue happens?

What does this mean?

Aaron Thomas said:

Based on the above discussion, this looks to be an issue with the Codesys MainDevice/Controller.

I also observed this issue with AM437x and DRA821.

This issue does not occur with EtherCAT slaves other than TI.

Therefore, I'm unable to identify the cause of this issue.

Aaron Thomas said:

As Pratheesh mentioned above, Codesys run time is not a TI supported product but 3P software. So, support needs to come from Codesys/3S in this case.

Could you please provide some material to ask Codesys?
Currently, all I have is that there is nothing unusual in the Wireshark logs, but the frequent status changes are causing the TI slave CPU or PRU to hang.

Best regard,

Oyama

Hi Oyama,

Tomoya Oyama said:

What does this mean?

I mean if the R5 is hanging when this issue occurs?

On an additional note, we have verified the setup with AM64x as the EtherCAT Controller in the following topology:

Codesys System Project (Windows) <> AM64x (EtherCAT Controller) <> TI ESC (running EtherCAT SubDevice Beckhoff Demo)

With this, we enabled a script to continuously toggle between INIT and PREOP after reaching OP. Attaching the PLC_PRG and the wireshark log for reference: Codesys_AM64x_State_Transition.zip

The above configuration ran for 30+ minutes and we haven't seen an instance where the state transition fails or the PRU halts after continuous state transitions.

EtherCAT Controller Setup (AM64x): 

• 6.6 kernel RT image
• CODESYS 64 3.5.21.40

Hi Aaron,

Thank you for recreating the environment on AM64.

Did you use AM64-evm or sk-AM64B? Is it a green board?

Which port did you use for EtherCAT?

The demo I use is not the Beckhoff demo, but the ethercat demo below with a 1h patch applied.

sdk\examples\industrial_comms\ethercat_subdevice_demo\device_profiles\401_simple\am243x-lp

If you are running a different program, please check again.

Best regard,

Oyama

Hi Oyama,

Tomoya Oyama said:

Did you use AM64-evm or sk-AM64B? Is it a green board?

We used the TMDS64EVM (green EVM board) for our testing.

Tomoya Oyama said:

Which port did you use for EtherCAT?

Tomoya Oyama said:

sdk\examples\industrial_comms\ethercat_subdevice_demo\device_profiles\401_simple\am243x-lp

If you are running a different program, please check again.

We've run the 401 Simple Demo from INDUSTRIAL-COMMUNICATIONS-SDK-AM243X 11.00.00.08 Software development kit (SDK) | TI.com. Attaching the wireshark logs for the same: Codesys_AM64x_State_Transition_TI_Simple_Demo.zip

Regards,
Aaron

Hi Aaron,

I also tried this with AM64, but it still didn't work.

Could you please provide more details so that I can fully replicate your environment?

My current environment is as follows:

############################################

AM243x-lp : PROC109A(001)

INDUSTRIAL-COMMUNICATIONS-SDK: 11.00.00.08 (default)

SysConfig: 1.22.0

Compiler: TI Clang v4.0.1.LTS

TMDS64EVM: PROC101C(004)

sdk linux rt am64 : 11.02.08.02 (tisdk default image rt am64xx evm)

CODESYS : SP21 Patch 1

CODESYS runtime : codesyscontrol linux arm64 4.18.0.0

##############################################

Did you change any settings when setting up the environment?

Best regard,

Oyama

Hi Oyama,

Aaron Thomas said:

EtherCAT Controller Setup (AM64x): 

• 6.6 kernel RT image
• CODESYS 64 3.5.21.40

This is the Codesys Environment and RT image version used. I'll confirm the SDK version for ti-processor-sdk-linux-rt.T

Aaron Thomas said:

With this, we enabled a script to continuously toggle between INIT and PREOP after reaching OP. Attaching the PLC_PRG and the wireshark log for reference: Codesys_AM64x_State_Transition.zip

This is the PLC script used for continuously toggling the states.

Regards,
Aaron

Aaron Thomas said:

This is the Codesys Environment and RT image version used. I'll confirm the SDK version for ti-processor-sdk-linux-rt.T

I just checked internally and looks like we're using sdk linux rt am64 from 10.00 SDK. 

Hi Aaron,

I will change it to 10.00.07.04 and try it.

Could you tell me what version of Codesys runtime you have?

And, Could you please send me the compiled app image of the EtherCAT Example you are using?

Best regard,

Oyama

Hi Oyama,

Tomoya Oyama said:

Could you tell me what version of Codesys runtime you have?

I'll get back on this by Thursday. My colleague who tested this is currently out on vacation.

Tomoya Oyama said:

And, Could you please send me the compiled app image of the EtherCAT Example you are using?

I had built the default example from 11.00.00.08 SDK. Let me see if I have the appimage available (since I switch between projects). Will share it with you.

Regards,
Aaron 

Hi Oyama,

Tomoya Oyama said:

Could you tell me what version of Codesys runtime you have?

CODESYS 64 3.5.21.40
CODESYS Control for Linux ARM64 SL 4.19.0.0.package

Tomoya Oyama said:

Could you please send me the compiled app image of the EtherCAT Example you are using?

As mentioned above, I had compiled the default 401 Simple Demo from INDUSTRIAL-COMMUNICATIONS-SDK-AM243X 11.00.00.08. I'm not sure if I need to share the compiled app image if there are no changes in the source code or firmware headers.

Let me know if you're facing any issue.

Regards,
Aaron

Hi Aaron,

I've matched the AM64 settings to yours.

I downloaded the SDK and compiled the default example again, but the toggle test result was the same.

Aaron Thomas said:

I'm not sure if I need to share the compiled app image if there are no changes in the source code or firmware headers.

Conversely, could you check my appimage whether what I compiled in my environment works in your environment?
I'm worried that my compilation environment may have changed due to some mistake.

Best regard,

Oyama

By the way, when I performed the same toggle test on ethercat_slave_beckhoff_ssc_demo, no problems occurred.

Does this give any clues as to the cause?