CCS/AM3359: EtherCAT Slave device reset affects network

The RTOS team have been notified. They will respond here.

Hi Biser,

Thank you for the reply. I am uncertain though, whether this falls on the RTOS team, or the AM3359 processor team, to respond next with how either the RST_GLOBAL_COLD_SW/RST_GLOBAL_WARM_SW might or might not affect the PRU of the SOC, which in this case serves as the EtherCAT interface of our slave device, and executes the pre-built TI library code (contained in our overall application firmware).

In any case, I'll look forward to further replies. Thank you.

John

Hi John, let me see if I understand your system requirement. Do you want to reset AM3359 ARM but not PRU-ICSS? If so, what is your use case, in other words, when/why do you want/need to reset ARM and not PRU?

Thank you,
Paula

Hi Paula,

Out project here (at Lam Research), would require that our slave device to successfully process an FoE firmware upgrade/download, without disrupting other slave devices connected on the same EtherCAT network, upstream or downstream to our device. At this time though, our firmware triggered a reset at the end of FoE download processing, by toggling the RST_GLOBAL_COLD_SW bit, such that the new firmware image just downloaded can take effect and start executing on the AM3359. However, this will kick other slave devices on the same EtherCAT out of OP Mode momentarily, before being restored by the master.

Our current thought is: Can there be a way to reset the AM3359 and bring the new main firmware into effect, with the PRU_ICSS still running its own image (binary library obtained from TI), and hence be left "undisturbed"? If this is not the correct approach, please advise otherwise.

Again, the goal here is not to disrupt other slave devices on the same EtherCAT network at all, while our own slave device goes through a firmware-triggered reset. Thank you.

John

Hi John, thanks for confirming, FoE upgrade was my main suspicious =).
Just FYI, this is currently not supported on latest PRU-ICSS EtherCAT slave, however, this was supported under the old Industrial SDK 1.1.x. If you have the old ISDK 1.1.x you might want to take a look on how it was implemented there.

I found a summary of the process used in ISDK 1.1.x example for upgrading EtherCAT application via FoE, without resetting PRU-ICSS:
"This is possible by maintaining a common loader section across application software versions, which will load the new software from the flash. Roughly, the idea is that the loader is the entry point to the application. And, during the firmware upgrade process, i.e. once flashing is completed, the application will branch to the loader (during BOOT to INIT transition) which in turn will load the new application to memory. INIT transition will be complete when loading and application initialization is complete (PRU firmware load is bypassed in this mode). Then master can initiate a transition to OPERATIONAL (OP) state. Since PRU firmware is not reloaded or PRU-ICSS is not reset during this whole sequence downstream devices will continue to communicate with master. "

In other words:
(1) Flashing will happen in BOOTSTRAP state – no other mailbox operations can happen in parallel as this is a SAFE mode
(2) Master change the state from BOOTSTRAP to INIT state
(3) Slave reloads the newly flashed application in to memory and branch to start of the application - requirement is that loader part of application is always running from predefined location and this section is not relocated/overwritten
(4) Master change the state from INIT to OP state
When Host is ready in INIT state, AL control write will take effect and master waits till slave transitions to requested state.

Hope this helps. Thank you,
Paula

Hi Paula,

Thank you for the good information in your reply. Just to confirm:

1. We are using the TI AM3359 SDK v.01.01.03.03. Does this fall into the "SDK 1.1.x" category that you mentioned? The FoE firmware upgrade implementation has been completed and working with our slave device.
2. Just curious: What are the "latest PRU-ICSS EtherCAT slave" products from TI? They do not support FoE upgrade, just because of the exact problems that we encountered?
3. Our on-board bootloader is indeed at a fixed location in Flash. So, the solution that you suggested, will have the main firmware simply branch to the bootloader for execution, after the FoE upgrade has been processed, hence neither RST_GLOBAL_COLD_SW nor RST_GLOBAL_WARM_SW bit will be toggled in the entire process, correct? ([Note:] We also implemented a Flash file system, to store the FoE-downloaded firmware upgrade images as files.)

Please advise. Thank you.


John

Hi John, ah OK your are still using ISDK 1.1.x. Then, for ISDK 1..1.x FoE firmware upgrade w/o resetting PRU-ICSS is part of EtherCAT slave demo.
About Q2) This feature is not required by standard, and we think it is more application dependent (only few customers request it). So it was de-scoped.
For Q3) I believe neither RST_GLOBAL_COLD_SW, nor RST_GLOBAL_WARM_SW are required, but I will confirm.

thank you,
Paula

Hi Paula,

Please do confirm, regarding Q3.

Now, you have also got me curious again: When you mentioned "Then, for ISDK 1.1.x FoE firmware upgrade w/o resetting PRU-ICSS is part of EtherCAT slave demo.", what demo are you exactly referring to? So, a solution without resetting the PRU-ICSS is already known? How do we get hold of such demo source code?

Looking forward to your reply. Thanks.


John

Hi Paula,

Asserting the RST_GLOBAL_WARM_SW bit via firmware on our slave device, does cause another slave device connected (upstream) on the same EtherCAT, to go through OP->INIT->SAFEOP->OP transition. Could you let us know why this is the case, please?

Also, the TwinCAT gave the following console messages, as our slave device is being warm-reset:

Warning 1 5/22/2018 11:17:26 AM 294 ms | Device 2 (EtherCAT): Frame missed 10 times (frame no. 0)
Message 2 5/22/2018 11:17:33 AM 104 ms | Device 2 (EtherCAT): Frame returned -> force reinitialization!
Message 3 5/22/2018 11:17:35 AM 620 ms | 'Box 1 (LAM ECAT LDS CONT) (1001)' Communication re-established
Message 4 5/22/2018 11:17:35 AM 624 ms | 'Box 2 (Lam ECAT ADVCI) (1002)' Communication re-established
Error 5 5/22/2018 11:17:44 AM 724 ms | 'Box 2 (Lam ECAT ADVCI)' (1002): 'PREOP to SAFEOP' failed! Error: 'check device state for SAFEOP'. AL Status '0x0002' read and '0x0004' expected.
Error 6 5/22/2018 11:17:44 AM 724 ms | 'Box 2 (Lam ECAT ADVCI)' (1002): state change aborted (requested 'SAFEOP', back to 'PREOP').

Thank you.

John

Hi John, could you try the steps mentioned on my post (05/21) w/o WARM RESET, and let us know if this works?

thank you,

Paula

John, let me share with you a possible process flow:

Then, answering one of your questions, you can use a flag to check if the application is being reloaded. About the WARM RESET flag, I will confirm and come back to you.

thank you,

Paula

Hi Paula,

Yes, please do confirm whether asserting RST_CLOBAL_COLD_SW or RST_GLOBAL_WARM_SW will also reset the PRU.

In your suggested process flow above, the app_reload flag is not relevant to judging whether the PRU is currently running, correct?

Lastly, our bootloader resides in the on-board SPI Flash. Our understanding is the the AM3359 ROM code will first execute at power-up, and then it "finds" the bootloader in SPI Flash, to start executing that. So, for our own firmware application to branch to the bootloader for executing without first invoking the AM3359 ROM code, how should it be done?

Please advise. Thank you.

Hi John, Please see below summary I got from our developers, which might be more clear:

1) ISDK1.1.x based method don't reset the PRU while doing firmware update
a. This application has a section of code which is essentially a copy of SPI bootloader. (This code is in the application)
b. After receiving and writing the new application in flash we branch to this section of code which will load the application from flash to the RAM and branch to the starting point of the application
c. The important point to note here is this new section of code resides in a memory different from that mentioned in the original application (basically the loading address is different from execution address)
i. If we don’t do this then while loading the new application, even this section of code will get overwritten. To prevent this we move this section post loading the application to a new memory location
d. This method doesn’t require any kind of reset of the SOC.

Hope this helps,
Paula

Hi Paula,

I see that you have described above, the process of branching to the bootloader to execute, and not doing any reset.

But, have you confirmed whether asserting RST_GLOBAL_WARM_SW bit, or RST_GLOBAL_COLD_SW bit, in the PRM_RSTCTRL register, will or will not also cause the PRU to reset?

Please let me know. Thank you.

John

Thank you, Paula, for getting us that confirmation, which we suspected to be the case.

Well, this should mean that in order to avoid disrupting other slave devices on the same EtherCAT network, as our own Sitara AM3359 based slave device has completed its FoE firmware download processing, our only option is to take the approach of NOT asserting either RST_GLOBAL_COLD_SW or RST_GLOBAL_WARM_SW bit in the PRM_RSTCTRL Sitara SoC register.

John

Paula,

We are considering another approach: If we only want to ensure that all upstream slave devices do NOT get disrupted, while our own device goes through the reset at the end of FoE fw download, then what can we do? Is there a particular PRU firmware patch by TI, that we can apply in our overall slave device firmware, to achieve the purpose?

Please advise. Thank you.

John

John, let me ask a question, to see If I am understanding correctly. Does the new approach consist in not disturbing upstream devices, but downstream will get disturbed?

thank you,
Paula

Yes, Paula, that is what we are willing to take, which is: To leave the upstream devices completely undisturbed on the EtherCAT when we reset our own slave device, but let the downstream devices go out of OP Mode and then get restored.

We are under the impression, that TI had actually developed a PRU firmware patch, to achieve this behavior? Please let me know. Thank you.


John

Hi John, one idea could be to close the Port 1 on the last upstream device. This should ideally cutoff all the communication to the DUT and all the devices after that.
One thing, for resetting the DUT, we assume you need to send a command from master, right?. If so, one option is that master send a reset command with some delay, in order to get enough time for the master to cut off the port 1 without losing frames.

As far as I know we don't have any PRU firmware patch that could achieve this behavior. Please let me know from where you get this impression so I can dig deeper.

thank you,
Paula

Paula,

Consulting with another group here at Lam Research that also worked on their Sitara based Ecat slave device, I got the following reply:

"The problem is that the slave is throwing out frames or returning corrupted frames preventing the previous slave to detect an open link. The correct behavior is that as soon as a slave disconnects; the previous one heals the connection by closing the port as soon as it detects the link to the next slave is broken."

Is this "...preventing the previous slave to detect an open link." problem, corrected by any PRU firmware patch available from TI? If yes, could we obtain such a patch, and be instructed as far as how to apply the patch?

Thanks.

John

Hi John, I am not aware of a patch or fix to this specific scenario. But if you want to you can take a look of ISDK EtherCAT erratas to see if anything there could apply to your case.

But please let me ask few questions, as I think I am a little bit lost in what you have tried, and what is working / non working

1) Want to confirm your SDK. I believe you are using AM335x ISDK 1.1x? Is this correct? if not which version are you using?

2) If 1) is correct, then have you tried FoE from there? If so, did it work?

2) If 1) Are you still using WARM RESET? I think you mentioned that no, but want to confirm.

3) if 1) is correct, could you move to PRU-ICSS EtherCAT + TI RTOS PSDK? (our latest supported model)

thank you,

Paula

Paula,

I am not sure whether you received my reply to your questions above, hence I'm doing it again:

1) We are using the SDK v.01.01.03.03.

2) Yes, FoE is working, and we can use it for downloading files to our slave device, including an upgrade firmware image.

3) Currently, we are asserting the RST_GLOBAL_COLD_SW bit in the Sitara PRM_RSTCTRL register, at the completion of processing a firmware image downloaded via FoE. As you had made clear to us, either asserting that bit, or the RST_GLOBAL_WARM_SW bit, will also reset the PRU that serves the EtherCAT interface of our device. And this, is what we consider as the reason for both the upstream and downstream devices to be disrupted.

4) Moving to another TI software platform, is out of the question for our project at this point in time. Instead, we are inquiring whether there has been any PRU firmware patch, that will close its port properly when the above reset bits were asserted, such that at least the upstream devices won't be disrupted/impacted, since all the EtherCAT packets will be just re-routed, and bypass our device as it is going through a reset triggered by its own firmware. (The PRU firmware is pre-built by TI as a binary image, and linked in by our own device firmware, per our understanding.).

Please let us know. Thank you.


John

John, I will come back to you. but let me clarify my question 2). Actually, the question is if you use FoE w/o PRU-ICSS reset from ISDK 1.1.x. I was instructed it was already there..

thank you,
Paula

Paula,

Yes, please do come back to us, after you have obtained further information, regarding the PRU properly closing its port when being reset.

As for question 2): No, we hesitate to bypass asserting the RST_GLOBAL_COLD_SW bit, and take the "no reset" approach for the FoE firmware upgrade to work. Again, to not disrupt just the upstream devices is our current goal now, and hopefully there can be a PRU firmware patch that can be applied for achieving the goal. Unless, of course, you know of no such port closing issue, on the AM3359 Sitara PRU.

Please let us know. Thank you.


John

Hi Paula,

Have you obtained further information on the PRU (not properly closing its port, when being reset)?

Please let me know. Thanks.

John

John,

We don't have the PRU patch in ISDK 1.x to close its port properly after setting RST_GLOBAL_COLD_SW.

ISDK 1.x is no longer supported/maintained as announced here ().

The latest PRU-ICSS industrial SW is available here - 

We have the feature of FoE without communication break on K2G device currently and can be a reference for Sitara device. But this is based on PRU-ICSS SW instead of ISDK 1.x.

Regards, Garrett

Hi Garrett,

Thanks for the feedback, and I certainly understand your viewpoint. As one last attempt on my end to resolve the issue: The above attached 4 files (contained in the zip file), came from another division here at Lam Research, which also worked on a Ti Sitara based EtherCAT slave device, and they used the identical SDK as we do. We were informed by them, that these source files contain a PRU firmware fix, which will indeed close the EtherCAT port properly, when being reset. Could we have your thoughts on these source files? Are they applicable to our case?

Please advise. Thank you.

John

firmware.zip

John,

Trying to find the out the history of the firmware you attached. In the meanwhile, have you tried to turn off the Port 0 link of DUT through the power down register of PHY or asserting the PHY reset GPIO till the new application starts. This can also be done from ECAT master side using ECAT MDIO interface similar to ECAT EEPROM interface, see MII Management Control/Status (0x0510-0x0511) - processors.wiki.ti.com/.../PRU_ICSS_EtherCAT_Slave_Controller_Register_List

Also forcing the port 1 of the slave before DUT to close from master side should help as well.

Regards,
Garrett

Garrett,

Given that our other group here also based their EtherCAT slave device on TI Sitara, and informed us that the patch (previously forwarded to you) resolved the very same issue of the PRU not closing its port properly, I will have to first try out the patch. I was given the information that as long as the 'merge_appl.bat'' file under our project in the CCS tool, is modified to refer to the source files containing the PRU firmware patch, we can build a full image file of our device firmware, with the patched PRU firmware being part of it.

However, our 'merge_appl.bat' is as follows:

---

@ECHO OFF
rem merge_appl.bat
rem Copyright (c) 2012 Texas Instruments Incorporated ALL RIGHTS RESERVED

rem ******************************************************
rem This batch file merges three binary files into one by
rem calling isdk_merge utility
rem ******************************************************

rem setlocal enabledelayedexpansion
setlocal

set out_file="%CD%\Debug\ecat_appl_pru.bin"
set infile="%CD%\Debug\ecat_appl_SPI.bin"
set pru0_firware_name="%IA_SDK_HOME%\protocols\ethercat_slave\firmware\ecat_frame_handler_flash.bin"
set pru1_firware_name="%IA_SDK_HOME%\protocols\ethercat_slave\firmware\ecat_host_interface_flash.bin"
set pru0_offset="78000"
set pru1_offset="4000"

rem echo %out_file%
rem echo %infile%
rem echo %pru0_firware_name%
rem echo %pru1_firware_name%
rem echo %pru0_offset%
rem echo %pru1_offset%

%IA_SDK_HOME%\tools\isdk_merge\isdk_merge.exe %out_file% %infile% %pru0_offset% %pru0_firware_name% %pru1_offset% %pru1_firware_name%


:FINISH
echo

---

And I do not even see 'ecat_appl_pru.bin' being generated as an output file in the corresponding directory. How do we build our device firmware image file, with the PRU firmware image being part of it?

Please advise. Thank you.

John

Hi John, I honestly don't recall if we have to merge the firmware and app in order to flash it.. this was long time ago. However, did you try to run app with JTAG and CCS, or via SD card? Just to confirm firmware shared by your colleagues behaves as you expected?

thank you,
Paula

Hi Paula,

Thank you for getting back to me again. I very much would like to try out the PRU firmware patch provided by our other group here, on our own Sitara based slave device. However, I still have not found the correct way, or script to use, to build the patched PRU firmware source, into the overall firmware image of our device. So, on your end, you also cannot recall how that was usually done?

Please advise. Thank you.

John

Hi John, how are you currently doing it? I assume you just need to replace Firmware binary and header files (C:\TI\am335x_sysbios_ind_sdk_1.1.xxx\sdk\protocols\ethercat_slave\firmware) and rebuild. But in any case, from AM335x SYSBIOS Industrial SDK 01.01.xxx User Guide I see another option:

NOTE 2: By Default, Both EtherCAT applications embed the PRU firmware binaries into the application binary.

However, loading PRU firmware from SPI flash is also supported in this version. In this case, the EtherCAT

application will load PRU firmware from SPI flash memory (if available) during run-time. This saves almost 16KB

memory by reducing the binary size. The EtherCAT binary expects the PRU0 firmware at an offset of 0x98000 from

the beginning of SPI memory and PRU1 firmware at an offset of 0x9C000 from the beginning of SPI memory

Could you try to load PRU Firmaware to those SPI offsets and see if it works?

Also, our developers mentioned you can potentially use latest PRU-ICSS EtherCAT slave firmware. This will help us to have the latest and greatest FW, which have several bug fixed and includes new features. If you would like to try our latest FW, you can get it from

hope this helps,

Paula

  

Paula,

Would you advise on how the below shown script file (originally from TI) should be used, in the build process of any given Sitara AM3359 based firmware image? Supposedly, the pru0_firware_name and pru1_firware_name referenced in the script, just need to be modified to point to the new or different versions, and the PRU images will be incorporated into the overall slave device firmware image. However, I have tested it to see that the script file is actually not used by our build process at all, and hence I would not think the new PRU image files got used at all.
Please advise. Thank you.

=====
@ECHO OFF
rem merge_appl.bat
rem Copyright (c) 2012 Texas Instruments Incorporated ALL RIGHTS RESERVED

rem ******************************************************
rem This batch file merges three binary files into one by
rem calling isdk_merge utility
rem ******************************************************

rem setlocal enabledelayedexpansion
setlocal

set out_file="%CD%\Debug\ecat_appl_pru.bin"
set infile="%CD%\Debug\ecat_appl_SPI.bin"
set pru0_firware_name="%IA_SDK_HOME%\protocols\ethercat_slave\firmware\ecat_frame_handler_flash.bin"
set pru1_firware_name="%IA_SDK_HOME%\protocols\ethercat_slave\firmware\ecat_host_interface_flash.bin"
set pru0_offset="78000"
set pru1_offset="4000"

rem echo %out_file%
rem echo %infile%
rem echo %pru0_firware_name%
rem echo %pru1_firware_name%
rem echo %pru0_offset%
rem echo %pru1_offset%

%IA_SDK_HOME%\tools\isdk_merge\isdk_merge.exe %out_file% %infile% %pru0_offset% %pru0_firware_name% %pru1_offset% %pru1_firware_name%


:FINISH
echo
=====


John

John, I think that script is not necessary, maybe it was used in some point in time... I believe only replacing FW files should be enough. I see in tiescutils.c firmware is loaded as a header file.

// get the firmware loaded as header files!
#include "ecat_frame_handler_bin.h" // EtherCAT frame handler
#include "ecat_host_interface_bin.h" // EtherCAT host interface

Also, in memory map (from ourt latest EtherCAT slave) I see
.rodata.FrameProc
0x8002f8a4 0x1fc8 ./tiescutils.o
0x8002f8a4 FrameProc
Could you please give a try just replacing header files from C:\TI\am335x_sysbios_ind_sdk_1.1.xxx\sdk\protocols\ethercat_slave\firmware and rebuild your project?

Also could you please try to use latest firmware header files from PRU-ICSS EtherCAT slave 1.0.5? (C:\TI\PRU-ICSS-EtherCAT_Slave_01.00.05.00\protocols\ethercat_slave\firmware\v1.0)

thank you,
Paula

Paula,

Thanks for the information again. Our 'tiescutils.c' does also have the identical #include statements, so I'll have to assume that the script file is no longer used. However, dropping in the PRU firmware source files provided by our other group, still does not help our own issue.

To confirm with you: Which link on the page that you pointed me to, for the latest PRU firmware v.1.0.5, actually leads to source files that I can download? I have attached the entire page snapshot here. Please let me know.

Thanks.

John

Hi John, I think you work in windows environment, if so, please download and install PRU-ICSS EtherCAT slave Windows installer. From there you can get the firmware files from this location C:\TI\PRU-ICSS-EtherCAT_Slave_01.00.05.00\protocols\ethercat_slave\firmware\v1.0

thank you,
Paula

Hi Paula,

Thank you for the advice, and with that Windows installer, I was able to retrieve the v.1.0.5 PRU firmware source files. So, I used the 'ecat_frame_handler_bin.h' and 'ecat_host_interface_bin.h' from the 'C:\ti\PRU-ICSS-EtherCAT_Slave_01.00.05.00\protocols\ethercat_slave\firmware\v1.0' folder, in place of the corresponding files in our 'C:\ti\am335x_sysbios_ind_sdk_01.01.03.03\sdk\protocols\ethercat_slave\firmware' folder, and then did a clean rebuild of the project for our slave device firmware, under the CCS toolset However, the issue persists. I tested it simply by issuing a soft reset on our device, and then still observing another device connected at the upstream, to go through the OP->INIT->SAFEOP->OP transition.

Let me know if you see any problem in how I incorporated the v.1.0.5 PRU firmware source code into our own project, as described above. Thank you again.

John

John, to be sure firmware was updated, you can check FW version via TwinCAT. For PRU-ICSS Ethercat slave you should see ESC Type:0x90 Revision: 3  Build: E5 (0x03E5). Attached and old snapshot I found in my files, just to illustrate it. 

For your reference: PRU-ICSS EtherCAT slave Release Notes FW revision 

I will check with our EtherCAT developers, if it is expected or not that upstream devices get affected after our slave resets.

thank you,

Paula  

Post correction in red..

Hi John, we tested three boards + TwinCAT as shown below:
TwinCAT <-> AM3 <-> AM4 <-> AM3
Last slave (ICEv2 board) was reset (hard/soft) and the other two slaves didn't go out of OP state. 

We used latest PRU-ICSS EtherCAT slave 1.0.6 (published in ti.com yesterday), 2 ICEv2 boards and 1 AM4379 IDK board. For booting we used SD card.

Questions:
- For you test, are you using TI's ICEv2 board? or your custom board?. If the second, could you try to test your app on an ICEv2 board and let us know if you still see the same behavior?

thank you,
Paula

Paula,

Thank you for the effort. Let me confirm:

• So, by "...the other two slaves didn't go to OP state", did you mean that you have indeed reproduced the issue which I had described? In our test here, again, the other upstream slave device will go through the OP->INIT->SAFEOF->OP transition, whenever our slave device is being reset.
• When/where is the 1.0.6 PRU firmware available?
• Lastly, I'm sure that there might have been an ICEv2 board available, when the project first started. At this point though, we have been working with exclusively our own custom AM3359-Sitara based board.

Please let me know. Thanks.

John

John, sorry for not being clear. I meant to say the two other slaves didn't go out of OP state.

PRU-ICSS EtherCAT slave 1.0.6 is our latest, you can get it from here: software-dl.ti.com/.../index_FDS.html

Thank you,
Paula

Paula,

However, in your testing, there was no AM3359-based board used, correct?

Thanks.


John

John, as mentioned we used 2 AM3 (TMDSICE3359 a.k.a ICEv2 boards)

Thank you,

Paula

Thanks, Paula, for the clarification. In that case, it can only be our own custom AM3359-based board, that somehow introduced the issue by mistake/accident.


John

John, maybe you can try our OOB PRU-ICSS Ethercat slave 1.0.6  on your custom board? and see how it behaves. You can get EtherCAT slave prebuild binaries from download webpage and MLO file, for SD card booting, from C:\TI\processor_sdk_rtos_am335x_4_03_00_05\prebuilt-sdcards\evmAM335x\sd_card_files) . This test could help to rule out if it is HW or SW problem..

Paula

Paula, we are only using am335x_sysbios_ind_sdk_01.01.03.03. 

John