AM2434: AM243 enhanced link communication issue for Ethercat

Hi Aaron

As discussed via email, this issue only happen after merge the patch from you to fix previous issue.

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1650989/am2432-am243-enhance-link-mode-will-configure-phy-to-power-down-status-and-not-recover/6369166

Thanks

Zekun

Hi Zekun, 

"We have conducted test verification and confirmed that this issue can be resolved" - May I know how was the patch verified at the customer's end with Enhanced Link enabled earlier? 

Also, has the customer enabled "MDIO Manual Mode" in their application?

Please share the complete ICSSG1 Memory dump when the issue occurs - How to Capture a Full ICSSG Memory Dump. 

Regards,
Aaron

Hi  Aaron

Aaron Thomas said:

"We have conducted test verification and confirmed that this issue can be resolved" - May I know how was the patch verified at the customer's end with Enhanced Link enabled earlier? 

Directly replace the original file at the corresponding location within the existing SDK with the new patch file.

Network setup: Beckhoff PLC + an AM2432 device with Enhanced Link enabled.
The device is detected normally; after the EtherCAT state switches to OP, the device is power-cycled (repeatedly powered off and on).

Aaron Thomas said:

Also, has the customer enabled "MDIO Manual Mode" in their application?

Yes

Aaron Thomas said:

Please share the complete ICSSG1 Memory dump when the issue occurs - How to Capture a Full ICSSG Memory Dump. 

Will share later.

Thanks

Zekun

Hi Zekun,

Zekun Bai said:

The device is detected normally; after the EtherCAT state switches to OP, the device is power-cycled (repeatedly powered off and on).

Understood. Thank you for the details on the test environment.

Another follow-up query, is the PHY asserting RX_ERR in the test environment? This will help us boil down the particular scenario within the firmware. I assume RX_ERR is asserted in the customer's setup, assuming similar environment was handled in the previous test run for PHY Power down issue, and the fix/patch was done in that particular section where RX_ERR is asserted.

To give a background on my initial hypothesis, what I suspect is that in the patch we provided earlier, we made a change in sequence of how the PRU access the PHY register through the MDIO_USER_ACCESS_REG Register. In order to make sure stale data is not read during the Read-Modify-Write of the PHY BMCR Register, we introduced a GO bit check (description of the bit below) to make sure the transaction is complete and PRU does not read stale value from the MDIO_USER_ACCESS_REG:

To confirm if this is the issue, we would like you to monitor the MDIO User Access 0 Register within the Emulated space for MDIO Registers (0x30090EC0-0x30090EC3) when the issue occurs to see if the GO bit gets cleared or not:

Do let me know if the customer have any more follow-up questions.

Regards,
Aaron

Zekun Bai said:

where the Beckhoff master fails to receive returning data frames

Additionally, is it possible to confirm if the SubDevice has processed the forward frame successfully and if the frame is lost in the return path? The customer can keep an ET2000 or any tap before and after the AM243x:

The activity will help us in locating if the issue is with respect to forward path (LOC1) or reverse path (LOC2).

Regards,
Aaron

Hi Zekun,

Apologies for spamming with a lot of details. We had a discussion internally and would like to rule out some more possibilities. Since the issue is stemming out of the patch I provided earlier, following are the initial hypothesis:

• MDIO GO bit is not cleared internally when the PRU firmware initiates Auto-negotiation restart. This can be ruled out by checking the corresponding bit as mentioned above and also through the PHY Register (BMCR and BMSR) polling during the RX_ERR event. This is the section modified within the patch, so just wanted to rule out any HW related issue.
• On restart of the SubDevice, the PHY polarity will be swapped or the PHY address may be changed, due to some variation in the initialization sequence or race condition. Couple of things to check on is:
  • When the issue occurs, remove and reconnect the Ethernet cables and check if the PHY links has been recovered. Monitor the behavior of MDIO Alive Register and MDIO Link Register:
    • 
    • When the issue occurs, is the PHY address or polarity getting changed? The polarity can be interpreted from the steps mentioned in AM243x INDUSTRIAL COMMUNICATIONS SDK: EtherCAT SubDevice FWHAL
  • Change MII_LINK0_EVENT and MII_LINK0_EVENT system event type to SYS_EVT_TYPE_PULSE in source\industrial_comms\ethercat_subdevice\icss_fwhal\tiesc_pruss_intc_mapping.h:
    •  

      {MII_LINK0_EVENT, CHANNEL1, SYS_EVT_POLARITY_HIGH, SYS_EVT_TYPE_PULSE},\
      {MII_LINK1_EVENT, CHANNEL1, SYS_EVT_POLARITY_HIGH, SYS_EVT_TYPE_PULSE},\

    • Rebuild the FWHAL as required.

Further I can check on the system based on the memory dump we are yet to receive from the customer.

Let me know the status of the above points.

Regards,
Aaron

Hi  Aaron

Aaron Thomas said:

To confirm if this is the issue, we would like you to monitor the MDIO User Access 0 Register within the Emulated space for MDIO Registers (0x30090EC0-0x30090EC3) when the issue occurs to see if the GO bit gets cleared or not:

MDIO Registers value is 0x22630000.

Aaron Thomas said:

Additionally, is it possible to confirm if the SubDevice has processed the forward frame successfully and if the frame is lost in the return path? The customer can keep an ET2000 or any tap before and after the AM243x:

The issue is with respect to forward path (LOC1).

Aaron Thomas said:

When the issue occurs, remove and reconnect the Ethernet cables and check if the PHY links has been recovered

It can be recovered

The PHY LINK signal is active‑low. The value of register 0x0E48 is 0x0000000C, and the value of register 0x0E4C is 0x00000008.

 Hi  Aaron

Aaron Thomas said:

Please share the complete ICSSG1 Memory dump when the issue occurs - How to Capture a Full ICSSG Memory Dump. 

start_fail.dat

Hi  Aaron

MDIO Registers value is 0x22630000.

The PHY LINK signal is active‑low. The value of register 0x0E48 is 0x0000000C, and the value of register 0x0E4C is 0x00000008.

When reproducing the issue, the value of the PHY RX_ERR(phyRegIn[21]) register is 0.

Thank you Yijin for the details.

yijin zhao said:

The PHY LINK signal is active‑low. The value of register 0x0E48 is 0x0000000C, and the value of register 0x0E4C is 0x00000008.

So the PHY Address 0 is 2 and PHY Address 1 is 3 with both PHY polarity at "Active Low". I hope MLINK mode is set in "MDIO Manual Mode Link Status Update":

yijin zhao said:

start_fail.dat

I'll review this.

yijin zhao said:

When reproducing the issue, the value of the PHY RX_ERR(phyRegIn[21]) register is 0.

Understood. So can rule out any changes with respect to the recent fix for PHY Power Down. Also, are you sure that this issue is not observed with the default firmware from ICSDK v2026.00 ? Is the error observed with the patch only?

Regards,
Aaron

Hi  Aaron

Aaron Thomas said:

So the PHY Address 0 is 2 and PHY Address 1 is 3 with both PHY polarity at "Active Low". I hope MLINK mode is set in "MDIO Manual Mode Link Status Update":

Yes. PHY Address 0 is 2 and PHY Address 1 is 3. MLINK mode is set in "MDIO Manual Mode Link Status Update".

Aaron Thomas said:

Also, are you sure that this issue is not observed with the default firmware from ICSDK v2026.00 ? Is the error observed with the patch only?

I am not sure whether this issue exists in the ICSDK v2026.00 . Previously, we frequently encountered the problem that the PHY link could not be established.

Hi Yijin,

Aaron Thomas said:

Change MII_LINK0_EVENT and MII_LINK0_EVENT system event type to SYS_EVT_TYPE_PULSE in source\industrial_comms\ethercat_subdevice\icss_fwhal\tiesc_pruss_intc_mapping.h:

Can you provide an update for this action item as well? Is PULSE instead EDGE mode helping in this scenario?

Regards,
Aaron

Hi  Aaron

Aaron Thomas said:

Change MII_LINK0_EVENT and MII_LINK0_EVENT system event type to SYS_EVT_TYPE_PULSE in source\industrial_comms\ethercat_subdevice\icss_fwhal\tiesc_pruss_intc_mapping.h:

Yes, modifying the trigger event from edge to pulse can resolve this issue.

I have several questions regarding this modification and would appreciate your clarification:

1、What is the root cause why modifying the trigger event from edge to pulse can resolve this issue?
2、What potential impacts will this modification bring to the overall system?
3、Are there any other system events that also require the same edge-to-pulse adjustment?

Thank you.

Hi Yijin,

yijin zhao said:

1、What is the root cause why modifying the trigger event from edge to pulse can resolve this issue?

There can be a scenario where the PRU is late to be enabled due to some race condition in the initialization sequence and due to the race condition, the link signal may be missed by the PRU.

yijin zhao said:

2、What potential impacts will this modification bring to the overall system?

I will have to review the impact analysis regarding this modification. Will review and get back soon.

yijin zhao said:

3、Are there any other system events that also require the same edge-to-pulse adjustment?

No, only MII link event needs this update.

Regards,
Aaron

Hi Yijin,

Can you provide more details on the initialization sequence of PHY, MCU and the PRU core. Is it different from the default boot-up sequence from the SDK for TI EVM?

What are the changes between TI EVM and the custom board the customer is using? I understand the PHY used is YT8522, apart from that, what all delats as there in the HW initializations?

Regards,
Aaron

Hi  Aaron

Aaron Thomas said:

Is it different from the default boot-up sequence from the SDK for TI EVM?

The boot-up sequence is mostly the same as the SDK for TI EVM, except our software doesn’t handle PHY power down.

Hi Yijin,

yijin zhao said:

The boot-up sequence is mostly the same as the SDK for TI EVM, except our software doesn’t handle PHY power down.

Any specific reason why PHY Power Down is not handled in your software? It is a critical requirement to keep the PHY in power down till the ESC is ready.

Aaron Thomas said:

Any specific reason why PHY Power Down is not handled in your software?

We used ESC  from another vendor, which does not impose special requirements for PHY power‑down.

Hi  Aaron

I would like to know whether this issue is related to the PHY power‑down operation?

What is the final solution for this issue?

Hi  Aaron

Aaron Thomas said:

It is a critical requirement to keep the PHY in power down till the ESC is ready.

We have added PHY power‑down handling following the reference routine from the TI EVM SDK. However, the issue of continuous frame loss still reproduces after more than 70 power‑cycle tests on the device.

Hi Yijin,

Apologies for the delay in response as I was out of office for couple of weeks.

yijin zhao said:

We have added PHY power‑down handling following the reference routine from the TI EVM SDK. However, the issue of continuous frame loss still reproduces after more than 70 power‑cycle tests on the device.

Is this with SYS_EVT_TYPE_PULSE or SYS_EVT_TYPE_EDGE configuration?

In earlier replies, you haven't mentioned how many retries was to be done to get a failing scenario. In this test sequence, are you seeing failure after 70 iterations of power-cycle or you see couple of failure throughout the 70 iterations of power-cycle?

With SYS_EVT_TYPE_PULSE configuration, are you seeing failure in multiple iterations of power-cycle?

Regards,
Aaron

Hi  Aaron

Aaron Thomas said:

Is this with SYS_EVT_TYPE_PULSE or SYS_EVT_TYPE_EDGE configuration?

This configuration is SYS_EVT_TYPE_EDGE.

Aaron Thomas said:

In earlier replies, you haven't mentioned how many retries was to be done to get a failing scenario. In this test sequence, are you seeing failure after 70 iterations of power-cycle or you see couple of failure throughout the 70 iterations of power-cycle?

The failure was observed after 70 iterations of power-cycle .

Aaron Thomas said:

With SYS_EVT_TYPE_PULSE configuration, are you seeing failure in multiple iterations of power-cycle?

No failures were found in all 5000 power cycles after applying SYS_EVT_TYPE_PULSE.

Understood. Thank you for the clarification.

We would recommend you to configure SYS_EVT_TYPE_PULSE going forward.

Based on our analysis on the use case of EDGE mode, we see that in the older device baseline, it was kept at SYS_EVT_TYPE_PULSE mode, and due to the below mentioned issue, it was changed to SYS_EVT_TYPE_EDGE:

So we suspect PINDSW-3120 will re-surface in certain scenario when PULSE mode is used.

In order to keep the implementation clean, we had a review of this internally and aligned to make changes in the PRU EtherCAT Firmware to not use INTC for Lost Link Counter, rather, update the register value based on the MDIO Link Status Register, thereby removing Lost Link Counter Register dependency with MII_LINK0_EVENT/MII_LINK1_EVENT. 

Regards,
Aaron

Hi  Aaron

I understand. The proposed solution is to use the SYS_EVT_TYPE_PULSE configuration, and you will release a new firmware to fix the inaccurate link loss counter issue.

Could you advise when this firmware will be available to us?

Hi Aaron

As discuss in email, this issue have blocked customer for a long time. So we need below information be clear ASAP.

1. What is the proposed solution to fix this?

2. If a SYS_EVT_TYPE_PULSE + new Firmware is our final solution, when can we provide it.

3. What is influence change about change to SYS_EVT_TYPE_PULSE?

4. What is the change point in the new firmware release? What is the influence of these change points?

Thanks

Zekun

Thank you for your understanding.

yijin zhao said:

Could you advise when this firmware will be available to us?

Our suggestion would be to use the firmware from the upcoming SDK release (we can provide an Early Access) for the above mentioned issue, along with few other fixes made on top of 333MHz configuration and DC Sync related bug fixes.

Do let me know which is the best approach for your further development.

Regards,
Aaron 

Hi Zekun,

Zekun Bai said:

1. What is the proposed solution to fix this?

2. If a SYS_EVT_TYPE_PULSE + new Firmware is our final solution, when can we provide it.

3. What is influence change about change to SYS_EVT_TYPE_PULSE?

4. What is the change point in the new firmware release? What is the influence of these change points?

These are clarified in this E2E thread.

Regards,
Aaron

Hi  Aaron

Aaron Thomas said:

Our suggestion would be to use the firmware from the upcoming SDK release (we can provide an Early Access) for the above mentioned issue

Is this Early Access firmware stable?  When can we receive the firmware?

Additionally, we have two more requests:
1. Please provide a document summarizing the changes and impact scope between this firmware and the previous version x0690_055F (6.5.95).
2. Could you explain why the port cannot be opened when configured to SYS_EVT_TYPE_EDGE?

Hi  Yijin

I am pushing this issue internally and will give you update soon.

Hi Aaron

As discussed in email, let us sync up this request via tomorrow meeting.

Thanks

Zekun

Hi Aaron

Thanks for meeting yesterday.

Besides the solution we offered, could you explain question 2? Customer need these info to evaluate the project influence.

yijin zhao said:

2. Could you explain why the port cannot be opened when configured to SYS_EVT_TYPE_EDGE?

Thanks

Zekun

Hi Aaron

Firmware x0690_056C (6.5.108) contains an issue. Directly replacing the x0690_055F (6.5.95) firmware binary with x0690_056C (6.5.108) causes EtherCAT communication failure. The PHY link is up, but ESC register 0x111 reads 0x56, the port stays closed, and frames are dropped continuously. We hope this can be resolved as soon as possible. Thank you.

Hi Yijin,

Is this observed on a custom hardware or on AM243x TI EVM/LP?

Is the Enhanced Link enabled? If yes, how is the link polarity - active high or active low?

The reason for asking the above is due to the fact that the drop has been validated with CTT v2.8.2 along with ESC Test Suite and TwinCAT free run. I’m trying to understand the scenario of this issue you are observing.

Regards,
Aaron

Aaron

Customer will feedback about the test case. But the whole project is blocked by this issue. They can not move forward due to this issue and need TI to release a new version today for them to do pressure test during the weekend. 
They are requesting a on-line meeting this afternoon to sync up all the test details. Please let me know your available time. 

Thanks

Zekun

Hi Zekun,

Zekun Bai said:

They can not move forward due to this issue and need TI to release a new version today for them to do pressure test during the weekend. 

As discussed over call, the fix is just in the diagnostics. Customer can always fall back to the previously working firmware if it is blocking. 

Zekun Bai said:

They are requesting a on-line meeting this afternoon to sync up all the test details

Sure, we can have one. But getting the test case details prior to the meeting will help us understand the issue with more clarity. I will re-check on the firmware with an older firmware baseline and see if the issue is with respect to any driver change.

One more clarification required - is the customer using TI Stack demo or Beckhoff SSC Demo?

Regards,
Aaron

Hi Yijin,

Couple of clarifications:

yijin zhao said:

The PHY link is up, but ESC register 0x111 reads 0x56, the port stays closed, and frames are dropped continuously.

Is this with respect to a single port connection or are both ports connected? In a single port connection, 0x56 is expected (bit8 = 0 which means port0 open, bit9 = 1 which means communication established)

Are you using LRD/LWR datagrams? 

Regards,
Aaron

Hi Aaron

Please find the clarifications below:

1. The issue is reproduced on our hardware.
2. It can be reproduced consistently, **whether Enhanced Link is enabled or disabled**.
3. The link signal is active-low.
4. We are using the Beckhoff SSC demo project.
5. Only one port is connected. My previous description was not accurate: the port opens normally. The problem is that after the master switches to SafeOP state, the EPU does not process packets correctly and no response packets are returned.
6. LWR and LRD datagrams are used.
7. The test environment remains unchanged and is the same as mentioned earlier in this E2E thread.

Hi Yijin,

Thank you for the details. It confirms my initial hypothesis. Please test with the latest drop and let us know of the status.

Regards,
Aaron

Hi Aaron

We have tested with the latest firmware. The continuous frame loss issue is fixed, and the subinstance can transition to the OP state normally now.

When Enhanced Link is enabled:

- The IN port link loss counter in register 0x310 is correct.
- The OUT port link loss counter shows abnormal behavior. Its value increases substantially upon each cable plug/unplug action.

When Enhanced Link is disabled:

- Both IN and OUT port link loss counters in register 0x310 work properly.

We hope this can be resolved as soon as possible. Thank you.

Hi Yijin,

Thank you for the feedback.

yijin zhao said:

When Enhanced Link is enabled:

- The IN port link loss counter in register 0x310 is correct.
- The OUT port link loss counter shows abnormal behavior. Its value increases substantially upon each cable plug/unplug action.

I am not seeing an abnormal behavior with Port1 link up/down. Are you correctly configuring SYS_EVT_TYPE_PULSE for MII_LINK0_EVENT/MII_LINK1_EVENT?

Regards,
Aaron

Hi Aaron

Aaron Thomas said:

Are you correctly configuring SYS_EVT_TYPE_PULSE for MII_LINK0_EVENT/MII_LINK1_EVENT?

Yes.

Test setup: Both the IN port and OUT port are connected to devices. We plug and unplug the cable on the OUT side.