TM4C1294NCPDT: Ethernet TCP Communication Lost and Device Hang

Mukund Kumbhar said:

It is TCP Client. We are facing issue of Ethernet communication, at one point communication stops, device get hang and it is not recovering or reconnecting ever. Ethernet Link Status LEDs - OFF , and there is no ping. In code we have also used Watch Dog timer , it seems it is also not Resetting controller,

Can you provide more information? Your answers to these questions will help narrow down the problem area.

Was the MCU running the TCP client application fine for a while before it hung? 

If you use the debugger to examine the processor, where is it stuck?

Is the problem happening on only one board or all boards you have running the same application?

Does a power cycle restore the device?

If you load a different application, will it hang or only this particular TCP client application will hang?

Can you try some TivaWare Ethernet examples. If you try these examples, will it also hang?

Mukund Kumbhar said:

Having compiler Version below :

CCS Version - 7.4.0

As our application is based on Ethernet TCP Protocol, using LWIP Stack - 1.4.1.

Although no issue with using CCS 7.4.0, this is an extremely old version.

What TivaWare version are you using? The latest version is 2.2.0.295 which can be downloaded from https://www.ti.com/tool/SW-TM4C

Was the MCU running the TCP client application fine for a while before it hung?

Yes ,Normally it is working fine ,We don't know due to high data traffic or other reason it hung, As this issue we have observed at client site,

If you use the debugger to examine the processor, where is it stuck?

We are not able to replicate the same issue using our set up in the lab, communication is working as intended, 

Is the problem happening on only one board or all boards you have running the same application?

At site our 26 devices( TCP Telnet Server - Modbus TCP) are used to communicate with SCADA system(TCP Telnet Client - Modbus TCP). Same Application is running on all devices and issue observed on any random device 

Does a power cycle restore the device?

Yes , After giving power cycle , device works fine 

If you load a different application, will it hang or only this particular TCP client application will hang?

We have not checked on any other application.

Can you try some TivaWare Ethernet examples. If you try these examples, will it also hang?

Yes , we can try, however issue is not getting replicate at test lab with existing firmware

What TivaWare version are you using? 

TivaWare Version  - 2.1.4.178 ,  Can we port our existing firmware to Version 2.2.0.195 without changing Application layer.

Please suggest solution or work around for this problem.

Thanks for your clarification. In the future, please describe your problem as clearly as possible like you just did. Your original post has not much information beside the the client is somehow stuck. 

Mukund Kumbhar said:

At site our 26 devices( TCP Telnet Server - Modbus TCP) are used to communicate with SCADA system(TCP Telnet Client - Modbus TCP). Same Application is running on all devices and issue observed on any random device 

Is Tiva MCU a Modbus client or a Modbus server? You seem to suggest the 26 devices are Tiva MCU but you said the problem is on the client for the SCADA system. 

 I will suggest you use the wireshark to see if it will provide some hints. TivaWare 2.2.0.295 also uses lwIP 1.4.1. I doubt it will produce any difference as far as your problem is concerned. 

Mukund Kumbhar said:

We are not able to replicate the same issue using our set up in the lab, communication is working as intended, 

So the problem only happens on the field? If you can't even replicate the problem in your lab, I really don't know what to suggest. Perhaps, check your cables and as suggested use Wireshark for some hints as to when and where it fails. 

Hello Charles ,

All 26 devices are Modbus TCP Servers, not Clients. Sorry, that’s my mistake. These devices are connected to a SCADA system, which acts as the Modbus TCP Client.

We have tried a couple of things on our end to intentionally disable the PHY and read its status:

We attempted to disable EMAC0 and PHY0 using the EMAC and PHY disable commands on a push button event. After executing the instructions MAP_SysCtlPeripheralDisable(SYSCTL_PERIPH_EMAC0) and MAP_SysCtlPeripheralDisable(SYSCTL_PERIPH_EPHY0), the MCU hangs, serial debugging stops, no further code execution occurs, and a WatchDog Reset happens. These instructions should simply disable EMAC and PHY and then execute the next instructions. What could be causing the MCU to hang and the WatchDog Reset to occur?
The purpose of this exercise is to implement a workaround for when the PHY is not working, we want to soft reset the MCU and reinitialize the PHY.

Please suggest a way to get the status of the PHY at runtime when the PHY is not working or has any issues.

I will suggest you reset EMAC and PHY instead. Calling lwIPInit again will do reset both EMAC and PHY and initiate lwIP stack again. See below snippet.

void
lwIPInit(uint32_t ui32SysClkHz, const uint8_t *pui8MAC, uint32_t ui32IPAddr,
uint32_t ui32NetMask, uint32_t ui32GWAddr, uint32_t ui32IPMode)
{
//
// Check the parameters.
//
#if LWIP_DHCP && LWIP_AUTOIP
ASSERT((ui32IPMode == IPADDR_USE_STATIC) ||
(ui32IPMode == IPADDR_USE_DHCP) ||
(ui32IPMode == IPADDR_USE_AUTOIP));
#elif LWIP_DHCP
ASSERT((ui32IPMode == IPADDR_USE_STATIC) ||
(ui32IPMode == IPADDR_USE_DHCP));
#elif LWIP_AUTOIP
ASSERT((ui32IPMode == IPADDR_USE_STATIC) ||
(ui32IPMode == IPADDR_USE_AUTOIP));
#else
ASSERT(ui32IPMode == IPADDR_USE_STATIC);
#endif

//
// Enable the ethernet peripheral.
//
MAP_SysCtlPeripheralEnable(SYSCTL_PERIPH_EMAC0);
MAP_SysCtlPeripheralReset(SYSCTL_PERIPH_EMAC0);

//
// Enable the internal PHY if it's present and we're being
// asked to use it.
//
if((EMAC_PHY_CONFIG & EMAC_PHY_TYPE_MASK) == EMAC_PHY_TYPE_INTERNAL)
{
//
// We've been asked to configure for use with the internal
// PHY. Is it present?
//
if(MAP_SysCtlPeripheralPresent(SYSCTL_PERIPH_EPHY0))
{
//
// Yes - enable and reset it.
//
MAP_SysCtlPeripheralEnable(SYSCTL_PERIPH_EPHY0);
MAP_SysCtlPeripheralReset(SYSCTL_PERIPH_EPHY0);
}
else
{
//
// Internal PHY is not present on this part so hang here.
//
while(1)
{
}
}
}

//
// Wait for the MAC to come out of reset.
//
while(!MAP_SysCtlPeripheralReady(SYSCTL_PERIPH_EMAC0))
{
}

//
// Configure for use with whichever PHY the user requires.
//
EMACPHYConfigSet(EMAC0_BASE, EMAC_PHY_CONFIG);

//
// Initialize the MAC and set the DMA mode.
//
MAP_EMACInit(EMAC0_BASE, ui32SysClkHz,
EMAC_BCONFIG_MIXED_BURST | EMAC_BCONFIG_PRIORITY_FIXED,
4, 4, 0);

//
// Set MAC configuration options.
//
MAP_EMACConfigSet(EMAC0_BASE, (EMAC_CONFIG_FULL_DUPLEX |
EMAC_CONFIG_CHECKSUM_OFFLOAD |
EMAC_CONFIG_7BYTE_PREAMBLE |
EMAC_CONFIG_IF_GAP_96BITS |
EMAC_CONFIG_USE_MACADDR0 |
EMAC_CONFIG_SA_FROM_DESCRIPTOR |
EMAC_CONFIG_BO_LIMIT_1024),
(EMAC_MODE_RX_STORE_FORWARD |
EMAC_MODE_TX_STORE_FORWARD |
EMAC_MODE_TX_THRESHOLD_64_BYTES |
EMAC_MODE_RX_THRESHOLD_64_BYTES), 0);

//
// Program the hardware with its MAC address (for filtering).
//
MAP_EMACAddrSet(EMAC0_BASE, 0, (uint8_t *)pui8MAC);

//
// Save the network configuration for later use by the private
// initialization.
//
g_ui32IPMode = ui32IPMode;
g_ui32IPAddr = ui32IPAddr;
g_ui32NetMask = ui32NetMask;
g_ui32GWAddr = ui32GWAddr;

//
// Initialize lwIP. The remainder of initialization is done immediately if
// not using a RTOS and it is deferred to the TCP/IP thread's context if
// using a RTOS.
//
#if NO_SYS
lwIPPrivateInit(0);
#else
tcpip_init(lwIPPrivateInit, 0);
#endif
}

Hello Charles ,

Thank you for your suggestion , but question here is , till I get what is status of my PHY, how and exactly where I can re-initialize it. In this case, there is Link Status bit from which I will get the status of physical link only, suppose physical connection is fine, and still there is problem with communication, so what are that registers from which I can get to know that there is issue with PHY or EMAC. 

Hi,

  In lwIP there is the link timer that will check for link status periodically. Please refer to the lwIPServiceTimers() function in lwiplib.c file. 

#if NO_SYS
static void
lwIPServiceTimers(void)
{
//
// Service the host timer.
//
#if HOST_TMR_INTERVAL
if((g_ui32LocalTimer - g_ui32HostTimer) >= HOST_TMR_INTERVAL)
{
g_ui32HostTimer = g_ui32LocalTimer;
lwIPHostTimerHandler();
}
#endif

//
// Service the ARP timer.
//
#if LWIP_ARP
if((g_ui32LocalTimer - g_ui32ARPTimer) >= ARP_TMR_INTERVAL)
{
g_ui32ARPTimer = g_ui32LocalTimer;
etharp_tmr();
}
#endif

//
// Service the TCP timer.
//
#if LWIP_TCP
if((g_ui32LocalTimer - g_ui32TCPTimer) >= TCP_TMR_INTERVAL)
{
g_ui32TCPTimer = g_ui32LocalTimer;
tcp_tmr();
}
#endif

//
// Service the AutoIP timer.
//
#if LWIP_AUTOIP
if((g_ui32LocalTimer - g_ui32AutoIPTimer) >= AUTOIP_TMR_INTERVAL)
{
g_ui32AutoIPTimer = g_ui32LocalTimer;
autoip_tmr();
}
#endif

//
// Service the DCHP Coarse Timer.
//
#if LWIP_DHCP
if((g_ui32LocalTimer - g_ui32DHCPCoarseTimer) >= DHCP_COARSE_TIMER_MSECS)
{
g_ui32DHCPCoarseTimer = g_ui32LocalTimer;
dhcp_coarse_tmr();
}
#endif

//
// Service the DCHP Fine Timer.
//
#if LWIP_DHCP
if((g_ui32LocalTimer - g_ui32DHCPFineTimer) >= DHCP_FINE_TIMER_MSECS)
{
g_ui32DHCPFineTimer = g_ui32LocalTimer;
dhcp_fine_tmr();
}
#endif

//
// Service the IP Reassembly Timer
//
#if IP_REASSEMBLY
if((g_ui32LocalTimer - g_ui32IPReassemblyTimer) >= IP_TMR_INTERVAL)
{
g_ui32IPReassemblyTimer = g_ui32LocalTimer;
ip_reass_tmr();
}
#endif

//
// Service the IGMP Timer
//
#if LWIP_IGMP
if((g_ui32LocalTimer - g_ui32IGMPTimer) >= IGMP_TMR_INTERVAL)
{
g_ui32IGMPTimer = g_ui32LocalTimer;
igmp_tmr();
}
#endif

//
// Service the DNS Timer
//
#if LWIP_DNS
if((g_ui32LocalTimer - g_ui32DNSTimer) >= DNS_TMR_INTERVAL)
{
g_ui32DNSTimer = g_ui32LocalTimer;
dns_tmr();
}
#endif

//
// Service the link timer.
//
#if LWIP_AUTOIP || LWIP_DHCP
if((g_ui32LocalTimer - g_ui32LinkTimer) >= LINK_TMR_INTERVAL)
{
g_ui32LinkTimer = g_ui32LocalTimer;
lwIPLinkDetect();
}
#endif
}
#endif

Hello Charles,

Using this link status I already  checked and I am getting status of physical link that is fine. other than that is there any way or registers by reading that we can identify that there is some issue with PHY / EMAC so Reset controller. 

I was checking different registers , found Remote Fault Bit in EPHYBMSR register and read that bit before Ethernet Initialization and after communication established , getting same value 0x00000040, indicates that , this bit is high ( Verified by disconnecting ethernet cable and Mod Poll connection)

one more register EPHYCONFIG1 having TDR and LLR bit. Can DMA or EMAC registers be useful to detect reception or transmission error , can you please suggest exact registers.