TM4C1294NCPDT: Ehthernet communication fails after many hours of operation in TM4C1294NCPDT (continuing from earlier thread)

Hi Krishna,

  Unfortunately, I really don't know the reason that is causing it to fail after 30 hours. In the last thread you were trying to investigate if changing a different switch will make a difference? Can you provide some updates?

 I have some questions and suggestions.

 - Will changing a switch make a difference?

 - Do all of your boards have the same problem? If this is the case, then I would focus on the software side.

 - You said your application is based off the tcpecho example. If you were to run the tcpecho example as is will you see any failure after the number of hours of operation you have seen when you run your own application.

 - Here I'm just trying to understand if the TivaWare tcpecho example could be a problem since you said you reference it to develop your application. Can you reference another tcpecho example from https://github.com/dreamcat4/lwip/blob/master/contrib/apps/tcpecho_raw/echo.c. This is a stock lwIP tcpecho example. Looks more complicated but will work. All you need is to call echo_init() from this file. If your application were to base on this echo example, will it make a difference? I just wanted to know if it is a software issue or there are other issues to look at. What I'm trying is a process of elimination to narrow down the cause. 

Hi Charles,

Charles Tsai said:

- Will changing a switch make a difference?

I have tried with a different  set of switch and cables and the problem did not appear even after a week. But with this set of switch and cables the problem occurred twice. Once after 30 hours and then after 6 hours, To eliminate the cable problem , I checked with the 3 cables individually without switch and they all  worked for a day each without a problem, I am strongly suspecting the switch. To confirm it I have to go back to the other switch and observe again. 

In the mean time, I have modified my program ( which is fairly complex with RTC, SD card storage , 2 SPI communications with  MSP432 and a Beagle Bone Black etc and NOSYS and no Ethernet interrupts. Ethernet is handled in the main super loop) so that it works on the Connected launch pad (without the other hardware) and kept it running since yesterday.  I will observe for a couple of days and let you know. If the problem appears here, it may be a bit easier to find the root cause as I am logging the UART outputs using teraterm.

I will try the other suggestion of running a simpler program on the launch pad.

Charles Tsai said:

- You said your application is based off the tcpecho example. If you were to run the tcpecho example as is will you see any failure after the number of hours of operation you have seen when you run your own application.

I will try this too.

 Thank you for your suggestions.

Hi,

Krishna Kumar A V said:

Ethernet is handled in the main super loop

Please read carefully the response by David Wilson in this post   https://e2e.ti.com/support/microcontrollers/other/f/908/t/319674?Optimizing-UDP-in-lwIP-on-TIVA-how-to-determine-when-a-packet-has-been-sent-.  Basically, you should never call LwIP such as tcp_send from a context other than the Ethernet interrupt handler as the LwIP is not re-entrant so you must only call from one context. 

Update:

The program running on Launchpad also stopped communicating on TCP after 37 hours. I have logged the status at regular intervals of 20 minutes and found that at the time of hang EMACIntStatus  becomes zero. I have reset the launch pad to observe again for the next many hours. Later I will replace the switch and try again and report. 

Once the firmware detects the hang it waits for  30 seconds and  calls lwIPNetworkConfigChange with the old values in an attempt to restart. But this has no effect.

However if we do a software reset the MCU it recovers. But this is not a desirable work around as we lose some data during reset.

Is there any other parameter I have to log to get an insight into the problem? 

Hi,

Krishna Kumar A V said:

Ethernet is handled in the main super loop

You said you handle Ethernet in the main loop. Did you have a chance to read the other post answered by David Wilson. I've copied his answer below again. I'm suspecting if your problem is due to the reason as stated. 

The biggest problem here is the fact that you are calling lwIP from a context other than the Ethernet interrupt handler. lwIP is not re-entrant so you must only call it from one context. When using an RTOS, this means only calling the lwIP APIs from a single thread. When not using an RTOS (like most of our examples), you must make calls in the context of the Ethernet interrupt handler. This can be tricky to do but there's a timer callback implemented that makes it a bit easier. Make sure you set HOST_TMR_INTERVAL to some non-zero value (I think it's a number of milliseconds) and implement lwIPHostTimerHandler in your application then make all your lwIP calls from the lwIPHostTimerHandler function.

This may sound awkward but it's vital. Calling udp_send from the main loop will likely work for a while but I guarantee you that it will crash after some period of time as some internal data structure in lwIP gets corrupted. This kind of problem is VERY difficult to debug so it's far better to fix this now rather than trying to pretend everything is OK and have it fail catastrophically later.

Hi Charles,

I think I am doing this part correctly. I will cross check again. Our requirement is that the timer interrupt (10ms) is used for data capture which cannot be interrupted by any other task, even the Ethernet. For this reason we have taken pains to see that Ethernet functionality is done without the Hardware Ethernet Interrupt. 

I will  make a concise file showing the modifications made to achieve this. 

Essentially I have followed this article by High 12 noon Blog  to the letter 

high12noon.neocities.org/lwip_polled_tm4c129.html

BTW the second test with Launch pad failed after 10 hours.

Now we have replaced the suspicious switch (D-Link D1008D) with another one (D-Link DES1008C) . We will observe this for a couple of days and report.

Thank you.

Ok, let us now the result of using a different switch.