NDK stops working in case of a big bidirectional traffic.

Victor Ivanov

Other Parts Discussed in Thread: OMAPL138

Hi All!

This thread was created as a replacement of http://e2e.ti.com/support/embedded/bios/f/355/t/223029.aspx?pi239031349=4. I was told by our local TI distributor that due to the fact that the previous thread was marked as answered it doesn't get enough attention.

Brief description of the problem:

Network Development Kit (NDK) working on DM648 stops receiving packets in case of significant bidirectional traffic and does not recovers until restart.

In our project, we digitize analog signal and send it to a PC via Ethernet. The traffic is about 150 Mb/s - 200 Mb/s. Also, the board and the PC exchange commands and status messages. While testing the system we noticed that it stops communicating in a few hours. Later we reproduced the problem on TMS320DM648DVDP bought from TI company using slightly modified example project provided by TI (helloWorld). With help of hping tool we were able to reproduce the problem faster, without waiting for a few hours.

Configuration:

CCS 5.1.1.00031

bios_5_42_00_07

ndk_2_20_06_35

ethss_dm648 from ndk_2_0_0. According to TI's advice all "printf" were removed

The project was taken from

ndk_2_0_0\packages\ti\ndk\example\network\helloWorld

This project originally was for CCS 3.3 so I had to convert it to use with CCS 5.1.

I did the following changes in the pfoject:

- an IP address is set statically

- dtask_udp_hello demon and its creation were commented out

- A MainTask thread was created

MainTask sends UDP packets (1000 bytes). The stream is about 150 Mb/s. We also reproduced the problem with 80 Mb/s.

The evaluation module DM648 DVDP is directly connected to a PC. To have more control on the PC we use Free BSD UNIX now. On the PC's side we use hping in flood mode to send packets to the board, netstat to see what is going on on the net and normal ping to test if the board responds (when hping works in flood mode it reports nothing). hping can provide a very big amount of packets and it makes the problem appears faster. Usually we start hping in two threads to use both CPUs of the PC and create maximum traffic. In this case the problem appears within minutes.

There is no application on the PC which would receive data from DM648.

As we understand the situation now, it is necessary that packets were sent in both ways from DM648 and to DM648. The problem happens with both 100 Mb and 1Gb connections.

How to reproduce the problem:

1. Connect the board and the PC directly with a cable. All modern cards don't require cross cable anymore.

2. Set desired IP addressed on the board and PC.

3. Run the project as it is on the board.

4. Run ping on the PC to watch the state of the board.

5. Start two terminals on the PC and in each run hping –flood –udp 192.168.1.100 (where 192.168.1.100 is the board's IP address)

The board will stop responding to the ping within a few minutes. After another few minutes, when the timeout expires in the route table on the board, the application will not be able to send data and sendto function will return 64.

Investigation:

Investigating this problem we found out that there is no Rx interrupts and emacDequeueRx function is not called.

Previous communication with TI:

I have placed the information about this problem on TI's e2e forum: http://e2e.ti.com/support/embedded/bios/f/355/t/223029.aspx?pi239031349=2 in 24th of November. This thread was created by another engineer who has a similar problem but on the other processor.

Then there was a short discussion with Steven Connell from TI and a pause till 23rd of January.

In 23rd of January Steven told me that he was ready to try to reproduce the issue and asked about the details.

In 7th of February Steven wrote the following:

Hi Victor,

I just wanted to update you. We are able to reproduce this issue and are seeing it on different hardware platforms. We are currently debugging the issue and will let you know as soon as we have something ... thanks again for your patience.

Steve

Since that moment we haven't received any updates about the status of the issue.

Also I tried to communicate with TI's support [Service Request# 1-926930884] but they advised me to continue to use e2e.

We are running out of time, our project is behind of the scheduler and we need at list to know what to expect:

1. Is TI working on it?

2. Is it software or hardware problem?

3. Is it going to be fixed and when?

Best Regards

Victor Ivanov

over 11 years ago

0 Victor Ivanov over 10 years ago in reply to old Wang

Expert 1160 points

Hello Sudhakar!

I confirm that the fix works fine in the test but I'm still investigating why it doesn't solve the problem in the real system. Until the problem is solved in the real system I am not allowed to mark the question as answered. I will let you know as soon as I find anything.

Best Regards

Victor

0 Victor Ivanov over 10 years ago in reply to Victor Ivanov

Expert 1160 points

Hello Sudhakar!

As you have found the hardware sometimes generates an interrupt before the previous ISR is completed
and the interrupt is acknowledged. In the provided fix this interrupt is just skipped. However, this is not an odd interrupt, it just must be generated later when the previous one is acknowledged. Thus we can't just ignore it, we must handle it later when the previous ISR is finished.

To do it I have created a new thread in the driver and moved all ISR code to this thread.

void NDKISRTask(Arg arg0)
{
Uint32 intflags;
Uint32 i;

while(1)
{
SEM_pend(ISRSem, SYS_FOREVER);
if( hEMAC )
{
//
// Read the interrupt cause
//
intflags = CPSW3G_REGS->CPDMA_IN_VECTOR;

//
// Look for TX interrupt
//
if (intflags & CSL_CPSW_MACINVECTOR_TXPEND_MASK)
{
i = EMAC_TxServiceCheck(hEMAC);

if(i)
{
if (i == CPSW3G_ERR_MACFATAL)
emac_fatal_error++;
else
csl_errors++;
}
// write the EOI register
EMAC_txEoiWrite(coreNum);
}

//
// Look for RX interrupt
//
if (intflags & CSL_CPSW_MACINVECTOR_RXPEND_MASK)
{
i = EMAC_RxServiceCheck(hEMAC);

if( i )
{
if (i == CPSW3G_ERR_MACFATAL)
emac_fatal_error++;
else
csl_errors++;
}
// write the EOI register
EMAC_rxEoiWrite(coreNum);
}
}
}
}

This thread has the highest priority because actually it runs an ISR code.

The ISR only set the trigger for the thread:

void HwInt(void *stub)
{
SEM_post(ISRSem); //Here we only trigger NDKISRTask
return;
}

The semaphore is countable so we will not lose the interrupt if it is generated before the previous one is finished.

I have been testing this fix for one day now and it works fine so far in both test and the real system.

Here is the modified ethdriver.c

I will continue testing of the fix. If everything is OK I will close the question.

Thanks a lot for your help!

Victor

0 Sudhakar Ayyasamy over 10 years ago in reply to Victor Ivanov

TI__Intellectual 1010 points

Victor,

Thank you. Please let me know if you see this issue fixed. I will try it on my end too.

Sincerely,

Sudhakar Ayyasamy

0 Sudhakar Ayyasamy over 10 years ago in reply to Sudhakar Ayyasamy

TI__Intellectual 1010 points

Wang,

Can you try the fix on your system and see if the issue still occurs? Please let me know. Thanks.

Sincerely,

Sudhakar Ayyasamy

0 old Wang over 10 years ago in reply to Sudhakar Ayyasamy

Intellectual 465 points

Hi Sudhakar and Victor

Thanks for your fix.

My system stops reponding in about 5~10 minitures without the fix (2 hping3 threads with --flood).

I have tried the fix 7455.etherdriver.c on my system.

It can works more than one night under 2 hping3 threads with --flood option.

But I have still seen one board stops responding in 5 hours,

the value DMASTATUS =0x80000000, RX_HDP=0.

I will try the fix 6254.etherdriver.c later.

0 Sudhakar Ayyasamy over 10 years ago in reply to old Wang

TI__Intellectual 1010 points

Victor,

Thank you for the file. The fix works on the test project on my set up with flooding from four windows.

Sudhakar

0 old Wang over 10 years ago in reply to Sudhakar Ayyasamy

Intellectual 465 points

Hi Victor & Sudhakar

Thank you for your fix.

My board still stops responding afer 16 hours( 6254.etherdriver.c).

and the DMASTAUS=0x80000000, RX_HDP=0.

0 Sudhakar Ayyasamy over 10 years ago in reply to old Wang

TI__Intellectual 1010 points

Victor, Wang,

We have modified the driver to disable all interrupts (not just ethernet subsystem interrupts) when we handle EMAC interrupts and this prevents the generation of any interrupt until interrupt servicing is completed. Once completed, the interrupts are restored to the state in which they were before being disabled.

Please try the attached ethdriver.c file on your side and let me know.

10/25/2013: This issue is fixed. Please follow the Wiki link posted at the end of this thread to get the final version of the driver that provides the fix. That is same as the file attached here.

Thank you.

Sincerely,

Sudhakar Ayyasamy

0 old Wang over 10 years ago in reply to Sudhakar Ayyasamy

Intellectual 465 points

Hi Sudhakar

Thanks a lot for your working.

I'll try the 5670.ethdriver.c today.

By the way, would you like to pay attention on why RX_HDP become 0 and why can't restore to a valide value?

0 Victor Ivanov over 10 years ago in reply to old Wang

Expert 1160 points

Hello Sudhakar and Wang

I haven't finished testing of 5670.ethdriver.c yet but I occasionally found another issue. I was reconnecting one of the boards during the testing and when I plugged in an Ethernet cable the data stream didn't resume. The link was established but no data were transmitted. The ISR was called regularly so the problem was in some other place.

I have reproduced this issue using all the versions of the ethdriver.c. Usually it requires from 10 to 100 connections-disconnections of the cable.

Wang, didn't your problem appeared at the reconnection moment?

Victor

0 Sudhakar Ayyasamy over 10 years ago in reply to Victor Ivanov

TI__Intellectual 1010 points

Victor and Wang,

Please try the latest 5670.ethdriver.c on your side as we believe this would fix the issue. We see that the ISR is not re-entered before running to completion with this. Please let me know after trying this.

Thanks.

Sudhakar

0 Victor Ivanov over 10 years ago in reply to Sudhakar Ayyasamy

Expert 1160 points

I'm testing it now. I just haven't finished my test. It works fine so far.

0 old Wang over 10 years ago in reply to Victor Ivanov

Intellectual 465 points

Hi Sudhakar and Victor

I've tried 5670.ethdriver.c on two boards.

One stills stop responding after 46 hours under 2 window hping.

Another board works well over 4 days without hping and then I stops the test because the other task

And I havn't found the reconnecting cable issue.

0 Sudhakar Ayyasamy over 10 years ago in reply to old Wang

TI__Intellectual 1010 points

Wang,

Does the system fail the same way as it did previously?

Thanks.

Sudhakar

0 Sudhakar Ayyasamy over 10 years ago in reply to Sudhakar Ayyasamy

TI__Intellectual 1010 points

Wang, Victor,

If you are seeing issues on 5670.ethdriver.c, we would like to make sure if it is because of the same reason: ISR re-entered before running into completion. Because we haven't seen the issue on our side. To make sure why this is happening, please use the attached ethdriver.c file. This is a slightly modified version of 5670.ethdriver.c to use a static variable to monitor if the ISR is re-entered.

Please add a breakpoint near (line 1127) the infinite while loop as below:

if ( hEMAC )
   {
       if(isrEntry)
       {
           while(1);
       }
       isrEntry++;

Please let us know if the program ever reaches the while loop when the issue occurs. This means the ISR is re-entered.

8475.ethdriver.c

10/25/2013: This issue is fixed. Please follow the Wiki link posted at the end of this thread to get the final version of the driver that provides the fix.

Thank you.

Sincerely,

Sudhakar Ayyasamy

0 old Wang over 10 years ago in reply to Sudhakar Ayyasamy

Intellectual 465 points

Hi Sudhakar

I'll try it today.

Sincerely,

Wang

0 Victor Ivanov over 10 years ago in reply to old Wang

Expert 1160 points

Hello Wang,

I still can't reproduce the issue with 5670.ethdriver.c or 6254.ethdriver.c. Could you please describe your test installation in more details. There must be something different in your test which helps to reveal the problem.

Thanks!

Victor

0 old Wang over 10 years ago in reply to Victor Ivanov

Intellectual 465 points

Hello Victor

My test environment :

5Mega pixels video capture then encode to JPEG, frame rate 15 frms/sec.

Send the encoded jpeg stream to a PC client over ethernet.

2 window hping with --flood option to board.

ping to board.

Wang

0 Victor Ivanov over 10 years ago in reply to old Wang

Expert 1160 points

Well, it is quite similar to what I'm doing in the real system. I have raw video (radar) which I send to PC (about 150-200 Mb/s) almost without processing now. Also I use a simple test which just sends an equal amount of data to PC to help TI reproduce the issue on their side.

How much data hping generates in your tests?

0 old Wang over 10 years ago in reply to Victor Ivanov

Intellectual 465 points

Hi Sudhakar and Victor,

I have test 8475.ethdriver.c in the last weekend,but my emulation lost communication with the board this time.

It seems that the dm648 works over 50 hours from my PC client.

I'll try it again today.

I open 2 window for hping, and where to see how much data hping generates?

Thanks.

Wang

0 Victor Ivanov over 10 years ago in reply to old Wang

Expert 1160 points

Hello Wang,

on Unix I use "netstat" command. It shows amount of data which is going from the certain interface of the PC and into this interface. I'm asking because I'm also trying to reproduce the issue but still can't.

Victor

0 Sudhakar Ayyasamy over 10 years ago in reply to Victor Ivanov

TI__Intellectual 1010 points

Victor, Wang,

Please let me know after using the latest driver source code (8475.ethdriver.c with error check), and if you don't see any issues for more than two days. Thanks.

Sudhakar

0 old Wang over 10 years ago in reply to Sudhakar Ayyasamy

Intellectual 465 points

Hi Sudhakar,

My test 8475.ethdriver.c has worked more than 3 days, and this time I didn't see any issue.

thanks.

Wang

0 Victor Ivanov over 10 years ago in reply to old Wang

Expert 1160 points

I also haven't seen this problem with the fixed driver.

Victor

0 Sudhakar Ayyasamy over 10 years ago in reply to Victor Ivanov

TI__Intellectual 1010 points

Victor, Wang,

Thank you. So, do you think it is time to close this thread? Thanks.

Sudhakar

0 old Wang over 10 years ago in reply to Sudhakar Ayyasamy

Intellectual 465 points

Sudhakar,

Thanks for your effort on this issue.

And thanks Victor for the long time focusing on this issue.

Wang

0 Victor Ivanov over 10 years ago in reply to old Wang

Expert 1160 points

Hello Sudhakar!

The issue is closed. Thanks a lot for your help!

Wang, thank you for participating in the testing!

Victor

0 Victor Ivanov over 10 years ago in reply to Victor Ivanov

Expert 1160 points

PS. In this thread I have mentioned problems with reconnection of the Ethernet cable. I found out that this was my mistake. There is a socket in my system for receiving UDP from the PC. The PC wasn't sending the messages so I stopped reading from this socket but left it opened, then I reconnected the board to another PC which were still sending UDP to this socket. Eventually, the Ethernet stopped working. Now I always close sockets when I don't read from them and I don't see any problems now.

Victor

0 Sudhakar Ayyasamy over 10 years ago in reply to Victor Ivanov

TI__Intellectual 1010 points

We now have a TI Wiki page that talks about this issue. The page discusses why the problem occurs, procedure followed to reproduce it with a test project and provides a fix. Here it is:

http://processors.wiki.ti.com/index.php/DM648_Fix_for_NDK_Crash_in_High_Network_Traffic

Thanks Victor and Wang for working on this!!!

Thank you.

Sincerely,

Sudhakar Ayyasamy

Processors

Processors forum

NDK stops working in case of a big bidirectional traffic.