udps.RcvFull and task switching

PDK 1.1.1.4

NDK 2.22.3.20

NSP 1.10.2.09

Sys/Bios 6.33.6.50

CCStudio 5.4.0.00091

Milan,

you might want to take a look at the NDK User's Guide. In section 3.5.` Troubleshooting Common Problems, there's a paragraph on "UDP application drops packets on recv() calls".

UDP application drops packets on recv() calls:

• Make sure you have plenty of packet buffers available (see Section 5.3.1).
• Make sure the packet threshold for UDP is high enough to hold all UDP data received in between calls to recv() (see CFGITEM_IP_SOCKUDPRXLIMIT in the NDK Programmer’s Reference Guide).
• Verify you do not have any scheduling issues. Try running the scheduler in high priority (via NC_SystemOpen()).
• It is possible that packets are being dropped by the Ethernet device driver. Some device drivers have adjustable RX queue depths, while others do not. Refer to the source code of your Ethernet device driver for more details (device driver source code is provided in NDK Support Package for your hardware platform).

I think that the default receive buffer size is 8k, which I think might be pretty much 5 UDP packets. Can you try increasing the receive data buffers, or reducing the rate at which you are sending UDP packets to the target?

Tom,

Thanks for your suggestions.  I still have the problem with LOW or HIGH priority.  I doubled the size of the CFG_IP_SOCKUDPRXLIMIT and still have a problem dropping packets.  I have no other tasks running, so the entire DSP is just trying to receive UDP packets.

I'm using a priority of 7 on the call to CreateTask.  The documentation says not to run above that.

Milan

Did you see this FAQ page?

Tom,

Yes.  Thanks.  I've been though all of that to no avail.

I'm working at getting the execution graph and task switching coded in.  I don't have any other tasks running, so I'm investigating why the packets aren't being processed as they arrive. 

Milan

Hi Milan,

Are you using the MCSDK?  If so, which version?

Can you attach your BIOS configuration file (*.cfg )?  (if you don't want to publicly post it, you can send me a friend request and after I accept you will be able to send to me privately).

Can you please describe your network topology, too?

Milan Yagodich said:

I doubled the size of the CFG_IP_SOCKUDPRXLIMIT and still have a problem dropping packets.

How big did you make the buffer?

Milan Yagodich said:

Some times, my task runs and processes all 4 packets, but other times a fifth packet calls UdpInput and then the socket buffer is full and the packet is discarded.

Can you elaborate a bit on the data you are trying to send?  I'm not sure how the socket buffer could be filling up with only 4 packets.  Maybe the socket buffer is not very big?

Steve

Steve,

McSDK 2.1.1.4

The buffer is 16384.

I'm send 1400 bytes of data; a 16 bit counter followed by 16 bits ints [ 0,1,2,3....]

As I said in the original post, after I had problems, I compiled in the source for the ndk into my project.  I set a breakpoint at the buffer full point and it hits.  I further traced through the code and found that my task is not running when a packet gets processed by the ndk stack.  My processing task is at priority 7.  I have this problem with NDK Interrupt mode, High priority and Low priority and polling mode.

In all cases, this is the only thing that is running.  The only background task should be the TI idle.  But I do find that I drop more packets, when I run in polled mode.  I even went as far as putting a 10 Mb switch between the two computers and still drop packets.  That cuts the inter-arrival time to around 1 msec.

Any insights appreciated.

Milan

/*
 * helloWorld.cfg
 *
 * Memory Map and Program intiializations for the helloWorld example Utility
 *
 * Copyright (C) 2010-2011 Texas Instruments Incorporated - http://www.ti.com/
 *  
 *  Redistribution and use in source and binary forms, with or without
 *  modification, are permitted provided that the following conditions
 *  are met:
 *
 *    Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 *
 *    Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the   
 *    distribution.
 *
 *    Neither the name of Texas Instruments Incorporated nor the names of
 *    its contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
 *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 *
*/

/*
 *   @file  helloWorld.cfg
 *
 *   @brief   
 *      Memory Map and Program intiializations for the hello world NDK unit test.
 *
 */
 
/********************************************************************************************************************
*  Specify all needed RTSC MOudles and ocnfigure them.                                                                *
********************************************************************************************************************/

var Memory  =   xdc.useModule('xdc.runtime.Memory');

var BIOS    =   xdc.useModule('ti.sysbios.BIOS');

var Task    =   xdc.useModule('ti.sysbios.knl.Task');

var HeapBuf =   xdc.useModule('ti.sysbios.heaps.HeapBuf');

var Log     =   xdc.useModule('xdc.runtime.Log');

var Mailbox =   xdc.useModule('ti.sysbios.knl.Mailbox');
/*
** Allow storing of task names. By default if you name a task with a friendly display name it will not be saved
** to conserve RAM. This must be set to true to allow it. We use friendly names on the Task List display.
*/
//Defaults.common$.namedInstance = true;
Task.common$.namedInstance = true;

var Clock   =   xdc.useModule ('ti.sysbios.knl.Clock');

/*
** Interface with IPC. Depending on the version of BIOS you are using the
** module name may have changed.
*/
/* Use this for pre BIOS 6.30 */
/* var Sem        =      xdc.useModule ('ti.sysbios.ipc.Semaphore'); */

/* Use this for BIOS 6.30 plus to get the IPC module */
var Sem    = xdc.useModule ('ti.sysbios.knl.Semaphore');

var Hwi    = xdc.useModule ('ti.sysbios.hal.Hwi');

var Ecm = xdc.useModule ('ti.sysbios.family.c64p.EventCombiner');

/*
** Configure this to turn on the CPU Load Module for BIOS.
**
*/
/*
var Load       =   xdc.useModule('ti.sysbios.utils.Load');
Load.common$.diags_USER4 = Diags.ALWAYS_ON;
*/

var Diags       = xdc.useModule('xdc.runtime.Diags');


/* Load the CSL package */
var Csl                         =     xdc.useModule('ti.csl.Settings');

/* Load the CPPI package */
var Cppi                        =   xdc.loadPackage('ti.drv.cppi');     

/* Load the QMSS package */
var Qmss                        =   xdc.loadPackage('ti.drv.qmss');

/* Load the PA package */
var Pa                             =     xdc.useModule('ti.drv.pa.Settings');

/* Load the Platform/NDK Transport packages */
var PlatformLib  = xdc.loadPackage('ti.platform.evmc6670l');
var NdkTransport = xdc.loadPackage('ti.transport.ndk');

/*
** Sets up the exception log so you can read it with ROV in CCS
*/
var LoggerBuf = xdc.useModule('xdc.runtime.LoggerBuf');
var Exc = xdc.useModule('ti.sysbios.family.c64p.Exception');
Exc.common$.logger = LoggerBuf.create();
Exc.enablePrint = true; /* prints exception details to the CCS console */

/*
**  Give the Load module it's own LoggerBuf to make sure the
**  events are not overwritten.
*/
/* var loggerBufParams = new LoggerBuf.Params();
loggerBufParams.exitFlush = true;
loggerBufParams.numEntries = 64;
Load.common$.logger = LoggerBuf.create(loggerBufParams);
*/

/*
** Use this load to configure NDK 2.2 and above using RTSC. In previous versions of
** the NDK RTSC configuration was not supported and you should comment this out.
*/
var Global       = xdc.useModule('ti.ndk.config.Global');

/*
** This allows the heart beat (poll function) to be created but does not generate the stack threads
**
** Look in the cdoc (help files) to see what CfgAddEntry items can be configured. We tell it NOT
** to create any stack threads (services) as we configure those ourselves in our Main Task
** thread hpdspuaStart.
*/  
Global.enableCodeGeneration = false;


/* Define a variable to set the MAR mode for MSMCSRAM as all cacheable */
var Cache       =   xdc.useModule('ti.sysbios.family.c66.Cache');
//Cache.MAR224_255 = 0x0000000f;

var Startup     =   xdc.useModule('xdc.runtime.Startup');

var System      =   xdc.useModule('xdc.runtime.System');



/*
*/
var HeapMem = xdc.useModule('ti.sysbios.heaps.HeapMem');
var heapMemParams = new HeapMem.Params();
heapMemParams.size = 0x300000;
heapMemParams.sectionName = "systemHeap";
Program.global.heap0 = HeapMem.create(heapMemParams);

/* This is the default memory heap. */
Memory.defaultHeapInstance  =   Program.global.heap0;
Program.sectMap["sharedL2"] = "DDR3";
Program.sectMap["systemHeap"] = "DDR3";
Program.sectMap[".sysmem"]  = "DDR3";
Program.sectMap[".args"]    = "DDR3";
Program.sectMap[".cio"]     = "DDR3";
Program.sectMap[".far"]     =     "DDR3";
Program.sectMap[".rodata"]     =     "DDR3";
Program.sectMap[".neardata"]     =     "DDR3";
Program.sectMap[".cppi"]     =     "DDR3";
Program.sectMap[".init_array"]     =     "DDR3";
Program.sectMap[".qmss"]     =     "DDR3";
Program.sectMap[".cinit"]     =     "DDR3";
Program.sectMap[".bss"]        =    "DDR3";
Program.sectMap[".const"]    =    "DDR3";
Program.sectMap[".text"]    =    "DDR3";
Program.sectMap[".code"]    =    "DDR3";
Program.sectMap[".switch"]    =    "DDR3";
Program.sectMap[".data"]    =    "DDR3";
Program.sectMap[".fardata"] =     "DDR3";
Program.sectMap[".args"]     =     "DDR3";
Program.sectMap[".cio"]     =     "DDR3";
Program.sectMap[".vecs"]     =     "DDR3";
Program.sectMap["platform_lib"]     =     "DDR3";
Program.sectMap[".far:taskStackSection"] = "L2SRAM";
Program.sectMap[".stack"]    =    "L2SRAM";
Program.sectMap[".nimu_eth_ll2"] = "L2SRAM";
Program.sectMap[".resmgr_memregion"] = {loadSegment: "L2SRAM", loadAlign:128};    /* QMSS descriptors region     */
Program.sectMap[".resmgr_handles"] = {loadSegment: "L2SRAM", loadAlign:16};    /* CPPI/QMSS/PA Handles            */
Program.sectMap[".resmgr_pa"]    = {loadSegment: "L2SRAM", loadAlign:8};        /* PA Memory                    */
Program.sectMap[".far:IMAGEDATA"] = {loadSegment: "L2SRAM", loadAlign: 8};
Program.sectMap[".far:NDK_OBJMEM"] = {loadSegment: "L2SRAM", loadAlign: 8};
Program.sectMap[".far:NDK_PACKETMEM"] = {loadSegment: "L2SRAM", loadAlign: 128};

/* Required if using System_printf to output on the console */
SysStd                  =   xdc.useModule('xdc.runtime.SysStd');
System.SupportProxy     =   SysStd;

/********************************************************************************************************************
* Define hooks and static tasks  that will always be running.                                                       *
 ********************************************************************************************************************/

/*
** Register an EVM Init handler with BIOS. This will initialize the hardware. BIOS calls before it starts.
**
** If yuo are debugging with CCS, then this function will execute as CCS loads it if the option in your
** Target Configuraiton file (.ccxml) has the option set to execute all code before Main. That is the
** default.
*/
Startup.lastFxns.$add('&EVM_init');

/*
** Create the stack Thread Task for our application.
*/
var tskNdkStackTest          =   Task.create("&StackTest");
tskNdkStackTest.stackSize      =     0x1400;
tskNdkStackTest.priority    =     0x7;

//var tskTest          =   Task.create("&Test");
//tskTest.stackSize      =     0x14000;
//tskTest.priority    =     0x6;

/*
** Create a Periodic task to handle all NDK polling functions.
** If you are using RTSC configuration with NDK 2.2 and above, this is done by default and
** you do not need to do this.
*/
/*var prdNdkClkParams         =   new Clock.Params ();
prdNdkClkParams.period      =   0x64;   
prdNdkClkParams.startFlag   =   true;
Program.global.clockInst1   =   Clock.create("&llTimerTick", 5, prdNdkClkParams);
*/

/*
** If you are using RTSC configuration with NDK 2.2 and above, this is done by default, else
** register hooks so that the stack can track all Task creation
Task.common$.namedInstance  =   true;
Task.addHookSet ({ registerFxn: '&NDK_hookInit', createFxn: '&NDK_hookCreate', });

/* Enable BIOS Task Scheduler */
BIOS.taskEnabled            =   true;

/*
 * Enable Event Groups here and registering of ISR for specific GEM INTC is done
 * using EventCombiner_dispatchPlug() and Hwi_eventMap() APIs
 */

Ecm.eventGroupHwiNum[0] = 7;
Ecm.eventGroupHwiNum[1] = 8;
Ecm.eventGroupHwiNum[2] = 9;
Ecm.eventGroupHwiNum[3] = 10;

Global.ndkTickPeriod = 1;

// Create the mailboxes for the sockets
//for (i = 0; i < 32; i++)
//{
//    Program.global['mbx_task' + i] = Mailbox.create(4, 25);
//}

Global.IPv6 = false;

Hi Milan,

Milan Yagodich said:

The buffer is 16384.

Can you try increasing the buffer to a really large number?  E.g.:

        Int receiveBufSize = 49,152;
        CfgAddEntry(hCfg, CFGTAG_IP, CFGITEM_IP_SOCKUDPRXLIMIT,
            CFG_ADDMODE_UNIQUE, sizeof(uint), (UINT8 *)&receiveBufSize, 0);

Can you view the global structure 'udps' in the CCS 'Expressions' window?  Can you check the values of udps.RcvTotal and udps.RcvFull when you see the problem happen?

In UdpInput(), there is a check to see if the socket buffer is full just before the call to SockNotify.  I think you should be hitting this if the receive buffer is indeed full.  Are you falling into this if block?

    /* If there's no space on the receiver queue, then discard the packet */
    /* with an ICMP error */
    if( SBGetSpace( hSBRx ) < (INT32)UDPLen )
    {
        if( !(pPkt->Flags & FLG_PKT_MACBCAST) )
            ICMPGenPacket( pIpHdr, pPkt->hIFRx,
                           ICMP_SOURCEQUENCH, 0, 0 );
        udps.RcvFull++;
        PBM_free( pPkt );
        return;
    }

Steve

Steve,

I increased the SOCKUDPRXLIMIT to 49152 and see no difference.

The first time I hit the breakpoint in the code you pointed out, udps.RcvFull=0 and udps.RcvTotal is 198.

Milan

Are you able to see what the values of UDPLen and SBGetSpace() are at this point?

Are you already re-compiling udp.c as part of your project? To see SBGetSpace()'s return value, you may need to modify the code from:

if( SBGetSpace( hSBRx ) < (INT32)UDPLen )

to be something like:

INT32 x = SBGetSpace(hSBRx);

if(x  < (INT32)UDPLen) ...

Do you have a Wireshark capture of this issue?

Steve

OK.  I'll get you a pcap tomorrow.

I think something is just grossly misconfigured at a  high level.  It seems like a task priority problem  or polling vs interrupt issue, even though they are both correctly configured.  I'm not using any of the network accelerators, just the stock helloWorld program modified to echo back udp packets.

I've run this on two different dev boards and two different host computers, so I think I can safely rule out a hardware problem.

I'll noodle in your hacks and  send you an update tomorrow.

Thanks

Milan

Steve,

The return from SBGetSpace() is 1147.

How do I get the wireshark data to you?

Milan

Milan,

You should be able to just attach the Wireshark capture file to this thread.

If you have problems with that, then just send me a friend request on the forum.  Once we're friends, it allows us to email files to one another.

Steve

7875.udp_rcvfull.zip

Steve,

We made two captures showing about identical behavior.  A test app is sending udp packets to the ti evm board, and the evm board simply echoes them back.

The data is piped through a 10 Mb hub that limits the transmission speed to about a packet per msec.

Wireshark is running on a different laptop attached to the hub.

Thanks for the help

Milan

Milan,

Thanks for sending those captures, as well as the nice detailed description of the apps and your network topology. 

This is definitely helping the picture become clearer and I have a guess as to what's going on - the DSP is simply dropping UDP packets because it is being sent more than it can handle.

...

Looking at the capture file "udp_echo_1.pcapng", I see that the DSP (192.168.1.2) sends the PC (192.168.1.3) an ICMP source quench message.  This happens at packet 169.

If you look at that ICMP packet, you will see that it contains the original offending packet (the one sent by the PC that caused the problem for the DSP) encapsulated within it.  You can see this in Wireshark by expanding the ICMP part of packet #169, where you should see the IP and UDP data of that offending packet.

In the offending packet's IP header, it shows that it has an ID number of 26913.  Using this IP ID and looking back through the packet caputure, we can see that the packet with IP ID number 26913 corresponds to packet #81 in Wireshark.

Now, looking further back in the trace, it looks like the transfer of UDP packets from PC to DSP starts at packet number #34, with each packet containing 1499 bytes.

Taking out header sizes (1499 - 14 (Eth header) - 20 (IP header) - 8 (UDP header)) gives us 1457 bytes of actual data per packet.

So, from packet 34 to 81, that gives us 48 packets (inclusive), which is a total of 69,936 bytes sent to the DSP.

Of course, the DSP is processing this data as it's being received, but it looks like the rate of reception is greater than the rate of consumption, so eventually (at packet #81), your 14 Kbyte UDP receive buffer can't hold any more, and so it sends the ICMP message.

This is expected and normal for the stack to behave this way.  If data is coming in too fast, it sends a response to the sending host to slow the send rate (that's the ICMP source quench, see this web page for some more info).

It may also help to check for the source quench in your PC app, throttling down the data rate when you see it (the above link also discusses that).

Hope this helps.

Steve

Steve,

I follow your logic, but are you saying the 1 GHz processor cannot keep up with a 10Mbps link?  At an arrival rate of about 1 msec between packets, I would think the DSP isn't doing much.

If I'm not using the packet accelerator, is the DSP being interrupted on every byte across the network?   I haven't dug into the ethernet code yet to see how much the DSP is actually processing.  Does the DSP actually have to do the level 2 framing or does that happen in the PHY silicon?

I just cannot come to grips with this superfast DSP not being able to keep up with such a slow data rate.

Thanks for your help

Milan

Milan Yagodich said:

I follow your logic, but are you saying the 1 GHz processor cannot keep up with a 10Mbps link?  At an arrival rate of about 1 msec between packets, I would think the DSP isn't doing much.

Good point.  1GHz should be able to handle it.  Can you verify that you're running at 1GHz?  You should be able to see this in ROV (tools -> RTOS Object Viewer (ROV)), under the BIOS module.  Or by calling the getCpuFreq().  E.g.:

Types_FreqHz freq;

BIOS_getCpuFreq(&freq);

/* Then check freq.lo/.hi) */

Can you also check that the Clock module is running at the right rate?  This can also be done using the ROV tool.  The default value for 1 tick is 1 ms.  So, if you halt the program, observe the Clock module's "ticks" field.  If you run for 10 seconds of wall time, then the "ticks" value should increase by 10,000.  This will verify that the Clock is running at the correct speed.

Milan Yagodich said:

If I'm not using the packet accelerator, is the DSP being interrupted on every byte across the network?   I haven't dug into the ethernet code yet to see how much the DSP is actually processing.  Does the DSP actually have to do the level 2 framing or does that happen in the PHY silicon?

I believe that the PHY should be handling the frames, not the DSP. I'm not very familiar with the driver for the 6678, so I can't comment on how many interrupts may occur or how the packet accelorator might affect things.  I'll ask and get back to you on this.

Steve

Hi Steve,

My name is Raj and I'm working with Milan to help resolve this issue.

I just tried your tips and everything appears to be at 1GHz. The BIOS_getCpuFreq call resulted in a Lo value of 1000000000 and a Hi value of 0. Manually starting and halting the program over 10s (using a stopwatch as a guide) did move the tick count in the Clock module appropriately (11500 ticks).

Also just to be clear - we are running on a 6670 eval board.

Raj

The 6670 CPU is 1GHz and the Ethernet port is 1G. Many UDP throughput test can reach ~700 Mbps, so there should not any issue for supporting only 10 Mbps link. The NDK uses NIMU driver for the C6670, the NIMU driver only uses PA for L2 MAC address matching, L3/L4 is handled by NDK stack. When an Ethernet packets is received, an rxISR is invoked (EmacRxPktISR in C:\ti\pdk_C6670_1_1_2_6\packages\ti\transport\ndk\nimu\src\nimu_eth.c). Will this ISR hit whenever you received a UDP packet?

- Eric  

Yes.  EmacRxPktISR is hit.

I have found that if I make the UDP buffer 32*8192 ~~ 260 kBytes, then I have good success.  I'm suspecting that with the NDK stack only being called every 100 msec instead of for each packet, a lot of storage is needed.

I looked at changing the timer tick so something other that 100 msec, but llTimerTick (ndk/hal/timer_bios/lltimer.c) has it hardcoded to 100 msec.  I suspect this will screw up any TCP timers.

Is there a way to invoke the stack on the receipt of every packet in addition to the timer.  I understand the need for the timer, but being called every 100msec introduces a lot of latency ( and buffering requirements).

Milan

Hi Steve,

We are still scratching our heads as to why we are dropping packets. In our latest test, I'm loading the DSP (setting up the network, socket, etc... and waiting on recvfrom() ). After that, I configure a laptop to send UDP packets at a rate of 1 packet every 2 seconds for a total of 12 packets. After receiving a packet, I read the buffer and issue another recvfrom(). I've noticed the following behavior under different scenarios

1) If I let the the code run (without setting any breakpoints) I will get about 1 - 3 packets and then I wont get anything. After the test timeout period is reached (25 secs), the test is reset and is again waiting on recvfrom(). I can restart the packet transmission again from the laptop and get the same behavior repeatedly. I can observe EmacRxPktISR being hit after all 12 packets.

2) I set a breakpoint after the first packet is received. I wait for all packets to be sent by the laptop - effectively having the 11 remaining packets buffered up in the DSP, the test completes properly and all packets are received. The EmacRxPktISR is hit 5 times.

What could be causing the DSP to miss processing the data when it's clearly hitting the ISR? The recvfrom does have the MSG_WAITALL option set so it should block and wait for data to arrive right?

Hi Raj,

What do you see in the udps statistics struct between these 2 scenarios?  (for example, you can see this struct in the screen shot your colleague posted earlier in this thread).

Are you checking the value returned by recvfrom()?  What does it return for each scenario?

Rajan Patel said:

In our latest test, I'm loading the DSP (setting up the network, socket, etc... and waiting on recvfrom() )

It may be helpful to post the code for this part.

Steve

Raj,

Another thing, there should be some EMAC statistics you can check on the 6670.  I see from this thread that the stats are at the following locations:

"You can dump the CPSW statistics at address 0x2090b00 and 0x2090c00"

It may be helpful to compare numbers between these two scenarions.

Steve

Hi Steve,

Thanks for the suggestions. I've attached a file with the results of udps and the memory dump of the CPSW statistics for the two scenarios.There are differences but I'm not sure what they mean.

In all cases recvfrom() always returns the number of bytes that were read for the message. There is never any error returned from it. In the case where I'm not getting the data...the process is blocked at recvfrom() and never wakes up.

The code I'm working with is not as straight forward to follow - there is a lot of message passing between tasks going on. We were hoping to have this issue resolved by the time we got to this point. I'll simplify this test case and post that code (hopefully later today).

Raj

Here is the attachment with the test results - it didn't work during the last post.

I was able to simply the code. In the original code, the timeout on the socket was 0. I was able to determine my data timeout while pending on a mailbox when I should have gotten data. In this simplified example, I'm explicitly setting a timeout period of 25 secs. 

Using the simplified code for scenario 1 (no breakpoints), I do see a recvfrom error of -1. Calling FdError on it shows and error of 35 - EWOULDBLOCK.

I've attached the code.

Raj,

EWOULDBLOCK just means that your 25 second timeout hit.

Thanks for attaching that.  The udps values are helpful.  However, I can't read the register dump ... maybe I told you to go about that the wrong way.  Can you instead try opening the register view in CCS?

There you will find core registers, as well as peripheral registers.  Can you look for the CPSW there?  Or maybe there's an EMAC register?

These register views should also be labeled once you expand them, so you will know what the numbers are.

Also thanks for making a simplified test case.  I'm wondering what other Tasks are running in your system with that simple test case?  Is ET_Udp_v4 the only Task running?  O are you running that test case (ET_Udp_v4) as part of your main app (with the Mailboxes, etc)?

If so, can you try running your simple test case in a simpler app (such as the NDK helloWorld example that's in your MCSDK product)?

Lastly, I was digging around a bit and came across an old post that I was helping a customer with earlier this year.  Their problem was similar to yours, and was on a different h/w platform, but the issue they were seeing (UDP packets weren't going out onto the wire) turned out to be due to a DMA configuration problem.  Not sure if this applies here but wanted to point it out just in case:

http://e2e.ti.com/support/embedded/bios/f/355/t/244852.aspx?pi199400=3

Steve

Steve,

Our code is built from the NDK helloWorld example. In this simplified example, I removed all other tasks/mailboxes etc. The ET_Udp_v4 function is the only other task running (aside from the NDK ones). Here is the task list from the ROV after receiving the first message in the test:

,0x00865540,,8,Blocked,StackTest,0x00000000,0x00000000,5120,0x85eb00
,0x00865588,,5,Terminated,ti_ndk_config_Global_stackThread,0x00000000,0x00000000,8192,0x85ff00
,0x008655d0,ti.sysbios.knl.Task.IdleTask,0,Ready,ti_sysbios_knl_Idle_loop__E,0x00000000,0x00000000,2048,0x861f00
,0x81cd4d78,udp_test,5,Running,ET_Udp_v4_simple,0x00000000,0x00000000,8192,0x81cd4dc0

I have tried setting the NC_SystemOpen with both low and high priorities, poll vs interrupt and they all seem to yield the same behavior.

I see the register view but can't find any of the registers you mentioned. Do you know specifically where they are located?

Thanks for pointing out that post - I will dig around and see if that applies to the helloWorld example.

Raj

Raj,

I think at this point I will need to try to reproduce the problem on my side.  Unfortunately I didn't have time to do it today, but will pick this up next week.

If you could share your PC side app that you use to test, and even your simple project test case it would be very helpful.

Steve

Hi Steve,

I've attached a zip file containing both our source and the test application that generates the packets. I wrote up a quick readme file that describes how to use the test app and what is included in the zip. If you have any questions / issues don't hesitate to ask.

Raj

Hi Steve,

Just wondering if there have been any updates on the situation.

Thanks,
Raj

Hi Raj,

I apologize for taking so long to get back to you.

I was able to download your updated helloWorld app as well as the Ostinato packet builder app.

Since I don't have a 6670 EVM on hand, I did a quick port of your app to the ARM9 on the evmOMAPL138 board.

With this board, I was not able to reproduce the issue that you are seeing.  This could mean that the problem is happening in the driver, or it could be due to a few minor differences between my ARM9 app and yours.  Or, it could mean I'm doing something incorrectly.

Here's the summary of what I did and some of the differences in my app.  Does this sound right to you or did I miss anything in these steps?

1. Start with the NDK 'client' example for the evmL138 ARM9
2. Configure the app for my private test network: change to use static IP address 10.90.90.3, subnet mask = 255.255.255.0
3. Test PC has IP address 10.90.90.5
4. I'm running the NDK in low priority in interrupt mode:
   1.     rc = NC_SystemOpen(NC_PRIORITY_LOW, NC_OPMODE_INTERRUPT);
5. NDK stack thread runs at priority 5
6. I created the ET_Udp_v4_simple Task in the NDK's service report function (the function that is called whenever an IP address is removed or added to the system). Also priority 5.
7. I am using the default UDP receive buffer size (8K).
8. Fixed a couple of syntax errors in the udpTest.c file (also changed it to a C file instead of C++ file)
9. Updated the Ostinato settings to match my EVM and test PC's MAC and IP addresses

Observations:

• When I run the app, if I do nothing I see the error 35 (EWOULDBLOCK) after 25 seconds.  This is expected due to the recvfrom hitting the 25 second timeout.
• When I run the app and quickly run the Ostinato, I see it successfully send 12 UDP packets
• I also see this in Wireshark (attached, however, note that b/c of port mirroring on the switch I'm using, each UDP packet is duplicated).
• I was able to hit "play" in Ostinato several times and sent 48 packets.
• I only saw the error once I stopped sending packets via Onstinato (due to the timeout).

Steve

2664.updTest_OMAPL138_ARM9.zip

3414.output_OMAPL138_ARM9.txt

Raj,

Thank you for attaching your project in earlier posts. We are going to reproduce your issue on C6670 EVM first.

best regards,

David Zhou

Raj,

I am working on your issue. I have 6670 EVM (192.168.1.2) and my PC (192.168.1.4) in a static network. I was able to use the Ostinato to send the packets at 2 sec interval.

When I ran the C:\ti\mcsdk_2_01_02_06\examples\ndk\client\evmc6670l client program on EVM, I verfied that:

1) SW statistics @0x2090b00/2090c00 regions are correct

2) EmacRxPktIsr was hit 12 times

3) ips and udps stats showed 12 packets received.

When I ran your pre-build helloworld_evm6670.out, I can only verify point 1) above, but less 12 packets are received. Can you clarify if this is your issue? If yes, I want to build your binary from the source code, please clarify:

The CCS project showed PDK 1.1.2.6, ndk 2.21.2.43, NSP 1.10.2.9, CCS 5.1.0.1. They are different from your earlier report of issue on PDK 2.22.03.20 and PDK 1.1.1.4? What is the actual package version to show the problem? Also, you ccs project includes NSP and UIA, they are not needed.

Regards, Eric

Hi Eric,

On our code, we observed both 1) and 2) but 3) was not occuring. This post describes what we saw:
http://e2e.ti.com/support/embedded/bios/f/355/p/288260/1029558.aspx#1029558

The package components for this build are:
PDK 1.1.2.6, ndk 2.21.2.43, NSP 1.10.2.9, CCS 5.4.0.00091

We had other versions of packages before, but tried to stay with all the components that were in the MCSDK for consistency to help narrow down the problem.

Raj

Raj,

I was able to re-produce your issue with 6670 EVM and hello world example, something I found the code work with smaller packets (say 503 bytes instead of 1503 bytes), I am investigating it.

Regards, Eric 

Hi Iding,

lding said:

I found the code work with smaller packets (say 503 bytes instead of 1503 bytes)

I just wanted to double check on this ... can you clarify on your test of size 1503?  Is 1503 the payload size?  If so, that's expected to fail due to the max payload size of 1500 on an Ethernet frame.

Steve

The payload is 1485 bytes long. The size of the frame with payload, FCS, etc is 1503 bytes. You can see it in the stream properties in Ostinato, and can also verify through wireshark capture.

Raj

Steve,

The wire capture is 1499 bytes. IP length is 1485, UDP length is 1465, data payload is 1457 bytes. There is 4 bytes FCS, so this is where 1503 came from. Attached is a packet capture.

Regards, Eric

http://e2e.ti.com/cfs-file.ashx/__key/communityserver-discussions-components-files/355/8322.udp.pcapng

Hi guys,

Just checking in to see the status of this issue. Are we certain that the problem is at some software level or is it related to the hardware on the EVM board?

Raj

Our developer is analyzing the issue and will keep you updated, I thought the issue is SW related..

Regards, Eric

Hi Eric,

Just checking in. Any progress on this issue?