NDK and ICMP frames

Hello,

Just following up with this.  Any advice?

Hi Brandy,

Can you try putting a break point in udp.c at this code?  Are you hitting this?

    // If there's no PCB, send an ICMP error
    if( !hSock )
    {
        if( !(pPkt->Flags & FLG_PKT_MACBCAST) && UDP_SEND_ICMP_PORTUNREACH)
            ICMPGenPacket( pIpHdr, pPkt->hIFRx, ICMP_UNREACH,
                           ICMP_UNREACH_PORT, 0 );
        udps.RcvNoPort++;
        PBM_free( pPkt );
        return;
    }

(Note you may need to rebuild the NDK libs in debug mode to try this.)

Steve

Hi Steve,

I am using CCSv5 and I rebuild the ndk using this step here. 

http://e2e.ti.com/support/embedded/bios/f/355/t/176512.aspx

Can you recommend how best to build it in debug mode?  I will keep looking through the internets but I thought if you were online already you might know off the top of your head or have a link available.

Thanks!

Hello Steve,

I have not been able to build in debug mode yet.  I did however see that the code does (obvioulsy) go into the UdpInput() function.  I also changed the default from one to zero.

#define DEF_UDP_SEND_ICMP_PORTUNREACH 0     // 1 = Send

This did not help. 

I am not sure why it says unreachable.  Is this because I don't have an application that is receiving the data directly?  In my case, the receiving application may not always be there, but I still have function.  I suspect these ICMP frames are causing me to not make my bandwidth.  I need to 1000 pkts (1500bytes each) every 500ms in UDP.  What I see is that some get about 2/3rds of the packets on wireshark.

If you don't think so, please suggest another way to make my bandwidth.

Thanks,

Brandy

Brandy,

Steve is on vacation so Im going to see if I can help out. To build NDK in debug mode go to our wiki and look at the users guide for MCSDK (building NDK section):

http://processors.wiki.ti.com/index.php/BIOS_MCSDK_2.0_User_Guide#Building_NDK

Basically you have to modify a configuration file and then build NDK on the command line using "xdc". If you get stuck let me know. Also, there is quite a bit of code there so depending on your pc it may take a little bit of time.

I looked at your protocol trace.  It looks like the packets should be able to get through but occasionally NDK is responding with port unreachable and at others times it is receiving them okay? If so, what is your data rate? Are you generating these packets from a line tester?

Thanks,

Jack

Brandy,

Im looking through the NDK code. I suspect that the message is being sent where there is no check to see if UNreachable should be sent or not. That is why the flag has no effect. It seems I was right. If you look in ipin.c and ipout.c there are several locations where we will send Unreachable and there is no check on the flag (I can follow upw ith steve on that as Im not sure yet if we should be chekcing there).  I suspect you are hitting one of these cases (although not sure why yet).

Jack

Hi Jack,

The debug version is building now.  Thanks for the link.

I am trying to send out a max of 1000 packets via UDP at close to max mtu.  What I see happening, if I put a counter in the data bytes is that some are just skipped.  For example if the total needed is 691, I am getting somewhere between 300 and 400 packets on wire shark and i can see my counter skip them.

I am wondering if the ICMP has something to do with the speed not being fast enough.  It seems reasonable to be able to send this many packets in 500 ms.  I am using the C6678 EVM (for reference).

If I slow the sending down alot with a wait loop, I see that every packet will send out a destination unreachable port message. 

I will check to see if I can figure out which of those statements in th ipin.c and ipout.c I am hitting.  It looks like those are reflecting host and net unreachable.  Perhaps I will just add the check and see if that helps.

I have also increased the PKT_NUM_FRAMEBUF to 2000 and that has increase the amount of packets I recieve but i am still not seeing all of them.  Is it reasonable to increase this even more?  I think there must be another option, that is already quite alot of memory.

Thanks for your help!  I appreciate it.

Brandy

Brandy,

Im curious to see what condition we are hitting. As to what sizes to use I think 2000 for now for the framebufs is fine and overly large. Even under stress with a smart bits blasting Ive found it doesnt need to be above six times the default value (the number escapes me at the moment but Im thinking a little over 1k) as there are other choke points.  We may need to increase the size of the socket buffers. By default they are fairly low at 8K. You can do that as follows (I know this helps with TCP. Not sure it wil help here but if the receiving task goes off for a bit then it might be needed and is a simpel thing to try).

  /* UDP Receive limit */
    rc = 8192;  <-- Change this to say 64000
    CfgAddEntry( hCfg, CFGTAG_IP, CFGITEM_IP_SOCKUDPRXLIMIT,
                 CFG_ADDMODE_UNIQUE, sizeof(uint), (uint8_t *)&rc, 0 );

Jack

Hi Jack,

I tried lots of breakpoints in many locations to confirm a few things:

1.  I am calling and using IPRxPacket() and UdpInput()

2.  In IPRxPacket, I set many breakpoints.  I never hit any regarding Unreachable circumstances.  I set them at line 288,314 and 337.  To be sure everything was working I also set and hit breakpoints prior to these conditionals but never went in them.

3.  In UdpInput, I did the same thing.  I never hit line 361 or even 354 for that matter.  It seems to always fall into this conditional:

if( IPDst == INADDR_BROADCAST || IN_MULTICAST( IPDst ) ||      (pPkt->Flags & (FLG_PKT_MACMCAST|FLG_PKT_MACBCAST)) )

and then returns.

Here is an odd thing, if I add a listener application on this port, then they seem to go away.  Maybe I misinterpreted what was meant in the wireshark.  However, I still am not able to send all my packets out without dropping some.  I changed the CFGITEM_IP_SOCKUDPRXLIMIT to 64000 as suggested and left the number of packets at 2000.  Now, I am getting more but still not all.  I drop about 150 packets.

Thanks for your help!

Brandy

Hi Jack,

I also think that the socket should be blocking to make this work.  I changed in between both blocking and non-blocking and it does not seem to change the behavoir, I still get dropped packets.

THanks,

Brandy

Brandy,

I meant ot get back to you sonner but I got tied up this morning. Yes. I dont think whether the socket is blocking or non - blocking holds the key. I see three things here we are looking at. 1) Why is unreachable being sent, 2) The socket option and 3) getting the data rate you need.

With respect to 1 How are you handling the socket? Are you opening and closing it? If the socket is not open then the port will not be bound so I could see unreachable being sent in that case. I was thinking you were always bound on the socket but that may not be the case.

With respect to 2.  I need to look at that. Ive always used a Timeout on the sockets (bever set the option for BLOCKING) and then on the recv did a wait all. Ill dig into the socket setting a bit more.

With respect to 3. Can you look in the look in the following file and tell me what you see (C:\Program Files\Texas Instruments\pdk_C6678_*\packages\ti\platform\resource_mgr.h. I need the value for NIMU_NUM_RX_DESC.

It would also help if I understood a little more of what you are trying to do.  Your data rate seems failry low, around 2000k packet a second? How are you processing that data. Are you in a task that gets n packets and then goes off and does something and then returns to get more? Or do you have a receiver task which then queues the packets to another task? Is all this running on one core or multiple cores? 

Jack

Jack

Hi Jack,

Yes, I think you summarized quite well.  Simply, I need to send an uncompressed image (approximately 720*480 sometimes more) even 500 ms to another user via UDP packets.  This equates to about 700 packets in 500ms, about 15mbps (if I did my math right).  This is basically Core 0 only job.  There is a task running on Core 0 that receives SRIO doorbells and sends a pointer to other Cores to act on the data.  Once the other cores are done, they signal back to core 0 to send out the frame (same task as before).  one frame comes in every 500ms, so Core 0 has to be done sending in 500ms.  It seems like Core 0 is done by then since on wire shark, there is time between the incomplete frames.

To be specific:

1.  Here is how I open the socket:

SOCKET mySocketCreate(int portno,struct sockaddr_in * destinfo)

{

                struct sockaddr_in           myinfo;

                struct timeval                    to;

                int                          i;

                char                                                       myBuffer[25];

                int                          opt, rc;

                SOCKET                                                                mySocket;

                mySocket = socket(AF_INET, SOCK_DGRAM, 0);

                to.tv_sec  = 0;

                to.tv_usec = 0;

                setsockopt( mySocket, SOL_SOCKET, SO_SNDTIMEO, &to, sizeof( to ) );

                setsockopt( mySocket, SOL_SOCKET, SO_RCVTIMEO, &to, sizeof( to ) );

                i = sizeof(opt);

                opt = 1;

                setsockopt( mySocket, SOL_SOCKET, SO_BLOCKING, &opt, sizeof( rc ) );

                memset(&myinfo, 0, sizeof(destinfo));

                myinfo.sin_family = AF_INET;

                myinfo.sin_port = htons(portno);

                myinfo.sin_addr.s_addr = htonl(INADDR_ANY);

                if( bind(mySocket, (PSA) &myinfo, sizeof(myinfo)) < 0)

                {

                                LOG_ERROR("Unable to bind to port %i", portno);

                                return (SOCKET)-1;

                }

                for(i = 0; i < 25; i++)

                {

                                myBuffer[i] = 0xAA;

                }

                i = sendto(mySocket, &myBuffer[0], 25, 0, (PSA)destinfo, sizeof(struct sockaddr_in));

                if(i < 0)

                {

                                LOG_ERROR("Sendto Failed. Error code %i",fdError());

                                return (SOCKET)-1;

                }

                Task_sleep(100);

                LOG_TRACE("Socket Handle %i created.", mySocket);

                return mySocket;

}

When I send the data later, I use the same sendto function call (destinfo contains a fixed IP and port #):

numBytes = sendto(hSocket[i], ptr, 1424, 0, (PSA)&destinfo, sizeof(struct sockaddr_in));

if(numBytes < 1424)

{LOG_ERROR("Sendto Failed. Return code %i Error code %i", numBytes, fdError() );

2.  I am never attempting to recv anything on these sockets.  Merely send it out to the ether where there may or may not be someone waiting for it.

3.  Here is the #defines you asked for:

#define NIMU_NUM_TX_DESC                16u /**< Maximum number of TX descriptors used by NIMU */

#define NIMU_NUM_RX_DESC                110u /**< Maximum number of RX descriptors used by NIMU */

#define NIMU_MAX_NUM_TX_CMD_DESC        2u  /**< Maximum number of TX Command descriptors used by NIMU */

#define MAX_NUM_NIMU_DESC   (NIMU_NUM_TX_DESC + NIMU_NUM_RX_DESC + NIMU_MAX_NUM_TX_CMD_DESC) /**< Maximum number of descriptors used by NIMU */

#define MAX_NUM_DESC        (MAX_NUM_NIMU_DESC)   /**< Maximum number of descriptors used by all the modules */

#define MAX_DESC_SIZE       128 /**< Maximum size of descriptor in bytes */

regarding this topic should the _INCLUDE_NIMU_CODE be defined or not?  Right now, i think it is.

Currently I am trying to see if the SO_SNDBUF option has any effect.

Thanks for your help!
Brandy

Brandy,

Thanks that is helpfull and I will look it over. Just to confirm, how are you counting the number of dropped packets? Is this based on the number of unreachables in wire shark?

The numbes for the queueing  (the defines) look good. Based on measurements in the past and your packet size we should be able to send and receive simulatenously 45, 000 + packets per second at 1024 bytes in size.  However that will consume pretty much the entire CPU. However, given that your rate is much lower I would expect you have the extra cycles to do what you need with the packets.  In looking for a reasonable explanation, Im thinking that the receive i soccasionaly not  fast enough and thus some packets are dropped due to a queueing point being full? There may be a path here that every now and then takes too long. Ive included a diagram of the tasks and paths used for the measurment above.

Jack

Jack

Hi Jack,

I was simply counting the packets I received for each frame on wireshark.  Then I added a counter to the payload and was able to verify that there were missing counter values.

I added a listening application to collect these UDP packets and the ICMP went away but I am still not receiving all the packets. 

Currently my packet size is 1424.  That shouldn't be a problem though.  It is just different than what you suggest above.

thanks,

Brandy

Brandy,

Okay. At that size we should be able to do almost line rate bidirectionally on 1Gig (around 950 Mbps). For some reason the image didnt upload so I will try that again. In the line rate tests done we used the default UDP socket size and around 700 NDK framebuffers. Much less than what you are using .

In the code you posted above that looks like it was a socket function to send?

So in counting the packets, you add up the number on wireshark that are  for your device.  The you have some counter in your code and when you send a packet back out that counter is added to the payload so you can see it on wireshark and cross reference it?

You said "I  added a listening application to collect these UDP packets and the ICMP went away but I am still not receiving all the packets. ". By listening you mean receieving? If so, have you added a counter into the receive there or was the counter added somewhere els,e possibly in the funciton where you send the data back out after it was modified?

Jack

Jack,

Let me pause your thinking for a bit.  I am starting to think it is my switch/network adapter.  Let me look into that some.  It will take me time to get a setup without a switch.  But if I change my network adapter, I get alot of improvement, so I think that the missing packets will appear with the removal of the switch.  Please hold.

Thanks,
brandy

Hi Jack,

Ok.  I have now eliminated variables.

I removed the network adapter and the switch.  What I found then is that if the DSP sends packets full speed, I will not get them all.   this is the output of my listener (reciever):

Ready to compare frame 136

last frame had 375 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 137

last frame had 691 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 139

last frame had 571 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 138

Ready to compare frame 140

last frame had 1293 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 141

last frame had 691 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 142

last frame had 462 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 144

last frame had 577 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 143

last frame had 670 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 145

last frame had 691 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 146

last frame had 691 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 147

last frame had 678 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 149

last frame had 313 packets. Expected 691.

 Waiting for next frame.

Ready to compare frame 148

last frame had 683 packets. Expected 691.

 Waiting for next frame.

If I add a short wait loop before I send each packet then I get them all

j = 0;

while (j < 5000) j++; 

I am not sure whose problem this is then.  My network card in the laptop or the DSP.  What do you think?

Thanks,
brandy

Oh, and just to bring us back on topic, I still have no idea about these ICMP frames and why they come.  I think it must be my network card because the NDK never hits the break points regarding the sending of the these frames.

Brandy,

Okay I'm a little confused as to the setup. I think I was making a few assumptions which I shouldn't have : /.  Let me ask a few more questions. This is what I think you are doing.

I recall you describing how the application was was working. However based on  he description above and the code snippet you posted (which was a send socket call with a packet using a particular pattern) I think you are measuring this. You are sending packets back with a pattern in them.  You are then reading through the receive packets and expect to get 691 of those "frames" in each receive?

Am I close here? If so, I know the answer and why slowing down the DSP send makes this work.

Jack