NDK: Dropped Raw Ethernet frames with SOCK_RAWETH socket

Corey,

Which version of the NDK and BIOS are you using?

What is the CPU load of the target?

Why are you using recvnc, if you are just going to copy the packet. It seems like you should just be using the standard recv.

Todd

NDK version: 2.20.03.24

BIOS version:  5.41.07.24

The CPU is virtually idle while this loop is executing.

I am aware that recvnc() combined with a copy is more-or-less equivalent to recv().

The reason I am doing this is that I am under the impression that Raw sockets do not support recv(). Is this true?

To make sure we are clear, my socket is created with the following code line:

hRawSocket = socket(AF_RAWETH, SOCK_RAWETH, 0x801);

In any case, why would this all lead to packets being dropped? I assume there is an underlying buffering system (i.e. "socket buffer") that will stow away received packets until the next time you call recvnc().

Thanks

Corey

Sorry. I missed the SOCK_RAWETH in the original post.

What does the raw stats look like? Put "raweths" in your watch window. Hopefully this will shed some light.

Todd

OK got the stats.

Procedure A: Boot the DSP and send it 5 packets in a row. The receive stats are:

RcvTotal 2
RcvDrops 0

Note: My application also processed 2 out of the 5 packets.

Procedure B. Boot the DSP and remotely bring down and up, again, ehe link (in linux, use ifconfig down/up).  The receive stats are:

RcvTotal 5

RcvDrops 5

This suggests:

1. The driver does not have any idea that in Procedure A, above, it lost 3 packets.
2. The RcvDrops field counts packets received by the ethernet driver but explicitly ignored (i.e. dropped), and RcvTotal counts all packets known to the driver, whether dropped or not.

Why does the driver miss these packets? Perhaps the driver has only two buffers being ping-pong'd for DMA.  Can someone take a look at the Ethernet driver and determine the length of the buffer chain set up in the buffer descriptors?

Thanks

Corey

Corey,

The "raweths" statistics are maintained in the NDK stack. 

Have you plugged a statistics function into the driver? Look in the csl_emac.c file for a description.

ecfg.pfcbStatistics            = &StatisticsUpdate;

There is a dummy one in the ethdriver.c called StatisticsUpdate.The statistics are in the

EMAC_Statistics Stats;

field.

You can call the EMAC_getStatistics() directly also, but the stats are cleared after every Rx and Tx in the driver.

I'll find a evm6748  and reproduce the problem here.

Todd

I have not plugged a statistics function into the driver as indicated by the fact that grep'ing my project files for "pfcbStatistics" returns only the line you mentioned:

ecfg.pfcbStatistics         = &StatisticsUpdate;

That function is defined as:

static void StatisticsUpdate( Handle hApplication )
{
    if( (Uint32)hApplication != 0x12345678 )
    {
        return;
    }
}

Looking at the numbers recorded in my previous post for "RcvTotal" and "RcvDrops", it is apparent that the statistics are NOT cleared after every Rx and Tx. Are you sure we are looking at the same version of NDK. Remember, my version is: 2.20.03.24.

I appreciate that you will be trying to reproduce the problem with the c6748.  When trying to reproduce the problem, it is important that the remote send multiple packets in one atomic operation so that there will be know pause between the transmission of packets, since (as mentioned in my first post) the problem does NOT occur when a small pause occurs between the transmission of packets.  Hence, it appears that the problem is a timing-driven one. Let me know if you are having trouble reproducing the problem and I will try to help.

Thanks

Corey

Corey,

The pfcbStatistics field is in the driver code, not the NDK. Which NSP are you using (NSP supply the driver, NDK is the generic stack)?

In the NSP I'm looking at, 1.10.00.03, the pfcbStatistics function is called after each transmit and receive with the following comment

Uint32 EMAC_TxServiceCheck(Handle hEMAC)
{
  ...
        /* Read the stats and reset to zero         */
        /* This is necessary to clear the interrupt */
        emacUpdateStats(pd);

        /* Tell the application */
        (*localDev.Config.pfcbStatistics)(pd->hApplication);

Todd

I am using NSP 1_00_00_09, but it looks like we have modified code in the "ethdriver.c". I am not sure why it is modified (by the previous developer of the project) but I suspect it was done to allow promiscuous mode or possibly to allow a custom protocol field in the MAC header.

Here is the diff from the original version of "ethdriver.c" (from 1.00.00.09) to my version:

36c36
< #define     PKT_MAX                     64
---
> #define     PKT_MAX                     190
405c405
<
---
> volatile Int32 pktSize = 0;
422a423
>                 pktSize = sizeof(hPkt);
439c440,441
<                                  EMAC_CONFIG_MODEFLG_RMII;
---
>                                  EMAC_CONFIG_MODEFLG_RMII |
>                                  EMAC_CONFIG_MODEFLG_PASSALL;

Should this modification have a bad effect?

Is there any other information you need in order to help on this issue?

Thanks

Corey

Thanks. Nothing jumps out. Do you have a simple PC or Linux app that sends the data? Can you attach it (along with the source)? I want to limit the variables when I test it out here (I found a board).

Todd

Attached is the source for sending raw packets to the DSP.  Unzip the the archive and do:

# cd ethtest

# make clean

# make

To send packets, e.g. 5 packets and assuming eth1, do:

./eth-send   eth1    5

There are a couple things to keep in mind that could prevent this from working with your c6748 receiving code.

1. The code assumes the receiving device is in promiscuous mode, i.e. will accept packets with a non-matching destination MAC address.

If this seems to be the problem, run instead, "eth-send-mac", as follows, and substitute the MAC address that the c6748 is set as:

# ./eth-send-mac eth1 1 01:23:45:67:89:AB

2. The code fills in a custom protocol field in the MAC header.  If this is the problem, look at line 21 of ethlib.c:

#define CUSTOM_PROTOCOL 0x0801

You should modify this to be a protocol that the c6748's raw ethernet stack will accept.

For example, 0x0800 will identify the packet as an ordinary IP packet.

Recompile and try again.

Let me know if this works.

Thanks

Corey

Todd,

I don't get it. It told me that it attached the code. It was a .zip file. What happened. Do you have an email address to which I can send the code?

Corey

I understand that this thread has moved outside the forum, so I will mark it as closed.

^{We have moved to email correspondence, but for the benefit of the community, we will post the results when a solution is found. This thread may turn out to be quite useful to developers at that time.}

Hi Corey,

This issue was recently brought to my attention.  I am trying to reproduce the problem locally.  I received the updated driver files mentioned in the post, as well as some Linux side apps that will communicate with the target board.

However, I believe the NDK you are using was updated for promiscuous mode (I suspect I was working with the previous developer of your project, Mitch, in order to get promiscuous mode working).

Do you know if this is true?  The reason I'm inquiring is because I get the following run time error when building against the file "preprocessor.c":

NIMUIoctl NIMU_ETH_PROMISC failed, retcode = -1

The standard version of nimu.c doesn't have a case for NIMU_ETH_PROMISC.  I'm guessing that this was added in order to get promiscuous mode working.  Or maybe I'm missing something else?

Thanks,

Steve

Indeed, what we have is a result of your previous work helping Mitch. It is the same project that I am working on.

Since I inherited this project in the company without knowing all the details, I don't know specifically what was done to accomplish the PROMISC functionality.

The project seems to be linked with version 2.20.03.24 of the NDK. For my sake, do you know in what way the NDK has been updated?

Is the update manifested in the modified versions of nimu.c, nimu_eth.c, nimu_eth.h found in my project?

If you think modifications have been applied to the NDK project itself, I am happy to send you the source of the NDK I'm linking against.

Thanks

Corey

Corey,

Yes, there were most certainly changes made to the NDK and the driver as well, I believe.

I hunted around and found this forum post that has the "final answer" of all the back and forth exchanges to that it took to get everything into promiscous mode.

The post has some details about which files were updated, however I think it would be easiest just to zip up the *entire* NDK you have and post it to the forum.  This would allow me to build against the exact same thing as you are.  Would it be possible to do that?  It may also be a good idea to do the same for the NSP you have.

Thanks,

Steve

Happy to send you my copy of the NDK and NSP. 

I have had trouble posting files to the forums before.  Do you have any hints for how to do this? zip format, etc?

Corey,

A zip format should work ... however I'm wondering if there's some sort of file size limit for the forums.

An alternative would be to use a cloud file share service such as drop box.  As long as you are not putting any proprietary code into the zip (it should just be NDK stuff, which is open sourced) then I don't see any issue with that.

Steve

Here's my copy of the packages:

http://dl.dropbox.com/u/47211568/ndk_2_20_03_24.zip

http://dl.dropbox.com/u/47211568/nsp_1_00_00_09.zip

Other possibly relevant modifications may be found in the code you received which I had sent to your co-worker.

Specifically, I think the version of ethdriver.c in my code will override the version of ethdriver.c contained in the above-linked NSP sources.

Thanks

Corey

Corey,

Is there a project file you could share?

So far I've taken the sources you gave Todd and put them into an existing NDK project.

It looks like I need the file "listlib.h".  Maybe just sharing that would work.

Also, the preprocessor.tcf file requires the file "uppSample.tci".

I see in the PSP drivers there are two versions of this file.  Do you know which one your application is using?

The files are in either "evmA" or "evmB" directories:

C:\Program Files\Texas Instruments\pspdrivers_01_30_01\packages\ti\pspiom\examples\evmOMAPL138\uppEvm\{evmA, evmB}\build

Thanks,

Steve

"C:/ti/ccsv5/tools/compiler/c6000/bin/cl6x" -mv6740 -g --include_path="C:/ti/ccsv5/tools/compiler/c6000/include" --include_path="C:/Documents and Settings/a0323418/workspace_forums/ndk_evm6748_bios5_client" --include_path="C:/Program Files/Texas Instruments/ndk_2_20_03_24/packages/ti/ndk/inc" --include_path="C:/Program Files/Texas Instruments/ndk_2_20_03_24/packages/ti/ndk/inc/tools" --include_path="C:/Documents and Settings/a0323418/workspace_forums/ndk_evm6748_bios5_client/Debug" --include_path="C:/ti/bios_5_41_11_38/packages/ti/bios/include" --include_path="C:/ti/bios_5_41_11_38/packages/ti/rtdx/include/c6000" --diag_warning=225 --preproc_with_compile --preproc_dependency="preprocessor.pp"  "../preprocessor.c"
"../preprocessor.c", line 11: fatal error: could not open source file "listlib.h"
1 fatal error detected in the compilation of "../preprocessor.c".
Compilation terminated.

Steve,

listlib.h: http://dl.dropbox.com/u/47211568/listlib.h

Note: This is identical to the one found at: packages\ti\ndk\inc\stack\inc\listlib.h within ndk version I previously posted (http://dl.dropbox.com/u/47211568/ndk_2_20_03_24.zip).  You probably just need to add "packages\ti\ndk\inc\stack\inc" to your project's included directories.

uppSample.tci: http://dl.dropbox.com/u/47211568/uppSample.tci

Thanks

Corey

Hi Corey,

Thanks for posting those.

I'm now hitting the following:

"../preprocessor.c", line 39: fatal error: could not open source file "gpiofunctions.h"

I'm not sure where this file is coming from.  I thought it would be part of the PSP drivers, but I'm not finding it in there.  Do you know where I could find this file?

Also, it may be easier if you zip up and attach your entire project.  This would allow me to see all of the dependencies you are using so I would be able to find things like this on my own (assuming they are part of a TI product).

Thanks,

Steve

Steve,

Thank you for your persistence.

I agree, it would be easier to send you the whole project.  As it may include proprietary and sensitive content, I need a way to send it to you directly, rather than posting it publicly.

Maybe I can send it to your email?

Thanks

Corey

Corey,

I've had some set up issues but finally have an Ubuntu machine where I'm able to run the eth-send application you shared.

However, I'm having trouble reproducing the issue still, but maybe you have an idea.  Here's where I'm at so far ...

I'm able to build your project that you sent.

I've got my PC and OMAPL138 hooked up to a hub which in turn is going through a switch that connects to the LAN.  The PC is used for CCS and also running Wireshark so that I can see packets being sent to the DSP.  The Ubuntu box is on the LAN.

I am able to ping the DSP (146.252.161.26) from the Ubuntu box (146.252.161.36) and I see those packets being captured in Wireshark.

However, when I run the eth-send app, I don't see any packets showing up in Wireshark.

I'd like to avoid fully digging into debugging the eth-send program as I'm short on time and you probably know this program well ... How does the app find the DSP?  Do I need to add the DSP's IP address <--> MAC Address mapping into the ARP table of the Ubuntu box? Or do I need a different network set up?

Thanks,

Steve

From the Ubuntu box:

a0323418@dta0868361-ubuntu1:~/ethtest$ ping 146.252.161.26
PING 146.252.161.26 (146.252.161.26) 56(84) bytes of data.
64 bytes from 146.252.161.26: icmp_seq=1 ttl=255 time=1.38 ms
64 bytes from 146.252.161.26: icmp_seq=2 ttl=255 time=0.380 ms
64 bytes from 146.252.161.26: icmp_seq=3 ttl=255 time=0.378 ms
^C
--- 146.252.161.26 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.378/0.714/1.385/0.474 ms
a0323418@dta0868361-ubuntu1:~/ethtest$ sudo ./eth-send eth0 5
sent 5 frames
a0323418@dta0868361-ubuntu1:~/ethtest$

And I only see the pings ...

The configuration is a point-to-point link, so connect the DSP to the Linux box directly a cross-over cable (it's possible that a regular patch cable will work, too).

In the "eth-send" call make sure you reference the ethernet port through which they are connected. If it's eth0, then "eth0". If eth1, then "eth1".

"eth-send" may need to be run as root to allow promiscuous mode.

Let me know how it goes.

Thanks

Corey

Hi Corey,

Over the past few weeks I've been spending a few hours here and there trying to get a good set up in order to reproduce and then debug this problem, but I have not been able to suceed so far.

I first tried to use a physical Ubuntu machine which we already have had set up.  However, TI has restricted the Linux box to prevent users like me to set a static IP address on it.

So, I next tried a different route.  I've downloaded Ubuntu and got it running on my PC as a virtual machine.  I'm able to ping the board from within the virtual machine with the default DHCP assigned IP address.  However, once I try to configure the virtual Ubuntu image to have a static IP address, I can no longer reach the board with ping.  (maybe I need to set my windows machine to use a static IP as well...)

Before I spent too much time on trying to debug this set up, I wanted to know if you believe it is even valid?  Will I be able to reproduce this using a virtual machine?

My physical network set up is as follows.  I'm using a hub, which I don't think should be an issue here but I'll mention it just in case:

• PC running windows

• connected to a hub via Ethernet cable
• Also running Wireshark

• VirtualBox running Ubuntu virtual machine

• this is running on the PC mentioned above
• all networking is channeled through the PC's network connection

• OMAPL138

• connected to hub  via Ethernet cable

Steve

Steve,

It seems to me that it should be able to work with a virtual machine.

Remember, this has nothing to do with IP. The packets we are sending are not IP packets; they are raw MAC-layer packets.

Things to consider:

1. The hub could be a problem; I recommended an Ethernet cable running directly between the PC (running Ubuntu as VM) and the c6748.
2. If the PC grants the Ubuntu VM the permissions to send Raw packets (and run in promiscuous mode) it should work. The test program must be run as root.
3. After taking care of (1) and (2) we should be able to test whether the packets are being sent from the Ubuntu VM.
1. If Wireshark reports outgoing packets, try running it on the PC and see if the outgoing packets show up.
2. Unplug the Ethernet cable from the c6748 and plug it into another PC running Wireshark and look for the incoming packets. I know for sure (as I have done this before) that (if Wireshark is configured correctly) Wireshark does show incoming packets sent by my test problem. I don't remember about the configuration of Wireshark but make sure it will show packets with the wrong destination MAC address (aka "Promiscuous mode")

Thanks

Corey

Corey,

Do you have a Windows side version of the test apps (eth-send, etc.)?

I went and bought a crossover cable to eliminate the hub from the equation.

However, I'm still not able to see anything coming through on Wireshark on the set up mentioned above.  I run the command (sudo ./eth-send eth0 5) and it says that it sends the 5 packets.  But, I don't see anything show up in Wireshark.  I also ensured that the network settings within VirtualBox are set to "allow all" for promiscuous mode, as well as in the Wireshark settings.  Is there something else that I'm missing??

I've also tried booting Ubuntu from a USB drive to eliminate the virtual machine + Windows setup.  I'm able to successfully boot; however I'm not able to mount a USB stick that has a Wireshark bundle for Linux on it (actually not even sure that Wireshark will work on Ubuntu ... and I'm not sure how I will be able to know that the packets are going through without Wireshark).

I can keep trying with this approach, but it's taking a lot of time here, and I want to use the time I have allocated for your issue as wisely as possible.  So, I'm hoping there is a way to reproduce this issue from my Windows machine as it would eliminate all of this set up overhead ...

Thanks,

Steve

Steve,

I appreciate all the effort and time trying to get this to work. Honestly, I never had this much difficulty trying to do this. On the other hand, I have had access to Linux machines as well as root permissions.

I think the Ubuntu thumb drive approach is better, since then you don't have any questions about Virtual Machines and network interface permissions. Ideally you would be given a Linux box on which you could run programs as "root", since this is needed for raw socket communication and promiscuous mode, but if you can run as "root" with a Ubuntu thumb drive then I guess that's the way to go.

Instead of worrying about putting Wireshark on the Linux box, see if the sent packets can be seen on the receiving end. To do this, instead of connecting the cable to the c6748, connect it to a Windows machine running Wireshark; Wireshark should be able to see the incoming packets.

Also before running "eth-send", if the Ubuntu machine has multiple ethernet ports, make sure you are sending through the correct network device.

If there is only one the correct command is probably: sudo ./eth-send eth0 5

But, if there are two ethernet ports, the correct command may be: sudo ./eth-send eth1 5

Also, what version of Ubuntu are you running?

Let me know what happens...

Thanks

Corey