Hello,
Sorry for my English, first of all...
We just have started with the C6678 EVM and the first task is to verify that our system design is viable. We are going to use C6678 DSP for intensive real-time data processing. In our system we receive raw sensor data via Ethernet (connected internally in chassis) then process it and send the results via a second Ethernet port. Actually our target throughput on input is quite low: near about 70 Mbit/s (but we really need a lot of computation power).
We are now experimenting with the NDK and our test bench is based on the 'Client' example and iperf. We slightly modified the example so that it could serve as a server for iperf (modified client.c is attached). Please note that for now we are interested only in NDK throughput so we use non-copy (SOCK_STREAMNC) sockets.
So far so good. Our NDK application works (wow!) but the results are a bit frustrating: measured throughput is as low as ~1Mbit/s and even this throughput is not stable. Here's a sample of the iperf client output:
> iperf.exe -c 192.168.0.200 -n 100K -i 1
------------------------------------------------------------
Client connecting to 192.168.0.200, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[172] local 192.168.0.2 port 53122 connected with 192.168.0.200 port 5001
[ ID] Interval Transfer Bandwidth
[172] 0.0- 1.0 sec 24.0 KBytes 197 Kbits/sec
[ ID] Interval Transfer Bandwidth
[172] 1.0- 2.0 sec 0.00 Bytes 0.00 bits/sec
[ ID] Interval Transfer Bandwidth
[172] 2.0- 3.0 sec 16.0 KBytes 131 Kbits/sec
[ ID] Interval Transfer Bandwidth
[172] 3.0- 4.0 sec 16.0 KBytes 131 Kbits/sec
[ ID] Interval Transfer Bandwidth
[172] 4.0- 5.0 sec 32.0 KBytes 262 Kbits/sec
[ ID] Interval Transfer Bandwidth
[172] 0.0- 5.3 sec 104 KBytes 160 Kbits/sec
And here's a sample screenshot of Wireshark:
(zipped .pcap log is attached as well). For completeness, tcp, ip and GbE stats are given below.
We've spent a couple of days investigating the problem and found that NDK feels bad when MSS value is greater than ~1KByte (Iperf allows to explicitly define MSS but it does not work in Windows, so we simply change MTU setting for the connection):
> netsh interface ipv4 set subinterface "Local Area Connection" mtu=1000 store=persistent
Ok.
> iperf.exe -c 192.168.0.200 -n 100K -i 1
------------------------------------------------------------
Client connecting to 192.168.0.200, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[184] local 192.168.0.2 port 53856 connected with 192.168.0.200 port 5001
[ ID] Interval Transfer Bandwidth
[184] 0.0- 0.1 sec 104 KBytes 6.83 Mbits/sec
With enough data to send (-n 1G, for example - 1GBytes to transmit) measured throughput is about 140Mbit/s (it actually varies a lot from time to time, we've even seen ~400Mbit/s).
It would be ok for us, but unfortunately, it works for about 20 minutes and then the throughput suddenly drops to ~100-200Kbit/s:
...
[188] 1042.0-1043.0 sec 22.1 MBytes 185 Mbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1043.0-1044.0 sec 22.0 MBytes 185 Mbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1044.0-1045.0 sec 14.1 MBytes 118 Mbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1045.0-1046.0 sec 10.3 MBytes 86.4 Mbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1046.0-1047.0 sec 4.70 MBytes 39.5 Mbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1047.0-1048.0 sec 16.0 KBytes 131 Kbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1048.0-1049.0 sec 32.0 KBytes 262 Kbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1049.0-1050.0 sec 40.0 KBytes 328 Kbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1050.0-1051.0 sec 24.0 KBytes 197 Kbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1051.0-1052.0 sec 0.00 Bytes 0.00 bits/sec
[ ID] Interval Transfer Bandwidth
[188] 1052.0-1053.0 sec 16.0 KBytes 131 Kbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1053.0-1054.0 sec 32.0 KBytes 262 Kbits/sec
[ ID] Interval Transfer Bandwidth
[188] 1054.0-1055.0 sec 16.0 KBytes 131 Kbits/sec
...
When this happened NDK we always have ~100-200Kbit/s throughput and only reboot helps
Our setup:
NDK: ndk_2_20_04_26
PDK: pdk_C6678_1_0_0_17
BIOS: bios_6_32_05_54
CCS: ccs_base_5.0.3.00028
MCSDK: mcsdk_2_00_05_17
Boot modes: via IBL/TFTP, via CCS/XDS100
Please note that we have experimented with point-to-point connections (no intermediate router/switch), we've verified that our cables are ok, we use large TCP window size (48K) and we use NC_PRIORITY_HIGH priority in NC_SystemOpen(). We've googled a lot and we've read this topic (http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/178524.aspx) :)) We really need your help here.
Thanks in advance,
Dmitry
TCP/IP stats (via telnet console):
TCP/IP iPerf compatible server
Welcome connection : 192.168.0.2:53458
Welcome to the console program.
Enter '?' or 'help' for a list of commands.
>stat tcp
TCP Statistics:
RcvTotal = 0000000106 RcvShort = 0000000000
RcvHdrSize = 0000000000 RcvBadSum = 0000000001
RcvAfterClose = 0000000000 RcvDupAck = 0000000000
RcvPack = 0000000080 RcvByte = 0000097667
RcvAckPack = 0000000012 RcvAckByte = 0000000193
RcvDupPack = 0000000006 RcvDupByte = 0000005840
RcvPartDupPack = 0000000001 RcvPartDupByte = 0000001371
RcvAfterWinPack= 0000000000 RcvAfterWinByte= 0000000000
RcvOOPack = 0000000025 RcvOOByte = 0000036500
RcvWinUpd = 0000000000 RcvWinProbe = 0000000000
RcvAckTooMuch = 0000000000 SndNoBufs = 0000000000
SndTotal = 0000000084 SndProbe = 0000000000
SndPack (data) = 0000000011 SndByte (data) = 0000000191
SndRexmitPack = 0000000001 SndRexmitByte = 0000000018
SndAcks = 0000000059 SndCtrl = 0000000002
SndUrg = 0000000000 SndWinUp = 0000000011
SegsTimed = 0000000013 RttUpdated = 0000000011
Connects = 0000000002 ConnAttempt = 0000000000
Drops = 0000000001 ConnDrops = 0000000000
Accepts = 0000000002 TimeoutDrops = 0000000001
KeepDrops = 0000000000 DelAck = 0000000002
KeepProbe = 0000000000 PersistTimeout = 0000000000
KeepTimeout = 0000000000 RexmtTimeout = 0000000013
>stat ip
IP Statistics:
Total = 0000000248 Odropped = 0000000000
Badsum = 0000000000 Badhlen = 0000000000
Badlen = 0000000001 Badoptions = 0000000000
Badvers = 0000000000 Forward = 0000000000
Noproto = 0000000000 Delivered = 0000000247
Cantforward = 0000000000 CantforwardBA = 0000000000
Expired = 0000000000 Redirectsent = 0000000000
Localout = 0000000116 Localnoroute = 0000000000
CacheHit = 0000000009 CacheMiss = 0000000003
Fragments = 0000000000 Fragdropped = 0000000000
Fragtimeout = 0000000000 Reassembled = 0000000000
Ofragments = 0000000000 Fragmented = 0000000000
Cantfrag = 0000000000 Filtered = 0000000000
>
GbE stats (the function that prints this was adapted from 8765.cpsw_stats_print.gel posted by a TI employee here: http://e2e.ti.com/support/embedded/bios/f/355/t/234454.aspx):
RX Good Frames ................ 0x87d4
RX Broadcast Frames ........... 0x27b
RX Multicast Frames ........... 0x70b
RX Pause Frames ............... 0x0
RX CRC Errors ................. 0x0
RX Align/Code Errors .......... 0x0
RX Oversized Frames ........... 0x0
RX Jabber Frames .............. 0x0
RX Undersized Frames .......... 0x0
RX Fragments .................. 0x0
RX Octets ..................... 0x11c49d5
TX Good Frames ................ 0x7e35
TX Broadcast Frames ........... 0x6
TX Multicast Frames ........... 0x0
TX Pause Frames ............... 0x0
TX Deferred Frames ............ 0x0
TX Collision Frames ........... 0x0
TX Single Collision Frames .... 0x0
TX Multiple Collision Frames .. 0x0
TX Excessive Collision Frames . 0x0
TX Late Collisions ............ 0x0
TX Underrun ................... 0x0
TX Carrier Sense Errors ....... 0xb
TX Octets ..................... 0x21920a
64 Byte Octet Frames .......... 0x443
65 to 127 Byte Octet Frames ... 0x7f42
128 to 255 Byte Octet Frames .. 0x3ac
256 to 511 Byte Octet Frames .. 0x36
512 to 1024 Byte Octet Frames . 0x7e39
Over 1024 Byte Octet Frames . 0x6f
Net Octets .................... 0x13df1a9
RX Start of Frame Overruns .... 0x0
RX Middle of Frame Overruns ... 0x0
RX DMA Overruns ............... 0x0
/* * ======== client.c ======== * * TCP/IP Network Client example ported to use BIOS6 OS. * * Copyright (C) 2007, 2011 Texas Instruments Incorporated - http://www.ti.com/ * * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the * distribution. * * Neither the name of Texas Instruments Incorporated nor the names of * its contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * */ #include "client.h" #include <stdio.h> #include <ti/ndk/inc/netmain.h> #include <ti/ndk/inc/_stack.h> #include <ti/ndk/inc/tools/servers.h> #include <ti/ndk/inc/tools/console.h> /* BIOS6 include */ #include <ti/sysbios/BIOS.h> /* Platform utilities include */ #include "ti/platform/platform.h" /* Resource manager for QMSS, PA, CPPI */ #include "ti/platform/resource_mgr.h" #include "gbestat.h" //--------------------------------------------------------------------------- // Title String // char *VerStr = "\nTCP/IP iPerf compatible server\n"; // Our NETCTRL callback functions static void NetworkOpen(); static void NetworkClose(); static void NetworkIPAddr( IPN IPAddr, uint IfIdx, uint fAdd ); static void ServiceReport( uint Item, uint Status, uint Report, HANDLE h ); static int dtask_tcp_iperf( SOCKET s, UINT32 unused ); /* Platform Information - we will read it form the Platform Library */ platform_info gPlatformInfo; //--------------------------------------------------------------------------- // Configuration // char *HostName = "tidsp"; char *LocalIPAddr = "192.168.0.200"; char *LocalIPMask = "255.255.255.0"; // Not used when using DHCP char *GatewayIP = "192.168.0.1"; // Not used when using DHCP char *DomainName = "local"; // Not used when using DHCP char *DNSServer = "0.0.0.0"; // Used when set to anything but zero /************************************************************************* * @b EVM_init() * * @n * * Initializes the platform hardware. This routine is configured to start in * the evm.cfg configuration file. It is the first routine that BIOS * calls and is executed before Main is called. If you are debugging within * CCS the default option in your target configuration file may be to execute * all code up until Main as the image loads. To debug this you should disable * that option. * * @param[in] None * * @retval * None ************************************************************************/ void EVM_init() { platform_init_flags sFlags; platform_init_config sConfig; /* Status of the call to initialize the platform */ int32_t pform_status; /* * You can choose what to initialize on the platform by setting the following * flags. Things like the DDR, PLL, etc should have been set by the boot loader. */ memset( (void *) &sFlags, 0, sizeof(platform_init_flags)); memset( (void *) &sConfig, 0, sizeof(platform_init_config)); sFlags.pll = 0; /* PLLs for clocking */ sFlags.ddr = 0; /* External memory */ sFlags.tcsl = 1; /* Time stamp counter */ sFlags.phy = 1; /* Ethernet */ sFlags.ecc = 0; /* Memory ECC */ sConfig.pllm = 0; /* Use libraries default clock divisor */ pform_status = platform_init(&sFlags, &sConfig); /* If we initialized the platform okay */ if (pform_status != Platform_EOK) { /* Initialization of the platform failed... die */ while (1) { (void) platform_led(1, PLATFORM_LED_ON, PLATFORM_USER_LED_CLASS); (void) platform_delay(50000); (void) platform_led(1, PLATFORM_LED_OFF, PLATFORM_USER_LED_CLASS); (void) platform_delay(50000); } } } //--------------------------------------------------------------------- // Main Entry Point //--------------------------------------------------------------------- int main() { /* Start the BIOS 6 Scheduler */ BIOS_start (); } // // Main Thread // // int StackTest() { int rc; int i; HANDLE hCfg; QMSS_CFG_T qmss_cfg; CPPI_CFG_T cppi_cfg; /* Get information about the platform so we can use it in various places */ memset( (void *) &gPlatformInfo, 0, sizeof(platform_info)); (void) platform_get_info(&gPlatformInfo); (void) platform_uart_init(); (void) platform_uart_set_baudrate(115200); (void) platform_write_configure(PLATFORM_WRITE_ALL); /* Clear the state of the User LEDs to OFF */ for (i=0; i < gPlatformInfo.led[PLATFORM_USER_LED_CLASS].count; i++) { (void) platform_led(i, PLATFORM_LED_OFF, PLATFORM_USER_LED_CLASS); } /* Initialize the components required to run this application: * (1) QMSS * (2) CPPI * (3) Packet Accelerator */ /* Initialize QMSS */ if (platform_get_coreid() == 0) { qmss_cfg.master_core = 1; } else { qmss_cfg.master_core = 0; } qmss_cfg.max_num_desc = MAX_NUM_DESC; qmss_cfg.desc_size = MAX_DESC_SIZE; qmss_cfg.mem_region = Qmss_MemRegion_MEMORY_REGION0; if (res_mgr_init_qmss (&qmss_cfg) != 0) { platform_write ("Failed to initialize the QMSS subsystem \n"); goto main_exit; } else { platform_write ("QMSS successfully initialized \n"); } /* Initialize CPPI */ if (platform_get_coreid() == 0) { cppi_cfg.master_core = 1; } else { cppi_cfg.master_core = 0; } cppi_cfg.dma_num = Cppi_CpDma_PASS_CPDMA; cppi_cfg.num_tx_queues = NUM_PA_TX_QUEUES; cppi_cfg.num_rx_channels = NUM_PA_RX_CHANNELS; if (res_mgr_init_cppi (&cppi_cfg) != 0) { platform_write ("Failed to initialize CPPI subsystem \n"); goto main_exit; } else { platform_write ("CPPI successfully initialized \n"); } if (res_mgr_init_pass()!= 0) { platform_write ("Failed to initialize the Packet Accelerator \n"); goto main_exit; } else { platform_write ("PA successfully initialized \n"); } // // THIS MUST BE THE ABSOLUTE FIRST THING DONE IN AN APPLICATION before // using the stack!! // rc = NC_SystemOpen( NC_PRIORITY_HIGH, NC_OPMODE_INTERRUPT ); if( rc ) { platform_write("NC_SystemOpen Failed (%d)\n",rc); for(;;); } // Print out our banner platform_write(VerStr); // // Create and build the system configuration from scratch. // // Create a new configuration hCfg = CfgNew(); if( !hCfg ) { platform_write("Unable to create configuration\n"); goto main_exit; } // We better validate the length of the supplied names if( strlen( DomainName ) >= CFG_DOMAIN_MAX || strlen( HostName ) >= CFG_HOSTNAME_MAX ) { platform_write("Names too long\n"); goto main_exit; } // Add our global hostname to hCfg (to be claimed in all connected domains) CfgAddEntry( hCfg, CFGTAG_SYSINFO, CFGITEM_DHCP_HOSTNAME, 0, strlen(HostName), (UINT8 *)HostName, 0 ); // Manually configure IP and Gateway { CI_IPNET NA; CI_ROUTE RT; IPN IPTmp; // Setup manual IP address bzero( &NA, sizeof(NA) ); NA.IPAddr = inet_addr(LocalIPAddr); NA.IPMask = inet_addr(LocalIPMask); strcpy( NA.Domain, DomainName ); NA.NetType = 0; // Add the address to interface 1 CfgAddEntry( hCfg, CFGTAG_IPNET, 1, 0, sizeof(CI_IPNET), (UINT8 *)&NA, 0 ); // Add the default gateway. Since it is the default, the // destination address and mask are both zero (we go ahead // and show the assignment for clarity). bzero( &RT, sizeof(RT) ); RT.IPDestAddr = 0; RT.IPDestMask = 0; RT.IPGateAddr = inet_addr(GatewayIP); // Add the route CfgAddEntry( hCfg, CFGTAG_ROUTE, 0, 0, sizeof(CI_ROUTE), (UINT8 *)&RT, 0 ); // Manually add the DNS server when specified IPTmp = inet_addr(DNSServer); if( IPTmp ) CfgAddEntry( hCfg, CFGTAG_SYSINFO, CFGITEM_DHCP_DOMAINNAMESERVER, 0, sizeof(IPTmp), (UINT8 *)&IPTmp, 0 ); } // Specify TELNET service for our Console example CI_SERVICE_TELNET telnet; bzero( &telnet, sizeof(telnet) ); telnet.cisargs.IPAddr = INADDR_ANY; telnet.cisargs.pCbSrv = &ServiceReport; telnet.param.MaxCon = 2; telnet.param.Callback = &ConsoleOpen; CfgAddEntry( hCfg, CFGTAG_SERVICE, CFGITEM_SERVICE_TELNET, 0, sizeof(telnet), (UINT8 *)&telnet, 0 ); // // Configure IPStack/OS Options // // Print all messages rc = DBG_INFO; CfgAddEntry( hCfg, CFGTAG_OS, CFGITEM_OS_DBGPRINTLEVEL, CFG_ADDMODE_UNIQUE, sizeof(uint), (UINT8 *)&rc, 0 ); // TCP Transmit buffer size rc = 8192; CfgAddEntry( hCfg, CFGTAG_IP, CFGITEM_IP_SOCKTCPTXBUF, CFG_ADDMODE_UNIQUE, sizeof(uint), (UINT8 *)&rc, 0 ); // TCP Receive limit (non-copy mode) rc = 48*1024; CfgAddEntry( hCfg, CFGTAG_IP, CFGITEM_IP_SOCKTCPRXLIMIT, CFG_ADDMODE_UNIQUE, sizeof(uint), (UINT8 *)&rc, 0 ); // // Boot the system using this configuration // // We keep booting until the function returns 0. This allows // us to have a "reboot" command. // do { platform_write( "NC_NetStart...\n"); rc = NC_NetStart( hCfg, NetworkOpen, NetworkClose, NetworkIPAddr ); } while( rc > 0 ); // Delete Configuration CfgFree( hCfg ); // Close the OS main_exit: NC_SystemClose(); return(0); } // // System Task Code [ Server Daemon Servers ] // static HANDLE hIperf; // // NetworkOpen // // This function is called after the configuration has booted // static void NetworkOpen() { // Create our local servers hIperf = DaemonNew( SOCK_STREAMNC, 0, 5001, dtask_tcp_iperf, OS_TASKPRIHIGH, OS_TASKSTKNORM, 0, 3 ); } // // NetworkClose // // This function is called when the network is shutting down, // or when it no longer has any IP addresses assigned to it. // static void NetworkClose() { DaemonFree( hIperf ); } // // NetworkIPAddr // // This function is called whenever an IP address binding is // added or removed from the system. // static void NetworkIPAddr( IPN IPAddr, uint IfIdx, uint fAdd ) { IPN IPTmp; if( fAdd ) platform_write("Network Added: "); else platform_write("Network Removed: "); // Print a message IPTmp = ntohl( IPAddr ); platform_write("If-%d:%d.%d.%d.%d\n", IfIdx, (UINT8)(IPTmp>>24)&0xFF, (UINT8)(IPTmp>>16)&0xFF, (UINT8)(IPTmp>>8)&0xFF, (UINT8)IPTmp&0xFF ); } static void ServiceReport( uint Item, uint Status, uint Report, HANDLE h ) { static char *TaskName[] = { "Telnet","HTTP","NAT","DHCPS","DHCPC","DNS" }; static char *ReportStr[] = { "","Running","Updated","Complete","Fault" }; static char *StatusStr[] = { "Disabled","Waiting","IPTerm","Failed","Enabled" }; platform_write( "Service Status: %-9s: %-9s: %-9s: %03d\n", TaskName[Item-1], StatusStr[Status], ReportStr[Report/256], Report&0xFF ); } static int dtask_tcp_iperf( SOCKET s, UINT32 unused ) { struct timeval to; int i; char *pBuf; HANDLE hBuffer; (void)unused; platform_write( "Connected!\n"); // Port 1 is available on the EVM Print_Ethernet_Statistics( 1, (printf_func_t) platform_write ); // Configure our socket timeout to be 5 seconds to.tv_sec = 5; to.tv_usec = 0; setsockopt( s, SOL_SOCKET, SO_SNDTIMEO, &to, sizeof( to ) ); setsockopt( s, SOL_SOCKET, SO_RCVTIMEO, &to, sizeof( to ) ); for(;;) { i = (int)recvnc( s, (void **)&pBuf, 0, &hBuffer ); // If we read data, echo it back if( i > 0 ) { //platform_write( "%d bytes received\n", (int) i ); recvncfree( hBuffer ); } // If the connection got an error or disconnect, close else { platform_write( "recvnc() flagged error. Disconnect\n"); break; } } fdClose( s ); // Return "0" since we closed the socket return(0); }