This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5728: Low TCP/IP throughput over Gbit Ethernet from AM572x EVM

Part Number: AM5728

Hi,

I've been working on some simple code to test the TCP/IP throughput we can get from the AM572x EVM using one of the Gigabit Ethernet ports. I wrote a simple socket application on windows to receive data from the EVM, and I modified one of the NIMU FTP example projects to get the configuration and setup. I removed the FTP code and replaced it with a simple socket application that loops over a "send()" call to send a message as fast as possible. I'm only able to get roughly 200Mbit/s, which is much lower than I would expect. Using the same recv() application but sending from another Windows machine, I'm able to get 800-900 Mbit/s, so the issue doesn't seem to be in how I'm receiving the messages, the Ethernet cable (CAT5e), or my PC's network card. 

I'm getting the best throughput with message sizes around 500 bytes, even though I'd expect the best throughput from messages right at the payload limit for Ethernet/TCP/IP (1460 bytes). When I send messages closer to or above that limit, my throughput is slightly lower, and eventually slows down drastically as I start getting a large volume of TCP retransmissions. One strange behavior I've seen is that if I kill my recv() application on my PC, the TCP retransmissions sent by the EVM jump from 200Mbit/s up to 800Mbit/s, suggesting that it should be capable of sending messages much faster while connected.

I'm using Processor SDK RTOS 6.03.00.106 and I modified the NIMU_FtpExample_evmAM572x_armExampleproject. I haven't changed much in the configuration, I've mostly tried optimizing my own code to send messages as quickly as possible. Is there anything else I need to change to optimize throughput? I can send you some of my code and configuration if it would help.

Thanks,

Jon

  • Hi Jon,

    Can you try increasing your socket buffers ? I don't see any underlying issue with the driver or HW since I have checked throughput with iperf on Linux for GMAC.

    Also, please share your custom app changes as a patch and I can take a look. Please share PC side application as well.

    Regards

    Vineet

  • Hi Vineet,

    I increased the TCP send buffer to 65,536 from 16,384 and noticed a substantial throughput increase up to ~500 Mbps. I increased it again to 131,072 and saw a further increase to 585 Mbps, but there was no change after doubling it again to 262,114. This is much better, but still less than the 800-900 Mbps I was hoping to get. I tried increasing the send buffer from 65,535 to 131,072 just to see if it would change anything even though I'm not receiving any data, but there was no change. I also tried changing the Network Scheduler task priority from Low to High, as well as setting the socket option TCP_NODELAY, but saw no throughput change.

    I'll attach my code example modifications and Windows TCP/IP application to this reply. The main am572x example modifications will be in main.c and QuickSend.c, which creates the task to rapidly generate messages. The cfg file likely also has some changes. In case you try to build and run the code, I've attached all the other important files as well. Anything not included can be commented out.

    The Windows TCP/IP application is in "listener.c". If you wind up compiling it, you can just remove any checksum references. I call it with the command "listener listen" to have it listen on 192.168.1.5:6500. I also have a separate Linux version I've used but did not include, since it seems to perform similarly.

    Thanks,

    Jon

    /**
     * References: https://docs.microsoft.com/en-us/windows/win32/winsock/getting-started-with-winsock
     *
     *
     */
    
    /* Custom includes */
    #include "../checksums/checksums.h"
    
    /* Windows includes */
    #include <winsock2.h>
    #include <ws2tcpip.h>
    #include <iphlpapi.h>
    
    /* Std */
    #include <stdio.h>
    #include <stdbool.h>
    
    #pragma comment(lib, "Ws2_32.lib")
    
    #define PRINT_LOOP_PERIOD 100000
    
    #define LISTEN_IP_ADDR "192.168.1.5"
    #define LISTEN_PORT 6500
    
    #define RECV_BUF_SIZE 11680
    
    //#define DO_CHECKSUM
    
    /* Function prototypes */
    void process_connection(int new_fd);
    void print_stats(uint32_t msgs_corr_cyc, uint32_t msgs_recvd_cyc, uint32_t msgs_corr_tot,
    		uint32_t msgs_recvd_tot);
    float calc_pct_corr(uint32_t msgs_corr, uint32_t msgs_recvd);
    
    int main(int argc, char * argv[])
    {
    	int sockfd, retval;
    	struct sockaddr_in my_addr;
    
    	/* From tutorial */
    	struct addrinfo *result = NULL,
    			*ptr = NULL,
    			hints;
    
    	WSADATA wsaData;
    	int iResult;
    
    	// Initialize Winsock
    	iResult = WSAStartup(MAKEWORD(2,2), &wsaData);
    	if (iResult != 0)
    	{
    		printf("WSAStartup failed: %d\n", iResult);
    		return 1;
    	}
    
    	sockfd = socket(PF_INET, SOCK_STREAM, 0);
    	if (-1 == sockfd)
    	{
    		perror("socket() error");
    		return -1;
    	}
    	printf("Socket created\n");
    
    	my_addr.sin_family = AF_INET;
    	my_addr.sin_port = htons(LISTEN_PORT);
    	my_addr.sin_addr.s_addr = inet_addr(LISTEN_IP_ADDR);
    	memset(my_addr.sin_zero, '\0', sizeof my_addr.sin_zero);
    
    	retval = bind(sockfd, (struct sockaddr *)&my_addr, sizeof my_addr);
    	if (-1 == retval)
    	{
    		perror("bind() error");
    		return -1;
    	}
    	printf("Bound\n");
    
    	retval = listen(sockfd, 1);
    	if (-1 == retval)
    	{
    		printf("listen() call failed with errno %d\n", errno);
    		return -1;
    	}
    	else
    	{
    		int new_fd;
    		//struct sockaddr_storage their_addr;
    		struct sockaddr their_addr;	
    		//socklen_t addr_size;
    		int addr_size;
    
    		printf("Listening on %s:%u\n", LISTEN_IP_ADDR, LISTEN_PORT);
    
    		while(1)
    		{
    			new_fd = accept(sockfd, (struct sockaddr *)&their_addr, &addr_size);
    			if (-1 == new_fd)
    			{
    				printf("accept() call failed with errno %d\n", errno);
    				return -1;
    			}
    			else
    			{
    				process_connection(new_fd);	
    			}
    		}
    	}
    
    	printf("Hello, world!\n");
    
    	return 0;
    }
    
    void process_connection(int new_fd)
    {
    	char recv_buf[RECV_BUF_SIZE];
    	int recv_bytes;
    	uint32_t msgs_recvd_tot = 0;
    	uint32_t msgs_corr_tot = 0;
    	uint32_t msgs_recvd_cyc = 0;
    	uint32_t msgs_corr_cyc = 0;
    	uint32_t loop_count = 0;
    	uint64_t bytes_recvd_cyc = 0;
    	uint64_t bytes_recvd_tot = 0;
    
    	printf("Accepted connection\n");
    
    	while(1)
    	{
    		loop_count++;
    
    		//recv_bytes = recv(new_fd, recv_buf, sizeof(recv_buf), 0);
    		recv_bytes = recv(new_fd, recv_buf, RECV_BUF_SIZE, 0);
    		if (0 < recv_bytes)
    		{
    			msgs_recvd_cyc++;
    			/* Succees! Test message now */
    #ifdef DO_CHECKSUM	
    			if (!checksum_correct(recv_buf, recv_bytes, 1))
    #else
    			if (0) // Check number of bytes instead?
    #endif
    			{
    			//	printf("CS incorrect: %d bytes\n", recv_bytes);
    			}
    			else
    			{
    				msgs_corr_cyc++;
    				bytes_recvd_cyc += recv_bytes;
    			}
    			//printf("Received %d bytes\n", recv_bytes);
    			if (0 == loop_count % PRINT_LOOP_PERIOD)
    			{
    				bytes_recvd_tot += bytes_recvd_cyc;
    				printf("Bytes recvd cyc: %f M tot: %f M\n", (float)bytes_recvd_cyc / 1e6, (float)bytes_recvd_tot / 1e6);
    				msgs_corr_tot += msgs_corr_cyc;
    				msgs_recvd_tot += msgs_recvd_cyc;
    				print_stats(msgs_corr_cyc, msgs_recvd_cyc, msgs_corr_tot,
    						msgs_recvd_tot);
    				msgs_corr_cyc = 0;
    				msgs_recvd_cyc = 0;
    				bytes_recvd_cyc = 0;
    			}
    		}
    		else if (-1 == recv_bytes)
    		{
    			printf("recv() failed\n");
    			//return -1;
    			return;
    		}	
    		else if (0 == recv_bytes)
    		{
    			printf("Connection closed\n");
    			break;
    		}
    	}
    	printf("Shutting down connection\n");
    	shutdown(new_fd, SD_BOTH);
    }
    
    float calc_pct_corr(uint32_t msgs_corr, uint32_t msgs_recvd)
    {
    	if (0 == msgs_corr || 0 == msgs_recvd)
    	{
    		return 0;
    	}
    
    	return 100.0 * (float) msgs_corr / (float) msgs_recvd;
    }
    
    void print_stats(uint32_t msgs_corr_cyc, uint32_t msgs_recvd_cyc, uint32_t msgs_corr_tot,
    		uint32_t msgs_recvd_tot)
    {
    	float pct_corr_cyc, pct_corr_tot;
    
    	pct_corr_cyc = calc_pct_corr(msgs_corr_cyc, msgs_recvd_cyc);
    	pct_corr_tot = calc_pct_corr(msgs_corr_tot, msgs_recvd_tot);
    
    	printf("Success: CYC: %f (%u/%u) TOT: %f (%u/%u)\n", 
    			pct_corr_cyc, msgs_corr_cyc, msgs_recvd_cyc,
    			pct_corr_tot, msgs_corr_tot, msgs_recvd_tot);	
    }
    
    /*
     * QuickSend.c
     *
     * Send messages to a foreign machine as quickly as possible
     *
     *  Created on: Mar 12, 2021
     *      Author: jon.lake
     */
    
    /*
     * Header includes
     */
    // Custom
    #include <thermo_utils.h>
    #include <QuickSend.h>
    // NDK
    #include <ti/ndk/inc/netmain.h>
    // C Std
    #include <stdlib.h> // For free
    
    
    /*
     * Macro defines
     */
    // Task configuration
    #define QS_TASK_STACK_SIZE 8192 //2048 // Arbitrary
    #define QUICKSEND_CONNECT_WAIT_S 2
    #define QUICKSEND_STARTUP_WAIT_MS 15000 // t + 5s for worst case Ethernet negotiation
    // Network configuration
    #define FOREIGN_IP_ADDR "192.168.1.5"
    #define FOREIGN_PORT 6500
    
    
    /*
     * struct definitions
     */
    
    /*
     * Function prototypes
     */
    void quicksend_task_fxn(UArg arg0, UArg arg1);
    
    /*
     * File scope variables
     */
    static volatile MODULE_STATE fg_qsModule_state = MODSTATE_INIT;
    /*
     * Function definitions
     */
    
    
    /*
     * Desc
     *
     * @param priority
     * @param p_eb
     *
     * @return Task_Handle
     */
    Task_Handle QuickSend_create(int priority, Error_Block *p_eb)
    {
        Task_Params taskParams;
        Task_Handle quickSend_task;
    
        // Set up parameters for the task
        Task_Params_init(&taskParams);
        Error_init(p_eb);
        taskParams.stackSize = QS_TASK_STACK_SIZE;
        taskParams.priority = priority;
        taskParams.instance->name = "QuickSend";
    
        quickSend_task = Task_create(quicksend_task_fxn, &taskParams, p_eb);
    
        return quickSend_task;
    }
    
    
    void quicksend_task_fxn(UArg arg0, UArg arg1)
    {
        /***********
         * Variables
         ***********/
    
        SOCKET send_sock;   // Socket we'll send on
        struct sockaddr_in send_sockaddr_in;    // Configuration for the foreign sock
        int sockaddr_in_sz = sizeof(send_sockaddr_in);    // Size of fsock config
        int bytes_sent;     // Bytes sent to the other socket
        // Socket options
        int32_t nodelay_opt = 1;
        uint32_t snd_recv_buf_sz_opt = 0; // Using this to check size for now
        int snd_recv_buf_sz_opt_sz = sizeof(snd_recv_buf_sz_opt);
    
        uint32_t msg_len = 500; // 1000;
        char *p_msg = set_arbitrary_msg(msg_len, 1);//"Hello!";
        //int msg_len = strlen(p_msg);
    
        /**********
         * Function
         **********/
    
        fg_qsModule_state = MODSTATE_DISCONNECTED;
    
        print_fxn("Quicksend: Sleeping for %u ms to wait for Ethernet negotiation\n",
                  QUICKSEND_STARTUP_WAIT_MS);
        Task_sleep(QUICKSEND_STARTUP_WAIT_MS);
    
        // Necessary for sockets
        fdOpenSession(TaskSelf());
    
        // Create the send socket
        if (INVALID_SOCKET == (send_sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)))
        {
            print_fxn("QuickSend: socket() failed\n");
            fdCloseSession(TaskSelf());
            return;
        }
    
        // Disable Nagle algorithm
        if (SOCKET_ERROR == setsockopt(send_sock, IPPROTO_TCP, NDK_TCP_NODELAY, &nodelay_opt, sizeof(nodelay_opt)))
        {
            print_fxn("Quicksend: setsockopt() failed to disable Nagle algorithm\n");
            fdCloseSession(TaskSelf());
            return;
        }
    
        if (0 != getsockopt(send_sock, SOL_SOCKET, SO_SNDBUF, &snd_recv_buf_sz_opt, &snd_recv_buf_sz_opt_sz))
        {
            print_fxn("Failed to get SO_SNDBUF socket option\n");
            fdCloseSession(TaskSelf());
            return;
        }
        print_fxn("Socket send buffer size is %u\n", snd_recv_buf_sz_opt);
    
        if (0 != getsockopt(send_sock, SOL_SOCKET, SO_RCVBUF, &snd_recv_buf_sz_opt, &snd_recv_buf_sz_opt_sz))
        {
            print_fxn("Failed to get SO_RCVBUF socket option\n");
            fdCloseSession(TaskSelf());
            return;
        }
        print_fxn("Socket receive buffer size is %u\n", snd_recv_buf_sz_opt);
    
        // Check our socket options
    
    
        // Configure foreign connection
        send_sockaddr_in.sin_family = AF_INET;
        send_sockaddr_in.sin_addr.s_addr = inet_addr(FOREIGN_IP_ADDR);
        send_sockaddr_in.sin_port = NDK_htons(FOREIGN_PORT);
    
        // Loop forever while not broken
        while (MODSTATE_BROKEN != fg_qsModule_state)
        {
            // Attempt connection
            print_fxn("Attempting socket connection\n");
            while (MODSTATE_DISCONNECTED == fg_qsModule_state)
            {
                if (-1 == connect(send_sock, (struct sockaddr*)&send_sockaddr_in,
                                  sockaddr_in_sz))
                {
                    print_fxn("QuickSend: connect failure. Let's take %d...\n",
                              QUICKSEND_CONNECT_WAIT_S);
                    Task_sleep(QUICKSEND_CONNECT_WAIT_S);
                }
                else
                {
                    print_fxn("Quicksend: connected!\n");
                    fg_qsModule_state = MODSTATE_RUNNING;
                }
            } // Exit MODSTATE_DISCONNECTED
    
            print_fxn("Quicksend: Entering running state\n");
            while (MODSTATE_RUNNING == fg_qsModule_state)
            {
                bytes_sent = send(send_sock, p_msg, msg_len, 0);
    
                if (bytes_sent <= 0) // Not checking much besides any sort of success
                {
                    print_fxn("Send failed. QuickSend exiting running state\n");
                    fg_qsModule_state = MODSTATE_DISCONNECTED;
    
                    if (-1 == shutdown(send_sock, SHUT_WR))
                    {
                        print_fxn("Socket shutdown failed\n");
                    }
                    else
                    {
                        print_fxn("Socket shutdown succeeded\n");
                    }
                }
            } // Exit MODSTATE_RUNNING
    
        } // Exit !MODSTATE_BROKEN
    
        print_fxn("QuickSend module broken. Task ending.\n");
    
        free(p_msg);
    
        return;
    }
    
    QuickSend.h
    /*
     * thermo_utils.c
     *
     *  Created on: Mar 5, 2021
     *      Author: jon.lake
     */
    
    /* Custom includes */
    #include <thermo_utils.h>
    #include <stdlib.h>         // For calloc, free
    
    THERMO_ERROR gen_checksum_invsum(char *input_string, uint32_t input_size,
                                            uint32_t num_checksum_bytes)
    {
        /* Decision: keep c-style string representation, or replace null term w/checksum? */
        /* 8-bit checksum */
        char sum = 0;
        int i;
        /* This needs to be redone for non-8-bit checksums */
        for (i = 0; i < (input_size - num_checksum_bytes); i++)
        {
            /* ignoring 1's complement, 2's complement, etc. considerations. Straight sum. */
            sum += input_string[i];
        }
    
        // invert here. one's complement inverse?
        sum = ~sum;
    
        /* Put checksum at end */
        input_string[input_size - 1] = sum;
    
        /* Always succeed. Could we have failed anywhere? */
        return THERMO_NO_ERR;
    }
    
    
    /**
     * Desc
     *
     * @param[in]   num_bytes
     *
     * @return
     */
    char* set_arbitrary_msg(uint32_t num_bytes, uint32_t checksum_bytes)
    {
        // Add one character for NULL termination
        char *arbitrary_msg = calloc(num_bytes, sizeof(*arbitrary_msg) + 1);
    
        int i;
    
        for (i = 0; i < num_bytes - 1; i++)
        {
            arbitrary_msg[i] = 'a' + ( i % 26 ); // Iterate over all lowercase letters
        }
    
        gen_checksum_invsum(arbitrary_msg, num_bytes, checksum_bytes);
    
        //print_fxn("Arbitrary message checksum: %d (%c)\n", arbitrary_msg[num_bytes - 1],
        //          arbitrary_msg[num_bytes - 1]);
    
        arbitrary_msg[num_bytes] = '\0';
    
        //print_fxn("Generated arbitrary message %s\n", arbitrary_msg);
    
        return arbitrary_msg;
    }
    
    thermo_utils.hthermo_types.hEthSend.cfg
    /*
     * Copyright (C) 2017 - 2018 Texas Instruments Incorporated - http://www.ti.com/
     *
     *
     *  Redistribution and use in source and binary forms, with or without
     *  modification, are permitted provided that the following conditions
     *  are met:
     *
     *    Redistributions of source code must retain the above copyright
     *    notice, this list of conditions and the following disclaimer.
     *
     *    Redistributions in binary form must reproduce the above copyright
     *    notice, this list of conditions and the following disclaimer in the
     *    documentation and/or other materials provided with the
     *    distribution.
     *
     *    Neither the name of Texas Instruments Incorporated nor the names of
     *    its contributors may be used to endorse or promote products derived
     *    from this software without specific prior written permission.
     *
     *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
     *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
     *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
     *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
     *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
     *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
     *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
     *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
     *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
     *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
     *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
     *
    */
    
    /* ========================================================================== */
    /*                             Include Files                                  */
    /* ========================================================================== */
    
    #include <stdio.h>
    #include <string.h>
    #include <stdlib.h>
    #include <xdc/std.h>
    #include <xdc/runtime/Error.h>
    #include <xdc/runtime/System.h>
    #include <ti/sysbios/BIOS.h>
    #include <ti/sysbios/knl/Task.h>
    
    
    
    #include <ti/csl/soc.h>
    #include <ti/csl/cslr_device.h>
    #include <ti/csl/soc/am572x/src/csl_device_xbar.h>
    
    #include <ti/ndk/inc/netmain.h>
    #include <ti/ndk/inc/stkmain.h>
    #include "ti/ndk/inc/os/osif.h"
    
    #include <ti/board/board.h>
    
    #include <ti/drv/uart/UART.h>
    #include <ti/drv/uart/UART_stdio.h>
    
    #include <ti/drv/emac/emac_drv.h>
    #include <ti/drv/emac/src/v4/emac_drv_v4.h>
    
    #ifdef NIMU_FTP_APP
    #include <ti/transport/ndk/nimu/example/ftpApp/ftpserver/ftpserver.h>
    #endif
    
    /* Custom modules */
    #include <MShipMsgr.h>
    #include <MsgGenerator.h>
    #include <QuickSend.h>
    
    extern char *LocalIPAddr;
    extern void app_stats(UArg arg0, UArg arg1);
    
    /* ========================================================================== */
    /*                             Macros                                  */
    /* ========================================================================== */
    /* Enable the below macro to have prints on the IO Console */
    
    //#define IO_CONSOLE
    
    #ifndef IO_CONSOLE
    #define NIMU_log                UART_printf
    #else
    #define NIMU_log                printf
    #endif
    
    #define MAX_TABLE_ENTRIES   3
    
    /* ========================================================================== */
    /*                            Global Variables                                */
    /* ========================================================================== */
    
    /**Task handle for EIP*/
    Task_Handle main_task;
    
    static int nimu_device_index = 0U;
    
    NIMU_DEVICE_TABLE_ENTRY NIMUDeviceTable[MAX_TABLE_ENTRIES];
    
    void TaskFxn(UArg a0, UArg a1);
    extern int CpswEmacInit (STKEVENT_Handle hEvent);
    
    void raw_eth_task(UArg a0, UArg a1);
    
    
    /**
     *  \name main
     *  \brief Main Function
     *  \param none
     *  \return none
     *
     */
    int main()
    {
        /* Call board init functions */
        Board_initCfg boardCfg;
        Task_Params taskParams;
    
        boardCfg =  BOARD_INIT_PINMUX_CONFIG |
            BOARD_INIT_MODULE_CLOCK | BOARD_INIT_UART_STDIO;
    
        Board_init(boardCfg);
    
        CSL_xbarMpuIrqConfigure(CSL_XBAR_INST_MPU_IRQ_92, CSL_XBAR_GMAC_SW_IRQ_RX_PULSE);
        CSL_xbarMpuIrqConfigure(CSL_XBAR_INST_MPU_IRQ_93, CSL_XBAR_GMAC_SW_IRQ_TX_PULSE);
    
        /* Select RGMII 2 ports GMIIx_SEL = 2 for RGMII*/
        CSL_FINS (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->CONTROL_IO_1,
              CONTROL_CORE_CONTROL_IO_1_GMII1_SEL, 2U);
        CSL_FINS (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->CONTROL_IO_1,
              CONTROL_CORE_CONTROL_IO_1_GMII2_SEL, 2U);
    
        /*GMAC RESET ISOLATION Enable*/
        CSL_FINS (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->CONTROL_IO_2,
              CONTROL_CORE_CONTROL_IO_2_GMAC_RESET_ISOLATION_ENABLE, 0U);
        CSL_FINS (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->CONTROL_IO_2,
              CONTROL_CORE_CONTROL_IO_2_GMAC_RESET_ISOLATION_ENABLE, 1U);
    
        EMAC_HwAttrs_V4 cfg;
        EMAC_socGetInitCfg(0, &cfg);
        cfg.port[0].phy_addr = GMAC_PORT1_ETHERNET_PHY_ADRESS;
        cfg.port[1].phy_addr = GMAC_PORT2_ETHERNET_PHY_ADRESS;
        EMAC_socSetInitCfg(0, &cfg);
    
        Task_Params_init(&taskParams);
        taskParams.priority = 1;
        taskParams.stackSize = 0x1400;
        main_task = Task_create (TaskFxn, &taskParams, NULL);
    
        NIMUDeviceTable[nimu_device_index++].init =  &CpswEmacInit;
    
        NIMUDeviceTable[nimu_device_index].init =  NULL;
    
        BIOS_start();
    
        return -1;
    }
    
    
    /**
     *  \name TaskFxn
     *  \brief Task Fxn
     *  \param a0
     *  \param a1
     *  \return none
     *
     */
    void TaskFxn(UArg a0, UArg a1)
    {
        NIMU_log("\n\rSYS/BIOS Ethernet/IP (CPSW) Sample application, EVM IP address I/F 1: %s\n\r", LocalIPAddr);
        Error_Block eb;
    
        //ftpserver_init();
        /*if (NULL == MShipMsgr_create(7, &eb))
        {
            // print Error_getMsg(eb);
        }
    
        if (NULL == MsgGenerator_create(6, &eb))
        {
            // print error
        }*/
    
        if (NULL == QuickSend_create(1, &eb))
        {
            // print error
        }
    }
    
    /*
     * ======== stackInitHook ========
     * hook called from ti_nkd_config_Global_stackThread() to run user setup code
     */
    void stackInitHook(void* hCfg)
    {
        NIMU_log("In stackInitHook");
    }
    
    8032.main.h

  • Hi Vineet,

    I increased the TCP send buffer to 65,536 from 16,384 and noticed a substantial throughput increase up to ~500 Mbps. I increased it again to 131,072 and saw a further increase to 585 Mbps, but there was no change after doubling it again to 262,114. This is much better, but still less than the 800-900 Mbps I was hoping to get. I tried increasing the send buffer from 65,535 to 131,072 just to see if it would change anything even though I'm not receiving any data, but there was no change. I also tried changing the Network Scheduler task priority from Low to High, as well as setting the socket option TCP_NODELAY, but saw no throughput change.

    I'll attach my code example modifications and Windows TCP/IP application to this reply. The main am572x example modifications will be in main.c and QuickSend.c, which creates the task to rapidly generate messages. The cfg file likely also has some changes. In case you try to build and run the code, I've attached all the other important files as well. Anything not included can be commented out.

    The Windows TCP/IP application is in "listener.c". If you wind up compiling it, you can just remove any checksum references. I call it with the command "listener listen" to have it listen on 192.168.1.5:6500. I also have a separate Linux version I've used but did not include, since it seems to perform similarly.

    Edit: actually, my first reply was flagged as spam, so I've just included the most important files.

    Thanks,

    Jon

    /*
     * QuickSend.c
     *
     * Send messages to a foreign machine as quickly as possible
     *
     *  Created on: Mar 12, 2021
     *      Author: jon.lake
     */
    
    /*
     * Header includes
     */
    // Custom
    #include <thermo_utils.h>
    #include <QuickSend.h>
    // NDK
    #include <ti/ndk/inc/netmain.h>
    // C Std
    #include <stdlib.h> // For free
    
    
    /*
     * Macro defines
     */
    // Task configuration
    #define QS_TASK_STACK_SIZE 8192 //2048 // Arbitrary
    #define QUICKSEND_CONNECT_WAIT_S 2
    #define QUICKSEND_STARTUP_WAIT_MS 15000 // t + 5s for worst case Ethernet negotiation
    // Network configuration
    #define FOREIGN_IP_ADDR "192.168.1.5"
    #define FOREIGN_PORT 6500
    
    
    /*
     * struct definitions
     */
    
    /*
     * Function prototypes
     */
    void quicksend_task_fxn(UArg arg0, UArg arg1);
    
    /*
     * File scope variables
     */
    static volatile MODULE_STATE fg_qsModule_state = MODSTATE_INIT;
    /*
     * Function definitions
     */
    
    
    /*
     * Desc
     *
     * @param priority
     * @param p_eb
     *
     * @return Task_Handle
     */
    Task_Handle QuickSend_create(int priority, Error_Block *p_eb)
    {
        Task_Params taskParams;
        Task_Handle quickSend_task;
    
        // Set up parameters for the task
        Task_Params_init(&taskParams);
        Error_init(p_eb);
        taskParams.stackSize = QS_TASK_STACK_SIZE;
        taskParams.priority = priority;
        taskParams.instance->name = "QuickSend";
    
        quickSend_task = Task_create(quicksend_task_fxn, &taskParams, p_eb);
    
        return quickSend_task;
    }
    
    
    void quicksend_task_fxn(UArg arg0, UArg arg1)
    {
        /***********
         * Variables
         ***********/
    
        SOCKET send_sock;   // Socket we'll send on
        struct sockaddr_in send_sockaddr_in;    // Configuration for the foreign sock
        int sockaddr_in_sz = sizeof(send_sockaddr_in);    // Size of fsock config
        int bytes_sent;     // Bytes sent to the other socket
        // Socket options
        int32_t nodelay_opt = 1;
        uint32_t snd_recv_buf_sz_opt = 0; // Using this to check size for now
        int snd_recv_buf_sz_opt_sz = sizeof(snd_recv_buf_sz_opt);
    
        uint32_t msg_len = 500; // 1000;
        char *p_msg = set_arbitrary_msg(msg_len, 1);//"Hello!";
        //int msg_len = strlen(p_msg);
    
        /**********
         * Function
         **********/
    
        fg_qsModule_state = MODSTATE_DISCONNECTED;
    
        print_fxn("Quicksend: Sleeping for %u ms to wait for Ethernet negotiation\n",
                  QUICKSEND_STARTUP_WAIT_MS);
        Task_sleep(QUICKSEND_STARTUP_WAIT_MS);
    
        // Necessary for sockets
        fdOpenSession(TaskSelf());
    
        // Create the send socket
        if (INVALID_SOCKET == (send_sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)))
        {
            print_fxn("QuickSend: socket() failed\n");
            fdCloseSession(TaskSelf());
            return;
        }
    
        // Disable Nagle algorithm
        if (SOCKET_ERROR == setsockopt(send_sock, IPPROTO_TCP, NDK_TCP_NODELAY, &nodelay_opt, sizeof(nodelay_opt)))
        {
            print_fxn("Quicksend: setsockopt() failed to disable Nagle algorithm\n");
            fdCloseSession(TaskSelf());
            return;
        }
    
        if (0 != getsockopt(send_sock, SOL_SOCKET, SO_SNDBUF, &snd_recv_buf_sz_opt, &snd_recv_buf_sz_opt_sz))
        {
            print_fxn("Failed to get SO_SNDBUF socket option\n");
            fdCloseSession(TaskSelf());
            return;
        }
        print_fxn("Socket send buffer size is %u\n", snd_recv_buf_sz_opt);
    
        if (0 != getsockopt(send_sock, SOL_SOCKET, SO_RCVBUF, &snd_recv_buf_sz_opt, &snd_recv_buf_sz_opt_sz))
        {
            print_fxn("Failed to get SO_RCVBUF socket option\n");
            fdCloseSession(TaskSelf());
            return;
        }
        print_fxn("Socket receive buffer size is %u\n", snd_recv_buf_sz_opt);
    
        // Check our socket options
    
    
        // Configure foreign connection
        send_sockaddr_in.sin_family = AF_INET;
        send_sockaddr_in.sin_addr.s_addr = inet_addr(FOREIGN_IP_ADDR);
        send_sockaddr_in.sin_port = NDK_htons(FOREIGN_PORT);
    
        // Loop forever while not broken
        while (MODSTATE_BROKEN != fg_qsModule_state)
        {
            // Attempt connection
            print_fxn("Attempting socket connection\n");
            while (MODSTATE_DISCONNECTED == fg_qsModule_state)
            {
                if (-1 == connect(send_sock, (struct sockaddr*)&send_sockaddr_in,
                                  sockaddr_in_sz))
                {
                    print_fxn("QuickSend: connect failure. Let's take %d...\n",
                              QUICKSEND_CONNECT_WAIT_S);
                    Task_sleep(QUICKSEND_CONNECT_WAIT_S);
                }
                else
                {
                    print_fxn("Quicksend: connected!\n");
                    fg_qsModule_state = MODSTATE_RUNNING;
                }
            } // Exit MODSTATE_DISCONNECTED
    
            print_fxn("Quicksend: Entering running state\n");
            while (MODSTATE_RUNNING == fg_qsModule_state)
            {
                bytes_sent = send(send_sock, p_msg, msg_len, 0);
    
                if (bytes_sent <= 0) // Not checking much besides any sort of success
                {
                    print_fxn("Send failed. QuickSend exiting running state\n");
                    fg_qsModule_state = MODSTATE_DISCONNECTED;
    
                    if (-1 == shutdown(send_sock, SHUT_WR))
                    {
                        print_fxn("Socket shutdown failed\n");
                    }
                    else
                    {
                        print_fxn("Socket shutdown succeeded\n");
                    }
                }
            } // Exit MODSTATE_RUNNING
    
        } // Exit !MODSTATE_BROKEN
    
        print_fxn("QuickSend module broken. Task ending.\n");
    
        free(p_msg);
    
        return;
    }
    
    /*
     * Copyright (C) 2017 - 2018 Texas Instruments Incorporated - http://www.ti.com/
     *
     *
     *  Redistribution and use in source and binary forms, with or without
     *  modification, are permitted provided that the following conditions
     *  are met:
     *
     *    Redistributions of source code must retain the above copyright
     *    notice, this list of conditions and the following disclaimer.
     *
     *    Redistributions in binary form must reproduce the above copyright
     *    notice, this list of conditions and the following disclaimer in the
     *    documentation and/or other materials provided with the
     *    distribution.
     *
     *    Neither the name of Texas Instruments Incorporated nor the names of
     *    its contributors may be used to endorse or promote products derived
     *    from this software without specific prior written permission.
     *
     *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
     *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
     *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
     *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
     *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
     *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
     *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
     *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
     *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
     *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
     *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
     *
    */
    
    /* ========================================================================== */
    /*                             Include Files                                  */
    /* ========================================================================== */
    
    #include <stdio.h>
    #include <string.h>
    #include <stdlib.h>
    #include <xdc/std.h>
    #include <xdc/runtime/Error.h>
    #include <xdc/runtime/System.h>
    #include <ti/sysbios/BIOS.h>
    #include <ti/sysbios/knl/Task.h>
    
    
    
    #include <ti/csl/soc.h>
    #include <ti/csl/cslr_device.h>
    #include <ti/csl/soc/am572x/src/csl_device_xbar.h>
    
    #include <ti/ndk/inc/netmain.h>
    #include <ti/ndk/inc/stkmain.h>
    #include "ti/ndk/inc/os/osif.h"
    
    #include <ti/board/board.h>
    
    #include <ti/drv/uart/UART.h>
    #include <ti/drv/uart/UART_stdio.h>
    
    #include <ti/drv/emac/emac_drv.h>
    #include <ti/drv/emac/src/v4/emac_drv_v4.h>
    
    #ifdef NIMU_FTP_APP
    #include <ti/transport/ndk/nimu/example/ftpApp/ftpserver/ftpserver.h>
    #endif
    
    /* Custom modules */
    #include <MShipMsgr.h>
    #include <MsgGenerator.h>
    #include <QuickSend.h>
    
    extern char *LocalIPAddr;
    extern void app_stats(UArg arg0, UArg arg1);
    
    /* ========================================================================== */
    /*                             Macros                                  */
    /* ========================================================================== */
    /* Enable the below macro to have prints on the IO Console */
    
    //#define IO_CONSOLE
    
    #ifndef IO_CONSOLE
    #define NIMU_log                UART_printf
    #else
    #define NIMU_log                printf
    #endif
    
    #define MAX_TABLE_ENTRIES   3
    
    /* ========================================================================== */
    /*                            Global Variables                                */
    /* ========================================================================== */
    
    /**Task handle for EIP*/
    Task_Handle main_task;
    
    static int nimu_device_index = 0U;
    
    NIMU_DEVICE_TABLE_ENTRY NIMUDeviceTable[MAX_TABLE_ENTRIES];
    
    void TaskFxn(UArg a0, UArg a1);
    extern int CpswEmacInit (STKEVENT_Handle hEvent);
    
    void raw_eth_task(UArg a0, UArg a1);
    
    
    /**
     *  \name main
     *  \brief Main Function
     *  \param none
     *  \return none
     *
     */
    int main()
    {
        /* Call board init functions */
        Board_initCfg boardCfg;
        Task_Params taskParams;
    
        boardCfg =  BOARD_INIT_PINMUX_CONFIG |
            BOARD_INIT_MODULE_CLOCK | BOARD_INIT_UART_STDIO;
    
        Board_init(boardCfg);
    
        CSL_xbarMpuIrqConfigure(CSL_XBAR_INST_MPU_IRQ_92, CSL_XBAR_GMAC_SW_IRQ_RX_PULSE);
        CSL_xbarMpuIrqConfigure(CSL_XBAR_INST_MPU_IRQ_93, CSL_XBAR_GMAC_SW_IRQ_TX_PULSE);
    
        /* Select RGMII 2 ports GMIIx_SEL = 2 for RGMII*/
        CSL_FINS (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->CONTROL_IO_1,
              CONTROL_CORE_CONTROL_IO_1_GMII1_SEL, 2U);
        CSL_FINS (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->CONTROL_IO_1,
              CONTROL_CORE_CONTROL_IO_1_GMII2_SEL, 2U);
    
        /*GMAC RESET ISOLATION Enable*/
        CSL_FINS (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->CONTROL_IO_2,
              CONTROL_CORE_CONTROL_IO_2_GMAC_RESET_ISOLATION_ENABLE, 0U);
        CSL_FINS (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->CONTROL_IO_2,
              CONTROL_CORE_CONTROL_IO_2_GMAC_RESET_ISOLATION_ENABLE, 1U);
    
        EMAC_HwAttrs_V4 cfg;
        EMAC_socGetInitCfg(0, &cfg);
        cfg.port[0].phy_addr = GMAC_PORT1_ETHERNET_PHY_ADRESS;
        cfg.port[1].phy_addr = GMAC_PORT2_ETHERNET_PHY_ADRESS;
        EMAC_socSetInitCfg(0, &cfg);
    
        Task_Params_init(&taskParams);
        taskParams.priority = 1;
        taskParams.stackSize = 0x1400;
        main_task = Task_create (TaskFxn, &taskParams, NULL);
    
        NIMUDeviceTable[nimu_device_index++].init =  &CpswEmacInit;
    
        NIMUDeviceTable[nimu_device_index].init =  NULL;
    
        BIOS_start();
    
        return -1;
    }
    
    
    /**
     *  \name TaskFxn
     *  \brief Task Fxn
     *  \param a0
     *  \param a1
     *  \return none
     *
     */
    void TaskFxn(UArg a0, UArg a1)
    {
        NIMU_log("\n\rSYS/BIOS Ethernet/IP (CPSW) Sample application, EVM IP address I/F 1: %s\n\r", LocalIPAddr);
        Error_Block eb;
    
        //ftpserver_init();
        /*if (NULL == MShipMsgr_create(7, &eb))
        {
            // print Error_getMsg(eb);
        }
    
        if (NULL == MsgGenerator_create(6, &eb))
        {
            // print error
        }*/
    
        if (NULL == QuickSend_create(1, &eb))
        {
            // print error
        }
    }
    
    /*
     * ======== stackInitHook ========
     * hook called from ti_nkd_config_Global_stackThread() to run user setup code
     */
    void stackInitHook(void* hCfg)
    {
        NIMU_log("In stackInitHook");
    }
    
    0827.EthSend.cfg
    /**
     * References: https://docs.microsoft.com/en-us/windows/win32/winsock/getting-started-with-winsock
     *
     *
     */
    
    /* Custom includes */
    #include "../checksums/checksums.h"
    
    /* Windows includes */
    #include <winsock2.h>
    #include <ws2tcpip.h>
    #include <iphlpapi.h>
    
    /* Std */
    #include <stdio.h>
    #include <stdbool.h>
    
    #pragma comment(lib, "Ws2_32.lib")
    
    #define PRINT_LOOP_PERIOD 100000
    
    #define LISTEN_IP_ADDR "192.168.1.5"
    #define LISTEN_PORT 6500
    
    #define RECV_BUF_SIZE 11680
    
    //#define DO_CHECKSUM
    
    /* Function prototypes */
    void process_connection(int new_fd);
    void print_stats(uint32_t msgs_corr_cyc, uint32_t msgs_recvd_cyc, uint32_t msgs_corr_tot,
    		uint32_t msgs_recvd_tot);
    float calc_pct_corr(uint32_t msgs_corr, uint32_t msgs_recvd);
    
    int main(int argc, char * argv[])
    {
    	int sockfd, retval;
    	struct sockaddr_in my_addr;
    
    	/* From tutorial */
    	struct addrinfo *result = NULL,
    			*ptr = NULL,
    			hints;
    
    	WSADATA wsaData;
    	int iResult;
    
    	// Initialize Winsock
    	iResult = WSAStartup(MAKEWORD(2,2), &wsaData);
    	if (iResult != 0)
    	{
    		printf("WSAStartup failed: %d\n", iResult);
    		return 1;
    	}
    
    	sockfd = socket(PF_INET, SOCK_STREAM, 0);
    	if (-1 == sockfd)
    	{
    		perror("socket() error");
    		return -1;
    	}
    	printf("Socket created\n");
    
    	my_addr.sin_family = AF_INET;
    	my_addr.sin_port = htons(LISTEN_PORT);
    	my_addr.sin_addr.s_addr = inet_addr(LISTEN_IP_ADDR);
    	memset(my_addr.sin_zero, '\0', sizeof my_addr.sin_zero);
    
    	retval = bind(sockfd, (struct sockaddr *)&my_addr, sizeof my_addr);
    	if (-1 == retval)
    	{
    		perror("bind() error");
    		return -1;
    	}
    	printf("Bound\n");
    
    	retval = listen(sockfd, 1);
    	if (-1 == retval)
    	{
    		printf("listen() call failed with errno %d\n", errno);
    		return -1;
    	}
    	else
    	{
    		int new_fd;
    		//struct sockaddr_storage their_addr;
    		struct sockaddr their_addr;	
    		//socklen_t addr_size;
    		int addr_size;
    
    		printf("Listening on %s:%u\n", LISTEN_IP_ADDR, LISTEN_PORT);
    
    		while(1)
    		{
    			new_fd = accept(sockfd, (struct sockaddr *)&their_addr, &addr_size);
    			if (-1 == new_fd)
    			{
    				printf("accept() call failed with errno %d\n", errno);
    				return -1;
    			}
    			else
    			{
    				process_connection(new_fd);	
    			}
    		}
    	}
    
    	printf("Hello, world!\n");
    
    	return 0;
    }
    
    void process_connection(int new_fd)
    {
    	char recv_buf[RECV_BUF_SIZE];
    	int recv_bytes;
    	uint32_t msgs_recvd_tot = 0;
    	uint32_t msgs_corr_tot = 0;
    	uint32_t msgs_recvd_cyc = 0;
    	uint32_t msgs_corr_cyc = 0;
    	uint32_t loop_count = 0;
    	uint64_t bytes_recvd_cyc = 0;
    	uint64_t bytes_recvd_tot = 0;
    
    	printf("Accepted connection\n");
    
    	while(1)
    	{
    		loop_count++;
    
    		//recv_bytes = recv(new_fd, recv_buf, sizeof(recv_buf), 0);
    		recv_bytes = recv(new_fd, recv_buf, RECV_BUF_SIZE, 0);
    		if (0 < recv_bytes)
    		{
    			msgs_recvd_cyc++;
    			/* Succees! Test message now */
    #ifdef DO_CHECKSUM	
    			if (!checksum_correct(recv_buf, recv_bytes, 1))
    #else
    			if (0) // Check number of bytes instead?
    #endif
    			{
    			//	printf("CS incorrect: %d bytes\n", recv_bytes);
    			}
    			else
    			{
    				msgs_corr_cyc++;
    				bytes_recvd_cyc += recv_bytes;
    			}
    			//printf("Received %d bytes\n", recv_bytes);
    			if (0 == loop_count % PRINT_LOOP_PERIOD)
    			{
    				bytes_recvd_tot += bytes_recvd_cyc;
    				printf("Bytes recvd cyc: %f M tot: %f M\n", (float)bytes_recvd_cyc / 1e6, (float)bytes_recvd_tot / 1e6);
    				msgs_corr_tot += msgs_corr_cyc;
    				msgs_recvd_tot += msgs_recvd_cyc;
    				print_stats(msgs_corr_cyc, msgs_recvd_cyc, msgs_corr_tot,
    						msgs_recvd_tot);
    				msgs_corr_cyc = 0;
    				msgs_recvd_cyc = 0;
    				bytes_recvd_cyc = 0;
    			}
    		}
    		else if (-1 == recv_bytes)
    		{
    			printf("recv() failed\n");
    			//return -1;
    			return;
    		}	
    		else if (0 == recv_bytes)
    		{
    			printf("Connection closed\n");
    			break;
    		}
    	}
    	printf("Shutting down connection\n");
    	shutdown(new_fd, SD_BOTH);
    }
    
    float calc_pct_corr(uint32_t msgs_corr, uint32_t msgs_recvd)
    {
    	if (0 == msgs_corr || 0 == msgs_recvd)
    	{
    		return 0;
    	}
    
    	return 100.0 * (float) msgs_corr / (float) msgs_recvd;
    }
    
    void print_stats(uint32_t msgs_corr_cyc, uint32_t msgs_recvd_cyc, uint32_t msgs_corr_tot,
    		uint32_t msgs_recvd_tot)
    {
    	float pct_corr_cyc, pct_corr_tot;
    
    	pct_corr_cyc = calc_pct_corr(msgs_corr_cyc, msgs_recvd_cyc);
    	pct_corr_tot = calc_pct_corr(msgs_corr_tot, msgs_recvd_tot);
    
    	printf("Success: CYC: %f (%u/%u) TOT: %f (%u/%u)\n", 
    			pct_corr_cyc, msgs_corr_cyc, msgs_recvd_cyc,
    			pct_corr_tot, msgs_corr_tot, msgs_recvd_tot);	
    }
    

  • Hi Jon,

    Re-opening this thread after some time. Apologies.

    Is this issue still open ?

    Regards

    Vineet

  • Hi Vineet,

    I'm still having some throughput issues. I have an application sending at 30-50 Mbit/s, using roughly 22% Cpu, which runs well for a few minutes but then the throughput drops dramatically to ~700 kbit/s. It seems to be related to the AM572x dropping packets and taking a long time to recognize duplicate acks from the other machine. I'm seeing that its TCP congestion window tracking variable snd_cwnd is very low. Here's what I'm seeing in wireshark once the throughput drops: