AM57x NDK: Lots of TCP retransmissions when sending data to device

Hi!

Thanks for your reply.

No I haven't tried what you suggest. I have just used the default settings that is supplied by TI setup.

The FTP server is an "own" implementation that I have used on several projects. I have ported this to work on the BasicExamle for am57xx using the socket API just to elliminate any other possible issues. 

Of course you can have the ccs6 project, no problem. I have tested this implementation (using LWIP as TCPIP stack) and one using the GreenHill fusion TCPIP stack without.any problems.

I will send you the CCS6 project tomorrow.

Best Regards.

Per-Anders

Hi!

Here is the code that i'm using to test this. It's actually an modification to the NIMU_BasicExamle_evmAM572x project in the PDK. I tried what you suggested, but it didn't help so much.

I have been playing with the buffersizes in the send() and recv() calls and found out if I set DATA_BUFFER_SIZE to 512 bytes, the sending from evm works, but anything above the fails.

int32_t ftp_filerout_read(io_handler_t *ioh, char *path)
{
	uint32_t i;
	volatile uint32_t j;

	for (i=0; i<5000;i++)
	{
		send(ioh->data_socket, ioh->DataBuf, DATA_BUFFER_SIZE, 0);
		Task_yield();
	}

	return CSL_SOK;
}

int32_t ftp_filerout_write(io_handler_t *ioh, char *path)
{
    uint32_t bytesRead = 0;

	do
	{
		bytesRead = recv(ioh->data_socket, ioh->DataBuf, DATA_BUFFER_SIZE, 0);

		if (bytesRead > 0)
		{
			Task_yield();
		}
	} while (bytesRead > 0);

	return 0;
}

To get the receive to "work" I had to set the Tcp.receiveBufSize = 2048;, well it doesn't work so well :-( 

Global.memRawPageSize = 12288;
Ip.maxReassemblySize = 12288;
//Tcp.transmitBufSize = 4096;
Tcp.receiveBufSize = 2048;
//Tcp.receiveBufLimit = 2048;
Global.netSchedulerPri = Global.NC_PRIORITY_HIGH;

otherwise the ftp client on the PC-side times out. It's the same here DATA_BUFFER_SIZE > 512 doesnt work.

I Attach the CCS6 project so you can verify teh strange behaviour.

Username: user

Password: password

Use put <file on pc side> or get <dummy file from evm>

but you can see this in the code :-)

Hope you can help me to solve this issue, because the whole new project based on the am5716 depends on a working TCP communication.

BR:

Per-Anders Strand

PS: I'm using the following components:

NDK_2_25_00_09

xdctools_3_32_00_06_core

bios_6_45_01_29

and CCS6 (6.1.2)

3482.NIMU_BasicExample_evmAM572x_armExampleproject.7z

Per-Anders,

Were you able to get negotiated connection FullDuplex "1000Mbps" during the application start-up? Do you connect the PC and EVM directly or through a router?

I could reproduce the TCP re-transmission issue, but with different size of packet (>4KB), will continue to investigate it as well as the thread you opened: .

Regards, Garrett

Hi!

Thanks for your cooperation :-).

I have tested both with a 'switch' between the EVM and PC and with a direct connection.

One thing about the connection (it's not a big issue yet), but we can't establish a working connection if we attach the EVM to a 100Mb switch. It says 'connected at 100Mbit' but we can't ping the EVMam57xx or the IDK437x.

If we just remove the ethernet cable and connect it to a 1000Mb switch, no reboot of the EVM, it says connected at 1000Mb and everything works fine. This is done with the BasicExample project also.

This might be an issue futher on, but not at the moment.

Yes, everything larger the 512 bytes. What I have seen is that it seems to 'ACK' the wrong package. I'm not that good at TCPIP stacks or analyzing Wireshark logs, but I slowly learns, so I can be wrong about that.

I have previous worked with LWIP since some years back. 

I used LWIP our previous project. We used the 'TMS570LS3137' in that one.

BR:

Per-Anders

Hi!

Have you found anything yet? Do you need any more information from me to continue?

BR:

Per-Anders

Hi!

Any feedback yet? This will soon be a quite complicated issue for us because the whole project is dependent on a fast and stable TCPIP connection.

BR:

Per-Anders

I want to inform that our apps team was able to reproduce the issue and they are working on it. More comments will be posted here soon.

Per-Anders,

The TCP re-transmission appears to be addressed after updating the followings in nimu_evm.cfg. Would you try the same on your system and check if any FTP connection and throughput behavior changes?

       Mmu.initDescAttrsMeta(attrs1);

       attrs1.type = Mmu.DescriptorType_BLOCK;    // BLOCK descriptor

       attrs1.shareable = 2;                  // sharerable

        attrs1.attrIndx = 1;                       // non-Cached, normal memory, original value 2

Regards, Garrett

Hi!

I just made some basic test with this change and saw that the thoughput  goes down from about 46000kbyte/second to ~1300kbyte/second doing a GET xxxx with the same buffersize (512 bytes).

This is NOT so good. Also the PUT command was realy slow, about the same speed (1300 kbyte/second) with the change you suggested

I also tested to remove the "Tcp.receiveBufSize = 2048;" line I added and then I can't do a PUT "filename". Alot of errors in the wireshark log.

Please try yourself and remove this line.

I will do more tests tomorrow because it's quater to eleven here in Sweden :-)

Do we have any other way to discuss this issue outside the E2E forum, that might speed things up?

BR:

Per-Anders Strand

Per-Anders,

I removed 'Tcp.receiveBufSize=2048' and used the default size 8192, but was still able to PUT, would you post your wireshark log here?

We are actively debugging the cacheability issue related to the low throughput rate...

Regards, Garrett

Hi!

This is a snapshot of my *.cfg file

// descriptor attribute structure
var attrs1 = new Mmu.DescriptorAttrs();

Mmu.initDescAttrsMeta(attrs1);
attrs1.type = Mmu.DescriptorType_BLOCK;    // BLOCK descriptor
attrs1.shareable = 2;                      // sharerable
attrs1.attrIndx = 1;                       // Cached, normal memory

// Set the descriptor for each entry in the address range
for (var i=0x80000000; i < 0xA0000000; i = i + 0x00200000) {
    // Each 'BLOCK' descriptor entry spans a 2MB address range
    Mmu.setSecondLevelDescMeta(i, i, attrs1);
}

Program.sectMap["BOARD_IO_DELAY_DATA"] = "OCMC_RAM1";
Program.sectMap["BOARD_IO_DELAY_CODE"] = "OCMC_RAM1";
Global.memRawPageSize = 12288;
Ip.maxReassemblySize = 12288;
//Tcp.transmitBufSize = 4096;
//Tcp.receiveBufSize = 2048;
//Tcp.receiveBufLimit = 2048;
Global.netSchedulerPri = Global.NC_PRIORITY_HIGH;

I try to do a PUT from the PC and I get alot of retransmissions/errors. The file is about 2,5 Mb large.

I attach the wireshark-log and screendump.

The GET (receive data from EVM->PC) seems to work with larger buffersize then 512, but with a bad throughput.ndk-test-1.zip

Per-Anders,

Yes, we also observe the TCP re-transmission if setting TCP buffer size to default 8192 and transferring a large size data as what you tried, thought 'A lot of errors' you described is a new issue after cache is disabled. We have been looking into the cache operation to packet buffers/descriptors, and modifying packet receiving ISR but don't have conclusion yet at this point, will update you as soon as we can.

Regards, Garrett

Hi Per-Anders,

PROCESSOR-SDK-RTOS-AM57X 03_00_00_04 is now available at

The release enhances the EMAC driver and NIMU transport, resolves the FTP put TCP-retrainsmission and FTP get hang issue. We have tested it with your FTP server application on AM572x PG2.0 GP-EVM. To check the PG version:

uint32_t pgVersion;

pgVersion = (HW_RD_REG32(CSL_MPU_CTRL_MODULE_WKUP_CORE_REGISTERS_REGS + CTRL_WKUP_ID_CODE) & 0xf0000000) >> 28;

pgVersion should return 1 or 2. You can also check if your board is revision A3 (printed on white sticker).

• To get optimal throughput, we set the TCP Tx/Rx buffer size as followings:

Global.pktNumFrameBufs = 384;

Tcp.transmitBufSize = 8192;

Tcp.receiveBufSize = 65536;

Tcp.receiveBufLimit = 65536;

• In FTP server application side, you need to update the func_port() and func_retr() in ftp_commands.c as below:

static int32_t func_port (io_handler_t *ioh)

{

......

if (!check_authentificated (ioh)) return -1;

if (ioh->data_socket) fdClose(ioh->data_socket);

ioh->data_socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);

......

}

static int32_t func_retr (io_handler_t *ioh)

{

...

if (ftp_filerout_read (ioh, filename) == CSL_SOK)

{

sprintf(ioh->send_buffer,"226 File transfer completed... \r\n");

}

else

{

sprintf(ioh->send_buffer,"450 Requested file action not taken\r\n");

}

bytes = send(ioh->socket, ioh->send_buffer, strlen(ioh->send_buffer), 0);

Task_sleep(1);

fdClose(ioh->data_socket);

......

}

Note: In case you experience unstable connection on ETH0 (only), try to adjust the IO delay in pdk_am57xx_1_0_3\packages\ti\drv\emac\src\v4\emac_cpsw.c as the followings (4 line _ENETPHY_UserAccessWrite), and add the file to your CCS project and re-build.

static void EMAC_cpgmacMacOpen(Cpsw3gPort *hPort)

{

......

  /* set the Phy to given mode based on Config Params */

   EMAC_cpswSetPhyMode(gmacObject,instId);

   _ENETPHY_UserAccessWrite(hPhyDev, (0xD), cpsw3gCfg->baseConfig.port[instId].phy_addr, 0x2);

   _ENETPHY_UserAccessWrite(hPhyDev, (0xE), cpsw3gCfg->baseConfig.port[instId].phy_addr, 0x8);

   _ENETPHY_UserAccessWrite(hPhyDev, (0xD), cpsw3gCfg->baseConfig.port[instId].phy_addr, 0x4002);

   _ENETPHY_UserAccessWrite(hPhyDev, (0xE), cpsw3gCfg->baseConfig.port[instId].phy_addr, (0x18 << 5) | 0x18);

   /* Finally Set the Mac Control register. Enable MII */

   CSL_CPGMAC_SL_enableGMII((CSL_cpgmacSlHandle) cpsw3gCfg->baseConfig.port[instId].sliver_base);

}

Regards, Garrett

Hi, I will test this ASP. I´m on vacation right now, but I will try to do this later this week. I just downloaded the new SDK and made a quick "diff". I will come back to you later this week.

BR:

Per-Anders

Hi!

Sorry for the late answer. I started to work with this during the summer period, but was interrupted because of a license issue with CCS.

But back to some status. I have tried to get the new PDK working with the EVM for am572x, but without any success.

When I use the code I sent to you (ftp server) based on pdk 1_0_1, it works as before, but with the new PDK i don't get any communication, not even a ping result.

I have downloaded and started the NIMU_BasicExample_evmAM572x (unmodified, original project), I get the following console output:

[CortexA15_0] Using MAC Address: 47-d7-0c-74-da-ea

Using MAC Address: 47-d7-0c-74-da-ea

Network Added: If-1:192.168.1.4

If I try to ping this IP, nothing happens and no trace i wireshark.

One thing I have noticed is that only the upper RJ45 connector seems to work. No light on the lower connector if I connect the ethernet cable.

Another thing is that the version number on the sticker is (A2) on our board. You mention A3 in your earlier post. Could this make a differens?

I also put some breakpoints in EMAC_cpswHwIntRx/Tx, but they are never trigged.

Using NDK_2_25_00_09

Do you have any suggestions?

Hi Per-Anders Strand,

From the log '[CortexA15_0] Using MAC Address: 47-d7-0c-74-da-ea', it appears the function EVMMACAddrGet() doesn't read the MAC address properly. In ti/drv/emac/src/v4/emac_cpsw.c, the function EMAC_cpswOpen() checks 'MAC address must be supplied and not a multicast address', can you please try to update the EVMMACAddrGet() in main_evmAM572x.c per the attached macAddrGet.c? Revision A2 Board doesn't make difference and should be able to pass a basic PING test.

/* ========================================================================== */ 		
/*                          Function Definitions                              */ 		
/* ========================================================================== */ 		
		
/** 		
 * \brief   This function returns the MAC address for the EVM 		
 * 		
 * \param   addrIdx    the MAC address index. 		
 * \param   macAddr    the Pointer where the MAC address shall be stored 		
 *     'addrIdx' can be either 1 or 2 		
 * 		
 * \return  0 on sucess, -1 on error. 		
*/ 		
		
int32_t EVMMACAddrGet(uint32_t addrIdx, uint8_t *macAddr) 		
{ 		
    int32_t retVal = 0; 		
    switch(addrIdx) 		
    { 		
        case 1U: 		
            macAddr[0U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_1 		
                      >> 16U) & 0xFFU; 		
            macAddr[1U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_1 		
                      >> 8U) & 0xFFU; 		
            macAddr[2U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_1) 		
                      & 0xFFU; 		
            macAddr[3U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_0 		
                      >> 16U) & 0xFFU; 		
            macAddr[4U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_0 		
                      >> 8U) & 0xFFU; 		
            macAddr[5U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_0) 		
                      & 0xFF; 		
        break; 		
		
        case 2U: 		
            macAddr[0U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_3 		
                      >> 16U) & 0xFFU; 		
            macAddr[1U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_3 		
                      >> 8U) & 0xFFU; 		
            macAddr[2U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_3) 		
                      & 0xFFU; 		
            macAddr[3U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_2 		
                      >> 16U) & 0xFFU; 		
            macAddr[4U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_2 		
                      >> 8U) & 0xFFU; 		
            macAddr[5U] =  (((CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS)->MAC_ID_SW_2) 		
                      & 0xFF; 		
            break; 		
		
        default: 		
            retVal = -1; 		
            break; 		
    } 		
    return retVal; 		
} 		

Hi Garrett

Yes ofcourse, I should have checked this. I have debugged this previously och made changes in the other PDK:s. The PING works as it should now and I will test the complete application today. I will come back to you with the result.

Thank you for your fast reply :-)

Best Regard

Per-Anders

Hi!

I have made some testings last week and it seems to work fine. There is still one open issue. If we try to connect the EVM to a 100 Mbs Switch/PC, we don't get any connection. It looks like it's connected, no errors. The same behavour was observed with the earlier PDK:s (1.0.1/1.0.2). 

The output on the serial port says:

Connected to a 100 Mbs

SYS/BIOS Ethernet/IP (CPSW) Sample application
ENETPHY_FindingState: PhyNum: 1
ENETPHY_FindingState: PhyNum: 2
ENETPHY_DisablePhy(1)
Enable Phy to negotiate external connection
NWAY Advertising: FullDuplex-1000 FullDuplex-100 HalfDuplex-100 FullDuplex-10 HalfDuplex-10
ENETPHY_DisablePhy(2)
Enable Phy to negotiate external connection
NWAY Advertising: FullDuplex-1000 FullDuplex-100 HalfDuplex-100 FullDuplex-10 HalfDuplex-10
Phy: 2, NegMode 01e1, NWAYadvertise 01e1, NWAYREadvertise 45e1
Negotiated connection: FullDuplex 100 Mbs
Phy: 1, NegMode 01e1, NWAYadvertise 01e1, NWAYREadvertise 45e1
Negotiated connection: FullDuplex 100 Mbs

Here I change to a Gigabit interface and everything works fine.
Negotiated connection: FullDuplex 1000 Mbs
SetPhyMode:000021e1 Auto:1, FD10:64, HD10:32, FD100:256, HD100:128, FD1000:8192 LPBK:0
SetPhyMode:000021e1 Auto:1, FD10:64, HD10:32, FD100:256, HD100:128, FD1000:8192 LPBK:0

Is this a known bug? Are there any workarounds for this issue?

BR:

Per-Anders Strand

Hi Per-Anders,

Thanks for the update. Yes, the 100Mbps ethernet connection is a known issue and have been addressed, will be published in next PSDK release. The fix is around the function EMAC_cpswUpdatePhyStatus() in drv\emac\src\v4\emac_cpsw.c, which does not handle the case where 100 Mbps link is established between EVM and PC.

Regards,
Garrett

#if defined  (SOC_AM572x) && defined(__ARM_ARCH_7A__)
                CSL_a15InvDataCache((void*)pPktNew->pDataBuffer, pPktNew->BufferLen);
                CSL_a15WbInvDataCache((void*)pPktNew, sizeof(EMAC_PKT_DESC_T));
#endif

--- should be (I think)

#if defined  (SOC_AM572x) ||  defined(SOC_AM571x)
#ifdef __ARM_ARCH_7A__
                CSL_a15InvDataCache((void*)pPktNew->pDataBuffer, pPktNew->BufferLen);
                CSL_a15WbInvDataCache((void*)pPktNew, sizeof(EMAC_PKT_DESC_T));
#endif
#endif

Hi!

I have still had problems with the pdk 1.0.3 lately running on our HW. The difference from the evmAM572x is that we are using am571x.

The testing has been delayed because I was waiting on new HW that uses the correct ethernet port (port 0) that can handle Gb ethernet.

When I started testing on the new HW I still had the problem with retransmissions.

The test-application is running fine on the evm, but not on our new HW.

This week I sat down and started debugging the problem and found the following in the emac_cpsw.c.

In the function EMAC_dequeueRx() the cache is not handled correctly for SOC_AM571x (I think)

Can you verify that this change I made (on line 721, addes SOC_AM571x in the #if defined) is correct and if so, 

see that this will be added in the next release if PDK..

The system seems to work fine after this change (with SOC_AM571x)

BR:

Per-Anders Strand

Hi Per-Anders

Your change for the data buffer and descriptor cache operation on SOC_AM571x with pdk_am57xx_1_0_3 is correct. The issue has been addressed in next PDK release, where we update the CSL cache functions to common osal functions e.g. EMAC_osalCacheInv(), so no need to differentiate devices as in the release you are working on.

Thanks,
Garrett

Hi!

Thanks for your reply.

Do you know when this release is scheduled to be available?

I have encounted another problem. When I have a lot of opened sockets receiving and transmitting data, sometimes the transmission stops working. Data is received and processed correctly in the NDK what I can see.

When this happens, the dmaInProgress flag is set to "1" and stays there. This is flag checked in the EMAC_enqueueTx() function.

I have followed into the EMAC_sendPacket() where it tries to enqueue the TX packet

Only a reboot solves the problem :-)

Is this something you are aware of?

BR:

Hi!

This is a large "test-system" that I can't run on the EVM. I will try to reproduce this om the EVM, but it might take a week or so before I have time to port some needed modules to the "test" project (the one that I used to reproduce the earlier TCP retransmission problem).

I will come back to you as soom I hame more info.

BR:

Hi again!

I have run into another NDK problem. It seems that if we are calling send() in a very tight loop, the NDK will crash in the PBMQ_enq() function.

The Pq->pTail->pNext pointer is NULL.

I have tracked this down (and think) it happens when PBMQ_free list runs out of buffers. I've put some code in the PBMQ_deq () function to break if the

PBMQ_free.pHead and PBMQ_free.pTail is to be set to NULL. The PBMQ_free.Count is in this case set to "1" (Last free item ?).

I attach some screendumps of memory views and stack traces.

I hope this can make it easier for you to understand the problem.

If I make a Task_sleep(1) between the send() calls, this never happens.

BR:

Per-AndersPBMQ problems.zip

Yes, of course'

/PAS