Intermittent "No Tx free descriptor" error after integrating EMAC example project

Chris,
It seems, you are modifying the PA example code according your requirement.
It is not simple to get your query and give the technical suggestion as per your experient.
Is it possible provide your modified code and testing scenario? So that the expert will provide detailed information for your query.

Pubesh,

I was able to cut out all of our software and was left with only the network I/O portion which is a modification of the original EMAC multicore example project in the PDK. The problem still occurs with this project. 

I have attached both the project and the script (loader.sh) that I use to run the project. I place the script and the .out file in the dsp_utils binary directory inside the TI Desktop Linux SDK and run it with the following command:

while true; do ./loader.sh 1 8 0 C66xx_RTAF_SYSBIOS_CCSv54.out ../../scripts/initcfg.txt; done

It can also be ran without an infinite loop:

./loader.sh 1 8 0 C66xx_RTAF_SYSBIOS_CCSv54.out ../../scripts/initcfg.txt

Running it in an infinite loop seems to make the problem occur quicker. 

The script usage is as follows:

Usage: ./loader.sh <number_of_chips> <number_of_cores> <boot_entry_address> <image_file> <config_file>

The script will use the dspallreset.sh and init_dsps.sh scripts in the scripts directory in the TI Desktop Linux SDK then use the dsp_utils load command to get the .out file running on the cores. I have only tested this on one chip as shown in the above example commands.

The project source can be modified to change the packet contents if needed, but capturing the packets on another device on the same LAN as the PCIe card is not necessary to reproduce the problem.

Chris

3056.tx_desc_debug.rar

I can repeat this test many times and it seems to be mostly reliable except for the occasional missed packets on the capture interface, but this doesn't seem to be an issue with the C66x side.

Titus S,

Thanks for looking into this issue for us.

I didn't see any issues with testing the internal loop back code, but I did not extensively test this. I will do some more testing with that mode today. At this point, I have further simplified the project that I previously attached. I can not find a difference between that project and what I had working from the PA_multicore_example, but I am not able to reproduce the problem using the modified PA_multicore_example. I even ran that in a loop with the TI SDK overnight.

The capture interface I am using is Wireshark. I don't think there's anything to look into there. I think the issue with occasionally missing packets is just due to the rate of packets and the hardware I am using or other congestion on the network (In some tests I am sending packets between public IPs instead of within our LAN).

Regards,
Chris

Titus, Pubesh-

Thanks for assisting.  If we need to get to you a DSPC-8681E card please let us know and we can coordinate that.

At this point, if you guys can simply reproduce the problem on the 6678 EVM and/or DSPC-8681E with Chris' minimal project code, and if you don't think it's a C66x chip level issue, give us your best guess at where to look or how to debug, that would be great.  We can then get Advantech guys and our customers who are using high performance network I/O more involved.

Some update notes:

1) We have tried power cycling the server, and a full POR does not help.

2) If we disconnect the RJ-45 on the card, we don't see the problem.  It might have something to do with Broadcom PHY initialization or reset, or flow control from PHY to EMAC0 to PA.

Thanks.

-Jeff
 Signalogic

I have further simplified the project that I previously attached. I can not find a difference between that project and what I had working from the PA_multicore_example, but I am not able to reproduce the problem using the modified PA_multicore_example. I even ran that in a loop with the TI SDK overnight.

Titus,

After more closely comparing the example multicore project with my own project, I finally found some differences in the compiler settings that seemed to be the reasons for the different behavior I was seeing between the two projects. It seems that for the cpsw_mgmt.c file specifically, I need to use the compiler options --opt_level=off and --symdebug:dwarf. The rest of the project uses --opt_level=3 and --symdebug:skeletal.

The simplest case that I have found to reproduce this problem is this:
1. Start with clean PA_multicore_example project
2. Change NUM_CORES defined in multicore_example.h to 1

#define         NUM_CORES      1

3. Change MAX_NUM_PACKETS defined in multicore_example.c to 300u

#define                     MAX_NUM_PACKETS                         300u

4. Change cpswLpbkMode declared/initialized in multicore_example.c to CPSW_LOOPBACK_NONE

Int cpswLpbkMode = CPSW_LOOPBACK_NONE;

5. Set project optimization level to 3 (--opt_level=3)

With these few changes, I am able to reproduce the problem using the previously sent loader.sh script ran in an infinite shell loop. If I then change the optimization level just for the cpsw_mgmt.c file to off (--opt_level=off), the problem does not occur.

I am not sure if 300 packets is the minimum needed to reproduce the problem, but I have tested at values of 10 and 100 and didn't see the problem even after 100+ iterations.

I have also tested with adding a 10ms delay after SendPacket() is called in multicore_example.c, but this did not make a difference.

Even with these test results, I do not think that the optimization level was actually causing the problem, but somehow removing any optimization for cpsw_mgmt.c helps avoid the problem. This is because in one of our full projects that we are using this network I/O code in, simply changing the optimization levels for the cpsw_mgmt.c file did not help. We still need to find the root cause of this problem.

Please let me know if you are able to reproduce the problem or if you have any suggestions on what to test further or possibly how to better debug this problem.

Titusrathinaraj Stalin said:

Have you tried to increase the "NUM_HOST_DESC" to 512 in "net_io.h" ?

When I try this in the original PA_multicore_example, I see this from core 0:

************************************************
*** PA Multi Core Example Started on Core 0 ***
************************************************
Initializing Free Descriptors.
QMSS successfully initialized
CPPI successfully initialized
PASS successfully initialized
Ethernet subsystem successfully initialized
Error allocating Tx free descriptors
Tx setup failed

Regards,
Chris

Initializing Free Descriptors.
QMSS successfully initialized
CPPI successfully initialized
PASS successfully initialized
Ethernet subsystem successfully initialized
Error allocating Tx free descriptors
Tx setup failed

I have some more info on this problem:

The problem seems to be related to other network traffic. If I use tcpreplay to blast a large number of packets at a high speed on the LAN, then the problem occurs instantaneously. This seems to be in line with what Jeff had mentioned where if we initially disconnect the cable running to the card and reconnect it after the C66x code has finished initialization and has already sent some packets then we don't see the problem.

Chris Johnson1 said:

3. Change MAX_NUM_PACKETS defined in multicore_example.c to 300u

#define                     MAX_NUM_PACKETS                         300u

Chris Johnson1 said:

I am not sure if 300 packets is the minimum needed to reproduce the problem, but I have tested at values of 10 and 100 and didn't see the problem even after 100+ iterations.

With MAX_NUM_PACKETS set to the default value of 10u and the aforementioned method of blasting a larger number of packets at a high rate on the network, I am able to reproduce the problem.

Also, as a note, the change to setting NUM_CORES to 1 was done to simplify the test and to avoid having to use any extra code to get the other cores started since I am loading/running over PCIe with the TI SDK. 

Regards,
Chris

Titus,

I do have a Shannon EVM C6678 board, but I don't currently have access to a JTAG Debugger. I can try to get a hold of one, but this will take some time on our part to locate one.

I have attached the modified project that is modified from the original project with the few steps I laid out previously.

I noticed that in the case of blasting a large number of packets at a high rate, the problem occurs even with the original unmodified PA_multicore_example project. In this case there isn't a "No Tx free descriptor" error message, but the dspresetall.sh script in the TI SDK will fail with the same aforementioned pscstate error messages. It seems like network traffic can have a negative effect when initializing the network I/O portion of the chip. 

Were you able to reproduce the problem at all on the EVM or PCIe card with the previous project that I sent?

If you are able to reproduce the problem on the PCIe card, but not the EVM, then we can get Advantech more involved in debugging this issue. 

Also, if you are trying to reproduce this problem on the EVM board, it will be better if you use a soft reset similar to what takes place with the TI SDK for the PCIe card. I think the problem is less likely to occur (if it will occur at all) when a hard reset takes place.

Note: I am blasting a large number of packets at a high rate using tcpreplay with the following command:

tcpreplay -i Auto_eth0 --unique-ip --loop 2 bigFlows_bcast.pcap

The .pcap file that I use can be found here: 

I used tcprewrite to change all destination MAC addresses in the original pcap file to broadcast MAC.

Regards,
Chris

pdk_C6678_1_1_2_6.rar

Titus-

Chris mentions that he can reproduce the issue with the TI unmodified "PA_multicore_example" project, using tcpreplay to simulate network congestion.

Can you try this on your EVM board and PCIe card?  It should work on both unmodified (i.e. TI distribution code, no changes), since the example is using internal loopback.  It's unclear how external congestion can affect this example.

Thanks.

-Jeff

Hi Jeff and Chris

Chris mentions that he can reproduce the issue with the TI unmodified "PA_multicore_example" project, using tcpreplay to simulate network congestion.

Can you try this on your EVM board and PCIe card?  It should work on both unmodified (i.e. TI distribution code, no changes), since the example is using internal loopback.  It's unclear how external congestion can affect this example.

Jeff, its not possible.

I've tried it but I'm not getting any problem on Shannon EVM (C66748).

I think that Chris might have used "CPSW_LOOPBACK_NONE".

Chris, Is that right ?

Else you were facing the problem even with TI's example (unmodified "PA_multicore_example" project") and when you broadcasting the large number of packets at a high rate using "tcpreplay"  as like Jeff said ?

Note that I'm using C66748 Shannon EVM with XDS560 emulator and using "System Reset" (ctrl+shift+S)  option in CCS to reset the board.

One time, I'm able to see the "No Tx free descriptor. Cant run send/rcv test" error when I enabled "CPSW_LOOPBACK_NONE"

Also I used tcprewrite to change all destination and source (machine IP) MAC addresses in the original pcap file to broadcast MAC and SRC and DSP IP addresses.

Still I'm trying to reproduce the issue with C6678 Shannon EVM by enabling "CPSW_LOOPBACK_NONE" and broadcasting the large number of packets at a high rate using "tcpreplay".

I'm not at all facing any problem with "CPSW_LOOPBACK_INTERNAL"

Also I'm not able to work with DSPC8681E board since this board is new to me.

I've used your loader script to load the .out to DSP.

./loader.sh 1 4 0x0084f6e0 pa.out initcfg_1000.txt

Where, 0x0084f6e0 this is the ENTRY point taken from pa.map file.

I've attached the log that tried in DSPC8681E board and not able to get any console log to check the error.

Could you please tell to me where I can get the console log if you knows.

Note that I've checked the onboard COM1 (UART) port for console prints but got nothing so I'm not able to check with DSPC board.

titus@titus-desktop:~/ti/Titus_PA_issue$ sudo ./loader.sh 1 4 0x0084f6e0 pa.out initcfg_1000.txt
[sudo] password for titus:
Num of devices 4

 Iterations waited for entry point to clear 1
Dsp 0:  DSP Reset success !

 Iterations waited for entry point to clear 1
Dsp 1:  DSP Reset success !

 Iterations waited for entry point to clear 1
Dsp 2:  DSP Reset success !

 Iterations waited for entry point to clear 1
Dsp 3:  DSP Reset success !
Number of devices: 4
Device : 0 : dspc8681

boot config file initcfg_1000.txt
 DSP boot config addr 0x86ff00
 DSP boot config size in bytes 8
Boot config words: 0xbabeface,
Boot config words: 0x14,
 Overriding image entry point with input 860000 Download image success !
Device : 1 : dspc8681

boot config file initcfg_1000.txt
 DSP boot config addr 0x86ff00
 DSP boot config size in bytes 8
Boot config words: 0xbabeface,
Boot config words: 0x14,
 Overriding image entry point with input 860000 Download image success !
Device : 2 : dspc8681

boot config file initcfg_1000.txt
 DSP boot config addr 0x86ff00
 DSP boot config size in bytes 8
Boot config words: 0xbabeface,
Boot config words: 0x14,
 Overriding image entry point with input 860000 Download image success !
Device : 3 : dspc8681

boot config file initcfg_1000.txt
 DSP boot config addr 0x86ff00
 DSP boot config size in bytes 8
Boot config words: 0xbabeface,
Boot config words: 0x14,
 Overriding image entry point with input 860000 Download image success !
Load to chip 0, core 1

boot config file initcfg_1000.txt
 DSP boot config addr 0x86ff00
 DSP boot config size in bytes 8
Boot config words: 0xbabeface,
Boot config words: 0x14,
 Overriding image entry point with input 1184f6e0 Download image success !
Load to chip 0, core 2

boot config file initcfg_1000.txt
 DSP boot config addr 0x86ff00
 DSP boot config size in bytes 8
Boot config words: 0xbabeface,
Boot config words: 0x14,
 Overriding image entry point with input 1284f6e0 Download image success !
Load to chip 0, core 3

boot config file initcfg_1000.txt
 DSP boot config addr 0x86ff00
 DSP boot config size in bytes 8
Boot config words: 0xbabeface,
Boot config words: 0x14,
 Overriding image entry point with input 1384f6e0 Download image success !
Load to chip 0, core 0

boot config file initcfg_1000.txt
 DSP boot config addr 0x86ff00
 DSP boot config size in bytes 8
Boot config words: 0xbabeface,
Boot config words: 0x14,
 Overriding image entry point with input 1084f6e0 Download image success !
titus@titus-desktop:~/ti/Titus_PA_issue$

Titus,

Titusrathinaraj Stalin said:

I think that Chris might have used "CPSW_LOOPBACK_NONE".

Chris, Is that right ?

In the modified version where I send 300 packets and am not using tcpreplay to create network congestion, I am using CPSW_LOOPBACK_NONE. In the other case where I use tcpreplay to create network congestion, I am using the original project unmodified which is CPSW_LOOPBACK_INTERNAL.

Titusrathinaraj Stalin said:

Could you please tell to me where I can get the console log if you knows.

To get the console log on a PCIe card, I modify the .cfg file to use a SysMin buffer instead of SysStd for logging, and save the memory location used by the SysMin buffer. I don't think the dsp_utils app in the TI SDK has the capability to do this so I use Advantech's dsp_loader app with the savebinary command instead.

/* var SysStd   = xdc.useModule('xdc.runtime.SysStd');
System.SupportProxy = SysStd; */
var SysMin = xdc.useModule('xdc.runtime.SysMin');
SysMin.bufSize = 10240; 

The buffer will have the symbol name: "xdc_runtime_SysMin_Module_State_0_outbuf__A" 

I don't usually check the console log to determine that the error occurred. I look for the reset to fail on the next iteration of the test. 

Regards,
Chris

Titus,

I don't have any update from my end. I've been trying to find a workaround using the PHY's registers, but haven't had any luck yet.

I tried the sequence that you mentioned and I was able to reproduce the problem. That sequence is actually a test sequence that I have already tried. I am using a modified version of the bigFlows.pcap file with the destination MAC address for all packets changed to broadcast.

Note: Previously when I mentioned using the original unmodified PA_multicore_example project, I have actually changed the NUM_CORES define to 1. This is the only change I have made and is necessary to avoid having to add additional code for core 0 to start cores 1-7 since I am loading over PCIe.

Regards,
Chris

Titus-

I would like to add to Chris' update.  It's good to keep in mind that the problem can be avoided by simply disconnecting the Shannon EVM RJ-45 cable during EMAC / PA initialization, and then re-connecting.  For example, disconnect, run C6678 code (wait a couple of seconds), reconnect.  This works regardless of the level of network congestion.

This seems to imply a flow control issue between PHY and 6678 during initialization.  To achieve a work-around, I think the question might be:  how to make the PHY "appear to be" disconnected?  Internal loopback?  Other method? 

Thanks.

-Jeff
Signalogic

Titus-

Can you give some suggestions on this?  What register(s) to set in the PHY to simulate "no cable connected"?  Thanks.

-Jeff

Titus, Pubesh-

This is becoming a critical thing for us.  One of our customers is considering to use an Enet switch with remote control power, so they can "programmatically disconnect" the card's RJ-45 during initialization.  As you can guess they're not happy with that.

Here is some more info, plus related questions:

1) After a server reboot, which resets the PCIe card and C6678, it appears that PA init can sometimes succeed one time regardless of network traffic / congestion.  After that, successive C6678 soft reboots and PA inits can fail depending on network traffic (as documented in this thread).

2) Given this, we have tried the following "shutdown sequence":

  -disable packet output, delay for 1 sec
  -SGMII and PHY reset, delay for 1 sec
  -soft reset C6678

and that did seem to help on some systems (reduce the init failure rate), but not on all systems / networks.  Can you suggest additional shutdown steps or sequence?

3) The issue may at least be somewhat dependent on variation in C6678 devices.  We got a new shipment of PCIe cards, and found one card that was somewhat more resilient than others.  (This was a 1.25 GHz card, but other 1.25 GHz cards from the same batch did not show any improvement).  My understanding is there have been no errata changes on 6678 for some time, so what would you suggest that we look for or measure?   Anything at all that you want us to probe, please let us know.

4) Have you tried programming the PHY in loopback or other modes that might simulate an RJ-45 disconnected condition?  Was there any difference in the problem with this?  Although the problem has been demonstrated in conjunction with both Marvell and Broadcom PHYs, and we don't think it's a PHY problem, any suggestions you can give us for PHY programming we're happy to try.

Thanks.

-Jeff

Hi Jeff,

I have taken a look at this issue, and it seems that the switch port does not know how to handle the large influx of packets prior to the device execution. I’m going off the example in this thread where tcpreplay is sending the packets specifically to the C6678 Eth port, and using the PA_multicoreExample in CPSW_LOOPBACK_NONE; The screenshot below was taken with Core0 connected, code loaded (prior to running) and tcpreplay running.

Questions:

-          Is this what you also observe?
-          Are you using a GEL to initialize?
-          I’m not sure what version of the C6678 GEL you are using, but is there a configSGMIISerdes() step in the sequence?

Additionally I have been thinking of ways to work around this problem. Essentially find a way to reset the SGMII just before transmitting the packets. The thought here is to somehow clear the switch status by doing the SGMII re-init. Please see if the below suggestion helps.

1) In cpsw_mgmt.c (around line 134) Please add a routine that basically does the configSGMIISerdes in C code instead of the GEL. You can comment out this step in the GEL if it is being done there. You will now be doing it in the code instead as described below.

if (!cpswSimTest)

    { do

       {
       Init_SGMII_SERDES(); //SGMII config SERDES routine
      CSL_SGMII_getStatus(macPortNum, &sgmiiStatus);
     } while (sgmiiStatus.bIsLinkUp != 1);

 2) Place the Init_SGMII_SERDES function in somewhere in cpsw_mgmt.c. Its essentially the same routine as in the GEL. The changeRegister function is a quick way to poke the relevant registers without much hassle.

 void changeRegister( unsigned int x, unsigned int val)

{
unsigned int *p5 ;
p5 = (unsigned int *) x ;
*p5 = (unsigned int) val ;
}

int Init_SGMII_SERDES(void)
{
printf("Initialzing SGMII SERDES...\n");

    CSL_BootCfgUnlockKicker(); //unlock

    changeRegister(0x2090210,0x0); //SGMII_SERDES_CONTROL_PORT1 = 0x0;

    changeRegister(0x2090110,0x0); //SGMII_SERDES_CONTROL_PORT0 = 0x0;

    CSL_BootCfgSetSGMIIConfigPLL (0x00000041); //SGMII_SERDES_CFGPLL = 0x00000041;

     CycleDelay(100);

     CSL_BootCfgSetSGMIIRxConfig (0, 0x00700621); //SGMII_SERDES_CFGRX0 = 0x00700621;

    CSL_BootCfgSetSGMIIRxConfig (1, 0x00700621); // SGMII_SERDES_CFGRX1 = 0x00700621;

     CSL_BootCfgSetSGMIITxConfig (0, 0x000108A1); //SGMII_SERDES_CFGTX0 = 0x000108A1;

    CSL_BootCfgSetSGMIITxConfig (1, 0x000108A1); //SGMII_SERDES_CFGTX1 = 0x000108A1;

     changeRegister(0x2090138,0x41); //SGMII_SERDES_AUX_CFG_PORT0 = 0x00000041;

    changeRegister(0x2090238,0x41); //SGMII_SERDES_AUX_CFG_PORT1 = 0x00000041;

    CycleDelay(100);      

    changeRegister(0x2090118,0x1); //SGMII_SERDES_MR_ADV_PORT0 = 0x1;

    changeRegister(0x2090218,0x1); //SGMII_SERDES_MR_ADV_PORT1 = 0x1;

    CycleDelay(100);      

    changeRegister(0x2090210,0x1); //SGMII_SERDES_CONTROL_PORT1 = 0x1;

    changeRegister(0x2090110,0x1); //SGMII_SERDES_CONTROL_PORT0 = 0x1;

    changeRegister(0x2090950,0x2520); //SGMII_SLIVER_MAXLEN2 = 0x2520;

    changeRegister(0x2090910,0x2520); //SGMII_SLIVER_MAXLEN1 = 0x2520;

     changeRegister(0x2090944,0xA1); //SGMII_SLIVER_MACCONTROL2 = 0xA1;

    changeRegister(0x2090904,0xA1); //SGMII_SLIVER_MACCONTROL1 = 0xA1;

     CycleDelay(100000000);

    CSL_BootCfgLockKicker(); //lock

        /* SGMII SERDES Configuration complete. Return. */

       return 0;

}

Mind you this isn’t polished code. Just an experiment to see if the descriptor errors don’t show if you re-init the SGMII SERDES beofre TX.
Please let us know if this helped.

Thanks,
Lali

Lali-

Thanks very much, we'll try this right away.

In addition, we've been working on a parallel approach:  isolate the PHY from incoming packets during initialization.  We tried placing the TRD+/- lines in high impedance state but somehow that impacted initialization.  Now we're trying to isolate the PHY from the GMII interface, which seems to show early promise, but needs a lot more testing.

-Jeff

Lali,

I observe similar behavior in my tests where those stats registers have values like you show. I run the reset script, the init script, then start tcpReplay and see those values increment.

In my test case, I am using Advantech's 8681 card and loading code over PCIe with the TI Desktop Linux SDK. I am not using a GEL file during this.

I tried adding the Init_SGMII_SERDES() function as you had shown. I had to change a couple lines for it to work at all with the 8681 card:

    CSL_BootCfgSetSGMIIConfigPLL (0x00000051); //SGMII_SERDES_CFGPLL = 0x00000041;

    changeRegister(0x2090904,0x400A1); //SGMII_SLIVER_MACCONTROL1 = 0xA1;

Without these changes, I wasn't able to see any packets being transmitted at all. After making those changes to the code you sent, I tried calling Init_SGMII_SERDES() in the location that you suggested as well as prior to init and immediately before sending packets. In all these cases, I still ran into the same problem: once I started running tcpReplay, within a few iterations of my test, the chip soft reset would fail.

Did the modifications that you suggested help in your tests? I do not have access to a JTAG and an EVM board right now so I want to find out if there are differences seen with this code added between the EVM board and the 8681 card.

Regards,

Chris

Chris,
To add to my above posts, please also note:

In the original cpsw_mgmt.c file around line 451, a call to Init_SGMII_SERDES() exists inside inside Init_Cpsw. Please remove this so as to avoid calling Init_SGMII_SERDES() twice in your code.

Please let me know if these suggestions fixed the issue.
I have tried this on my C6678 EVM and they work.

Lali

Lali,

I tried your suggestion of modifying the platorm_lib. I rebuild both platform_lib and platform_init. I ended up taking the code from platform_lib and using that as the Init_SGMII_SERDES() function and left the call at line 451. I didn't add the function to that do-while loop in Init_SGMII(). This seems to work for the simplified test case of using tcpReplay, but doesn't fix the problem in our main project. 

Your explanation of the problem has helped though. I think the problem should be similar in the case of our main project because, as Jeff mentioned, disconnecting the Ethernet cable and reconnecting it after init has finished and the first hundred packets or so have been "sent" prevents the issue from occurring. 

We're currently looking into a way of "disconnecting" the cable through software by writing to the PHY registers as a workaround in this case.

Chris

Pavlo-

We put the PHY in high-impedance mode; i.e. as if external Rx and TX were disconnected (same as if cable was disconnected).  This ensures that no Rx packet activity occurs during PA initialization.

The c66x NetCP and PA truly need "absolute quiet" to be reliably/consistently initialized.

-Jeff
Signalogic