TDA4VH-Q1: Adding flow control inside the CPSW9g switch of the SOC Jacinto

Hi,

Matthias said:

/* Clear fields not supported by hardware */
CSL_FINS(macControl, XGE_CPSW_PN_MAC_CONTROL_REG_EXT_RX_FLOW_EN, 0U);
CSL_FINS(macControl, XGE_CPSW_PN_MAC_CONTROL_REG_EXT_TX_FLOW_EN, 0U);

These bits might enable Flow control based on signaling mechanism. Might need H/W for activating flow control.

Matthias said:

CSL_FINS(macControl, XGE_CPSW_PN_MAX_BLKS_REG_RX_MAX_BLKS, 2U);
CSL_FINS(macControl, XGE_CPSW_PN_MAX_BLKS_REG_TX_MAX_BLKS, 2U);

This will change the FIFO size to 2KB for both Tx & Rx.
But in Tx we have 8 logical Tx Queues each corresponding to different priority (0 to 7).

We have not changes this value anytime during our test.

Matthias said:

CSL_FINS(macControl, XGE_CPSW_PN_MAC_CONTROL_REG_RX_FLOW_EN, 1U);
CSL_FINS(macControl, XGE_CPSW_PN_MAC_CONTROL_REG_TX_FLOW_EN, 1U);

Matthias said:

I believe the commands to add in the cpsw_macport.c are : 

Yes.

Best Regards,
Sudheer

Thank you,

I also believe I have to define a limit in the FIFO to trigger a flow control when it has been reached, like this : 

CSL_FINS(macControl, XGE_CPSW_PN_RX_FLOW_THRESH_REG_COUNT, 0x000000F0U);

I run some test with wireshark to see if I can get some pause frame when I data flood a processor. (I sent data from PC to R5F)

On Wireshark I detected no trace of pause frame so I wonder if I'm missing something.

Is there more modification to be done to receive pause frame ?

Regards,

Matthias

Hi,

Matthias said:

I also believe I have to define a limit in the FIFO to trigger a flow control when it has been reached, like this : 

CSL_FINS(macControl, XGE_CPSW_PN_RX_FLOW_THRESH_REG_COUNT, 0x000000F0U);

Threshold configuration register is different, which is not part of MAC CONTROL register.
Above will overwrite the MAC CONTROL register values.

We not have any CSL API for this, as we have not configured this at all (not tested at our side)
You can access this using below.
hCpswRegs->ENETPORT[portNum].PN_RX_FLOW_THRESH_REG = value;

Best Regards,
Sudheer

Hi, thank you for your response.

I tried to implement the command ligne you suggested, but I still have no trace of pause frame...

Is this command really connected to the CPSW_PN_RX_FLOW_THRESH_REG[8-0] COUNT bit field ? 

Regards,

Matthias

Hi,

Threshold configuration is not available for CPSW9G.

Default it will be FIFO level. If FIFO is full then you will see pause frames, if not you can't.

May be you can test by connecting debugger and halt the core, and then send the data to CPSW. After all DMA descriptors fill then FIFO will get full you may observe pause frames from CPSW.

Best Regards,
Sudheer

Hello,

Ok, I thought it was possible because the documentation SPRUJ52C p.1327 say that we can activate flow control and we have to set this bit field to do so. If this bit is not set it seems flow control can't work in reception. 

I did multiple tries, stopping a core and sending UDP messages from a PC to a R5F core ; sending too much of UDP messages in order to fill the FIFOs ; but both case doesn't return pause frames. 

I have a question related to the last case. Does DMA send automatically the data from port 0 FIFO to R5F's lwip stack, without checking if there is room for the data ? And is there a mechanism in the lwip stack memory that junk the overflow of data ?

Thank you.

Regards,

Matthias Durand

Hi,

Matthias said:

Ok, I thought it was possible because the documentation SPRUJ52C p.1327 say that we can activate flow control and we have to set this bit field to do so. If this bit is not set it seems flow control can't work in reception. 

Can you try to set buffer size as below and check once, by halting the core using debugger.

Matthias said:

I have a question related to the last case. Does DMA send automatically the data from port 0 FIFO to R5F's lwip stack, without checking if there is room for the data ? And is there a mechanism in the lwip stack memory that junk the overflow of data ?

DMA will send data from Host Port FIFO to R5F when it has free descriptors.
When all descriptors are filled, then DMA can't empty the Host Port FIFO.

Best Regards,
Sudheer

Hi, thank you for your response.

I implemented the configuration as you suggested :

CSL_CPSW_setPortMaxBlksReg(regs,portNum,0x00005U,0x0000BU);

I tried multiples couple of valeur : 0x007U, 0x00DU ; 0x006U, 0x00CU and 0x005U, 0x00BU but none of them seems to work.

I also tried to use directly those commande instead on going with the function but no result as well :
CSL_FINS (regs->ENETPORT[portNum].PN_MAX_BLKS_REG, XGE_CPSW_PN_MAX_BLKS_REG_RX_MAX_BLKS, 0x00005U);
CSL_FINS (regs->ENETPORT[portNum].PN_MAX_BLKS_REG, XGE_CPSW_PN_MAX_BLKS_REG_TX_MAX_BLKS, 0x0000BU);

Maybe there is something I'm missing...

Regards,

Matthias

Hi,

Please allow some time for us to confirm on our side and will update you in next week.

Best Regards,
Sudheer

Hello,

Do you have news regarding this topic ?

Regards,

Matthias

Hi,

Sorry, we have not tested this at our side.
But, I got some information from Expert it can trigger when No buffers at destination port for the transmission of incoming data then only it can occur. 

Can you check as per below once.

$pn_rx_max_blks should be written with a value of 7 (receive FIFO allocation for runout) and the $pn_tx_max_blks should be written with a value of 13 (20 blocks total in the combined RX and TX FIFOs).  If Pn_Rx_Bottom_of_FIFO drops occur then the $pn_rx_max_blks can be increased to a maximum value of 9 ($pn_tx_max_blks value should be written with 11 for a total of 20).

Best Regards,
Sudheer

Hello,

I implemented the configuration you suggested but it doesn't work. I'm currently looking to print switch statistic to get more informations about the current state of the CPSW9g. I did found that it is possible to use : "ethtool -S" on A72 cores but no stats are available for eth2.

Regards,

Matthias

Hi,

Matthias said:

I'm currently looking to print switch statistic to get more informations about the current state of the CPSW9g. I did found that it is possible to use : "ethtool -S" on A72 cores but no stats are available for eth2.

CPSW9G statistics are not available at A72 via ethtool command, when using EthFw.

Please refer to FAQ for collection of CPSW statistics, use patch from below FAQ and integrate into EthFw. (statistics can be seen on EthFw debug terminal) 

Best Regards,
Sudheer

Hello,

Thank you, I didn't know there was a patch but I implemented it myself. So it gave me an additional confirmation that there is no pause frame generated. Can you check on your side to confirm that the flow control is rightly usable ?

Regards,

Matthias

Hi,

Matthias said:

Can you check on your side to confirm that the flow control is rightly usable ?

We have not tested our side and SDK is not supported by default this feature.
Let me try same thing once, and check whether pause frames observed or not?  Please allow us some time to test this.

Best Regards,
Sudheer

Hello,

I have done an additional experiment regarding this issue : 

- Sending UDP frames to a client

- Pause the client (I also tried to pause both the server and the client)

- Analyse with Wireshark for pause frames (no pause frame seen)

-Analyse the statistics printed by the server :

It seems like, the Host port still proceed to send data despite having a destination, because FIFO transfert the same amount of frames between Rx FIFO and Tx FIFO. So the Tx FIFO pursue to send data and doesn't pile up and that why there is no pause frame. So we loose packet and that is an issue. 

Is this comportement normal ? I thought that if DMA has no where to send packet, it doesn't send them...

Can you please confirm on your side this comportement ? 

Regards

Matthias

Hi,

Host Port tx packet count is packet received from external port and sending to any of TDA4 cores (server or client).

if any other port is active, and few data movement also will be captures at Host Port Tx.

Host Port Tx is same as sum of all External Ports Rx packet count.

Matthias said:

I have done an additional experiment regarding this issue : 

- Sending UDP frames to a client

- Pause the client (I also tried to pause both the server and the client)

- Analyse with Wireshark for pause frames (no pause frame seen)

-Analyse the statistics printed by the server :

I think it may crate Top of FIFO Drops not pause frames.

Best Regards,
Sudheer

 Hello,

Yes, there is a little difference of trames numbers surely due to other client (like the A72). But in other ports there is no significant action compared to the number of frames sent from port 3 to client.

But the issue is regarding the lost of the large number of frames from the switch CPSW9g to the client.

Doredla Sudheer Kumar said:

I think it may crate Top of FIFO Drops not pause frames.

The statistic table show the number of Top of FIFO Drops if it's different from zero but in our case it is not shown so there is no Top of FIFO Drops in the stat. 

In the experiment, the client is stopped so it can no longer receive frames. So the number of Host port TxGoodFrame counter should stop but it's not the case, the Host continu to send frames to a stopped client. So that why there is no pause frame and no Top of FIFO drops.

How can we proceed in order to stop the Host port from sending frames to a dead end ?

Thank you.

Regards,

Matthias 

Hi,

Matthias said:

The statistic table show the number of Top of FIFO Drops if it's different from zero but in our case it is not shown so there is no Top of FIFO Drops in the stat. 

In the experiment, the client is stopped so it can no longer receive frames. So the number of Host port TxGoodFrame counter should stop but it's not the case, the Host continu to send frames to a stopped client. So that why there is no pause frame and no Top of FIFO drops.

Until Host Port find the free descriptors it will send the packets to the client. If packets received belongs to client MAC address.

Matthias said:

How can we proceed in order to stop the Host port from sending frames to a dead end ?

Can you send Broadcast frames to the CPSW Port and Halt the MCU2_0 server core and Client cores MCU2_1 and check the statistics once..


Best Regards,
Sudheer

Hello, 

Doredla Sudheer Kumar said:

Until Host Port find the free descriptors it will send the packets to the client. If packets received belongs to client MAC address.

Yes, that what I think too but I don't understand how the packets are lost between host port and the client...

Here are the statistics for broadcast frames with all the cores stopped. The comportement stay the same. 

Thanks for your time.

Regards,

Matthias

Hi,

Matthias said:

Here are the statistics for broadcast frames with all the cores stopped. The comportement stay the same. 

Can you confirm, are you still sending the Broadcast frames even after all cores stopped. If so, you would have observed some error frames on the Broadcast receive port. Can you please dump other active Ports statistics as well.

Best Regards,
Sudheer

Hello, 

Yes I sent Broadcast frames when all cores were stopped, and I confirmed it with Wireshark, but I find it weird too that the switch is unphased. Even in Unicast there is no error. 

Other ports aren't used, Statistics are empty. 

Regards,

Matthias

Hi,

Matthias said:

Yes I sent Broadcast frames when all cores were stopped, and I confirmed it with Wireshark, but I find it weird too that the switch is unphased. Even in Unicast there is no error. 

As cores are halted, and you are sending some frames..
After few frames sending, have you stopped sending frames & ran core and check the statistics. As above cores are halted, you many not get the updated statistics.

Best Regards,
Sudheer

Hello,

My protocol is : 

1.  Start all cores with bootloader with SD card. It contain the app and they are linked to the right core.

2.  When the network finish it init and the network is stable I stop all the cores.

3.  I send on port 3, 37 000 Multicast messages.

4. Once all the messages are sent, I start the cores.

5. The mcu2_0 write on logs the statistics of the switch I have shown before.

The result : frames are sent from the switch even if the cores are stopped.

This is an issue so if you have more information on your side, I'm eager to know your thought.

Regards,

Matthias

Hi,

Matthias said:

3.  I send on port 3, 37 000 Multicast messages.

4. Once all the messages are sent, I start the cores.

During core Halt, CPSW switch is still active and it can forward the packets to other ports.
But, not to Host Port if packets intended core is halted (ex. packet are sending to MCU2_0 and MCU2_0 is halt) after all available DMA descriptors are used.

Also, can you please confirm the Multicast packets are intended to any of core? Have you verified ALE table?
Also, can you check on Wireshark side, whether CPSW is sending nay pause frames or not at sender side? Ideally drops should be expected if not controlling the flow would result in sending pause frames.

Let me check internally with H/W, how to trigger pause frames from CPSW and update you soon.

Best Regards,
Sudheer

Hello,

Indeed, host shouldn't send the data because there is no destination. I'm sure I have stopped them because they are not responding in any way. And on Wireshark the data are sent but no responses, that is normal because the are stopped. The switch keep sending despite that and that's why there is no pause frame, because the FIFO stay empty.

Did you get more information on your side ?

Regards,

Matthias

Hi, 

Matthias said:

Did you get more information on your side ?

When cores are stopped, FIFO should get overflow and observe drops in ports.

But in your log we could not see any drop statistics. 

If no drops ideally pause frame should sent. 

What is the use case for flow control in your system. 

Best Regards, 

Sudheer

Hello,

Doredla Sudheer Kumar said:

But in your log we could not see any drop statistics.

Indeed, there is no drop statistics, no pause frame statistics, and the statistics say all frames are sent despite having no destination.

Doredla Sudheer Kumar said:

What is the use case for flow control in your system

In our system frames are very important, so we need to have no data drop. The Pause frame tenet response to our need, so we would very much like it to work.

Regards,

Matthias

Hi, 

Matthias said:

Doredla Sudheer Kumar said:

But in your log we could not see any drop statistics.

Indeed, there is no drop statistics, no pause frame statistics, and the statistics say all frames are sent despite having no destination.

It look very odd. 

Ideally we should observe FIFO drops when no core is reading the data from the Host Port. 

When Host Port FIFO is empty, there could not be any data movement from external port FIFO to Host port, ideally in this scenario pause frame should sent from external port to avoid drops. 

Best Regards, 

Sudheer