PROCESSOR-SDK-AM62X: CPSW: Broadcast Rate Limit: Burst behavior

Daolin Qiu said:

An alternative method using different tc configurations is show below, where the rate can be limited to 10Mbits/sec and allowing for bursting of 20k. It might also be a starting point for you to experiment around with your specific use case. 

Fullscreen

1

2

tc qdisc add dev eth0 ingress

tc filter add dev eth0 parent ffff: u32 match u32 0 0 police rate 10mbit burst 20k

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: u32 match u32 0 0 police rate 10mbit burst 20k

Please let us know if you have additional follow up questions.

Hi Daolin,

thanks for your response. I am afraid the suggested tc command will not work, because the driver only allows for packets rate to be limited and the burst parameter is not used. This is hinted by the commit message you referred to. I also dug through the code a bit (am65-cpsw-qos.c) beforehand, only pkts_rate is evaluated. The driver sets  the field I0_REG_P0_BCAST_LIMIT in the CPSW_NU_CPSW_NU_ALE_I0_ALE_PORTCTL0_N registers.

Is it possible to change the measurement interval? We build our own firmware so patching the kernel would be possible. Currently it seems to be tied to a 1ms interval.

From the technical reference manual:

"Broadcast Packet Rate Limit - Each prescale pulse loads this field
into the port broadcast rate limit counter. The port counters are
decremented with each packet received or transmitted depending on
whether the mode is transmit or receive. If the counters decrement
to zero, then further packets are rate limited until the next prescale
pulse. Broadcast rate limiting is enabled by a non-zero value in this
field."

Is it possible to change the prescaler pulse frequency or would that mess with all ALE operations?

Currently we tell our customers to disable the broadcast rate limit when they are experiencing related networking problems.

It would be nice if future iterations of the integrated CPSW switch would implement a more sophisticated rate-limiting scheme (leaky bucket or so)

Thanks,

Jan

Hi Jan, 

Jan Soundhouse said:

I also dug through the code a bit (am65-cpsw-qos.c) beforehand, only pkts_rate is evaluated. The driver sets  the field I0_REG_P0_BCAST_LIMIT in the CPSW_NU_CPSW_NU_ALE_I0_ALE_PORTCTL0_N registers.

Can you point me out to where in the am65-cpsw-qos.c code indicates only pkts_rate is evaluated and not burst? Where exactly in the driver modifies the PORTCTL0_N registers?

Jan Soundhouse said:

I am afraid the suggested tc command will not work, because the driver only allows for packets rate to be limited and the burst parameter is not used.

May I ask if there is any method through which you test if burst parameter takes an impact? 

There are two variations of using the tc tool that I shared in my previous message. The first variation was shown in the commit message which we haven't evaluated extensively before. The second variation uses the "burst" parameter as opposed to the "pkts_burst" parameter. This is the one we have done some first level evaluation with to show that rate-limiting on ingress can be achieved. However, we have not extensively evaluated if/how the burst parameter works. It would help us understand what sort of tests are typically done to test if bursting is accounted for.

What we do know from feedback from the software development team about the second variation of the tc tool is that it doesn't actually perform the ingress rate limiting on the CPSW hardware. However, despite the hardware not running rate-limiting, we still find that CPU load on the destination is significantly reduced even if software is handling rate-limiting. This indicates that the CPSW driver may not even be handling the burst parameter either and it might be some higher level Linux driver that is handling it. However, this is just speculation as we have not yet looked into this further.

Additionally, for clarification, are you talking about rate-limiting on ingress or egress of packets?

-Daolin

Daolin Qiu said:

Can you point me out to where in the am65-cpsw-qos.c code indicates only pkts_rate is evaluated and not burst? Where exactly in the driver modifies the PORTCTL0_N registers?

In the am65_cpsw_qos_clsflower_policer_validate function, there is a check at the end that only lets rate_pkt_ps through and in am65_cpsw_qos_configure_clsflower only the rate_pkt_ps is taken from the police (pkts_burst is not used).

When the pkts_rate is set in cpsw_ale_rx_ratelimit_bc there is some indirection going on, the actual register is determined by this:

	[ALE_PORT_BCAST_LIMIT]	= {
		.name		= "bcast_limit",
		.offset		= ALE_PORTCTL,
		.port_offset	= 4,
		.shift		= 24,
		.port_shift	= 0,
		.bits		= 8,
	},

Daolin Qiu said:

The second variation uses the "burst" parameter as opposed to the "pkts_burst" parameter. This is the one we have done some first level evaluation with to show that rate-limiting on ingress can be achieved. However, we have not extensively evaluated if/how the burst parameter works. It would help us understand what sort of tests are typically done to test if bursting is accounted for.

What we do know from feedback from the software development team about the second variation of the tc tool is that it doesn't actually perform the ingress rate limiting on the CPSW hardware. However, despite the hardware not running rate-limiting, we still find that CPU load on the destination is significantly reduced even if software is handling rate-limiting. This indicates that the CPSW driver may not even be handling the burst parameter either and it might be some higher level Linux driver that is handling it. However, this is just speculation as we have not yet looked into this further.

Additionally, for clarification, are you talking about rate-limiting on ingress or egress of packets?

I have missed that the second command is not offloaded to the cpsw switch hw (and did not try it, because I thought it would be rejected by the cpsw driver). In that case it should not go into the cpsw driver at all and would be handled by some other part of the kernel. The software based ingress rate limit might work for us, but I have to verify that with our Product Management.

We only need ingress rate-limiting.

Thanks,

Jan

Hi Jan, 

Apologies for the delayed response. 

Jan Soundhouse said:

In that case it should not go into the cpsw driver at all and would be handled by some other part of the kernel. The software based ingress rate limit might work for us, but I have to verify that with our Product Management.

If it helps you some additional information, we did some testing using the second command with a simple setup and discovered the CPU load can be reduced to about 4% load compared to ~90% load when the command was not used to rate limit. You should keep in the mind that the CPU load of course can vary depending on your setup and what applications you are running, and that we achieved these numbers with a very simplistic setup.

-Daolin