DRA821: CPSW CBS configuration and Linux standard tc-cbs interface

Hi Gunter,

Putting Siddharth's response in the mail thread here.

First, it looks like the driver does not support the normal Linux interface for the CBS, tc-cbs, and has its own way of configuring this. Is there a reason why this
does not use the standard interface?
=> As documented on the tc-cbs man page at: man7.org/.../tc-cbs.8.html,
"CBS is meant to be installed under another qdisc that maps packet flows to traffic classes, one
example is mqprio." The driver supports hw offload when the tc command is used for mqprio, with hw
parameter set to 1. The driver supports offloading the specific parameters: min_rate and max_rate
to the CPSW hardware using the CIR (Committed Information Rate) and EIR (Excess Information Rate)
registers, with the CIR register configured for min_rate and EIR configured for max_rate - min_rate.
All tc commands including cbs are supported in software. The specific example used in the
Documentation with min_rate and max_rate is the subset of tc commands which can be offloaded to CPSW
Hardware.

It appears to use a combination of the sysfs interface and the mqprio qdisc's "min_rate" and "max_rate" parameters. It's unclear how exactly these map to the 802.1Qav shaper parameters?
=> The sysfs interface is for the Host Port DMA channel rate-limiting, which controls the rate at
which the traffic is sent to CPSW HW from Host Port via the DMA channels. The min_rate and max_rate
parmeters which are a part of the tc mqprio command can be offloaded to HW. The tc-cbs commands in
software can be used as is, with the mqprio min_rate and max_rate parameters being used to enforce
rate limits in hardware as well. 802.1Qav aims to shape traffic in terms of the rate allocated to
each priority. The same can be achieved with the min_rate and max_rate parameters as well per
priority, with the ability to offload them to CPSW Hardware on the Forwarding path, while the DMA
channel rate limiting is used to control Host Port traffic being sent to the CPSW Hardware.

The official interface, tc-cbs, has parameters which directly correspond to the expected parameters, but it's not at all clear how these can be realized with the existing interface.
=> The tc mqprio command with the min_rate and max_rate parameters aren't being mapped to the tc cbs
command. They both can be used indpendently and are supported in Software. The only difference is
that the tc mqprio command with the min_rate and max_rate paramters can be used for rate-limiting
traffic per priority and can be offloaded to CPSW HW for rate-limiting traffic in the Forwarded
Path as well as the External Port Egress path.

In the section "3.2.3.6.3.3.2.3.1.1. Host port ingress Rate Limiting offload," the "/sys/class/net/eth0/queues/tx-7/tx_maxrate" sysfs value is set, but the section is described as "ingress rate limiting." This implies that it limits the receive rate, as a policer, but the parameter name is "tx_maxrate," which implies an egress rate limit. Is this an ingress or an egress rate limit? Experimentally, it appears to be an egress limit.
=> The phrase "ingress" here has to be understood in conjunction with the words "Host port" that
precede it. The Host Port corresponds to the port in CPSW facing the Software/DMA, from which the
traffic generated by the Host ingresses into the CPSW Hardware. Thus, Host Port ingress rate
limiting offload corresponds to Rate Limiting the traffic being sent from Host to CPSW's Host Port
through DMA channels, effectively implying DMA channel rate limiting.

The following section, "3.2.3.6.3.3.2.3.1.2. Switch egress, Ethernet Port Transmit Rate Limiting offload," suggests that it is configuring an egress limit and sets values both through the sysfs interface and the "mqprio" interface. What is the relationship among these values?
=> The sysfs commands are used to configure the CPSW Host Port Ingress Rates which correspond to the
DMA channel rate-limiting.
The mqprio commands can be used both in Software and Hardware. In the Documentation, with the
parameter "hw" set to 1, the min_rate and max_rate rate limits are offloaded to the CPSW Hardware.
With this, for all traffic that is either being sent from the Host and Egressing out the External
Port, or the traffic that is being forwarded and Egressing out the External Port, can be rate
limited in CPSW Hardware itself.

The document describes "min_rate" as the CIR and the "max_rate" as the EIR. Following the example, where min_rate matches tx_maxrate, the transmission rate is constrained to the min_rate. What does the max_rate do, if not allowing the sender to exceed the min_rate when bandwidth permits?
=> CIR corresponds to min_rate, while EIR corresponds to the Excess rate which is max_rate - min_rate.
min_rate and max_rate correspond to the minimum rate that should be allowed for traffic of the
highest priority at the given moment. For example, the command could specify the min_rate for
priorities 0, 1 and 2. If there is sufficient traffic of priority 2, then it can be expected to
meet the min_rate. On the other hand, if there is no traffic of priority 2 and only traffic of
priority 1 happens to be the highest priority traffic at the moment, then it can be expected to
meet the min_rate criteria provided there is sufficient traffic of priority 1. Additionally, the
min_rate parameter allows switching to other priority traffic intermittently, while ensuring that
the min_rate is met for the highest priority traffic. max_rate ensures that traffic of the corresponding priority never exceeds the permitted value
configured by the max_rate parameter.

What are the constraints, as it's easy to enter an invalid config, but there's no feedback as to what the constraints actually are?
=> The values of min_rate and max_rate are postive integers in the units of Mbps. The minimum value
for min_rate is 1, while the maximum allowed value depends on the link bandwidth. Similarly,
max_rate has to be at least 2, considering that is should be greater than min_rate. The sum of
all max_rate values across priorities cannot exceed link bandwidth.

Finally, performing packet capture with these setting, the VLAN header always has the Priority Code Point (PCP) set to zero in the captured packets. This is expected for TC0, but not for TC6 and TC7, in the example. Can the hardware be configured to set the PCP to the correct value? This is a problem, as without the correct value, the switch can't provide guaranteed resources.
=> If the source interface is a VLAN interface, and the skbedit command is used to edit the PCP value
of the packets, and the ip command used to map vlan egress is used as follows:
ip link set eth0.100 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
then the VLAN headers will have the corresponding priority tags for the PCP value added in the
packet using skbedit, that were sent using the VLAN interface.

Regards,
Tanmay