This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

PROCESSOR-SDK-AM64X: AM64x: Configuration of CPSW3G for TSN

Part Number: PROCESSOR-SDK-AM64X

I confirmed the operation of "Enhancements for Scheduled Traffic (EST) Offload" in PROCESSOR-SDK-AM64X 08_00_00_21.

And, I use TMDS64GPEVM.

I executed the example configuration for "CPSW2g" (not CPSW3g) written in the following link.

https://software-dl.ti.com/processor-sdk-linux/esd/docs/08_00_00_21/linux/Foundational_Components/Kernel/Kernel_Drivers/Network/CPSW3g.html#enhancements-for-scheduled-traffic-est-offload

However, the TAPRIO settings have not been completed correctly.

when I ran the example settings, I got the following error:

ERROR 1
"netlink error: Device or resource busy"

I checked "Query Channels", I confirmed that the hardware tx queue settings have not changed.

# ethtool -l eth0
Channel parameters for eth0
Pre-set maximums:
RX:        1
TX:        8
Other:        n/a
Combined:    n/a
Current hardware settings:
RX:        1
TX:        1
Other:        n/a
Combined:    n/a

Therefore, I added a command to stop eth1 in the configuration.

#Setup interface and queue configuration
ip link set dev eth1 down
ip link set dev eth0 down
ethtool -L eth1 tx 3

When the above command was executed, the hardware tx queue settings were reflected.

In the case of CPSW3g, it is necessary to bring down the eth1 side as well.

# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:             1
TX:             8
Other:          n/a
Combined:       n/a
Current hardware settings:
RX:             1
TX:             3
Other:          n/a
Combined:       n/a

However, when I ran the modified configuration, I got another error about the mode of taprio.

ERROR 2
Error: Device failed to setup taprio offload.

This error was output when full offload is enabled (flag 2).

Q1. Does PROCESSOR-SDK-AM64X 08_00_00_21 not support the full-off load function?

Q2. Will there be any settings for CPSW3g in the future?

  • Hi,

    You are correct, both eth0 and eth1 (the CPSW3G Ethernet ports by default) need to be down in order to modify the HW queues in one of the ports. If you only have one Ethernet cable connected the interface is down, but I'm assuming you have both eth0 and eth1 cables connected. The number of HW ports in use is expected to be a static configuration, we don't expect this restriction to be an issue. It is not explicitly stated in the SW documentation, we'll look to clarify in a future revision.

    With regards to the error

    Error: Device failed to setup taprio offload.

    the first thing that comes to mind is did you bring the interface back up after modifying the HW queuing with ethtool -L ? I get the same error if I skip bringing the interface up in teh example at https://software-dl.ti.com/processor-sdk-linux/esd/docs/08_00_00_21/linux/Foundational_Components/Kernel/Kernel_Drivers/Network/CPSW3g.html#enhancements-for-scheduled-traffic-est-offload . The step "ip link set dev eth0 up" in the below sequence.

    ip link set dev eth0 down
    ethtool -L eth0 tx 3
    
    #disable rrobin
    ethtool --set-priv-flags eth0 p0-rx-ptype-rrobin off
    
    #bring up eth0 interface
    ip link set dev eth0 up
    #Setup EST schedule with 3 Gates (Q0-Q2). For description of Command parameters, see manual page for taprio.
    #TC0 <-> Q0, TC1 <-> Q1, and TC2 <-> Q2
    tc qdisc replace dev eth0 parent root handle 100 taprio \
       num_tc 3 \
       map 0 0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 \
       queues 1@0 1@1 1@2 \
       base-time 0000 \
       sched-entry S 4 125000 \
       sched-entry S 2 125000 \
       sched-entry S 1 250000 \
       flags 2
    

    In case it is something different could you post the similar sequence of what you ran to get the error?

      Pekka

  • Hi Pekka,

    Thank you for your reply.

      Pekka Varis said:

      You are correct, both eth0 and eth1 (the CPSW3G Ethernet ports by default) need to be down in order to modify the HW queues in one of the ports.

    I understand how to set HW queues.

    With regards to the error

      Ryohei Yamada said:

      Error: Device failed to setup taprio offload.

    The error was improved by executing the sleep command.

    It seems that tc-taprio should not be run immediately after bring up eth0 interface.

    ip link set dev eth0 down
    ip link set dev eth1 down
    ethtool -L eth0 tx 3
    
    #disable rrobin
    ethtool --set-priv-flags eth0 p0-rx-ptype-rrobin off
    
    #bring up eth0 interface
    ip link set dev eth0 up
    sleep 5
    #Setup EST schedule with 3 Gates (Q0-Q2). For description of Command parameters, see manual page for taprio.
    #TC0 <-> Q0, TC1 <-> Q1, and TC2 <-> Q2
    tc qdisc replace dev eth0 parent root handle 100 taprio \
       num_tc 3 \
       map 0 0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 \
       queues 1@0 1@1 1@2 \
       base-time 0000 \
       sched-entry S 4 125000 \
       sched-entry S 2 125000 \
       sched-entry S 1 250000 \
       flags 2 

    I ran iperf3 with the above settings.

    A part of the capture at that time is as follows.

    Occasionally a periodic burst of frames is seen every 500 us.

    However, there may be about 1000us frame interval once every few times.

    How can I see a periodic burst of frames every 500 us? 

    Do I need to change iperf3 settings? or Do I need to enable IET FPE with MAC Verify?

  • The error was improved by executing the sleep command.

    In general with Ethernet interfaces once you enable an interface there is a delay of a second or maybe more when the Ethernet level auto-negotiation completes. So I'd suggest in general you have a sleep 1 or longer after executing interface up. You will see a printout in the standard console (serial port) that the link is actually up.

    I ran iperf3 with the above settings.

    I'm guessing you also ran the tc filter to force the UDP port used by your iperf3 command to use one of the queues you setup? Can you show the tc filter command and the iperf3 command. Basically how many frames per second is iperf3 asked to generate.

    Occasionally a periodic burst of frames is seen every 500 us.

    However, there may be about 1000us frame interval once every few times.

    How can I see a periodic burst of frames every 500 us? 

    What EST does is close gates for traffic to get on the wire. So EST guarantees that traffic using a given HW queue (often mapped to VLAN PCP priority by the user) does not show up when gate is closed. If frames show up when a gate is open is a combination of packet being sent (iperf3 in your case), and gate being open. The schedule set is 500us period (125+125+250), so packets packets showing up n * 500us is as expected. iperf3 in UDP tests produces frames in bursts to try to meet the rate you asked it to create. It does not create a smooth rate of 1 packet per say 500us, rather it tries to meet the rate ou requested over a longer period of time. The gates you have configured are open for 125us or 250us, so multiple packets will fit on the wire at a time and the avarege can be met even if there is no packets in one 500us period.

    or Do I need to enable IET FPE with MAC Verify?

    No. IET is the acronym for the feature preemption, and MAC verify is a feature related to initializing preemption. EST is the time aware shaper feature in TSN.

      Pekka

  • Hi Pekka,

    Thank you for your reply.

      Pekka Varis said:

      Can you show the tc filter command and the iperf3 command. 

    The tc filter and iperf3 commands have the same settings in the documentation of Processor SDK AM64X.

    tc filter command

    tc filter add dev eth0 egress protocol ip prio 1 u32 match ip dport 5003 0xffff action skbedit priority 3
    tc filter add dev eth0 egress protocol ip prio 1 u32 match ip dport 5002 0xffff action skbedit priority 2
    tc filter add dev eth0 egress protocol ip prio 1 u32 match ip dport 5001 0xffff action skbedit priority 1

    iperf3 command

    iperf3 -c 192.168.2.10 -u -b100M  -p 5003 -l1472 -t10 -i5&
    iperf3 -c 192.168.2.10 -u -b100M  -p 5002 -l1472 -t10 -i5&
    iperf3 -c 192.168.2.10 -u -b100M  -p 5001 -l1472 -t10 -i5&

      Pekka Varis said:

      What EST does is close gates for traffic to get on the wire.

      So EST guarantees that traffic using a given HW queue (often mapped to VLAN PCP priority by the user) does not show up when gate is closed.

      The gates you have configured are open for 125us or 250us, so multiple packets will fit on the wire at a time and the avarege can be met even if there is no packets in one 500us period.

    I was able to understand the function of EST.

    It was confirmed that the frames stored in each queue are no longer sent at the specified timing.

    Q2: UDP Port 5003 (125us)

    Q1: UDP Port 5002 (125us)

    Q0: UDP Port 5001 (250us)

    I solved all my problems of EST

    I appreciate your kind response.