• State
  • Locked Locked
  • Replies 13 replies
  • Subscribers 100 subscribers
  • Views 2312 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Using Packet Accelerator on 66AK2L06 to route Enet packets to ARM and DSP cores

Alan Morales
Intellectual 420 points

In our project, we would like to share a single ethernet port with the same IP address and MAC address (efuse MAC) for all cores (ARM & DSP)  and then use the Packet Accelerator to route packets to specific cores based on different criteria such as UDP port #s and custom header data.

But in our initial testing, we are seeing that the Linux kernel running on the ARM (Processor SDK 03.00.00.04) is setting it's own rules in the Packet Accelerator to basically send all packets with the destination MAC address matching the efuse MAC address into its own RX queue.  Here is an excerpt from the Linux kernel logs (with dynamic debug printing enabled) showing PA rules being set for the efuse MAC address:

Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: pa_open() called for port: 1
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: configuring data receive flow 22, queue 528
...
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: ethss adding address ff:ff:ff:ff:ff:ff, type 4
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: pa_add_addr, port 1
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: add mac, index 0, smac (null), dmac ff:ff:ff:ff:ff:ff, rule 2,
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: type 0000, port 1
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: buflen = 164(a4), cmd_size 120
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: waiting for command transmit complete
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: ethss adding address b4:99:4c:91:52:71, type 1
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: command response complete
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: pa_rx_compl_work_handler - end
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: pa_add_addr, port 1
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: add mac, index 1, smac (null), dmac b4:99:4c:91:52:71, rule 1,
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: type 0800, port 1
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: buflen = 164(a4), cmd_size 120
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: waiting for command transmit complete
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: add mac, index 2, smac (null), dmac b4:99:4c:91:52:71, rule 1,
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: command response complete
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: type 86dd, port 1
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: pa_rx_compl_work_handler - end
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: buflen = 164(a4), cmd_size 120
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: waiting for command transmit complete
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: add mac, index 3, smac (null), dmac b4:99:4c:91:52:71, rule 2,
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: command response complete
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: pa_rx_compl_work_handler - end
Jul 08 23:07:53 k2l-evm kernel: netcp-1.0 2620110.netcp: type 0000, port 1

So when our application code runs on either the ARM or DSP cores and calls into the PA LLD to set its own rules using this same efuse MAC address, they will always be ignored since the rules set by the Linux kernel are set first.

So is there any way to be able to insert our PA rules ahead of the rules set by the kernel so that they can be used to route packets into our own Rx queues?

Or is there some other preferred method for using the PA to route packets to the various cores when Linux is running on the ARM?

  • 0
    TI__Guru 65405 points
    Hi Alan,

    I've forwarded this to the ethernet experts. Their feedback should be posted here.

    BR
    Tsvetolin Shulev
  • 0
    TI__Expert 5805 points
    Hi Alan,

    Could you provide me with your DTS? I want to understand what default NetCP + QoS related configs you have currently in you system.

    In general how PA "filtering" works is that each Cluster has lookup tables where forwarding or "routing" rules are defined. The way that lookup works on LUT1's is First in Last Checked. So it is always recommended that the broader rules are entered first, then the more specific/narrow rules are entered at the top of the table.

    For what you want to do, you will need rules on Ingress0, Ingress1 and Ingress4 on their respective LUT1's and LUT2 (for UDP/TCP/GTPU etc.). Linux kernel will already have some default rules on Ingress0 LUT1a. You will need to add your rules of MAC+ (some other more narrow parameter such as VLAN, ethertype etc.) to get a "search hit" prior to the Linux default rule. Then, on Ingress1->Ingress4 you may need to add rules which link back to the Ingress0 LUT1 entry to continue parsing.

    The use of prevLink is to create an "&" operation rather than an "or" operation; logically speaking. It means that you want a specific MAC+IPV4+UDP port combination to match in order to forward the packet to a specific Destination queue. Another way of seeing this is that if the IPV4 entries and UDP entries are not linked back to the previous stages, then it means that it doesn't care (at that stage) what the address of the lower layer was. So, say that no LUT2 UDP rule entry links back to a IPV4 entry. Then all packets that pass Ingress 0 and Ingress 1 will pass Ingress4 LUT2 as long as they match any rule at that stage. Conversely, if they were linked, then only specific MAC+IPV4 combinations would pass LUT2.

    Another thing to remember is that you also need to consider the "fail routes", which are triggered for packets that fail to match any entry, then these will be forwarded to Linux directly and skip all steps after the fail (say MAC address match fail will force a packet from Ingress0 LUT1a -> RX queue; bypassing all other Ingress clusters).

    Regards,
    Javier
  • 0 in reply to Javier Malave-Bonet
    Intellectual 420 points
    Javier,

    We are using the default DTS for the K2L EVM that is provided in the Processor SDK 03.00.00.04. I can still send you a copy if you are not able to access it.

    The problem that we seem to be having is that the Linux kernel is configuring fairly broad rules in the the packet accelerator using the destination MAC address. So any rules that we try to add after the kernel boots up are ignored because the kernel's rules seem to be catching all the packets.
  • 0 in reply to Alan Morales
    TI__Expert 5805 points

    Hi Alan,


    Thanks for the reply. If you are using the default DTS, I can get the internally no worries. Yes, I do believe that it is looking like an issue with rule configs. Are you clear on how to add these new rules? I suggest to add some more narrow rules to test our hypothesis. Could you run the experiment and let me know of the results?


    Kind regards,

    Javier

  • 0 in reply to Javier Malave-Bonet
    Intellectual 420 points
    So how can I add PA rules such that they are inserted ahead of the Linux kernel's default rules?
  • 0 in reply to Alan Morales
    TI__Expert 5805 points

    Hi Alan,


    When the interface is brought up, Linux would have already set their default rules. So by just using pa_addCustomLUT1 (or pa_addMac) you will ensure these rules are "invoked" first. But, again, as explained earlier, these rules should be more narrow than the Linux rules (which are based, as I understand it on mac destination address filtering only). In that case the packets that do not match your rules will then "trickle down" to the default Linux rules, and if they don't match those, then the fail route will be invoked for forwarding.


    Regards,

    Javier

  • 0 in reply to Javier Malave-Bonet
    Intellectual 420 points

    OK, so what you are saying is that adding more narrow rules through the PA LLD even after Linux kernel boots up will still take precedence.  I will give it a try and let you know how it ends up.  Thanks.

  • 0 in reply to Alan Morales
    TI__Expert 5805 points

    That is correct. Yes, please, let me know how it goes.

  • 0 in reply to Javier Malave-Bonet
    Intellectual 420 points
    Hello Javier,

    I finally got a chance to test this out and I'm still not receiving the packets in the queue that I'm specifying for my Packet Accelerator rules when I use the same destination MAC address that Linux is using. I did as you suggested and added more narrow rules using the PA LLD after the Linux kernel has booted and configured the PA with its rules. One rule is to filter on UDP packets with a specific destination port #. Another rule I tried is for a custom LUT looking at specific bytes in the payload right after the MAC header.

    Neither of these rules would capture packets I send to the EVM with the matching criteria. But, if I change the destination MAC address in my PA LLD rules to something different than what Linux is using, and then I send packets to the EVM with this new MAC address, it will work, and I will see the packets being captured into the queue that I specified.

    I still feel like the Linux PA rules are taking precedence over my rules since Linux entered them into the PA first.
  • 0 in reply to Alan Morales
    TI__Expert 5805 points

    Hi Alan,

    This is strange behavior. After re-reading the PA LLD documentation one more time; (doxygen provided with PDK), I noticed some inconsistency with the explanation. 

    --

    The sub-system examines the L2 and L3 (LUT1) information (see netlayers) in packets based on internal table location. When function Pa_addMac and Pa_addIp are executed and the resulting packet forwarded to the sub-system, the sub-system places the new entries at the highest free table location [JMB -  as I explained earlier]. When incoming packets are examined, until the first matching entry is found. That entry is used to route the packet.

    Because of this it is required that entries into the table be made . For example, when adding a mac address it is common to want to route the following:

       dest mac only - Forward packet to host

       dest mac + ethertype - Continue parsing

       dest mac + source mac + ethertype - Forward packet to host

    To get the desired routing the Pa_addMac commands must be executed and the command packets forwarded to the sub-system in the order shown above [JMB - contradicts the. If they are entered in the reverse order then every packet which has the value dest MAC will be forwarded to the host since it matches the first entry in the list.

    The order dependency applies to calls to Pa_addMac and Pa_addIp, but not to calls between these functions. So all MAC entries can be made followed by all IP entries, or in the reverse order (provided the IP entries do not reference the MAC entries) without changing the operation of the sub-system.

    ----

    I recall that it should be Highest Entry checked first, not lowest entry checked first. This may be a "typo", but I will double check with the LLD designer.

    Could you try just adding a pa_addMAC entry with the same Linux DEST ADDR but adding a more narrow set of filters (i.e. SRC ADDR, Ethertype, VLAN?). That way we keep the test simple without the use of UDP Port entries or Custom L2 entries.

    P.S. I would add that if you are using UDP port rules, then you want to make sure you link "your" MAC entry to the UDP entry (using the paHandleL2L3_t parameter on pa_addPort) to ensure proper parsing and filtering.

    Kind regards,

    Javier

  • 0 in reply to Javier Malave-Bonet
    TI__Expert 5805 points

    --

    The sub-system examines the L2 and L3 (LUT1) information (see netlayers) in packets based on internal table location. When function Pa_addMac and Pa_addIp are executed and the resulting packet forwarded to the sub-system, the sub-system places the new entries at the highest free table location [JMB - as I explained earlier]. When incoming packets are examined, the table is searched from lowest entry location to highest entry location until the first matching entry is found [JMB - contradicts what I explained earlier]. That entry is used to route the packet.

    Because of this it is required that entries into the table be made in order from the most general to the most specific [JMB - as I explained earlier]. For example, when adding a mac address it is common to want to route the following:

    • dest mac only - Forward packet to host
    • dest mac + ethertype - Continue parsing
    • dest mac + source mac + ethertype - Forward packet to host

    To get the desired routing the Pa_addMac commands must be executed and the command packets forwarded to the sub-system in the order shown above [JMB - contradicts what I explained earlier]. If they are entered in the reverse order then every packet which has the value dest MAC will be forwarded to the host since it matches the first entry in the list.

    The order dependency applies to calls to Pa_addMac and Pa_addIp, but not to calls between these functions. So all MAC entries can be made followed by all IP entries, or in the reverse order (provided the IP entries do not reference the MAC entries) without changing the operation of the sub-system.

    ----

     

    Resending as previous message got corrupted when posted.

  • 0 in reply to Javier Malave-Bonet
    TI__Expert 5805 points

    Hi Alan,


    I confirmed that the search happens from top of stack to bottom. (i.e. new entries first, old entries last). As I had explained earlier. Could you please try the pa_addMac simple rule I suggested?

    Kind regards,

    Javier

  • 0 in reply to Javier Malave-Bonet
    Intellectual 420 points

    Javier,

    I did as you suggested and added just one simple MAC rule using "dest addr + ethertype", where dest addr is the same MAC address being used by Linux.  The results are the same, I am not receiving any packets on the queue I specify.  Again, if I change the dest addr to a value different than the one used by Linux kernel, I have no problems receiving the packets on my queue.

    Perhaps there is some difference in the way Linux kernel driver is adding rules to the PA that is different than the PA LLDs that I am using?  Can the linux drivers insert rules into the very highest entries of the LUTs such that they are always evaluated first?