• State Resolved
  • Locked Locked
  • Replies 30 replies
  • Subscribers 99 subscribers
  • Views 1221 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Original question:

AM6548: SR 2.0 PRU-ICSSG multicast address filter - operation and example/reference code

AM6548: SR 2.0 PRU-ICSSG multicast address filter support in Linux

Ian Willats
Prodigy 250 points
Part Number: AM6548


Tool/software:

Continuing the earlier related thread...

...I see that the Linux driver provided with the current AM65x Platform SDK (09.01.00.01) now implements multicast address filtering using the FDB.  However, reviewing the code in that driver has not answered (all) the questions outstanding from the previous thread.

Please would you clarify the following:

1: The driver defines (in icssg_switch_map.h) NUMBER_OF_FDB_BUCKET_ENTRIES as 4.  Am I right to assume that this must match the configuration of the FDB_BUCKET_SIZE field in the ICSSG_FDB_GEN_CFG1 register, which is 4 (field value 0b10) by default?  If not, how is the correct value of NUMBER_OF_FDB_BUCKET_ENTRIES chosen?

2: The driver also defines PRUETH_SWITCH_FDB_MASK as ((SIZE_OF_FDB / NUMBER_OF_FDB_BUCKET_ENTRIES) - 1) with SIZE_OF_FDB fixed to 2048.  This mask selects values in the range [0..512).  Is this mask required to correspond with the configuration of FDB_HASH_SIZE field in the FDB CFG1 register?  By default it looks to me as though it does not, since the FDB_HASH_SIZE field is 0b100 by default, selecting a hash size of 1024 rather than 512.  Is this a defect or are the two configurations unrelated?

3: From my previous thread:

Please would you also explain exactly the meaning of the following fields in the MII G RT FDB CFG1 register:

  • FDB_HASH_SIZE - is this related to the number of broadside slot entries?  does it have to match the mask (e.g. 0x1ff) used to convert the CRC over the multicast address and filter ID to a broadside slot index?  otherwise, what does it refer to?
  • FDB_BUCKET_SIZE - does this refer to the number of FDB entries which can be "attached" to the same hash value/broadside slot index?  otherwise, what does it refer to?

In the case where two different combinations of multicast address and filter ID hash to the same value (broadside slot index), how does FDB manage this collision?  Is the same FDB entry applied to all such combinations which hash to the same value?

4: The driver function emac_ndo_set_rx_mode_work() defined in icssg_prueth.c appears to erase all FDB entries.  As far as I can see this means that changing the receive configuration of one EMAC interface will obliterate the current multicast address filter configuration for the other EMAC interface on the same ICSSG - is that correct?  If so it seems to be a defect in the driver.  Is the proper way to do this to read-modify-write each FDB entry to reset the FID_C2 bit corresponding to the port being initialised, while leaving the bit corresponding to the other port unchanged?

5: From my previous thread:

  • In my experiments it seems that when peer ports on the same ICSSG are directly connected, inbound multicast frames are filtered differently to what happens if ports on different ICSSGs are directly connected.  In particular it seems that both port 0 and port 1 bits in the FDB entry need to be set in the first case.  Is that correct?  Please can you explain why this is so.

Thanks in advance for your help.

Ian

  • 0
    TI__Intellectual 1145 points

    Hello Ian,

    Please find below the answers to the questions:

    1. Yes, that is right.

    2. The mask will have to match the FDB_HASH_SIZE ( = SIZE_OF_FDB / NUMBER_OF_FDB_BUCKET_ENTRIES). Yes, this is a defect, as by default FDB_HASH_SIZE should be 512, when FDB_BUCKET_SIZE by default is 4.  Since SIZE_OF_FDB is fixed to 2048. This would be addressed in the AM64x 10.0 release, and we have no AM65x release planned.

    3. FDB_HASH_SIZE - This is the number of broadside slots. So yes, the Mask should match (FDB_HASH_SIZE -1).
      FDB_BUCKET_SIZE -  Correct. This is the number of FDB entries with the same hash value/broadside slot index.

    To clarify with an example: FDB_HASH_SIZE = 512 and FDB_BUCKET_SIZE  =4
    This means we have 512 broadside slots. And each slot has 4 FDB entries.

    Thus, when a collision occurs, then upto 4 unique FDB entries can be hashed to the same hash value. If there are greater than 4 combinations that have the same hash value (    or slot index), then there will be a replacement of last FDB bucket if that entry was non-static.

    4. Yes it erases all FDB entries, but also adds the FDB entries one-by-one back from the list of addresses maintained by the netdev. When there is a link change, say, then this driver function deletes the entire FDB table and then adds the modified value of entry back to FDB. Yes, Read-modify-write is the right way to update the FDB entry.

    5. Is this the right understanding of your query? -  A multicast frame is transmitted out of port A , looped back to port B and is to be received. This is same in both the same and different ICSSG case. Also can you explain the direction of traffic ( eg. Outbound ICSSG0 MAC2 -> inbound ICSSG 0 MAC1 ) and the bits set correspondingly in all cases?

    In dual-MAC mode, If inbound multicast frame is to be received on port1 (MAC1) then p1 bit needs be set and similarly for port 2 (MAC2) , p2 bit needs to be set.


    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    Hello Miriam,

    Thanks for your reply, that's very helpful.  To follow up:

    4: "it erases all FDB entries, but also adds the FDB entries one-by-one back from the list of addresses maintained by the netdev"

    But isn't it the case that erasing all FDB entries will affect multicast filtering for the /other/ EMAC on the same ICSSG, and this will not be restored when the multicast addresses are added for the /current/ EMAC?  In other words, I think this will disable receipt of any multicast frame on the /other/ EMAC, which is undesired behaviour.  Am I missing something?

    5: "A multicast frame is transmitted out of port A , looped back to port B and is to be received"  Yes.  In the case where A and B are on different ICSSGs this works as intended: the FDB entry corresponding to the destination multicast address must have the port B bit set to enable it to be received.  However, if A and B are on the same ICSSG the frame is not received unless the FDB entry for that MAC address has /both/ port B and port A bits set (even though the frame is not supposed to be received by port A).  This is undesired behaviour, at least from my point of view, because in principle the receive filtering should surely work independently of where the frame originated.

    Ian

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hello Ian,

    4. Yes, I understand your point, I shall clarify with our Linux Driver expert and revert back here.

    5. Yes, that is the right understanding, that port A bit does not have to be set in order for the multicast frame to be received on port B on the same ICSSG. From your previous thread "(Incidentally my experiments have been using EMAC firmware v02.02.12.01 as shipped with AM65x Processor SDK v08.06.00.047.)" We shall check this on our side and comeback by next week.

    Thanks and Regards,
    Miriam

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hi Ian,

    Ian Willats said:
    But isn't it the case that erasing all FDB entries will affect multicast filtering for the /other/ EMAC on the same ICSSG, and this will not be restored when the multicast addresses are added for the /current/ EMAC?  In other words, I think this will disable receipt of any multicast frame on the /other/ EMAC, which is undesired behaviour.  Am I missing something?

    Yes, this is the right understanding.

    Regarding the clarification on 5) this shall be tested at the earliest, as there was some delay since I was out-of-office for a few days too.

    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    Hello Miriam,

    Thanks for your message, and sorry it's taken me a while to reply.

    If I understand correctly, the relevant change is in the driver's emac_ndo_set_rx_mode_work function which now calls __dev_mc_sync instead of erasing all entries in the FDB.

    However, I don't think that achieves the goal of ensuring that there are no "random" entries in the FDB; as far as I can see, __dev_mc_sync will only remove entries with MAC addresses in the device's multicast address list which are considered stale, but it will not remove entries with MAC addresses that are not in the device's multicast address list.

    I believe the driver needs to initialise the FDB entries to a known, clean state, but it must only do so when bringing up the first interface associated with the PRU ICSSG.  In turn I think this requires the driver's shutdown routine to remove its current multicast addresses from the FDB, and it requires the driver to maintain a count of running interfaces (so that it can erase all FDB entries when transitioning from no running interfaces to one running interface).

    Or, better, the firmware should generalise the "erase all FDB entries" command so that it can specify that this should apply only to entries with specific port bit(s) set.

    Have I missed something?

    Ian

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hello Ian,

    Thanks for the query, I shall relay this to the Linux driver expert also. We shall review this scenario w.r.t the driver and firmware and get back to you.

    Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    TI__Intellectual 1145 points

    Hello Ian,

    Ian Willats said:
    the relevant change is in the driver's emac_ndo_set_rx_mode_work function which now calls __dev_mc_sync instead of erasing all entries in the FDB.

    Yes this is correct.

    I had a discussion with Linux expert and we have the following questions:

    a) What did you exactly mean by these "random" entries and how/when are they added to the FDB? Since all entries are either added by the Kernel during boot-up or during run-time an entry is added manually.

    b) Currently __dev_mc_sync will remove all the entries in the netDev list. Can you elaborate on what you mean by "stale" entry here ?
        Also all MAC addresses added to the FDB will be present in the netDev list. Can you explain which MAC addresses won't be present in the netDev list? 

    c)__dev_mc_unsync(ndev, icssg_prueth_del_mcast) in emac_ndo_stop is removing the multicast address from FDB and there is a count of the running interfaces here. Please check this here: git.ti.com/.../icssg_prueth.c

    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    >> a) What did you exactly mean by these "random" entries and how/when are they added to the FDB? Since all entries are either added by the Kernel during boot-up or during run-time an entry is added manually.

    I'm referring to the possibility that the FDB may contain entries which were not deliberately added by the EMAC driver - for example, if the contents of the FDB are not predictable after reset, or if some other agent (hardware or software) has previously written to the FDB before the driver brings up the first EMAC interface.  In other words, I'm assuming that the purpose of the original code which cleared the whole FDB was to ensure that it was guaranteed to be in a "clean" state before the driver adds multicast entries - which I think is correct, and good practice - although the original implementation was incorrect as we've already discussed and agreed.

    Even if the FDB is guaranteed to be cleared by hardware after a system reset, I think it is still necessary for the EMAC driver to perform this initialisation in case the FDB has been written by software before the EMAC interface(s) start.  For example, if the system is re-started without a system reset, after the re-start the FDB may contain entries which were added during the previous run.

    >> b) Currently __dev_mc_sync will remove all the entries in the netDev list. Can you elaborate on what you mean by "stale" entry here ?
        Also all MAC addresses added to the FDB will be present in the netDev list. Can you explain which MAC addresses won't be present in the netDev list? 

    As noted above, my concern is that the FDB may contain entries which were not put there by the EMAC driver, and therefore may refer to MAC addresses which the driver has no knowledge of.  These will not be cleared by the current code based on calling __dev_mc_sync, as I understand it.  Perhaps I've missed something?

    >> c)__dev_mc_unsync(ndev, icssg_prueth_del_mcast) in emac_ndo_stop is removing the multicast address from FDB and there is a count of the running interfaces here. Please check this here: git.ti.com/.../icssg_prueth.c

    Maybe so, but for the reasons mentioned above I believe the FDB is not being properly cleared to a known state.

    Thanks,

    Ian

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hello Ian,

    Ian Willats said:
    For example, if the system is re-started without a system reset, after the re-start the FDB may contain entries which were added during the previous run.

    In case the ICSSG Firmware is reloaded then the FDB contents will be cleared. And in the Dual-MAC mode the is no case where the hardware adds the entry to FDB before the driver brings the interface(s) up.
    And we do not expect any software agent to add an entry and that entry not be present in the netDev list. Hence the Erase FDB might not be needed in this case.

    However, in case you believe that there could be some scenario where stray/stale FDB entry is persistent and an Erase FDB is necessary, then we can discuss this here and/or over the meeting.

    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    Hello Miriam

    Miriam Santhosh Kadavan said:
    In case the ICSSG Firmware is reloaded then the FDB contents will be cleared

    Strictly speaking I guess the FDB is cleared when the ICSSG firmware is started (rather than loaded), right?

    In any case, if I've understood correctly I think this will cause problems in the following scenario, assuming two EMAC interfaces on the same ICSSG:

    * start interface 0 (will load and start firmware on PRUs for slice 0, which according to the above will clear the FDB contents)

    * add multicast address for interface 0 (will configure an FDB entry)

    * start interface 1 (will load and start firmware on PRUs for slice 1, which will clear the FDB contents again)

    At that point, the FDB entry configured for interface 0 multicast address will be lost, which is incorrect behaviour.

    Have I missed something?  How can the sequence (start first interface, add multicast address for first interface, start second interface) be supported correctly?

    Thanks,

    Ian

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hello Ian,

    Ian Willats said:

    At that point, the FDB entry configured for interface 0 multicast address will be lost, which is incorrect behaviour.

    Have I missed something?  How can the sequence (start first interface, add multicast address for first interface, start second interface) be supported correctly?

    Yes, The issue with the above sequence is that slice 1 firmware is loaded and started later in the sequence causing the FDB to be cleared. 
    The above sequence can be supported correctly by loading the firmware for both the slices when the first interface is started (i.e when running interfaces count goes from 0 to non-zero).

    Thus sequence would be:

    * start interface 0 (will load and start firmware on PRUs for slice 0 and slice1, which according to the above will clear the FDB contents)

    * add multicast address for interface 0 (will configure an FDB entry)

    * start interface 1

    Reference to the linux driver implementation: git.ti.com/.../icssg_prueth.c

    In emac_ndo_open()

    	if (!prueth->emacs_initialized) {
		if (prueth->emac[ICSS_SLICE0]) {
			ret = prueth_emac_start(prueth, prueth->emac[ICSS_SLICE0]);
			if (ret) {
				netdev_err(ndev, "unable to start fw for slice %d", ICSS_SLICE0);
				goto free_rx_irq;
			}
		}
		if (prueth->emac[ICSS_SLICE1]) {
			ret = prueth_emac_start(prueth, prueth->emac[ICSS_SLICE1]);
			if (ret) {
				netdev_err(ndev, "unable to start fw for slice %d", ICSS_SLICE1);
				goto halt_slice0_prus;
			}
		}
	}


    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    Hello Miriam,

    Thanks again for your comment.  I see the purpose of that recent change to the Linux driver, and the corresponding change in emac_ndo_stop, but I am nervous about the implications of the PRU firmware running before an EMAC device is opened and after is it closed, which is now a likely occurrence.  In your example, PRU firmware will be running on interface 1 before emac_ndo_open has been called for interface 1; if two interfaces have been opened, the PRU firmware will continue running on one interface after it is closed until the second interface is also closed.

    Given our other ongoing discussions about the PRU-ICSSG start-up and shut-down sequences, I am quite uncomfortable about making what seems to be a significant change to the relative sequence of events (as seen by interface 1 in your example) especially with respect to initialisation and finalisation of DMA channels.

    Can you provide any further information to reassure me that these changes to the start-up and shut-down sequences are guaranteed to operate correctly, please?

    Putting it another way, what are the pre-requisites which must be satisfied before the PRU firmware is started, and what can/cannot be changed while the PRU firmware is running?  Is there a user guide for the PRU EMAC firmware which documents such things?

    Ian

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hello Ian,

    If I understand correctly, your concern is that, if the DMA channels are not yet initialized for interface 1 while the firmware is running, it could cause issues if there are some incoming packets?

    However, this won't be an issue if the port is disabled until the DMA channels are initialized during open. And similarly if the port is disabled before the teardown of DMA channels during close.

    Also the reason it's necessary to load both the slice's firmware together is that, we have some feature implementation shared across two slices, like for timesync, and also the FDB is a single shared resource between both slices.

    Nevertheless, we shall also discuss the implications internally and get back with the details/information requested.

    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    Thanks again, Miriam.

    One follow-up question, if I may: is a new firmware version required to work with this revised start-up sequence, and if so what version is that?

    Ian

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hello Ian,

    Yes, you can use the latest firmware present in the 10.0 MCU+ SDK version for AM64x.

    Thanks and Regards,
    Miriam

  • 0 in reply to Ian Willats
    Prodigy 250 points

    Hello Miriam,

    Going back to one of my earlier questions:

    Ian Willats said:
    5: "A multicast frame is transmitted out of port A , looped back to port B and is to be received"  Yes.  In the case where A and B are on different ICSSGs this works as intended: the FDB entry corresponding to the destination multicast address must have the port B bit set to enable it to be received.  However, if A and B are on the same ICSSG the frame is not received unless the FDB entry for that MAC address has /both/ port B and port A bits set (even though the frame is not supposed to be received by port A).  This is undesired behaviour, at least from my point of view, because in principle the receive filtering should surely work independently of where the frame originated.
    Miriam Santhosh Kadavan said:
    5. Yes, that is the right understanding, that port A bit does not have to be set in order for the multicast frame to be received on port B on the same ICSSG. From your previous thread "(Incidentally my experiments have been using EMAC firmware v02.02.12.01 as shipped with AM65x Processor SDK v08.06.00.047.)" We shall check this on our side and comeback by next week.
    Miriam Santhosh Kadavan said:
    Regarding the clarification on 5) this shall be tested at the earliest, as there was some delay since I was out-of-office for a few days too.

    Please could you let me know the current status on this topic?

    Thanks,

    Ian

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hello Ian,

    As discussed in the meet, we shall reproduce this on our side and provide the status next week.
    Could you also try with the latest firmware available if you are able to reproduce the same issue, if not already?

    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    To confirm: I see the same (incorrect) behaviour when I test using EMAC firmware v02.02.14.03.

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hello Ian,

    Ian Willats said:
    5: "A multicast frame is transmitted out of port A , looped back to port B and is to be received" 

    Regarding the above query, can you share more details on the following:

    1. Is the device in Vlan-Aware or Unaware mode?
    2. Is the MC frame being sent out tagged or untagged?
    3. If it's tagged and vlan-aware, what is the VID-FID entry being added by Application/driver - specifically the Vlan-ID , FID, FID_C1 details? (relevant IOCTL that's used: ICSSG_PER_IOCTL_VLAN_SET_ENTRY)
    4. If it's untagged and vlan-aware, what are the default VLANs configured in this register? (6.5.14.12.20 MII_G_RT_FDB_DF_VLAN Register). And the corresponding FID, FIDC1 entry made for these VLAN IDs ?
    5. For the FDB entry made using the IOCTL (ICSSG_FDB_IOCTL_ADD_ENTRY),
      1. What are all the bits being set for FID_C2?
      2. what is the VLAN ID provided as input argument?
      3. Also what is the FID_C1 values configured for FID=0?

    Note:
    FID: Filtering ID value. Always mapped with Vlan-ID entry using the IOCTL ICSSG_PER_IOCTL_VLAN_SET_ENTRY
    FID_C1: Is the 8-bit value (hostMember, p1Member, p2Member, hostTagged, p1Tagged, p2Tagged, streamVid, floodToHost)
    FID_C2:  Is the 8-bit FDB entry value (hostMember, p1Member, p2Member, ageable, blocked, secure, touched, valid) 

    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    Hello Miriam,

    1: MII_G_RT_FDB_GEN_CFG1.SMEM_VLAN_OFFSET is zero, and MII_G_RT_FDB_GEN_CFG2.FDB_VLAN_EN is zero so I presume this is "VLAN-unaware" mode.

    2: Multicast frames in my test case do not include VLAN tags.

    3: N/A

    4: NA

    5: a: FID_C2 is set to 0x82 for (frames to be received by) EMAC #0 and 0x84 for EMAC #1; that is valid and either p1Member or p2Member respectively

        b: VLAN_ID is always zero

        c: I don't know what the FID_C1 value is; as far as I can see it's not directly set by the host, unless this is simply the filter ID in which case it is always 0.

    I hope that's helpful, but if you need anything else please let me know.

    Ian

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Thank you Ian for the details.

    Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    TI__Intellectual 1145 points

    Hello Ian,

    Can you please share the ICSSG0 dump in both cases:
    a) With only Rx Port bit set ( the case where the Multicast frame is not received )
    b) With the Rx port and Tx port bit set ( the case where the Multicast frame is received )

    ICSSG0 address range: 0x0B000000 to 0x0B03C100

    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    Hello Miriam - I've just sent the requested logs by e-mail.

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hello Ian, 

    We have identified a fix for the multicast filtering issue reported. The firmware patch with the fix and related details are sent over e-mail. Please let us know the results once you have tested this patch.

    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    Hello Miriam,

    As noted by e-mail, the updated firmware may have resolved the problem but seems to have introduced another which prevents the received frames being delivered to the host driver.

    Ian

  • 0 in reply to Ian Willats
    TI__Intellectual 1145 points

    Hello Ian,

    Thank you for the update on the test result, we shall look into this further and revert back.

    Thanks and Regards,
    Miriam

  • +1 in reply to Miriam Santhosh Kadavan
    TI__Intellectual 1145 points

    Hello Ian,

    Thank you for the update over e-mail that with the firmware patch2 the multicast test between EMAC ports of same ICSSG is passing now. 

    Ian Willats said:

    5: From my previous thread:

    • In my experiments it seems that when peer ports on the same ICSSG are directly connected, inbound multicast frames are filtered differently to what happens if ports on different ICSSGs are directly connected.  In particular it seems that both port 0 and port 1 bits in the FDB entry need to be set in the first case.  Is that correct?  Please can you explain why this is so.

    Can you please confirm if this has clarified/solved all your queries in this thread ? If yes, we can go ahead and mark this thread as resolved and close it.

    Thanks and Regards,
    Miriam

  • 0 in reply to Miriam Santhosh Kadavan
    Prodigy 250 points

    Hello Miriam,

    Yes, I believe that answers all my queries.  I would like to do some more thorough testing to be sure that everything is working correctly but I suggest you close this thread and I'll open a new one if anything unexpected crops up.

    Thanks again for your help,

    Ian