• State TI Thinks Resolved
  • Locked Locked
  • Replies 12 replies
  • Answers 1 answer
  • Subscribers 102 subscribers
  • Views 417 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Original question:

AM6548: PRU ICSSG (SR 2.0) receive stalls

AM6548: PRU-ICSSG MII G RT interface type configuration sequence/timing

Ian Willats
Prodigy 240 points
Part Number: AM6548
Other Parts Discussed in Thread: STRIKE

[Follow-up to AM6548: PRU ICSSG (SR 2.0) receive stalls.]

Since the last activity reported on the "parent" thread, we have:

  • updated the host driver for PRU-ICSSG EMAC to work with the current EMAC firmware
  • switched to use EMAC firmware v02.02.11.02 (as advised by TI)
  • confirmed that under the "receive stall" condition, the MII G RT hardware receive counters do not increment

This suggests that the receive stall involves a failure in the MII G RT and/or the connected PHY (TI DP83867).  With that in mind, we have:

  • carried out a detailed review of the hardware interface between the AM564x SoC and the Ethernet PHYs
  • verified PHY configuration (internal delays, etc.)
  • verified MDIO operation (timing, etc.)

Nothing was found to indicate any failures in operation of the PHY devices.

We have made some progress, though, by adjusting the sequence the host driver uses to initialise the MII G RT ICSSG G CFG register (offset 0 in the MII G RT register map).

Previously, this register was initialised in two stages:

  1. initialise all fields shared between interfaces (that is, all fields except MII[01]_MODE) - to be clear, this includes enabling TX_PRU, and both Tx and Rx L2 FIFOs
  2. initialise MII0_mode (to RGMII) when interface 0 is started
  3. initalise MII1_mode (to RGMII) when interface 1 is started

With that initialisation sequence, the receiver of one interface, typically slice 1, will go into the "stalled" state on start-up and will receive no frames (MII G RT hardware receive counters stuck at 0).

With the host driver changed to initialise the MII G RT ICSSG C CFG register in a single write, with no later update, we have not been able to reproduce the problem.

This suggests that leaving the MII G RT / PHY interface in a default configuration (MII) for some time before it is configured into the required mode (RGMII) may cause it "lock up" and fail to recover until a hardware reset occurs.

While this change to the driver seems to have stopped the immediate and frequent receive stall during start up, it is not clear whether this is in fact a correct solution to resolving the receive stall problem.

Please would you clarify whether there are undocumented requirements about the configuration sequence for MII G RT ICSSG C CFG - for example, MII[01]_mode fields must not be updated after Tx L2 FIFO and/or Rx L1 FIFO are enabled?  Is it expected that the MII G RT receive path may stall indefinitely if the MII_mode fields are updated after other aspects of MII G RT have been enabled?

At the moment we feel uncomfortable about assuming this change to the host driver is a correct and complete solution to the receive stall.  Clarification and advice from TI on this point would be much appreciated.

Thanks.

  • 0
    TI__Guru*** 135268 points

    Hello Ian Willats

    Thank you for the inputs.

    I am reassigning to the PRU expert.

    Regards,

    Sreenivasa

  • 0
    TI__Guru* 76435 points

    Hello Ian,

    I am sharing your questions with the firmware developers. Please ping the thread if I have not provided a response by Tuesday of next week.

    Regards,

    Nick

  • 0 in reply to Nick Saulnier
    Prodigy 240 points

    Hello Nick - nothing so far?

  • 0 in reply to Ian Willats
    TI__Guru* 76435 points

    Hello Ian,

    I sat down with the engineer who actually built the PRU_ICSSG (designed all the circuitry & logic, etc) to pick his brain. Keep in mind that it has been a few years since he touched the AM65x PRU_ICSSG design, so there may be some things he forgot. I am also sharing these notes with the firmware and Linux teams, and will let you know if I learn anything more from them. The designer said:

    1) MII G RT ICSSG C CFG only supports atomic 32 bit writes. The hardware looks for a write on the lowest bit before it applies the register settings. That means that a write that only affects some of the bits instead of all of the bits will not actually trigger an update.

    2) The MII G RT ICSSG C CFG register should probably only be written to once. He intended the register to be written before almost anything else was set up in the PRU Ethernet.

    3) I asked if there were any other potentially tricky registers to keep in mind. He pointed to TX0_IPG and TX1_IPG registers. Apparently the hardware will apply the settings from these two registers at the same time, but ONLY when IPG0 is written. So you should always program TX1_IPG register first, and then TX0_IPG second, and both settings will be applied after the TX0_IPG write.

    Regards,

    Nick

  • 0 in reply to Nick Saulnier
    Prodigy 240 points

    Hello Nick,

    Thanks for the update.

    To confirm, the driver is using 32-bit accesses to read and write the MII G RT ICSSG C CFG register, so from what you said there should be no possibility that the register update fails in the way you described in (1) above.

    It seems that (2) is the important point.  In particular it seems that the MII G can be upset by acitivy on the link to the PHY which does not correspond to the PHY mode configured in the ICSSG C CFG register, and once this occurs the MII G receive path effectively locks up until a hardware reset occurs.

    Our driver already handles the transmit IPG configuration "oddness", as we observed similar code in the Linux driver.

    Ian

  • 0 in reply to Nick Saulnier
    Prodigy 240 points

    Nick,

    A further comment on your point (2) above.  The Linux driver delivered with the Linux SDK 08.02.00.02 makes multiple writes to the register in question, for example in this code sequence from icssg_config_sr2:

    ...

    regmap_update_bits(prueth->miig_rt, ICSSG_CFG_OFFSET, ICSSG_CFG_DEFAULT, ICSSG_CFG_DEFAULT);
    icssg_miig_set_interface_mode(prueth->miig_rt, slice, emac->phy_if);

    ...

    The function icssg_miig_set_interface_mode reads the register, changes the value and writes it back to update one or other of the MII[01]_MODE fields depending on which slice the interface corresponds to.  That code sequence is essentially the same as what was in our driver when the problem arose, other than timing - in our case we have a longer time gap between the first write and subsequent writes.

    The Linux driver also writes to the same register again from icssg_class_disable which is called when the interface is brought up and down.

    That being the case, and assuming the LInux driver works correctly, it is not multiple writes to the ICSSG configuration register in themselves which cause the problem; rather I believe it is concerned with the sequence and/or timing of the ICSSG configuration due to those writes.  So my original question remains:

    "...[are there] undocumented requirements about the configuration sequence for MII G RT ICSSG C CFG - for example, MII[01]_mode fields must not be updated after Tx L2 FIFO and/or Rx L1 FIFO are enabled?  Is it expected that the MII G RT receive path may stall indefinitely if the MII_mode fields are updated after other aspects of MII G RT have been enabled?"

    Ian

  • 0 in reply to Ian Willats
    TI__Guru* 76435 points

    Hello Ian,

    Fair point, thank you for bringing it up. The write when the interface is brought down seems reasonable to me, but if we are doing multiple writes to the register at other points in time, that may be something we should double check. I will highlight your point to the Linux developer.

    Regards,

    Nick

  • 0 in reply to Nick Saulnier
    TI__Guru* 76435 points

    Hello Ian,

    From the Linux developer:

    I see that Linux driver does writes twice to MII G RT ICSSG C CFG register. But both of these writes take place even before PRU firmware is loaded/booted.

    1) In ndo_open(), emac_ndo_open() -> icssg_class_default() -> icssg_class_disable(), this to disable all the RX traffic and set all the registers to default values.
    2) While setting the MII G interface mode, emac_ndo_open() -> prueth_emac_start() -> icssg_config_sr2() -> icssg_miig_set_interface_mode, this is to set the MII G RT ICSSG C CFG register to the desired value ( MII mode, either RGMII or MII).

    Once the register is configured properly, the PRU firmware is booted and once the firmware is booted and running the CFG register is not touched.

    >> Do we want to adjust the Linux driver, or is the current programming intended?
    I think the current programming is intended.

    Does that answer your questions?

    Regards,

    Nick

  • 0 in reply to Nick Saulnier
    Prodigy 240 points

    Hello Nick,

    Since you asked, the questions at the end of my post of 9 Feb haven't been answered yet.

    The reason for pointing out that the Linux driver makes a similar sequence of writes to the MII G RT configuration register as our driver was to highlight that if there is a restriction imposed by the hardware (as you suggested) then the Linux driver is also violating that restriction.  That sequence of writes clearly caused a problem in our driver so there may be a latent defect in the Linux driver - possibly you don't see the failure in practice because the timing between writes is different.

    In any case, this does highlight again why proper documentation for the the hardware is required, and the Linux driver is no substitute for that.

    Ian

  • 0 in reply to Ian Willats
    TI__Guru* 76435 points

    Hello Ian,

    To confirm: Was your driver also doing multiple writes to the config register before the PRU firmware is loaded? Or were the writes happening at a different time?

    FYI the developer is on vacation the next two weeks, so driver-specific questions may take a while to get a response on. But at this point you are much more intimately familiar with the Linux driver than I am, so I appreciate your insight and questions as a way to sanity check our driver work so far.

    I hear you on hardware documentation. It is tricky to strike the right balance, since we don't really support customers developing custom Ethernet drivers and firmware. However, dedicated PRU networking third party developers do still need documentation. There are some small details that got lost between the IP designer's spec and the TRM, and I am taking actions to try to eliminate that difference. It may take a while for that to happen, however.

    Regards,

    Nick

  • 0 in reply to Nick Saulnier
    Prodigy 240 points

    Yes, the multiple writes in our driver to the MII G RT configuration register took place before the PRU firmware was loaded.

  • 0 in reply to Ian Willats
    TI__Guru* 76435 points

    Ok, thanks for confirming. I'll push the developer a bit more on this point once they return, because you're right, that does make it sound like there may be a latent bug in the TI driver.

    In terms of "undocumented requirements about the configuration sequence for MII G RT ICSSG C CFG", I have not found any guidance about a specific configuration sequence.

    Regards,

    Nick