• State Resolved
  • Locked Locked
  • Replies 6 replies
  • Subscribers 97 subscribers
  • Views 448 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Original question:

AM5728: AM57 hang during PCIe shutdown

AM5749: AM57 hang during PCIe shutdown (v2)

Romain Naour
Intellectual 586 points

Part Number: AM5749

Tool/software:

Hello,

I noticed the same issue than a previous post "am5728-am57-hang-during-pcie-shutdown".

Indeed, on a custom AM5749 based board with both PCIe interfaces enabled, the kernel hang most of the time during a reboot.
The last kernel log is normal but nothing happen after that:
    

      [   57.943328] systemd-shutdown[1]: Rebooting.
      [   58.075653] reboot: Restarting system

The kernel reboot correctly each time as soon as both PCIe interfaces are disabled from the devicetree.
The kernel hang once with only one PCIe interface enabled (seems strange).

In order to reduce the bug occurrence, I applied the suggested workaround by adding 1ms in
ti_pipe3_power_off() before calling regmap_update_bits().

Note:
The second PCIe interface is limited to gen1 following a previous issue reported here am5749-pcie-pcie-scan-may-fail-on-pcie_ss2-when-pcie_ss1-and-pcie_ss2-are-in-single-line-mode.

The BSP use uboot 2022.01-ti, ti-rt-linux-6.1.y.

Best regards,
Romain

  • 0
    TI__Guru 62443 points

    Hi Romain,

    Could you put a dev_dbg statement in each of the functions listed in this previous E2E post and share logs?  AM5728: AM57 hang during PCIe shutdown 

    I would like to know the sequence at which the functions are called by both PCIe interface (aka, whether the pci_write_config_word is called for both PCIe interfaces before SerDes shutdown, or whether the full shutdown sequence occurs for one interface before going to next interface, etc).

    Regards,

    Takuma

  • +1 in reply to Takuma Fujiwara
    Intellectual 586 points

    Hello Takuma,

    Its seems to be a timing issue, adding dev_dbg statement may have some side effects (I'll have a look).
    IIUC, there is a race in dra7xx_pcie_shutdown() between dra7xx_pcie_stop_link() (last time we write into the PCIe configuration) and dra7xx_pcie_disable_phy().
    I had only once a kernel hang with only one PCIe interface enabled... it's really difficult to reproduce it.

    dra7xx_pcie_shutdown()
{
	dra7xx_pcie_stop_link(dra7xx->pci);
	ret = pm_runtime_put_sync(dev);
	pm_runtime_disable(dev);

	+ mdelay(1);

	dra7xx_pcie_disable_phy(dra7xx);

    But I don't see such delay in other dwc based PCIe driver (imx6...).

    Best regards,
    Romain

  • 0 in reply to Takuma Fujiwara
    Intellectual 586 points

    Hello Takuma,

    As expected, adding debug print also "fix" the issue. I was not able to reproduce the kernel hang with debug logs.

    The full shutdown sequence occurs for one interface before going to next interface:

    [   42.219573] systemd-shutdown[1]: Rebooting.

    [   42.292327] pci_device_shutdown:
    [   42.295562] pcie_portdrv_remove:
    [   42.298828] pcie_port_device_remove:
    [   42.302734] pci_disable_device:
    [   42.305908] do_pci_disable_device:
    [   42.309356] do_pci_disable_device: clear PCI_COMMAND_MASTER
    [   42.315002] dra7xx_pcie_shutdown:
    [   42.318359] dra7xx_pcie_stop_link:
    [   42.321899] dra7xx_pcie_disable_phy:


    [   42.326660] pci_device_shutdown:
    [   42.329895] pcie_portdrv_remove:
    [   42.333160] pcie_port_device_remove:
    [   42.337005] pci_disable_device:
    [   42.340179] do_pci_disable_device:
    [   42.343627] do_pci_disable_device: clear PCI_COMMAND_MASTER
    [   42.349273] dra7xx_pcie_shutdown:
    [   42.352630] dra7xx_pcie_stop_link:
    [   42.356170] dra7xx_pcie_disable_phy:

    [   42.421417] reboot: Restarting system

    Best regards,
    Romain

  • 0 in reply to Romain Naour
    TI__Guru 62443 points

    Hi Romain,

    Thanks for the information and experiment.

    Could you try adding delay after do_pci_disable_device, similar to the suggestion made by Jian in past E2E thread? Let us know if that fixes the issue.

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    Intellectual 586 points

    Hello Takuma,

    I added the 1ms delay in dra7xx_pcie_shutdown() (after do_pci_disable_device()) just before calling dra7xx_pcie_disable_phy() (see my previous reply).
    With that the hang is gone.

    For now this workaround is enough but it would be great to fix it upstream.
    But we have to convince upstream maintainers to add this experimental delay :-/

    Best regards,
    Romain

  • 0 in reply to Romain Naour
    TI__Guru 62443 points

    Hi Romain,

    Ah, apologies. I had missed your previous reply. 

    I'll see if our SDK team can integrate into the SDK and upstream, but this will take some time. AM57x is quite old, so most new development is done on the next generation of AM6x devices. Nevertheless, it is good that we have this issue in the forums with a confirmed fix so that it is searcheable by others that may have the issue.

    Regards,

    Takuma