AM3352: DRVVBUS never going high when port set to host

Yet another DRVVBUS topic here but this seems slightly different to everything else. This is a custom board that has two Bluetooth radios soldered on board so there are no external USB interfaces, both ports on the AM3352 should be set to host mode.

I don't do any USB initialization in u-boot, the previous generation of this product didn't use USB for the radios, I'm just using the same u-boot we always have which pretty much only does board detection and then tells the kernel what dtb to use. This is all working correctly. This is just about the only thing I can think of though... maybe I need to initialize USB in u-boot?

I'm using a 5.4.106 kernel that came from a TI SDK a couple years ago when we developed the previous gen of this board. Everything has been working great except the USB. On boot we get the follow errors:

(NULL device *) no device provided for PHY

am335x-phy-driver: probe of 47401300.usb-phy failed with error -22

(NULL device *) no device provided for PHY

am335x-phy-driver: probe of 47401b00.usb-phy failed with error -22

Stepping through hardware debug I noticed that the DRVVBUS pin are never driven high to turn on VBUS, which I guess make sense since the phy never seems to be brought up.

This is the dts section pertaining to USB, which is pulled from a couple other boards with the same thing... which is why I'm totally lost.

Pinmux: (I don't think I really need to do this?)

	usb0_drvvbus: usb0_drvvbus {
        pinctrl-single,pins = <
            AM33XX_IOPAD(0xa1c, PIN_OUTPUT_PULLDOWN | MUX_MODE0) //0x21c (PIN_OUTPUT_PULLUP | MUX_MODE0)
        >;
    }; 

    usb1_drvvbus: usb1_drvvbus {
        pinctrl-single,pins = <
            AM33XX_IOPAD(0xa34, PIN_OUTPUT_PULLDOWN | MUX_MODE0) //0x234 ( PIN_OUTPUT_PULLUP | MUX_MODE0 )
        >;
    };

And USB nodes:

&usb {
	status = "okay";
};

&usb_ctrl_mod {
	status = "okay";
};

&usb0_phy {
	status = "okay";
};

&usb1_phy {
	status = "okay";
};

&usb0 {
	pinctrl-0 = <&usb0_drvvbus>;
	status = "okay";
    dr_mode = "host";
};

&usb1 {
	pinctrl-0 = <&usb1_drvvbus>;
	status = "okay";
    dr_mode = "host";
};

&cppi41dma {
	status = "okay";
};

I'm going to start to dig into the am335x-phy-drive to see if I can back track but thought I'd post here to see if anyone had any ideas in the short-term.

Thanks!