AM335x two USB controllers which are used as a device/peripheral

As Biser mentioned, there is no technical limitation that would prevent this topology from working, but it would require a custom driver to actually function. Also keep in mind that with the exception of isochronous transfers, USB is not bandwidth-guaranteed, so PC to PC connections may be performance limited.

Ethernet would seem to be a better option as AM335x has an embedded 3-port Ethernet switch that could much more easily be configured for this task.

I have used this configuratin before, it works. The only issue, if you use Linux, is that both controllers cannot use the same gadget driver, because the Linux gadget driver cannot be loaded more than once.

Sorry, I overlooked part of your message.

Yes, both usb controllers can work in device mode, but if you want to use them to communicate between two PCs, as DK said, Ethernet is a better option, especially if you use Linux.