Since the original topic is already locked and there is no way to answer there, I will write here about the solution I found. Hope admins can combine these topics.
We checked the pinout of the connectors, and it turned out to be non-standard. When using an adapter that swapped the upstream RX and TX, everything is in working order. Many thanks for the help.
It also matches the answer given by JMMN. Unfortunately, I read it a little bit later )
So be careful if you use this open source USB 3.0 hub schematic based on TUSB8020B.
