REG711-33 droops when switching to boost mode

Perhaps you can post some waveforms that show your issue?

Channel-1 is the output of the REG711

Channel-2 is the input to the REG711 from a hold up cap

Channel-3 is just a reset chip that senses the low level on the output of the REG711

It would be good to check your output load, as it seems you are overloaded since Vout drops.

You can also test the EVM in your system to make sure that your layout and component selection are ok.