According to the system user's guide, "After the boot process is complete, the external bus selection register must be modified only once".
The CSL writes to EBSR in the initialization procedure for each peripheral, possibly with different values. It would seem there is a weakness in its current design. There needs to be some sort of overall system initialization which is performed once.