Should the ground pin of analog mux be connected to digital ground or analog ground?
I think it should be connected to DGND, because DGND is the reference for the control pins which are digital signal. Although analog signal is in the main signal path, the ground pins has no relation with the main signal path.