AFE5809: Consideration for calibration?

Team,

Customer is evaluating the AFE5809 and trying to do some calibration in order to scale the sampled voltage accordingly.

A given test pattern is provided but the converted values have a lot of variation depending on the LNA, VCAT and PGA settings used.

Could you shade some lights on how calibration should be performed? Is the methodology correct? What are the consideration to take into account?

A custom PCB with AFE5809 is used to measure dynamic voltage in a frequency range up to 450 kHz. The AFE5809A is configured as follows:
-Sample frequency 10 MHz.
-LNA, VCAT and PGA gains are set according to the table below.
-The demodulators decimation filter is enabled, the down conversion block and the DC removal block are bypassed.
-The DC shift (DEMOD register 0x1D) is set according to the scaling of the filter coefficients loaded into the coefficient RAM.
-Decimation factor is 10.
-The internally-generate reference voltages is used for the ADC.

The data is collected with an FPGA and forwarded to a windows host.
The goal is to calibrate or scale the data to get voltage.
However, with the scaling factors, i.e LSB (least significant bit) calculated from the gains set for LNA, VCAT and PGA  the measurement results are different (for the same signal) when using different input ranges.  

For example for a 100 kHz sine with a RMS of 35.4 mV I measurements of:
34.6 mV at +-500 mV range (1000 in table below)
33.3 mV at +-250 mV range (501 in table below)
33.0 mV at +-125 mV range (251 in table below)
31.9 mV at +-95 mV range (189 in table below)
30.3 mV at +-50 mV range (95 in table below)

We cannot guaranty the signal is exactly 35.4 mV rms but we would expect, that my results are at least equal, regardless which gain settings I use as long as no stage is overloaded.

Is the LSB calculation wrong? Or is the assumption that the gain matches exactly the value set is too optimistic?

max. Input Voltage (nominal)                                    
LNA    1000    mVpp    at 12 dB                            real maximum input range of LNA seems to be about 1250 mVpp
ADC    1500    mVpp    => LSB    0,022888            at    16    bit        32768    digits half range

Thanks in advance,

A.