CC1101: CC1101 async receives perfectly at 433 and 868 MHz, but has a poor "sensitivity" at 303 MHz

My understanding is that the sensitivity is good on all 3 RF frequency bands when doing conducted measurements. When doing range testing, however, the 303 MHz solution is significantly worse than 868 MHz. This points to the antenna performance being worse on 303 MHz vs 868 MHz so this is were I would focus the debugging. The AGC works on the down-converted signal in the digital domain and this is the same regardless of RF frequency so I do not agree with your assumption that the degraded performance is due to the AGC.

The IF ADC has a finite dynamic range. At the lower end there will be 1/f noise and at the high end there will be quantization noise. The ADC noise transfer function is not flat between the 1/f and quantization noise and the IF should be as low as possible for best sensitivity (lowest SNR). The RX filter BW is centered around the IF.and should be >=RX filter BW / 2 to get away from the 1/f noise. At the same time it cannot be too high as you want to avoid the quantization noise If you have FSCTRL1 = 0x4 the IF is approx 100 kHz. This means that an RX filter BW of 270 kHz is too wide. Use the IF frequency suggested by SmartRF Studio for the RX filter BW you are using.

If you don't follow the design steps in DN022 you might get a solution which might have unstable reception. That is there might be regions in the input dynamic range where you start to get packet loss (there is a figure in the design note showing this). I am not saying our settings are not good, but make sure you test for input power levels ranging from the sensitivity limit up to the saturation level.  

There are 2 VCO’s (called “low” and “high”). Difference is the on-chip inductor value. The two VCO’s are connected to a MUX.

When TEST0.VCO_SEL_CAL_EN = 0, the VCO to use is set manually through FSCAL2[5] (0 = low, 1 = high).

When TEST0.VCO_SEL_CAL_EN = 1, the VCO used by CC1101 is always the lower one. FSCAL2[5] is dont't care in this case.

In SmartRF Studio we set FSCAL2.VCO_CORE_H_EN = 1 for both TEST0.VCO_SEL_CAL_EN = 0 and TEST0.VCO_SEL_CAL_EN = 1, but this setting is only relevant for TEST0.VCO_SEL_CAL_EN = 0.

Frequency range 779- 928.  

freq >= 861 => TEST0.VCO_SEL_CAL_EN = 0 & FSCAL2.VCO_CORE_H_EN = 1
freq < 861 => TEST0.VCO_SEL_CAL_EN = 1 & FSCAL2.VCO_CORE_H_EN = 1
(Alternative: TEST0.VCO_SEL_CAL_EN = 0 & FSCAL2.VCO_CORE_H_EN = 0 )

High VCO goes up to 928 x 6 MHz

Low VCO goes down to 779 x 6 MHz

There is an overlap between the two VCO’s around 861 x 6 MHz.

The AGC adjusts down the gain very fast, but increases the gain rather slowly. If this were not the case OOK demodulation would not work as the logic 0’s and logic 1’s (low and high power respectively) would have the same amplitude internal to the chip.

Your frame does not include a preamble nor a sync word. The first frame will then most likely not be received correctly across the full input dynamic range as the AGC needs a few bits to settle. I assume you send 2 or more frames with the same information (at least that is common in some older protocols). One thought is then to treat the first frame as a “preamble” and read the AGC gain at the end of the frame through register AGCTEST. Next freeze the AGC gain by setting AGCCTRL0. AGC_FREEZE[1:0] = 3 and write back AGCTEST values.

AGCTEST[7:6]: First LNA. “00” is max gain; “10” is minimum gain

AGCTEST[5:3]: Second LNA. “000” is mimimum gain; “111” is maximum gain

AGCTEST[2:0]: Digital LNA. “000” is minimum gain; “111” is max gain