SN74LV4046A: FSK Modulation / Demodulation Troubles

Part Number: SN74LV4046A

I'm trying to do binary FSK modulation / demodulation using two SN74LV4046A, one to modulate and one to demodulate.

I built a test circuit based on the example in SLAA618 (Implementation of FSK Modulation and Demodulation using CD74HC4046A).

I've also pored over SCHA003B (CMOS Phase-Locked-Loop Applications Using the CD54/74HC/HCT4046A and CD54/74HC/HCT7046A).

VCC = 5V

On both modulator and demodulator, I am using:
R1 = 4.42kOhm
R2 = open
C1 = 27pF

On demodulator, the low pass filter is connected as shown in figure 4 of above application note, with:
R3 = 36kOhm
C2 = 100pF

Input to the modulator is:
1.36V to represent binary 0 -> produces 10MHz at modulator VCO output
3.1V to represent binary 1 -> produces 20MHz at modulator VCO output

I am having trouble at the demodulator. I was hoping to recover approximately the same voltages as input to the modulator (or at least have good voltage separation), then use a Schmitt trigger logic gate to bring that output back to normal TTL levels.

I alternately tried using Phase Comparator 1, and Phase Comparator 2.

When using Phase Comparator 1:
* When input to modulator is 1.36V producing 10MHz, output voltage at DEMout is about 2.75V.
* When input to modulator is 3.1V producing 20MHz, output voltage at DEMout about 2.28V.

Therefore, there is not enough voltage separation, so the Schmitt trigger logic gate does not distinguish between the two states (its output is always high).

When using Phase Comparator 2:
* When input to modulator is 1.36V producing 10MHz, output voltage at DEMout seems to oscillate at high frequency (too high for my scope) between 880mV and 2V.
* When input to modulator is 3.1V producing 20MHz, output voltage at DEMout oscillates similarly between 2.2V and 4V.

This is producing totally meaningless output.

Suggestions?

Scope images:

All of the below are using Phase Comparator 1:

When input to modulator is 1.36V representing "binary 0":

Input to modulator VCO, 1.36V:

Modulated signal, 10MHz (9.8 MHz in this photo):

Output of Phase Comparator 1:

Output at DEMout:

When input to modulator is 3.1V representing "binary 1":

Input to modulator VCO, 3.1V:

Modulated waveform, 20 MHz:

Output of Phase Comparator 1:

Output at DEMout: