Still looking for the RF spectrum analyzer, but in the meantime I probed both the TI and custom PWB's balun and matching networks. With both devices set to receive in SmartRF, I see a constant potential on the nodes of the TI board (about 0.92V), but on my custom PWB, the potential varies depending on where I probe the circuit. This behavior is indicative of an oscillation. Touching the circuit with a second probe while probing with my Fluke voltmeter would squelch the oscillation and the circuit would give similar readings to the TI PWB.
In a SPICE simulation of the balun and matching network, capacitors C66 and C70 provide control of out-of-band response and I replaced these 1.5pF caps with 4.7pF parts. See below.
The RF link now works. However, I still need to open the RX Filter BW to roughly three times the tool (SmartRF Studio 7) recommended value (on the RX side of this unidirectional link). I'm speculating that there might still be a low level oscillation. A sniffer probe on the spectrum analyzer might show this . . .
For the next spin of the board, I will reduce the IPC pads to half the area and open windows in the inner two layers (of my 4 layer PWB) to reduce parasitic capacitance. IPC pads for 0402 capacitors measure 27W x 34L mils. (They are sized for manufacturing yield and minimize tombstone failures.) My new pads will measure 27x16mils--still not as small as used by TI, but the pad extension beyond the parts aid hand assembly. As space permits, I will pepper the area around the balun and matching network with as many ground vias as space permits and use poured copper to with multiple vias to ground for connecting the circuit.
It's not clear to me whether TI opened the inner layers on their multi-layer PWB. Any TI RF engineers out there care to weigh in?