How to realize the isolation in the datasheet of the HMC547?

I am dealing with a receiver circuit, in which I have to use the HMC547 to realize an about 40dB isolation for the frequency from 4GHz to 20 GHz. According to the datasheet of the HMC547, the isolation would meet the demand. However, when I use the Rogers 4350 PCB material, the isolation is very bad, such as about 15dB at 4 GHz, less than 10 dB at 20 GHz. The measuring method is that one port of the SPDT ( such as RF1) is loaded by a 51 Ohm resistor, and the other port is sent into the spectrum. The RFC port is provided with the input frequency. The control circuit is OK. So what is wrong with the circuit? I have used some absorbative materials.

What is more, the circuit I designed is according to the evaluation borad circuit in the datasheet.

Rogers 4350 material should work fine at least up to 10GHz.
To get good isolation at high frequencies must have the right terminating impedance at the other port (where you placed a 51 ohms resistor). The 50 ohms load resistor should have minimum parasitic inductance (preferable is to place 3 x 150 ohms SMD resistors in parallel), and also the resistor should be placed just near the switch port, to eliminate any other parasitic reactances.
PCB layout, grounding, and placing the switch in a shielded box (cavity) can affect the isolation at microwave frequencies.