How to suppress the leakage of the VCO ?
- - - Updated - - -
Further more, the whole circuit is a four-layered PCB. The PLL is on the bottom layer, and the main circuit, including the mixers is on the toplayer. The second layer is a whole ground plane. The third one is also a ground plane, in which there are some control lines and some power supply lines.
Good luck. You have some of that power floating around on the board, and you can no longer filter it out.
If you really need 80 dBc type rejection numbers, then the VCO/X2/HPF have to be enclosed in their own metal shielded box. You might also have to put the mixer in another box, with another HPF right at the LO input.
You might have some luck with a true differential feed of the LO signal, so the fundamental does not get onto the ground plane so much.
Why you multiply 2.4Ghz by 2 ? Instead, why don't you realize 4.8GHz VCO directly ?
Multipliers are very nonlinear and harmonic producing components...
As Bigboss suggests getting rid of the multiplier and generating 4.8 GHz directly is a good solution.
For the current board you could try terminating the 2.4 GHz at the multiplier. You probably have a bandpass filter between the multiplier and the second mixer. This will pass 4.8 GHz and be reflective (not absorptive) to 2.4 GHz. Instead you could implement a diplexer (LPF/HPF) where the 4.8 GHz goes through the HPF to the mixer and the 2.4 GHz goes through the LPF where it is terminated with 50 ohms.