What's the energy received by an open end cable with the inner conductor?

Hi everyone. I am designing a circuit for IQ modulator, covering from 3.8GHz to 4GHz. The LO power is about +5dBm. However, when the LO power, ie. 0dBm, is fed to the LO input port, there is some energy showing in the ouput port about -70dBm, even in case I cut off the power of the whole circuit. The outputport is also SMA type. The smallest signal I have to provide from the output port is about -85dBm. Then I am sure that the -85dBm can't be achieved. When I use an open end cable, touching the ground shielding box, there is also some energy about -56dBm in the neigborhood ground of the input port, and about -60dBm in the neigborhood ground of the output port. These two ports are separated by a metal wall. What is the energy showing in/on the ground?
How can I realize the -85dBm output?
There is some tiny or minor clearence between the metal wall and the ground plane of the PCB. Is this clearence that ruins the isolation?
I have used the microwave absorption material, which did not work at all.
Would someone like to help me? Any reply is welcome.

I fear, there's no simple answer to the title question. You can have common mode ground plane waves, a shielding not completely enclosing the circuit can show resonances etc. It's difficult to achieve zero input/output coupling.

The problem can't be discussed without analyzing the exact circuit geometry, and the critical effects are not always obvious. It's a matter of RF engineering experience or "craftmanship".

As you did not say anything about your I/Q modulator, the situation is pretty typical in any mixer type. It is called LO/RF leakage. This is the reason we need to use a RF band-pass filter, to prevent the LO signal as well as IF input signal to appear in the RF output.

I understand the original post in the way, that the -70 dBm leakage is achieved with the board without a mixer in place. 75 dB LO suppression won't be bad for the mixer itself, I think.

In fact I have never seen a mixer with 75 dB suppression without filters