What is wrong with this mixer circuits? Puzzling and for help.

I have designed a mixer circuit, of which the main mixer IC is sim-14+ from the mini-circuits. The PCB board is made up mulitlayers: the toplayer and the second layer ( designed to be as the gnd plane for the microstrip in the toplayer), made up of RO4350B substrate, and is followed by the third layer and the bottomlayer ( also is designed to be gnd plane), made up of FR4. Between the 4350B and the FR4, there is some FR4 material, all of the three substates are glued or suppressed together to form a multilayer. The 50 OHm microstrip line is calculted by the ESSOF lincalc software. I have put many many through vias to improve the gnd integrity of the second plane. The bottom layer and the top layer has no soler resisting material and is plated with sn. When I test the circuit using the PCB mount SMA connectors( without the shielding box, for the PCB mount SMA providing the gnd connections by solder the outter leads to the top layer gnd), just as in the attachment, the result of conversion loss and the isolations between the three ports is similar to that in the datasheet.
However, when I put the circuit in the shieding box, the conversion loss is varying noticeably from 30 dB to 7 dB. I have added the LO power when the conversion loss doesn't improve. What is wrong with the circuit? the multilayer circuits won't work in the microwave bands, not less than 10 GHz such as 4 GHz? Some one would like to verify it? Any advice is welcome.
I have to design a multilayer for I have other components to solder to the final PCB. If you have designed the multilayer in the C-band, and it works well or doesn't works, or you can advise me some books or links, please tell me.

It sounds like you may have a resonance inside your shielded box. You could try putting some RF absorber material on one of the inside walls.

http://www.tdk.co.jp/tefe02/absorber.htm

You mentioned about a lot of ground vias that you put on your PCB layout, but what about grounding the shielded can? For wideband and high frequency the shield may need to be grounded and soldered all around, and using multiple through-via to all the bottom ground layers.
Also because this is a mixer design, any mismatch to one of the ports will deteriorate the circuit performances.

Thanks for all replies. I have mistaken the IF port and the RF port. It shows normally when I connect the right port. It is a low level mistake. Thanks again.