Single Side-band Performance

Hi,

I understand that SSB mixer is capable to suppress or cancels the unwanted sideband without the need of the filter. Unfortunately, the carrier and the unwanted sideband of my designed mixer is still high (as shown below). The different between the real peak and the image peak is -28dB. Is this mixer sufficient enough? is there any book to recommend? or keyword for me to search around?

You need better amplitude match and more accurate quadrature. What you have is not bad for a first attempt.
Local oscillator rejection relative to your wanted output will depend on signal level and mixer imbalance. Injecting DC into the IF port of the mixer can null the leakage. Getting it all right is an iterative process of adjusting LO leakage, signal phase and signal ampilitude. The narrower the bandwidth the easier it will be.
Search for IQ mixer sideband rejection, that should get you plenty of application note hits.

In the plot below you can see how amplitude and phase mismatch can affect your sideband rejection.

A well documentation about IQ Modulators an their pitfalls due to Amplitude and Phase Errors.
http://cds.linear.com/docs/en/applic...ote/an102f.pdf