The third harmonic of the HMC497

I am now designing an IQ modulator. The main chip is HMC497, from 400MHz~4GHz. I use the E8267D signal generator of Agilent to transmit the two tune sinwaves as the I-channel, Q-channel. However, I find the third harmonic of the modulated signal is so large, just about 10dBc. For example, if I choose the LO frequency , from the E4438, to be 800MHz, there will be about -12 dBc spur in the 2400MHz, and about -18dBc spur in the 4000MHz. Would anyone tell me some suggestions to suppress such harmonics excluding the low pass filter? Is the output of the IQ direct modulator in some rectangluar wave in time domain? Thanks.

From the datasheet I see that inter-modulation products (and indirectly the harmonics) are increasingly a lot if the output power is greater than 0dBm.
So, try to decrease the input power, and optimize the LO input power.

For the usual device operation, a PA is involved and harmonic filtering can't be avoided anyway. So it's quite understandable, that Hittite didn't feel a need to achieve extraordinary harmonic supression, I think.

You have to filter. That is the nature of these things. either switched LPF bank, or a wadley loop bandpass approach.