Why use notch filter but not a attenuator?

When measuring GSM Tx spurious, the reason of use notch filter is that signal analyzer's dynamic rang is not enouugh.

I use R&S FSIQ7 which have noise floor at -88dBm(RBW 3M), and I want to measure a 30dBm EGSM signal.

First I use a notch filter to filte out main signal and measure the 2nd harmonic, all lose are considered.The 2nd harmonic is about -40dBm.

Since the 2nd harmonic is -40dBm and the GSM signal is 30dBm, if I use a 20dB attenuator ,the main signal and 2nd harmonic into the analyzer are 10dBm and -60dBm, these signal are in the analyzer's dynamic range.
So I use 20dBm attenuator, level offset set to 20dBm to shift back the signal, and measure in the same RBW, but the 2nd harmonic is greater than -20dBm!

Why the measured 2nd harmonic is so different

This usually happens because the 1st mixer is saturated by the carrier signal ( even if it is not existing on the screen, the 1st mixer is very wide band device ) and the mixer itself generates the harmonics.
Try this : increase the imput attenuator by 10 dB and check if the harmonics level decrease: if it is so the mixer itself is generating harmonics.

Mandi