amplifier ip2

The spectrum analyzer is single ended input, so we need to use transformer
to convert singal into single-ended format. But how to ensure the transformer
generate symmetrical result?

If you use a transformer, or balun, or differential to single ended amplifier in front of the spectrum analyzer, in each case you have to calibrate the gain of that converting device using careful measurements. You have to devise a set of experiments to accurately measure differential to single ended gain.