How can I simulate the Q of balun by ADS~?

Q factor is nothing but Im part /Re part.

See at what point u want tot know Q-factor( if ur using indcutors find the Q-factor of the inductor alone)

Which tool that you are using?
ADS not support q model for balun, however you can do it with S-parameters

If you mean to extract the Q from simulation, then just resonate the balun at the designated frequency, and from the transfer function Q=freq/BW (3dB).