How could the transmitter gain loss in measurement?

Hi, all,

The TX is designed to be with 5dB gain from analog baseband to RF sections, but it turns out to be -5dB, while the trace loss and cable loss have been considered. that's to say, there are 10dB difference between simulation and measurement. I could not figure out that how could this happen, anybody can give some hints, thanks a lot.

You can see the S22 of the Tx and make sure it is well below -10 dB. If there is no proper matching at your frequency of operation, you should expect higher loss.

Impedance matching is important, but the same importance have using the proper biasing for your RF amplifier.
5dB of gain for an RF amplifier is unusual low, unless is about a high power solid-state amplifier.