Re: power amplifier PAE at compression

I'd like to suggest to check not only the plot but also the raw matrix data. To check each response value and their stimulus index, you maybe could find more.
And just guess, if the ADS do the correct, the fold back on the power or PAE sweep maybe the indication of some un-stability at large signal drive. I mean the circuit stability.

Thanks for your contribution, this is what I though but I measured this PA and did not see any stability issue... It works pretty well and I need to improve the input matching at low frequency.

Any idea ?

obviously you don't get the essential of the broadband transmission line transformer balun. The transformer parameter used in your schematic looks not follow the theory.
I'd suggest you look for the book written by Servic and read the ADS component help to fine tune your input match.
I keep away from such type design for so long time that can't provide better suggestion.
Good luck.