what cause modulation fail?

modulation index

Modulation mask failure can be much words about the reason.
Except VCO and PLL, PA control loop stability at low PCL is one important thing worthy to be studied.

Some PAs need large capacitor to ensure loop stability for all PCLs, especially Mid& low PCLs. Some PA using power feedback would have problem of dynamic range of power detector on use. Make sure you PA control loop works without fret
of low margin.

Hope it works!

I use RF3146, but on an other project which use the RF3146, it is OK on modulation.