edges causes bad s11 at high frequency

Hello , in a coupler design the discontinuities causes bad s11 at high frequencies,
i tried many ways of smoothing but its not sufficient.
what could be done?

Which substrate do you use for that design ? The substrate losses and radiation losses may be higher than expected.

I think your problem could also be due to the abrupt bends I can see in the picture. You should use instead mitered bend microstrips.