Why are the higher order modes in waveguide highly attenuated?

In fact, the higher modes in a waveguide are not attenuated; in special designs named "beam-guides" are essentially multimode waveguides. Their transmission loss is very low, for this they are used in satellite-antenna feeds. They look like big chimney pipes behind large antennas.
The extremely low loss is due to that only the "superficial" modes interact with pipe wall and this causes the loss. Modes close to pipe center propagate with no additional loss as their energy is not dissipated.
The problem rather is that exciting such higher modes is difficult as well as picking the power back at line end.
I do not have a good reference at hand but you can surely find some in Google.

I saw one harmonic attenuator. The waveguide is split in horizonal direction, and have tunable screws on both up and down cavity. The base wave have insertion loss about 0.3dB, and the 2th harmonic is attenuated about 40dB.

This is possible. Structures like that are called low-pass filters, harmonic rejection filters, etc.
Modes in waveguides and cavities offer incalculable options. Then you can combine them with coaxial and other waveguiding structures, and invent wonderful devices.