how much isolation is required

This depends by multiple factors, as: TX power, RX IP3, modulation type, RX/TX mode (full or half-duplex), RX/TX frequency spacing (if they are not on the same frequency).
Generally an isolation greater than 20dB between RX and TX antennas is necessarily.

There are 2 chief things.

First, you have to have enough isolation so that the transmitter does not blow up the receiver front end.

Second, transmitters put out a lot of broadband noise. That broadband noise can leak to the receiver front end, and cause the system to have something like a 40 dB noise figure! Either you have to turn off the transmitter amplifier when receiving, or have enough isolation and/or bandpass filtering so that the broadband transmit noise is 10 dB less than the the receiver's equivalent input noise. The equivalent input noise might be thermal noise, or environmental noise if you are in a big city.

Rich

Suppose duplex communication is happenig in two separate freq bands? then how i have to caculate the separation between tese two? Whatever you explained must be true for time division duplex, in frequency division duplex indepenent bands are there, so broad band noise is not a problem? if i keep bandpass filters sensitivity will be reduced? then how can i approach?

Duplex communications is still pretty narrowband. Lets say you are transmitting on 10 GHz and receiving on 10.4 GHz. That means that any noise coming out of the transmit amplifier at 10.4 GHz will potentially jam the receiver for low level signals.


A 10 GHz amp will still have significant gain/output power at 10.4 GHz. So you need a combination of insolation and filtering to keep that from happening.

Rich