How to convert BER to dB?

Universal formula is impossible, it depends from modulation type, demodulataor characteristics, FEC .... look any classical communication book about this.

Do you mean the relationship between
BER and SNR (or Q)?
codec, modulation, demodulation, noise distribution are essential. And usually for physical system, it's just impossible to give an express to estimate all non-ideal factors, like BW limitation, colorred noise, nonlinearity, etc. The best you can do is give an upper bound.