VIA fence and gap between vias

what is the relationship between VIA gap(distance of two via) and frequecy in RF PCBs?

how we select the via hole size and its distance from other vias?what is typical values?

The PCB vendor will usually want a minimum of 1=via_diamater/layer_thickness to get good plating of the via walls.

0.010" is usually the smallest diameter via you can get with conventional drilling. Smaller than 0.010" requires laser drilling.

The equivalent circuit model in your simulator will also have limits on W/H.

You can probably get away with an edge-to-edge spacing of 1 to 2 diameters but they must be staggered to prevent fault lines.

There are papers that show the resonance of via-fences; maybe look those up. However ideally you want them close as possible.

Your PCB vendor will have the final say with design rules.

I would start with spacing on the order of 1/10 wavelength at the highest frequency of interest. Smaller spacing is better and multiple rows of vias is better still. You might consider staggering the rows as well. None of this is particularly popular when you are trying to reduce cost.

My experience with via diameter is that the size is usually driven by manufacturing constraints. I have not seen a strong diameter to performance connection and been involved with designs where the via diameters ranged from around 10 mils to on the order of 50 mils or more. No clear practical guidelines emerged but the debate still rages on.

When all the opinions are expressed I would try to model your particular situation with a good EM (3D) tool like HFSS and then go with what the simulation tells you is the best solution.