How to best seal a directional coupler to meet gross leak requirements
Just to give some answer...
Could flexing of the enclosure ever occur? Either in assembly (tightening fasteners), or in the field? Then you want to use a sealant that stays flexible.
Automotive silicone is what I think of. Engines are an extreme environment. There is vibrating, high temp, low temp, high pressure, solvents, etc.
For underwater exposure, use the type made for marine equipment.
If it will be subject to high pressure (or vacuum) then consider plumbing sealants. Screw threads will need a sealant such as thread-lock compound or teflon tape.
If it will be subject to high temperatures then use the type made to go around engine parts (or on the exhaust system).
Don't know about cold temperature limits.
Consider whether the enclosure must be opened in the field. Will a technician be able to remove old sealant, clean all surfaces, and apply new sealant? Consider making a gasket of suitable material that will be easy to remove and re-install.
You do not specify the leakage level. Good commercial couplers are sealed by machining a notch in body rim under the lid and filling the notch with either a wire cord or a special conductive plastic. You can also use the silver epoxy before tightening the lid. Using multiple lid screws separated by < 1 cm distance should be good for 2-18 GHz. Use four-screw connector flange for the connectors, or threaded connectors, also secure the contact by silver Epoxy. Even good connectors leak ~ 90-110 dBc; using silver Epoxy on them can also help.
You will need leak proof connectors (either hermetic seal, or maybe oring seal). You could try liberally gooping up the cover to body interface with RTV--assuming there is enough width past the cover hold down screws to form a seal.
Then gross leak test it as your specification says, and see if it leaked!
Worst case, you can go to true hermetic connectors (like sparkplugs with knife edge seals) and a laser welded cover.