The design of current-commutating passive mixer

RF you would choose experimentally to get the right K = Imeasured/Vout

AFTER RF is chosen, you then choose CF to be the corner frequency where the gain starts decreasing, and has decreased by 3 dB (or is it 6 dB, since it is a voltage out?). In any event at that "corner frequency" = Fc:
Rf = 1/(2 * Pi * Cf * Fc)

So if i wanted to limit the 3 dB bandwidth to 1 MHz, and Rf was chosen at 1 K ohm, you would then choose Cf=1/(2*3.14*1000*1*10^6)=159 pF

I think it is a good way to choose RF first. I would do this:
There is a filter thumb rule, that if you are using a feedback topology the open loop gain of your circuit should be at least 20dB bigger at the cut-off frequency (Fc) than the closed loop gain at Fc.
The loop gain of the OTA is depending on the Gm*Rout, where Rout will be close to RF. So to find RF you have to know the Gm of the OTA, and the desired gain first.
At higher output voltage swings a higher current will flow through RF and CF, so you have to be sure that the OTA's output stage has enough output current capability. If it is not enough, increase RF's value or the current of the output stage and be sure that it doesn't decrease the gain at Fc.
Be cautious with too big values of RF because it can make your circuit noisy, and sensitivity of filter parameters will be higher for parasitics.
To find CF use the formula by biff44.

I presume that the circuit will be integrated.
If you choose larger capacitor, a placement restriction problem occurs due to limited space, if you choose larger resistance, it will increase the noise.
So, you have to choose an optimum one which maintains the corner frequency.

for example，fLO=2.398GHz，fRF=2.4GHz,so IF output frequency is 2MHz, then how to find the corner frequency ？

If the modulation is not I/Q, 2MHz+- BW of the Baseband Signal ( DSB) .If the modulation is I/Q ,either 2MHz+BW or 2MHz-BW (SSB).

I cann't understand why the the corner frequency can be 2MHz-BW. the IF signal wouldn't be attenuated too much？