Waveguide Cutoff in HFSS

I'm not sure what do you mean ...

but If you are referring to how to demonstrate that a certain propagation mode has a certain cut-off frequency you have just to excite this mode and simulate it on the frequency range of interest.

simple, draw a piece of waveguide, do a full-band simulation, then you will see a "high-pass" response, which represents "cut-off"

What if you want an actual number, say for an optimization analysis?

You can you the prorogation constant, which is the im(gamma(portNumber:mode)). If you could be more specific as to what you are trying to optimize about the waveguide it would be helpful.