firsts mode of propagation in rectangular and circular

How are you modeling your waveguide? If you are using an off-the-shelf software package, I would read the documentation.

If you are using a homemade numerical code, you will be solving an eigen-value problem of the form A*x=b*x where b are the eigen-values. The eigen-value will give you the propagation constant of your modes perhaps with some scaling factor if your normalized anything. From, here you need to understand what eigen-values correspond to guided modes and which will be your fundamental mode. Eigen-value finding algorithms can be "told" to find the eigen-values with smallest complex component, largest real component, smallest real components, etc. You will need to search for whatever will limit the eigen-values you calculate to your guided modes.

-Tip