Excitement of Wave guide by magnetron, TEM, TE, TM mode propagation

I have been studying through microwave engineering books and RF cavity books trying to understand wave guide excitation, specifically to understand microwave emission and how the direction of emission can be controlled. In reading about wave guides I learned TEM do not propagate in wave guides, but TE and TM modes do. This is confusing to me, and I hoping someone could explain this to me. In a microwave oven, isn't there a wave guide after the magnetron directing the microwaves to the large closed area (oven)? What about resonant cavities; do they propagate TEM modes or just TE and TM modes too? I'm trying to understand this so that I can design a wave guide myself.

I do not believe you have studied anything. If so, you would certainly know!
Find basic textbooks on waveguides, to learn how EM waves propagate and what the modes are. You will also learn how to excite such waves and how transitions from TEM to TE and TM modes work.
Try Moreno, T., Microwave Design Data, 1948.
Many others are available.

a microwave oven has a magnetron tube to produce the power. The tube has two rf terminals, one is ground, one is a "probe" with RF power coming out of it. That probe is typically inserted into a waveguide that has a backshort around one quarter wavelength away on one side, and the opening to the microwave oven on the other. The piece of rectangular waveguide could be very long, but in practice, it is only a few inches long in a microwave oven.

I haven't studied enough to be correct. I've read Collin, Robert E., Foundations for Microwave Engineering, 2001, Pozar, David M., Microwave Engineering, 1998, and Wangler, Thomas P., RF Linear Accelerators. I'll look that book up as well as go through what I have again. Thank for letting me know what I should be looking for as I did not understand TEM modes can transition to TE or TM modes. Also, be careful of your tone of writing. Condescension isn't appreciated on this forum.

Do you have any other advice that will help me in designing a wave guide?

My tone of writing is proportional to your arguments. If you really studied the books you indicated, you should know better how EM modes occur in waveguides and cavities. It is impossible to instruct anyone of all many details only by writing on the Forum.
TEM to TE/TM transitions are clearly described in the above and other literature. THE TEM coaxial line enters a waveguide with TE10 mode in parallel with its electric line of force, or electrical component. Usually it is formed as a pin on the coaxial line end protruding into the longer waveguide wall. Transition is made more efficient by inserting the pin aprox. one-quarter wavelength from waveguide shorted end.
There are many varieties of such transitions between various modes. Again look for microwave books for detail. Try Harvey, MIcrowave Engineering, for example.

As far as I'm aware of, jiripolivka is acting courteously and polite in forum discussions. I guess he stumbled upon you "have been studying through microwave engineering books" and - apparently don't understand the nature of TEM waves.

To make it simple, TEM waves only propagate in a transmission line, two-wire line or coaxial with center conductor. In so far you can't have TEM waves in a microwave oven cavity.

The oven cavity is designed to get different and time varying modes for almost uniform heating, a mode mixer (a kind of fan with metal blades) is helping to achieve this purpose.

Regarding TEM (usually coaxial line) to wave guide transistion, I'm sure you've seen examples in your books. The most simple are a capacitive (open coaxial end) or inductive (shorted coaxial end) inserted into the waveguide, as already mentioned by bif44 for the magnetron coupler.