reflected wave in short and open termination
The same thing. All power is reflected. Difference between short circuit and open is that at short circuit the line voltage at load is zero and the current is double as it is at match. At open, the voltage at load is doubled and current is zero.
And open and a short circuit both have unity reflection coefficient magnitudes. The short has a 180 degree phase shift, while an open has a 0 degree phase shift.
It is sometimes hard to get a true open circuit fabricated. It is fairly common to fall somewhat short and have some radiation into free space, or some stray center conductor capacitance. Just leaving the sma connector open is NOT the same as a true "open circuit".
the picture shows an open termination and its incident and reflected waves (in phase in the open plane). Also, there is the addition of both signals to produce a standing wave (total voltage, in green). What does the "peak envelope voltage" refer to ?
The peak envelope voltage refer to the square of total voltage
this square product is to obtain effective voltages or what is the purpose ?
Well I don't think there is any physical meaning of this square product. It doesn't certainly refer to the total delivered power (as this is zero). But keep in mind that, when for example you want to find the reflection coefficient of a structure (in lab) you can do that by measuring the VSWR and also finding the distance of the first minimum of Voltage from the load. Taking the square product of voltage will make the minimums sharper, so it will become much easier to define this distance. But again I don't know if this is the actual meaning of the square product