Nonrefracting spherical atmosphere

Atmospheric effects on optical transmission can be modelled as if the atmosphere is concentrated in approximately the lower 9 km.

If atmospheric refraction is ignored, it can be shown from simple geometrical considerations (Schoenberg 1929, 173) that the path ${\displaystyle s}$ of a light ray at zenith angle ${\displaystyle z}$ through a radially symmetrical atmosphere of height ${\displaystyle y_{\mathrm {atm} }}$ above the Earth is given by

$${\displaystyle s={\sqrt {R_{\mathrm {E} }^{2}\cos ^{2}z+2R_{\mathrm {E} }y_{\mathrm {atm} }+y_{\mathrm {atm} }^{2}}}-R_{\mathrm {E} }\cos \,z\,}$$

or alternatively,

$${\displaystyle s={\sqrt {\left(R_{\mathrm {E} }+y_{\mathrm {atm} }\right)^{2}-R_{\mathrm {E} }^{2}\sin ^{2}z}}-R_{\mathrm {E} }\cos \,z\,}$$

where ${\displaystyle R_{\mathrm {E} }}$ is the radius of the Earth.

The relative air mass is then:

$${\displaystyle X={\frac {s}{y_{\mathrm {atm} }}}={\frac {R_{\mathrm {E} }}{y_{\mathrm {atm} }}}{\sqrt {\cos ^{2}z+2{\frac {y_{\mathrm {atm} }}{R_{\mathrm {E} }}}+\left({\frac {y_{\mathrm {atm} }}{R_{\mathrm {E} }}}\right)^{2}}}-{\frac {R_{\mathrm {E} }}{y_{\mathrm {atm} }}}\cos \,z\,.}$$

Homogeneous atmosphere

If the atmosphere is homogeneous (i.e., density is constant), the atmospheric height ${\displaystyle y_{\mathrm {atm} }}$ follows from hydrostatic considerations as:

$${\displaystyle y_{\mathrm {atm} }={\frac {kT_{0}}{mg}}\,,}$$

where ${\displaystyle k}$ is the Boltzmann constant, ${\displaystyle T_{0}}$ is the sea-level temperature, ${\displaystyle m}$ is the molecular mass of air, and ${\displaystyle g}$ is the acceleration due to gravity. Although this is the same as the pressure scale height of an isothermal atmosphere, the implication is slightly different. In an isothermal atmosphere, 37% (1/e) of the atmosphere is above the pressure scale height; in a homogeneous atmosphere, there is no atmosphere above the atmospheric height.

Taking ${\displaystyle T_{0}=\mathrm {288.15~K} }$, ${\displaystyle m=\mathrm {28.9644\times 1.6605\times 10^{-27}~kg} }$, and ${\displaystyle g=\mathrm {9.80665~m/s^{2}} }$ gives ${\displaystyle y_{\mathrm {atm} }\approx \mathrm {8435~m} }$. Using Earth's mean radius of 6371 km, the sea-level air mass at the horizon is

$${\displaystyle X_{\mathrm {horiz} }={\sqrt {1+2{\frac {R_{\mathrm {E} }}{y_{\mathrm {atm} }}}}}\approx 38.87\,.}$$

The homogeneous spherical model slightly underestimates the rate of increase in air mass near the horizon; a reasonable overall fit to values determined from more rigorous models can be had by setting the air mass to match a value at a zenith angle less than 90°. The air mass equation can be rearranged to give

$${\displaystyle {\frac {R_{\mathrm {E} }}{y_{\mathrm {atm} }}}={\frac {X^{2}-1}{2\left(1-X\cos z\right)}}\,;}$$

matching Bemporad's value of 19.787 at ${\displaystyle z}$ = 88° gives ${\displaystyle R_{\mathrm {E} }/y_{\mathrm {atm} }}$ ≈ 631.01 and ${\displaystyle X_{\mathrm {horiz} }}$ ≈ 35.54. With the same value for ${\displaystyle R_{\mathrm {E} }}$ as above, ${\displaystyle y_{\mathrm {atm} }}$ ≈ 10,096 m.

While a homogeneous atmosphere is not a physically realistic model, the approximation is reasonable as long as the scale height of the atmosphere is small compared to the radius of the planet. The model is usable (i.e., it does not diverge or go to zero) at all zenith angles, including those greater than 90° (see § Homogeneous spherical atmosphere with elevated observer). The model requires comparatively little computational overhead, and if high accuracy is not required, it gives reasonable results.^[5] However, for zenith angles less than 90°, a better fit to accepted values of air mass can be had with several of the interpolative formulas.

Variable-density atmosphere

In a real atmosphere, density is not constant (it decreases with elevation above mean sea level. The absolute air mass for the geometrical light path discussed above, becomes, for a sea-level observer,

$${\displaystyle \sigma =\int _{0}^{y_{\mathrm {atm} }}{\frac {\rho \,\left(R_{\mathrm {E} }+y\right)\mathrm {d} y}{\sqrt {R_{\mathrm {E} }^{2}\cos ^{2}z+2R_{\mathrm {E} }y+y^{2}}}}\,.}$$

Isothermal atmosphere

Several basic models for density variation with elevation are commonly used. The simplest, an isothermal atmosphere, gives

$${\displaystyle \rho =\rho _{0}e^{-y/H}\,,}$$

where ${\displaystyle \rho _{0}}$ is the sea-level density and ${\displaystyle H}$ is the pressure scale height. When the limits of integration are zero and infinity, the result is known as Chapman function. An approximate result is obtained if some high-order terms are dropped, yielding (Young 1974, p. 147),

$${\displaystyle X\approx {\sqrt {\frac {\pi R}{2H}}}\exp {\left({\frac {R\cos ^{2}z}{2H}}\right)}\,\mathrm {erfc} \left({\sqrt {\frac {R\cos ^{2}z}{2H}}}\right)\,.}$$

An approximate correction for refraction can be made by taking (Young 1974, p. 147)

$${\displaystyle R=7/6\,R_{\mathrm {E} }\,,}$$

where ${\displaystyle R_{\mathrm {E} }}$ is the physical radius of the Earth. At the horizon, the approximate equation becomes

$${\displaystyle X_{\mathrm {horiz} }\approx {\sqrt {\frac {\pi R}{2H}}}\,.}$$

Using a scale height of 8435 m, Earth's mean radius of 6371 km, and including the correction for refraction,

$${\displaystyle X_{\mathrm {horiz} }\approx 37.20\,.}$$

Polytropic atmosphere

The assumption of constant temperature is simplistic; a more realistic model is the polytropic atmosphere, for which

$${\displaystyle T=T_{0}-\alpha y\,,}$$

where ${\displaystyle T_{0}}$ is the sea-level temperature and ${\displaystyle \alpha }$ is the temperature lapse rate. The density as a function of elevation is

$${\displaystyle \rho =\rho _{0}\left(1-{\frac {\alpha }{T}}_{0}y\right)^{1/(\kappa -1)}\,,}$$

where ${\displaystyle \kappa }$ is the polytropic exponent (or polytropic index). The air mass integral for the polytropic model does not lend itself to a closed-form solution except at the zenith, so the integration usually is performed numerically.

Layered atmosphere

Earth's atmosphere consists of multiple layers with different temperature and density characteristics; common atmospheric models include the International Standard Atmosphere and the US Standard Atmosphere. A good approximation for many purposes is a polytropic troposphere of 11 km height with a lapse rate of 6.5 K/km and an isothermal stratosphere of infinite height (Garfinkel 1967), which corresponds very closely to the first two layers of the International Standard Atmosphere. More layers can be used if greater accuracy is required.^[6]

Refracting radially symmetrical atmosphere

When atmospheric refraction is considered, ray tracing becomes necessary (Kivalov 2007), and the absolute air mass integral becomes^[7]

$${\displaystyle \sigma =\int _{r_{\mathrm {obs} }}^{r_{\mathrm {atm} }}{\frac {\rho \,\mathrm {d} r}{\sqrt {1-\left({\frac {n_{\mathrm {obs} }}{n}}{\frac {r_{\mathrm {obs} }}{r}}\right)^{2}\sin ^{2}z}}}\,}$$

where ${\displaystyle n_{\mathrm {obs} }}$ is the index of refraction of air at the observer's elevation ${\displaystyle y_{\mathrm {obs} }}$ above sea level, ${\displaystyle n}$ is the index of refraction at elevation ${\displaystyle y}$ above sea level, ${\displaystyle r_{\mathrm {obs} }=R_{\mathrm {E} }+y_{\mathrm {obs} }}$, ${\displaystyle r=R_{\mathrm {E} }+y}$ is the distance from the center of the Earth to a point at elevation ${\displaystyle y}$, and ${\displaystyle r_{\mathrm {atm} }=R_{\mathrm {E} }+y_{\mathrm {atm} }}$ is distance to the upper limit of the atmosphere at elevation ${\displaystyle y_{\mathrm {atm} }}$. The index of refraction in terms of density is usually given to sufficient accuracy (Garfinkel 1967) by the Gladstone–Dale relation

$${\displaystyle {\frac {n-1}{n_{\mathrm {obs} }-1}}={\frac {\rho }{\rho _{\mathrm {obs} }}}\,.}$$

Rearrangement and substitution into the absolute air mass integral gives

$${\displaystyle \sigma =\int _{r_{\mathrm {obs} }}^{r_{\mathrm {atm} }}{\frac {\rho \,\mathrm {d} r}{\sqrt {1-\left({\frac {n_{\mathrm {obs} }}{1+(n_{\mathrm {obs} }-1)\rho /\rho _{\mathrm {obs} }}}\right)^{2}\left({\frac {r_{\mathrm {obs} }}{r}}\right)^{2}\sin ^{2}z}}}\,.}$$

The quantity ${\displaystyle n_{\mathrm {obs} }-1}$ is quite small; expanding the first term in parentheses, rearranging several times, and ignoring terms in ${\displaystyle (n_{\mathrm {obs} }-1)^{2}}$ after each rearrangement, gives (Kasten & Young 1989)

$${\displaystyle \sigma =\int _{r_{\mathrm {obs} }}^{r_{\mathrm {atm} }}{\frac {\rho \,\mathrm {d} r}{\sqrt {1-\left[1+2(n_{\mathrm {obs} }-1)(1-{\frac {\rho }{\rho _{\mathrm {obs} }}})\right]\left({\frac {r_{\mathrm {obs} }}{r}}\right)^{2}\sin ^{2}z}}}\,.}$$

Homogeneous spherical atmosphere with elevated observer

Air mass for elevated observer in homogeneous spherical atmosphere

In the figure at right, an observer at O is at an elevation ${\displaystyle y_{\text{obs}}}$ above sea level in a uniform radially symmetrical atmosphere of height ${\displaystyle y_{\mathrm {atm} }}$. The path length of a light ray at zenith angle ${\displaystyle z}$ is ${\displaystyle s}$; ${\displaystyle R_{\mathrm {E} }}$ is the radius of the Earth. Applying the law of cosines to triangle OAC,

$${\displaystyle {\begin{aligned}\left(R_{E}+y_{\text{atm}}\right)^{2}&=s^{2}+\left(R_{E}+y_{\text{obs}}\right)^{2}-2\left(R_{E}+y_{\text{obs}}\right)s\cos \left(180^{\circ }-z\right)\\&=s^{2}+\left(R_{E}+y_{\text{obs}}\right)^{2}+2\left(R_{E}+y_{\text{obs}}\right)s\cos z\end{aligned}}}$$

expanding the left- and right-hand sides, eliminating the common terms, and rearranging gives

$${\displaystyle {{s}^{2}}+2\left({R_{\text{E}}}+{y_{\text{obs}}}\right)s\cos z-2{R_{\text{E}}}{y_{\text{atm}}}-y_{\text{atm}}^{2}+2{R_{\text{E}}}{y_{\text{obs}}}+y_{\text{obs}}^{2}=0\,.}$$

Solving the quadratic for the path length s, factoring, and rearranging,

$${\displaystyle s=\pm {\sqrt {{{\left({R_{\text{E}}}+{y_{\text{obs}}}\right)}^{2}}{{\cos }^{2}}z+2{R_{\text{E}}}\left({y_{\text{atm }}}-{y_{\text{obs}}}\right)+y_{\text{atm}}^{2}-y_{\text{obs}}^{2}}}-({R_{\text{E}}}+{y_{\text{obs}}})\cos z\,.}$$

The negative sign of the radical gives a negative result, which is not physically meaningful. Using the positive sign, dividing by ${\displaystyle y_{\mathrm {atm} }}$, and cancelling common terms and rearranging gives the relative air mass:

$${\displaystyle X={\sqrt {{{\left({\frac {{R_{\text{E}}}+{y_{\text{obs}}}}{y_{\text{atm}}}}\right)}^{2}}{{\cos }^{2}}z+{\frac {2{R_{\text{E}}}}{y_{\text{atm}}^{2}}}\left({y_{\text{atm}}}-{y_{\text{obs}}}\right)-{{\left({\frac {y_{\text{obs}}}{y_{\text{atm}}}}\right)}^{2}}+1}}-{\frac {{R_{\text{E}}}+{y_{\text{obs}}}}{y_{\text{atm}}}}\cos z\,.}$$

With the substitutions ${\displaystyle {\hat {r}}=R_{\mathrm {E} }/y_{\mathrm {atm} }}$ and ${\displaystyle {\hat {y}}=y_{\mathrm {obs} }/y_{\mathrm {atm} }}$, this can be given as

$${\displaystyle X={\sqrt {{({\hat {r}}+{\hat {y}})}^{2}\cos ^{2}z+2{\hat {r}}(1-{\hat {y}})-{\hat {y}}^{2}+1}}\;-\;({\hat {r}}+{\hat {y}})\cos z\,.}$$

When the observer's elevation is zero, the air mass equation simplifies to

$${\displaystyle X={\sqrt {{{\left({\frac {R_{\text{E}}}{y_{\text{atm}}}}\right)}^{2}}\cos ^{2}z+{\frac {2R_{\text{E}}}{y_{\text{atm}}}}+1}}-{\frac {R_{\text{E}}}{y_{\text{atm}}}}\cos z\,.}$$

In the limit of grazing incidence, the absolute airmass equals the distance to the horizon. Furthermore, if the observer is elevated, the horizon zenith angle can be greater than 90°.

Maximum zenith angle for elevated observer in homogeneous spherical atmosphere