Nomenclature

Arc distance, ${\displaystyle D,\,\!}$ is the minimum distance along the surface of sphere/ellipsoid calculated between two points, ${\displaystyle P_{1}\,\!}$ and ${\displaystyle P_{2}\,\!}$. Whereas, the tunnel distance, or chord length, ${\displaystyle D_{\textrm {t}}}$, is measured along Cartesian straight line. The geographical coordinates of the two points, as (latitude, longitude) pairs, are ${\displaystyle (\phi _{1},\lambda _{1})\,\!}$ and ${\displaystyle (\phi _{2},\lambda _{2}),\,\!}$ respectively. Which of the two points is designated as ${\displaystyle P_{1}\,\!}$ is not important for the calculation of distance.

Latitude and longitude coordinates on maps are usually expressed in degrees. In the given forms of the formulae below, one or more values must be expressed in the specified units to obtain the correct result. Where geographic coordinates are used as the argument of a trigonometric function, the values may be expressed in any angular units compatible with the method used to determine the value of the trigonometric function. Many electronic calculators allow calculations of trigonometric functions in either degrees or radians. The calculator mode must be compatible with the units used for geometric coordinates.

Differences in latitude and longitude are labeled and calculated as follows:

${\displaystyle {\begin{aligned}\Delta \phi &=\phi _{2}-\phi _{1};\\\Delta \lambda &=\lambda _{2}-\lambda _{1}.\end{aligned}}\,\!}$

It is not important whether the result is positive or negative when used in the formulae below.

"Mean latitude" is labeled and calculated as follows:

${\displaystyle \phi _{\mathrm {m} }={\frac {\phi _{1}+\phi _{2}}{2}}.\,\!}$

Colatitude is labeled and calculated as follows:

For latitudes expressed in radians:

${\displaystyle \theta ={\frac {\pi }{2}}-\phi$ ;\,\!}

For latitudes expressed in degrees:

${\displaystyle \theta =90^{\circ }-\phi .\,\!}$

Unless specified otherwise, the radius of the Earth for the calculations below is:

${\displaystyle R\,\!}$ = 6,371.009 kilometers = 3,958.761 statute miles = 3,440.069 nautical miles.

${\displaystyle D_{\,}\!}$ = Distance between the two points, as measured along the surface of the Earth and in the same units as the value used for radius unless specified otherwise.

Singularities and discontinuity of latitude/longitude

Longitude has singularities at the Poles (longitude is undefined) and a discontinuity at the ±180° meridian. Also, planar projections of the circles of constant latitude are highly curved near the Poles. Hence, the above equations for delta latitude/longitude (${\displaystyle \Delta \phi \!}$, ${\displaystyle \Delta \lambda \!}$) and mean latitude (${\displaystyle \phi _{\mathrm {m} }\!}$) may not give the expected answer for positions near the Poles or the ±180° meridian. Consider e.g. the value of ${\displaystyle \Delta \lambda \!}$ ("east displacement") when ${\displaystyle \lambda _{1}\!}$ and ${\displaystyle \lambda _{2}\!}$ are on either side of the ±180° meridian, or the value of ${\displaystyle \phi _{\mathrm {m} }\!}$ ("mean latitude") for the two positions (${\displaystyle \phi _{1}\!}$=89°, ${\displaystyle \lambda _{1}\!}$=45°) and (${\displaystyle \phi _{2}\!}$=89°, ${\displaystyle \lambda _{2}\!}$=−135°).

If a calculation based on latitude/longitude should be valid for all Earth positions, it should be verified that the discontinuity and the Poles are handled correctly. Another solution is to use n-vector instead of latitude/longitude, since this representation does not have discontinuities or singularities.

Spherical Earth projected to a plane

This formula takes into account the variation in distance between meridians with latitude:

${\displaystyle D=R{\sqrt {(\Delta \phi )^{2}+(\cos(\phi _{\mathrm {m} })\Delta \lambda )^{2}}},}$

where:

${\displaystyle \Delta \phi \,\!}$ and ${\displaystyle \Delta \lambda \,\!}$ are in radians;

${\displaystyle \phi _{\mathrm {m} }\,\!}$ must be in units compatible with the method used for determining ${\displaystyle \cos(\phi _{\mathrm {m} }).\,\!}$

To convert latitude or longitude to radians use

${\displaystyle 1^{\circ }=(\pi /180)\,\mathrm {radians} .}$

This approximation is very fast and produces fairly accurate result for small distances . Also, when ordering locations by distance, such as in a database query, it is faster to order by squared distance, eliminating the need for computing the square root.

Ellipsoidal Earth projected to a plane

The above formula is extended for ellipsoidal Earth:^[2]

${\displaystyle D={\sqrt {(M(\phi _{\mathrm {m} })\Delta \phi )^{2}+(N(\phi _{\mathrm {m} })\cos \phi _{\mathrm {m} }\Delta \lambda )^{2}}},}$

where ${\displaystyle M\,\!}$ and ${\displaystyle N\,\!}$ are the meridional and its perpendicular, or "normal", radii of curvature of Earth. See also "Geographic coordinate conversion" for their formulas.

Polar coordinate flat-Earth formula

${\displaystyle D=R{\sqrt {\theta _{1}^{2}\;{\boldsymbol {+}}\;\theta _{2}^{2}\;\mathbf {-} \;2\theta _{1}\theta _{2}\cos(\Delta \lambda )}},}$

where the colatitude values are in radians. For a latitude measured in degrees, the colatitude in radians may be calculated as follows: ${\displaystyle \theta ={\frac {\pi }{180}}(90^{\circ }-\phi ).\,\!}$

Tunnel distance

A tunnel between points on Earth is defined by a Cartesian line through three-dimensional space between the points of interest. The tunnel distance ${\displaystyle D_{\textrm {t}}=2R\sin {\frac {D}{2R}}}$ is the great-circle chord length and may be calculated as follows for the corresponding unit sphere:

${\displaystyle {\begin{aligned}&\Delta {X}=\cos(\phi _{2})\cos(\lambda _{2})-\cos(\phi _{1})\cos(\lambda _{1});\\&\Delta {Y}=\cos(\phi _{2})\sin(\lambda _{2})-\cos(\phi _{1})\sin(\lambda _{1});\\&\Delta {Z}=\sin(\phi _{2})-\sin(\phi _{1});\\&D_{\textrm {t}}=R{\sqrt {(\Delta {X})^{2}+(\Delta {Y})^{2}+(\Delta {Z})^{2}}}.\end{aligned}}}$

Or

${\displaystyle {\begin{aligned}D_{\textrm {t}}&=2R{\sqrt {\sin ^{2}{\frac {\Delta \phi }{2}}+\left(\cos ^{2}{\frac {\Delta \phi }{2}}-\sin ^{2}\phi _{\textrm {m}}\right)\sin ^{2}{\frac {\Delta \lambda }{2}}}}\\&=2R{\sqrt {\left(\sin {\frac {\Delta \lambda }{2}}\cos \phi _{\textrm {m}}\right)^{2}+\left(\cos {\frac {\Delta \lambda }{2}}\sin {\frac {\Delta \phi }{2}}\right)^{2}}}.\end{aligned}}}$

For short distances (${\displaystyle D\ll R}$), this underestimates the great circle distance by ${\displaystyle D(D/R)^{2}/24}$.

Lambert's formula for long lines

Lambert's formulae^[15] give accuracy on the order of 10 meters over thousands of kilometers. First convert the latitudes ${\displaystyle \scriptstyle \phi _{1}}$, ${\displaystyle \scriptstyle \phi _{2}}$ of the two points to reduced latitudes ${\displaystyle \scriptstyle \beta _{1}}$, ${\displaystyle \scriptstyle \beta _{2}}$

${\displaystyle \tan \beta =(1-f)\tan \phi ,}$

where ${\displaystyle f}$ is the flattening. Then calculate the central angle ${\displaystyle \sigma }$ in radians between two points ${\displaystyle (\beta _{1},\;\lambda _{1})}$ and ${\displaystyle (\beta _{2},\;\lambda _{2})}$ on a sphere using the Great-circle distance method (haversine formula), with longitudes ${\displaystyle \lambda _{1}\;}$ and ${\displaystyle \lambda _{2}\;}$ being the same on the sphere as on the spheroid.

${\displaystyle P={\frac {\beta _{1}+\beta _{2}}{2}}\qquad Q={\frac {\beta _{2}-\beta _{1}}{2}}}$

${\displaystyle X=(\sigma -\sin \sigma ){\frac {\sin ^{2}P\cos ^{2}Q}{\cos ^{2}{\frac {\sigma }{2}}}}\qquad \qquad Y=(\sigma +\sin \sigma ){\frac {\cos ^{2}P\sin ^{2}Q}{\sin ^{2}{\frac {\sigma }{2}}}}}$

${\textstyle \mathrm {distance} =a{\bigl (}\sigma -{\tfrac {f}{2}}(X+Y){\bigr )}}$

where ${\displaystyle a}$ is the equatorial radius of the chosen spheroid.

On the GRS 80 spheroid Lambert's formula is off by

0 North 0 West to 40 North 120 West, 12.6 meters

0N 0W to 40N 60W, 6.6 meters

40N 0W to 40N 60W, 0.85 meter

Gauss mid-latitude method for short lines

It has the form of the arc length converted from tunnel distance. Detailed formulas are given by Rapp,^[12] §6.4.

${\displaystyle D=2N\left(\phi _{\textrm {m}}\right)\arcsin {\sqrt {\left(\sin {\frac {\Delta \lambda }{2}}\cos \phi _{\textrm {m}}\right)^{2}+\left(\cos {\frac {\Delta \lambda }{2}}\sin {\frac {M\left(\phi _{\textrm {m}}\right)\Delta \phi }{2N\left(\phi _{\textrm {m}}\right)}}\right)^{2}}}.}$

Bowring's method for short lines

Bowring maps the points to a sphere of radius R′, with latitude and longitude represented as φ′ and λ′. Define

${\displaystyle A={\sqrt {1+e'^{2}\cos ^{4}\phi _{1}}},\quad B={\sqrt {1+e'^{2}\cos ^{2}\phi _{1}}},}$

where the second eccentricity squared is

${\displaystyle e'^{2}={\frac {a^{2}-b^{2}}{b^{2}}}={\frac {f(2-f)}{(1-f)^{2}}}.}$

The spherical radius is

${\displaystyle R'={\frac {\sqrt {1+e'^{2}}}{B^{2}}}a.}$

(The Gaussian curvature of the ellipsoid at φ₁ is 1/R′².) The spherical coordinates are given by

${\displaystyle {\begin{aligned}\tan \phi _{1}'&={\frac {\tan \phi }{B}},\\\Delta \phi '&={\frac {\Delta \phi }{B}}{\biggl [}1+{\frac {3e'^{2}}{4B^{2}}}(\Delta \phi )\sin(2\phi _{1}+{\tfrac {2}{3}}\Delta \phi ){\biggr ]},\\\Delta \lambda '&=A\Delta \lambda ,\end{aligned}}}$

where ${\displaystyle \Delta \phi =\phi _{2}-\phi _{1}}$, ${\displaystyle \Delta \phi '=\phi _{2}'-\phi _{1}'}$, ${\displaystyle \Delta \lambda =\lambda _{2}-\lambda _{1}}$, ${\displaystyle \Delta \lambda '=\lambda _{2}'-\lambda _{1}'}$. The resulting problem on the sphere may be solved using the techniques for great-circle navigation to give approximations for the spheroidal distance and bearing. Detailed formulas are given by Rapp,^[12] §6.5 and Bowring.^[13]