In mathematical statistics, the Kullback–Leibler (KL) divergence (also called relative entropy and I-divergence^[1]), denoted ${\displaystyle D_{\text{KL}}(P\parallel Q)}$, is a type of statistical distance: a measure of how one probability distribution P is different from a second, reference probability distribution Q.^[2][3] A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model when the actual distribution is P. While it is a measure of how different two distributions are, and in some sense is thus a "distance", it is not actually a metric, which is the most familiar and formal type of distance. In particular, it is not symmetric in the two distributions (in contrast to variation of information), and does not satisfy the triangle inequality. Instead, in terms of information geometry, it is a type of divergence,^[4] a generalization of squared distance, and for certain classes of distributions (notably an exponential family), it satisfies a generalized Pythagorean theorem (which applies to squared distances).^[5]

In the simple case, a relative entropy of 0 indicates that the two distributions in question have identical quantities of information. Relative entropy is a nonnegative function of two distributions or measures. It has diverse applications, both theoretical, such as characterizing the relative (Shannon) entropy in information systems, randomness in continuous time-series, and information gain when comparing statistical models of inference; and practical, such as applied statistics, fluid mechanics, neuroscience and bioinformatics.

Introduction and context

Consider two probability distributions P and Q. Usually, P represents the data, the observations, or a measured probability distribution. Distribution Q represents instead a theory, a model, a description or an approximation of P. The Kullback–Leibler divergence ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is then interpreted as the average difference of the number of bits required for encoding samples of P using a code optimized for Q rather than one optimized for P. Note that the roles of P and Q can be reversed in some situations where that is easier to compute, such as with the expectation–maximization algorithm (EM) and evidence lower bound (ELBO) computations.

Etymology

The relative entropy was introduced by Solomon Kullback and Richard Leibler in Kullback & Leibler (1951) as "the mean information for discrimination between ${\displaystyle H_{1}}$ and ${\displaystyle H_{2}}$ per observation from ${\displaystyle \mu _{1}}$",^[6] where one is comparing two probability measures ${\displaystyle \mu _{1},\mu _{2}}$, and ${\displaystyle H_{1},H_{2}}$ are the hypotheses that one is selecting from measure ${\displaystyle \mu _{1},\mu _{2}}$ (respectively). They denoted this by ${\displaystyle I(1:2)}$, and defined the "'divergence' between ${\displaystyle \mu _{1}}$ and ${\displaystyle \mu _{2}}$" as the symmetrized quantity ${\displaystyle J(1,2)=I(1:2)+I(2:1)}$, which had already been defined and used by Harold Jeffreys in 1948.^[7] In Kullback (1959), the symmetrized form is again referred to as the "divergence", and the relative entropies in each direction are referred to as a "directed divergences" between two distributions;^[8] Kullback preferred the term discrimination information.^[9] The term "divergence" is in contrast to a distance (metric), since the symmetrized divergence does not satisfy the triangle inequality.^[10] Numerous references to earlier uses of the symmetrized divergence and to other statistical distances are given in Kullback (1959, pp. 6–7, §1.3 Divergence). The asymmetric "directed divergence" has come to be known as the Kullback–Leibler divergence, while the symmetrized "divergence" is now referred to as the Jeffreys divergence.

Definition

For discrete probability distributions P and Q defined on the same sample space, ${\displaystyle \ {\mathcal {X}}\ ,}$ the relative entropy from Q to P is defined^[11] to be

${\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in {\mathcal {X}}}P(x)\ \log \left({\frac {\ P(x)\ }{Q(x)}}\right)\ ,}$

which is equivalent to

${\displaystyle D_{\text{KL}}(P\parallel Q)=-\sum _{x\in {\mathcal {X}}}P(x)\ \log \left({\frac {\ Q(x)\ }{P(x)}}\right)~.}$

In other words, it is the expectation of the logarithmic difference between the probabilities P and Q, where the expectation is taken using the probabilities P.

Relative entropy is only defined in this way if, for all x, ${\displaystyle \ Q(x)=0\ }$ implies ${\displaystyle \ P(x)=0\ }$ (absolute continuity). Otherwise, it is often defined as ${\displaystyle +\infty }$,^[1] but the value ${\displaystyle \ +\infty \ }$ is possible even if ${\displaystyle \ Q(x)\neq 0\ }$ everywhere,^[12][13] provided that ${\displaystyle \ {\mathcal {X}}\ }$ is infinite in extent. Analogous comments apply to the continuous and general measure cases defined below.

Whenever ${\displaystyle \ P(x)\ }$ is zero the contribution of the corresponding term is interpreted as zero because

${\displaystyle \lim _{x\to 0^{+}}x\log(x)=0~.}$

For distributions P and Q of a continuous random variable, relative entropy is defined to be the integral^[14]

${\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{-\infty }^{\infty }p(x)\ \log \left({\frac {p(x)}{q(x)}}\right)\ \mathrm {d} \ \!x\ ,}$

where p and q denote the probability densities of P and Q.

More generally, if P and Q are probability measures on a measurable space ${\displaystyle \ {\mathcal {X}}\ ,}$ and P is absolutely continuous with respect to Q, then the relative entropy from Q to P is defined as

${\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}\ \log \left({\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\right)\ P(\mathrm {d} \ \!x)\ ,}$

where ${\displaystyle \ {\frac {\ P(\mathrm {d} \ \!x)\ }{Q(\mathrm {d} \ \!x)\ }}}$ is the Radon–Nikodym derivative of P with respect to Q, i.e. the unique Q almost everywhere defined function r on ${\displaystyle \ {\mathcal {X}}\ }$ such that ${\displaystyle \ P(\mathrm {d} \ \!x)=r(x)Q(\mathrm {d} \ \!x)\ }$ which exists because P is absolutely continuous with respect to Q. Also we assume the expression on the right-hand side exists. Equivalently (by the chain rule), this can be written as

${\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}{\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\ \log \left({\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\right)\ Q(\mathrm {d} \ \!x)\ ,}$

which is the entropy of P relative to Q. Continuing in this case, if ${\displaystyle \mu }$ is any measure on ${\displaystyle {\mathcal {X}}}$ for which densities p and q with ${\displaystyle \ P(\mathrm {d} \ \!x)=p(x)\mu (\mathrm {d} \ \!x)\ }$ and ${\displaystyle \ Q(\mathrm {d} \ \!x)=q(x)\mu (\mathrm {d} \ \!x)\ }$ exist (meaning that P and Q are both absolutely continuous with respect to ${\displaystyle \ \mu \ }$), then the relative entropy from Q to P is given as

${\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}p(x)\ \log \left({\frac {\ p(x)\ }{q(x)}}\right)\ \mu (\mathrm {d} \ \!x)~.}$

Note that such a measure ${\displaystyle \mu }$ for which densities can be defined always exists, since one can take ${\displaystyle \ \mu ={\frac {1}{2}}\left(P+Q\right)\ }$ although in practice it will usually be one that in the context like counting measure for discrete distributions, or Lebesgue measure or a convenient variant thereof like Gaussian measure or the uniform measure on the sphere, Haar measure on a Lie group etc. for continuous distributions. The logarithms in these formulae are usually taken to base 2 if information is measured in units of bits, or to base e if information is measured in nats. Most formulas involving relative entropy hold regardless of the base of the logarithm.

Various conventions exist for referring to ${\displaystyle \ D_{\text{KL}}(P\parallel Q)\ }$ in words. Often it is referred to as the divergence between P and Q, but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be described as the divergence of P from Q or as the divergence from Q to P. This reflects the asymmetry in Bayesian inference, which starts from a prior Q and updates to the posterior P. Another common way to refer to ${\displaystyle \ D_{\text{KL}}(P\parallel Q)\ }$ is as the relative entropy of P with respect to Q or the information gain from P over Q.

Basic example

Kullback^[3] gives the following example (Table 2.1, Example 2.1). Let P and Q be the distributions shown in the table and figure. P is the distribution on the left side of the figure, a binomial distribution with ${\displaystyle N=2}$ and ${\displaystyle p=0.4}$. Q is the distribution on the right side of the figure, a discrete uniform distribution with the three possible outcomes ${\displaystyle x=}$ 0, 1, 2 (i.e. ${\displaystyle {\mathcal {X}}=\{0,1,2\}}$), each with probability ${\displaystyle p=1/3}$.

Two distributions to illustrate relative entropy

x	0	1	2
Distribution ${\displaystyle P(x)}$	${\displaystyle {\frac {9}{25}}}$	${\displaystyle {\frac {12}{25}}}$	${\displaystyle {\frac {4}{25}}}$
Distribution ${\displaystyle Q(x)}$	${\displaystyle {\frac {1}{3}}}$	${\displaystyle {\frac {1}{3}}}$	${\displaystyle {\frac {1}{3}}}$

Relative entropies ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ and ${\displaystyle D_{\text{KL}}(Q\parallel P)}$ are calculated as follows. This example uses the natural log with base e, designated ln to get results in nats (see units of information):

${\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\sum _{x\in {\mathcal {X}}}P(x)\ln \left({\frac {P(x)}{Q(x)}}\right)\\&={\frac {9}{25}}\ln \left({\frac {9/25}{1/3}}\right)+{\frac {12}{25}}\ln \left({\frac {12/25}{1/3}}\right)+{\frac {4}{25}}\ln \left({\frac {4/25}{1/3}}\right)\\&={\frac {1}{25}}\left(32\ln(2)+55\ln(3)-50\ln(5)\right)\approx 0.0852996,\end{aligned}}}$

${\displaystyle {\begin{aligned}D_{\text{KL}}(Q\parallel P)&=\sum _{x\in {\mathcal {X}}}Q(x)\ln \left({\frac {Q(x)}{P(x)}}\right)\\&={\frac {1}{3}}\ln \left({\frac {1/3}{9/25}}\right)+{\frac {1}{3}}\ln \left({\frac {1/3}{12/25}}\right)+{\frac {1}{3}}\ln \left({\frac {1/3}{4/25}}\right)\\&={\frac {1}{3}}\left(-4\ln(2)-6\ln(3)+6\ln(5)\right)\approx 0.097455.\end{aligned}}}$

Interpretations

Motivation

Illustration of the relative entropy for two normal distributions. The typical asymmetry is clearly visible.

In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value ${\displaystyle x_{i}}$ out of a set of possibilities X can be seen as representing an implicit probability distribution ${\displaystyle q(x_{i})=2^{-\ell _{i}}}$ over X, where ${\displaystyle \ell _{i}}$ is the length of the code for ${\displaystyle x_{i}}$ in bits. Therefore, relative entropy can be interpreted as the expected extra message-length per datum that must be communicated if a code that is optimal for a given (wrong) distribution Q is used, compared to using a code based on the true distribution P: it is the excess entropy.

${\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\sum _{x\in {\mathcal {X}}}p(x)\log {\frac {1}{q(x)}}-\sum _{x\in {\mathcal {X}}}p(x)\log {\frac {1}{p(x)}}\\[5pt]&=\mathrm {H} (P,Q)-\mathrm {H} (P)\end{aligned}}}$

where ${\displaystyle \mathrm {H} (P,Q)}$ is the cross entropy of P and Q, and ${\displaystyle \mathrm {H} (P)}$ is the entropy of P (which is the same as the cross-entropy of P with itself).

The relative entropy ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ can be thought of geometrically as a statistical distance, a measure of how far the distribution Q is from the distribution P. Geometrically it is a divergence: an asymmetric, generalized form of squared distance. The cross-entropy ${\displaystyle H(P,Q)}$ is itself such a measurement (formally a loss function), but it cannot be thought of as a distance, since ${\displaystyle H(P,P)=:H(P)}$ is not zero. This can be fixed by subtracting ${\displaystyle H(P)}$ to make ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ agree more closely with our notion of distance, as the excess loss. The resulting function is asymmetric, and while this can be symmetrized (see § Symmetrised divergence), the asymmetric form is more useful. See § Interpretations for more on the geometric interpretation.

Relative entropy relates to "rate function" in the theory of large deviations.^[20][21]

Arthur Hobson proved that relative entropy is the only measure of difference between probability distributions that satisfies some desired properties, which are the canonical extension to those appearing in a commonly used characterization of entropy.^[22] Consequently, mutual information is the only measure of mutual dependence that obeys certain related conditions, since it can be defined in terms of Kullback–Leibler divergence.

Properties

• Relative entropy is always non-negative,
  $${\displaystyle D_{\text{KL}}(P\parallel Q)\geq 0,}$$

  
  a result known as Gibbs' inequality, with ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ equals zero if and only if ${\displaystyle P=Q}$ as measures.

In particular, if ${\displaystyle P(dx)=p(x)\mu (dx)}$ and ${\displaystyle Q(dx)=q(x)\mu (dx)}$, then ${\displaystyle p(x)=q(x)}$ ${\displaystyle \mu }$-almost everywhere. The entropy ${\displaystyle \mathrm {H} (P)}$ thus sets a minimum value for the cross-entropy ${\displaystyle \mathrm {H} (P,Q)}$, the expected number of bits required when using a code based on Q rather than P; and the Kullback–Leibler divergence therefore represents the expected number of extra bits that must be transmitted to identify a value x drawn from X, if a code is used corresponding to the probability distribution Q, rather than the "true" distribution P.

• No upper-bound exists for the general case. However, it is shown that if P and Q are two discrete probability distributions built by distributing the same discrete quantity, then the maximum value of ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ can be calculated.^[23]
• Relative entropy remains well-defined for continuous distributions, and furthermore is invariant under parameter transformations. For example, if a transformation is made from variable x to variable ${\displaystyle y(x)}$, then, since ${\displaystyle P(dx)=p(x)\,dx={\tilde {p}}(y)\,dy={\tilde {p}}(y(x))|{\tfrac {dy}{dx}}(x)|\,dx}$ and ${\displaystyle Q(dx)=q(x)\,dx={\tilde {q}}(y)\,dy={\tilde {q}}(y)|{\tfrac {dy}{dx}}(x)|dx}$ where ${\displaystyle |{\tfrac {dy}{dx}}(x)|}$ is the absolute value of the derivative or more generally of the Jacobian, the relative entropy may be rewritten:
  $${\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\int _{x_{a}}^{x_{b}}p(x)\log \left({\frac {p(x)}{q(x)}}\right)\,dx\\[6pt]&=\int _{x_{a}}^{x_{b}}{\tilde {p}}(y(x))|{\frac {dy}{dx}}(x)|\log \left({\frac {{\tilde {p}}(y(x))\,|{\frac {dy}{dx}}(x)|}{{\tilde {q}}(y(x))\,|{\frac {dy}{dx}}(x)|}}\right)\,dx\\&=\int _{y_{a}}^{y_{b}}{\tilde {p}}(y)\log \left({\frac {{\tilde {p}}(y)}{{\tilde {q}}(y)}}\right)\,dy\end{aligned}}}$$

  
  where ${\displaystyle y_{a}=y(x_{a})}$ and ${\displaystyle y_{b}=y(x_{b})}$. Although it was assumed that the transformation was continuous, this need not be the case. This also shows that the relative entropy produces a dimensionally consistent quantity, since if x is a dimensioned variable, ${\displaystyle p(x)}$ and ${\displaystyle q(x)}$ are also dimensioned, since e.g. ${\displaystyle P(dx)=p(x)\,dx}$ is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory^[24] (such as self-information or Shannon entropy), which can become undefined or negative for non-discrete probabilities.
• Relative entropy is additive for independent distributions in much the same way as Shannon entropy. If ${\displaystyle P_{1},P_{2}}$ are independent distributions, and ${\displaystyle P(dx,dy)=P_{1}(dx)P_{2}(dy)}$, and likewise ${\displaystyle Q(dx,dy)=Q_{1}(dx)Q_{2}(dy)}$ for independent distributions ${\displaystyle Q_{1},Q_{2}}$ then
  $${\displaystyle D_{\text{KL}}(P\parallel Q)=D_{\text{KL}}(P_{1}\parallel Q_{1})+D_{\text{KL}}(P_{2}\parallel Q_{2}).}$$

  
• Relative entropy ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is convex in the pair of probability measures ${\displaystyle (P,Q)}$, i.e. if ${\displaystyle (P_{1},Q_{1})}$ and ${\displaystyle (P_{2},Q_{2})}$ are two pairs of probability measures then
  $${\displaystyle D_{\text{KL}}(\lambda P_{1}+(1-\lambda )P_{2}\parallel \lambda Q_{1}+(1-\lambda )Q_{2})\leq \lambda D_{\text{KL}}(P_{1}\parallel Q_{1})+(1-\lambda )D_{\text{KL}}(P_{2}\parallel Q_{2}){\text{ for }}0\leq \lambda \leq 1.}$$

  
• The Taylor expansion is ${\displaystyle D_{\text{KL}}(P\parallel Q)={\frac {1}{2}}\sum _{x}{\frac {(P(x)-Q(x))^{2}}{P(x)}}-{\frac {1}{3}}\sum _{x}{\frac {(P(x)-Q(x))^{3}}{P(x)^{2}}}+\cdots }$.

Duality formula for variational inference

The following result, due to Donsker and Varadhan,^[25] is known as Donsker and Varadhan's variational formula.

For alternative proof using measure theory, see.^[26]

Examples

Relation to metrics

While relative entropy is a statistical distance, it is not a metric on the space of probability distributions, but instead it is a divergence.^[4] While metrics are symmetric and generalize linear distance, satisfying the triangle inequality, divergences are asymmetric in general and generalize squared distance, in some cases satisfying a generalized Pythagorean theorem. In general ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ does not equal ${\displaystyle D_{\text{KL}}(Q\parallel P)}$, and while this can be symmetrized (see § Symmetrised divergence), the asymmetry is an important part of the geometry.^[4]

It generates a topology on the space of probability distributions. More concretely, if ${\displaystyle \{P_{1},P_{2},\ldots \}}$ is a sequence of distributions such that

${\displaystyle \lim _{n\to \infty }D_{\text{KL}}(P_{n}\parallel Q)=0}$,

then it is said that

${\displaystyle P_{n}{\xrightarrow {D}}Q}$.

Pinsker's inequality entails that

${\displaystyle P_{n}\xrightarrow {D} P\Rightarrow P_{n}\xrightarrow {TV} P}$,

where the latter stands for the usual convergence in total variation.

Relation to other quantities of information theory

Many of the other quantities of information theory can be interpreted as applications of relative entropy to specific cases.

Bayesian updating

In Bayesian statistics, relative entropy can be used as a measure of the information gain in moving from a prior distribution to a posterior distribution: ${\displaystyle p(x)\to p(x\mid I)}$. If some new fact ${\displaystyle Y=y}$ is discovered, it can be used to update the posterior distribution for X from ${\displaystyle p(x\mid I)}$ to a new posterior distribution ${\displaystyle p(x\mid y,I)}$ using Bayes' theorem:

${\displaystyle p(x\mid y,I)={\frac {p(y\mid x,I)p(x\mid I)}{p(y\mid I)}}}$

This distribution has a new entropy:

${\displaystyle \mathrm {H} {\big (}p(x\mid y,I){\big )}=-\sum _{x}p(x\mid y,I)\log p(x\mid y,I),}$

which may be less than or greater than the original entropy ${\displaystyle \mathrm {H} (p(x\mid I))}$. However, from the standpoint of the new probability distribution one can estimate that to have used the original code based on ${\displaystyle p(x\mid I)}$ instead of a new code based on ${\displaystyle p(x\mid y,I)}$ would have added an expected number of bits:

${\displaystyle D_{\text{KL}}{\big (}p(x\mid y,I)\parallel p(x\mid I){\big )}=\sum _{x}p(x\mid y,I)\log \left({\frac {p(x\mid y,I)}{p(x\mid I)}}\right)}$

to the message length. This therefore represents the amount of useful information, or information gain, about X, that has been learned by discovering ${\displaystyle Y=y}$.

If a further piece of data, ${\displaystyle Y_{2}=y_{2}}$, subsequently comes in, the probability distribution for x can be updated further, to give a new best guess ${\displaystyle p(x\mid y_{1},y_{2},I)}$. If one reinvestigates the information gain for using ${\displaystyle p(x\mid y_{1},I)}$ rather than ${\displaystyle p(x\mid I)}$, it turns out that it may be either greater or less than previously estimated:

${\displaystyle \sum _{x}p(x\mid y_{1},y_{2},I)\log \left({\frac {p(x\mid y_{1},y_{2},I)}{p(x\mid I)}}\right)}$ may be ≤ or > than ${\displaystyle \displaystyle \sum _{x}p(x\mid y_{1},I)\log \left({\frac {p(x\mid y_{1},I)}{p(x\mid I)}}\right)}$

and so the combined information gain does not obey the triangle inequality:

${\displaystyle D_{\text{KL}}{\big (}p(x\mid y_{1},y_{2},I)\parallel p(x\mid I){\big )}}$ may be <, = or > than ${\displaystyle D_{\text{KL}}{\big (}p(x\mid y_{1},y_{2},I)\parallel p(x\mid y_{1},I){\big )}+D_{\text{KL}}{\big (}p(x\mid y_{1},I)\parallel p(x\mid I){\big )}}$

All one can say is that on average, averaging using ${\displaystyle p(y_{2}\mid y_{1},x,I)}$, the two sides will average out.

Discrimination information

Relative entropy ${\textstyle D_{\text{KL}}{\bigl (}p(x\mid H_{1})\parallel p(x\mid H_{0}){\bigr )}}$ can also be interpreted as the expected discrimination information for ${\displaystyle H_{1}}$ over ${\displaystyle H_{0}}$: the mean information per sample for discriminating in favor of a hypothesis ${\displaystyle H_{1}}$ against a hypothesis ${\displaystyle H_{0}}$, when hypothesis ${\displaystyle H_{1}}$ is true.^[32] Another name for this quantity, given to it by I. J. Good, is the expected weight of evidence for ${\displaystyle H_{1}}$ over ${\displaystyle H_{0}}$ to be expected from each sample.

The expected weight of evidence for ${\displaystyle H_{1}}$ over ${\displaystyle H_{0}}$ is not the same as the information gain expected per sample about the probability distribution ${\displaystyle p(H)}$ of the hypotheses,

${\displaystyle D_{\text{KL}}(p(x\mid H_{1})\parallel p(x\mid H_{0}))\neq IG=D_{\text{KL}}(p(H\mid x)\parallel p(H\mid I)).}$

Either of the two quantities can be used as a utility function in Bayesian experimental design, to choose an optimal next question to investigate: but they will in general lead to rather different experimental strategies.

On the entropy scale of information gain there is very little difference between near certainty and absolute certainty—coding according to a near certainty requires hardly any more bits than coding according to an absolute certainty. On the other hand, on the logit scale implied by weight of evidence, the difference between the two is enormous – infinite perhaps; this might reflect the difference between being almost sure (on a probabilistic level) that, say, the Riemann hypothesis is correct, compared to being certain that it is correct because one has a mathematical proof. These two different scales of loss function for uncertainty are both useful, according to how well each reflects the particular circumstances of the problem in question.

Relationship to available work

Pressure versus volume plot of available work from a mole of argon gas relative to ambient, calculated as ${\displaystyle T_{o}}$ times the Kullback–Leibler divergence.

Surprisals^[33] add where probabilities multiply. The surprisal for an event of probability p is defined as ${\displaystyle s=k\ln(1/p)}$. If k is ${\displaystyle \left\{1,1/\ln 2,1.38\times 10^{-23}\right\}}$ then surprisal is in ${\displaystyle \{}$nats, bits, or ${\displaystyle J/K\}}$ so that, for instance, there are N bits of surprisal for landing all "heads" on a toss of N coins.

Best-guess states (e.g. for atoms in a gas) are inferred by maximizing the average surprisal S (entropy) for a given set of control parameters (like pressure P or volume V). This constrained entropy maximization, both classically^[34] and quantum mechanically,^[35] minimizes Gibbs availability in entropy units^[36] ${\displaystyle A\equiv -k\ln(Z)}$ where Z is a constrained multiplicity or partition function.

When temperature T is fixed, free energy (${\displaystyle T\times A}$) is also minimized. Thus if ${\displaystyle T,V}$ and number of molecules N are constant, the Helmholtz free energy ${\displaystyle F\equiv U-TS}$ (where U is energy and S is entropy) is minimized as a system "equilibrates." If T and P are held constant (say during processes in your body), the Gibbs free energy ${\displaystyle G=U+PV-TS}$ is minimized instead. The change in free energy under these conditions is a measure of available work that might be done in the process. Thus available work for an ideal gas at constant temperature ${\displaystyle T_{o}}$ and pressure ${\displaystyle P_{o}}$ is ${\displaystyle W=\Delta G=NkT_{o}\Theta (V/V_{o})}$ where ${\displaystyle V_{o}=NkT_{o}/P_{o}}$ and ${\displaystyle \Theta (x)=x-1-\ln x\geq 0}$ (see also Gibbs inequality).