Using frame differencing

A motion detection algorithm begins with the segmentation part where foreground or moving objects are segmented from the background. The simplest way to implement this is to take an image as background and take the frames obtained at the time t, denoted by I(t) to compare with the background image denoted by B. Here using simple arithmetic calculations, we can segment out the objects simply by using image subtraction technique of computer vision meaning for each pixels in I(t), take the pixel value denoted by P[I(t)] and subtract it with the corresponding pixels at the same position on the background image denoted as P[B].

In mathematical equation, it is written as:

${\displaystyle P[F(t)]=P[I(t)]-P[B]}$

The background is assumed to be the frame at time t. This difference image would only show some intensity for the pixel locations which have changed in the two frames. Though we have seemingly removed the background, this approach will only work for cases where all foreground pixels are moving, and all background pixels are static.^[2] A threshold "Threshold" is put on this difference image to improve the subtraction (see Image thresholding):

${\displaystyle |P[F(t)]-P[F(t+1)]|>\mathrm {Threshold} }$

This means that the difference image's pixels' intensities are 'thresholded' or filtered on the basis of value of Threshold. ^[3] The accuracy of this approach is dependent on speed of movement in the scene. Faster movements may require higher thresholds.

Mean filter

For calculating the image containing only the background, a series of preceding images are averaged. For calculating the background image at the instant t:

${\displaystyle B(x,y,t)={1 \over N}\sum _{i=1}^{N}V(x,y,t-i)}$

where N is the number of preceding images taken for averaging. This averaging refers to averaging corresponding pixels in the given images. N would depend on the video speed (number of images per second in the video) and the amount of movement in the video.^[4] After calculating the background B(x,y,t) we can then subtract it from the image V(x,y,t) at time t = t and threshold it. Thus the foreground is:

${\displaystyle |V(x,y,t)-B(x,y,t)|>\mathrm {Th} }$

where Th is a threshold value. Similarly, we can also use median instead of mean in the above calculation of B(x,y,t).

Usage of global and time-independent thresholds (same Th value for all pixels in the image) may limit the accuracy of the above two approaches.^[2]

Running Gaussian average

For this method, Wren et al.^[5] propose fitting a Gaussian probabilistic density function (pdf) on the most recent ${\displaystyle n}$ frames. In order to avoid fitting the pdf from scratch at each new frame time ${\displaystyle t}$, a running (or on-line cumulative) average is computed.

The pdf of every pixel is characterized by mean ${\displaystyle \mu _{t}}$ and variance ${\displaystyle \sigma _{t}^{2}}$ . The following is a possible initial condition (assuming that initially every pixel is background):

${\displaystyle \mu _{0}=I_{0}}$

${\displaystyle \sigma _{0}^{2}=\langle {\text{some default value}}\rangle }$

where ${\displaystyle I_{t}}$ is the value of the pixel's intensity at time ${\displaystyle t}$. In order to initialize variance, we can, for example, use the variance in x and y from a small window around each pixel.

Note that background may change over time (e.g. due to illumination changes or non-static background objects). To accommodate for that change, at every frame ${\displaystyle t}$, every pixel's mean and variance must be updated, as follows:

${\displaystyle \mu _{t}=\rho I_{t}+(1-\rho )\mu _{t-1}}$

${\displaystyle \sigma _{t}^{2}=d^{2}\rho +(1-\rho )\sigma _{t-1}^{2}}$

${\displaystyle d=|(I_{t}-\mu _{t})|}$

Where ${\displaystyle \rho }$ determines the size of the temporal window that is used to fit the pdf (usually ${\displaystyle \rho =0.01}$ ) and ${\displaystyle d}$ is the Euclidean distance between the mean and the value of the pixel.

Gaussian distribution for each pixel

We can now classify a pixel as background if its current intensity lies within some confidence interval of its distribution's mean:

${\displaystyle {\frac {|(I_{t}-\mu _{t})|}{\sigma _{t}}}>k\longrightarrow {\text{foreground}}}$

${\displaystyle {\frac {|(I_{t}-\mu _{t})|}{\sigma _{t}}}\leq k\longrightarrow {\text{background}}}$

where the parameter ${\displaystyle k}$ is a free threshold (usually ${\displaystyle k=2.5}$ ). A larger value for ${\displaystyle k}$ allows for more dynamic background, while a smaller ${\displaystyle k}$ increases the probability of a transition from background to foreground due to more subtle changes.

In a variant of the method, a pixel's distribution is only updated if it is classified as background. This is to prevent newly introduced foreground objects from fading into the background. The update formula for the mean is changed accordingly:

${\displaystyle \mu _{t}=M\mu _{t-1}+(1-M)(I_{t}\rho +(1-\rho )\mu _{t-1})}$

where ${\displaystyle M=1}$ when ${\displaystyle I_{t}}$ is considered foreground and ${\displaystyle M=0}$ otherwise. So when ${\displaystyle M=1}$ , that is, when the pixel is detected as foreground, the mean will stay the same. As a result, a pixel, once it has become foreground, can only become background again when the intensity value gets close to what it was before turning foreground. This method, however, has several issues: It only works if all pixels are initially background pixels (or foreground pixels are annotated as such). Also, it cannot cope with gradual background changes: If a pixel is categorized as foreground for a too long period of time, the background intensity in that location might have changed (because illumination has changed etc.). As a result, once the foreground object is gone, the new background intensity might not be recognized as such anymore.

Background mixture models

Mixture of Gaussians method approaches by modelling each pixel as a mixture of Gaussians and uses an on-line approximation to update the model. In this technique, it is assumed that every pixel's intensity values in the video can be modeled using a Gaussian mixture model.^[6] A simple heuristic determines which intensities are most probably of the background. Then the pixels which do not match to these are called the foreground pixels. Foreground pixels are grouped using 2D connected component analysis.^[6]

At any time t, a particular pixel (${\displaystyle x_{0},y_{0}}$)'s history is:

${\displaystyle X_{1},\ldots ,X_{t}=\{V(x_{0},y_{0},i):1\leqslant i\leqslant t\}}$

This history is modeled by a mixture of K Gaussian distributions:

${\displaystyle P(X_{t})=\sum _{i=1}^{K}\omega _{i,t}N\left(X_{t}\mid \mu _{i,t},\sigma _{i,t}\right)}$

where:

${\displaystyle N\left(X_{t}\mid \mu _{i,t},\sigma _{i,t}\right)={\dfrac {1}{(2\pi )^{D/2}}}{1 \over |\sigma _{i,t}|^{1/2}}\exp \left(-{1 \over 2}(X_{t}-\mu _{i,t})^{T}\sigma _{i,t}^{-1}\left(X_{t}-\mu _{i,t}\right)\right)}$

First, each pixel is characterized by its intensity in RGB color space. Then probability of observing the current pixel is given by the following formula in the multidimensional case:

${\displaystyle P(X_{t})=\sum _{i=1}^{K}\omega _{i,t}\eta \left(X_{t},\,\mu _{i,t},\sigma _{i,t}\right)}$

Where K is the number of distributions, ω is a weight associated to the ith Gaussian at time t and µ, Σ are the mean and standard deviation of said Gaussian respectively.

${\displaystyle \eta \left(X_{t},\,\mu _{i,t},\sigma _{i,t}\right)={\dfrac {1}{(2\pi )^{D/2}}}{1 \over |\sigma _{i,t}|^{1/2}}\exp \left(-{1 \over 2}(X_{t}-\mu _{i,t})^{T}\sigma _{i,t}^{-1}\left(X_{t}-\mu _{i,t}\right)\right)}$

Once the parameters initialization is made, a first foreground detection can be made then the parameters are updated. The first B Gaussian distribution which exceeds the threshold T is retained for a background distribution:

${\displaystyle B=\operatorname {argmin} \left(\Sigma _{i=1}^{B}\omega _{i,t}>T\right)}$

The other distributions are considered to represent a foreground distribution. Then, when the new frame incomes at times ${\displaystyle t+1}$, a match test is made of each pixel. A pixel matches a Gaussian distribution if the Mahalanobis distance:

${\displaystyle \left(\left(X_{t+1}-\mu _{i,t}\right)^{T}\sigma _{i,t}^{-1}\left(X_{t+1}-\mu _{i,t}\right)\right)^{0.5}<k\cdot \sigma _{i,t}}$

where k is a constant threshold equal to ${\displaystyle 2.5}$. Then, two cases can occur:

Case 1: A match is found with one of the k Gaussians. For the matched component, the update is done as follows:^[7]

${\displaystyle \sigma _{i,t+1}^{2}=\left(1-\rho \right)\sigma _{i,t}^{2}+\rho \left(X_{x+1}-\mu _{x+1}\right)\left(X_{x+1}-\mu _{x+1}\right)^{T}}$

Power and Schoonees [3] used the same algorithm to segment the foreground of the image:

${\displaystyle \sigma _{i,t+1}=\left(1-\alpha \right)\omega _{i,t}+\alpha P\left(k\mid X_{t},\varphi \right)}$

The essential approximation to ${\displaystyle P\left(k\mid \ X_{t},\varphi \right)}$ is given by ${\displaystyle M_{k,t}}$:^[8]

${\displaystyle M_{k,t}={\begin{cases}1&{\text{match}},\\0&{\text{otherwise}}.\end{cases}}}$

Case 2: No match is found with any of the ${\displaystyle K}$ Gaussians. In this case, the least probable distribution ${\displaystyle K}$ is replaced with a new one with parameters:

${\displaystyle k_{i.t}={\text{low prior weight}}}$

${\displaystyle \mu _{i,t+1}=X_{t+1}}$

${\displaystyle \sigma _{i.t+1}^{2}={\text{large initial variance}}}$

Once the parameter maintenance is made, foreground detection can be made and so on. An on-line K-means approximation is used to update the Gaussians. Numerous improvements of this original method developed by Stauffer and Grimson^[6] have been proposed and a complete survey can be found in Bouwmans et al.^[7] A standard method of adaptive backgrounding is averaging the images over time, creating a background approximation which is similar to the current static scene except where motion occur.

Comparisons

Several comparison/evaluation papers can be found in the literature:

• A. Sobral, A. Vacavant. "A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos". Computer Vision and Image Understanding, CVIU 2014, 2014.
• A. Shahbaz, J. Hariyono, K. Jo, "Evaluation of Background Subtraction Algorithms for Video Surveillance", FCV 2015, 2015.
• Y. Xu, J. Dong, B. Zhang, D. Xu, "Background modeling methods in video analysis: A review and comparative evaluation', CAAI Transactions on Intelligence Technology, pages 43–60, Volume 1, Issue 1, January 2016.

Books

• T. Bouwmans, F. Porikli, B. Horferlin, A. Vacavant, Handbook on "Background Modeling and Foreground Detection for Video Surveillance: Traditional and Recent Approaches, Implementations, Benchmarking and Evaluation", CRC Press, Taylor and Francis Group, June 2014. (For more information: http://www.crcpress.com/product/isbn/9781482205374)
• T. Bouwmans, N. Aybat, and E. Zahzah. Handbook on Robust Low-Rank and Sparse Matrix Decomposition: Applications in Image and Video Processing, CRC Press, Taylor and Francis Group, May 2016. (For more information: http://www.crcpress.com/product/isbn/9781498724623)

Journals

• T. Bouwmans, L. Davis, J. Gonzalez, M. Piccardi, C. Shan, Special Issue on "Background Modeling for Foreground Detection in Real-World Dynamic Scenes", Special Issue in Machine Vision and Applications, July 2014.
• A. Vacavant, L. Tougne, T. Chateau, Special section on "Background models comparison", Computer Vision and Image Understanding, CVIU 2014, May 2014.
• A. Petrosino, L. Maddalena, T. Bouwmans, Special Issue on "Scene Background Modeling and Initialization", Pattern Recognition Letters, September 2017.
• T. Bouwmans, Special Issue on "Detection of Moving Objects", MDPI Journal of Imaging, 2018.

Workshops

• Background Learning for Detection and Tracking from RGB videos (RGBD 2017) Workshop in conjunction with ICIAP 2017. (For more information: http://rgbd2017.na.icar.cnr.it/)
• Scene Background Modeling and Initialization (SBMI 2015) Workshop in conjunction with ICIAP 2015. (For more information: http://sbmi2015.na.icar.cnr.it/)
• IEEE Change Detection Workshop in conjunction with CVPR 2014. (For more information: http://www.changedetection.net/)
• Workshop on Background Model Challenges (BMC 2012) in conjunction with ACCV 2012. (For more information: http://bmc.iut-auvergne.com/)

Contests

• IEEE Scene Background Modeling Contest (SBMC 2016) in conjunction with ICPR 2016 (For more information: http://pione.dinf.usherbrooke.ca/sbmc2016/)

Websites

• Background Subtraction website

The Background Subtraction Website (T. Bouwmans, Univ. La Rochelle, France) contains a comprehensive list of the references in the field, and links to available datasets and software.

Datasets

• ChangeDetection.net (For more information: http://www.changedetection.net/)
• Background Models Challenge (For more information: http://bmc.iut-auvergne.com/)
• Stuttgart Artificial Background Subtraction Dataset (For more information: http://www.vis.uni-stuttgart.de/index.php?id=sabs Archived 2015-03-27 at the Wayback Machine)
• SBMI dataset (For more information: http://sbmi2015.na.icar.cnr.it/)
• SBMnet dataset (For more information: http://pione.dinf.usherbrooke.ca/dataset/)

Libraries

• BackgroundSubtractorCNT

The BackgroundSubtractorCNT library implements a very fast and high quality algorithm written in C++ based on OpenCV. It is targeted at low spec hardware but works just as fast on modern Linux and Windows. (For more information: https://github.com/sagi-z/BackgroundSubtractorCNT).

• BGS Library

The BGS Library (A. Sobral, Univ. La Rochelle, France) provides a C++ framework to perform background subtraction algorithms. The code works either on Windows or on Linux. Currently the library offers more than 30 BGS algorithms. (For more information: https://github.com/andrewssobral/bgslibrary)

• LRS Library – Low-Rank and Sparse tools for Background Modeling and Subtraction in Videos The LRSLibrary (A. Sobral, Univ. La Rochelle, France) provides a collection of low-rank and sparse decomposition algorithms in MATLAB. The library was designed for motion segmentation in videos, but it can be also used or adapted for other computer vision problems. Currently the LRSLibrary contains more than 100 matrix-based and tensor-based algorithms. (For more information: https://github.com/andrewssobral/lrslibrary)
• OpenCV – The OpenCV library provides a number background/foreground segmentation algorithms.