Linear predictors

Each linear predictor is a quantity which incorporates information about the independent variables into the model. The symbol ${\displaystyle \eta _{j}}$ (Greek "eta") denotes a linear predictor and a subscript j is used to denote the jth one. It relates the jth parameter to the explanatory variables, and ${\displaystyle \eta _{j}}$ is expressed as linear combinations (thus, "linear") of unknown parameters ${\displaystyle {\boldsymbol {\beta }}_{j},}$ i.e., of regression coefficients ${\displaystyle \beta _{(j)k}}$.

The jth parameter, ${\displaystyle \theta _{j}}$, of the distribution depends on the independent variables, ${\displaystyle {\boldsymbol {x}},}$ through

${\displaystyle g_{j}(\theta _{j})=\eta _{j}={\boldsymbol {\beta }}_{j}^{T}{\boldsymbol {x}}.}$

Let ${\displaystyle {\boldsymbol {\eta }}=(\eta _{1},\ldots ,\eta _{M})^{T}}$ be the vector of all the linear predictors. (For convenience we always let ${\displaystyle {\boldsymbol {\eta }}}$ be of dimension M). Thus all the covariates comprising ${\displaystyle {\boldsymbol {x}}}$ potentially affect all the parameters through the linear predictors ${\displaystyle \eta _{j}}$. Later, we will allow the linear predictors to be generalized to additive predictors, which is the sum of smooth functions of each ${\displaystyle x_{k}}$ and each function is estimated from the data.

Link functions

Each link function provides the relationship between a linear predictor and a parameter of the distribution. There are many commonly used link functions, and their choice can be somewhat arbitrary. It makes sense to try to match the domain of the link function to the range of the distribution's parameter value. Notice above that the ${\displaystyle g_{j}}$ allows a different link function for each parameter. They have similar properties as with generalized linear models, for example, common link functions include the logit link for parameters in ${\displaystyle (0,1)}$, and the log link for positive parameters. The VGAM package has function identitylink() for parameters that can assume both positive and negative values.

Constraint matrices

More generally, the VGLM framework allows for any linear constraints between the regression coefficients ${\displaystyle \beta _{(j)k}}$ of each linear predictors. For example, we may want to set some to be equal to 0, or constraint some of them to be equal. We have

${\displaystyle {\boldsymbol {\eta }}=\sum _{k=1}^{p}\,{\boldsymbol {\beta }}_{(k)}^{}\,x_{k}=\sum _{k=1}^{p}\,{\boldsymbol {H}}_{k}\;{\boldsymbol {\beta }}_{(k)}^{*}\,x_{k}}$

where the ${\displaystyle {\boldsymbol {H}}_{k}}$ are the constraint matrices. Each constraint matrix is known and prespecified, and has M rows, and between 1 and M columns. The elements of constraint matrices are finite-valued, and often they are just 0 or 1. For example, the value 0 effectively omits that element while a 1 includes it. It is common for some models to have a parallelism assumption, which means that ${\displaystyle {\boldsymbol {H}}_{k}={\boldsymbol {1}}_{M}}$ for ${\displaystyle k=2,\ldots ,p}$, and for some models, for ${\displaystyle k=1}$ too. The special case when ${\displaystyle {\boldsymbol {H}}_{k}={\boldsymbol {I}}_{M}}$ for all ${\displaystyle k=1,\ldots ,p}$ is known as trivial constraints; all the regression coefficients are estimated and are unrelated. And ${\displaystyle \theta _{j}}$ is known as an intercept-only parameter if the jth row of all the ${\displaystyle {\boldsymbol {H}}_{k}=}$ are equal to ${\displaystyle {\boldsymbol {0}}^{T}}$ for ${\displaystyle k=2,\ldots ,p}$, i.e., ${\displaystyle \eta _{j}=\beta _{(j)1}^{*}}$ equals an intercept only. Intercept-only parameters are thus modelled as simply as possible, as a scalar.

The unknown parameters, ${\displaystyle {\boldsymbol {\beta }}^{*}=({\boldsymbol {\beta }}_{(1)}^{*T},\ldots ,{\boldsymbol {\beta }}_{(p)}^{*T})^{T}}$, are typically estimated by the method of maximum likelihood. All the regression coefficients may be put into a matrix as follows:

${\displaystyle {\boldsymbol {\eta }}_{i}={\boldsymbol {B}}^{T}{\boldsymbol {x}}_{i}={\begin{pmatrix}{\boldsymbol {\beta }}_{1}^{T}\,{\boldsymbol {x}}_{i}\\\vdots \\{\boldsymbol {\beta }}_{M}^{T}\,{\boldsymbol {x}}_{i}\\\end{pmatrix}}=\left({\boldsymbol {\beta }}_{(1)}^{},\ldots ,{\boldsymbol {\beta }}_{(p)}^{}\right)\;{\boldsymbol {x}}_{i}.}$

The xij facility

With even more generally, one can allow the value of a variable ${\displaystyle x_{k}}$ to have a different value for each ${\displaystyle \eta _{j}}$. For example, if each linear predictor is for a different time point then one might have a time-varying covariate. For example, in discrete choice models, one has conditional logit models, nested logit models, generalized logit models, and the like, to distinguish between certain variants and fit a multinomial logit model to, e.g., transport choices. A variable such as cost differs depending on the choice, for example, taxi is more expensive than bus, which is more expensive than walking. The xij facility of VGAM allows one to generalize ${\displaystyle \eta _{j}({\boldsymbol {x}}_{i})}$ to ${\displaystyle \eta _{j}({\boldsymbol {x}}_{ij})}$.

The most general formula is

${\displaystyle {\boldsymbol {\eta }}_{i}={\boldsymbol {o}}_{i}+\sum _{k=1}^{p}\,diag(x_{ik1},\ldots ,x_{ikM})\,\mathbf {H} _{k}\,{\boldsymbol {\beta }}_{(k)}^{*}.}$

Here the ${\displaystyle {\boldsymbol {o}}_{i}}$ is an optional offset; which translates to be a ${\displaystyle n\times M}$ matrix in practice. The VGAM package has an xij argument that allows the successive elements of the diagonal matrix to be inputted.

Maximum likelihood

We are maximizing a log-likelihood

${\displaystyle \ell =\sum _{i=1}^{n}\,w_{i}\,\ell _{i},}$

where the ${\displaystyle w_{i}}$ are positive and known prior weights. The maximum likelihood estimates can be found using an iteratively reweighted least squares algorithm using Fisher's scoring method, with updates of the form:

${\displaystyle {\boldsymbol {\beta }}^{(a+1)}={\boldsymbol {\beta }}^{(a)}+{\boldsymbol {\mathcal {I}}}^{-1}({\boldsymbol {\beta }}^{(a)})\,\,\mathbf {u} ({\boldsymbol {\beta }}^{(a)}),}$

where ${\displaystyle {\boldsymbol {\mathcal {I}}}({\boldsymbol {\beta }}^{(a)})}$ is the Fisher information matrix at iteration a. It is also called the expected information matrix, or EIM.

VLM

For the computation, the (small) model matrix constructed from the RHS of the formula in vglm() and the constraint matrices are combined to form a big model matrix. The IRLS is applied to this big X. This matrix is known as the VLM matrix, since the vector linear model is the underlying least squares problem being solved. A VLM is a weighted multivariate regression where the variance-covariance matrix for each row of the response matrix is not necessarily the same, and is known. (In classical multivariate regression, all the errors have the same variance-covariance matrix, and it is unknown). In particular, the VLM minimizes the weighted sum of squares

${\displaystyle \mathrm {ResSS} =\sum _{i=1}^{n}\;w_{i}\left\{\mathbf {z} _{i}^{(a-1)}-{\boldsymbol {\eta }}_{i}^{(a-1)}\right\}^{T}\mathbf {W} _{i}^{(a-1)}\left\{\mathbf {z} _{i}^{(a-1)}-{\boldsymbol {\eta }}_{i}^{(a-1)}\right\}}$

This quantity is minimized at each IRLS iteration. The working responses (also known as pseudo-response and adjusted dependent vectors) are

${\displaystyle \mathbf {z} _{i}={\boldsymbol {\eta }}_{i}+\mathbf {W} _{i}^{-1}\mathbf {u} _{i},}$

where the ${\displaystyle \mathbf {W} _{i}}$ are known as working weights or working weight matrices. They are symmetric and positive-definite. Using the EIM helps ensure that they are all positive-definite (and not just the sum of them) over much of the parameter space. In contrast, using Newton–Raphson would mean the observed information matrices would be used, and these tend to be positive-definite in a smaller subset of the parameter space.

Computationally, the Cholesky decomposition is used to invert the working weight matrices and to convert the overall generalized least squares problem into an ordinary least squares problem.

Generalized linear models

Of course, all generalized linear models are a special cases of VGLMs. But we often estimate all parameters by full maximum likelihood estimation rather than using the method of moments for the scale parameter.

Ordered categorical response

If the response variable is an ordinal measurement with M + 1 levels, then one may fit a model function of the form:

${\displaystyle g(\theta _{j})=\eta _{j}}$   where ${\displaystyle \theta _{j}=\mathrm {Pr} (Y\leq j),}$

for ${\displaystyle j=1,\ldots ,M.}$ Different links g lead to proportional odds models or ordered probit models, e.g., the VGAM family function cumulative(link = probit) assigns a probit link to the cumulative probabilities, therefore this model is also called the cumulative probit model. In general they are called cumulative link models.

For categorical and multinomial distributions, the fitted values are an (M + 1)-vector of probabilities, with the property that all probabilities add up to 1. Each probability indicates the likelihood of occurrence of one of the M + 1 possible values.

Unordered categorical response

If the response variable is a nominal measurement, or the data do not satisfy the assumptions of an ordered model, then one may fit a model of the following form:

${\displaystyle \log \left[{\frac {Pr(Y=j)}{\mathrm {Pr} (Y=M+1)}}\right]=\eta _{j},}$

for ${\displaystyle j=1,\ldots ,M.}$ The above link is sometimes called the multilogit link, and the model is called the multinomial logit model. It is common to choose the first or the last level of the response as the reference or baseline group; the above uses the last level. The VGAM family function multinomial() fits the above model, and it has an argument called refLevel that can be assigned the level used for as the reference group.

Count data

Classical GLM theory performs Poisson regression for count data. The link is typically the logarithm, which is known as the canonical link. The variance function is proportional to the mean:

${\displaystyle \operatorname {Var} (Y_{i})=\tau \mu _{i},\,}$

where the dispersion parameter ${\displaystyle \tau }$ is typically fixed at exactly one. When it is not, the resulting quasi-likelihood model is often described as Poisson with overdispersion, or quasi-Poisson; then ${\displaystyle \tau }$ is commonly estimated by the method-of-moments and as such, confidence intervals for ${\displaystyle \tau }$ are difficult to obtain.

In contrast, VGLMs offer a much richer set of models to handle overdispersion with respect to the Poisson, e.g., the negative binomial distribution and several variants thereof. Another count regression model is the generalized Poisson distribution. Other possible models are the zeta distribution and the Zipf distribution.

Reduced-rank vector generalized linear models

RR-VGLMs are VGLMs where a subset of the B matrix is of a lower rank. Without loss of generality, suppose that ${\displaystyle {\boldsymbol {x}}=({\boldsymbol {x}}_{1}^{T},{\boldsymbol {x}}_{2}^{T})^{T}}$ is a partition of the covariate vector. Then the part of the B matrix corresponding to ${\displaystyle {\boldsymbol {x}}_{2}}$ is of the form ${\displaystyle {\boldsymbol {A}}{\boldsymbol {C}}^{T}}$ where ${\displaystyle {\boldsymbol {A}}}$ and ${\displaystyle {\boldsymbol {C}}}$ are thin matrices (i.e., with R columns), e.g., vectors if the rank R = 1. RR-VGLMs potentially offer several advantages when applied to certain models and data sets. Firstly, if M and p are large then the number of regression coefficients that are estimated by VGLMs is large (${\displaystyle M\times p}$). Then RR-VGLMs can reduce the number of estimated regression coefficients enormously if R is low, e.g., R = 1 or R = 2. An example of a model where this is particularly useful is the RR-multinomial logit model, also known as the stereotype model. Secondly, ${\displaystyle {\boldsymbol {\nu }}={\boldsymbol {C}}^{T}{\boldsymbol {x}}_{2}=(\nu _{1},\ldots ,\nu _{R})^{T}}$ is an R-vector of latent variables, and often these can be usefully interpreted. If R = 1 then we can write ${\displaystyle \nu ={\boldsymbol {c}}^{T}{\boldsymbol {x}}_{2}}$ so that the latent variable comprises loadings on the explanatory variables. It may be seen that RR-VGLMs take optimal linear combinations of the ${\displaystyle {\boldsymbol {x}}_{2}}$ and then a VGLM is fitted to the explanatory variables ${\displaystyle ({\boldsymbol {x}}_{1},{\boldsymbol {\nu }})}$. Thirdly, a biplot can be produced if R = 2 , and this allows the model to be visualized.

It can be shown that RR-VGLMs are simply VGLMs where the constraint matrices for the variables in ${\displaystyle {\boldsymbol {x}}_{2}}$ are unknown and to be estimated. It then transpires that ${\displaystyle {\boldsymbol {H}}_{k}={\boldsymbol {A}}}$ for such variables. RR-VGLMs can be estimated by an alternating algorithm which fixes ${\displaystyle {\boldsymbol {A}}}$ and estimates ${\displaystyle {\boldsymbol {C}},}$ and then fixes ${\displaystyle {\boldsymbol {C}}}$ and estimates ${\displaystyle {\boldsymbol {A}}}$, etc.

In practice, some uniqueness constraints are needed for ${\displaystyle {\boldsymbol {A}}}$ and/or ${\displaystyle {\boldsymbol {C}}}$. In VGAM, the rrvglm() function uses corner constraints by default, which means that the top R rows of ${\displaystyle {\boldsymbol {A}}}$ is set to ${\displaystyle {\boldsymbol {I}}_{R}}$. RR-VGLMs were proposed in 2003.^[3]

Vector generalized additive models

Vector generalized additive models (VGAMs) are a major extension to VGLMs in which the linear predictor ${\displaystyle \eta _{j}}$ is not restricted to be linear in the covariates ${\displaystyle x_{k}}$ but is the sum of smoothing functions applied to the ${\displaystyle x_{k}}$:

${\displaystyle {\boldsymbol {\eta }}({\boldsymbol {x}})={\boldsymbol {H}}_{1}\,{\boldsymbol {\beta }}_{(1)}^{*}+{\boldsymbol {H}}_{2}\,{\boldsymbol {f}}_{(2)}^{*}(x_{2})+{\boldsymbol {H}}_{3}\,{\boldsymbol {f}}_{(3)}^{*}(x_{3})+\cdots \,\!}$

where ${\displaystyle {\boldsymbol {f}}_{(k)}^{*}(x_{k})=(f_{(1)k}^{*}(x_{k}),f_{(2)k}^{*}(x_{k}),\ldots )^{T}.}$ These are M additive predictors. Each smooth function ${\displaystyle f_{(j)k}^{*}}$ is estimated from the data. Thus VGLMs are model-driven while VGAMs are data-driven. Currently, only smoothing splines are implemented in the VGAM package. For M > 1 they are actually vector splines, which estimate the component functions in ${\displaystyle f_{(j)k}^{*}(x_{k})}$ simultaneously. Of course, one could use regression splines with VGLMs. The motivation behind VGAMs is similar to that of Hastie and Tibshirani (1990)^[6] and Wood (2017).^[7] VGAMs were proposed in 1996 .^[8]

Currently, work is being done to estimate VGAMs using P-splines of Eilers and Marx (1996) .^[9] This allows for several advantages over using smoothing splines and vector backfitting, such as the ability to perform automatic smoothing parameter selection easier.

Quadratic reduced-rank vector generalized linear models

These add on a quadratic in the latent variable to the RR-VGLM class. The result is a bell-shaped curve can be fitted to each response, as a function of the latent variable. For R = 2, one has bell-shaped surfaces as a function of the 2 latent variables---somewhat similar to a bivariate normal distribution. Particular applications of QRR-VGLMs can be found in ecology, in a field of multivariate analysis called ordination.

As a specific rank-1 example of a QRR-VGLM, consider Poisson data with S species. The model for Species s is the Poisson regression

${\displaystyle \log \,\mu _{s}(\nu )=\eta _{s}(\nu )=\beta _{(s)1}+\beta _{(s)2}\,\nu +\beta _{(s)3}\,\nu ^{2}=\alpha _{s}-{\frac {1}{2}}\left({\frac {\nu -u_{s}}{t_{s}}}\right)^{2},}$

for ${\displaystyle s=1,\ldots ,S}$. The right-most parameterization which uses the symbols ${\displaystyle \alpha _{s},}$ ${\displaystyle u_{s},}$ ${\displaystyle t_{s},}$ has particular ecological meaning, because they relate to the species abundance, optimum and tolerance respectively. For example, the tolerance is a measure of niche width, and a large value means that that species can live in a wide range of environments. In the above equation, one would need ${\displaystyle \beta _{(s)3}<0}$ in order to obtain a bell-shaped curve.

QRR-VGLMs fit Gaussian ordination models by maximum likelihood estimation, and they are an example of direct gradient analysis. The cqo() function in the VGAM package currently calls optim() to search for the optimal ${\displaystyle {\boldsymbol {C}}}$, and given that, it is easy to calculate the site scores and fit a suitable generalized linear model to that. The function is named after the acronym CQO, which stands for constrained quadratic ordination: the constrained is for direct gradient analysis (there are environmental variables, and a linear combination of these is taken as the latent variable) and the quadratic is for the quadratic form in the latent variables ${\displaystyle {\boldsymbol {\nu }}}$ on the ${\displaystyle {\boldsymbol {\eta }}}$ scale. Unfortunately QRR-VGLMs are sensitive to outliers in both the response and explanatory variables, as well as being computationally expensive, and may give a local solution rather than a global solution. QRR-VGLMs were proposed in 2004.^[10]