Abstract form

A convex optimization problem is defined by two ingredients:^[6][7]

• The objective function, which is a real-valued convex function of n variables, ${\displaystyle f:{\mathcal {D}}\subseteq \mathbb {R} ^{n}\to \mathbb {R} }$.;
• The feasible set, which is a convex subset ${\displaystyle C\subseteq \mathbb {R} ^{n}}$.

The goal of the problem is to find some ${\displaystyle \mathbf {x^{\ast }} \in C}$ attaining

${\displaystyle \inf\{f(\mathbf {x} ):\mathbf {x} \in C\}}$.

In general, there are three options regarding the existence of a solution:^{[5]: chpt.4}

• If such a point x* exists, it is referred to as an optimal point or solution; the set of all optimal points is called the optimal set; and the problem is called solvable.
• If ${\displaystyle f}$ is unbounded below over ${\displaystyle C}$, or the infimum is not attained, then the optimization problem is said to be unbounded.
• Otherwise, if ${\displaystyle C}$ is the empty set, then the problem is said to be infeasible.

Standard form

A convex optimization problem is in standard form if it is written as

${\displaystyle {\begin{aligned}&{\underset {\mathbf {x} }{\operatorname {minimize} }}&&f(\mathbf {x} )\\&\operatorname {subject\ to} &&g_{i}(\mathbf {x} )\leq 0,\quad i=1,\dots ,m\\&&&h_{i}(\mathbf {x} )=0,\quad i=1,\dots ,p,\end{aligned}}}$

where:^{[5]: chpt.4}

• ${\displaystyle \mathbf {x} \in \mathbb {R} ^{n}}$ is the vector of optimization variables;
• The objective function ${\displaystyle f:{\mathcal {D}}\subseteq \mathbb {R} ^{n}\to \mathbb {R} }$ is a convex function;
• The inequality constraint functions ${\displaystyle g_{i}:\mathbb {R} ^{n}\to \mathbb {R} }$, ${\displaystyle i=1,\ldots ,m}$, are convex functions;
• The equality constraint functions ${\displaystyle h_{i}:\mathbb {R} ^{n}\to \mathbb {R} }$, ${\displaystyle i=1,\ldots ,p}$, are affine transformations, that is, of the form: ${\displaystyle h_{i}(\mathbf {x} )=\mathbf {a_{i}} \cdot \mathbf {x} -b_{i}}$, where ${\displaystyle \mathbf {a_{i}} }$ is a vector and ${\displaystyle b_{i}}$ is a scalar.

The feasible set ${\displaystyle C}$ of the optimization problem consists of all points ${\displaystyle \mathbf {x} \in {\mathcal {D}}}$ satisfying the inequality and the equality constraints. This set is convex because ${\displaystyle {\mathcal {D}}}$ is convex, the sublevel sets of convex functions are convex, affine sets are convex, and the intersection of convex sets is convex.^{[5]: chpt.2}

Many optimization problems can be equivalently formulated in this standard form. For example, the problem of maximizing a concave function ${\displaystyle f}$ can be re-formulated equivalently as the problem of minimizing the convex function ${\displaystyle -f}$. The problem of maximizing a concave function over a convex set is commonly called a convex optimization problem.^[8]

Standard form with linear objective

In the standard form it is possible to assume, without loss of generality, that the objective function f is a linear function. This is because any program with a general objective can be transformed into a program with a linear objective by adding a single variable t and a single constraint, as follows:^[9]: 1.4

${\displaystyle {\begin{aligned}&{\underset {\mathbf {x} ,t}{\operatorname {minimize} }}&&t\\&\operatorname {subject\ to} &&f(\mathbf {x} )-t\leq 0\\&&&g_{i}(\mathbf {x} )\leq 0,\quad i=1,\dots ,m\\&&&h_{i}(\mathbf {x} )=0,\quad i=1,\dots ,p,\end{aligned}}}$

Conic form

Every convex program can be presented in a conic form, which means minimizing a linear objective over the intersection of an affine plane and a convex cone:^[9]: 5.1

${\displaystyle {\begin{aligned}&{\underset {\mathbf {x} }{\operatorname {minimize} }}&&c^{T}x\\&\operatorname {subject\ to} &&x\in (b+L)\cap K\end{aligned}}}$

where K is a closed pointed convex cone, L is a linear subspace of Rⁿ, and b is a vector in Rⁿ. A linear program in standard form in the special case in which K is the nonnegative orthant of Rⁿ.

Unconstrained and equality-constrained problems

The convex programs easiest to solve are the unconstrained problems, or the problems with only equality constraints. As the equality constraints are all linear, they can be eliminated with linear algebra and integrated into the objective, thus converting an equality-constrained problem into an unconstrained one.

In the class of unconstrained (or equality-constrained) problems, the simplest ones are those in which the objective is quadratic. For these problems, the KKT conditions (which are necessary for optimality) are all linear, so they can be solved analytically.^{[5]: chpt.11}

For unconstrained (or equality-constrained) problems with a general convex objective that is twice-differentiable, Newton's method can be used. It can be seen as reducing a general unconstrained convex problem, to a sequence of quadratic problems.^{[5]: chpt.11}Newton's method can be combined with line search for an appropriate step size, and it can be mathematically proven to converge quickly.

Other efficient algorithms for unconstrained minimization are gradient descent (a special case of steepest descent).

General problems

The more challenging problems are those with inequality constraints. A common way to solve them is to reduce them to unconstrained problems by adding a barrier function, enforcing the inequality constraints, to the objective function. Such methods are called interior point methods.^{[5]: chpt.11}They have to be initialized by finding a feasible interior point using by so-called phase I methods, which either find a feasible point or show that none exist. Phase I methods generally consist of reducing the search in question to a simpler convex optimization problem.^{[5]: chpt.11}

Convex optimization problems can also be solved by the following contemporary methods:^[12]

• Bundle methods (Wolfe, Lemaréchal, Kiwiel), and
• Subgradient projection methods (Polyak),
• Interior-point methods,^[1] which make use of self-concordant barrier functions ^[13] and self-regular barrier functions.^[14]
• Cutting-plane methods
• Ellipsoid method
• Subgradient method
• Dual subgradients and the drift-plus-penalty method

Subgradient methods can be implemented simply and so are widely used.^[15] Dual subgradient methods are subgradient methods applied to a dual problem. The drift-plus-penalty method is similar to the dual subgradient method, but takes a time average of the primal variables.