Ranking method

Suppose ${\displaystyle \mathbb {C} }$ is a data set containing ${\displaystyle N}$ elements ${\displaystyle c_{i}}$. ${\displaystyle r}$ is a ranking method applied to ${\displaystyle \mathbb {C} }$. Then the ${\displaystyle r}$ in ${\displaystyle \mathbb {C} }$ can be represented as a ${\displaystyle N\times N}$ binary matrix. If the rank of ${\displaystyle c_{i}}$ is higher than the rank of ${\displaystyle c_{j}}$, i.e. ${\displaystyle r\ c_{i}<r\ c_{j}}$, the corresponding position of this matrix is set to value of "1". Otherwise the element in that position will be set as the value "0".

Kendall's tau^[3][4]

Kendall's Tau also refers to Kendall tau rank correlation coefficient, which is commonly used to compare two ranking methods for the same data set.

Suppose ${\displaystyle r_{1}}$ and ${\displaystyle r_{2}}$ are two ranking method applied to data set ${\displaystyle \mathbb {C} }$, the Kendall's Tau between ${\displaystyle r_{1}}$ and ${\displaystyle r_{2}}$ can be represented as follows:

${\displaystyle \tau (r_{1},r_{2})={P-Q \over P+Q}=1-{2Q \over P+Q}}$

where ${\displaystyle P}$ is the number of concordant pairs and ${\displaystyle Q}$ is the number of discordant pairs (inversions). A pair ${\displaystyle d_{i}}$ and ${\displaystyle d_{j}}$ is concordant if both ${\displaystyle r_{a}}$ and ${\displaystyle r_{b}}$ agree in how they order ${\displaystyle d_{i}}$ and ${\displaystyle d_{j}}$ . It is discordant if they disagree.

Information retrieval quality^[5][6][7]

Information retrieval quality is usually evaluated by the following three measurements:

1. Precision
2. Recall
3. Average precision

For a specific query to a database, let ${\displaystyle P_{relevant}}$ be the set of relevant information elements in the database and ${\displaystyle P_{retrieved}}$ be the set of the retrieved information elements. Then the above three measurements can be represented as follows:

${\displaystyle {\begin{aligned}&{\text{precision}}={\frac {\left|P_{\text{relevant}}\cap P_{\text{retrieved}}\right|}{\left|P_{\text{retrieved}}\right|}};\\[6pt]&{\text{recall}}={\frac {\left|P_{\text{relevant}}\cap P_{\text{retrieved}}\right|}{\left|P_{\text{relevant}}\right|}};\\[6pt]&{\text{average precision}}=\int _{0}^{1}{\text{Prec}}({\text{recall}})\,d{\text{recall}},\\\end{aligned}}}$

where ${\displaystyle {\text{Prec}}({\text{Recall}})}$ is the ${\displaystyle {\text{Precision}}}$ of ${\displaystyle {\text{Recall}}}$.

Let ${\displaystyle r^{*}}$ and ${\displaystyle r_{f(q)}}$ be the expected and proposed ranking methods of a database respectively, the lower bound of Average Precision of method ${\displaystyle r_{f(q)}}$ can be represented as follows:

${\displaystyle \operatorname {AvgPrec} (r_{f(q)})\geqq {1 \over R}\left[Q+{\binom {R+1}{2}}\right]^{-1}\left(\sum _{i=1}^{R}{\sqrt {i}}\right)^{2}}$

where ${\displaystyle Q}$ is the number of different elements in the upper triangular parts of matrices of ${\displaystyle r^{*}}$ and ${\displaystyle r_{f(q)}}$ and ${\displaystyle R}$ is the number of relevant elements in the data set.

SVM classifier^[8]

Suppose ${\displaystyle ({\vec {x}}_{i},y_{i})}$ is the element of a training data set, where ${\displaystyle {\vec {x}}_{i}}$ is the feature vector and ${\displaystyle y_{i}}$ is the label (which classifies the category of ${\displaystyle {\vec {x}}_{i}}$). A typical SVM classifier for such data set can be defined as the solution of the following optimization problem.

${\displaystyle {\begin{aligned}&{\text{minimize }}V({\vec {w}},{\vec {\xi }})={1 \over 2}{\vec {w}}\cdot {\vec {w}}+CF\sum \xi _{i}^{\sigma }\\[6pt]&{\text{subject to}}\\[6pt]&{\begin{array}{l}\sigma \geqq 0;\\\forall y_{i}({\vec {w}}{\vec {x}}_{i}+b)\geqq 1-\xi _{i}^{\sigma };\end{array}}\\[6pt]&\mathrm {where} \\[6pt]&{\begin{array}{l}b{\text{ is a scalar;}}\\\forall y_{i}\in \left\{-1,1\right\};\\\forall \xi _{i}\geqq 0;\\\end{array}}\end{aligned}}}$

The solution of the above optimization problem can be represented as a linear combination of the feature vectors ${\displaystyle x_{i}}$s.

${\displaystyle {\vec {w}}^{*}=\sum _{i}\alpha _{i}y_{i}x_{i}}$

where ${\displaystyle \alpha _{i}}$ is the coefficients to be determined.

Loss function

Let ${\displaystyle \tau _{P(f)}}$ be the Kendall's tau between expected ranking method ${\displaystyle r^{*}}$ and proposed method ${\displaystyle r_{f(q)}}$, it can be proved that maximizing ${\displaystyle \tau _{P(f)}}$ helps to minimize the lower bound of the Average Precision of ${\displaystyle r_{f(q)}}$.

• Expected loss function^[9]

The negative ${\displaystyle \tau _{P(f)}}$ can be selected as the loss function to minimize the lower bound of average precision of ${\displaystyle r_{f(q)}}$

${\displaystyle L_{\text{expected}}=-\tau _{P(f)}=-\int \tau (r_{f(q)},r^{*})\,dPr(q,r^{*})}$

where ${\displaystyle Pr(q,r^{*})}$ is the statistical distribution of ${\displaystyle r^{*}}$ to certain query ${\displaystyle q}$.

• Empirical loss function

Since the expected loss function is not applicable, the following empirical loss function is selected for the training data in practice.

${\displaystyle L_{\text{empirical}}=-\tau _{S}(f)=-{1 \over n}\sum _{i=1}^{n}\tau (r_{f(q_{i})},r_{i}^{*})}$

Collecting training data

${\displaystyle n}$ i.i.d. queries are applied to a database and each query corresponds to a ranking method. The training data set has ${\displaystyle n}$ elements. Each element contains a query and the corresponding ranking method.

Feature space

Labelled points in feature space

A mapping function ${\displaystyle \Phi (q,d)}$^[10][11] is required to map each query and the element of database to a feature space. Then each point in the feature space is labelled with certain rank by ranking method.

Optimization problem

The points generated by the training data are in the feature space, which also carry the rank information (the labels). These labeled points can be used to find the boundary (classifier) that specifies the order of them. In the linear case, such boundary (classifier) is a vector.

Suppose ${\displaystyle c_{i}}$ and ${\displaystyle c_{j}}$ are two elements in the database and denote ${\displaystyle (c_{i},c_{j})\in r}$ if the rank of ${\displaystyle c_{i}}$ is higher than ${\displaystyle c_{j}}$ in certain ranking method ${\displaystyle r}$. Let vector ${\displaystyle {\vec {w}}}$ be the linear classifier candidate in the feature space. Then the ranking problem can be translated to the following SVM classification problem. Note that one ranking method corresponds to one query.

${\displaystyle {\begin{aligned}&{\text{minimize }}V({\vec {w}},{\vec {\xi }})={1 \over 2}{\vec {w}}\cdot {\vec {w}}+{\text{constant}}\cdot \sum \xi _{i,j,k}\\[6pt]&{\text{subject to}}\\[6pt]&{\begin{array}{l}\forall \xi _{i,j,k}\geqq 0\\\forall (c_{i},c_{j})\in r_{k}^{*}\\{\vec {w}}(\Phi (q_{1},c_{i})-\Phi (q_{1},c_{j}))\geqq 1-\xi _{i,j,1};\\\,\,\,\vdots \\{\vec {w}}(\Phi (q_{n},c_{i})-\Phi (q_{n},c_{j}))\geqq 1-\xi _{i,j,n};\\{\text{where }}k\in \left\{1,2,\ldots ,n\right\},\ i,j\in \left\{1,2,\ldots \right\}.\end{array}}\end{aligned}}}$

The above optimization problem is identical to the classical SVM classification problem, which is the reason why this algorithm is called Ranking-SVM.

W candidate

Not a w candidate

Retrieval function

The optimal vector ${\displaystyle {\vec {w}}^{*}}$ obtained by the training sample is

${\displaystyle {\vec {w}}^{*}=\sum \alpha _{k,\ell }^{*}\Phi (q_{k},c_{i})}$

So the retrieval function could be formed based on such optimal classifier.
For new query ${\displaystyle q}$, the retrieval function first projects all elements of the database to the feature space. Then it orders these feature points by the values of their inner products with the optimal vector. And the rank of each feature point is the rank of the corresponding element of database for the query ${\displaystyle q}$.