Algorithm Overview

For every frame in the input stream the algorithm computes the corresponding high-resolution (upsampled) frame by combining information from the original low-resolution frame (the central frame) and several low-resolution frames preceding and following the central frame in the video stream. The algorithm consists of 2 parts:

Registration, or finding position in the central frame of each pixel of each low-resolution frame. That is, the dense optical flow from each surrounding frames to the central frame is calculated.
Robust estimation of value of each pixel in the high-resolution frame using the Bayesian framework.

The Intel IPP functions described below implement only second part.

Once the optical flow is computed, the Bayesian formulation is reduced to the following optimization problem:

where:

g - unknown high-resolution frame represented as a 1D vector which elements are pixel intensities in the raster scan order.

v_i,j - confidence weights that make certain links between low- and high-resolution frames more or less important depending on the accuracy of the estimated motion field at corresponding areas and so on.

f_i - i-th low-resolution frame, represented as a 1D vector.

f_i,j - intensity of the j-th pixel of f_i.

A_i - mapping from g to f_i , also known as the point spread function PSF (see “Point Spread Function” for more details). It combines optical flow, blurring and downsampling. As A_i is a linear mapping (each low-resolution pixel is a linear combination of several pixels in high-resolution image, A_i,jg is a dot product of the j-th row of matrix A_i and vector g.

σ_L, σ_p - standard deviations in the likelihood and prior parts of the target function, respectively.

φ_L, φ_p - robust error functions (see “Error Functions”) for the likelihood and prior parts of the target function, respectively.

λ - parameter that regulates smoothness, or the relative weight of the prior part.

(i, j): i< j & |x (i)-x (j)|+| y(i)- y(j)|=1 means that the summation is done over all ordered pairs (g_i , g_j) where g_i and g_j are intensities of the horizontally or vertically adjacent pixels of g.

This problem is solved using the conjugate gradient algorithm, which main steps are:

Computing gradient of C(g ):

where ψ_L and ψ_p are derivatives of the σ_L and σ_p, respectively, and I_i is the i-th column of the identity matrix.

Solving 1D optimization problem:

where d is the current “conjugate” direction. The search is performed using the Newton-Raphson method, if it fails then the golden-section search (GSS) is used.

Computing cost C(g) to determine when to stop the algorithm (as soon as the decrease of the cost function becomes small enough or even negative).