Degradation and Super-Resolution (SR) model. The original HR video frames are related to each other by motion fields. The HR sequence is then degraded to generate the observed sequence. Our proposed dictionary based video SR algorithm estimates the HR sequence, as well as the motion field.
Project Description
Video SR, namely estimating the highresolution (HR) frames from low-resolution (LR) input sequences, is one of the fundamental problems in image and video processing and has been extensively studied for decades. With the popularity of high-definition display devices, such as High-definition television (HDTV), or even Ultra-high-definition television (UHDTV), on the market, there is an avid demand for transferring LR videos into HR videos so that they are displayed on high resolution TV screens. In the figure above, the degradation model relating the HR sequence to the LR sequence which is the input to the SR algorithms.
In this project, we propose an approach for video SR, according to which multiple LR observations of an HR video frame are utilized for both designing coupled dictionaries connecting the sparse representation of LR and HR image frames, as well as for reconstructing an HR frame. We borrow two ideas from single frame SR, namely, bilevel coupled dictionary and multipledictionaries. We incorporate them into a multiple frame SR framework, according to which the non-redundant information contained in LR frames which are typically related by sub-pixel shifts among them is utilized to generate an HR frame. We propose a multiple dictionary multiple frame video SR algorithm utilizing sub-pixel accurate motion estimation. With our proposed SR approach, the estimated optical flow is utilized to obtain multiple frame high accuracy registration and an HR frame is reconstructed from multiple LR frames. The moving parts in a scene can be super-resolved by the sub-pixel shift information while for the stationary parts, the SNR improves due to the multiple observation of the same scene. As far as registration error is concerned, we address it by adapting the weight parameter that enforces the similarity of multiple LR observations, so that our proposed algorithm has the ability to move between single frame bilevel coupled dictionary SR approach and multi-frame SR approach, and perform at least as good as any of these two approaches.
Publications
"Dictionary-based multiple frame video super-resolution"
Dai, Qiqin, Seunghwan Yoo, Armin Kappeler, and Aggelos K. Katsaggelos.
In Image Processing (ICIP), 2015 IEEE International Conference on, pp. 83-87. IEEE, 2015.
[PDF]
"Sparse representation-based multiple frame video super-resolution"
Dai, Qiqin, Seunghwan Yoo, Armin Kappeler, and Aggelos K. Katsaggelos.
IEEE Transactions on Image Processing 26, no. 2 (2017): 765-781.
[PDF]
Codes
TBA