[2205] [CVPR 2022 Workshop NTIRE] Residual Local Feature Network for Efficient Super-Resolution

paper
code

Content

• • Abstract
  • Method
  • • • network architecture
      • • residual feature distillation block (RFDB)
        • residual local feature block (RLFB)
      • contrastive loss
      • warm-up strategy
  • Experiment
  • • • setup
      • comparison with SOTA
      • ablation studies
      • • effectiveness of architecture optimization
        • effectiveness of contrastive loss
        • effectiveness of warm-start strategy
        • generalization
      • NTIRE 2022 challenge

Abstract

• feature extractor using 3 conv-relu with shortcut extract local information
  • features from shallow layers are critical for PSNR oriented models
• refined contrastive loss
  • replace ReLU with Tanh after features extractor in VGG-19
• multi-stage warm-start training
  • utilize trained weights from previous stages to boost performance

Illustration of PSNR, inference time and parameter numbers of different SISR models on the Urban100 dataset for 4 × 4times 4× SR.

Method network architecture

The architecture of residual local feature network.

3 parts in RFLN

• shallow features extraction: a 3x3 conv
• deep features extraction: several RLFBs and a 3x3 conv with shortcut
• reconstruction: a 3x3 conv and a pixel-shuffle

residual feature distillation block (RFDB)

step 1 refinement (SBs) and distillation (1x1 conv)
F d 1 , F r 1 = D M 1 ( F i n ) , R M 1 ( F i n ) F d 2 , F r 2 = D M 2 ( F r 1 ) , R M 2 ( F r 1 ) F d 3 , F r 3 = D M 3 ( F r 2 ) , R M 3 ( F r 2 ) F d 4 = D M 4 ( F r 3 ) F d = c o n c a t ( F d 1 , F d 2 , F d 3 , F d 4 ) begin{aligned}F_{d1}, F_{r1}&=DM_1(F_{in}), RM_1(F_{in})\F_{d2}, F_{r2}&=DM_2(F_{r1}), RM_2(F_{r1})\F_{d3}, F_{r3}&=DM_3(F_{r2}), RM_3(F_{r2})\F_{d4}&=DM_4(F_{r3})\F_d&=mathrm{concat}(F_{d1}, F_{d2}, F_{d3}, F_{d4})end{aligned} Fd1​,Fr1​Fd2​,Fr2​Fd3​,Fr3​Fd4​Fd​​=DM1​(Fin​),RM1​(Fin​)=DM2​(Fr1​),RM2​(Fr1​)=DM3​(Fr2​),RM3​(Fr2​)=DM4​(Fr3​)=concat(Fd1​,Fd2​,Fd3​,Fd4​)​

where, D M j , R M j DM_j, RM_j DMj​,RMj​ are j-th distillation or refinement module, F d j , F r j F_{dj}, F_{rj} Fdj​,Frj​ are j-th distilled or refined features, c o n c a t mathrm{concat} concat is channel-wise concatenation
step 2 channel reduction (1x1 conv)
step 3 enhanced spatial attention (ESA) block with a residual connection

note that features distillation (1x1 convs + concat) reduces parametres but deteriorates inference speed
RFDB → remove distillation layers xrightarrow{text{remove distillation layers}} remove distillation layers ​ RFDB_R_48 → increase channel to 52 xrightarrow{text{increase channel to 52}} increase channel to 52 ​ RFDB_R_52 → remove shortcuts in SRB xrightarrow{text{remove shortcuts in SRB}} remove shortcuts in SRB ​ RLFB

The blocks used in ablation study. RFDB_R represent the refinement part of RFDB.

residual local feature block (RLFB)

aim reduce inference time and maintain model capacity based on RFDB

Conv-1 denotes 1 × 1 1times1 1×1 convolution and Conv-3 denotes 3 × 3 3times3 3×3 convolution. (a) RFDB: residual feature distillation block. (b) RLFB: residual local feature block. © ESA: Enhanced Spatial Attention. In (a) and (b) channel numbers of output feature maps are shown next to each layer.

step 1 refinement (1x1 conv + ReLU) with a residual connection
F r 1 = R M 1 ( F i n ) F r 2 = R M 2 ( F r 1 ) F r 3 = R M 3 ( F r 2 ) F r = F r 3 + F i n begin{aligned}F_{r1}&=RM_1(F_{in})\F_{r2}&=RM_2(F_{r1})\F_{r3}&=RM_3(F_{r2})\F_{r}&=F_{r3}+F_{in}end{aligned} Fr1​Fr2​Fr3​Fr​​=RM1​(Fin​)=RM2​(Fr1​)=RM3​(Fr2​)=Fr3​+Fin​​

where, R M j RM_j RMj​ is j-th refinement module, F r j F_{rj} Frj​ are j-th refined features
step 2 a 1x1 conv and an ESA block

Parameter redundancy analysis of ESA modules on the Urban100 dataset. The red lines represent convolutional layers located in ConvGroups.

use a pruning sensitivity analysis to analyse redundancy of ESA blocks in RFDB
for each ESA block, reduce number of convolution layers in ConvGroups to 1

contrastive loss

contrastive learning has shown impressive performance in self-supervised learning
basic idea push positives closer to anchors and push negatives away from anchors in the latent space
contrastive loss is defined as
C L = ∑ i = 1 n λ i d ( ϕ i ( Y a n c h o r ) , ϕ i ( Y p o s ) ) d ( ϕ i ( Y a n c h o r ) , ϕ i ( Y n e g ) ) CL=sum_{i=1}^nlambda_ifrac{d(phi_i(Y_{anchor}), phi_i(Y_{pos}))}{d(phi_i(Y_{anchor}), phi_i(Y_{neg}))} CL=i=1∑n​λi​d(ϕi​(Yanchor​),ϕi​(Yneg​))d(ϕi​(Yanchor​),ϕi​(Ypos​))​

where, ϕ i phi_i ϕi​ is intermediate features from i-th layer, λ i lambda_i λi​ is balancing weight for i-th layer, d ( x , y ) d(x, y) d(x,y) is L 1 L_1 L1​-distance between x x x and y y y

note that experimentally find that PSNR is decreased when contrastive loss is directly employed
   ⟹    implies ⟹investigate the reason to explain discrepancy
visualize difference map of features extracted by a pre-trained model
D M A P i , j = ∑ k = 1 K ( ϕ ( Y 1 ) i , j , k − ϕ ( Y 2 ) i , j , k ) 2 DMAP_{i, j}=sqrt{sum_{k=1}^K(phi(Y_1)_{i, j, k}-phi(Y_2)_{i, j, k})^2} DMAPi,j​=k=1∑K​(ϕ(Y1​)i,j,k​−ϕ(Y2​)i,j,k​)2 ​

where, i , j i, j i,j are spatial coordinates of Y 1 , Y 2 Y_1, Y_2 Y1​,Y2​, k k k is channel index of Y 1 , Y 2 Y_1, Y_2 Y1​,Y2​
Y 1 Y_1 Y1​ are 100 validation HR images in DIV2K, Y 2 Y_2 Y2​ are the corresponding images degraded blur kernels

Visualization of extracted features from the 1st, 3rd, 5th, 9th and 13th layer of the pre-trained VGG-19.

features from deep layers improve performance of perceptual quality by providing more semantic guidance
features from shallow layers preserve more accurate details and textures, which are critical for PSNR-oriented models
   ⟹    implies ⟹utilize features from shallow layers to improve PSNR of trained model

original contrastive loss minimize distance between 2 activated features after ReLU
ReLU is unbounded above and activated feature map is sparse, resulting in information loss and weaker supervision
   ⟹    implies ⟹replace ReLU in feature extractor with Tanh

performance will drop if ReLU is replaced with Tanh without any training    ⟸    impliedby ⟸VGG-19 is pre-trained with ReLU
a randomly initialized network with good architecture is sufficient to capture perceptual details
   ⟹    implies ⟹build a randomly initialized 2-layer feature extractor: Conv_k3s1-Tanh-Conv_k3s1

The difference maps of the pre-trained VGG-19 D M A P p DMAP_p DMAPp​ and our proposed feature extractor D M A P r DMAP_r DMAPr​. D M A P r DMAP_r DMAPr​ has stronger response and can capture more details and textures compared with D M A P p DMAP_p DMAPp​.

proposed feature extractor has stronger response and captures more details and textures
a randomly initialized feature extractor captures some structural information; pre-training is not necessary

warm-up strategy

previous works use 2 × 2times 2× model as a pre-trained network to train 3 × 3times 3× or 4 × 4times 4× model, instead of training them from scratch
   ⟸    impliedby ⟸initialized weights to accelerate convergence and improve final performance
the benefit is only enjoyed once    ⟸    impliedby ⟸scale factors of pre-trained models and target models are different

multi-stage warm-start training strategy

• in the first stage, train RLFN from scratch
• in the next stage, load weights of RLFN of previous stage

enjoy the benefit of pre-trained weight of models from all previous stages in each stage
training settings (batch size or learning rate) follow exactly the same in the first stage

Experiment setup

training sets DIV2K; on RGB channels; random flipping, 90 rotation
testing sets Set5, Set14, BSD100, Urban100; PSNR and SSIM on Y channel of YCbCr
degradation down-sampling by bicubic interpolation
input size 256 × 256 256times256 256×256, randomly cropped from GT
mini-batch size 64
optimizer Adam: β 1 = 0.9 , β 2 = 0.999 , ϵ = 1 0 − 8 beta_1=0.9, beta_2=0.999, epsilon=10^{-8} β1​=0.9,β2​=0.999,ϵ=10−8
learning rate initial 5e-4, halved every 2e5 iters
loss function 1st+2nd: L1, 3rd: L1+CL

comparison with SOTA

Quantitative results of the state-of-the-art models on four benchmark datasets. The best and second-best results are marked in red and blue colors, respectively.

ablation studies effectiveness of architecture optimization

Comparison of RFDB, its two variants: RFDB_R_48, RFDB_R_52, and our RLFB for 4 × 4times 4× SR. Runtime is the average of 10 runs on DIV2K validation set.

key findings

• RFDB_R_48 reduces 25% inference time than RFDB
• RFDB_R_52 surpasses RFDB_R_48 largely in PSNR with just a slight increase of inference time
• RFDB_R_52 yields comparable results with RFDB but shows great superiority in inference speed

Comparison of RLFB_esa_g3 and our RLFB for 4 × 4times 4× SR. RLFB_esa_g3 uses three convolution layers in ConvGroups, while our RLFB uses just one. Runtime is the average of 10 runs on DIV2K validation set for 4 × 4times 4× SR.

key findings

• there is redundancy of ESA blocks in RFDB
• reducing convolution layers in ConvGroups to 1 does not reduce performance but accelerates inference

effectiveness of contrastive loss

Effect of contrastive loss for 4 × 4times 4× SR. RLFN-S_ws_2 applies warm-start twice and RLFN-S_ws_2+CL employs contrastive loss in the second warm-start stage. Contrastive loss improves the performance in terms of both PSNR and SSIM.

key findings

• contrastive loss consistently improves performance of PSNR and SSIM on 4 benchmark datasets

effectiveness of warm-start strategy

Effect of learning rate strategy for 4 × 4times 4× SR. RLFN-S_e2000 and RLFN-S_clr set the total epochs to 2000 to be compared with our proposed strategy RLFN-S_ws_1. RLFN-S_e2000 halves the learning rate every 4e5 iterations. RLFN-S clr_applies a cyclical learning rate policy. The best and second-best results are marked in red and blue colors, respectively.

key findings

• warm-start strategy helps to jump out of the local minimum during optimization process

generalization

generalization of proposed contrastive loss and warm-start strategy

Generalization of our proposed contrastive loss and warm-start strategy. We compare EDSR and its two variants which employ the contrastive loss and warm-start strategy, respectively. It can be seen that our proposed methods are generic to existing SISR models.

NTIRE 2022 challenge

in NTIRE 2022 efficient super-resolution (ESR) challenge

• 1st in main track (Runtime Track)
• 2nd in sub-track2 (Overall Performance Track)

Runtime track results of NTIRE 2022 efficient SR challenge. Only the top five methods are included.