Region Normalization for Image Inpainting

  • Tao Yu University of Science and Technology of China
  • Zongyu Guo University of Science and Technology of China
  • Xin Jin University of Science and Technology of China
  • Shilin Wu University of Science and Technology of China
  • Zhibo Chen University of Science and Technology of China
  • Weiping Li University of Science and Technology of China
  • Zhizheng Zhang University of Science and Technology of China
  • Sen Liu University of Science and Technology of China

Abstract

Feature Normalization (FN) is an important technique to help neural network training, which typically normalizes features across spatial dimensions. Most previous image inpainting methods apply FN in their networks without considering the impact of the corrupted regions of the input image on normalization, e.g. mean and variance shifts. In this work, we show that the mean and variance shifts caused by full-spatial FN limit the image inpainting network training and we propose a spatial region-wise normalization named Region Normalization (RN) to overcome the limitation. RN divides spatial pixels into different regions according to the input mask, and computes the mean and variance in each region for normalization. We develop two kinds of RN for our image inpainting network: (1) Basic RN (RN-B), which normalizes pixels from the corrupted and uncorrupted regions separately based on the original inpainting mask to solve the mean and variance shift problem; (2) Learnable RN (RN-L), which automatically detects potentially corrupted and uncorrupted regions for separate normalization, and performs global affine transformation to enhance their fusion. We apply RN-B in the early layers and RN-L in the latter layers of the network respectively. Experiments show that our method outperforms current state-of-the-art methods quantitatively and qualitatively. We further generalize RN to other inpainting networks and achieve consistent performance improvements.

Published
2020-04-03
Section
AAAI Technical Track: Vision