Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families

Authors

  • Yibo Yang Peking University
  • Jianlong Wu Shandong University
  • Hongyang Li Peking University
  • Xia Li Peking University
  • Tiancheng Shen Peking University
  • Zhouchen Lin Peking University

DOI:

https://doi.org/10.1609/aaai.v34i04.6141

Abstract

The correspondence between residual networks and dynamical systems motivates researchers to unravel the physics of ResNets with well-developed tools in numeral methods of ODE systems. The Runge-Kutta-Fehlberg method is an adaptive time stepping that renders a good trade-off between the stability and efficiency. Can we also have an adaptive time stepping for ResNets to ensure both stability and performance? In this study, we analyze the effects of time stepping on the Euler method and ResNets. We establish a stability condition for ResNets with step sizes and weight parameters, and point out the effects of step sizes on the stability and performance. Inspired by our analyses, we develop an adaptive time stepping controller that is dependent on the parameters of the current step, and aware of previous steps. The controller is jointly optimized with the network training so that variable step sizes and evolution time can be adaptively adjusted. We conduct experiments on ImageNet and CIFAR to demonstrate the effectiveness. It is shown that our proposed method is able to improve both stability and accuracy without introducing additional overhead in inference phase.

Downloads

Published

2020-04-03

How to Cite

Yang, Y., Wu, J., Li, H., Li, X., Shen, T., & Lin, Z. (2020). Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 6648-6655. https://doi.org/10.1609/aaai.v34i04.6141

Issue

Section

AAAI Technical Track: Machine Learning