SkipNet: Learning Dynamic Routing in Convolutional Networks

Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, Joseph E. Gonzalez
ECCV 2018

SkipNet: Learning Dynamic Routing in Convolutional Networks

Abstract

While deeper convolutional networks are needed to achieve maximum accuracy in visual perception tasks, for many inputs shallower networks are sufficient. We exploit this observation by learning to skip convolutional layers on a per-input basis. We introduce SkipNet, a modified residual network, that uses a gating network to selectively skip convolutional blocks based on the activations of the previous layer. We formulate the dynamic skipping problem in the context of sequential decision making and propose a hybrid learning algorithm that combines supervised learning and reinforcement learning to address the challenges of non-differentiable skipping decisions. We show SkipNet reduces computation by 30-90% while preserving the accuracy of the original model on four benchmark datasets and outperforms the state-of-the-art dynamic networks and static compression methods. We also qualitatively evaluate the gating policy to reveal a relationship between image scale and saliency and the number of layers skipped.

Paper

Code

paper
github.com/ucbdrive/skipnet

Citation

@inproceedings{wang2018skipnet,
  title={Skipnet: Learning dynamic routing in convolutional networks},
  author={Wang, Xin and Yu, Fisher and Dou, Zi-Yi and Darrell, Trevor and Gonzalez, Joseph E},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  year={2018}
}

Related


Deep Mixture of Experts via Shallow Embedding

Deep Mixture of Experts via Shallow Embedding

UAI 2019 We explore a mixture of experts (MoE) approach to deep dynamic routing, which activates certain experts in the network on a per-example basis.


IDK Cascades: Fast Deep Learning by Learning not to Overthink

IDK Cascades: Fast Deep Learning by Learning not to Overthink

UAI 2018 We introduce the “I Don’t Know” (IDK) prediction cascades framework to accelerate inference without a loss in prediction accuracy.


TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning

TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning

CVPR 2019 We propose Task-Aware Feature Embedding Networks (TAFE-Nets) to learn how to adapt the image representation to a new task in a meta learning fashion.


Deep Layer Aggregation

Deep Layer Aggregation

CVPR 2018 Oral We augment standard architectures with deeper aggregation to better fuse information across layers.


Dilated Residual Networks

Dilated Residual Networks

CVPR 2017 We show that dilated residual networks (DRNs) outperform their non-dilated counterparts in image classification without increasing the model’s depth or complexity.