Abstract

The development of effective vision-based algorithms has been a significant challenge in achieving autonomous drones, which promise to offer immense potential for many real-world applications. This paper investigates learning deep sensorimotor policies for vision-based drone racing, which is a particularly demanding setting for testing the limits of an algorithm. Our method combines feature representation learning to extract task-relevant feature representations from high-dimensional image inputs with a learning-by-cheating framework to train a deep sensorimotor policy for vision-based drone racing. This approach eliminates the need for globally-consistent state estimation, trajectory planning, and handcrafted control design, allowing the policy to directly infer control commands from raw images, similar to human pilots. We conduct experiments using a realistic simulator and show that our vision-based policy can achieve state-of-the-art racing performance while being robust against unseen visual disturbances. Our study suggests that consistent feature embeddings are essential for achieving robust control performance in the presence of visual disturbances. The key to acquiring consistent feature embeddings is utilizing contrastive learning along with data augmentation.

Video

Paper

Jiawei Fu, Yunlong Song, Yan Wu, Fisher Yu, and Davide Scaramuzza Learning Deep Sensorimotor Policies for Vision-Based Autonomous Drone Racing IROS 2023

Code

github.com/SysCV/drone-racing

Citation

@article{fu2022learning,
  title={Learning deep sensorimotor policies for vision-based autonomous drone racing},
  author={Fu, Jiawei and Song, Yunlong and Wu, Yan and Yu, Fisher and Scaramuzza, Davide},
  journal={arXiv preprint arXiv:2210.14985},
  year={2022}
}