Learning-Based NMPC for Agile Navigation and Obstacle Avoidance in Unstructured Environments

Autonomous drones operating in unstructured environments face challenges that demand agile navigation and obstacle avoidance. However, the lack of robust vision-based algorithms has hindered the development of effective control strategies. This paper presents a novel approach that combines Learning-Based NMPC (Nonlinear Model Pre-dictive Control) with deep learning policies to address this problem. In our approach, a deep learning policy is trained using an expert NMPC that has access to the environment's map in simulation. The trained policy is then integrated with the NMPC as a feed-forward control signal, ensuring obstacle avoidance while reaching the desired waypoints. By leveraging NMPC, our approach provides essential control and local motion planning capabilities, while the deep learning policy enhances perception and decision-making based on learned representations from input depth images. Experimental results demonstrate the effectiveness of our approach in achieving agile navigation and obstacle avoidance in unstructured environments. This work represents a significant advancement towards the development of intelligent vision-based autonomous systems capable of operating reliably in challenging real-world scenarios. Video: https://www.youtube.com/playlist?list=PLF6aVOnw8UMyxwgsV3nlgBSOpASshK8_b