Rapid adaptation of deep learning teaches drones to survive any weather

To be truly useful, drones—that is, autonomous flying vehicles—will need to learn to navigate real-world weather and wind conditions.

Right now, drones are either flown under controlled conditions, with no wind, or are operated by humans using remote controls. Drones have been taught to fly in formation in the open skies, but those flights are usually conducted under ideal conditions and circumstances.

However, for drones to autonomously perform necessary but quotidian tasks, such as delivering packages or airlifting injured drivers from a traffic accident, drones must be able to adapt to wind conditions in real time—rolling with the punches, meteorologically speaking.

To face this challenge, a team of engineers from Caltech has developed Neural-Fly, a deep-learning method that can help drones cope with new and unknown wind conditions in real time just by updating a few key parameters.

Neural-Fly is described in a study published on May 4 in Science Robotics. The corresponding author is Soon-Jo Chung, Bren Professor of Aerospace and Control and Dynamical Systems and Jet Propulsion Laboratory Research Scientist. Caltech graduate students Michael O’Connell and Guanya Shi are the co-first authors.

Neural-Fly was tested at Caltech’s Center for Autonomous Systems and Technologies (CAST) using its Real Weather Wind Tunnel, a custom 10-foot-by-10-foot array of more than 1,200 tiny computer-controlled fans that allows engineers to simulate everything from a light gust to a gale.

“The issue is that the direct and specific effect of various wind conditions on aircraft dynamics, performance, and stability cannot be accurately characterized as a simple mathematical model,” Chung says. “Rather than try to qualify and quantify each and every effect of turbulent and unpredictable wind conditions we often experience in air travel, we instead employ a combined approach of deep learning and adaptive control that allows the aircraft to learn from previous experiences and adapt to new conditions on the fly with stability and robustness guarantees.”

O’Connell adds, “We have many different models derived from fluid mechanics, but achieving the right model fidelity and tuning that model for each vehicle, wind condition, and operating mode is challenging. On the other hand, existing machine learning methods require huge amounts of data to train yet do not match state-of-the-art flight performance achieved using classical physics-based methods. Moreover, adapting an entire deep neural network in real time is a huge—if not currently impossible—task.”

Neural-Fly, the researchers say, gets around these challenges by using a so-called separation strategy, through which only a few parameters of the neural network must be updated in real time.

“This is achieved with our new meta-learning algorithm, which pre-trains the neural network so that only these key parameters need to be updated to effectively capture the changing environment,” Shi says.

After obtaining as little as 12 minutes of flying data, autonomous quadrotor drones equipped with Neural-Fly learned how to respond to strong winds so well that their performance significantly improved (as measured by their ability to precisely follow a flight path). The error rate following that flight path is around 2.5 times to 4 times smaller compared to the current state-of-the-art drones equipped with similar adaptive control algorithms that identify and respond to aerodynamic effects but without deep neural networks.

Neural-Fly, which was developed in collaboration with Caltech’s Yisong Yue, Professor of Computing and Mathematical Sciences, and Anima Anandkumar, Bren Professor of Computing and Mathematical Sciences, is based on earlier systems known as Neural-Lander and Neural-Swarm. Neural-Lander also used a deep-learning method to track the position and speed of the drone as it landed and modify its landing trajectory and rotor speed to compensate for the rotors’ backwash from the ground and achieve the smoothest possible landing; Neural-Swarm taught drones to fly autonomously in close proximity to each other.

Though landing might seem more complex than flying, Neural-Fly, unlike the earlier systems, can learn in real time. As such, it can respond to changes in wind on the fly, and it does not require tweaking after the fact. Neural-Fly performed as well in flight tests conducted outside the CAST facility as it did in the wind tunnel. Further, the team has shown that flight data gathered by an individual drone can be transferred to another drone, building a pool of knowledge for autonomous vehicles.

At the CAST Real Weather Wind Tunnel, test drones were tasked with flying in a pre-described figure-eight pattern while they were blasted with winds up to 12.1 meters per second—roughly 27 miles per hour, or a six on the Beaufort scale of wind speeds. This is classified as a “strong breeze,” in which it would be difficult to use an umbrella. It ranks just below a “moderate gale,” in which it would be difficult to move and whole trees would be swaying. This wind speed is twice as fast as the speeds encountered by the drone during neural network training, which suggests Neural-Fly could extrapolate and generalize well to unseen and harsher weather.

The drones were equipped with a standard, off-the-shelf flight control computer that is commonly used by the drone research and hobbyist community. Neural-Fly was implemented in an onboard Raspberry Pi 4 computer that is the size of a credit card and retails for around $20.

Paper co-authors include Anandkumar and Yue, as well as Xichen Shi, and former Caltech postdoc Kamyar Azizzadenesheli, now an assistant professor of computer science at Purdue University.