Behavioural Intelligence Based on Reinforcement Learning and Real-time System Identification for Complex UAV Tasks

Abstract

Reinforcement Learning (RL) has demonstrated significant potential for addressing complex robotic tasks, yet its real-world application to Unmanned Aerial Vehicles (UAVs) with slung loads remains challenging. This dissertation addresses four critical gaps in current research: (1) limitations of existing sim-to-real transfer methods, which are often computationally intensive or require additional fine-tuning, (2) lack of control theory-based closed-loop stability analyses for RL-controlled systems, (3) insufficient transfer of RL policies across related systems, and (4) underutilization of curriculum learning for mastering complex tasks. The study aims to develop and validate a comprehensive RL framework that enhances UAV slung-load performance in complex tasks involving navigation, load swing suppression, and obstacle avoidance. The methodology progresses systematically from simpler to more complex systems, all within the context of developing a comprehensive RL framework for UAVs with slung loads. Initially, the study applies RL to a Tower Crane System (TCS) as a simplified analogue to the UAV slung-load system, introducing a novel method for sim-to-real transfer that incorporates detailed dynamic modeling. This is complemented by a novel closed-loop stability analysis of RL-controlled systems, utilizing Lyapunov theory and a linear-quadratic polynomial approximation of the RL agent. This analysis provides quantifiable stability margins and operational boundaries, which are crucial for ensuring system reliability under various dynamic conditions. Further, the dissertation presents an innovative zero-shot transfer technique that adapts RL agents across quasi-similar systems without retraining. This is achieved by training an RL agent on the TCS with tailored compensatory dynamics, facilitating its seamless transition to the UAV slung-load system. Subsequently, curriculum learning is introduced, beginning with the transferred 1D navigation and load swing suppression tasks. The curriculum then scales the complexity, leading to comprehensive 2D navigation, load suppression and obstacle avoidance, thereby enhancing the agent’s performance and adaptability in the context of the UAV slung-load system. Extensive simulations and real-world experiments demonstrate the effectiveness of these approaches. Key findings include successful sim-to-real transfer, investigation of closed-loop stability in RL-controlled systems with quantifiable margins, efficient policy transfer between quasi-similar systems, and significant performance enhancements through curriculum learning over traditional approaches. This research advances RL application in complex robotic tasks, addressing critical gaps and extending aerial robotics capabilities. The developed methods provide a foundation for more reliable and adaptable autonomous robotic control, with implications beyond UAVs. Practical implications suggest improved safety, efficiency, and adaptability in real-world deployments of autonomous robotic systems.

Date of Award	11 Dec 2024
Original language	American English
Supervisor	Yahya Zweiri (Supervisor)