Augmented Reality for Teleoperation of Unmanned Vehicles Using 5G and Hybrid 2D-3D Approach

Abstract

For autonomous vehicles to be viable for everyday transportation, they must first address the safety concerns associated with edge cases. Teleoperation is a mechanism that allows remote driving of the vehicle, thereby allowing human intervention if risky situations arise. The main issue with teleoperation is the presence of a time delay between the teleoperator and vehicle. Many approaches involve indirect driving and supervisory control, such as the teleoperator planning a path for the vehicle to follow. On the other hand, this paper proposes a direct teleoperation system, where a predictive display can aid in real-time teleoperation of the vehicle. To present a more accurate view of the environment, we leverage a predictive display, which enhances the teleoperator view by adding visual guides. We utilize a swin transformer model for lane semantic segmentation and road object detection, so as to determine the free space in the environment. Based on this free space, we construct a collision-free path to guide the teleoperator. Our results are in three parts. First, we evaluate our model on image data, to observe its accuracy in correctly identifying the free space in an image. Next, we test it in simulation, to observe the effects of different network conditions. Finally, we test our model on a live video stream via a 5G connection from within a moving vehicle, to evaluate how the model adjusts to a rapidly changing environment. Future work will include implementing user controls within the vehicle, so as to fully test the real-time teleoperation system.

Date of Award	8 May 2024
Original language	American English
Supervisor	Jorge Dias (Supervisor)