Semantic-aware Mapping and Exploration of Unknown Indoor Environments Utilizing Multi-model Sensing for Object Labeling and Risk Assessment

Abstract

Robot Urban Search and Rescue (USAR) is a challenging yet promising research area that has potential applications in real-world rescue and recovery operations. USAR response is always a race against time; hence, it suffers a trade-off between time and safety. The USAR response should be fast enough to reach all potential survivors and locate all sources of danger, yet it should be conducted carefully and diligently in order to avoid adding additional risk to the rescuers and victims. The main challenge for rescuers is to adapt to the unique conditions of the indoor USAR environment, which could include inflexible navigation and harsh conditions. The navigation challenge is embodied in the existence of collapsed structures, narrow openings, water leakage, and obstructed pathways. While, the harsh conditions could include the presence of hazardous materials, radiations, dust, poisonous gases, and extreme temperatures. Therefore, rescue robots are considered a promising solution to reduce human risk, extend current human capabilities, and perform complex search and rescue missions safely and efficiently. In this context, rescue robots are devoted to i) explore the environment, ii)locate victims and source of danger, iii) neutralize hazards and sources of danger, vi) build a situation-awareness knowledge of the environment, and v) provide a danger-free route for the first responders to intervene and save victim's life. In the scope of this thesis, we developed three contributions using an autonomous UAV platform to assist the first responders in rapidly exploring an unknown indoor such as USAR in order to identify the location of victims, objects of interests, and risk resources and constructing an informative semantic map to intervene accordingly. The first contribution focuses on the rapid exploration of the environment intending to detect and map the presence of humans (potential victims). This approach utilizes the information collected from various complementary sensors and integrates the gathered data from deep learning-based person detection, wireless signal mapping, and thermal signature mapping in order to create a global human location map accurately. Moreover, the approach employs the next best view (NBV) strategy using a multi-objective cost function to iteratively create a global map to locate the existence of humans quickly. The second contribution focuses on exploring and gathering high-level contextual information about the environment. This high-level information is captured in a 3D semantic map, where, in addition to capturing the geometric representation of the environment, color codes the surfaces in the environment based on a deep-learning-based semantic classification. Moreover, this approach uses these semantic 3D maps to explore the USAR environment in order to detect and label various objects using novel semantic utility functions. The third contribution focuses on increasing situational awareness of the indoor environment by detecting, labeling, and mapping the different types and levels of hazards presented in USAR environments in a new 3D semantic-occupied risk map data structure. The proposed map encapsulates both occupancy and semantic risk annotations. The mapping approach uses a deep learning model to segment risk types and levels semantically, creates an annotated 3D point cloud, and generates a 3D semantic-occupied risk map. The proposed approaches were evaluated in a realistically simulated indoor environment, and results were benchmarked against the state of the art approaches. The results demonstrated that the proposed strategies outperform the state of art methods in several performance measures. The future directions of this research include performing comprehensive real-world testing and evaluation as well as developing a combining all three contributions for multi-sensor based exploration with 3D semantic mapping that includes risk assessment.

Date of Award	Apr 2020
Original language	American English