Neuromorphic Vision based Instance Segmentation of Unknown Objects in Cluttered Environments for Robotic Grasping

Abstract

In recent years, robotic grasping has been widely deployed in various industries driven by necessity and opportunity. However, grasping unknown objects in a cluttered environment is still an open research challenge. The characteristics of target objects are required to be known for successful robotic grasping, driving the motivation behind vision-based instance segmentation. Conventional vision-based robotic vision suffers from motion blur and low sampling rate, and hence may not meet the automation needs of evolving industrial requirements. Neuromorphic vision, on the other hand, has the advantage of low latency, no motion blur, high dynamic range, and low power consumption. It introduces new opportunities as well as challenges in terms of processing and coping with challenging environmental conditions such as poor illumination and complex dynamics.

This research aims to investigate an instance segmentation approach of unknown tabletop objects and consequently develop a robotic grasping system using neuromorphic vision in cluttered environments. Particularly, four research contributions are made to achieve this aim: (1) an event-based method is developed to detect and suppress slip between the object and gripper in a timely manner, that involves noise sampling for slip threshold calibration, object features sampling while caging for controller calibration, grasping, slip detection and suppression for maintaining grasp stability; (2) the first complete and robust event-based grasping framework for robotic manipulators with neuromorphic eye-in-hand configuration is developed involving visual servoing and grasp plan and validated with various objects in different lighting conditions; (3) a novel frame-based segmentation refinement model is developed based on an encoder-decoder structure for unknown objects; (4) a 3D spatial-temporal event-based segmentation dataset (ESD) of tabletop objects is constructed, to support research and benchmarking in the field of learning-based instance segmentation since only few datasets exist for this particular application. This research provides a complete pipeline of neuromorphic vision-based robotic grasping from perception to actuation, the first in this field. It has the potential to be employed in other perception-leading applications, such as harvesting and picking in agriculture, object sorting in warehouses, and robotic surgery in the medical field.

Date of Award	Apr 2023
Original language	American English
Supervisor	Yahya Zweiri (Supervisor)