HYPER-DRIVE - Hyperspectral Imaging for Robots in Unstructured Environments
Published on Wed Aug 23 2023
Robot Delivery Shake Shack | Phillip Pessar on FlickrA new research paper titled "HYPER-DRIVE: Visible-Short Wave Infrared Hyperspectral Imaging Datasets for Robots in Unstructured Environments" introduces a groundbreaking system architecture that allows robots to operate onboard with snapshot hyperspectral cameras and point spectrometers. The system efficiently generates composite datacubes, which are three-dimensional representations of data, from a robotic base. The datacubes collected span the visible to shortwave infrared spectrum and simultaneously capture the ambient solar spectrum reflected off a white reference tile.
The researchers collected and analyzed a large dataset of over 500 labeled datacubes from both on-road and off-road terrains. The analysis showed that hyperspectral imaging (HSI) is an invaluable tool for increasing understanding of scene composition from a robot-centric perspective. The dataset is open-source and can be accessed online, along with the code and other data: https://river-lab.github.io/hyper_drive_data.
Terrain perception in robots is a critical area of research in order to enable actionable intelligence before traversing surfaces. Unlike objects, terrain is an abstract material that can vary significantly in size and shape. Traditional perception systems struggle to understand terrain due to these variations. For example, a farm field and a patch of soil may have the same appearance, but cover vastly different areas. This paper builds on previous work in terrain classification by incorporating hyperspectral imaging into mobile robot terrain understanding.
The HYPER DRIVE system developed in this research captures datacubes from a moving platform with variable illumination. The research shows that forward-facing HSI is a powerful tool for robotics and can be applied in a variety of terrain conditions. The main contributions of this paper are the development of a sensing system to incorporate hyperspectral cameras onto a mobile robot, an open-source software framework for the Robot Operating System, and a multi-modal dataset containing registered imaging products and ambient solar spectra data.
The researchers also conducted a t-SNE separability analysis on the dataset to demonstrate its usefulness in robotic terrain analysis. The analysis revealed clearer decision boundaries and tighter clusters among the dominant classes in the hyperspectral t-SNE embedding compared to the RGB color space alone. This suggests that semantic segmentation networks can exploit the natural distinctions among the classes to generate more accurate classifications.
In conclusion, this research paper presents an innovative system architecture for collecting and associating snapshot hyperspectral data from a moving vehicle. The team behind the study released a large dataset of visible-short wave infrared images that capture various operating conditions as seen from a mobile robot. Additionally, they provided open-source software to integrate hyperspectral imaging into robotics applications. This advancement has the potential to revolutionize the applicability of hyperspectral imaging in unstructured environments, and further research in this area is highly encouraged.
