Real-Time Mosaicking

Motivation

Introduction

Robotic exploration of remote environments often involves a video survey of scientifically-interesting sites, where the robot is tasked with providing complete video coverage of the sites. These video surveys are made challenging by the lack of reliable sensor data during remote exploration tasks, hampering robot navigation and hindering complete video coverage. The real-time mosaicking project aims to aid remote video surveys by building in real-time a navigation grade map (mosaic) of the survey area and using the map to correct unreliable sensor data.

This project aims to perform video surveys of terrain in the benthic environment (sea-floor) of Monterey Bay. The Monterey Bay Aquarium Research Institute (MBARI) owns a set of Remotely-Operated Vehicles (ROVs) that can operate in the benthic environment, and the ROVs contain a suite of sensors that are used for navigation:

The table shows that the ROV lacks sensors accurate enough to perform large scale navigation over the sea-floor. The USBL's position measurement cannot be used due to its low bandwidth and high noise, and the DVL's integrated position measurement is subject to drift. However, creating a mosaic of the benthic environment in real-time offers us additional information that can be used to improve the ROV's position measurement such that complete video coverage of a site can be guaranteed.

Sample navigational-grade mosaic of benthic terrain

Current Work

The video mosaicking system has successfully been used to survey nearly planar benthic sites, such as the rubble field of the USS Macon airship (above). The system is currently being extended to operate on sites that contain gentle curvature, yet look planar in the camera's view. A key issue in the extension lies in the vehicle navigation, and in particular, how to choose a ROV orientation that will point the camera perpendicular to the non-planar terrain.

Video Mosaic Creation

Real-Time Visual Processing

Navigation grade mosaics of nearly planar surfaces are created by tiling many individual images of the terrain. The tile-to-tile offsets are computed in real-time using a visual processing algorithm involving 2-D image texture correlation. At 5 Hz, the live image from the camera feed is fed through a Signum Laplacian over Gaussian (SLoG) filter to create a binary image representative of the image's texture. The SLoG-ed live image is then 2-D correlated with a SLoG-ed version of a reference image, and an image-to-image offset is computed in pixels. Once the live image has moved a certain number of pixels from the reference image, the live image is stored as the new reference image. The series of reference images and their associated location offsets can be displayed as a mosaic.

Global Mosaic Correction

As reference images are added to the mosaic, errors in computing tile-to-tile offsets accumulate to create large global errors. Even when images are tiled based on the (more accurate) DVL position measurement, the measurement is subject to drift and results in global errors. To correct for these global errors, an additional visual processing step is added to mosaic creation. Non-sequential but overlapping reference images from the mosaic are run through additional texture correlation to provide more information on the tile locations. The additional information is then used to globally correct the tile locations, and the updated tile locations are used to correct the DVL vehicle position estimate. The updated tile locations are also used to determine the regions of the terrain that have been surveyed in order to guarantee complete terrain coverage at survey's end.

To maximize the acquisition of additional information via texture correlation, a lawnmower pattern trajectory is driven over the planar terrain. After every swath of the lawnmower pattern, the images in the swath are correlated with the closest image in the last swath, yielding the formation of "side-to-side" links.

Tile locations in lawnmower pattern with side-to-side links shown in red. The tile locations before and after global correction as shown.

Current Work: Benthic Navigation

The video survey benthic navigation algorithm focuses on providing two pieces of information in real-time:

• Terrain-relative vehicle position estimate
• Camera orientation command that points the camera (ideally) perpendicular to the terrain to minimize perspective distortion in the mosaic's images

The navigation information is subsequently corrected by using the information from the global mosaic correction step.

Navigation Over Planar Terrain

Over planar terrain, the ROV can easily calculate the two required pieces of navigation information:

• Terrain-relative position: Integrate DVL velocities in the plane of interest
• Camera orientation command: Hold fixed orientation while driving lawnmower pattern

Navigation Over Non-Planar Terrain

• Terrain-relative position: With the terrain taking an arbitrary shape, terrain-relative navigation methods must be used to discern the terrain-relative vehicle position
• Camera orientation command: The camera must change orientation as the vehicle traverses the terrain. One way to perform this task is to sense the terrain's orientation by fitting a plane to range measurements of the terrain. However, high spatial frequency content in the terrain (rocks, divots, etc.) leads to undesirable qualities in the camera orientation command: