Human-Robot Interaction

OpenGL GUI with context-sensitive popup dialog
(Actual robots shown in inset)

GUI Development

A Graphical User Interface (GUI) has been developed to enable the operation of multiple complex field robots. The interaction mechanism was inspired by interface techniques refined in the Real-Time Strategy (RTS) genre of video games that includes the popular titles Starcraft, Command & Conquer, and Strifeshadow. This mechanism follows three basic steps:

• The operator selects which robots to use
• The operator selects which objects to be acted on
• The operator selects a task to perform

However, the nature of field robotics requires some significant differences in the implementation of the RTS interface method. For instance, there is no single source of accurate global information -- each robot can only provide relative data that has to be fused together. In addition, the tasks that each robot can perform change dynamically and this information must be reflected in the choices presented by the GUI to the operator.

The GUI utilizes OpenGL to display the robot world in three dimensions. Development was significantly aided by Glt (by Nigel Stewart) and GLUI (by Paul Rademacher). Using Glt, which includes GLUI, is highly recommended, especially for C++ programmers new to OpenGL. The OpenGL picking mechanism was used in conjunction with GLUI dialog boxes to provide a direct manipulation interface for robot operation. Additional screenshots and system architecture diagrams are also available (see below).

In the background, real-time data is being handled by NDDS from RTI. The determination of robot capabilities, which change from moment to moment depending on robot capabilities and object characteristics, is performed by the Java Theorem Prover (JTP) developed at Stanford by Gleb Frank. Communication between the GUI and JTP is carried out by the Open Agent Architecture (OAA) from SRI.

SWAT Observations

To gain insight into how humans already manage distributed teams, this research observed field exercises of a police Special Weapons and Tactics (SWAT) team. The Palo Alto / Mountain View (California) Regional SWAT team provided access to its training exercises. The researchers were given free movement throughout the exercise area so that the activities of the commanders, the field units, the snipers, and the hostages and suspects could all be monitored. The tactical commander and field units play roles analgous to the robot operator and the field robots, respectively. The key observations made were:

• The role of the leader (commander or operator) has two primary components
  • Cultivating common ground
  • Coordinating action
• A natural and efficient interaction can be based on physical objects in the remote agents' (field units or robots) environment, just as with the RTS games

SWAT Commander briefing his team

Other interaction methods

Other human-robot interaction projects using the concepts previously developed in the Aerospace Robotics Laboratory are being considered, such as the implementation of other interface modalities.

Object-Based Interaction on a Handheld Device

Additional screenshots and system diagrams are available

Click here

Selected Publications

Extreme Work Groups: Using SWAT Teams As a Model for Coordinating Distributed Robots
Henry L. Jones, Pamela J. Hinds
Published in ACM 2002 Conference on Computer Supported Cooperative Work (CSCW 2002) Procceedings, New Orleans, LA, November 2002

Autonomous Robots in SWAT Applications: Research, Design, and Operations Challenges
Henry L. Jones, Stephen M. Rock, Dennis Burns, Steve Morris
Published in Proceedings of the 2002 Symposium for the Association of Unmanned Vehicle Systems International (AUVSI '02), Orlando, FL, July 2002

Dialogue-Based Human-Robot Interaction for Space Construction Teams
Henry L. Jones and Stephen M. Rock
Published in IEEE Aerospace Conference Proceedings, Big Sky, MT, 2002

Supervisory Control of Multiple Robots based on a Real-Time Strategy Game Interaction Paradigm
Henry Jones and Martin Snyder
Published in IEEE Systems, Man, and Cybernetics Proceedings, Tucson, AZ, 2001

Human-Robot Interaction for Field Operation of an Autonomous Helicopter
Henry Jones, Eric Frew, Bruce Woodley, Stephen Rock
Published in Mobile Robots XIII, Boston, MA, November 1998

A Real-Time Human Perception Interface for Task-Level Control of a Robot in Unfamiliar Environments
Eric S. Miles
PhD thesis, Stanford University, Stanford, CA 94305, June 1997

An Interactive Operator Interface for Task-Level Direction of a Robot in Uncertain Environments
Eric S. Miles and R. H. Cannon Jr.
Published in 2nd ASCE Conference on Robotics for Challenging Environments Proceedings, Albuquerque, NM, June 1996

Underwater Vehicle Control from a Virtual Environment Interface
Stephen D. Fleischer, Stephen M. Rock, Michael J. Lee
Published in Proceedings of the Symposium on Interactive 3D Graphics, Monterey, CA, April 1995