Pi Robot Trades Webcam for Kinect RGB-D Camera

One of the benefits of being a robot is that you can swap out your parts more-or-less at will when new gadgets appear on the market.  So when the $149 Kinect camera was released for the XBox game console, the robotics community took about a week to figure out how to

use the camera to give their robots 3D vision.  Pi Robot has actually had a Kinect since just before Christmas (no, he couldn't wait), but today he finally got to try it on.

What you are about to see may shock you at first--Pi's familiar face has been transformed by the new technology.  But as we will see, it is all for the best.  Though bigger and heavier than a typical webcam, the Kinect provides both a regular color video image as well as a depth measurement at each point.  It does this by projecting a pattern of structured infrared light out onto the world and uses a second camera tuned to infrared wavelengths to measure the distortion of this pattern caused by the shapes of objects in the scene.  The result is a full depth map of every pixel in the image and at 30 frames per second.

The image on the right shows Pi with his new head on.  The Kinect camera is the black horizontal device at the top.  It is wide because it has to accommodate two cameras and one projector.  It also has two microphones, one at either end, that can be used to locate sounds.  The tufts on top of the Kinect came as a little surprise, first discovered by Mike Ferguson at the University of Albany when he built his latest creation Maxwell shown on the left.

So what do the 3-dimensional images look like?  ROS includes a visualization tool called RViz which allows us to see both the regular video image and the 3D map from the Kinect.  The 3D map can be shown in one of three ways: as the original RGB colors overlayed on the depth measurements; as a grayscale image; or as a color-coded image with red indicating objects closer to the camera and blue/violet labeling far away points.  Here's what they look like:

RGB image superimposed on depth map:

Flat Color (gray): 

Color Coded Depth Map (Axis mode in RViz)

 
The last image perhaps best illustrates the quality of the depth map returned by the Kinect, with subtle changes in hue reflecting the distance of that point in the scene from the camera. Over the next several weeks and months, Pi will use this new 3D vision ability to begin recognizing real-world objects by size and shape, much as we do.   Stay tuned!