How does Pi's arm tracking work? The arm tracking algorithm depends on head tracking wherein the yellow ball is kept in the center of the field of view. Then the forward facing sonar module (the one with the green blinking light that looks like Pi's eyes) measures the distance to the ball by continually pinging it with a sonar pulse about 20 times per second. Combining this distance with the tilt angle of Pi's head and the geometry of Pi's body and limbs, a bit of high school three-dimensional geometry gives the x, y, z coordinates of the ball in Pi's head-centered coordinate system. I then use some more math to translate these coordinates into a frame of reference centered at Pi's shoulder joint. From these new coordinates, I can compute the angle of the arms required to point at the ball. All of these calculations are continually updated so that the arms can follow the movement of the ball.

As to how Pi knows when to grasp the ball, there is an infrared sensor embedded in Pi's right hand that faces Pi's left hand. With the arms open, the sensor measures a large gap between the hands. When the ball is placed between the hands, it cuts into the beam and Pi knows something is there and therefore begins closing his arms. The inside of Pi's hands are lined with force sensors that detect contact with the ball. Squeezing continues until a certain force level is obtained, thereby assuring a good grip on the ball. Once he has a firm hold on the ball, Pi lifts his arms upward so as to be able to carry the ball or hand it off.