ROS by Example: HTML5, Javascript Pi Face Tracker GUI
by
Overview
Advancements in browser technology now open the door to creating robust, platform independent GUI interfaces to your ROS projects. In this example, we create a GUI interface for the pi_face_tracker code. To make it all work, we leverage the following technologies:
rosbridge
mjpeg_server
html5 Canvas element
javascript
The pi_face_tracker_gui leverages the functionality of the pi_face_tracker code which exposes it's functionality through topics and services that we can interact with via javascript over rosbridge.
HTML and Javascript Note
When implementing javascript with html, you can either include the javascript as a section in the html page, or host it in a separate file and reference the path to it. For ease of visibility and usage, we will include our custom javascript in the same page.
We will also reference the rosbridge javascript which lives outside our html page using this line:
Components
The HTML page has the following main components which we will then discuss in turn:
Toggle buttons: buttons that run on off features that existed as hotkeys in the base pi_face_tracker
Dynamic ROI: GUI implementation of the opencv feature that lets you draw a rectangle over an image and then pass the coordinates of the rectangle to set a new ROI
Dynamic Resolution Adjustment: Using the output from the /camera/rgb/camera_info topic, we are able to dynamically read the camera resolution and adjust the canvas elements and location of the buttons relative to the height of the canvas.
Rosbridge: As mentioned before, it allows us to create javascript messages in JSON format that are then converted by the rosbridge to standard rostopic messages.
Mjpegserver: Used to create an image stream from the video source (in our case the Kinect) but it will also work with a standard webcam as well. The exciting bit about using the mjepgserver is that is is exposed via a socket connection, allowing the video source to be anywhere you can get to by IP address. This is important for future versions of pi_face_tracker_gui, which will working on mobile android enabled devices (or any device that can run google chrome or an equivalent browser).
Component and Features Code Details
Buttons: All of the buttons take the same basic format as the following code for the Toggle Text button.
This line of html creates a button that when selected, calls a custom method called toggle_markers()
The toggle_markers function:
function toggle_markers()
{
var connection = new ros.Connection("ws://127.0.0.1:9090");
connection.setOnOpen(function (e) {
connection.callService('/pi_face_tracker/key_command','["t"]',nop);
});
}
}
This function like the others behind all the buttons creates a connection to rosbridge and once it establishes the connection, it then send the text “t” over the rostopic of /pi_face_tracker/key_command
Showing Video on a Canvas
An HTML5 canvas by default does not play streaming video. In order to make that work, we create the following function called init(), which we call on page load
function init(){
kinect_img.src = "http://127.0.0.1:9191/stream?topic=/pi_face_tracker/image";
//call redraw of video canvas every 100 ms
setInterval(draw,100);
}
This function sets an image source to our streaming video and then uses the setInterval function to call the draw function every 100ms
function draw() {
// draw video on single canvas
var ctx = document.getElementById('video_canvas').getContext('2d');
//ctx.clearRect(0,0,300,300); // clear canvas...not needed for video feed to work
ctx.drawImage(kinect_img,0,0,width,height);
}
It first gets a handle (called a context) to the html object (ctx variable).
One you have the canvas elements context, you can call the drawImage function to display the current “grab” of the image on the canvas. By calling this every 100ms, you then get video.
Dynamically Setting the ROI
For this implementation, we needed the ability to draw a rectangle over the surface of the video image. To do so, we use 2 HTML5 Canvas elements. By default canvases are transparent, so we use the bottom canvas to display the video and another canvas of the exact same size that lays over the top, where we use the mouse to draw a rectangle. As you draw the rectangle to create a new ROI, you are really drawing on the top canvas and looking through at the video on the second canvas behind it.
We create the two canvas elements with the following html code
They use a default size of 640x480, but we will then later adjust that based on information from the CameraInfo topic.
Drawing a Rectangle on a Canvas and Sending Coordinates to pi_face_tracker
When you press and release the mouse buttons while over the drawing canvas, it then calls functions which draw the rectangle. As the rectangle is drawn, we can get the rectangles coordinates relative to the canvas and send them over rosbridge back to the pi_face_tracker. Pi_face_tracker then uses that input to call opencv to draw a rectangle on top of the video coming into pi_face_tracker from the kinect. This is all accomplished through the following functions and code.
function mouseDown(e) {
rect.startX = e.pageX - this.offsetLeft;
rect.startY = e.pageY - this.offsetTop;
drag = true;
}
function mouseUp() {
drag = false;
//clear rectangle on mouse up as opencv now has drawn it for us on the incoming image
ctx.clearRect(0,0,canvas.width,canvas.height);
}
function mouseMove(e) {
if (drag) {
rect.w = (e.pageX - this.offsetLeft) - rect.startX;
rect.h = (e.pageY - this.offsetTop) - rect.startY ;
ctx.clearRect(0,0,canvas.width,canvas.height);
draw_rect();
}
}
function draw_rect() {
// ctx.fillRect(rect.startX, rect.startY, rect.w, rect.h); //creates a filled in square
ctx.strokeRect(rect.startX, rect.startY, rect.w, rect.h); //creates a square with just the outline
//Call to pi face service to set roi
var connection = new ros.Connection("ws://127.0.0.1:9090");
connection.setOnOpen(function (e) {
connection.callService('/pi_face_tracker/set_roi','{"roi":{"x_offset":' + rect.startX + ',"y_offset":' + rect.startY + ',"width":' + rect.w + ',"height":' + rect.h + ' }}',nop);
Using the output from the /camera/rgb/camera_info topic, we are able to dynamically read the camera resolution and adjust the canvas elements and location of the buttons relative to the height of the canvas.
For this part, we will use rosbridge to READ info from a published topic, where so far we have only published to a topic over the bridge. For debugging purposes, we have also added some console output examples which you can see when you run in google Chrome by selecting Ctrl+Shift+J and then select the Console icon.
More Javascript Notes: A section of Javascript can be called on page load, as we do with the init() function, but from there, Javascript will then process from top down. So after the init() function is called, we then hit our custom Javascript below, when then sets up some global variables and executes and also contains various functions we have discussed.
For an overview and some good code example of rosbridge and JSON go to the following link:
Line of HTML code that points image source at the mjpegserver video stream (grabs a single image each time it is called, thus need to call on a timer as discussed above to get video)
The port used here (9191) must match the port on which mjpeg_server is launched (see below), otherwise it can be any available port.
Running the Code
Once mjpeg server and pi_face_tracker are installed, you can run the pi_face_tracker and pi_face_tracker_gui with the following steps in separate terminal windows.
