ofxPortalCam - Viewing your monitor as a window, not a picture

/

ofxPortalCam is an addon for openFrameworks that allows you to use your monitor as if it were a window through which you could see 3d content.  In other words, it transforms your monitor into a digital diorama using the Kinect for head tracking. [youtube]http://youtu.be/9hOUw3zqobc[/youtube]

Here's the project in a nutshell:

1) The Big Idea

A computer (that is hooked up to both the Kinect and the screen) gives the viewer an image based on these three different things:

  1. where the viewer's head is located
  2. where the screen is located
  3. the shape of the digital scene

With this information the computer knows what image to present to give the viewer the illusion that the screen is a portal into a 3d space.

2) The main problem: Monitor space and Kinect space

So, in isolation, I have all three criteria.  The Kinect gives me the user's head, openFrameworks defines where the screen is located, and the code I use to build my openFrameworks app defines the shape of the scene.

But there's one big problem.  The Kinect is defining everything according to one perspective, and openFrameworks is telling me everything in another perspective.  In other words, the location of my head according to the Kinect doesn't correlate to the location of my head in screen space.  So we need to do some calibration so that we can translate back and forth between the two types of space.

3) The solution: The Human Body

Luckily for us, we've got bodies!  Bodies are awesome for this purpose because they (A) can be tracked by the Kinect in Kinect space and (B) can define the location of the screen by assuming poses.

The calibration process works like this.  The screen presents a dot.  The user puts their hand between their eyes and that dot.  Once there, the user presses a button.  A program then uses the Kinect to record the shape of their body.  This creates a line that extends from the head out through the dot towards infinity.

This is done three times on different parts of the screen to complete one set of calibration points.  The points are: top left, right, and bottom left.

Once these readings are taken, we define a few other sets of points.

4) Moar Datums!!1

Here's the second set of calibration points:

And the third:

Note that every time a set of calibration points is taken, it's from a different place in the room.  Also note that from each set of points, it's always the same three points on the screen: the top left, right, and bottom left.

5) Boil down the data

Once you do this a number of times, you have a really useful dataset.  The data is defined in Kinect space (built off of the geometry of your body), but it describes the location of the screen!  When you put all the data together, you get a bunch of lines that all intersect in 3d space at three discrete points.  These three points define the plane of the screen in Kinect space.

ofxPortalCam then takes this plane definition and gives developers tools that allow them to easily translate from one space to another.

6) So Now What?

When you have the screen defined in Kinect space, it means that you can do some cool transformations to make it look like the screen isn't even there -- like it's a portal into a digital space.

The best part about all of this is that it's an OF addon -- so if you've got any interesting openFrameworks projects, they can integrate this addon and take a look at the project in a whole new light.