Remix your environment with DrawPen

DrawPen, as we've finally settled on calling it, is a project I've been working on for the last several months with the talented Maya Pillai, and it's included so much design variety, including product design, programming, electronics, and 3D design and printing. In fact, for two former visual designers, it's been refreshing to avoid pushing pixels around. 

Originally, we were driven to create something interactive and even learning-centric for children, and inspired by kids' natural inclination to draw on surfaces like tables, walls, and furniture. So we thought it would be interesting to make a kind of live collage environment, where children could bring objects from their world into their compositions.

Uncovering the challenges ahead

The basic specifics of the desired actions we first drew up would be to track a stylus or hand across the surface of a table or other surface, allowing users to "draw" and show the result using projection. In addition, we wanted to incorporate user-created images in a meaningful way, as objects in a composition. Maya and I looked at several ways to make this a reality, but we had to move quickly — The original idea for the project came to us during a 3-week course (led by the always great Matti Niinimäki).

After much experimentation, we arrived at a simple, debuggable solution: use an infrared camera pointed at the underside of a clear acrlyic table covered in a translucent white sheet. The camera would track an infrared light held by the user above the table, and the result would be projected onto the underside of the table, via a mirror in order to increase the size of the projection and trackable space on the table. Meanwhile, a webcam would be rigged up in a makeshift lightbox with green screen surfaces, so the user could place any object in the box and see the result appear on the table surface. 

This was much easier said than done — our initial setup of this concept took a few days to get right, and involved a couple of big breakthroughs — first setting the thresholding on the Xbox Kinect we used as an infrared camera, then achieving a repeatable way to calibrate the camera and the projected result.