We finished our programming project!
It turned out quite well in the end I reckon. Before we started out we had a long list of pretty ambitious things that we wanted to get in there so we were just hoping we’d get most of them in there. In the end, somehow, we got every single thing we wanted to in there which surprised us. Here’s a video of me using it:
It’s very weird, I know. When me and Mala had the idea in the first place it wasn’t far off what it ended up being. There was some weird stuff about making your hand into a triangle and I don’t know what else but we filtered it down to the coolest things. Firstly, my group were brilliant! We all worked really well together and everyone got their contributions working really well. Here’s my group members:
Mala Lal: Shader programming and look development
Jerry Lee: Head tracking and Delaunay triangulation
Sam Hiscock: Point cloud, animal head blend shapes
Leo Pettersson: Octree and flocking system
Me: Hand tracking, UI and general game flow, Collisions, Audio and Integration
Quick run-through of how it worked:
- Start menu with hand tracking button collisions. Choose an animal (we each made one, I was the giraffe).
- Point cloud from the kinect reads your face (or whatever you put in front of the camera) using depth sensors.
- Using Delaunay triangulation we make a mesh from the points.
- The mesh rotates round so you can see it properly (which wasn’t as easy as it sounds, each triangle is a separate object with a different pivot point so Sam had to translate it back to the centre, do the rotations, then move it back.
- You can reset and make a new head or carry on.
- The head mesh it used next with hand collisions. Rays are fired from the hand (five, some around the hand to give a bigger collision radius) into the head and each one hit becomes part of a flock and flies around, following your hand.
- One you’ve knocked off enough, the rest join the flock and follow your hand.
- After a time, you can continue.
- Head tracking starts and the animal you chose at the beginning appears and mimics your head movements. This bit is fun I think!
- The flock flies around the head in a slight spiral and flees the head if you get near.
- After a while, you can finish by moving your head to the side.
- Sort of ending credit screen with everyone’s animal as a constellation in the sky.
- When you hover over each constellation, our names appear.
- After a while, you can restart the program and do it again!
Stuff I did:
I explained about the hand tracking in a different post ( http://georgebatchelor.com/2014/03/12/hand-tracking/ ) so read that if you’re interested in how I did it.
There are quite a lot of things that needed to be collided with in the program, all the buttons for one, some of them advance the program, some change shape or play a sound. And then there was the break your face section. Basically all the collision works the same by firing a ray from the point of the hand into the z-axis, and using ray-triangle intersection (there’s quite a lot of info about this online). Because all the collision objects are triangles, we could return a hit or not. For the face triangles, if there was a hit I would switch off drawing that triangle and create a new boid (which was a triangle) to be added to Leo’s flock with it’s initial position being the centre point of the triangle, it worked quite well I think. We had concerns that it would just pop and appear or it wouldn’t look right but because it’s a fast motion you don’t notice any transition really.
The interface was a real pain. I rushed it at the end and hadn’t really thought of how long it would all take. There are so many buttons! And all the text prompts. It wasn’t complicated, just tedious. A lot of the buttons (the start menu ones for instance) would have a mesh and texture for the standard button, a separate textured mesh for when the button was being selected (to change to indicate to the user) and a collision triangle for calculating intersections. It just got a bit convoluted having timers start when you go into a new mode to have a new text prompt appear, then disappear, then a button show up after a while, or change after it had been selected for a certain amount of time. But I guess these things are always there.
Audio was another thing that was left really last minute. I didn’t really have much to do a few days before the hand-in and Mala found this song: http://highskies.bandcamp.com/track/the-world-forgotten which was pretty much perfect for the project, and the author gave us permission to use it. Using NGL (the NCCA Graphics Library) there isn’t a way in there to play audio, so it wasn’t really straight forward. I used the OpenAL library which is quite nice to use once you know how. You basically have to create your audio device and listener and a buffer for each audio file to store all the sound data in. Then you read in all the separate elements of a WAVE file such as the bits per minute and number of channels, assign memory for the song to occupy and read all your source data into the buffer. Then you can just play each audio whenever. The audio class I made as a singleton with an instance method so we could call it anywhere in the program and just call PlayWav(whichever) which worked nicely. For the buttons we had one sound which sounded quite good, and I just changed the pitch for each button (which you can do with OpenAL) which made a pretty fun space xylophone on the start and end menus.
I learned quite a lot on this project. Mainly through integrating all the code, so I worked with everyone trying to fit each bit of code into the main program, getting Delaunay triangulation to work and things like that. There were a lot of challenges through using NGL. If we’d used something a lot bigger and better documented such as OpenFrameworks we would have saved ourselves a lot of trouble, but we also would probably have been lazy and used a lot of libraries instead of writing everything ourselves. Also trying to get a program you can move through without using a keyboard or mouse, so having buttons you press with your hands. And there’s the bit at the end where you get the the end screen by moving your head to the right. It’s little things like that which takes some working out but paid off in the end.
Anyway, I had a lot of fun doing this and want to do a lot more stuff along these lines! Trying to think of ways to make my major project more technical, but I’m also drowning in the amount of work I’ve got to do so I’ll see what I can squeeze in.