In the previous article I introduced you to the FeED--or FiOS-enabled Emotion Detection--system. FeED is a System created to give online gamers an advantage by giving them a heads-up display of their opponent's mood. You may recall the research I began with the classification of music based on mood. The data gathered from the game contributed to some of the design decisions and helped structure the algorithms for the test version of the audio analysis software. I will begin with a brief on MoodSwings, followed with its connection to the Emotion Detection software.

Deducting Mood from Audio

MoodSwings is a collaborative, two-player game that incorporates each listener’s subjective judgments of the emotional content (mood) of music into the game play. You are partnered at random with someone online, and the goal is for you to reach an agreement on the mood of the song by the end of the 30 second clip.

The game board (shown below) is a representation of Mood in the Valence and Arousal Space. The Valence and Arousal model, originally theorized by Robert Thayer, is at the core of the game, and gives the game player a way to connect music with mood. The colors chosen reflect the mood of the given region. If you want to dive deeper into how MoodSwings works, play it at http://schubert.ece.drexel.edu/moodswings, and if you want to dive even deeper, read this paper. Try out the the board below to see how the Graph and colors represent different emotional content.

Making the Connection with the FeED System

Now we've established a method to retrieve mood information, lets use the data we've gathered to see if we can make a correlation with audio from XBOX online gaming to the color-mood graph above.

Emotion Detection Software

In the previous article we setup our hacked XBOX 360 headset, and we were able to get our opponents audio feed. Now we need to perform real-time analysis of the audio stream, and make our Color lights dance accordingly. I decided to use the Programming Language Python because of its cross-platform compatibility, high-level scripting syntax, and its performance with signal processing. For the first version, I decided to perform real-time analysis on the audio signal's loudness, and pitch. My emotion detection algorithm is structured around these variables. When you begin a new match with your opponent on the XBOX, start the software. If your opponent is quiet, the colors slowly change from calm to energetic. As soon as your opponent says something, the loudness and pitch of their audio is calculated and the colors change accordingly. I wont get into to much detail about the software, but I will start a project page for those of you interested in downloading and testing the code.

Color Kenetics Lighting System

Now we have established a method to approximate our opponent's emotion, lets setup our lights to dance for us. Alison found these great LED lighting company called Color Kenetics by Philips. There are a whole bunch of different types of lights available, and the software to configure the lights is pretty easy to use. Because we want our lights to automatically change colors based on whats going on in the software, we decided to use the iPlayer to interface the computer with the lights. Different color scenes can be programmed on the iPlayer; 255 to be exact. So for our FeED system, we divided the color-mood graph into an 8x8 grid, and programmed a scene for each part of the grid. Once that was complete, I added code to switch between the different scenes, and after some debugging, we have brought our opponent's emotions to light!

Home Theater PC

So all this magic is happening on a computer, but who wants a big tower in their living room. Not me. Thats where the Home Theater PC, or HTPC, comes in.

The HTPC looks like an ordinary DVD Player. Don't be fooled! This bad boy is packing more processing power than any other computer in the House. It is equipped with an AMD Quad Core processor, 640GB hard drive, 3GB DRAM, and Windows Vista Ultimate.

The HTPC comes with a Windows remote that gives the user access to the Windows Media Center. From there you can watch and record TV, listen to music, and view pictures from any networked computer in the house. But for our project, we really just need to utilize the processing power and memory speed.

The computer receives the audio from the XBOX 360 via the networked USB HUB. Because the HTPC and USB Hub are both connected to the FiOS router, the HTPC can utilize any device conned to the the Hub. As such, our USB sound card installed in the Convertible Coffee table can be used to feed audio to the HTPC.

 So there you have it! The result of the trio of Guru's coming together for a common goal. I hope I was able to give you an idea of how this complex system works. Feel free to leave feedback in the comments section below.

Posted by Guru Lloyd | tagged: FeED, Alison, Lloyd