Project Link: https://github.com/weiweikee/Voce-Snake
For my final project for Codes in Emergent Digital Practices at the University of Denver, we were required to create an interactive visualization and/or game. It involved conceiving and creating larger-scale Processing sketches that will utilize external libraries and an object oriented-approach with multiple tabs. The sketches must be interactive, and user interaction would directly affect the state of the sketches. The sketches required to be done in Java with Processing within 2 weeks.
Project Guidelines & Deliverables
One sketch that includes linking to all relevant classes and libraries
An external form of input is used to drive the output of the sketch (text files, XML, audio, video, images, etc..)
Code must be commented to demonstrate your understanding & intention
At least 1 library must be imported and used within the sketch
Classes should be used to structure the project
User interaction should be clear and have a discernible effect upon the experience
Initially I had 2 ideas for the project. The first idea was to develop an audio reactive art visualization. This was inspired by Ryoichi Kurokawa, who creates both the visual and the musical elements for his generative arts. He is one of the trendsetters in audiovisual language programming.
"He is a genre-transcending artist who continues to confound and transform stereotyped notions concerning the illustration of electronic image expression." - Naut Humon, founder of Rhythm & Noise and jury member at Ars Electronica.
I felt that Ryoichi's work could guide my compositional elements of the music visualization. In addition, I would like to expand audio-visualization to 3D realms which not many have done. I also wanted to utilize my first project (Particle System) and expand upon the idea in which audio input will affect particles' positions and movements.
The second Idea was to develop a simple 2D game that could be controlled using voice-control interface. I had two games in mind - the first was Snake and the second was FlappyBird. I was inspired by the video of the voice controller prototype from The Gamers' Charity. The Gamers' Charity is an organization that helps people with physical disabilities to play video games. For the purpose of the voice controller, I was using Voce as one of my external libraries. Voce is a speech synthesis and recognition library that is cross-platform, accessible from Java and C++, and has a very small API. It uses CMU Sphinx4 for the speech recognition and FreeTTS for the Speech synthesis.
I chose to build a 2D game with voice control because of its simplicity. In terms of the game, I chose Snake because of the limitation of the current speech recognition system. The Snake game pace was slower than the FlappyBird so it would not require an extremely reactive speech recognition system. The voce library had a 1 to 2 second delay when performing speech recognition, hence the FlappyBird would not be suitable as it is a hyper reactive game.
During the development, I divided the project into two smaller projects - the snake game and the voce speech recognition. The voce speech recognition project was created to reveal the command on the screen. While the snake project was created to function a simple snake game.
When I combined them, there were problems with the speed of voce speech recognition and its sensitivity to external noises which give false commands. In order to create a better control for the commands, I added a spacebar button as an activator of the commands. This allowed a more precise control of the snake and reduced external noise being received as commands. Furthermore, I added the command to the top right corner to identify which command is currently being activated. I also kept the keyboard control from the original snake file as a backup controller for the game.
After the main parts of the project were done, I added some finishing touches such as sound effect, menu system, and high score system. The menu system basically has 3 buttons that divide the game into 3 states: Start, Instruction, and High Score. There are additional states such as Paused, Gameover, and Menu that are not listed in the menu. The instructions are basically text on screen about the operation of the game. The high score system works with a csv file so the high score would not reset every time the program is closed.
Results & Final Thoughts
I am very pleased with how this project turned out. The slowness of the voice recognition libraries was fixed by the spacebar activation. The spacebar activation and menu system made the game even more polished. This allowed for better interactivity between the game and the player. The high score system created a more sophisticated system that could increase competition and playtime. The project was also well received by the professor as I was the only one that used the external libraries outside of the contributed libraries in Processing.