BitBender was commissioned by the Contemporary Gallery at Georgia Southern University. It has been shown at the Contemporary Gallery and at The Clock Tower Gallery in Manhattan New York.
BitBender is an interactive installation which invites viewers to play games while also generating complex sounds and interlinked video glitch effects by manipulating a custom joystick.
BitBender is an experimental music, performance and gaming interface which utilizes elements of game play, touch controlled electronic sound generation and modified video. The combination of these elements results in a unique, multi faceted experiences for both performer and audience.
It is designed to be a public installation. Therefore it has been designed with the laymen and first time user in mind. It also attempts to harness the social potential of public interaction.
BitBender is made up of three main blocks (see figure 1 for hardware layout):
1. Modified Nintendo game system
2. Analog voltage controlled signal generator
3. Game/synthesizer controllers
The user selects a game to play and presses the power button on the Nintendo. 1 or 2 player games may be selected. Once the game has loaded, using the game controller allows for standard game play. As long as the viewer only touches the joystick and buttons the game will function normally.
Pressing on the touch sensitive voltage source pads (see fig B) causes a few things to happen at once.
1. The game sound track from the Nintendo is silenced.
2. A complex sound pattern is generated by the synth and broadcast from a nearby amplifier. Characteristics such as rate and pitch or the entire kind of effect change as more pads are pressed the pressure on each pad changes.
3. The video becomes severely distorted. The kinds of distortion are determined by the pads being pressed as well as pressure applied. The distortion patters are controlled by the synth oscillators. Therefore what the user hears is also what they see.
Releasing the pads returns the system to the normal state and the user can continue playing the game in a traditional fashion.
PERFORMANCE, PLAY and COMPETITION:
My goal was to create a hybrid of differing kinds of interaction in order to explore a new form of music and performance. It incorporates electronic sound generation, video modulation controlled by the sound, is built around a gaming system and is intended to be installed in a public venue which introduces a strong social component to all of the aspects.
The result is a social, audio and visual experience which invites liberal exploration of each of these parameters.
Most importantly I want the experience to be intriguing and inspiring for the user. The key to successful installation and also experimental musical interfaces is to steer the user to a favorable outcome but also let them explore and come up with personal results. It is important to make the interface as easy to access as possible but to then offer a depth of control for those who wish to delve deeper. The pressure sensitive touch interface does this perfectly. It is incredibly easy to use but can also be extremely subtle.
As a musical interface I am interested in how the conventions of gaming translate into the relative timing and flow of musical output. For instance one may be making decisions based on musical composition as well as timing aspects of the game.
I am equally interested in how the coordination of sound and video impacts the musical output of the device. One hears sound very differently when that sound is represented visually.
The inclusion of gaming serves several purposes. First it is an accessible and intuitive form of interaction. The first challenge when making public installation is to get the public to interact with the piece. The next challenge is to get them to interact in the way you intended. One generally needs very little explanation or coaxing in order to play a video game. The controls were designed using standard arcade hardware which are big, simple and inviting.
Gaming brings with it lots of social, personal and thematic baggage. Many people have a history of gaming and some form of emotional connection with that experience. This is an important aspect which generates expectations. Expectations create the wonderful opportunity to defy those expectations, which is what this piece does through dramatic sound and video modification.
Gaming is also a social experience which more often than not is competitive in nature. This is an aspect I am especially interested in exploring as applied to music making. Each player is able to control the sound and video. One player can modulate the sound and video while the other is playing the game. Therefore there is an added level of competitive and even collaborative dynamic in this piece.
Lastly, game systems such as the Nintendo have easily modifiable video generators. By shorting out points on the video chips it is possible to generate many different kinds of video glitching effects. Some effects will inhibit game play while others simply change aspects of the game characters or landscape.
Nintendo Game System.
The nintendo has been modified to allow points on the circuit board associated with video generation and rendering to be shorted to the negative power supply. This results in a variety of video glitching effects ranging from resolution reduction to background and/or foreground scrambling.
Analog voltage controlled signal generator
This is an analog modular synthesizer. I have referred to it as a voltage controlled signal generator due to the fact that the term “synthesizer” is often associated with audio generating devices. In this case the synth generates audio as well as controlling the video.
Sound is simply a modulating voltage. This voltage may be sent to speaker to make sound but can also be used as a control signal for other circuitry and electronic devices.
The synth contains the following modules:
-1 low pass filter
– 2 amplifiers
– 1 ring modulator
– 1 sample and hold
-1 envelope generator
The challenge with this synth was to make a system which allowed the user to access a very wide range of sound effects and eliminate any setting combinations which would result in no sound outout.
To do this I drastically limited the number of user accessible controls and normalized many of the module ins and outs. I then designed a signal architecture which incorporates many layers of signal feedback in order to generate complex and interesting sounds. See figure 3 for a signal flow block diagram.
The end result is that the user can generate a great range of modulating tones and complex sequences by only adjusting the pitch of each of the 3 oscillators. There are no dead spots, the effects are extremely varied and the different sounds are animated and interesting.
There are two identical controllers. All details below apply to both controllers.
This device contains 4 buttons, a joystick and an array of 24 pressure sensitive touch pads. The 4 buttons and joystick are used to “talk” to the Nintendo. These function as a traditional game controller and allow the user to play games on the Nintendo.
The pressure sensitive touch pads “talk” to the synth. Touching each pad generates a discreet voltage relative to the pressure applied to that pad. These voltages are then used to do three things:
1. Modulate the frequency of one of the 3 voltage controlled oscillators.
2. Open a gate which allows that oscillator to activate 1 of 12 video glitching effects.
3. Trigger the envelope generator which in turn turns UP the synth volume while simultaneously turning DOWN the Nintendo game volume.
This is one of the more complex and interesting parts of the piece. Video glitches can be generated by connecting pins from the video generating ICs in the Nintendo to the negative power supply. In order to make the glitches more interesting I linked them to the 3 synth oscillators. This is also much viscerally more engaging as the mind perceives the link between the sound and the visuals.
To do this I connected each oscillator output to the gate of a transistor. This acts as a switch which simply connects a video glitch point to ground but it does so at the frequency of the control oscillator. This is significant because the rate of the oscillator will interact with the rate of video generation. This results in a density of horizontal lines (width and number of lines)of glitch patterns relative to the two frequencies.
I also wanted to make the glitch only activate when a pressure pad is being pressed. See Figure 2 for a schematic. I connected the voltage output of each pad to the gate of a transistor which then passes one of the oscillators to the base of another transistor which connects a video glitch point to ground.
There are only 3 oscillators which can be used to modulate the video but there are 12 different glitch points which the oscillators can be routed to. The routing scheme is a little crazy and is illustrated in figure 2.
The Nintendo joystick uses a serial protocol to communicate with the Nintendo. I was able to make a joystick by wiring the controls to a CD4021 shift register. Simple and effective.