What started off as taking my first steps in Papervision evolved into the creation of a visualization that maps audio spectrums to a 3D body. My goal was to display sound as a smoothly tangled structure as well as having it clearly react to highlights and different instruments in the song. The Candy Sound Visualizer is the result of those experiments.

1. In an old 2D dummy I drew ribbons across the screen for several frequency ranges. The louder a certain frequency was, the wider and brighter the ribbon of that frequency range got. On each frame a new spectrum was drawn on the right side of the screen and the old spectrums were shifted left.

2. Already a big part of the 2D version was normalizing the amplitudes of all frequency ranges so that the color ribbons fill the complete height of the drawing canvas. The first image shows non normalized spectrums with clear peaks and gaps. The second image shows normalized spectrums, filling most of the available area.

3. Since the 2D canvas used rectangles to draw the spectrum it was easy to take the same algorithm to Papervision and map the amplitudes to a mesh and modify the altitude of its vertices.

4. One big issue was that lower frequencies tend to be overweighted in most songs so I added a simple low pass filter to reduce their impact and smooth the overall shape of the landscape.

5. After tweaking the spectral data I went on to the next iteration which was taking the 3D plane and rolling it up to a tube and twisting it to a helix. That was simply done by taking the frequencies’ amplitudes as radius and transforming the vertices’ x/y coordinates with sine functions.

6. A problem I faced here was that even though the spectral data was filtered, the connection between the low and high frequencies wasn’t seamless and left a visible gap.

7. To remove the gap I used both the left and right channels instead of mixing them to one spectrum and mirrored the left channel, so that the lower frequencies were in the middle of the spectrum and the higher frequencies on the outsides. The first image below shows the mono spectrum, the second image the mirrored stereo spectrum.

8. With the mirrored spectrum the gap was practically gone.

9. As soon as I got the 3D shape running I started tweaking the filters, easing, sine functions and textures to generate a broader band of visual styles:

Update 06/02/09: Here’s the candy sound source code.

Since I couldn’t include the music you will have to edit the XML and add your own track. I also added some minor tweaks, camera control (Space+Drag Mouse) and a new texture:

The whole project evolved from a collection of ideas and inspirations, so I never really planned the structure of the code. Everything might seem a little scrambled and there’s probably a thousand things I could have done more elegantly to save performance. Anyways, it’s just an experiment

The FrequencyMapper3d.mxml is the main class which connects the options panel with the audio data and the visualization. The Audio class is responsible for getting and optimizing the sound spectrum. RibbonMap3d visualizes that data and that’s about it.