Augmented reality and sound


Yes I know, it’s the hype of 2009 and we’re in 2010 now, but I haven’t seen anything yet where augmented reality is combined with sound. If you do know examples of augmented reality and audio, please leave a comment. I’d love to see what other people do with it.

This is a small experiment I started half a year ago but never finished. I looked it up again last week because I needed a Flash AR example for a project at work. Here I improved it a bit to make it more presentable.

What it is

It’s basically a sort of theremin. Hold the marker in front of your computer’s webcam and a sound starts to play. Move the marker and the sound changes.

The Flash file

  Get Adobe Flash player

All source files can be downloaded here. They’re a bit messy however. This was just a quick update to an abandoned older experiment, I didn’t take care to write clean code.

To try this you’ll need the marker to move in front of the webcam. Download it here as a PDF file. Print the file and – for the best results – glue it on a piece of cardboard.

The video

Because I know it's a lot of hassle to download and print the marker I recorded a short demonstration screencap video to show how it all works.

How to use it

Click "Allow" on the Flash player settings screen. This allows Flash to switch on the webcam and use the images it films. Then hold the printed marker in front of the webcam to generate some sounds:

  • With the marker at the center of the video a sine wave plays a C note.
  • Move the marker slowly up and the pitch rises in steps, move down and the pitch goes down. It plays a pentatonic scale in the key of C minor. Two octaves up and two down. With added flattened fifth to make it sound more bluesy.
  • Move the marker to the right to add vibrato. The more to the right the faster and the more intense the vibrato gets.
  • Move left from the center and a 'note repeat' effect starts. At the outer left edge of the webcam the repeat effect doubles in speed. When the marker is moved vertically at the left new notes start and combine with the previous notes that still play and produce a harp-like sound. Very nice.

This idea could be taken a lot further. There's more information to be had from the marker:

  • The distance from the webcam can be measured. To lower the volume the farther the marker is away from the webcam for instance.
  • The three rotations in 3D are available: Rotation around the X, Y and Z axis.

The only obstacle for direct musical use is the latency. The delay between a movement and hearing the resulting sound is too big to be able to play along with a record for example.

For the augmented reality code I used FLARToolkit. More information can be found at the English FLARToolKit page and the Start Up Guide.



Hey great development! I have a couple questions. Do you know of any other AR with sound yet? Also, how do I get into the source code files? I am currently in some university CS classes and would love to see how you did this!


I found Your exciting app.
I am newbie in AR development, but wanted to now better Your app.
Did you made it with Eclipse or Flash Builder?
I just don’t understand what is needed to open it. Some files are .as what can be opened with Flash Builder. But in Audio v010 FlarToolkit.launch I found indication to Eclipse.
I looked for Saqoosha starter kit and there is .fla file, so idea is clear.

I would be very appreciate about your reply!

Thanks in advance!


@ปั๊มแอนด์วาล์ว: The app uses a sound engine I wrote myself. I haven’t heard of arsound before and couldn’t find any info about it. Do you have a link for more about arsound?

Please note that you can download the source files of the app and have a look if you’re interested.

Tamera Crick,

{Thanks|Cheers|Thankyou} {lots|very much}, {I’ve|I have} found {this article|this info|this} {very|extremely} {good|nice|useful}!



Ps. Je WordPress theme is trouwens ook mooi zeg!
Pss. Ik voeg je blog nu meteen toe aan mijn Google Reader!!

Comments are closed.