I posted a lot of experimental stuff on this weblog over the past year. Or components that might have their use when built into proper applications. But this is my first web app that’s useful on it’s own.

This is a simple Flash audio editor that works in the browser. Everything it does is done by ActionScript code inside the application. There’s no backend that does the difficult work. So it would function just as well in the standalone Flash Player or exported as an Air application as it does in the browser.

This audio editor has only a very basic set of features. Here’s the things it can do:

• Open mono and stereo WAV and MP3 files.
• Select part of the audio file by graphically adjusting the start and end point.
• Change the pitch by adjusting the playback speed.
• Reverse the file so it plays backwards.
• The result of these adjustments (start and end, pitch, reverse) can be saved as a new stereo WAV file.

The Flash file

Download the source files here.

Code

This project is built on the PureMVC framework again. It’s what I’ve been using for all my projects the last half year and I’m still really happy with it. The code gets more complicated with each project but I still have a clear overview of where everything is and I can even find my way quickly around older projects.

Here’s a short overview of the main building blocks of the audio engine:

Audio engine overview

• AudioDriverProxy listens to the SampleDataEvent. On each event it sends a request to the audio engine (AudioEngineProxy) to fill the buffer with samples.
• AudioEngineProxy holds all the instruments (of type IAudioInstrument) that produce sound. When it receives the request for samples from the audio driver it passes this request on to all the instruments that are registered with the engine.
• IAudioInstrument is (the interface for) a general instrument. An instrument doesn’t generate sound itself, but it maintains a list of IAudioProcessor objects that produce the actual audio. Each processor is a note that the instrument currently plays. So if an instrument was to play a three note chord three processors would be added to the list. And when a note is finished the processor is removed from the list. Besides that IAudioInstrument has an API for the instrument to be played. These are MIDI inspired functions like noteOn(), noteOff() and programChange().
• IAudioProcessor is the object that produces sound. When an instrument receives a buffer from the audio engine it passes the buffer on to each of its processors. The processor loops through all the samples in the buffer and adds its own values to it. An IAudioProcessor can generate sound like a synthesizer or play back existing sound like a sampler. It doesn’t matter as long as it implements the IAudioProcessor interface.
• AudioOutputProxy is the last stage in the audio generation process. Once all instruments have contributed their sounds to the buffer, the audio output is the last one that gets the buffer with sample data from the audio engine. Here final adjustments to the sound can be made. In the case of the audio editor the only adjustment that can be made is to set the overall output volume.

The audio editor is a really simple implementation of this audio engine: There is only one instrument – the editor itself (WaveEditorProxy) – and it can play only one sound at a time: the audio being edited (PitchedSampleProcessor as IAudioProcessor).

The Harrington 1200, a state of the art songwriting computer.

I thought about it a bit and came up with two different scenarios where audio and visuals need to be synchronized:

1. The first is where a sound plays and something visual on screen happens to accompany that audio event.
2. The second is the other way around. Something happens on screen that should be accompanied by a sound.

In the first case the animation is the initiator, in the second case the sound. For the second case I included a Flash example here with the source code available.

1. Animation follows audio

This is the simpler scenario. For example: There’s an online drum machine built in Flash that plays a drum sound on every beat. A kick drum in a techno track for example. On the machine’s interface there’s a small red LED light (a MovieClip) that should blink each time the drum hits.

The drum machine has a steady rhythm set in BPM (Beats Per Minute). So once the machine has started it’s always known when the next drum hit will happen. Here’s a graph of the events:

• (a), (b), (c) … are the moments in time when a SampleDataEvent is received and the buffer is filled with samples.
• (p) is where the drum hit should happen. Because the drum BPM and buffer refresh rate are completely independent the drum hit will almost always start somewhere between two buffer refreshes.
• (q) is where the drum sound is actually heard through the speakers. There is a delay between when the samples are sent out from Flash to when they are played through the speakers. This is the latency; the time needed for the soundcard to process the samples and send them as analog audio to the soundcard’s outputs. Latency is the duration from (p) to (q).

To synchronize the two it’s a simple addition. Duration (a)(p) plus duration (p)(q).

• (a)(p) is the time from the first sample in the buffer to the first sample of the drum hit (and to convert this duration in samples to a duration in milliseconds it must be divided by 44.1 (sample rate / 1000)).
• (p)(q) is the latency. That’s ((SampleDataEvent.position / 44.1) – SoundChannelObject.position). (See the AS3.0 Language reference on SampleDataEvent.)
• The result is the total delay in milliseconds. A Timer could be started when the buffer is filled at (a) so that the LED blinks at the frame nearest (q).

2. Audio follows animation

Download the source code of this example.

This second scenario is a bit more complicated. Hopefully someone else has a simpler solution. But until then this is the solution I use.

Click the ‘Start’ button in the example above: Balls bounce on the floor and each time one of them hits the floor a ‘boing’ sound plays. A simple implementation of Newton’s laws of motion.

A graph of the events:

• (a), (b), (c) … are the times when a buffer is requested and filled. Same as in the previous example.
• (1), (2), (3) … are frames where Flash updates the screen (Event.ENTER_FRAME).
• (p) is where the ball hitting the floor is calculated to happen.
• (q) is where the sound is heard.
• (r) is where the ball hitting the floor is drawn on screen.

What’s going on?

The trick is to delay drawing the animation for a ‘latency’ amount of time.

1. First calculate where the ball WOULD be (but NOT draw it).
2. Fire a sound if the calculation shows the ball would hit the floor.
3. Draw the animation with a delay determined by the latency.

That is the principle of how it works. The actual implementation is a bit more involved:

• At (a), (b), (c) … (each SampleDataEvent.SAMPLE_DATA) the position of the ball is calculated. The current time and position of the ball are stored in a Vector called _history.
• If the calculation shows the ball will hit the floor during the current buffer (at (p)):
  • The time from the start of the buffer till the hit is calculated (a)(p).
  • The sound is started at time offset (a)(p) in the buffer.
• Meanwhile at (1), (2), (3) … (at each Event.ENTER_FRAME) the screen is updated.
• To know where to draw the ball the _history of the ball is used.
• First get the current time and subtract the latency. This is the moment in the past that should be drawn on screen.
• Search in the balls _history for that moment in the past.
• It’s very unlikely however that the position at exactly the required millisecond is stored. More likely somewhere in between two stored moments in time. So get the closest positions before and after the requested time and use linear interpolation to draw the ball at the right position between the ‘before’ and ‘after’ records.

So what’s going on?

I’m afraid this explanation might be more complex than the actual code. So I’ll add some of the relevant ActionScript.

To understand it you should know that there is an AudioDriver class that listens to the SampleDataEvent. Each SampleDataEvent it sends an ENTER_BUFFER notification (this is PureMVC) to notify all Ball instances to store their position. To do this the function enterBuffer is called on each Ball instance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

public function enterBuffer() : void
{
    var hit : Boolean;
 
    if(_isDragging)
    {
        _vy = 0;
    }
    else
    {
        _vy += _gravity;
        _y += _vy;
 
        /** If the ball hits the floor. */
        if(_y > _floorY - _radius)
        {
            var distance : Number = _vy;
            var overlap : Number = _y + _radius - _floorY;
            _hitAlpha = (distance - overlap) / distance;
 
            _y = _floorY - _radius - overlap;
            _vy *= -1;
            hit = true;
 
            dispatchEvent(new Event(Ball.BOUNCE));
        }
    }
 
    /** Store as data for later use. */
    _history.splice(0, 0, new BallVO(_x, _y, _vx, _vy, hit));
 
    /** Keep maximum 100 history items so the maximum latency is buffer * 100. */
    if(_history.length > 100)
    {
        var n : int = _history.length;
        while(--n > 99)
        {
            _history.pop();
        }
    }
}

This function does what was expected: A new position is calculated and stored in a BallVO value object, the BallVO is added to the _history and a bit of maintenance is done: Keep _history’s length within bounds.

The only other thing is the _isDragging Boolean. Balls can be dragged to a new position.

The AudioDriver also sends an ENTER_FRAME notification. This is just like a regular Event.ENTER_FRAME from a DisplayObject except that it has a AudioStreamInfoVO value object as an argument. The AudioStreamInfoVO has a ‘latencyInMS’ property that holds (as you’ll have guessed) the latency in milliseconds that the animation should be delayed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

public function enterFrame(infoVO : AudioStreamInfoVO) : void
{
    graphics.clear();
    graphics.lineStyle(2, 0xFFFFFF);
    graphics.beginFill(0xBBBBBB, 0.1);
 
    if(_isDragging)
    {
        graphics.drawCircle(_x, _y, _radius);
    }
    else
    {
        var time : Number = new Date().getTime() - infoVO.latencyInMS;
 
        var i : int = -1;
        var n : int = _history.length;
        while(++i < n)
        {
            /** Get the data from immediately before and after 'time'. */
            if(_history[i].time < time && i > 0)
            {
                var before : BallVO = _history[i];
                var after : BallVO = _history[i - 1];
                var alpha : Number = (time - before.time) / (after.time - before.time);
 
                /** If a bounce on the floor happened between before and after. */
                if(before.vy > 0 && after.vy < 0)
                {
                    /** Linear interpolation to correct the vertical position. */
                    var maxY : Number = _floorY - _radius;
                    var partBeforeBounce : Number = maxY - before.y;
                    var partAfterBounce : Number = maxY - after.y;
                    var bounceAlpha : Number = partBeforeBounce / (partBeforeBounce + partAfterBounce);
 
                    if(alpha < bounceAlpha)
                    {
                        var y : Number = before.y + (before.vy * bounceAlpha);
                    }
                    else
                    {
                        y = after.y + (before.vy * (1 - bounceAlpha));
                    }
                }
                else
                {
                    /** Regular linear interpolation for a ball in mid flight. */
                    y = before.y + (alpha * (after.y - before.y));
                }
 
                if(before.hit) graphics.beginFill(0xFFFFFF);
                graphics.drawCircle(_x, y, _radius);
 
                break;
            }
        }
    }
}

So in this function there’s a variable ‘time’ which holds the moment in time in the past that must be drawn, ‘before.time’ and ‘after.time’ and a resulting ‘alpha’ which is the interpolation value between before and after. The resulting ball position calculation is this:

y = before.y + (alpha * (after.y – before.y));

Isaac Newton

All this code is not optimized at all. In this example I wanted to keep it all as clear as possible to read and understand.

This is a small experiment I started half a year ago but never finished. I looked it up again last week because I needed a Flash AR example for a project at work. Here I improved it a bit to make it more presentable.

What it is

It’s basically a sort of theremin. Hold the marker in front of your computer’s webcam and a sound starts to play. Move the marker and the sound changes.

The Flash file

All source files can be downloaded here. They’re a bit messy however. This was just a quick update to an abandoned older experiment, I didn’t take care to write clean code.

To try this you’ll need the marker to move in front of the webcam. Download it here as a PDF file. Print the file and – for the best results – glue it on a piece of cardboard.

The video

Because I know it’s a lot of hassle to download and print the marker I recorded a short demonstration screencap video to show how it all works.

How to use it

Click “Allow” on the Flash player settings screen. This allows Flash to switch on the webcam and use the images it films. Then hold the printed marker in front of the webcam to generate some sounds:

• With the marker at the center of the video a sine wave plays a C note.
• Move the marker slowly up and the pitch rises in steps, move down and the pitch goes down. It plays a pentatonic scale in the key of C minor. Two octaves up and two down. With added flattened fifth to make it sound more bluesy.
• Move the marker to the right to add vibrato. The more to the right the faster and the more intense the vibrato gets.
• Move left from the center and a ‘note repeat’ effect starts. At the outer left edge of the webcam the repeat effect doubles in speed. When the marker is moved vertically at the left new notes start and combine with the previous notes that still play and produce a harp-like sound. Very nice.

This idea could be taken a lot further. There’s more information to be had from the marker:

• The distance from the webcam can be measured. To lower the volume the farther the marker is away from the webcam for instance.
• The three rotations in 3D are available: Rotation around the X, Y and Z axis.

The only obstacle for direct musical use is the latency. The delay between a movement and hearing the resulting sound is too big to be able to play along with a record for example.

For the augmented reality code I used FLARToolkit. More information can be found at the English FLARToolKit page and the Start Up Guide.

MP3 wave display is orange.

Wave Display 2 is an improved version of the last one (in the previous post). A few things had to be rewritten in the code for waveform rendering, and there were some extra features I wanted to include to make the application a bit more interesting to use.

The new features in this version:

• Audio playback. You can now listen to the loaded MP3 file. Click ‘play mp3′ or press the SPACE bar to listen.
• Draggable start and end locators to select only part of the sound. Drag the ‘S’ and ‘E’ locators to set the start and end point.
• Reverse playback. Click the ‘reverse’ button to toggle reverse playback.
• Adjustable playback speed. Drag the ‘transpose’ slider to change the playback speed. The speed is adjusted so that the pitch changes from one octave down to one octave up in semitone steps.
• Drag to zoom. Press and drag the mouse on the display area for finer control: Drag vertical to zoom and horizontal to move the waveform. I always really liked this behaviour in Steinberg WaveLab where I first saw it, and where I now copied it from.

Click here to download all source files in a ZIP.

Transpose by playback speed

Transpose works fairly straightforward. The transpose value is set with the slider. And every time the slider is moved the new transpose value is stored.

Each time playback is started, the transpose value is passed to the class PitchedSamplePlayer that processes the sound. However, PitchedSamplePlayer uses a multiplier to set the playback speed instead of a semitone value. For example 0.5 is playback at half speed. So the transpose value in semitones must be converted to playback speed. That’s done with this conversion:

1
2
3
4
5
6
7
8

/**
 * Covert semitone shift to playback speed multiplier.
 * For example: -12 returns 0.5, 12 returns 2, 0 returns 1.
 */
public static function semitoneToRate(semitone : Number) : Number
{
    return Math.pow(2, semitone / 12);
}

Then inside PitchedSamplePlayer linear interpolation is used to change the playback speed and maintain acceptable sound quality. I took the code for pitching a sample from this MP3 Pitch example by Andre Michelle, which builds on Lee Brimelow’s post Adjusting audio pitch in Flash Player 10.

Reverse playback

The display and the sample player both read samples from a ByteArray. I had the choice of reversing the whole ByteArray or leaving the ByteArray as it is and read from it backwards. From end to start.

It seemed best to reverse the order of the ByteArray because the audio processing and display rendering code can stay the same and as short and fast as possible:

1
2
3
4
5
6
7
8
9
10
11
12

/** Reverse the order of samples in samplerPresetVO.soundData.byteArray. */
var bytesPerSample : int = AudioDriverProxy.BYTES_PER_SAMPLE;
var numBytes : Number = samplerPresetVO.soundData.numSamplesTotal * bytesPerSample;
var tempByteArray : ByteArray = new ByteArray();
samplerPresetVO.soundData.byteArray.position = 0;
tempByteArray.writeBytes(samplerPresetVO.soundData.byteArray);
samplerPresetVO.soundData.byteArray.position = 0;
 
for(var offset : Number = (numBytes - bytesPerSample); offset >= 0; offset -= bytesPerSample)
{
   samplerPresetVO.soundData.byteArray.writeBytes(tempByteArray, offset, bytesPerSample);
}

What happens here is:

• Copy all sample data to a temporary ByteArray.
• Read two samples (or one stereo sample) at a time from the temporary ByteArray, reading backwards from end to start.
• And write each stereo sample to the original ByteArray, writing normally from start to end.

The waveform display

2010! A new year, a new decennium even! This year I want to be more consistent in publishing articles on this weblog. That’s my new year’s resolution.

The first one of the new year is more of a utility than a complete application. It’s a wave display; It loads an MP3 file and displays the sound as a graph of the waveform. It makes no sound. A simple waveform display like this might be a useful component in a larger application where audio needs to be edited. Here it’s on its own because I was figuring out how to best program such a component. So this application is more experimental than useful.

Click here to download all source files in a ZIP.

How to use it

A default MP3 file is loaded and displayed at startup. But you can load your own MP3. Click the ‘load mp3′ button and browse to an MP3 file on your computer. Opening a large MP3 file might take a long time however. Best to try with short files. Five minutes or less for example.

Once the MP3 is loaded and displayed there’s several controls to zoom in closer on details of the waveform:

• Zoom in to a detail of the waveform with the ‘zoom horizontal’ slider.
• When zoomed in you can move through the waveform with the scrollbar directly under the waveform.
• For more precise movement once zoomed in drag the mouse left or right on the waveform.
• Magnify the waveform vertically with the ‘zoom vertical’ slider. This is useful to inspect very quiet parts of the sound.

Some information is shown above the display

• In Flash the sample rate is 44100 samples per second. When the waveform is at the highest zoom level, it shows one sample per pixel. (So one second of audio would take up 44100 pixels!)
• At lower zoom levels each pixel displays the average amplitude of several pixels. The amount is displayed at ’samples per pixel’.
• The first and last sample in view are shown at ‘first sample’ and ‘last sample’.

Some notes on the ActionScript

The biggest problem with the display was to get it to respond quick to zoom and position changes. It should use as little calculations as possible to refresh the display. Three things reduce the number of calculations (and so the processor load) here:

Each change in zoom level or position results in a changed portion of the waveform that is actually shown in the display. When the display needs updating, it’s first determined what the first and last sample are that will be displayed. Then only that portion of the file is processed. That means that especially at high zoom levels only a tiny fraction of the file needs to be calculated.

But at low zoom levels still a lot of samples need to be averaged: To show one minute of waveform 44100 x 60 = 2,646,000 samples must be processed. More than two and a half million! To reduce this number samples are skipped. The more samples there are, the more are skipped.

This bit of ActionScript shows how these two reductions are done:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

/**
 * Sets the zoom level of the display. Measured in how many samples one horizontal pixel of the display represents.
 * @param inSamplesPerPixel : How many samples one horizontal pixel of the display represents.
 * @param inCenterSamplePosition : The center or focus of the zoom. 0.5 is the horizontal center. 
 */
private function setCurrentSamplesPerPixel(inSamplesPerPixel : Number, inCenterSamplePosition : Number = 0.5) : void
{
  if(inSamplesPerPixel == mCurrentSamplesPerPixel) return;
 
  // get the sample that is the center of the zoom
  var centerSample : int = mFirstSampleInView + (mNumSamplesInView * inCenterSamplePosition);
 
  mCurrentSamplesPerPixel = Math.max(mMinSamplesPerPixel, Math.min(inSamplesPerPixel, mMaxSamplesPerPixel));
  mNumSamplesInView = mCurrentSamplesPerPixel * mDisplayWidth;
  mFirstSampleInView = centerSample - (mNumSamplesInView * inCenterSamplePosition);
 
  // polling stuff
  mPollingFrequency = Math.ceil(mCurrentSamplesPerPixel / 20);
  mNumPolls = int(mCurrentSamplesPerPixel / mPollingFrequency);
  mBytesToSkip = (mPollingFrequency - 1) * SoundProxy.BYTES_PER_SAMPLE;
 
  if(mFirstSampleInView < 0) mFirstSampleInView = 0;
  if((mFirstSampleInView + mNumSamplesInView) > mNumSamplesTotal) mFirstSampleInView = mNumSamplesTotal - mNumSamplesInView;
 
  updateGraphic();
}

And the third thing to reduce the processor load is a simple timer. There’s a maximum of how often the display can update. In this case a maximum of once every 50 milliSeconds was fine. The timer script was added at the start of the update function:

1

private var mUpdateDelayTimer : Timer = new Timer(50, 1);

1
2
3
4
5
6
7
8
9
10

/**
 * Redraws the display.
 */
private function updateGraphic() : void
{
  // delay between updates
  if(mUpdateDelayTimer.running) return;
  mUpdateDelayTimer.start();
 
  // etcetera ...

That’s it for now. Download the source files to have a closer look at the code or to use it in your own projects.

This project is build with PureMVC as framework again. I like it more and more.
The UI components I used here are very much based on Keith Peters‘ Minimal Comps.
To load an MP3 file and use it in Flash as a Sound object I used FlexibleFactory’s MP3FileReferenceLoader library.
For debugging I used the Yalog logging tool by Stephan Bezoen. Very nice Air application.