Pure Data OSC test patch

The main new feature in this version is support for Open Sound Control (OSC). After the last version a great offer came from programmer Michael Heuer to collaborate and add OSC support to the application. So this part of the program is very much his work. You can view more of his projects on GitHub.

The application

Euclidean Pattern Generator v1.1

As usual here are some download links for the app and the source files. Note that the application is now released under the GPL v3 license. If you’re a developer you might find it easiest to visit the GitHub repository.

Download Java JAR file.
Download Mac OS X APP file.
Download Java source files (Eclipse project), JAR and APP, all in one ZIP file.
GitHub repository for the Euclidean Pattern Generator.

The OSC implementation

All OSC messages are sent to ‘localhost’ on a port number you can specify yourself. In the top part of the application I added a text input labelled ‘OSC Port’. Here you can enter the port number of the receiving OSC server you want to send the OSC messages to.

Because there is a choice now between MIDI and OSC I added checkboxes ‘Send OSC’ and ‘Send MIDI’. Only when these are checked OSC and / or MIDI messages will actually be sent.

The receiving OSC application is able to identify which pattern sent the message if an address for the pattern is entered.
In the pattern’s MIDI / OSC Settings panel there is a new text input field labelled ‘OSC Address’. This field is automatically filled with a default address ‘/eu/patternX’ where ‘X’ is a number that is auto-increased with each new created pattern. Of course you can use your own preferred address naming scheme as well.

Here is a video that demonstrates sending OSC messages from the application to a Pure Data patch.

Other new features

Next to the Open Sound Control support there are a number of other new features, some of which were requested by users in comments on the previous version.

• Step duration (quantization) from 4 beats (one 4/4 measure) to 1/8 beat (1/32 note).
• Tempo maximum increased to 300 BPM.
• Pattern length up to 64 steps. The pattern grows in size to accommodate the number of step circles.
• Patterns can be named. New text input Pattern name in the pattern’s Other Settings panel. Whatever is typed here appears under the pattern graphic. When there are a lot of patterns in a project it’s easy to lose track of which pattern does what, so now you can name them ‘kick’, ‘snare’ etc.
• ‘All Notes Off’ button to kill hanging notes. This sends an ‘All Notes Off’ MIDI Channel Mode Message to each channel. All Notes Off is actually MIDI CC #123 with value 0 (see the MIDI specification).
• New patterns start with velocity 10 (handy in live situations where you don’t want a new pattern to come in at full volume).
• Preferences: Several settings are automatically stored and recalled the next time you open the application.
  • Window size and position.
  • MIDI enabled and MIDI Out port.
  • OSC enabled and OSC port number.
  • Last used directory to open or save a project.

And lastly, not really a new feature, but ‘behind the scenes’ the user interface now uses SwiXML, a layout engine that uses XML files to define an interface built with Java Swing components. This makes the interface much more flexible and easier to maintain for future changes.

Next version

In the next version I’ll add MIDI input and I hope it will be possible to get the app to synchronize to external MIDI clock. That would be cool.

And I’d like to add triplets to the quantization options. For even more unusual polyrhythms.

Screenshot

Since Flash doesn’t do MIDI I spent the last few months learning Java and rebuilt the pattern generator to add MIDI capabilities.

Running the application

The Java app doesn’t run in the browser, so I can’t show it embedded on the web page like I’m used to with Flash projects. Instead you’ll have to download the Java .jar file and run it on your computer like you would any regular desktop application. For Mac users there’s a OS X .app file as well.
I haven’t looked into creating an .exe file for Windows users yet, but double clicking the .jar file should start up the program just like an .exe.

Download Java JAR file. *
Download Mac OS X APP file.
Download Java source files (Eclipse project), JAR and APP, all in one ZIP file.

* The JAR file needs the runtime for Java v1.6 (also known as Java SE 6) to be installed. On Mac this version is automatically installed with OS X 10.6 (Snow Leopard), but OS X 10.5 (Leopard) has the older Java v1.5 (or J2SE 5.0). I tested on Windows XP with Java v1.6 and that runs fine. You can test your Java version by opening Command Prompt (Windows) or Terminal (OS X) and typing “java -version” (without the quatation marks and press return after typing).

To compensate for the lack of an embedded app here’s a video with a short overview of what the Euclidean Pattern Sequencer does:

Features

The features are basic in this first version:

• Double click anywhere in the lower panel to generate a new pattern. A new pattern is 16 steps with 4 notes by default.
• Click a pattern’s center circle to select the pattern. A selected pattern shows a double center ring and it’s settings are displayed on the panel at the right.
• Drag a pattern’s center circle to move the pattern. So you can visually reorganize your screen when there’s a lot of patterns.
• All patterns with their settings and location can be saved in a file. This is a regular XML text file. The File menu has the familiar New, Open, Save and Save As options. The project tempo is saved in the file as well.

Pattern generator control bar

Pattern Settings Panel

Once a pattern is selected it’s settings can be edited:

• The Steps and Fills sliders determine the pattern. Each time one of them changes the algorithm is recalculated.
• Rotate rotates the pattern. As you’ll note when you play with the Steps and Fills sliders, a pattern always starts with a note on the first step. To overcome this limitation you can rotate the pattern with this slider.
• MIDI settings Channel, Pitch and Velocity are parts of the MIDI Note On and Note Off messages that the pattern sends to the MIDI port.
• Length is the duration of the note. Once the end of a note is reached a MIDI Note Off message is sent to the MIDI port. The length of a note can be 16 steps maximum. At the moment length is measured in the internal sequencer resolution, which is 24 PPQN. Because one step is seen as a 16th note, the maximum slider value is ( 16 steps / 4 ) * 24 PPQN = 96. Not very intuitive, I know. I’ll think of something better in an update.
• Mute, Solo and Delete do what you’d expect them to do.

Examples

Here are two pieces of music I’ve already made with the app. I tried to get them to be a bit more like finished track than just demos of layered polyrhythms.

The first one is the same setup as in the video above but with more patterns. About sixteen of them sending MIDI to sixteen tracks in Ableton Live with mainly samples being played. I recorded the MIDI patterns in Live, did some arranging and automation and exported the result as audio.

Euclidean Patterns Demo 1 by Wouter Hisschemöller

The second one are patterns sending MIDI notes to a hardware drum machine via the soundcard’s MIDI output. This is my new Machinedrum UW that plays samples as well as synthesizing drum sounds. All sounds are the Machinedrum’s sixteen tracks playing and recorded on the computer.

Euclidean Patterns Demo 2 by Wouter Hisschemöller

E(5, 13) pattern

Here’s an updated version in which I rewrote the algorithm code. I checked it with several patterns and they all came out correct, so hopefully I have it right this time.

The new Flash app

Download the source files here.

I didn’t update the download ZIP file in the original post. So if you tire of the regular Euclidean patterns you can still download the old version and use those quirky ‘not quite right’ patterns. :-)

The difference

The difference between the old and new version can be seen in the outcome of the algorithm for a pattern of thirteen steps with five notes. The old version generates as outcome:

E(5, 13) = [x..x..x..x.x.]

while it should be:

E(5, 13) = [x..x.x..x.x..]

and the correct calculation broken down in iterations:

[x][x][x][x][x][.][.][.][.][.][.][.][.]
[x.][x.][x.][x.][x.][.][.][.]
[x..][x..][x..][x.][x.]
[x..x.][x..x.][x..]

It appears the old code stops one iteration too early: The wrong pattern is the same as the one before last iteration. Here’s the ActionScript code of the new version. It might be more compact and optimized, but I find it clearer to follow this way.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135

package com.hisschemoller.utils
{
   /**
    * @author Wouter Hisschemoller
    * (c) May 18, 2011
    */
   public class EuclidGenerator
   {
       private var _pattern : Vector.<Boolean> = new Vector.<Boolean>( );
 
       public function EuclidGenerator ( total : int, hits : int )
       {
           if ( hits >= total || hits == 1 || hits == 0 )
           {
               _pattern = new Vector.<Boolean>( );
               if ( hits >= total )
               {
                   for ( var i : int = 0; i < total; i ++ )
                   {
                       _pattern.push( true );
                   }
               }
               else if ( total == 1 )
               {
                   _pattern.push( hits == 1 );
               }
               else
               {
                   for ( i = 0; i < total; i ++ )
                   {
                       _pattern.push( false );
                   }
               }
 
               return;
           }
 
           var bigArray : Vector.<Vector.<Boolean>> = new Vector.<Vector.<Boolean>>( );
           for ( i = 0; i < hits; i ++ )
           {
               var boolArray : Vector.<Boolean> = new Vector.<Boolean>( );
               boolArray.push( true );
               bigArray.push( boolArray );
           }
 
           var smallArray : Vector.<Vector.<Boolean>> = new Vector.<Vector.<Boolean>>( );
           for ( i = 0; i < ( total - hits ); i ++ )
           {
               boolArray = new Vector.<Boolean>( );
               boolArray.push( false );
               smallArray.push( boolArray );
           }
 
           _pattern = bjorklund( bigArray, smallArray );
       }
 
       public function get pattern () : Vector.<Boolean>
       {
           return _pattern;
       }
 
       private function bjorklund ( bigArray : Vector.<Vector.<Boolean>>, smallArray : Vector.<Vector.<Boolean>> ) : Vector.<Boolean>
       {
           /** Done, flatten arrays. */
           if ( smallArray.length <= 1 )
           {
               var resultList : Vector.<Boolean> = new Vector.<Boolean>( );
               for ( var i : int = 0; i < bigArray.length; i ++ )
               {
                   for ( var j : int = 0; j < bigArray[ i ].length; j ++ )
                   {
                       resultList.push( bigArray[ i ][ j ] );
                   }
               }
               for ( i = 0; i < smallArray.length; i ++ )
               {
                   for ( j = 0; j < smallArray[ i ].length; j ++ )
                   {
                       resultList.push( smallArray[i][j] );
                   }
               }
               return resultList;
           }
 
           var fullRounds : int = Math.floor( smallArray.length / bigArray.length );
           for ( i = 0; i < fullRounds; i ++ )
           {
               for ( j = 0; j < bigArray.length; j ++ )
               {
                   var sourceBoolArray : Vector.<Boolean> = smallArray[0];
                   for ( var k : int = 0; k < sourceBoolArray.length; k ++ )
                   {
                       bigArray[j].push( sourceBoolArray[k] );
                   }
                   smallArray.splice( 0, 1 );
               }
           }
 
           var remainder : int = smallArray.length % bigArray.length;
           for ( i = 0; i < remainder; i ++ )
           {
               sourceBoolArray = smallArray[0];
               for ( k = 0; k < sourceBoolArray.length; k ++ )
               {
                   bigArray[i].push( sourceBoolArray[k] );
               }
               smallArray.splice( 0, 1 );
           }
 
           /** Split result in two new arrays. */
           var newBigArray : Vector.<Vector.<Boolean>> = new Vector.<Vector.<Boolean>>( );
           var newSmallArray : Vector.<Vector.<Boolean>> = new Vector.<Vector.<Boolean>>( );
           for ( i = 0; i < bigArray.length; i ++ )
           {
               if ( remainder == 0 )
               {
                   newBigArray.push( bigArray[i] );
               }
               else
               {
                   if ( i < remainder )
                   {
                       newBigArray.push( bigArray[i] );
                   }
                   else
                   {
                       newSmallArray.push( bigArray[i] );
                   }
               }
           }
 
           return bjorklund( newBigArray, newSmallArray );
       }
   }
}

Audio Editor 1.1

Its new feature is the loop function. Toggle the loop button and playback loops between the start marker (S) and the end marker (E). Just like you’d expect.

Next to this single new feature there’s a new waveform display that I really like.
In the old version the waveform shape was rendered by drawing vertical lines next to each other for each pixel from left to right. The new version draws a much nicer vector shape that is less processor intensive as well.

The Flash app

Download the ActionScript source code and files for this project here.

What happens is that for each pixel the maximum and minimum sample value is polled. Those values are stored in two arrays. Then the array of maximums is used to draw the top half of the waveform and the array of minimums for the bottom half. Together the two halves form a filled shape.

Like in the earlier waveform display the render method changes when zoomed in. If the zoom level is high enough (when one pixel represents less than forty samples) the filled shape is replaced by a single line. I find a single line easier for close inspection of an audio signal and most audio editors work like this.

The mouse interaction remains the same as well: Click on the waveform and drag up or down to zoom and left or right to move, and even both at the same time! I still like this behaviour a lot. First saw it in Steinberg WaveLab.

A note: This waveform display works well for audio files of a minute or less in length. The longer the file the less precise the waveform gets. This is because a fixed number of samples is polled for each pixel. If the file is long one pixel represents a lot of audio, and if it’s polled at too large intervals the polling results get less and less precise. I didn’t increase the number of polls because I found that slows down the screen redraws too much.

Snd - Systems Medley

Euclidean patterns

The formula is simple really. It’s an algorithm to distribute an amount of notes as evenly as possible over a period of time, where time is divided in equal parts. A very basic example: Say you have one measure of sixteen sixteenth notes and there are four notes to be played, then this is how those four notes would be equally distributed by the algorithm:

x . . . x . . . x . . . x . . .

A basic house or techno kick drum pattern. There’s four notes on sixteen steps, so the division is easy, 16 / 4 = 4. It gives you one note every four steps. But it gets more interesting when the numbers don’t divide so easily. Take five hits on sixteen steps. Then the pattern becomes more irregular:

x . . . x . . x . . x . . x . .

It’s these irregular patterns that create the best rhythms. Especially when several patterns are combined. In Ruin & Wesen’s article there’s a few audio examples and here’s a video on blip.tv of a MiniCommand MIDI controller generating Euclidean rhythms on a Machinedrum.

The algorithm

The algorithm is very well explained in the article I mentioned, so I won’t repeat it here. There’s another article on the weblog of Robin Price which includes a Max/MSP example, there’s a Pure Data example by Dave Poulter as well as one in Java by Kristopher Reese.

Absolutely worth reading is the paper by Godfried Toussaint who as far as I know first recognized the connection between the Euclidean algorithm and musical rhythms. It’s downlaodable as a PDF: “The Euclidean Algorithm Generates Traditional Musical Rhythms”.

The Flash app

Download the full source code here.

This is a Flash app I made over the weekend to quickly try out combining rhythms. It’s by no means complete, but it does work:

• To create a new pattern doubleclick somewhere in the lower panel. A basic sixteen step quarter note pattern appears where you clicked.
• To edit the pattern doubleclick it’s center circle. A settings popup appears with sliders for pattern length and number of hits. Change these to quickly get a feel for how these Euclidean patterns work. The ‘Sound’ dropdown menu lets you select a different sample for the pattern to play.
• Click and drag the center circle to move a pattern.
• The controls at the top are obvious: Start / stop playback, adjust volume and change tempo (in beats per minute).

You will notice that when you create a new pattern while patterns are already playing, the new pattern is not synchronized. To synchronize all patterns stop and restart playback.

Euclid of Alexandria

I posted a lot of experimental stuff on this weblog over the past year. Or components that might have their use when built into proper applications. But this is my first web app that’s useful on it’s own.

This is a simple Flash audio editor that works in the browser. Everything it does is done by ActionScript code inside the application. There’s no backend that does the difficult work. So it would function just as well in the standalone Flash Player or exported as an Air application as it does in the browser.

This audio editor has only a very basic set of features. Here’s the things it can do:

• Open mono and stereo WAV and MP3 files.
• Select part of the audio file by graphically adjusting the start and end point.
• Change the pitch by adjusting the playback speed.
• Reverse the file so it plays backwards.
• The result of these adjustments (start and end, pitch, reverse) can be saved as a new stereo WAV file.

The Flash file

Download the source files here.

Code

This project is built on the PureMVC framework again. It’s what I’ve been using for all my projects the last half year and I’m still really happy with it. The code gets more complicated with each project but I still have a clear overview of where everything is and I can even find my way quickly around older projects.

Here’s a short overview of the main building blocks of the audio engine:

Audio engine overview

• AudioDriverProxy listens to the SampleDataEvent. On each event it sends a request to the audio engine (AudioEngineProxy) to fill the buffer with samples.
• AudioEngineProxy holds all the instruments (of type IAudioInstrument) that produce sound. When it receives the request for samples from the audio driver it passes this request on to all the instruments that are registered with the engine.
• IAudioInstrument is (the interface for) a general instrument. An instrument doesn’t generate sound itself, but it maintains a list of IAudioProcessor objects that produce the actual audio. Each processor is a note that the instrument currently plays. So if an instrument was to play a three note chord three processors would be added to the list. And when a note is finished the processor is removed from the list. Besides that IAudioInstrument has an API for the instrument to be played. These are MIDI inspired functions like noteOn(), noteOff() and programChange().
• IAudioProcessor is the object that produces sound. When an instrument receives a buffer from the audio engine it passes the buffer on to each of its processors. The processor loops through all the samples in the buffer and adds its own values to it. An IAudioProcessor can generate sound like a synthesizer or play back existing sound like a sampler. It doesn’t matter as long as it implements the IAudioProcessor interface.
• AudioOutputProxy is the last stage in the audio generation process. Once all instruments have contributed their sounds to the buffer, the audio output is the last one that gets the buffer with sample data from the audio engine. Here final adjustments to the sound can be made. In the case of the audio editor the only adjustment that can be made is to set the overall output volume.

The audio editor is a really simple implementation of this audio engine: There is only one instrument – the editor itself (WaveEditorProxy) – and it can play only one sound at a time: the audio being edited (PitchedSampleProcessor as IAudioProcessor).

The Harrington 1200, a state of the art songwriting computer.

I thought about it a bit and came up with two different scenarios where audio and visuals need to be synchronized:

1. The first is where a sound plays and something visual on screen happens to accompany that audio event.
2. The second is the other way around. Something happens on screen that should be accompanied by a sound.

In the first case the animation is the initiator, in the second case the sound. For the second case I included a Flash example here with the source code available.

1. Animation follows audio

This is the simpler scenario. For example: There’s an online drum machine built in Flash that plays a drum sound on every beat. A kick drum in a techno track for example. On the machine’s interface there’s a small red LED light (a MovieClip) that should blink each time the drum hits.

The drum machine has a steady rhythm set in BPM (Beats Per Minute). So once the machine has started it’s always known when the next drum hit will happen. Here’s a graph of the events:

• (a), (b), (c) … are the moments in time when a SampleDataEvent is received and the buffer is filled with samples.
• (p) is where the drum hit should happen. Because the drum BPM and buffer refresh rate are completely independent the drum hit will almost always start somewhere between two buffer refreshes.
• (q) is where the drum sound is actually heard through the speakers. There is a delay between when the samples are sent out from Flash to when they are played through the speakers. This is the latency; the time needed for the soundcard to process the samples and send them as analog audio to the soundcard’s outputs. Latency is the duration from (p) to (q).

To synchronize the two it’s a simple addition. Duration (a)(p) plus duration (p)(q).

• (a)(p) is the time from the first sample in the buffer to the first sample of the drum hit (and to convert this duration in samples to a duration in milliseconds it must be divided by 44.1 (sample rate / 1000)).
• (p)(q) is the latency. That’s ((SampleDataEvent.position / 44.1) – SoundChannelObject.position). (See the AS3.0 Language reference on SampleDataEvent.)
• The result is the total delay in milliseconds. A Timer could be started when the buffer is filled at (a) so that the LED blinks at the frame nearest (q).

2. Audio follows animation

Download the source code of this example.

This second scenario is a bit more complicated. Hopefully someone else has a simpler solution. But until then this is the solution I use.

Click the ‘Start’ button in the example above: Balls bounce on the floor and each time one of them hits the floor a ‘boing’ sound plays. A simple implementation of Newton’s laws of motion.

A graph of the events:

• (a), (b), (c) … are the times when a buffer is requested and filled. Same as in the previous example.
• (1), (2), (3) … are frames where Flash updates the screen (Event.ENTER_FRAME).
• (p) is where the ball hitting the floor is calculated to happen.
• (q) is where the sound is heard.
• (r) is where the ball hitting the floor is drawn on screen.

What’s going on?

The trick is to delay drawing the animation for a ‘latency’ amount of time.

1. First calculate where the ball WOULD be (but NOT draw it).
2. Fire a sound if the calculation shows the ball would hit the floor.
3. Draw the animation with a delay determined by the latency.

That is the principle of how it works. The actual implementation is a bit more involved:

• At (a), (b), (c) … (each SampleDataEvent.SAMPLE_DATA) the position of the ball is calculated. The current time and position of the ball are stored in a Vector called _history.
• If the calculation shows the ball will hit the floor during the current buffer (at (p)):
  • The time from the start of the buffer till the hit is calculated (a)(p).
  • The sound is started at time offset (a)(p) in the buffer.
• Meanwhile at (1), (2), (3) … (at each Event.ENTER_FRAME) the screen is updated.
• To know where to draw the ball the _history of the ball is used.
• First get the current time and subtract the latency. This is the moment in the past that should be drawn on screen.
• Search in the balls _history for that moment in the past.
• It’s very unlikely however that the position at exactly the required millisecond is stored. More likely somewhere in between two stored moments in time. So get the closest positions before and after the requested time and use linear interpolation to draw the ball at the right position between the ‘before’ and ‘after’ records.

So what’s going on?

I’m afraid this explanation might be more complex than the actual code. So I’ll add some of the relevant ActionScript.

To understand it you should know that there is an AudioDriver class that listens to the SampleDataEvent. Each SampleDataEvent it sends an ENTER_BUFFER notification (this is PureMVC) to notify all Ball instances to store their position. To do this the function enterBuffer is called on each Ball instance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

public function enterBuffer() : void
{
    var hit : Boolean;
 
    if(_isDragging)
    {
        _vy = 0;
    }
    else
    {
        _vy += _gravity;
        _y += _vy;
 
        /** If the ball hits the floor. */
        if(_y > _floorY - _radius)
        {
            var distance : Number = _vy;
            var overlap : Number = _y + _radius - _floorY;
            _hitAlpha = (distance - overlap) / distance;
 
            _y = _floorY - _radius - overlap;
            _vy *= -1;
            hit = true;
 
            dispatchEvent(new Event(Ball.BOUNCE));
        }
    }
 
    /** Store as data for later use. */
    _history.splice(0, 0, new BallVO(_x, _y, _vx, _vy, hit));
 
    /** Keep maximum 100 history items so the maximum latency is buffer * 100. */
    if(_history.length > 100)
    {
        var n : int = _history.length;
        while(--n > 99)
        {
            _history.pop();
        }
    }
}

This function does what was expected: A new position is calculated and stored in a BallVO value object, the BallVO is added to the _history and a bit of maintenance is done: Keep _history’s length within bounds.

The only other thing is the _isDragging Boolean. Balls can be dragged to a new position.

The AudioDriver also sends an ENTER_FRAME notification. This is just like a regular Event.ENTER_FRAME from a DisplayObject except that it has a AudioStreamInfoVO value object as an argument. The AudioStreamInfoVO has a ‘latencyInMS’ property that holds (as you’ll have guessed) the latency in milliseconds that the animation should be delayed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

public function enterFrame(infoVO : AudioStreamInfoVO) : void
{
    graphics.clear();
    graphics.lineStyle(2, 0xFFFFFF);
    graphics.beginFill(0xBBBBBB, 0.1);
 
    if(_isDragging)
    {
        graphics.drawCircle(_x, _y, _radius);
    }
    else
    {
        var time : Number = new Date().getTime() - infoVO.latencyInMS;
 
        var i : int = -1;
        var n : int = _history.length;
        while(++i < n)
        {
            /** Get the data from immediately before and after 'time'. */
            if(_history[i].time < time && i > 0)
            {
                var before : BallVO = _history[i];
                var after : BallVO = _history[i - 1];
                var alpha : Number = (time - before.time) / (after.time - before.time);
 
                /** If a bounce on the floor happened between before and after. */
                if(before.vy > 0 && after.vy < 0)
                {
                    /** Linear interpolation to correct the vertical position. */
                    var maxY : Number = _floorY - _radius;
                    var partBeforeBounce : Number = maxY - before.y;
                    var partAfterBounce : Number = maxY - after.y;
                    var bounceAlpha : Number = partBeforeBounce / (partBeforeBounce + partAfterBounce);
 
                    if(alpha < bounceAlpha)
                    {
                        var y : Number = before.y + (before.vy * bounceAlpha);
                    }
                    else
                    {
                        y = after.y + (before.vy * (1 - bounceAlpha));
                    }
                }
                else
                {
                    /** Regular linear interpolation for a ball in mid flight. */
                    y = before.y + (alpha * (after.y - before.y));
                }
 
                if(before.hit) graphics.beginFill(0xFFFFFF);
                graphics.drawCircle(_x, y, _radius);
 
                break;
            }
        }
    }
}

So in this function there’s a variable ‘time’ which holds the moment in time in the past that must be drawn, ‘before.time’ and ‘after.time’ and a resulting ‘alpha’ which is the interpolation value between before and after. The resulting ball position calculation is this:

y = before.y + (alpha * (after.y – before.y));

Isaac Newton

All this code is not optimized at all. In this example I wanted to keep it all as clear as possible to read and understand.

This is a small experiment I started half a year ago but never finished. I looked it up again last week because I needed a Flash AR example for a project at work. Here I improved it a bit to make it more presentable.

What it is

It’s basically a sort of theremin. Hold the marker in front of your computer’s webcam and a sound starts to play. Move the marker and the sound changes.

The Flash file

All source files can be downloaded here. They’re a bit messy however. This was just a quick update to an abandoned older experiment, I didn’t take care to write clean code.

To try this you’ll need the marker to move in front of the webcam. Download it here as a PDF file. Print the file and – for the best results – glue it on a piece of cardboard.

The video

Because I know it’s a lot of hassle to download and print the marker I recorded a short demonstration screencap video to show how it all works.

How to use it

Click “Allow” on the Flash player settings screen. This allows Flash to switch on the webcam and use the images it films. Then hold the printed marker in front of the webcam to generate some sounds:

• With the marker at the center of the video a sine wave plays a C note.
• Move the marker slowly up and the pitch rises in steps, move down and the pitch goes down. It plays a pentatonic scale in the key of C minor. Two octaves up and two down. With added flattened fifth to make it sound more bluesy.
• Move the marker to the right to add vibrato. The more to the right the faster and the more intense the vibrato gets.
• Move left from the center and a ‘note repeat’ effect starts. At the outer left edge of the webcam the repeat effect doubles in speed. When the marker is moved vertically at the left new notes start and combine with the previous notes that still play and produce a harp-like sound. Very nice.

This idea could be taken a lot further. There’s more information to be had from the marker:

• The distance from the webcam can be measured. To lower the volume the farther the marker is away from the webcam for instance.
• The three rotations in 3D are available: Rotation around the X, Y and Z axis.

The only obstacle for direct musical use is the latency. The delay between a movement and hearing the resulting sound is too big to be able to play along with a record for example.

For the augmented reality code I used FLARToolkit. More information can be found at the English FLARToolKit page and the Start Up Guide.

MP3 wave display is orange.

Wave Display 2 is an improved version of the last one (in the previous post). A few things had to be rewritten in the code for waveform rendering, and there were some extra features I wanted to include to make the application a bit more interesting to use.

The new features in this version:

• Audio playback. You can now listen to the loaded MP3 file. Click ‘play mp3′ or press the SPACE bar to listen.
• Draggable start and end locators to select only part of the sound. Drag the ‘S’ and ‘E’ locators to set the start and end point.
• Reverse playback. Click the ‘reverse’ button to toggle reverse playback.
• Adjustable playback speed. Drag the ‘transpose’ slider to change the playback speed. The speed is adjusted so that the pitch changes from one octave down to one octave up in semitone steps.
• Drag to zoom. Press and drag the mouse on the display area for finer control: Drag vertical to zoom and horizontal to move the waveform. I always really liked this behaviour in Steinberg WaveLab where I first saw it, and where I now copied it from.

Click here to download all source files in a ZIP.

Transpose by playback speed

Transpose works fairly straightforward. The transpose value is set with the slider. And every time the slider is moved the new transpose value is stored.

Each time playback is started, the transpose value is passed to the class PitchedSamplePlayer that processes the sound. However, PitchedSamplePlayer uses a multiplier to set the playback speed instead of a semitone value. For example 0.5 is playback at half speed. So the transpose value in semitones must be converted to playback speed. That’s done with this conversion:

1
2
3
4
5
6
7
8

/**
 * Covert semitone shift to playback speed multiplier.
 * For example: -12 returns 0.5, 12 returns 2, 0 returns 1.
 */
public static function semitoneToRate(semitone : Number) : Number
{
    return Math.pow(2, semitone / 12);
}

Then inside PitchedSamplePlayer linear interpolation is used to change the playback speed and maintain acceptable sound quality. I took the code for pitching a sample from this MP3 Pitch example by Andre Michelle, which builds on Lee Brimelow‘s post Adjusting audio pitch in Flash Player 10.

Reverse playback

The display and the sample player both read samples from a ByteArray. I had the choice of reversing the whole ByteArray or leaving the ByteArray as it is and read from it backwards. From end to start.

It seemed best to reverse the order of the ByteArray because the audio processing and display rendering code can stay the same and as short and fast as possible:

1
2
3
4
5
6
7
8
9
10
11
12

/** Reverse the order of samples in samplerPresetVO.soundData.byteArray. */
var bytesPerSample : int = AudioDriverProxy.BYTES_PER_SAMPLE;
var numBytes : Number = samplerPresetVO.soundData.numSamplesTotal * bytesPerSample;
var tempByteArray : ByteArray = new ByteArray();
samplerPresetVO.soundData.byteArray.position = 0;
tempByteArray.writeBytes(samplerPresetVO.soundData.byteArray);
samplerPresetVO.soundData.byteArray.position = 0;
 
for(var offset : Number = (numBytes - bytesPerSample); offset >= 0; offset -= bytesPerSample)
{
   samplerPresetVO.soundData.byteArray.writeBytes(tempByteArray, offset, bytesPerSample);
}

What happens here is:

• Copy all sample data to a temporary ByteArray.
• Read two samples (or one stereo sample) at a time from the temporary ByteArray, reading backwards from end to start.
• And write each stereo sample to the original ByteArray, writing normally from start to end.

The waveform display

2010! A new year, a new decennium even! This year I want to be more consistent in publishing articles on this weblog. That’s my new year’s resolution.

The first one of the new year is more of a utility than a complete application. It’s a wave display; It loads an MP3 file and displays the sound as a graph of the waveform. It makes no sound. A simple waveform display like this might be a useful component in a larger application where audio needs to be edited. Here it’s on its own because I was figuring out how to best program such a component. So this application is more experimental than useful.

Click here to download all source files in a ZIP.

How to use it

A default MP3 file is loaded and displayed at startup. But you can load your own MP3. Click the ‘load mp3′ button and browse to an MP3 file on your computer. Opening a large MP3 file might take a long time however. Best to try with short files. Five minutes or less for example.

Once the MP3 is loaded and displayed there’s several controls to zoom in closer on details of the waveform:

• Zoom in to a detail of the waveform with the ‘zoom horizontal’ slider.
• When zoomed in you can move through the waveform with the scrollbar directly under the waveform.
• For more precise movement once zoomed in drag the mouse left or right on the waveform.
• Magnify the waveform vertically with the ‘zoom vertical’ slider. This is useful to inspect very quiet parts of the sound.

Some information is shown above the display

• In Flash the sample rate is 44100 samples per second. When the waveform is at the highest zoom level, it shows one sample per pixel. (So one second of audio would take up 44100 pixels!)
• At lower zoom levels each pixel displays the average amplitude of several pixels. The amount is displayed at ‘samples per pixel’.
• The first and last sample in view are shown at ‘first sample’ and ‘last sample’.

Some notes on the ActionScript

The biggest problem with the display was to get it to respond quick to zoom and position changes. It should use as little calculations as possible to refresh the display. Three things reduce the number of calculations (and so the processor load) here:

Each change in zoom level or position results in a changed portion of the waveform that is actually shown in the display. When the display needs updating, it’s first determined what the first and last sample are that will be displayed. Then only that portion of the file is processed. That means that especially at high zoom levels only a tiny fraction of the file needs to be calculated.

But at low zoom levels still a lot of samples need to be averaged: To show one minute of waveform 44100 x 60 = 2,646,000 samples must be processed. More than two and a half million! To reduce this number samples are skipped. The more samples there are, the more are skipped.

This bit of ActionScript shows how these two reductions are done:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

/**
 * Sets the zoom level of the display. Measured in how many samples one horizontal pixel of the display represents.
 * @param inSamplesPerPixel : How many samples one horizontal pixel of the display represents.
 * @param inCenterSamplePosition : The center or focus of the zoom. 0.5 is the horizontal center. 
 */
private function setCurrentSamplesPerPixel(inSamplesPerPixel : Number, inCenterSamplePosition : Number = 0.5) : void
{
  if(inSamplesPerPixel == mCurrentSamplesPerPixel) return;
 
  // get the sample that is the center of the zoom
  var centerSample : int = mFirstSampleInView + (mNumSamplesInView * inCenterSamplePosition);
 
  mCurrentSamplesPerPixel = Math.max(mMinSamplesPerPixel, Math.min(inSamplesPerPixel, mMaxSamplesPerPixel));
  mNumSamplesInView = mCurrentSamplesPerPixel * mDisplayWidth;
  mFirstSampleInView = centerSample - (mNumSamplesInView * inCenterSamplePosition);
 
  // polling stuff
  mPollingFrequency = Math.ceil(mCurrentSamplesPerPixel / 20);
  mNumPolls = int(mCurrentSamplesPerPixel / mPollingFrequency);
  mBytesToSkip = (mPollingFrequency - 1) * SoundProxy.BYTES_PER_SAMPLE;
 
  if(mFirstSampleInView < 0) mFirstSampleInView = 0;
  if((mFirstSampleInView + mNumSamplesInView) > mNumSamplesTotal) mFirstSampleInView = mNumSamplesTotal - mNumSamplesInView;
 
  updateGraphic();
}

And the third thing to reduce the processor load is a simple timer. There’s a maximum of how often the display can update. In this case a maximum of once every 50 milliSeconds was fine. The timer script was added at the start of the update function:

1

private var mUpdateDelayTimer : Timer = new Timer(50, 1);

1
2
3
4
5
6
7
8
9
10

/**
 * Redraws the display.
 */
private function updateGraphic() : void
{
  // delay between updates
  if(mUpdateDelayTimer.running) return;
  mUpdateDelayTimer.start();
 
  // etcetera ...

That’s it for now. Download the source files to have a closer look at the code or to use it in your own projects.

This project is build with PureMVC as framework again. I like it more and more.
The UI components I used here are very much based on Keith Peters‘ Minimal Comps.
To load an MP3 file and use it in Flash as a Sound object I used FlexibleFactory‘s MP3FileReferenceLoader library.
For debugging I used the Yalog logging tool by Stephan Bezoen. Very nice Air application.