PROCESS PIECES AS PAPER PROGRAMS

A ScoreCard Workshop at NIME'24

with your host, ijc

Welcome, dear participants, to Process Pieces as Paper Programs!

Before we begin, here are some handy links that you might want to refer back to over the course of the workshop:

Schedule

This is a half-day workshop (14:30-18:30), and we’ll be following this rough schedule (which may be adjusted as needed):

Introduction to ScoreCard

In this segment, I’ll talk about a bit about what ScoreCard is and what you can do with it!

I will be presenting with these slides, if you’d like to follow along on.

Hands-on Tutorial

Now that you know a bit about ScoreCard, let’s see how we can create our own. We’ll work through several examples of increasingly complexity.

Creating ScoreCards

In order to create a scorecard, we need to produce a WebAssembly (Wasm) binary that can generate audio samples.

This requires a compiler toolchain capable of targeting Wasm, such as clang (possibly wrapped by Emscripten). But, to save you the hassle of installing this on your machine during this workshop, you can instead use the web-based ScoreCard Creator.

Oh, don’t worry about me—the compiler still runs on your machine. (I am much too cheap/lazy to set up a backend compilation service. ☺) But the compiler itself has been compiled to WebAssembly, so that it can run in the browser… to produce binaries that run in the browser!

Aside from being really nifty, this trickery will allow you to create ScoreCards without first installing (per se) Emscripten. Hooray! (This is possible thanks to Jorge Prendes’s work on Emception!)

If you would like to set up Emscripten to run on your machine natively, or if you’d like to try creating scorecards with languages other than C (such as Rust or Zig), we can do that too! The goal of the Creator is just to get you up and running as quickly as possible, and to ensure that nobody is blocked by installation issues. ♥

Structure of a ScoreCard

Okay, enough prelude, let’s get some actual code on a card already!

// 4ever'33"
float process() {
    return 0;
}

This is about as simple as it gets: the silent card.

This demonstrates the one thing that a program must do to serve as a scorecard: generate samples by exporting a process() function.

(Technically the function is ultimately exported as p to save precious bytes, but you don’t need to worry about that.)

The above example outputs the same sample forever. To make things more interesting, we need some state, so that subsequent calls to process() (which takes no arguments) can yield different results.

// bzzz
float t = 0.0f;

float process() {
    t += 0.01f;
    if (t > 1.0f) t -= 2.0f;
    return t;
}

(Note that you can click “Show Card” above to see the compiled card for this program!)

In this case, our state is explicitly declared as a global variable, which process() mutates every time it is called.

However, we can also have implicit state if we use standard library functions that have side-effects, such as C’s rand().

// pssshhh
#include <stdlib.h>

float process() {
    return rand() / (float)RAND_MAX * 2 - 1;
}

Notice that the size of the binary has actually increased from the previous one, despite the program getting shorter. This is due to our use of rand(); the implementation of rand() must now be included in the binary by the linker. Because these programs are statically-linked, we pay for what we use (and nothing else)!

This principle is an important constraint when composing scorecards, and it’s why we’re using a language like C rather than, say, Python or Pure Data. To analogize, if you want a grand piano at the gig, you’re going to have bring it in your van. It won’t fit in your van? Perhaps consider an electronic keyboard instead.

Fun exercise: see what happens if you call sin() or sinf().

Interlude: The Value Shopper

At this point, things may be feeling a little cramped, a bit spartan. Our examples so far have included… silence, an aliased sawtooth, and noise. Trying to play a single sine wave totally broke the bank!

But don’t give up hope. I promise we really can do some interesting things that fit within the confines of a card’s QR code; we just have to get a little creative. In particular, we must shop for value. We don’t need to do less, but we need to make less do more.

Composing a card is an exercise in concision. How much code does it take to explain your idea to the computer? Are you repeating yourself? Use the tools at your disposal keep it brief. These are the core classics of abstraction: loops and functions. (Beware of macros.) Code is compression.

Beyond coding techniques, hunt for deals in the realms of synthesis and music theory. Often, great insights require only a brief description. For example, here are some cheap synthesis classics that have withstood the test of time: FM synthesis. Karplus-Strong. The State Variable Filter.

These are great value: they’ll give you a lot of bang for your buck (or byte, as it were). These techniques take just a few lines of code to describe, but they open up worlds of timbral possibilities.

deck.h

To aid you in your quest for value, I have created a little library (in the form of a header file) to give you some cheap utilities.

Case study: Quirky FM ramps

Let’s modulate some frequencies.

First, let’s just play a single frequency. Rather than writing our oscillation function from scratch, we’ll use one of the cheapo waveforms provided by deck.h.

#include "deck.h"

float phase = 0;
float freq = 500;

float process() {
    return sqr(&phase, freq);
}

Ta-da. sqr() takes in phase and frequency, and it generates a square wave at the requested frequency. (deck.h includes the ScoreCard sample rate, which is fixed at 48000 kHz). saw() and tri() are also included.

Note that the state is in phase, which sqr() mutates (hence passing a pointer).

We can combine two instances of sqr() — using the output of one to modulate the frequency of the other — to get a more complex timbre.

#include "deck.h"

float phase = 0;
const float base_freq = 500;
float mod_phase = 0;
const float mod_ratio = 10;
float mod_freq = base_freq / mod_ratio;
float mod_depth = 0.5;

float process() {
    float mod = mod_depth * sqr(&mod_phase, mod_freq);
    float freq = base_freq * (1 + mod);
    float out = sqr(&phase, freq);
    return out;
}

We can apply some envelopes and ramps to make the sound more dynamic…

#include "deck.h"

float t = 0, dur = 1.0, start = 300, end = 900, phase = 0;
float mod_depth = 0.5, mod_phase = 0, mod_ratio = 10;

float process() {
    if (t > dur) return 0;
    float freq = ramp(t, dur, start, end);
    float mod_freq = freq / mod_ratio;
    freq *= 1 + mod_depth*sqr(&mod_phase, mod_freq);
    float out = sqr(&phase, freq) * env(t, dur);
    t += dt;
    return out;
}

Now we have a sound with a bit of character. We can get a lot of different sounds with character by randomizing some of the parameters. While we’re at it, let’s give this thing a name.

#include "deck.h"

card_title("Quirky FM ramps");

float t, dur, start, end, phase;
float mod_depth, mod_phase, mod_ratio;
float dur_options[] = {0.25, 0.5, 1.0, 2.0, 4.0};

void setup(unsigned int seed) {
    srand(seed);
    dur = choice(dur_options);
    start = uniform(0, 1000);
    end = uniform(0, 1000);
    mod_ratio = uniform(1, 100);
    mod_depth = uniform(0, 1);
}

float process() {
    if (t > dur) return 0;
    float freq = ramp(t, dur, start, end);
    float mod_freq = freq / mod_ratio;
    freq *= 1 + mod_depth*sqr(&mod_phase, mod_freq);
    float out = sqr(&phase, freq) * env(t, dur);
    t += dt;
    return out;
}

This version of the card introduces the optional things that a ScoreCard can do. (Recall that the one thing it must do is generate audio samples via process().)

In addition to process(), we define setup(), which can take a random seed and use it however it likes. In this case, we use it to seed the standard library’s PRNG via srand(), and then use the convenience function uniform() and the macro choice() to leave some things up to (pseudo-)chance. (Try pressing the reset button to hear different variants! You can also try locking the seed or setting it manually.)

Finally, we’ve added a call to card_title(), a macro which allows us to declare a string that the ScoreCard player can display to the user. (This string is embedded in the WebAssembly binary.)

Higher-Level Structure

At this point, we’ve seen how to make some cards that make some nifty little sounds. But this work has remained firmly in the realm of synthesis, perhaps edging into sound design. How can we represent musical elements beyond timbre, such as notes, rhythms, and events, or even phrases and sections?

Well, we’re programming, so if we want to talk about higher levels of abstraction… we can just build the abstraction! For example, we can readily define structures to represent notes or other musical materials, and then write functions that operate on these data types.

But, there’s a catch. process() is still called once per sample by the player, and this drives the entire program. This means that when process() is called to generate the next sample, we have to figure out where we are in the higher-level structure and react accordingly. If we’re in the middle of a note, for example, we should run all the oscillators and envelopes and so on in order to generate the next sample of that note. But if we’re at the end of a note, or at the very beginning of playback, we need to execute some other logic to determine what note to play!

If we approach this naively, we’ll end up with a big old state machine, probably in the form of a gnarly switch branching on the various values of an enum. In practice, this is quite awkward, as it forces us to flatten out all of our control flow and manually connect the states. If the card were driving instead of the player, we wouldn’t have to deal with this inversion of control and we could simply use ordinary control flow constructs (such as loops).

Fake Generators Yield Real Fun

The good news: languages have figured out to deal with the problem of preserving control flow and state across multiple “entries” into a program. The general answer is coroutines. Different languages have different flavors and names (e.g. “generators” in Python and JS), but the key point is that you have something that looks like a function, and which can use all the conventional control flow you’d expect, but which can “return” (yield) multiple times. Each time it’s called again (resumed), it simply picks up from where it left off.

The bad news: C does not have coroutines.

The funny news: Simon Tatham has demonstrated that we can fake coroutines reasonably well by abusing other language features, particularly macros and Duff’s device. What a language.

Anyhow, deck.h includes an implementation of generators based on Tatham’s work, and you can use them to encode higher-level structure in (apparently) normal control flow!

Let’s look at a quick example. We’ll play a melody repeatedly, with some logic to vary subsequent repetitions.

#include "deck.h"

card_title("lick spiral");

struct note_t {
    uint8_t pitch;
    uint8_t dur;
};

struct note_t notes[] = {
    {62, 1}, {64, 1}, {65, 1}, {67, 1},
    {64, 2}, {60, 1}, {62, 1},
};

float process() {
    static float phases[2];
    static float freq, dur, base_dur = 0.25, t = 0;
    static int offset = 0, i;
    gen_begin;
    for (;; offset = (offset + 1) % 12) {
        for (i = 0; i < SIZEOF(notes); i++) {
            freq = m2f(notes[i].pitch + offset);
            dur = notes[i].dur * base_dur;
            for (; t < dur; t += dt) {
                float pos = t / dur;
                pos = (i + pos) / SIZEOF(notes);
                pos = (offset + pos) / 12;
                float x = pos*tri(&phases[0], freq/2)
                    + (1-pos)*tri(&phases[1], freq);
                yield(x * ad(t, dur/8, dur*7/8));
            }
            t -= dur;
        }
    }
    gen_end(0);
}

We’re using three generator-related macros here. gen_begin and gen_end() are basically boilerplate that you need to turn your function into a “generator”. The argument to gen_end() defines what your generator will return after it’s exhausted (reached the end of the control flow), so it needs to match the function’s return type. yield() is where the action is, though — when we yield(), we “suspend” the function’s execution, returning the argument passed to yield(), and we’ll pick up from the same point when the function is called again.

Thus, in this example, we can simply iterate through our melody using a for loop, yield()ing whenever (from “our perspective”) we feel like it.

Note that there’s one other change we add to make to our function: all the locals are static. We have to do this because, ultimately, process() really is just a normal function getting called multiple times, and locals are destroyed when a function exits. We need them to persist between calls, hence — static.

If you’re familiar with static and thinking ahead a bit, you might wonder how this will work when we have multiple instances of a generator (an issue that doesn’t affect process() but might affect other generator functions). The answer is, it won’t! You need to put your state somewhere else. Put it in a struct, and you can have as many instances of generator running at the same time as you want. deck.h includes re-entrant versions of the generator macros (cleverly prefixed with re-) for just this purpose, although the resulting code is slightly more awkward.

Composing Generators

Here’s a more involved example in which we compose two generators together. process() itself is a generator function, but it just deals with synthesizing notes (and rests). It pulls notes from another function, arp(), which is also a generator: a higher-level one that emits events!

#include "deck.h"

card_title("arpeggios");
setup_rand;

const uint8_t scale[] = {0, 2, 3, 5, 7, 9, 10, 12, 14};

const uint8_t chords[][5] = {
    {0, 2, 4, 6, 7},
    {0, 2, 3, 5, 7},
    {0, 1, 3, 6, 7},
    {1, 3, 4, 6, 8}
};

typedef struct {
    uint8_t length;
    const uint8_t *order;
} pattern_t;

const pattern_t patterns[] = {
    {3, (uint8_t []){0, 1, 2}},
    {3, (uint8_t []){2, 1, 0}},
    {4, (uint8_t []){0, 1, 2, 3}},
    {4, (uint8_t []){4, 3, 2, 1}},
    {4, (uint8_t []){0, 1, 2, 4}},
    {4, (uint8_t []){0, 2, 1, 2}},
    {6, (uint8_t []){0, 1, 2, 3, 2, 1}},
    {8, (uint8_t []){0, 1, 2, 3, 4, 3, 2, 1}},
};

typedef struct {
    int pitch;
    float dur;
} event_t;

event_t arp() {
    const float dur = 60.0 / 110;
    static int c, i, j, octave;
    static pattern_t pattern;
    gen_begin;
    for (;;) {
        octave = rand() % 3;
        pattern = choice(patterns);
        for (c = 0; c < SIZEOF(chords); c++) {
            for (i = 0; i < 4; i++) {
                for (j = 0; j < pattern.length; j++) {
                    int deg = chords[c][pattern.order[j]];
                    yield((event_t){
                        scale[deg] + 48 + octave*12,
                        dur / pattern.length
                    });
                }
            }
        }
    }
    gen_end((event_t){});
}

float process() {
    static event_t event;
    static float t = 0, freq, phase;
    gen_begin;
    for (;;) {
        event = arp();
        if (event.pitch == 0) {
            sleep(t, event.dur);
            continue;
        }
        freq = m2f(event.pitch);
        for (; t < event.dur; t += dt) {
            float amp = ad(t, 0.03, event.dur - 0.03);
            yield(saw(&phase, freq) * amp);
        }
        t -= event.dur;
    }
    gen_end(0);
}

Break

Phew! After all that, it’s probably a good time to get up, stretch, and walk around. Consider hydrating!

Card Tricks

At this point, if you’re not an incurable C hacker, you may be asking a reasonable question:

“Okay, it’s kinda cool (and maybe a bit freaky) that you can abuse C to make it sort of usable for composing cards, but… do I really have to write this stuff in C? I just want to make little musical cards!”

The answer is no: you don’t have to write this stuff in C. As mentioned previously, you pay for whatever you bring with you, so you need a language that allows you to only bring what you need into a self-contained executable. (E.g. if we have a Pd patch and only use osc~ and dac~, the “self-contained” size is the size of our patch + the size of Pd, and Pd doesn’t get any smaller just because our patch is simpler.) But other compiled languages let you only bring what you need: in particular, Zig and Rust (with some coaxing) can produce sufficiently tiny Wasm binaries to fit in a card. Faust and Pd (via the Heavy Compiler) may also prove viable. (And of course you can always write Wasm by hand. ☺)

But, there’s another option: start with something else entirely, and generate C. Or start with a C template, and fill in one small but crucial fragment. Following this strategy, I’ve created a few tiny tools/templates you can use to create ScoreCards without writing C yourself. Each lets you do something specific, like play with bytebeat expressions, or record a short sample, or tinker with Tidal-esque patterns, and they take care of the boilerplate for you. (You can also use the resultant C as a starting point for further hacking.)

Create Your Own Cards!

Create cards! Collaborate or compete! Preserve your code as a printed card, or just use it as a sticker.

If the pacing has worked out as intended, there should be ample time for this segment of the workshop. Feel free to ask questions, try weird things, or provoke interesting tangents.

Trading Cards

Share with the class. ☺

Discuss: fun? frustrations? fantasies?