ijc

GSoC'23: Better Faust on the Web

a summer retrospective

2023-08-27

Background

This summer, I participated in Google Summer of Code, working with GRAME (and more specifically Stéphane Letz) for better Faust on the web!

Faust is a functional audio language intended to efficiently express digital signal processing algorithms. I mean “efficiently” in the sense of both programmer time (expressively, concisely) and machine time (generating highly-optimized code). I was eager to work on Faust in particular because it is relevant to my work in music technology and to my long-standing interests in programming language design and compilers. One of the really neat things about Faust is that it seeks to target many different architectures, platforms, and environments — from spitting out C code using JACK on Linux, to building an audio plugin with C++ and JUCE, to targeting an Android phone or a Bela board or a SuperCollider unit generator.

It also supports generating code that can run on the web: first through C++ to asm.js using Emscripten, then directly to asm.js using a new backend, and finally to WebAssembly. What’s more, the Faust compiler itself has been compiled to WebAssembly (via Emscripten) and thus can compile DSP code in the browser, on-the-fly, generating new Web Audio nodes on-demand! This nifty feature has enabled a number of web-based projects that make it easy to work with Faust without setting up any toolchains, compilers, or SDKs.

Project Goals

The overarching goal of my project was to build on Faust’s existing support for the web platform and enhance it. Under this general goal, I identified four subgoals in my proposal:

  1. Transition the Faust web tools (editor, IDE, and playground) from using faust2webaudio (which produces a lot of glue code and is no longer actively maintained) to the newer faustwasm. This would involve updating faustwasm to use the latest version of libfaust, making it the official successor to faust2webaudio, and building on some of the existing work done by Shihong Ren.
  2. Extend the Faust Web IDE to make it a more useful environment for audio development. Potential extensions include UI improvements to the built-in plots (zooming, log-scale axes, pop-out windows) and additional support for audio and MIDI input (e.g. the ability to record and playback live input).
  3. Enhance Faust’s online documentation so that all Faust examples can be executed and played inline, without requiring the reader to leave the documentation.
  4. Prototype an experimental platform for sharing Faust code on the web. This subgoal was inspired by Freesound, a huge collaborative database of sounds with open licenses; it would be nice to have a similarly collaborative repository for DSP code, such that users can experiment with audio code in their browser and easily import it into their project.

Execution

Tackling all four subgoals proved a bit too ambitious for a medium-sized project. Ultimately, I ended up focusing the most on subgoals 1 and 3, which both increased in scope at the expense of 2 and 4.

Infrastructure Work

Most of the summer was spent on infrastructural work (related to subgoal 1), which requires a bit of background information. Shihong Ren created faust2webaudio1 in 2019 during an internship at GRAME. In 2020, GRAME rewrote the Faust/web glue in TypeScript on the wasm2 branch with the intention of switching over and making it (faust2wasm) the officially-supported way to use Faust on the web. Then Shihong took this work and independently developed it further into faustwasm, resulting in a divergence between wasm2 and faustwasm and incomplete integration of either into the Faust ecosystem.

If this all sounds confusing (faust2webaudio vs. faust2wasm vs. faustwasm all in varying states of maintenance and adoption), that’s because it is! Thus, my initial goal in the infrastructural work was to help resolve the situation by doing whatever was needed to get faustwasm to official status and then migrating the Faust web tools (IDE, editor, playground) to faustwasm. This also involved reconciling the feature branch wasm2, which had not been touched since 2021, with the main branch (master-dev), in order to get faustwasm unstuck from an old version of libfaust. (See the branch ijc/wasm-new on my fork, PR faust#912, and the branch wasm2-june23-rebase.)

I then worked on to the main task of migrating the web projects, starting from Shihong’s various branches and draft PRs from last year: migrate-faustwasm on a Faust IDE fork, fausteditor#3, and faustplayground#7. In addition to migrating the code to use faustwasm, I found several opportunities to clean up and employ best practices, such as using package managers (i.e. npm) instead of storing copies of dependencies in the repository, using build tools to generate minified bundles, and setting up GitHub actions to build artifacts rather than manually generating them and checking them into source code version control. Other work included migrating fausteditor from _f4u$t to faust-ui, converting code to ES modules, testing, and updating documentation.

Much of this work was not included in the original scope of subgoal 1, and thus infrastructure work took considerably more time than anticipated. Nonetheless, the work needed doing, and one upshot is that the final PRs have line count changes like this:

fausteditor#4 changed lines

faustplayground#9 changed lines

faustide#72 changed lines

which I always like to see.

I also came across a couple of interesting bugs:

The migration of the Faust web tools occurred in the following PRs: faustide#72 (open, blocked on discussion with another stakeholder), fausteditor#4 (merged & deployed), and and faustplayground#9 (merged & deployed).

faust-web-component

All of this infrastructure work is important for the health of Faust on the web, but it’s mostly behind-the-scenes. None of it is directly visible to users who actually encounter Faust on the web! Thus, after working on infrastructure for most of the summer, I was excited to switch gears in late July to work on subgoal 3. Initially the idea was just to make the Faust documentation more interactive: allowing users to try out and modify examples without leaving the docs. When discussing this idea with Stéphane, we realized that it could be more generally useful if I created a Web Component to embed executable and editable Faust examples on any web page, not just the Faust documentation. Then, people discussing and demonstrating DSP on their own websites could easily embed Faust just by adding a <script> tag.

So, I designed and implemented new web components for embedding Faust on the web. Using them looks something like this:

<p><em>Here's an embedded editor!</em></p>

<faust-editor>
<!--
import("stdfaust.lib");
ctFreq = hslider("cutoffFrequency",500,50,10000,0.01);
q = hslider("q",5,1,30,0.1);
gain = hslider("gain",1,0,1,0.01);
process = no.noise : fi.resonlp(ctFreq,q,gain);
-->
</faust-editor>

<p><em>And here's a simple DSP widget!</em></p>

<faust-widget>
<!--
import("stdfaust.lib");
ctFreq = hslider("[0]cutoffFrequency",500,50,10000,0.01) : si.smoo;
q = hslider("[1]q",5,1,30,0.1) : si.smoo;
gain = hslider("[2]gain",1,0,1,0.01) : si.smoo;
t = button("[3]gate") : si.smoo;
process = no.noise : fi.resonlp(ctFreq,q,gain)*t <: dm.zita_light;
-->
</faust-widget>

<script src="faust-web-component.js"></script>

(The Faust source is embedded in an HTML comment <faust-editor> to prevent it from being treated as HTML; this is inspired by @strudel.cycles/embed.)

This result looks like this (view the page source to compare against the example code):

Here's an embedded editor!

And here's a simple DSP widget!

As you can see (assuming you have JavaScript enabled), <faust-editor> gives you a little editor (CodeMirror 6) that allows you to compile, run, and hear Faust code right on the page, along with tabs for controls, block diagrams, and plots. Perfect for observing and experimenting with some Faust code. If you want to get more serious, you can click the button that opens the current contents of the embedded editor in the more full-featured Faust IDE. <faust-widget> is a more minimal component that just gives you audio & controls (which may be more useful when the point is more the DSP itself rather than its implementation in Faust). These web components live in the faust-web-component repository.

Future Work

A few loose ends remain to be tied up regarding the infrastructure work. While the fausteditor and faustplayground PRs have been merged and deployed, the faustide PR (which is usable here in the meantime) and the wasm2-june23 branch are awaiting discussion with other stakeholders related to old export options. Beyond getting these merged, future work could include more cleanup to make the projects more consistent, such as converting fausteditor to TypeScript (to match the playground and IDE) and updating the build process of faustide to use vite (to match the editor and playground).

As for faust-web-component, we still need to transfer or fork my repository to grame-cncm, publish a package to npm, and merge my fork of faustdoc that uses faust-web-component (currently viewable here) upstream. Beyond that, features that may be nice to have in the future include support for polyphony and MIDI, audio input via file (including some stock signals), and greater configurability via HTML attributes.

As discussed previously, subgoals 2 and 4 were sidelined in favor of 1 and 3 as the summer unfolded, but I still think they’re worthwhile. Regarding subgoal 2 (extensions to the Faust IDE), some of the features that people have requested should be fairly straightforward to implement, and I may take a stab at these in the coming weeks now that my GSoC is over. Subgoal 4 was always something of a stretch goal, as it entailed building a new full-stack application for sharing and browsing DSP code. I still would like to work on this, but given the exploratory nature of the idea and its ambitions beyond Faust (it would be nice to make the platform general enough to support other languages and systems), it may be a better fit for my work as a graduate student anyway.

Conclusion

I had a good experience working on Faust during this Summer of Code, and I look forward to seeing how the work I did impacts Faust on the web. Somewhat ironically, I didn’t actually have all that much opportunity to use Faust the language in my work (beyond example code for testing & debugging), despite working on Faust all summer. But I did gain valuable familiarity with the tooling and ecosystem, and I’ve already had the chance to share the ways Faust can be used on the web with others. What’s more, I’m eager to spend more time getting my hands dirty with Faust in my personal and research projects.

Before I go, here are some brief remarks on my GSoC experience that might be useful to future participants:

Thanks to Google for sponsoring and organizing this great program, thanks to GRAME for making Faust fast, flexible, and free, and thanks also to:

  1. Not to be confused with the earlier faust2webaudio by Myles Borins, which goes from Faust ⇨ C++ ⇨ asm.js using Emscripten and predates WebAssembly. ↩