Why would I use R for music?

With Christmas around the corner, and in the spirit of spreading some joy out into the world, I decided not to write about shiny, or data workflows, or developments in base R for a change. Rather, this post is about something that brings me joy: music.

Not that R doesn’t bring me joy. Hey, I’ve ‘done data’ in other languages and in the point-and-click world. Solving the data problem with R brings a very different kind of joy….

As with most of my blogs, this one started with a daft project. I wanted to make an app that printed out musical notation, with randomly-sampled notes, that I could use as improvisation prompts when playing piano at my local experimental music open mic. A problem we’ve all faced.

This felt like something I could build in shiny, though it proved a little more difficult than I expected. Solving the problem completely might need a second blog post, a htmlwidget, and a bit of Javascript knowledge.

Here, we’ll talk about music in R, what packages are available, how to represent musical notation, and what people are actually doing with music data in R. We’ll maybe round off with a public domain Christmas carol or two, for good measure.

Computer World: Musical scores, sequencers and beeping chips

Home computers have been making music since the 70’s. At a time when dedicated sound chips belonged to a distant future, electromagnetic interference from bit switches in an Altair was hacked to play “Fool on the Hill” through a neighbouring radio (outlined in “Bits and Pieces”, KB McAlpine, p154). Even earlier than this, people had made music on research computers at universities (“Bits and Pieces”, p12). The development, hand-in-hand, of electronic music, computer sound chips, music software and video games is a fascinating story. But that’ll have to wait for another day.

Fundamental to those developments, was a simple question: how do you represent a piece of music inside a computer? Converting this representation into sound is a separate issue, because there are things you can do with music beyond listening to it. You can compare different aspects of a collection of songs (keys, harmonies, lyrics etc), you can (attempt to) get a computer to compose new music, or you can rearrange a given piece or print out sheet music for musicians to play from. For example here Kris Shaffer analyses chords in 100 rock songs using R, and here is a presentation analysing chords, lyrics and spotify data by Bruna Wundervald and Julio Trecenti using packages from the r-music organization.

Nowadays, most of the music stored on your computer will be stored as recordings, such as mp3s. This wasn’t originally the case, early games encoded music directly using note pitches and durations - much like you find in sheet music. A modern view of this representation is provided by the Humdrum format. The following contains the chorus melody for “Jingle Bells”.

**kern
=1
4e
4e
2e
=2
4e
4e
2e
=3
4e
4g
4c
4d
=4
1e
=5
*-

We can view that melody in an online tool, and we get traditional music notation back out:

The pairs “4g”, “2e”, “1e”, and so on, represent the duration (4, 2, 1 in increasing length; 4 being a crotchet or ‘quarter-note’) and pitch (e, g). The “=1” lines separate bars, and the "**kern" and "*-" delimit the whole sequence. To represent multiple notes playing at the same time, you can use additional vertical tracks (spines) to represent the additional notes. The syntax can get pretty complicated but so does sheet music….

Solid State “S”urvivor: Sounds in R - {beepr}, {audio}, {tuneR}

When it evolved from S in 1993, creating music might not have been on the horizon for R.

R wasn’t really on my horizon at the time either, I was at school, and spent quite a bit of spare time writing music in OctaMED on a Commodore Amiga - again, involving multiple vertical tracks of pitches and durations.

Can R even make a sound? Aside from the groans that Error in mean[1:3]: object of type 'closure' is not subsettable can evoke?

Yes it can. There are a few packages available for producing sound in R. My favourite is {beepr}. If you’ve got a long-running script burning away on your computer, what better way to celebrate its completion than with a fanfare, or with the Super Mario Bros “Level Complete” tune:

source("my-beautiful-script.R")
beepr::beep(sound = "mario")

Doodly-doodly-doo!

You could similarly have a cymbal crash when you’ve finally loaded that big dataset if you install {drumr}:

cars = {Sys.sleep(5); mtcars}
drumr::beat("crash")

We aren’t going to go any further into emitting sounds or analysing music from R here. But there are a few packages like {audio} and {tuneR} that can be used for this purpose.

Replicas: Representing music and making sheet music in R

{tabr} is a CRAN package providing the ability to handle musical scores as data. It also provides the ability to render sheet music from this data, by integrating with a system dependency ‘LilyPond’. Once you have installed both LilyPond and {tabr}, you can construct sheet music from R. The syntax for encoding melodies in tabr is similar but different from that used in Humdrum, above.

library("tabr")
melody = as_music("e4 e e2 e4 e e2 e4 g c d e1")
plot_music(melody)

So again, we encode notes with both pitch and duration, though now the duration comes after the pitch (‘e4’ is a crotchet E). We don’t need to specify the duration of a note, if it is the same as the preceding note. {tabr} has added a time-signature and tempo using some default values. This particular tempo might not help Santa get his sleigh off the ground though - that’s about half the speed that Bing Crosby recorded it. The notes are written out in a lower octave than in the Humdrum example, too.

We can fix all that though. While we’re at it let’s make that final run a bit sassier:

melody <- as_music(
  "e'4 e' e'2 e'4 e' e'2 e'4 g' c'~ c'8 d'8 e'1",
  tempo = "2 = 120"
)
plot_music(melody)

You can find out the syntax used in the music strings using the tabrSyntax data-frame.

tabrSyntax

##                 description    syntax    example
## 1                note/pitch a b ... g          a
## 2                     sharp         #         a#
## 3                      flat         _         a_
## 4  drop or raise one octave    , or '    a, a a'
## 5             octave number   0 1 ...   a2 a3 a4
## 6                tied notes         ~       a~ a
## 7             note duration       2^n 1 2 4 8 16
## 8               dotted note         .     2. 2..
## 9                     slide         -         2-
## 10                     bend         ^         2^
## 11          muted/dead note         x         2x
## 12     slur/hammer/pull off        ()      2( 2)
## 13                     rest         r          r
## 14              silent rest         s          s
## 15       expansion operator         * ceg*8, 1*4

For guitarists, there’s also the ability to plot out guitar tab (hence the name; strangely the notes have been transposed by an octave):

plot_music_guitar(melody, header=list(title = "Jingle Bells"))

It should be noted that {tabr} is not as flexible as LilyPond when it comes to creating musical scores, and indeed, the author recommends that “If you are only creating sheet music on a case by case basis, write your own LilyPond files manually”. The truth is, I got a lot of errors while experimenting with {tabr}, but it was still a fun experiment.

Blue Lines: Adding scores to an app

I originally wanted to randomly-generate music phrases that I could interpret myself. And {tabr} looked like a good fit for just printing out notes to an app.

We sample from two octaves of the ‘white notes’ of the C major scale:

# C major notes from G below middle-C
notes <- c("g", letters[1:7], letters[1:7]) |>
  paste(
    c(rep("", 3), rep("'", 7), rep("''", 5)),
    sep = ""
  )

To get a valid musical string, we can do the following:

sample_notes = function(x, n) {
  sample(x, size = n, replace = TRUE) |> paste("4", sep = "")
}

rand_melody = sample_notes(notes, 8)
rand_melody

## [1] "b4"   "c''4" "f'4"  "g''4" "e'4"  "b4"   "a'4"  "b4"

rand_melody |> as_music() |> plot_music()

As a way of sampling melodies this is as simple as it gets. And it works in an app quite nicely too:

library("shiny")
library("tabr")

ui = fluidPage(
  plotOutput("music")
)

server = function(input, output, session) {
  melody = reactive({
    invalidateLater(10000) # sample a new melody every 10s
    sample_notes(notes, 8)
  })
  output$music = renderPlot({
    melody() |> as_music() |> plot_music()
  })
}

There was a couple of issues with the app (and I made it a bit more complicated before I realised this).

The main issue was that, if I deployed to shinyapps.io, LilyPond wasn’t available - so to use the app for real, I would have had to take a laptop, rather than just my phone, to the open-mic with me - and I’m a rather heavy-handed pianist so something expensive could well have broken….

The other issue was that rendering the music was a little slow and updating the score was glitchy - a png is created on the server side and transferred to the browser every 10 seconds. There are JavaScript libraries that can render musical scores, for example, the Humdrum library has a JavaScript plugin. Using such a library would mean that our shiny app could transfer some Humdrum notation to the browser, which might speed up rendering. The website for the Humdrum plugin includes an example of how to use it in a Shiny app - however, extending these examples to dynamically update after a new melody was sampled didn’t work for me. So, my next project is to work out how to write an htmlwidget package for Humdrum….

Endtroducing: Why didn’t the app deploy?

When you deploy an app to shinyapps.io, any packages it depends upon are installed on the shinyapps.io server. This would typically include {shiny}, {bslib} and a few other app-related things, but could include packages for any number of other things: numerics, data processing, visualisation. Many of these packages will depend on system libraries - the {quarto} package requires the Quarto command-line tool to be installed on a machine, for example. These system dependencies are encoded in the SystemRequirements section of the R package DESCRIPTION file, the same content you see on CRAN when looking at a single package. For {quarto} , for example, the SystemRequirements state “Quarto command line tool (https://github.com/quarto-dev/quarto-cli).”.

Now, the SystemRequirements is a freely-structured text field. As a package author you can write whatever you want in there, and it is up to the users of your package to ensure that their system has the SystemRequirements available. This makes sense because on different operating systems, the system libraries have different names. But it’s a little problematic when attempting to deploy to a server - if you need an R package that has a system-requirement that isn’t already available on that server, and you can’t log in to the server to install system libraries, how do you ensure it gets installed?

The {pak} package helps here. This provides an enhanced way to install R packages. When {pak} installs packages, it uses the free-text SystemRequirements field to determine the OS-specific system libraries that an R package needs. It does this by making use of rules specified in the r-hub/r-system-requirements repository. This is outlined in a blog post by Hugo Gruson.

Ultimately what happened, is that while {pak} was installing the R packages for my {tabr}-dependent app to shinyapps.io it saw that there was a dependency on LilyPond, but because there is no LilyPond rule at r-hub/r-system-requirements, it couldn’t work out what libraries or system tools it needed to install. So {tabr} installed, but the ‘lilypond’ library that it depends upon didn’t.