Tag: rubberband

Programs for Music

Performance improvements in Rubber Band Library

Chris CannamLeave a comment

Today marks version 3.1 of the audio time-stretching and pitch-shifting library Rubber Band. This release focuses primarily on performance improvements.

In version 3.0 we introduced a totally new, higher-quality processing engine, which I’ll refer to as the R3 engine. The older one is still included, and I’ll call that R2.

Although the output of R3 typically sounds much better than R2, it uses a lot more CPU power to run. Measuring sustained throughput in frames-per-second for common fixed stretch factors, we find R2 to be typically about three times as fast as R3. Both are eminently usable in real-time on hardware from the last decade, but the headroom available for R2 can make a big difference.

It would be nice to do better, but the R3 code was already quite heavily optimised before release — it is simply a fairly CPU-intensive method. Still, as it turns out, there are a few things we can do.

Measuring performance

Sustained throughput is not the only measure. Rubber Band is often used in real-time situations where the worst-case time per processed block is what matters most.

To measure this, I set up a test case that simulates a typical sound processing callback, passing a music recording through a stretcher and emitting a fixed 512 sample frames from each processing cycle, while varying the time and pitch ratios and measuring how long each cycle takes to return. The stretcher is initialised with typical parameters for this activity (in code terms, OptionProcessRealTime | OptionPitchHighConsistency | OptionFormantPreserved) and it is primed with an initial pad before entering the cycle loop, as otherwise the first call would dominate results.

The results for R2 and R3, as of the 3.0 release, look like this:This is a graph of processing cycle count (x-axis) against time taken per 512-frame cycle (y-axis).  The y-axis is linear in time with zero at the bottom, so lower is better. No units are shown because they are totally system-dependent — this is purely a comparative visualisation, we’re only interested in the relative heights. Obviously the relative heights may also vary from system to system, so this is still quite tentative.

The test runs in four consecutive phases with different pitch and time modifications, and so the x-axis is divided into four (uneven) quadrants: raising pitch, lowering pitch, slowing down, and speeding up.

In the first quadrant, the pitch rises smoothly and then falls again, reaching a peak at two octaves up; in the second it falls smoothly and then rises again, reaching a trough at two octaves down; in the third the pitch is unchanged but the tempo slows to just under a third of the original speed and then returns to normal; and in the fourth quadrant the tempo gradually speeds up to 8x the original speed and then returns to normal.

The plots for R2 (orange) and R3 (purple) reveal significant differences in behaviour:

  • R2 is usually faster, sometimes much faster, especially for modest stretch factors.
  • R3’s long internal processing buffers and step size mean that it hops between “modes” depending on how many processing increments (1, 2, 3, 4 or occasionally 0) are required for each output block.
  • R2 has less widely-spaced distinct “modes”, because it uses smaller increments. It’s still faster because it does so much less work for each increment.
  • R2’s processing time becomes very variable, and relatively high, when speeding up the audio by a large factor (above about 3x). This may be because it continues to perform transient detection and adjust its input and output steps accordingly, and at those rates our test file contains a lot of transients. R3 is very predictable in this area by comparison.
  • Both stretchers use increasingly more CPU when pitch-shifting further upward, but not when shifting down.

The last point happens because we are using OptionPitchHighConsistency. This option ensures that the resampler used for the pitch-shift part of the operation is always engaged, so that there are no discontinuities when changing ratio (particularly to or from the 1x ratio). We’ll come back that later.

A Draft Mode for Finer Mode

The main novelty in version 3.1 is an option to deactivate R3’s multi-window processing system, dropping down to a single shorter processing window and potentially running much faster, while retaining its more advanced signal analysis and some of its output characteristics.

This is enabled using the OptionWindowShort flag when constructing a stretcher, or the --window-short argument to the command-line tool. It’s an option that already existed in R2, and conceptually it does something similar there, but the effect on performance is much greater with R3.

Here’s a plot comparing R2, R3, and the new R3 single window option (“R3short”):

With this new option we get both performance comparable to R2 and the more predictable behaviour at high tempo ratios found in R3. Splendid.

What does it sound like? Not as good as R3; it loses some percussive clarity and quite a lot of low-end stability. For some material, particularly acoustic instruments and vocals without too much bass content, it can sounds markedly better than R2. It’s not a universal substitute, but it’s really not bad given the CPU budget.

Here are some ten-second audio clips to give you an idea. Both are stretched to 140% of their original duration using R2, R3 with short window, and full R3. Neither of these is trivial to handle, though the second is far harder than the first.

Resamplers and FFTs

Rubber Band makes heavy use of audio resampler and fast Fourier transform (FFT) implementations. Originally it used external libraries for both, but in June 2021 a built-in FFT was added and in October 2021 a built-in resampler appeared as well.

These are both slower than the best external libraries, but they make Rubber Band simpler to build and integrate. And the built-in resampler is also designed to reduce clicky artifacts and maintain tempo integrity on ratio changes, at some further expense in performance, so if you do have the headroom it is worth defaulting to.

Here’s a performance comparison of the built-in resampler with libsamplerate in the “draft” short-window R3 mode described above.

Clearly libsamplerate is both faster and more predictable. It’s faster even when changing only the tempo, which doesn’t involve resampling, because of our previously-mentioned use of OptionPitchHighConsistency which keeps the resampler running at all ratios.

(Incidentally all of the other performance plots in this post were made using libsamplerate, unless otherwise specified. Its smoother performance profile makes other comparisons easier.)

I’ve mentioned OptionPitchHighConsistency a couple of times now. If we use OptionPitchHighSpeed instead, we get quite different behaviour:

The relation between the amount of pitch shift and the CPU effort is totally gone. All pitch shifts are roughly equal, and the time-stretching quadrants are faster. The tradeoff, unfortunately, is that there are now audible discontinuities every time the pitch ratio reaches or crosses 1.0.

Traditionally the alternative to libsamplerate in Rubber Band has been a resampler implementation cribbed from the Speex audio codec and provided with Rubber Band as a compile-time option. This resampler was a bit unsatisfactory for various reasons, but a much improved version of it has for a while been available in a library called speexdsp.

As of v3.1 Rubber Band now includes support for speexdsp as well, and it works well — audio quality seems good, and so is performance on my test hardware, shown here against libsamplerate:

I don’t think this is well-exercised enough to be a standard recommendation yet, but it’s promising.

The built-in FFT fares better than the resampler, but in addition to the previously-supported external libraries (FFTW, IPP, and Apple’s vDSP) this release also adds support for FFTs from SLEEF, a library which looks as if it should be competitive on platforms that have been short on good options in the past.

To summarise:

  • The R3 time-stretcher and pitch-shifter engine introduced in Rubber Band 3.0 sounds great, but is relatively CPU-intensive compared to the older R2
  • The new 3.1 release introduces a draft mode (“short-window” or single window mode) for the R3 engine, that retains some of its good qualities while running much faster and with more predictable CPU usage
  • You may be able to speed up your implementation by using an external resampler or FFT library, and the 3.1 release adds support for a couple of new ones with good performance.

See the Rubber Band Library site for more information about the library.

Thank you for your time. Perhaps we can help you make more of it.

* * *

Many thanks to Davy Wentzler for valuable feedback on the 3.1 development process.

 

Code · Programs for Music · Work

Rubber Band Library: a thrilling new release

Chris Cannam1 Comment

Rubber Band is a software library I wrote a while ago for changing audio recordings, typically of music, by altering their speed or pitch independently of one another — often known as time-stretching and pitch-shifting.

There’s a new release out, version 3.0, and I think it’s terrific and sounds great and I’m very proud of it. (Audio examples here.) But I should warn you that I find time-stretching an endlessly fascinating idea, so before I say more about the new release I’m going to digress around it for a bit.

Time-stretching

If you speed up or slow down a recording by “naive” means such as by sample rate conversion (the computational equivalent of playing an old-school tape or record at the wrong speed) its tempo and pitch change together. As it gets slower it gets lower, as it gets higher it goes faster. The result is mathematically precise and perfectly sensible but not always auditorily useful.

Time-stretching in contrast is often useful but marvellously ill-defined. I think of it as answering the question “what would this sound like if the same musicians had played it at a different tempo?” But there isn’t enough information in the signal to answer that, and people’s expectations about it are subjective and inconsistent.

Say you’re making a recording slower. If a singer sings a note with vibrato, do you expect the vibrato also to slow down? Or should it wobble at the original speed while the note gets longer? If the drummer hits a cymbal, and you slow it down, do you expect the whole sound to be fuzzily smooshed out? Or do you expect the first percussive hit to sound like the original but the decay to be extended? Or do you expect both hit and decay to be preserved exactly as the original, because if they had been playing at a different speed they would still have been hitting the same cymbal? Whatever your opinion is, would it be the same for both a recording of a real cymbal and a synthetic cymbal-like sound from a noise generator?

We have already ruled out the straightforwardly mathematical answers to these questions, because those involved changing the pitch as well. The answers appear to be essentially aesthetic.

Time-stretching software has come to a sort of consensus on these things, but it’s still largely based on what is practical rather than what an audience might expect. They slow down the vibrato, but really because it’s so much more difficult not to. They try to preserve the hit of the cymbal and extend the decay. There are many other interesting possible choices.

No doubt before too long such software will be replaced by deep learning systems that re-dream the original performance as a mere side-effect of visualising the band playing it at a different tempo or just in a different posture. But that moment does not appear to have quite arrived yet.

Back to the subject

So yes, there’s a new release of Rubber Band out. After the above, I’m sorry to admit that it doesn’t totally redefine the time-stretcher consensus, but it does do an acceptable job with that consensus and that’s good enough to delight me.

The aim with this update was to bring Rubber Band back to the same relationship to the state-of-the-art as it had when first released a shocking 15 years ago. That is: not state-of-the-art, but as close as can reasonably be expected in a nicely-licensed portable library that is fast enough for real-time use on ordinary CPUs of the day.

For the original release, that meant it was a phase vocoder (a frequency domain technique) which tries to maintain horizontal phase continuity for harmonic partials within the signal, but also detects transients (noisy instants) and resets all phases when one is found, so that the transients sound good. That’s a nice approach for signals that have a clear distinction between steady and transient sounds, like drum loops or a lot of electronic music. It’s problematic for more organic sounds or complex mixes, in which it can have trouble deciding which bit is the transient and in which its incorrect decisions are all too obvious.

That processing engine is still there in the new release. It’s good. It’s nicely fast on current hardware and has a lot of practical uses, and for reasons of compatibility it is still the default method used — so if you update the library but don’t change your code, you’ll still get the same results.

But there is also a new engine that’s just like the original one was when it appeared. That is, it still isn’t the literal state-of-the-art, but it is once again as good as can be had in a nicely-licensed portable library that is fast enough for real-time use on ordinary CPUs.

The new engine is still a phase vocoder, but it splits the signal into up to multiple frequency bands with different window lengths and shapes, and seeks limited areas of transience within the frequency spectrum rather than applying its transient phase reset across the whole signal at once.

It does use a lot more CPU power than the older one. I had aimed to get it within twice the CPU budget, but at the moment it’s more like 3 or 4 times. There may be improvements to come — as it stands this is fast enough for real-time in a responsive application on desktop or laptop, but probably not for mobile platforms, where the original Rubber Band engine has been and continues to be very suitable.

Our listening tests found that it sounded really good: it wasn’t considered the best available for every test case, but it was the best in test for some, for the rest it was close to it, and in every case it improved on the existing method. I hope you’ll agree, but time-stretching is both very subjective and very dependent on the source material and ratio. Despite our tests, it’s totally possible you might listen to the new version and hear something that offends you straight away — I hope you won’t, but people have amazingly different levels of receptivity for different audible artifacts. It might be interesting to hear about it if that happens.

If you’d like to try out the new engine (or indeed the old one) we have a little desktop application called Rubber Band Audio that you can use to load an audio file and mess with the tempo and pitch as you listen. It has a free demo version for Windows, Mac, and Linux.

Programs for Music

Rubber Band Library v1.8.2

Chris Cannam

I have finally managed to get together all the bits that go into a release of the Rubber Band library, and so have just released version 1.8.2.

The Rubber Band library is a software library for time-stretching and pitch-shifting of audio, particularly music audio. That means that it takes a recording of music and adjusts it so that it plays at a different speed or at a different pitch, and if desired, it can do that by changing the speed and pitch “live” as the music plays. This is impossible to do perfectly: essentially you are asking software to recreate what the music would have sounded like if the same musicians had played it faster, slower, or in a different key, and there just isn’t enough information in a recording to do that. It changes the sound and is absolutely not a reversible transformation. But Rubber Band does a pretty nice job. For anyone interested, I wrote a page (here) with a technical summary of how it does it.

I originally wrote this library between 2005 and 2007, with a v1.0 release at the end of 2007. My aim was to provide a useful tool for open source GPL-licensed audio applications on Linux, like Ardour or Rosegarden, with a commercial license as an afterthought. As so often happens, I seriously underestimated the work involved in getting the library from “working” (a few weeks of evening and weekend coding) to ready to use in production applications (two years).

It has now been almost six years since the last Rubber Band release, and since this one is just a bugfix release, we can say the library is pretty much finished. I would love to have the time and mental capacity for a version 2: there are many many things I would now do differently. (Sadly, the first thing is that I wouldn’t rely on my own ears for basic testing any more—in the intervening decade my hearing has deteriorated a lot and it amazes me to think that I used to accept it as somehow authoritative.)

In spite of all the things I would change, I think this latest release of version 1 is pretty good. It’s not the state-of-the-art, but it is very effective, and is in use right now in professional audio applications across the globe. I hope it can be useful to you somehow.

 

Programs for Music

Rubber Band Audio v1.9.0

Chris Cannam

Some three years after its last release (!), I present to you version 1.9.0 of Rubber Band Audio:

rbap-1.9.0-win32-full-hiresRubber Band Audio is a little audio-processing application that does time-stretching and pitch-shifting using the Rubber Band Library. It’s quite a neat tool for adjusting loops, adapting recordings to a new key, or slowing down sections for closer listening.

The previous version still works perfectly fine — I don’t really subscribe to the idea that you need to update an application constantly even though it’s basically done. But this one does add a couple of nice features. It now includes little start and end playback markers, that you can drag to set a looping range within the recording — useful when studying something closely, and a feature users have asked for. It works better with retina displays (on OS/X) and hi-dpi ones (on Windows). Audio file input and output have been updated to support a wider range of formats on Windows and to save a bit faster.

Rubber Band Audio is available direct from my little company (for Windows or Mac, with a pretty full-featured demo version available for both) or through the Mac App Store, for £7.99 or currency equivalent.