SoundSoftware tutorial at AES 53

I’ll be co-presenting the first tutorial session at the Audio Engineering Society 53rd Conference on Semantic Audio, this weekend.

(It’s the society’s 53rd Conference, and it happens to be about semantic audio. It’s not their 53rd conference about semantic audio. In fact it’s their second: that was also the theme of the AES 42nd Conference in 2011.

What is semantic audio? Good question, glad you asked. I believe it refers to extraction or estimation of any semantic material from audio, including speech recognition and music information retrieval.)

My tutorial, for the SoundSoftware project, is about developing better and more reliable software during research work. That’s a very deep subject, so at best we’ll barely hint at a few techniques during one programming exercise:

  • making readable experimental code using the IPython Notebook, and sharing code for review with colleagues and supervisors;
  • using version control software to manage revisions and escape disaster;
  • modularising and testing any code that can be used in more than one experiment;
  • packaging, publishing, and licensing code;
  • and the motivations for doing the above.

We presented a session at the Digital Audio Effects (DAFx) conference in 2012 which covered much of this material in presentation style, and a tutorial at the International Society for Music Information Retrieval (ISMIR) in 2012 which featured a “live” example of test-driven development in research software. You can find videos and slides from those tutorials here. The theme of this one is similar, and I’ll be reusing some code from the ISMIR tutorial, but I hope we can make this one a bit more hands-on.

 

On Vodafone

Vodafone UK still haven’t fixed the problem I grumbled about in my previous post. They’ve left me unable to use my primary mobile number for 7 weeks now.

The phone works and has the right number, but it has become detached from the billing account somehow and can’t be topped up, making it effectively useless.

After talking to the friendly folk on Vodafone’s support lines got me nowhere, I switched to dealing with the people in my local Vodafone store. They’ve also been as helpful as they can possibly be. Unfortunately, all that means is that they, rather than I, have been getting nowhere by trying to talk to the technical team.

I did get to see what my account looks like on Vodafone’s systems though. It’s blank. They can search for my mobile number, and the search succeeds, but the record that comes back has nothing in it—even the phone number field is empty.

I also finally managed to get a ticketing number for the problem with their technical team. The ticket has a “72-hour SLA”, but they haven’t managed to update it in over a fortnight despite several reminders from the store staff.

Vodafone’s customer accounts website has been down for most of this week. They had network outages in several areas, last week and this.

Their systems are looking pretty fragile.

(Other networks have their problems too: EE screwed up billing for many, many pay-as-you-go customers last week. But at least they fixed that one fairly quickly.)

Edit: Vodafone fixed this on the 27th January, just after I posted this and tweeted about it. The combination of social media pressure with an actual ticket number for an open technical ticket seems to have done the trick.

 

Vodafone’s fiction of customer service

A couple of weeks ago I switched my mobile number to Vodafone, because they have better signal coverage in my workplace than T-Mobile.

That was a mistake. Vodafone screwed up my account, and their support systems have only made it worse. And it’s strange, because Vodafone work hard at making their customer service look good. Let me explain.

I use a pay-as-you-go phone. Vodafone, like most UK networks, have a system by which topping up by £10 qualifies you for “free” texts and data for the month, with calls charged separately. I don’t make many calls, so that suits me.

cmJust before my main number was ported in, I topped up £10 expecting one of these text and data packs to appear. I got a weirdly-phrased text message in reply (see picture), but no pack.

There seems to be a problemSince then I’ve dealt with at least 15 Vodafone support staff to try to get this fixed. All they’ve managed to do is confuse their accounts system so that it no longer recognises my phone number. Now when I log in on their online system, I get a different number and an error message.

I can’t top up my phone, and I can’t query its balance. I know it’s about to run out of cash, though, because Vodafone have sucked all the money out of my account for text and data charges. Their charges for texts without an active pack are pretty high.

All in all, this is crappy service from Vodafone.

What is strange is that they obviously work quite hard to supply all the trappings of good customer support:

  • I’ve never had to wait ages for anyone to pick up the phone. Every time I’ve used their online support line or phone support, I’ve been put through to a real human in a minute or two.
  • I’ve never had to convince a rude and surly operator that there was a problem here. Always, the person I’ve spoken to has sounded friendly and helpful and prepared to try to sort it out for me.
  • I’ve been able to contact them easily through social media. I tweeted about this problem and got an immediate reply with a secret-looking contact URL, suggesting that if I got in touch, they’d get it sorted right away.

But every time I communicate with someone—including that super-secret contact URL from Twitter—they react as if this is the first they’ve ever heard of it. And although they promise things, those things never happen.

None of Vodafone’s customer support people has ever been able to do anything.

I’ve been told a lot of things. I’ve been told that my problem couldn’t be escalated, because it had already been escalated and so was already in a queue somewhere. I’ve had a support adviser submit a request to fix something, then put me through to another support adviser who told me the problem was that there were now too many outstanding requests.

I’ve been put through to a “technical team” member who told me she would walk over to the technicians on the other side of the office and try to persuade one of them to look at it. And I’ve been put through to another “technical team” member who told me the technicians were in a different part of the country and they only communicated with them by email.

None of the people who told me these have achieved a single thing.

Not only are the support people powerless to fix anything, they also appear to be powerless to find out anything. Every time I’ve asked for an estimate of how long I’d have to wait, I’ve had the same reply: “usually 24 to 48 hours, sometimes as long as 72.” (We’re over 370 hours so far.) The technicians who make the fixes just don’t seem to be answerable to anyone.

Vodafone have some of the friendliest, most accessible customer service people I’ve dealt with. But they’re maintaining a fiction. They can’t provide any service.

These good people seem to be employed solely to give the customer somebody to talk fruitlessly to. They’re a kind of customer anti-service, a customer placation department.

QM Vamp Plugins in MIREX

During the past 7 years or so, we in the Centre for Digital Music have published quite a few audio analysis methods in the form of Vamp plugins: bits of software that you can download and use yourself with Sonic Visualiser, run on a set of audio recordings with Sonic Annotator, or use with your own applications.

Some of these methods were, and remain, pretty good. Some are reasonably good, simplified versions of work that was state-of-the-art at the time, but might not be any more. Some have always been less impressive. They are all available free, with source code—or with commercial licences for companies that want to incorporate them into their products.

This year we thought we should give them a trial against the current state-of-the-art in academia. Luis Figueira and I prepared a number of entries for the annual Music Information Retrieval Evaluation Exchange (or MIREX), submitting a Vamp plugin from our group in every category where we had one available.

MIREX, which is an excellent large-scale community endeavour organised by J Stephen Downie at UIUC, works by running your methods across a known test dataset of music recordings, comparing the results against “ground truth” produced in advance by humans, and publishing scores for how well each method compares.

Here’s how we got on for each evaluation task.

Audio Onset Detection

(That is, identifying the times in the music recording where each of the individual notes begin.)

We submitted two plugins here: the QM Onset Detector plugin implementing a number of (by now) standard methods, from Juan Bello and others back in 2005; and OnsetsDS, a refinement by Dan Stowell aimed at real-time use (so not directly relevant to this task). Both did modestly well. These methods have been published for a long time and become widely known, so it would be a disappointment if current work didn’t improve on them.

Audio Beat Tracking

(Tapping along with the beat.)

Here we entered the QM Tempo Tracker plugin, based on the work of Matthew Davies, and a Vamp plugin implementation of Simon Dixon‘s BeatRoot beat tracker. Both of these are now quite old methods (especially BeatRoot, although the plugin is new). The results for three datasets are here, here and here.

Both the original BeatRoot and a different version of Matthew Davies’ work were included in the MIREX evaluation back in 2006, and the ’06 dataset is one of the three used this year. So you can compare the 2006 versions here and the 2013 evaluations over here. They perform quite similarly, which is a relief. You can also see that the state of the art has moved on a bit.

Audio Tempo Estimation

(Coming up with an overall estimate in beats-per-minute of the tempo of a recording. Presumably the evaluation uses clips in which the tempo doesn’t vary.)

We entered the same QM Tempo Tracker plugin, from Matthew Davies, as used in the Beat Tracking evaluation. It doesn’t quite suit the evaluation metric, because the plugin estimates tempo changes rather than the two fixed tempo estimates (higher and lower, to allow for beat-period “octave” errors) the task calls for—but it performed pretty well. Again, a related method was evaluated on the same dataset in MIREX ’06 with quite similar results.

Audio Key Detection

(Estimating the overall key of the piece, insofar as that makes sense.)

We entered the QM Key Detector plugin for this task. This plugin, from Katy Noland back in 2007, is straightforward and fast, and is intended to detect key changes rather than the overall key.

To everyone’s surprise (including Katy’s) it scored better than any other entry, and indeed better than any entry from the past four years! The test dataset is pretty simplistic, but this is a nice result anyway.

Audio Melody Extraction

(Writing down the notes for the main melody from a recording which may have more than one instrument.)

Here we submitted my own cepstral pitch tracker plugin. This is not actually a melody extractor at all, but a monophonic pitch tracker with note estimation intended for solo singing. And it was developed as an exercise in test-driven development, rather than as a research outcome. It was not expected to do well. It actually did come out well in one dataset (solo vocal?), but it got weak results in the other three. I’m quite excited about having submitted something all-my-own-work to MIREX though.

Audio Chord Estimation

(Annotating the chord changes in a piece based on the recording.)

For this task we entered the Chordino plugin from Matthias Mauch. This plugin is much the same as the “Simple Chord Estimate” method that Matthias entered for MIREX in 2010; it got the same excellent results then and now for the dataset that was used in both years, and it also got the highest scores in the other dataset.

Structural Segmentation

(Dividing a song up into parts based on musical structure. The parts might correspond to verse, chorus, bridge, etc—though the segmenter is not required to label them, only to identify which ones have similar structural purpose.)

Two entries here. The Segmentino plugin from Matthias Mauch is fairly new, and is the only submission we made for which plugin code has not yet been released—we hope to remedy that soon. And we also entered Mark Levy‘s QM Segmenter plugin, an older and more lightweight method.

The results for different test datasets are here, here, here and here. The evaluation metrics are slightly baffling (for me anyway). I have been advised to concentrate on

  • Frame pair clustering F-measure: how well corresponding sections correspond; this measures getting matching segment types right. Segmentino does very well here, except in one dataset for some reason. The QM Segmenter is not as good, but actually not so bad either.
  • Segment boundary recovery evaluation measures: how accurately the segmenters report the precise locations of segment boundaries. Neither of our submissions does this very well, although Segmentino does well on precision at 3 seconds, meaning the segment boundaries it does report are usually fairly close to the real ones.

This is a pretty good result—I think!

Release Week ahead

I’ve built up quite a pile of almost-released new versions of software, in the lab at C4DM. It’s time to get some of it released, and I’m planning to spend the next week or two doing exactly that. This means spending a lot of time getting cross about platform builds not working right (that’s where the term “cross-platform” comes from you see).

This sort of work always calls for endless to-do lists; I thought this time I’d write up a list here, so I can talk a bit about the work we’ve been doing that I want to release. I’m hoping to get much of this done before the Digital Music Research Network meeting on the 17th December.

Vamp Plugins: MIREX edition

We entered a number of Vamp plugin implementations of audio analysis methods into the annual MIREX evaluation—I’ll write about that separately. But now we need to make properly packaged releases of them:

  • one submission, Segmentino from Matthias Mauch, has not yet been published at all, and I think we’d like to see a release of it go out;
  • one submission, the BeatRoot Vamp Plugin, has been sitting in “testing” status for a few years on our code site: MIREX gave us the impetus to finish it up, and now that it’s been through that evaluation as well, it’s surely ready to release;
  • the other submissions were from the standard QM Vamp Plugins set which has been publicly available for years, but we found a few bugs and had to make some small changes to suit the output formats needed for MIREX: we should make a release that corresponds exactly to the MIREX results as published.

Sonic Visualiser

We have a number of fixes (e.g. save window size correctly when maximised) and a couple of small new features (e.g. export audio as data) since the last release. I’d like to deal with a couple more of the outstanding bugs as well.

Sonic Visualiser “IM AF build”

A project headed by Panos Kudumakis at C4DM recently added support for loading MPEG-A Interactive Music Application Format files to Sonic Visualiser. This isn’t ready to go into mainstream builds yet, but we’d like to get a preview build released from the IM AF branch.

Klapuri Constant-Q Plugin

I’ve been working on a (causal) C++ implementation of this constant-Q transform method from Christian Schörkhuber and Anssi Klapuri.

(The constant-Q transform, like the FFT, transforms a sampled time-domain signal into a set of frequency components. But while the FFT produces output bins at constant frequency increments, the constant-Q transform produces them at constant log-frequency increments so that every bin has identical ratio of centre frequency to bandwidth—that ratio is known as Q factor, hence constant-Q. This means output bins are effectively spaced evenly in terms of musical pitch, which is handy.)

Anyway, I’d like to get this released as a library and plugin, and then use it in some other work (still pending).

Slightly longer-term…

“Tonioni”

A mysterious project involving interactive editing of pitch estimations. This is something we’re working on for at least an internal release, and it might end up being public too. We’ll see.

“Sonic Vector”

This program (you can find source here) has been around for ages. It needs brushing up and making properly available, though even a rough release would be more useful than none. It’s a comparative viewer for multiple audio recordings, most relevant where those recordings represent different versions of the same underlying music.

 

Repairs

a dishwasher, todayWaiting for a new hinge for the dishwasher door to be delivered.

We got this thing in 2002. At the end of 2009 it started cutting out and we got a repairman in to fix it. He came twice, failed to diagnose the problem, changed the motherboard, quickly gave up. We decided to replace the dishwasher, but I thought I should have a look inside it first.

Turned out the bottom of the machine, under the tub, was full of water. It had sprung a leak, and simply cut out because the leak detector had tripped. I got in touch with a nice person at a parts company who forwarded me the schematic for the dishwasher, ordered a new tiny O-ring for 69p to fix the leak, and it worked again.

Obvious lesson: replacing something like an integrated dishwasher is a pain in the arse anyway, so you might as well look inside it first and see if there’s anything visibly wrong. There’s something amazingly satisfying about getting a “free” working dishwasher by fixing it yourself, especially if, like me, you’re not a very practical chap and wouldn’t normally expect to be able to. Admittedly you do have to spend a few evenings with a dishwasher in bits in the middle of your kitchen floor.

In 2012 the door lock broke. Inspired by my earlier success, I took apart the door, found the bit I needed, bought a new one, and got it working again. Now part of the hinge has sheared through, so that once you open the door fully you can’t close it again. I’m not very good with hinges—or anything where a lot of force is involved—and not sure I know how to replace this one, but the part’s on the way, so I’ll give it a go. This machine washes perfectly well, and I like the thought of being able to avoid scrapping it.

In other pointless domestic news, I’m about to run out of this year’s marmalade and we’re still almost two months from the oranges being back in season.

The extraordinary success of git(hub)

The previous post, How I developed my git aversion, talked about things that happened in the middle of 2007.

That was nearly a year before the launch of github, which launched publicly in April 2008. I know that because I just looked it up. I’m not sure I would have believed it otherwise: git without github seems like an alien idea.

Still, it must be true that github didn’t exist then, because it would have solved the two problems that I had with git. It answers the question of where to push to, when you’re using a random collection of computers that aren’t always on; and it provides the community of people you can ask questions of when you find yourself baffled.

And that community is? All developers. Or at least, all those who ever work in the open.

The amazing success of github—and it is facilitated by the architecture of git, if not the syntax of its tools—is to produce a public use of version control software that is completely out of proportion to the number of developers who ever cared about it before.

That’s because github has so many users who are not classic “software developers”, but I suspect that it’s also because so many software developers would never otherwise use version control at all. I can’t believe that very many of github’s current users are there for the version control. They’re there for lightweight and easy code sharing. Version control is a happy accident, a side-effect of using this social site for your code.

I still don’t really use github myself, partly because I don’t really use git and partly because of a social network antipathy. (I don’t use Facebook or Google+ either.) But it’s a truly extraordinary thing that they’ve done.