Functional programming and the joy of learning something again

Twenty years ago, as a maths-and-computing undergraduate at the university of Bath, I was introduced to functional programming using the ML language by the excellent Julian Padget. We undergrads were set the traditional assignment of writing a sed-like text processor in ML, and found it first baffling and then, if we were lucky, rather exciting. I got all enthusiastic about functional languages and, like any proper enthusiast, spent a delightful pointless while trying to design a toy language of my own.

Then I went off and got a job as a C++ programmer.

C++ is a practical, useful language, but it’s also complex, verbose, and baroque, with many different schools of practice. I’ve done a fair bit of work in other languages as well (equally boring ones, like Java) but effectively, I’ve spent much of the past two decades simply continuing to learn C++. That’s a bit crazy.

So when messing about with Android recently, I decided I wanted to try to get some of that old sense of joy back. I went looking for a language with the following properties:

  • It should be primarily functional rather than object-oriented
  • It should be strongly-typed, ideally with Hindley-Milner typing (the most exciting thing about ML, for the undergraduate me)
  • It should have a compiler to JVM bytecode, so I could use it directly in Android apps, and should be able to use Java library code
  • It should have a REPL, an interactive evaluation prompt for productive messing around
  • It should be nice to read—it should be obviously a language I wanted to learn, and I was going to be happily guided by personal whim
  • It must be simple enough for the old, stupid me to have a chance of getting to grips with it
  • And, while I wasn’t going to care very much how mainstream it was, it did need to be reasonably complete and reliable.

There are lots of languages out there for the JVM, including quite a few functional ones. Scala and Clojure are the best-known.

Scala (here in Wikipedia) is a multi-paradigm language that, for me, has shades of C++ in that it feels like it’s designed to improve on all sorts of things in Java rather than be something simple of its own. It also looks object-oriented first and functional second; doesn’t prioritise interactive evaluation; and although it has a sophisticated type system, it doesn’t do inference on function parameter types. All in all, it just seemed a bit chunky to me.

Clojure (here in Wikipedia) looks more fun. It has a very vibrant community and seems well-loved. It’s basically a Lisp for the JVM, and I’ve written Lisp before. That’s definitely interactive and functional. But I wasn’t really setting out to find Lisp again.

Yeti

Having sifted through a few other possibilities, the one that really seemed to fit the bill was Yeti.

Yeti is a functional language with Hindley-Milner type inference, for the JVM, with a relatively simple syntax and interoperability with Java, that has an interactive REPL up front. (See the snappy tutorial.) It seems to be basically the work of one programmer, but a programmer with taste.

The syntax of Yeti looks a bit like the way I remember ML—although on reviewing ML, it turned out not to be as similar as I’d thought. Functions are defined and applied with just about the simplest possible syntax, and the language deduces the types of all values except Java objects. The lack of commas in function application syntax makes it obvious how to do partial application, a straightforward way to obtain closures (functions with context).

Here’s a trivial function, an application of it, and a partial application. The lines starting > are my typing, and the others are type deductions returned by the evaluation environment. Comments start //, as in Java.

> f a b = a + b   // f is a function taking two arguments
f is number -> number -> number = <code$f>
> f 2 3           // apply f to two numbers
5 is number
> f 2             // apply f to one number, returning a new function
<yeti.lang.Fun2_> is number -> number
> g = f 2         // assign that function to g
g is number -> number = <yeti.lang.Fun2_>
> g 3             // apply g to the second number
5 is number

So far, so academic toy language. But the more I apply Yeti to practical problems, the more I find it does work as a practical language.

What is challenging, of course, is that every language or language family has its set of idioms for handling everyday problems, and on the whole I simply don’t know those idioms yet in a functional language. This is the first time I’ve really tried to do anything serious with one. I know the language, roughly, but I don’t really speak the language. I’m still leafing through the phrasebook. My hovercraft is still full of eels.

With most of my incredibly stupid questions on the Yeti mailing list—which get very patient responses, but I really do need to cut back on the late-night stupidity and start reviewing my code in the morning instead—the answer turns out to be, “it’s simpler than I thought”. And I’m really enjoying it.

Why type inference?

A footnote. Why did I want a language with type inference?

Well, I’m lazy of course, and one of the most obvious sources of tedium in C++ and Java is having to type everything out over and over again.

And I’m a bit nostalgic about those undergrad days, no doubt.

But also, I’m increasingly mistrustful of my own work. In languages such as Python and Objective-C the concept of duck typing is widely used. This essentially means employing objects on the basis of their supported methods rather than their nominal class (“if it walks like a duck…”). This relaxed approach reaches a bit of a pinnacle in Ruby on Rails, which I’ve been working with a bit recently—and I find the magic and the assumptions go a bit far for me. I like to have some of the reassurance of type checking.

So, type inference gives you—in theory—the best of both worlds. You get to write your code using duck-typing principles, and the compiler proof-reads for you and checks that your types really do work out.

That’s the theory. Does it scale? Not sure. And if it was so great, wouldn’t it have become more mainstream during the past 30 years? Some moderately widely-used languages, like Haskell, use it, but they’re still only moderately widely-used. So, we’ll see.

There are some obvious immediate disadvantages to type inference. Long error messages, for a start.

And as a C++ guy I somewhat miss function overloading and even (sometimes) operator overloading. A function argument can take more than one type, of course—that’s the whole point—but only if the types can be unified in terms of their properties; you can’t just reuse a function name for a second function that has a similar effect but works on unrelated types.

Most of the situations in which I want function overloading can be handled instead using variant types or module namespaces, both of which work well in Yeti, but sometimes it seems to come down to inventing slightly more awkward function names than I’d really like.

“Various nifty functions”

Further to the code-literate judge in Oracle v Google, via Groklaw we now have his ruling that the Java APIs are not copyrightable.

It’s an exceptionally clear piece of work and a good introduction to the subject. I certainly couldn’t have written a better technical summary, although I’m sure there are bits that a non-programmer would still struggle with—for example, the judge uses the term “subroutine” without explanation.

I like the jaunty language:

After Java’s introduction in 1996, Sun […] wrote hundreds more programs to carry out various nifty functions

And he is certainly decisive. The section describing the code at issue (rangeCheck) is introduced thus:

Oracle has made much of nine lines of code that crept into both Android and Java. This circumstance is so innocuous and overblown by Oracle that the actual facts, as found herein by the judge, will be set forth below for the benefit of the court of appeals.

And in the closing remark,

[It] is important to step back and take in the breadth of Oracle’s claim. Of the 166 Java packages, 129 were not violated in any way. Of the 37 accused, 97 percent of the Android lines were new from Google and the remaining three percent were freely replicable under the merger and names doctrines. Oracle must resort, therefore, to claiming that it owns, by copyright, the exclusive right to any and all possible implementations of the taxonomy-like command structure for the 166 packages and/or any subpart thereof — even though it copyrighted only one implementation. To accept Oracle’s claim would be to allow anyone to copyright one version of code to carry out a system of commands and thereby bar all others from writing their own different versions to carry out all or part of the same commands. No holding has ever endorsed such a sweeping proposition.

As an aside, nice to see our old friend Sega v Accolade cited again. I haven’t read all that many US legal opinions on software copyright, but I think pretty much all the ones I have seen have referred to Sega v Accolade.

You can read the whole thing on Groklaw.

Wrapping a C++ library with JNI, part 4

In this series…

  • Introduction, outlining the general steps from starting with a C++ library to being able to build and run simple tests on some JNI wrappers;
  • Part 1, in which I design some simple Java classes and generate the stub wrapper code;
  • Part 2, in which I add just enough of the implementation to be able to do a test build;
  • Part 3, discussing object lifecycles in C++ and Java;
  • Part 4 (this post), the final episode covering a few remaining points of interest.

A More Complex Method

The most complex function in our API is the process method in the Plugin class. In C++, this is

typedef std::vector<Feature> FeatureList;
typedef std::map<int, FeatureList> FeatureSet;

virtual FeatureSet process(const float *const *inputBuffers,
                           RealTime timestamp) = 0;

That is, the process method of a Plugin object takes a two-dimensional array of floats and a RealTime object as arguments, and returns a map from an integer to a FeatureList, which is a sequence of Feature objects stored in a vector. That’s fairly complicated.

Java lacks typedef or any very satisfactory alternative. So, we render this as

public native Map<Integer, ArrayList<Feature>>
    process(float[][] inputBuffers, RealTime timestamp);

where Feature and RealTime are classes we must provide separately. (It’s good form to declare the return type using an interface such as Map in cases where the caller doesn’t need to care which specific container implementation is used. In this case our concrete return type should probably be a TreeMap.)

I’m not going to explain every detail of the implementation of this in the JNI wrapper, but I want to illustrate a couple of aspects:

Constructing Java objects from JNI code

In this particular case, the objects we want to return take rather a lot of work to construct. However, we can illustrate a simple example. At the innermost level, all of our returned objects have type Feature, a Java class we define which has a constructor that needs no arguments.

To construct one of these, first we need to look up its class:

jclass featClass = env->FindClass("org/vamp_plugins/Feature");

Then we find the method in the class that corresponds to the constructor. The GetMethodID function is used for looking up all non-static methods, with the method name supplied as the first string argument. For a constructor, the method name is <init>. The final argument gives the signature of the method to look up; in this case ()V means a method taking no arguments with a void return type.

jmethodID ctor = env->GetMethodID(featClass, "<init>", "()V");

Then, to construct the object we simply call the constructor:

jobject feature = env->NewObject(featClass, ctor);

Handling Generics through JNI

This is dead easy. Generic types are there only for the purpose of compiler type-checking: they don’t exist in the virtual machine or in the .class file.

In terms of Java objects, including from the perspective of any JNI code, our ArrayList<Feature> is simply ArrayList.

So, to construct the TreeMap<Integer, ArrayList<Feature>> we intend to return from our function, we only need to do this:

jclass treeMapClass = env->FindClass("java/util/TreeMap");
jmethodID treeMapCtor = env->GetMethodID(treeMapClass, "<init>", "()V");
jobject map = env->NewObject(treeMapClass, treeMapCtor);

Of course we then need to make sure the objects we put in it are of the right type, as the Java compiler is no longer there to help us with type-checking.

Similarly, if a generic container gets passed to a native function, we can (and must) just ignore the type specialisation when unpacking it.

Getting Data from Multi-Dimensional Arrays

My process function takes a two-dimensional array of floats as one of its arguments. (This actually represents multi-channel audio sample data.)

Multi-dimensional arrays in Java are easier to deal with reliably than in C++. An array is an object, so a two-dimensional array is an array of array objects. It appears in the JNI implementation as a jobjectArray:

jobject
Java_org_vamp_1plugins_Plugin_process
    (JNIEnv *env, jobject obj,
     jobjectArray inputBuffers, jobject timestamp);

JNI provides simple accessor methods for getting at array elements: to pull out an object from an array of objects, we use GetObjectArrayElement. That will enable us to get hold of the second dimension of our arrays.

Then, to access a sequence of non-object type elements such as our float values all at once, we need to use a symmetrical pair of calls: one to lock the elements in place so the garbage collector can’t get at them, and the other to release them again. To wit, GetFloatArrayElements and ReleaseFloatArrayElements.

Putting these together to retrieve our two-dimensional float array in C++, we have as the body of our process function:

int channels = env->GetArrayLength(data);
float **input = new float *[channels];

for (int c = 0; c < channels; ++c) {
    jfloatArray cdata =
        (jfloatArray)env->GetObjectArrayElement(data, c);
    input[c] = env->GetFloatArrayElements(cdata, 0);
}

Then we do something with the C array that is now stored in input, hang on to the output for a moment, and tidy up:

for (int c = 0; c < channels; ++c) {
    jfloatArray cdata =
        (jfloatArray)env->GetObjectArrayElement(data, c);
    env->ReleaseFloatArrayElements(cdata, input[c], 0);
}

delete[] input;

and we’re done. It remains only to construct and return our rather complex return value based on whatever we calculated with the input array earlier.

That’s all

That’s all for this series—thanks for reading, and I hope it’s been useful.

Wrapping a C++ library with JNI, part 3

In this series…

  • Introduction, outlining the general steps from starting with a C++ library to being able to build and run simple tests on some JNI wrappers;
  • Part 1, in which I design some simple Java classes and generate the stub wrapper code;
  • Part 2, in which I add just enough of the implementation to be able to do a test build;
  • Part 3 (this post), discussing object lifecycles in C++ and Java;
  • Part 4, the final episode covering a few remaining points of interest.

My previous post on this topic ended with a very simple test program built and run successfully, wrapping a small subset of the C++ library I was referring to.

That’s all I want to go through step-by-step, but there are still a couple more things to explain. The first one is…

Disposing of C++ objects

Java objects are garbage-collected: you don’t have to delete them when you’ve finished with them.

C++ objects are not. Unless they’re locally scoped objects allocated on the stack (i.e. created without a call to new), you have to delete them explicitly when you’ve finished with them, or else you’re creating a resource leak.

The wrapper code I’ve been describing uses Java objects that “own” their corresponding C++ objects. My Plugin object for example contains an opaque nativeObject handle which, on the C++ side, gets interpreted as a pointer to the C++ object that provides the plugin implementation. This object belongs to the Java Plugin wrapper, and it should have a corresponding lifetime which we need to manage. Nothing else is going to delete it if we forget to.

This means we need the Java object to delete its C++ object when it is itself deleted. But a Java object isn’t deleted explicitly, and it has no C++-style destructor function we could make this happen from.

We have two options:

  1. Add a method to the Java class that deletes the C++ object, and insist that users of our library remember to call it;
  2. Delete the C++ object from the Java class’s finalize method, which is called by the JVM when the Java object is garbage collected.

The second option, using finalize, is attractive because it reduces the burden on our users. Deleting the object would be automatic. If we take the first option, any method we add to delete our object will be one that isn’t required for most Java classes, and that also doesn’t exist in the same form in the C++ library we’re wrapping—so developers coming from either Java or C++ will quite likely overlook it.

But there are several problems with using finalize for this.

One problem is that, because the JVM only controls memory and resources within the Java heap, it can’t take into account any resources allocated in the native layer when determining which Java objects it needs to garbage-collect: our Java objects will seem smaller than they effectively are. So it might not finalize our objects quickly enough to prevent memory pressure.

A second problem is that we can’t guarantee which thread the finalize method will be called from. It’s quite important that our C++ object is deleted from the same thread that allocated it.

Finally, it’s possible that the C++ library we are wrapping might expect objects to be deleted in some particular order relative to one another, and we can’t enforce that if we leave the garbage collector to do it. When in C++, we should do as C++ programmers do.

And that means we need to get the user of our library to tell us when to delete objects: we must at least take our option 1. That is, we add a method called dispose to the Java class: when it’s called, it deletes the underlying native object. But, of course, our user—the programmer writing to our library—must remember to call it.

(Why “dispose”? Well, it’s a good name, and it’s the name used by some other libraries that rely on native code, such as the SWT widget toolkit.)

In our Plugin class, the dispose call would be simply

    public native void dispose();

and the implementation in Plugin.cpp, using the native handle helper method referred to in my earlier post, would be along the lines of

void
Java_org_vamp_1plugins_Plugin_dispose(JNIEnv *env, jobject obj)
{
    Plugin *p = getHandle(env, obj);
    setHandle(env, obj, 0);
    delete p;
}

In principle we could also make the finalize method call dispose in case the user forgets. But that’s probably unwise, as the library’s behaviour could become quite unpredictable in subtle ways if the pattern and order of deallocations depended on whether the user had remembered to dispose an object or had left it to the garbage collector.

Another possibility might be to test within finalize whether the object has been disposed, and print a warning to System.err if it hadn’t.

Coming next

In my next (and final) post on this subject, I’ll give an illustration of a function with rather more complex argument and return types than the ones we’ve seen so far.

Wrapping a C++ library with JNI, part 2

In this series…

  • Introduction, outlining the general steps from starting with a C++ library to being able to build and run simple tests on some JNI wrappers;
  • Part 1, in which I design some simple Java classes and generate the stub wrapper code;
  • Part 2 (this post), in which I add just enough of the implementation to be able to do a test build;
  • Part 3, discussing object lifecycles in C++ and Java;
  • Part 4, the final episode covering a few remaining points of interest.

By the end of my previous post, I had cooked up some simple Java classes corresponding to those parts of the C++ library API that I wanted to wrap up first.

(That means as little of the API as is necessary to get a simple build and test going: I can add to it after that.)

Now I need to fill in the JNI function implementations, code up a little test program, build it, and see what ensues.

4. Write JNI function bodies

This is the point at which we need to start referring quite seriously to the JNI specification and Programmer’s Guide.

Our task is to write function bodies for the functions whose declarations have been generated by javah. The first thing we’ll need is a way to get and set the native handles we are storing in each Java class, so that we can keep track of the native object that each Java object corresponds to.

This can go in a common header, handle.h:

#ifndef _HANDLE_H_INCLUDED_
#define _HANDLE_H_INCLUDED_

jfieldID getHandleField(JNIEnv *env, jobject obj)
{
    jclass c = env->GetObjectClass(obj);
    // J is the type signature for long:
    return env->GetFieldID(c, "nativeHandle", "J");
}

template <typename T>
T *getHandle(JNIEnv *env, jobject obj)
{
    jlong handle = env->GetLongField(obj, getHandleField(env, obj));
    return reinterpret_cast<T *>(handle);
}

template <typename T>
void setHandle(JNIEnv *env, jobject obj, T *t)
{
    jlong handle = reinterpret_cast<jlong>(t);
    env->SetLongField(obj, getHandleField(env, obj), handle);
}

#endif

For PluginLoader.cpp, we start with the appropriate headers and namespaces:

#include "org_vamp_plugins_PluginLoader.h"
#include <vamp-hostsdk/PluginLoader.h>
#include "handle.h"

using Vamp::Plugin;
using Vamp::HostExt::PluginLoader;

Now we need two function implementations. We have the declarations for these already: they were generated by javah in the previous post.

One, the initialise function called from the Java class constructor, simply creates a native object and stows it in the Java object.

void
Java_org_vamp_1plugins_PluginLoader_initialise(JNIEnv *env, jobject obj)
{
    PluginLoader *inst = PluginLoader::getInstance();
    setHandle(env, obj, inst);
}

The other does the real work. Here we grab the previously stowed native object from the handle in the Java object, convert the plugin key argument from its opaque jstring type to a C string, call out to the native loadPlugin, and return the result.

jlong
Java_org_vamp_1plugins_PluginLoader_loadPluginNative(JNIEnv *env, jobject obj,
                             jstring key, jfloat rate)
{
    PluginLoader *inst = getHandle<PluginLoader>(env, obj);
    const char *kstr = env->GetStringUTFChars(key, 0);
    Plugin *p = inst->loadPlugin(kstr, rate);
    env->ReleaseStringUTFChars(key, kstr);
    return (jlong)p;
}

In contrast, Plugin.cpp is largely a cut-and-paste job:

#include "org_vamp_plugins_Plugin.h"
#include <vamp-hostsdk/Plugin.h>
#include "handle.h"

using Vamp::Plugin;
using std::string;

jstring
Java_org_vamp_1plugins_Plugin_getIdentifier(JNIEnv *env, jobject obj)
{
    Plugin *p = getHandle<Plugin>(env, obj);
    return env->NewStringUTF(p->getIdentifier().c_str());
}

jstring
Java_org_vamp_1plugins_Plugin_getName(JNIEnv *env, jobject obj)
{
    Plugin *p = getHandle<Plugin>(env, obj);
    return env->NewStringUTF(p->getName().c_str());
}

jstring
Java_org_vamp_1plugins_Plugin_getDescription(JNIEnv *env, jobject obj)
{
    Plugin *p = getHandle<Plugin>(env, obj);
    return env->NewStringUTF(p->getDescription().c_str());
}

jint
Java_org_vamp_1plugins_Plugin_getPluginVersion(JNIEnv *env, jobject obj)
{
    Plugin *p = getHandle<Plugin>(env, obj);
    return p->getPluginVersion();
}

5. Make a test program

This should be a Java program that refers only to the Java classes I’ve put together. It just needs to load a plugin using the PluginLoader class, and call some test methods on the resulting plugin.

package org.vamp_plugins;

public class test
{
    public static void main(String[] args) {

        // This is the name of a Vamp plugin we know we have installed
        String key = "vamp-example-plugins:percussiononsets";

        PluginLoader loader = PluginLoader.getInstance();

        try {
            Plugin p = loader.loadPlugin(key, 44100);
            System.out.println("identifier: " + p.getIdentifier());
            System.out.println("name: " + p.getName());
            System.out.println("description: " + p.getDescription());
            System.out.println("version: " + p.getPluginVersion());
        } catch (PluginLoader.LoadFailedException e) {
            System.out.println("Plugin load failed");
        }
    }
}

That should be enough.

6. Build and test

So, we need to compile and link our C++ wrapper implementations into a dynamic library (DLL, or shared object); and we need to build our Java test program.

How you would do this depends on the platform and development environment you’re using. For the purposes of this post, I’m using Linux and command-line tools.

So, for the C++ parts I’ll create a small Makefile:

LIBRARY := libvamp-jni.so
OBJFILES := src/PluginLoader.o src/Plugin.o
INCLUDES := -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/linux
CXXFLAGS := $(INCLUDES)

$(LIBRARY): $(OBJFILES)
    $(CXX) -shared -o $@ $^ -lvamp-hostsdk

Thus:

$ make
g++ -I/usr/lib/jvm/java-6-openjdk/include -I/usr/lib/jvm/java-6-openjdk/include/linux   -c -o src/PluginLoader.o src/PluginLoader.cpp
g++ -I/usr/lib/jvm/java-6-openjdk/include -I/usr/lib/jvm/java-6-openjdk/include/linux   -c -o src/Plugin.o src/Plugin.cpp
g++ -shared -o libvamp-jni.so src/PluginLoader.o src/Plugin.o -lvamp-hostsdk
$

Now build the Java classes and test program, and run it:

$ javac org/vamp_plugins/*.java
$ java -classpath . org.vamp_plugins.test
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.vamp_plugins.PluginLoader.initialise()V
    at org.vamp_plugins.PluginLoader.initialise(Native Method)
    at org.vamp_plugins.PluginLoader.<init>(PluginLoader.java:23)
    at org.vamp_plugins.PluginLoader.getInstance(PluginLoader.java:10)
    at org.vamp_plugins.test.main(test.java:11)
$

Uh-oh. I forgot to load the JNI wrapper DLL. To make this happen, I need to add a static section to the end of the PluginLoader class:

    static {
        System.loadLibrary("vamp-jni");
    }

The JVM class loader is quite sensitive to the exact name and location of the library file.

If I supply the name vamp-jni to loadLibrary as above, then on my Linux machine the JVM will expect the library to be called libvamp-jni.so (and not, say, vamp-jni.so).

Other rules apply on other platforms.

Rebuilding and trying again:

$ javac org/vamp_plugins/*.java
$ java -classpath . org.vamp_plugins.test
identifier: percussiononsets
name: Simple Percussion Onset Detector
description: Detect percussive note onsets by identifying broadband energy rises
version: 2
$

Hurrah!

Next

  • In my next post on this subject, I talk about managing the disposal of our underlying C++ objects to avoid memory leaks.

Wrapping a C++ library with JNI, part 1

In this series…

  • Introduction, outlining the general steps from starting with a C++ library to being able to build and run simple tests on some JNI wrappers;
  • Part 1 (this post), in which I design some simple Java classes and generate the stub wrapper code;
  • Part 2, in which I add just enough of the implementation to be able to do a test build;
  • Part 3, discussing object lifecycles in C++ and Java;
  • Part 4, the final episode covering a few remaining points of interest.

In my introductory post I outlined the steps I’d need to take in order to get a JNI wrapper working for a simple subset of a C++ library. Time to get started.

1. Identify the C++ classes and methods

The C++ library I’m wrapping is documented here.

I’ll initially need to wrap parts of two classes: PluginLoader, in the Vamp::HostExt namespace, and Plugin, in the Vamp namespace.

PluginLoader is a singleton class which loads and returns an instance of a Vamp plugin on request, given the name (or “key”) of the plugin. It’s essentially an object factory.

Then Plugin has both methods that return information about the specific plugin that has been loaded, and methods that are called with blocks of audio data in order to do the actual analysis.

In my first draft, I want to make a singleton Java class to correspond to PluginLoader and implement in it the one method that actually loads the plugin; and I want to make a Plugin class with a couple of the methods that return simple metadata about the plugin.

The methods that do real work involve some complicated return types, so I’ll leave those until later.

So the basic library API I’m starting with looks like this, in the existing C++ API:

class PluginLoader
{
public:
    static PluginLoader *getInstance();
    Plugin *loadPlugin(PluginKey key,
                       float inputSampleRate,
                       int adapterFlags = 0);
    // ... other methods for later
};

class PluginBase
{
public:
    virtual std::string getIdentifier() const = 0;
    virtual std::string getName() const = 0;
    virtual std::string getDescription() const = 0;
    virtual int getPluginVersion() const = 0;
    // ... other methods for later
};

That’s plenty to be going on with.

Note that the PluginBase with its pure virtual methods is somewhat like a C++ equivalent of a Java interface; the plugins themselves have a subclass type, namely Plugin.

2. Design the Java classes

There may be several ways to render a native library into Java classes, even where the native library API is already written as a set of C++ classes.

In this case, PluginBase is similar to a Java interface, so I could turn it into one, or I could just put everything straight into the Plugin class in the Java version.

I’ll start with the latter, because it’s simpler. At this stage my only goal is to get something built that I can run and test. I can refactor later.

So, my first draft of Plugin is:

package org.vamp_plugins;

public class Plugin
{
    public native String getIdentifier();
    public native String getName();
    public native String getDescription();
    public native int getPluginVersion();
}

The methods are all declared native and left unimplemented in the Java; they’ll be implemented in our C++ JNI wrapper later on.

Meanwhile I want my PluginLoader to look a bit like this:

package org.vamp_plugins;

public class PluginLoader
{
    public class LoadFailedException
        extends Exception { };
    public static synchronized PluginLoader getInstance() {
        // some magic here
    }
    public Plugin loadPlugin(String key, float inputSampleRate)
        throws LoadFailedException {
        // and here
    }
}

The C++ PluginLoader returns a null pointer if it can’t load a plugin. I want to throw an exception instead, and for this reason I haven’t marked loadPlugin as native—instead it will call a native method that returns a null or non-null value, throwing the exception if the returned value is null.

I’ve made other simplifications too. I happen to know that the PluginKey type referred to in the C++ API is just a typedef for std::string, so I’ve translated it to a Java string type for the moment. I’ve omitted the options argument for now as well.

Native object handles

I have two Java classes, each instance of which should “manage” a C++ object of a corresponding class from the native library. Every time a method is called on a given Java instance, it should call through to the same instance of the underlying C++ class.

To make this happen, we need to make the Java object remember which C++ object it is managing.

For this purpose we’ll have both classes simply contain a “handle” field, which will hold a value that (on the native side of the fence) will be interpreted as a pointer to the C++ object on the heap, and (on the Java side) will be an opaque integer value.

We just need to make sure the handle is stored in an integer type big enough for a pointer; the Java long type will do.

Also, we need a way to construct Plugin objects. The PluginLoader will make plugins by calling its native-code implementation, getting back a C++ plugin pointer, and using that to construct our Java Plugin, so our plugin class needs a constructor that works from our opaque handle type.

Adding these details and filling in our singleton implementation gives us:

public class PluginLoader
{
    public class LoadFailedException
        extends Exception { };

    public static synchronized PluginLoader getInstance() {
        if (inst == null) {
            inst = new PluginLoader();
            inst.initialise();
        }
        return inst;
    }

    public Plugin loadPlugin(String key, float inputSampleRate)
        throws LoadFailedException {
        long handle = loadPluginNative(key, inputSampleRate);
        if (handle != 0) return new Plugin(handle);
        else throw new LoadFailedException();
    }

    private PluginLoader() { initialise(); }
    private native long loadPluginNative(String key, float inputSampleRate);
    private native void initialise();
    private static PluginLoader inst;
    private long nativeHandle;
}

and

public class Plugin
{
    private long nativeHandle;
    protected Plugin(long handle) {
        nativeHandle = handle;
    }

    public native String getIdentifier();
    public native String getName();
    public native String getDescription();
    public native int getPluginVersion();
}

The initialise method in PluginLoader is there to construct a native object and set up its nativeHandle field. Plugin doesn’t need a similar method, because its handle is given to it in the constructor.

Hang on! How does the native object get deleted? — I give over a later post entirely to discussion of this question, so skip there if you’re interested now.

3. Generate JNI function declarations

For each of the native methods in our Java classes, we need an implementation in the JNI wrapper code. This will be a function in C or C++, with a name encoding the Java class and method name and an appropriate signature to receive the JNI mappings for the Java argument types, and with C linkage.

It’s perfectly possible to write these from scratch, but the JDK includes a tool called javah that can help by producing the function prototypes. It requires compiled class files rather than Java files as input:

$ javac org/vamp_plugins/*.java
$ javah -jni org.vamp_plugins.Plugin org.vamp_plugins.PluginLoader
$ ls -1
org
org_vamp_plugins_Plugin.h
org_vamp_plugins_PluginLoader.h
sample.h
$

As you see, javah generates a header file for each Java class; they contain declarations like

JNIEXPORT jstring JNICALL
Java_org_vamp_1plugins_Plugin_getIdentifier
 (JNIEnv *, jobject);

The JNIEXPORT and JNICALL macros simply ensure traditional C linkage regardless of the build platform.

The method’s return type, String in the Java code, has been converted to the opaque JNI type jstring.

The function name is an encoding of the package, class, and method name. These must appear exactly as generated by javah, as the JVM relies on them to find the implementation code.

And the two arguments, which are common to all JNI implementation functions, provide the environment object which rolls up the JNI function namespace, and an opaque jobject referring to the this object on which the method was called.

If we leave these headers untouched and simply include them in our implementation files, then we’ll be able to regenerate them when we extend our Java API without losing any of our work.

Administrative note: If you haven’t already, this is a good moment to put the code directory under version control, so as to be able to mess around without fear of losing any working code.

I like Mercurial, so I would either hg init; hg add org/vamp_plugins/*.java and hg commit, or else run up EasyMercurial and follow roughly the explanation in this video.

Next

  • In my next article I talk about writing the JNI wrapper implementations for these functions, and testing them from Java.