F♯ has possibilities

A couple of months ago, Microsoft announced that they were buying a company called Xamarin, co-founded by the admirable Miguel “you can now flame me, I am full of love” de Icaza. (No sarcasm — I think Miguel is terrific, and the delightfully positive email linked above really stuck with me; if only I could have that attitude more often.)

As I understand it, Xamarin makes

  1. the Mono runtime, a portable third-party implementation of Microsoft’s .NET runtime for the C# and F# programming languages
  2. the eponymous Xamarin frameworks, which can be used with .NET to develop mobile apps for iOS and Android
  3. plugins for the Visual Studio IDE on Windows and the MonoDevelop IDE on OS/X to support mobile platform builds using Xamarin (the MonoDevelop-plus-plugins combo is known as Xamarin Studio).

Then a couple of days ago, the newly-acquired Xamarin declared

  1. that the Mono runtime was switching from LGPL/GPL licenses to MIT, allowing no-cost use in commercial applications
  2. that Microsoft were providing a patent promise (which I have not closely read) to remove concerns for commercial users of Mono
  3. that the Xamarin frameworks for iOS and Android, and the IDE plugins, were now free (of cost)
  4. that at some future point the Xamarin frameworks would be open sourced

I’m trying to unpick exactly what this could mean to me.

According to this discussion on Hacker News, the IDE plugins are remaining proprietary (which appears to mean that no IDE on Linux will be supported, since the IDE plugins are not currently available for Linux) but that “the Xamarin runtime and all the commandline tools you need to build apps” will be open sourced.

What this means

as I understand it,

  • Developers working on proprietary .NET applications will be able to build and release versions for other platforms than Windows, using Mono, at no extra cost
  • Developers working on open source .NET applications will be able to publish the ensemble with Mono under the MIT license if desired and will (apparently) be free of patent concerns
  • Developers will be able to make both proprietary and open source .NET applications for iOS and Android at no cost using Windows and OS/X
  • There is a possibility of being able to do builds of the above using Linux as well once the SDK is open, though probably without an IDE

Unrelatedly, there are separate projects afoot to provide native code and to-Javascript compilers for .NET bytecode.

What I’m interested in

I do a range of programming including a mixture of signal-processing and UI work, and am interested in exploring comprehensible, straightforward functional languages in the ML family (I wrote a little post about that here). Unlike many audio developers I have relatively limited demands on real-time response, but everything I write really wants to be cross-platform, because I’ve got specialised users on pretty every common platform and I have limited time and funding. (I understand that cross-platform apps are often inferior to single-platform apps, but they’re better than no apps.)

Xamarin doesn’t quite meet my expectations because it’s not really a cross-platform framework in the manner of Qt (which I use) or JUCE (which is widely used by others in my field). Instead of providing a common “widget set” across all platforms, Xamarin provides a separate thin interface to the native UI logic for each platform. It’s hard to judge how much more work this is, without knowing where the abstraction boundaries lie, but it may be a more relevant and sensible distinction on mobile platforms (where the differences are often in interaction and layout) than desktops (where the differences are mostly about how large numbers of individual widgets look).

An ideal combination of language and framework for me goes something like

  • strongly-typed, mostly functional, mostly immutable data structures
  • efficient unboxed support for floating-point vector types, including SIMD support
  • simple syntax (SML is nice)
  • low-cost foreign-function interface for C integration
  • high-level approach to multithreading
  • can work with gross UI layout in HTML5 (possibly DOM-update reactive UI style?)
  • good libraries for e.g. audio file I/O, signal processing, matrix algebra
  • can develop on Linux and deploy to all of Linux, Windows, OS/X, iOS, Android
  • free (or cheap, for proprietary apps) and open source (for open source apps)
  • has indenting Emacs mode

Where F# appears to score

F#, Microsoft’s ML-derived functional language for the .NET CLR, hits several of these. It has the typing, mostly-functional style, syntax, FFI, multithreading, libraries, deployment and licensing, and potentially the development platform (if the open source Xamarin framework should lead to the ability to build mobile apps directly from Linux).

I’m not sure about floating-point and vectors or about reusable HTML-style UI. I’d like to make the time to do another comparison of some ML-family languages, focusing on DSP-style float activity and on threading. I’ve done a bit of related work in Standard ML, which I could use as a basis for comparison.

Unless and until I get to do that, I’d love to hear any thoughts about F# as a general-purpose DSP-and-UI language, for a developer whose home platform is Linux.

My impression from the feedback on my earlier post was that the F# community is both enthusiastic and polite, and I notice that F# is the third most-loved language in the StackOverflow’s 2016 survey. Imagine a language that is useful no matter what platform you’re targeting, and whose developers love it. I can hope.

 

Feedback on “Four MLs (and a Python)”

My last post got an unprecedented amount of attention after appearing on the popular site Hacker News. It spent about 14 hours on the front page there and got almost 11,000 unique visitors that day — a fair way above my usual daily average, for this whole blog, of about a hundred.

I combed through the comments on the Hacker News post, as well as entries on lobste.rs and Reddit. Counting only comments directly about the post (rather than to other commenters), I reckoned there were:

  • Nine comments responding only, or primarily, to performance differences in the language implementations;
  • Six suggestions to hoist regexp construction out of the split function in the OCaml version, to make it faster: four of them included code, and they were all quite polite;
  • Six objections that my Python code was un-idiomatic, un-Pythonic, or generally strange. Five of those proposed rewriting it to use the CSV parser module or pandas — good advice in general, though not really in line with the purpose of the post;
  • Six code rewrites in, enthusiasms about, or defences of, F♯;
  • Six comments suggesting other languages to look at (4 x Haskell; Scala; Mythryl);
  • Three observations that the Python code would run faster with PyPy;
  • Three people sharing my general feeling that it would be a good thing if Standard ML had more practical things going for it;
  • Two people having difficulty with my use of the term “Big Data” to describe a 300MB CSV file. (Sorry. I really only ever use the term ironically, but you weren’t to know that.)

I must say, OCaml developers seem like a keen bunch. Actually the vibe I get about both OCaml and F♯, from these few comments, is a happy one. About SML it’s more a sense of distant yearning. The Python comments were a bit grumpier, but that’s reasonable given that my post didn’t exactly engage with the way things actually get done in Python.

It’s quite obviously hard to resist a speed contest. I wasn’t initially going to include the runtime measurements in the post, but I couldn’t help myself either. Still, none of the coding decisions I made when writing the test programs had anything to do with speed. (Scalability, in a simplistic way, yes: I made sure not to read the whole file into memory, and to avoid non-tail-recursive code.) Then of course more people responded about performance than anything else. And even though I can see that that way lies madness, I’d still like to find time to follow up on the question of why the SML code didn’t go as quickly as the OCaml did.

Since I wrote that post, I’ve had a piece of real work come up for which these languages are quite well suited, involving parsing and manipulating a modest amount of symbolic data with a very specialised structure. I decided to do it using Standard ML this time, and hope to find out whether there’s a point at which it becomes more impractical than delightful.

Four MLs (and a Python)

I wrote a small command-line text processing program in four different ML-derived languages, to try to get a feel for how they compare in terms of syntax, library, and build-run cycles.

ML is a family of functional programming languages that have grown up during the past 40 years and more, with strong static typing, type inference, and eager evaluation. I tried out Standard ML, OCaml, Yeti, and F#, all compiling and running from a shell prompt on Linux.

The job was to write a utility that:

  • accepts the name of a CSV (comma-separated values) file as a command-line argument
  • reads all the lines from that file, each consisting of the same number of numeric columns
  • sums each column and prints out a single CSV line with the results
  • handles large inputs
  • fails if it finds a non-numeric column value or an inconsistent number of columns across lines (an uncaught exception is acceptable)

A toy exercise, but one that touches on file I/O, library support, string processing and numeric type conversion, error handling, and the build-invocation cycle.

I tested on a random Big Data CSV file that I had to hand; running the wc (word count) utility on it gives the size and a plausible lower bound for our program’s runtime:

$ time wc big-data.csv 
 337024 337024 315322496 big-data.csv

real 0m3.086s
user 0m3.050s
sys 0m0.037s
$

I’ve included timings throughout because I thought a couple of them were interesting, but they don’t tell us much except that none of the languages performed badly (with the slowest taking about 16 seconds on this file).

Finally I wrote the same thing in Python as well for comparison.

Practical disclaimer: If you actually have a CSV file you want to do things like this with, don’t use any of these languages. Do it with R instead, where this exercise takes three lines including file I/O. Or at least use an existing CSV-mangling library.

Here are the programs I ended up with, and my impressions.

Standard ML

Standard ML, or SML, is the oldest and “smallest” of the four and the only one to have a formal standard, fixed since 1997. Its standard library (the Basis library) is a more recent addition.

fun fold_stream f acc stream =
    case TextIO.inputLine stream of
	SOME line => fold_stream f (f (line, acc)) stream
      | NONE => acc
    
fun to_number str =
    case Real.fromString str of
	SOME r => r
      | NONE => raise Fail ("Invalid real: " ^ str)
		      
fun values_of_line line =
    let val fields = String.fields (fn c => c = #",") line in
	map to_number fields
    end

fun add_to [] values = values
  | add_to totals values =
    if length totals = length values then
	ListPair.map (Real.+) (totals, values)
    else raise Fail "Inconsistent-length rows"

fun sum_stream stream =
    fold_stream (fn (line, tot) => add_to tot (values_of_line line)) [] stream
		    
fun sum_and_print_file filename =
    let val stream = TextIO.openIn filename in
	let val result = sum_stream stream in
	    print ((String.concatWith "," (map Real.toString result)) ^ "\n")
	end;
	TextIO.closeIn stream
    end
							     
fun main () =
    case CommandLine.arguments () of [filename] => sum_and_print_file filename
      | _ => raise Fail "Exactly 1 filename must be given"

val () = main ()

(Note that although I haven’t included any type annotations, like all ML variants this is statically typed and the compiler enforces type consistency. There are no runtime type errors.)

This is the first SML program I’ve written since 23 years ago. I enjoyed writing it, even though it’s longer than I’d hoped. The Basis library doesn’t offer a whole lot, but it’s nicely structured and easy to understand. To my eye the syntax is fairly clear. I had some minor problems getting the syntax right first time—I kept itching to add end or semicolons in unnecessary places—but once written, it worked, and my first attempt was fine with very large input files.

I had fun messing around with a few different function compositions before settling on the one above, which takes the view that, since summing up a list is habitually expressed in functional languages as an application of fold, we could start with a function to apply a fold over the sequence of lines in a file.

More abstractly, there’s something delightful about writing a language with a small syntax that was fixed and standardised 18 years ago and that has more than one conforming implementation to choose from. C++ programmers (like me) have spent much of those 18 years worrying about which bits of which sprawling standards are available in which compiler. And let’s not talk about the lifespans of web development frameworks.

To build and run it I used the MLton native-code compiler:

$ time mlton -output sum-sml sum.sml

real 0m2.295s
user 0m2.160s
sys 0m0.103s
$ time ./sum-sml big-data.csv 
150.595368855,68.9467923856,[...]

real 0m16.383s
user 0m16.370s
sys 0m0.027s
$

The executable was a 336K native binary with dependencies on libm, libgmp, and libc. Although the compiler has a good reputation, this was (spoiler alert!) the slowest of these language examples both to build and to run. I also tried the PolyML compiler, with which it took less than a tenth of a second to compile but 26 seconds to run, and Moscow ML, which was also fast to compile but much slower to run.

OCaml

OCaml is a more recent language, from the same root but with a more freewheeling style. It seems to have more library support than SML and, almost certainly, more users. I started taking an interest in it recently because of its use in the Mirage OS unikernel project—but of these examples it’s the one in which I’m least confident in my ability to write idiomatic code.

(Edit: at least two commenters below have posted improved versions of this—thanks!)

open Str

let read_line chan =
  try Some (input_line chan)
  with End_of_file -> None

let rec fold_channel f acc chan =
  match read_line chan with
  | Some line -> fold_channel f (f line acc) chan
  | None -> acc

let values_of_line line =
  let fields = Str.split (Str.regexp ",") line in
  List.map float_of_string fields
  
let add_to totals values =
  match totals with
  | [] -> values
  | _  ->
     if List.length totals = List.length values then
       List.map2 (+.) totals values
     else failwith "Inconsistent-length rows"

let sum_channel chan =
  let folder line tot = add_to tot (values_of_line line) in
  fold_channel folder [] chan
	      
let sum_and_print_file filename =
  let chan = open_in filename in
  (let result = sum_channel chan in
   print_string ((String.concat "," (List.map string_of_float result)) ^ "\n");
   close_in chan)

let main () =
  match Sys.argv with
  | [| _; filename |] -> sum_and_print_file filename
  | _ -> failwith "Exactly 1 filename must be given"
    
let () = main ()

I’m in two minds about this code. I don’t much like the way it looks and reads. Syntax-wise there are an awful lot of lets; I prefer the way SML uses fun for top-level function declarations and saves let for scoped bindings. OCaml has a more extensive but scruffier library than SML and although there’s lots of documentation, I didn’t find it all that simple to navigate—as a result I’m not sure I’m using the most suitable tools here. There is probably a shorter simpler way. And my first attempt didn’t work for long files: caught out by the fact that input_line throws an exception at end of file (ugh), I broke tail-recursion optimisation by adding an exception handler.

On the other hand, writing this after the SML and Yeti versions, I found it very easy to write syntax that worked, even when I wasn’t quite clear in my head what the syntax was supposed to look like. (At one point I started to worry that the compiler wasn’t working, because it took no time to run and printed no errors.)

I didn’t spot at first that OCaml ships with separate bytecode and optimising native-code compilers, so my first tests seemed a bit slow. In fact it was very fast indeed:

$ time ocamlopt -o sum-ocaml str.cmxa sum.ml

real 0m0.073s
user 0m0.063s
sys 0m0.003s
$ time ./sum-ocaml big-data.csv 
150.595368855,68.9467923856,[...]

real 0m7.761s
user 0m7.740s
sys 0m0.027s
$

The OCaml native binary was 339K and depended only on libm, libdl, and libc.

Yeti

Yeti is an ML-derived language for the Java virtual machine. I’ve written about it a couple of times before.

valuesOfLine line =
    map number (strSplit "," line);
    
addTo totals row =
    if empty? totals then array row
    elif length totals == length row then array (map2 (+) totals row)
    else failWith "Inconsistent-length rows"
    fi;

rowsOfFile filename =
    readFile filename "UTF-8"
        do handle: map valuesOfLine (handle.lines ()) done;

sumFile filename =
    fold addTo (array []) (rowsOfFile filename);

sumAndPrintFile filename =
    println (strJoin "," (map string (sumFile filename)));

case (list _argv) of
     [filename]: sumAndPrintFile filename;
     _: failWith "Exactly 1 filename must be given";
esac

I love Yeti’s dismissive approach to function and binding declaration syntax—no let or fun keywords at all. Psychologically, this is great when you’re staring at an empty REPL prompt trying to decide where to start: no syntax to forget, the first thing you need to type is whatever it is that you want your function to produce.

The disadvantage of losing let and fun is that Yeti needs semicolons to separate bindings. It also makes for a visually rather irregular source file.

As OCaml is like a pragmatic SML, so Yeti seems like a pragmatic OCaml. It provides some useful tools for a task like this one. Although the language is eagerly evaluated, lazy lists have language support and are interchangeable with standard lists, so the standard library can expose the lines of a text file as a lazy list making a fold over it very straightforward. The default map and map2 functions produce lazy lists.

Unfortunately, this nice feature then bit me on the bottom in my first draft, as the use of a lazy map2 in line 6 blew the stack with large inputs (why? not completely sure yet). The standard library has an eager map as well as a lazy one but lacks an eager map2, so I fixed this by converting the number row to an array (arguably the more natural type for it).

The Yeti compiler runs very quickly and compiles to Java .class files. With a small program like this, I would usually just invoke it and have the compiler build and run it in one go:

$ time yc ./sum.yeti big-data.csv 
150.59536885458684,68.9467923856445,[...]

real 0m14.440s
user 0m26.867s
sys 0m0.423s
$

Those timings are interesting, because this is the only example to use more than one processor—the JVM uses a second thread for garbage collection. So it took more time than the MLton binary, but finished quicker…

F♯

F♯ is an ML-style language developed at Microsoft and subsequently open-sourced, with a substantial level of integration with the .NET platform and libraries.

(Edit: as with the OCaml example, you’ll find suggestions for alternative ways to write this in the comments below.)

let addTo totals row =
    match totals with
    | [||] -> row
    | _ ->
       if Array.length totals = Array.length row then
         Array.map2 (+) totals row
       else failwith "Inconsistent-length rows"

let sumOfFields fields =
    let rows = Seq.map (Array.map float) fields in
    Seq.fold addTo [||] rows

let fieldsOfFile filename = 
    seq { use s = System.IO.File.OpenText(filename)
          while not s.EndOfStream do yield s.ReadLine().Split ',' }

let sumAndPrintFile filename =
    let result = fieldsOfFile filename |> sumOfFields in
    printfn "%s" (String.concat "," (Array.map string result))

[<EntryPoint>]
let main argv = 
    match argv with
    | [|filename|] -> (sumAndPrintFile filename; 0)
    | _ -> failwith "Exactly 1 filename must be given"

F♯ also has language support for lazy lists, but with different syntax (they’re called sequences) and providing a Python-style yield keyword to generate them via continuations. The sequence generator here came from one of the example tutorials.

A lot of real F♯ code looks like it’s mostly plugging together .NET calls, and there are a lot of capital letters going around, but the basic functional syntax is almost exactly OCaml. It’s interesting that the fundamental unit of text output seems to be the formatted print (printfn). I gather F♯ programmers are fond of their |> operator, so I threw in one of those.

I’m running Linux so I used the open source edition of the F♯ compiler:

$ time fsharpc -o sum-fs.exe sum.fs
F# Compiler for F# 3.1 (Open Source Edition)
Freely distributed under the Apache 2.0 Open Source License

real 0m2.115s
user 0m2.037s
sys 0m0.063s
$ time ./sum-fs.exe big-data.csv 
150.595368854587,68.9467923856445,[...]

real 0m13.944s
user 0m13.863s
sys 0m0.070s
$

The compiler produced a mere 7680-byte .NET assembly, that of course (like Yeti) requires a substantial managed runtime. Performance seems pretty good.

Python

Python is not an ML-like language; I include it just for comparison.

import sys

def add_to(totals, values):
    n = len(totals)
    if n == 0:
        return values
    elif n == len(values):
        return [totals[i] + values[i] for i in range(n)]
    else:
        raise RuntimeError("Inconsistent-length rows")
        
def add_line_to(totals, line):
    values = [float(s) for s in line.strip().split(',')]
    return add_to(totals, values)

def sum_file(filename):
    f = open(filename, 'r')
    totals = []
    for line in f:
        totals = add_line_to(totals, line)
    f.close()
    return totals

if __name__ == '__main__':
    if len(sys.argv) != 2:
        raise RuntimeError("Exactly 1 filename must be given")
    result = sum_file(sys.argv[1])
    print(','.join([str(v) for v in result]))

Feels odd having to use the return keyword again, after using languages in which one just leaves the result at the end of the function.

This is compact and readable. A big difference from the above languages is invisible—it’s dynamically typed, without any compile-time type checking.

To build and run this, I just invoked Python on it:

$ time python ./sum.py ./big-data.csv 
150.59536885458684,68.9467923856445,[...]

real 0m10.939s
user 0m10.853s
sys 0m0.060s
$

That’s Python 3. Python 2 was about a second faster. I was quite impressed by this result, having expected to suffer from my decision to always return new lists of totals rather than updating the values in them.

Any conclusions?

Well, it was a fun exercise. Although I’ve written more in these languages than appears here, and read quite a bit about all of them, I’m still pretty ignorant about the library possibilities for most of them, as well as about the object support in OCaml and F♯.

I am naively impressed by the OCaml compiler. For language “feel”, it gave me the least favourable first impression but I can imagine it being pleasant to use daily.

F♯ on Linux proved unexpectedly straightforward (and fast). Could be a nice choice for web and server applications.

I have made small web and server applications using Yeti and enjoyed the experience. Being able to integrate with existing Java code is good, though of course doubly so when the available libraries in the language itself are so limited.

Standard ML has a clarity and simplicity I really like, and I’d still love to try to use it for something serious. It’s just, well, nobody else seems to—I bet quite a lot of people have learned the language as undergrads (as I did) but it doesn’t seem to be the popular choice outside it. Hardly anyone uses Yeti either, but the Java interoperability means you aren’t so dependent on other developers.

Practically speaking, for jobs like this, and where you want to run something yourself or give someone the source, there’s not much here to recommend anything other than Python. Of course I do appreciate both compile-time typechecking and (for some problems) a more functional style, which is why I’m writing this at all.

But the fact that compilers for both SML and OCaml can generate compact and quick native binaries is interesting, and Yeti and F♯ are notable for their engagement with other existing frameworks.

If you’ve any thoughts or suggestions, do leave a comment.