When the Dogecoin Foundation’s libdogecoin had its first release a couple of weeks ago, it included support for programming languages C (with C++ implicitly), Go, and Python.
More languages are better, so a few of us hackers started to develop support other languages like JavaScript, Perl, and PHP.
I’ve done this thing a couple of times. Way back when, I wrote some Ruby bindings for the company building I Want Sandy and Stikkit. Twitter bought the company. You might have heard of Twitter. (I mistimed things, sadly.)
This work is tooling work; making things easier for other developers, so that they can go on to build other things. Many of those other things are going to be amazing. They’ll be a lot more amazing if those other developers don’t have to repeat all of the tooling work.
(Patrick and I talk about this occasionally; we’d both also like to build amazing things on top of good tools, but someone has to build the tools, and I’m assured that we are both someones.)
In the Repos Bind Them
What are language bindings? Let’s go down to the details.
The project libdogecoin is a set of functions written in the C programming
language. These functions do basic things to work with the Dogecoin network and
protocols. For example, there are functions to generate private and public keys
and to validate and verify those keys. There are also functions to interact
with the network, such as creating transactions and broadcasting them to other
nodes.
This project exists so that everyone everywhere can have a simple, reliable, standardized way to interact with the Dogecoin network in various forms. It doesn’t compete with Dogecoin Core; it exists so that other projects don’t have to rely on the Core to get things done.
In other words, if you want to build an arcade that takes Dogecoin instead of
quarters or tokens or swipes of a plastic card, you could write (or run)
software that uses libdogecoin on a tiny little piece of hardware and still
get the job done, even if you run a full Core wallet on your laptop in the back
office.
The more opportunities to collaborate, the more interesting opportunities to build useful things, and the more useful things, the better the network becomes.
YABI YAPI DO!
Why C?
In basic, nerdy-but-not-too-nerdy terms, C is the lowest common denominator language most computers understand these days, in the sense that pretty much any machine anywhere, from a smart watch to a supercomputer to your smart phone to a microchip running inside the dashboard of your car understands some variety of C.
This isn’t actually true in the way that “two plus two is four, give or take” is true. It’s true in the same way that “the sky is blue” is true. It’s true because we’ve collectively decided it should be true, and in those cases where it’s not true, we pretty much ignore them.
What’s actually true is that a modern operating system like a Windows or a Mac OS or a Linux or a BSD supports something called libraries. Sometimes that library is a bunch of code that’s bundled together into a single program you can run and other time’s it’s a separate file that another program loads when it runs.
In both cases, the operating system has to know that this library exists, that it has a bunch of functions, and that the program you want to run can use the functions from that library. The operating system has a specific way of identifying these libraries and the functions they contain, and if you’ve never heard of something before called an Application Binary Interface (ABI), then this is the set of rules the operating system has for how these libraries work.
If you have heard of this before, you know that I’m stretching the truth, but you ought to know why I’m stretching the truth.
When it comes time to write in a language like Perl, PHP, Python, Ruby, JavaScript, Lua, or countless other languages, they behave differently from how C behaves. Even C++ (the language in which most of Dogecoin Core is written) behaves a little differently from C. All of these languages behave consistently with themselves, but they don’t behave exactly like C does.
What we do know about how C behaves is that it has documented behavior, in
the sense that to make libdogecoin work and work with other programs (no
matter what they’re written in), the library includes a series of header
files
that tell computers and humans what the library provides.
When I say “Perl should be able to ask libdogecoin to generate a
private/public keypair for the testnet”, I know the name of a function to call
in the library, the values to provide that function, and the values I will
receive.
If you’ve never heard of something called an Application Programming Interface (API), this is what one is. If you have heard of this before… aw okay, I fib sometimes, but for good didactic reasons.
A Box of “Lost In Translation” DVDs
Wait a minute, though. I said that Perl and Python and Ruby et cetera don’t
have the same ABI as C does, or at least I implied it heavily. (It’s true!) If
I know the API of libdogecoin, how do I get from that to the ABI of
libdogecoin on my Linux machine?
I’m glad you asked! Aren’t you glad I put words in your mind?
What Perl thinks of as a public key is a string. In Perl terms, that’s a chunk of memory that has data like “treat it like this character set” or “it’s this many bytes long in memory” or “if you want to turn this into a number, it corresponds to this number”. In C terms, that’s “here’s a memory address”.
It’s not easy to get from one to the other, but it’s a mechanical translation.
Similarly, what Perl thinks of as a boolean true-or-false value is sometimes a boolean true-or-false value but other times “here’s a chunk of data, and if you ask if it’s true I’ll tell you”. What C thinks of as a boolean true-or-false value is (sometimes) a single on-or-off bit. Translating between those is a little bit tricky.
When you call a Perl function in a Perl program, the Perl core takes the arguments you provide, puts them in a special place in memory, calls the function, does some work, and puts the results back in that special place in memory in a slightly different way. When you call a C function, the computer takes the arguments you provide and treats them in an entirely different way, sometimes in a very different place in the computer (not really memory per se) and things are entirely different.
How do you bridge that gap?
You might already start to see the answer, now that I’ve said “it’s a mechanical translation”, but it’s also largely a pain in the neck.
Something has to sit between Perl and C (or Python and C or Ruby and C) and translate data from the host language’s data format to the C ABI data format and the function call from the host language’s function call mechanism to the C ABI function call mechanism and are you feeling tired yet?
Most of these languages have an extension interface where you can write a little bit of C or C++ code to load a platform-specific library and manually translate data and function calls to and from the host language. Some are easier than others. Tcl makes it really easy. Ruby wasn’t bad. I enjoyed that work when I did it for Stikkit/I Want Sandy. It’s not super fun with Perl, but it’s not too bad.
At least, it’s not too bad the old way. There’s a better way.
Future Foundation Interface
A couple of years back when I wrote the fourth edition of the Modern Perl book, I wrote something like “If you find yourself writing language bindings to shared libraries, look at FFI::Platypus. I had played with it briefly myself, but never for anything serious. Even so, I recognized that the pattern it uses could make a lot of difficult things easy.
It took me longer to write the introduction to this blog post than it did to
get the first couple of libdogecoin functions working from Perl, thanks to
the Platypus.
Python’s bindings work in a similar way. A little library called libffi knows
how the current operating system’s ABI works. It loads a library and lets you
refer to functions inside that library. If you tell it the specific details of
the values you provide to a function, the name of that function, and the values
you receive from the function, libffi will generate the mechanical
transformation code you need to call those functions.
FFI stands for Foreign-Function Interface, and if you read the wisdom of old Lisp hackers sometime, you’ll find that there’s a good history behind this idea. At least, that’s what some old Lisp hackers told me once when I drew this idea out on a whiteboard in wine country in California one summer afternoon.
The real magic in Python’s cffi and Perl’s FFI-Platypus and other systems
like this is taking a Perl or Python or Dart or Ruby or PHP variable, making it
available to the C ABI, and doing the same operation in reverse. This can be
really difficult in some situations, but in the best of situations it’s
tedious.
I mostly didn’t have to worry about this at all, and it’s really spectacular when it works, especially because I know how much effort this could have been without this tooling that feels almost magical.
Good Tools are Like Onions and Ogres
The libdogecoin hackers who wrote the Python bindings in the repository and
me and the other hackers who’ve written other bindings could have done the hard
work ourselves. It’s all pretty well understood work. Yes, it’s tedious, but
it’s not like inventing a completely new consensus algorithm to solve the
Byzantine Generals problem for a multi-billion dollar market value currency in
a field rife with scammers, spammers, schemers, bad actors, and the occasional
stooge.
It’s a lot easier with good tools at hand, though.
Similarly, my hope is that someone else will take the work that us language binders did and build something on top of that and say “I could have done this other work, but what’s there already exists and works and feels a little magical, so I could focus on other things”.
In this way, all of our work builds on other work, and we make it easier to focus on bigger, more interesting, more creative, more useful problems instead of solving the same problem all over and over and over again.
Even if you don’t care about the nuances of ABIs and APIs and FFIs and all of this nerdy systems-level programming stuff, I hope you see the goal we have in mind: make it easy to be creative, to play, to experiment, to have fun, to build useful things without having to reinvent things that already exist and work.
See Alien-Libdogecoin for the
Perl code that bundles up the libdogecoin library for Perl users to install
from the CPAN and
Finance-Libdogecoin for the
Perl code that provides the foreign-function interface to use libdogecoin in
Perl programs.
