Red Echo

December 15, 2016

Welcome to 2030. I own nothing, have no privacy, and life has never been better

There is a good business idea in this article of speculative fiction, which was written by an MP in Denmark:

Welcome to 2030. I own nothing, have no privacy, and life has never been better

Once in awhile, I will choose to cook for myself. It is easy — the necessary kitchen equipment is delivered at my door within minutes. Since transport became free, we stopped having all those things stuffed into our home. Why keep a pasta-maker and a crepe cooker crammed into our cupboards? We can just order them when we need them.

Well, why not? All my tools, which used to take up space in my house, now live at a makerspace, where they can be used by other people during the great majority of the time that I am not personally using them. How many kitchen appliances – or even just pots and pans – could be similarly homed elsewhere, if it could be assured that I could have them delivered whenever I actually needed to use them?

I cook a full meal for guests once a week. I cook a little bit, for myself, a handful of other times a week. The only pieces of kitchen equipment which are reliably used every single day in my house are the kettle and the coffee press. Yet the kitchen – which takes up about a quarter of one floor in the building I call home – is largely comprised of storage space, much of which serves to contain objects that are used at most once or twice a month.

Well, why not store it somewhere else? There are already banquetting services which will deliver a flat of glassware or plates or whatever if you are going to host a dinner party, and then pick them up again afterward. Why not do the same with boxes full of pots and pans, or with specialized hardware like a slow-cooker, a fondue pot, a panini press, or a waffle maker? I don’t own any of those objects and so I never cook the foods they are designed for, but I totally would if I could just have one dropped off whenever I needed it, without needing to purchase it, own it, and store it afterward.

August 16, 2016

I am going to Burning Man this year, and I have mixed feelings about it. This will be my tenth burn since 2001; I last visited Black Rock City in 2012. This experience has changed me, thoroughly; but the years have also changed the city, and the ways we have changed are not always congruent. I’m not sure how well I fit in there any more. I wonder – to what degree was the activating sense of freedom and possibility that ignited my current life pattern an artifact of that moment, rather than an ongoing characteristic of the event? Do I still need Burning Man? Does it still need me?

I’m looking forward to working hard in the dust with my friends. I’m looking forward to camping with AJ. Dancing under the stars, exploring creative art projects, meeting interesting people, finding creative solutions to unexpected problems; I love this world. It will be a good time.

I don’t think that Burning Man still has the power to change my life, because it already did that. I kept on bringing it home, over and over, until I’d rebuilt my life around it. Would the 2016 burn experience have that effect on me if I were coming to it for the first time? I know I’ve changed, and I know the burn has changed, but it’s hard to know which feelings to associate with which changes.

August 1, 2016

Xvisor is an open-source hypervisor targeting various flavors of ARM as well as AMD64. I’m curious whether it might be a useful component for Fleet.

July 11, 2016

Wow, do I feel loved. Adam H. just showed up at my house to coordinate delivery of a gift from a group of friends – I didn’t catch all the names he rattled off, and will have to catch up to thank them properly – a hot tub! Wow! This tickles me just right – it will be fun and relaxing for me and the others who live here, but I’m also delighted that I’ll be able to offer a warm refreshing soak to my guests after dinners and parties. – Including, of course, the friends who bought it for me. What a lovely virtuous cycle!

June 18, 2016

Fractal Fryday psytrance

The Seattle Psy Ops crew invited me to play the opening set at their monthly Fractal Fryday event, and I recorded the mix for your listening pleasure. As much fun as I’ve been having with glitch-hop and downtempo lately, progressive psytrance remains my main squeeze, and getting to kick the night off for a club full of psytrance aficionados was a real treat.

June 13, 2016

I played a Doomtruck glitch-hop/electroswing set for Lesley & Steve’s birthday party this weekend and was delighted by the reception it got. People were in a mood to dance, light-hearted fun the order of the day, so I gave ’em the swingiest, funkiest, bassiest, booty-shakingest glitch in my library, and they loved it. Oh, so satisfying. As usual I recorded the set so you can stream or download.

May 22, 2016

Kinky Salon DJ mix set

Check out the set I recorded last night for Kinky Salon at the Factory Luxe – an hour of dirty, dubby downtempo for a Barbarella-style sexy-sci-fi party. I had fun mixing this up and I hope you’ll enjoy it too.

May 21, 2016

While I can’t say I ever had any particular interest in the idea of an ocean cruise, after reading the David Foster Wallace classic “A Supposedly Fun Thing I’ll Never Do Again” it seemed completely obvious that cruise ships were not a thing I would ever care to do. But I just read a really funny article on hackernews which describes a transatlantic crossing as the perfect environment for writing code – it sounds totally bonkers, but I can see what he’s getting at. Flipping it all around, he turns the isolation into focus and the scripted monotony into intellectual freedom; all he has to do is think and write, while the ship and its crew take care of everything else.

May 10, 2016

Saturday night DJ mix set

I posted a two-hour recording and track list from Saturday night over on Mixcrate: May 2016 at the Dungeon. Gritty, dubby, 90-105 bpm downtempo, with an industrial edge and traces of psychedelia.

April 18, 2016

I spent all weekend working on the Doom Truck with AJ. We are upgrading the shop box, making it a bit nicer to camp in, which mostly means cleaning up all the interior surfaces and building in a permanent bed frame with storage underneath. We’re not going to go full RV style with cabinets and fold-outs everywhere, because we like the open feeling it currently has, and we want it to continue being versatile.

First of all we have to stop all the water coming in, which caused a bunch of problems over the winter. One of the window frames leaks – but there are also lots of holes drilled through the walls and floor where previous owners had bolted down various workbenches and storage containers. Our plan was to weld plugs into all these holes, then grind ’em flat. We can always weld tie-down tabs onto the sheet metal if we want to.

Well… it was a long complicated process, and I started to bore myself trying to recount the details, so I’ll just say that it took half a dozen trips to various hardware stores and over $1000 spent on tools and supplies before nonstop labor finally got us to the point that we could start welding around 4 PM on Sunday. Oh, my.

We did end up making a fair bit of progress once we got going: all the floor holes are filled and ground down, so the wheels won’t kick water up into the box anymore, which means we can put down that nice cork flooring without having it rot from underneath… And I patched up the assortment of bolt holes down one side of the truck with big round rivety-looking carriage bolt heads, welded in place. The welds look terrible and it’s going to need a bunch of cleanup, but at least there’s no more daylight leaking in.

I didn’t get to the bolt holes on the other side of the truck; AJ thinks she will try to tackle those during the week. There are also a couple of big square access panels for equipment long since removed, with extremely janky covers… we’re going to weld the covers on, stuff some insulation in the gap, and weld them shut on the inside as well. Finally, we need to cut away and rebuild the threshold, which has leaked enough water in that the wood is a little rotten…

But hey, this is not a bad set of problems to have. The truck works great; we don’t have any mechanical problems to fix right now. It took a long time to get our metalwork process going but now things are moving quickly. And wow is it going to be nice to park our giant clanky apocalypse-proof monster truck at every festival and camping trip we go on this summer, then throw open the doors and have a clean, stylish, convenient, and actually pretty damn swanky home-away-from-home inside. I’m looking forward to it.

Also looking forward to installing a muffler, now that I can weld stuff in my back yard, because *damn* does that thing need one.

Oh yeah! I just discovered the wonder that is turbodiesel propane injection. It’s like nitrous for a gas engine, giving a significant horsepower boost, which will be super useful when climbing up hills – but not only that, it actually improves fuel efficiency, cleans up the exhaust, *and* puts no additional stress on the engine, since it actually *lowers* burn temperature slightly. Way simpler and cheaper than any of the other power-boosting improvements we’d been considering, and we’re already planning to have a propane tank on board to power the stove, heaters, etc. Super cool, it should mean it’s no longer a hassle trying to convoy out to an event with other vehicles.

April 1, 2016

I CAN’T STOP: it’s a Mackie SWA1801 for $650, JUST WHAT I ALWAYS WANTED (since the last time I got what I always wanted, temporarily(?) sating my PA-equipment lust with a pair of SWA1501s).

No, I’m not going to buy it. I’m not. Where would I PUT it? What about the fact that the gear I’ve collected will already max out a 20-amp circuit? For that matter, would this beast even fit through my basement door? Why do I keep looking at these things?

Well, I guess that one’s obvious: MORE BASS.

March 19, 2016

I don’t hate aggravator any more and may eventually have to change its name. I blamed the hardware, since it’s running the same Ubuntu 14.04 I’ve used successfully on three previous Thinkpads, but installing a different video driver fixed the “crash on resume” issue, which was really a “crash when the machine discovers you’ve plugged in its power cord” issue, and uninstalling whatever the hell “apport” is made the relentless barrage of “the machine has crashed, do you want to send us all your personal data so we can pretend we care” messages stop accosting me at startup.

With a second battery replacing the DVD player, I feel comfortable leaving the power supply at home, now, too.

Yay.

March 12, 2016

I recorded my DJ set last night – dubby, gritty downtempo – and have just finished uploading it for your enjoyment:

March 2016 at the Dungeon

March 6, 2016

KreativeKorp Relay Fonts has a glorious array of retrocomputing typefaces pulled from sources I haven’t thought about in years.

March 5, 2016

The junk removal people have come and gone. With them went all the random objects that have been sitting in the backyard at Rock Star Parking, unclaimed and unloved, since last summer’s Floodland truck unloading. In addition, they departed with an assortment of mysterious objects that had taken up residence under the porch for no apparent reason. Why, exactly, did we have two portable charcoal barbecues? Where did that knit bag of rubber stamps come from? Who put that big black plastic bag next to my motorcycle, and what was inside it? (None of us were brave enough to open it and find out.) And what kind of person brings a goddamn swamp cooler to a party and then forgets to take it home again afterward?

Also gone: the massive slab of seating furniture which would have been the first component of the now-cancelled couch-fortress project, the coffee table I’ve disliked since the day my ex-wife surprised me with it, and some bags of clothing which did not belong to any of this house’s past or present occupants. Alas: the cans of paint, piles of lead-acid batteries, and terrifyingly rusty propane cylinder were problems they were unwilling to solve. I guess it’s time to break out the box of thermite.

March 1, 2016

After sixteen years of membership, I just closed my Metafilter account. End of an era: it was the last piece of the old Web, the one I was excited about and emotionally invested in, the one that felt like a community – or a community of communities, really. No more; Metafilter’s culture has been growing steadily crankier over the years, less fun and more nitpicky, and has finally sunk below the “never read the comments” threshold.

The web is over; it’s basically just television, now, with a few exceptions. Reddit is still fun, though it’s more of a playground than a community. Blogs are over, message boards all seem to have gotten swallowed up by Facebook, and Facebook itself doesn’t have very much going on anymore.

I’m not sure what comes next, but I’m looking forward to finding out about it.

February 29, 2016

I didn’t get any photos, but the verdict was clear after Saturday night’s impromptu gathering: you totally can make delicious grilled-cheese sandwiches with a flamethrower. Use a soft cheese and keep the slices thin (so it’ll melt quickly), don’t butter the outside of the bread (it catches fire), and smash the sandwich between a couple of cast iron skillets (or use a panini press) if you’d rather not char the outside. 15-20 seconds of heat will do it, but you can go longer if you want the bread to get super crunchy.

Also, don’t assume that those propane cylinders you haven’t touched since last summer are still full.

February 16, 2016

A nice solid side-table lamp made of oak and epoxy, with step-by-step photos.

February 8, 2016

Colorization using optimization:

In this paper we present a simple colorization method that requires neither precise image segmentation, nor accurate region tracking. Our method is based on a simple premise: neighboring pixels in space-time that have similar intensities should have similar colors. We formalize this premise using a quadratic cost function and obtain an optimization problem that can be solved efficiently using standard techniques. In our approach an artist only needs to annotate the image with a few color scribbles, and the indicated colors are automatically propagated in both space and time to produce a fully colorized image or sequence. We demonstrate that high quality colorizations of stills and movie clips may be obtained from a relatively modest amount of user input.

O-expressions are a new syntax for Lisp-like languages, with an implementation called Liso which is based on Racket.

I’ve been playing with similar ideas recently, from a slightly more functional-programming perspective. Everyone agrees, for the most part, that delimiter pairs create structure, and that commas have higher precedence than semicolons. The idea that any consecutive sequence of non-delimiter punctuation characters constitutes an “operator” token is useful, and I like the use of INDENT/DEDENT as another form of bracketed delimiters. Is it possible, given the additional constraint of “no reserved identifiers”, to construct a language sufficiently general that a language like Python, Javascript, or (my own) Radian could be constructed on top of it as a macro library?

January 29, 2016

I’ve just discovered a language called Sisal, developed in the early ’80s, which was based on the same ideas that inspired Radian. It was quickly forgotten, as most languages are, but a group of students resurrected it in 2010 and built a fine-grained parallelism backend for it, based on pthreads. Which was, again, the whole point of Radian. It looks like they pulled it off about a year before I got Radian doing the same tricks, but the project seems to have lapsed back into obscurity.

January 4, 2016

I had a startling thought on the bus this morning which led to a pretty exciting idea. What if I combined the work I’ve been doing on lightweight unikernels with all the work I did on Radian, turning Radian into a tool for building lightweight services?

I still think the Radian language is a good one; it’s clean and neat, and its semantics are constrained in just such a way that it’s easy to interact with asynchronous processes, and thus to split work up among multiple workers. As an experienced developer, I could do a lot of productive work with that language – but I think it’d be just as easy for a newbie to pick up as would python or javascript.

As a new language, though, it suffers from the unavoidable defect of being new: there is no language community, there are no libraries, there are no example programs, there are no tutorials, there is barely any documentation. Not if I made it my full-time job and worked the kind of hours I put in when I was doing Object Basic could I make up for that lack myself.

As a tool, it also has significant limitations. The constraints of its immutable dataflow semantics make it very difficult to interact with code written in other, less-constrained languages. That makes it very difficult to interact with the operating system and all of its libraries. Its parallel execution model and its dataflow semantics make it very difficult to debug using a conventional debugger: the traditional mode of stepping through line by line and watching the variables change is basically impossible, because there is nothing in the compiled code which cleanly matches up with the original source lines or variable names.

Finally, and worst of all, I was never able to find a problem for which Radian was a uniquely powerful solution. I looked into datamining, and then into numerics, but in each case I got bogged down in the immense amount of work necessary to get the core libraries up to the point that someone not already dedicated to the language would care to use it.

So, the new idea.

Unikernel-based services are still quite new, and *lightweight* unikernel services are still quite a cutting-edge activity. I’ve been having a great time poking around in the gritty guts of the x86 architecture but that’s because it reminds me of coding in the wild west of the late ’80s and early ’90s. Quite a barrier to entry!

What if I turned radian into a dedicated tool for building standalone microservices? The output would not be an executable program, but a bootable disk image you could start up inside a VM.

This would neatly turn all of radian’s disadvantages into strengths – or inevitable constraints of the target environment, at least. Can’t interact with system libraries? Well, isolation is the whole point of a lightweight unikernel service. Can’t step through with a debugger? Who cares, it’s probably running in the cloud anyway, so you debug it via log files – which is pretty much exactly the way I used to debug Radian programs anyway. Can’t link your favorite library into Radian? Doesn’t matter, just boot it up in a separate microservice, then make them sling packets at each other!

Radian’s strengths, meanwhile, would remain strengths. It’s all about slinging data between parallel asynchronous processes, and what else are microservices? Lightweight tasks are built right into the language, as is a synchronization mechanism which would fit perfectly on top of a socket-based communication mechanism.

Furthermore, there’s basically nothing else out there that does this. The only comparable system is MirageOS, which basically wraps OCaml and a bunch of unix libraries into a bootable VM. And that’s great, but it’s OCaml, and it’s a *much* heavier solution than I’d be presenting.

Radian, under this vision, would become shell scripting for the cloud: letting anyone with the tiniest trace of programming knowledge go from idea to running microservice in a matter of seconds. I am undoubtedly feeling unreasonably excited about this idea right now, but if I could accomplish this, it seems like it might be the most valuable thing I ever did with my career.

So. What would I have to do in order to bring this idea into reality? The radian compiler wouldn’t need any changes; all the work would be on the runtime side. I would certainly have to write some better documentation, though. I’d then replace the fleet kernel’s C runtime with the Radian runtime and rework the radian allocator and threads accordingly. I’d also have to write a little bit of code for emitting a disk image, and for invoking qemu on it.

As far as the radian standard library goes, I’d scrap the fancy but unfinished bignums package and bang out something simple and reliable based on good ol’ int64 and float64. The major piece of new work I’d have to do would be the development of a socket interface. Since I’d be going for an all-in-one runtime library, though, I could scrap the clunky ffi workarounds that used to make this sort of thing so laborious, and just build it right in! Call straight into the kernel, why not.

At that point… we’ve got sockets, tools for working with data going in and out of sockets, tools for managing complex asynchronous processes involving multiple sockets, and a tool that turns source code into a running VM in a snap of the fingers. What’s next? Time to play.

This would be an insanely formidable task if I were thinking about starting it from scratch, but I’ve already done all the hard parts! What I’ve just described sounds more like… assembling pieces.

Wow. Could I actually make all those years of effort pay off? I think I might try it.

January 1, 2016

everything wrong with the 2015 web

The Website Obesity Crisis: starts off pointing out just how absurdly large websites have become, then continues – in remarkably straightforward, plain-spoken terms – to highlight everything I hate about modern web design. Such a satisfying read.

December 31, 2015

I’ve made significant progress toward the completion of a new, non-POSIX-oriented unikernel library in the form of a baremetal C99 runtime, but even as I’m cruising down the home stretch I think I will put the project aside. Sure, my unikernel library would be tighter and lighter than the rest, if I finished it, but it really doesn’t add all that much, and working on it means I haven’t been making any progress on the stuff that is truly unique to my vision.

IncludeOS is marketed as a cloud-service tool, but when I look at what they’re actually building, it looks very much like what I’m doing. Their project is C++, mine is C; their architecture is a bit more full-featured, where mine is deliberately minimal… but it’s still basically the same idea, and I generally agree with all of their architectural choices. So… why not join forces? I can always dust off my fleet-kernel later, if there’s some problem it would solve better; one of the great things about this VM/unikernel architecture is that you can have as many kernel architectures as you like running simultaneously.

The part I really need to build, in order to get my vision for fleet off the ground, is the router/shell system that will make unikernel-based programs useful as everyday workstation tools. I’ve grown increasingly antsy as development of my unikernel library has dragged on, preventing me from getting to the shell; it’s time to refocus and get on with it.

Maybe I can contribute some of my stuff to IncludeOS along the way, too.

October 24, 2015

A few solid hours hacking on fleet last night got me to a basic C “hello world”. The C library has everything from string.h and the simple character I/O functions from stdio.h, and the kernel has just enough of a driver interface to make simple file operations work. I’m using the legacy serial ports for now, since the drivers are trivial; stdin maps to COM1, stdout is COM2, and stderr writes to the host console via port E9.

This is the part where it starts to get really interesting. If you handwave past the part where most of the C library isn’t there yet, it’s now possible to compile a couple of ordinary C programs, link them against the fleet library, run them in VMs, and pipe data between them in grand old Unix shell style. It’s all very normal- except that these processes are just as comprehensively isolated from each other as if they were running on separate physical machines.

October 21, 2015

This fleet project is a lot of fun, combining a shiny new idea with an excuse to take a crack at a lot of classic problems.

The next layer after the startup code should be something to do with drivers and the low-level kernel apparatus, but it all felt a bit vague, so I decided to start with the C standard library interface and work my way down, letting that drive the rest of the kernel architecture.

There are dozens of free C library implementations available, but I have not been able to find one that will work for my project. I don’t want POSIX support, don’t need hardware portability, and won’t have a Unix-style system call interface underneath. And while I’m building this in the style of an embedded firmware project, it’s actually designed to run on a purely virtual machine, so I don’t need or want a lot of code dealing with legacy PC hardware.

Oh, well, I’m writing my own C library. Of course I’ll fill it in with a lot of existing components, but this architecture is apparently weird enough that the framework is up to me.

I did write the string library myself, though, because I thought it would be fun. There sure is a lot of weirdness in there – it’s been 23 years since I learned C, and I can’t say I had ever noticed the existence of strspn, strxfrm, or strcoll – but now I’ve written ’em and built test suites too.

October 19, 2015

I posted a thing: a piece of fleet called ‘startc’

I factored the lowest-level portion of the fleet code out as a standalone library which I’ve named ‘startc’ and posted on github. I also announced it on reddit.com/r/programming and on hackernews. Of course it feels a wee bit nerve-racking to post something experimental like this for the world to examine, but it’s a good exercise as it forces me to get all the loose ends tied up and to really think carefully about the interfaces between modules. So far the reception has been generally positive, which is nice. I have no idea whether anyone will actually use the library, but perhaps someone will get through the early stages of a similar project more quickly by looking at its source code, and that would make me feel good.

October 12, 2015

I spent four hours hanging a TV on the wall yesterday. Yes, really. I thought I’d simplify the project and save myself a bunch of work by purchasing a wall-mount swivel arm for the TV instead of building what I wanted from scratch.

As soon as I got started, it was clear that the wall-mount was designed to be mounted on a solid wood or brick wall (seriously? how many of those do you find in the USA?), so I started with a trip to the hardware store for a plank and some lag screws. After some careful measuring and a lot of exploratory drilling, I found the right spot and bolted the anchor panel firmly into the studs.

Next, I discovered that the wall-mount was a little bit too small for the TV. What!? I thought I measured it before I ordered it! Well… the wall mount listed a diagonal measurement range which includes the size of my TV, and its mounting bracket style is the same as that of the bracket I formerly used to attach the TV to the entertainment center, but it was designed for TVs with square bolt patterns and it just doesn’t spread out enough.

So… back to the hardware store, for another handful of bolts and some aluminum bars. I cut and drilled until I had a workable pair of adapter brackets.

Finally, I bolted the adapter brackets onto the TV, bolted the swivel-arm brackets onto the adapter brackets, screwed the swivel-arm brackets onto the arm head, bolted the swivel-arm base onto the anchor panel, which I’d previously bolted onto the wall.

Sure saved myself a lot of work there!

October 1, 2015

Text editing as a wire protocol

I spend a lot of my computer time editing text files, and so I’ve thought a lot about how one might go about that in a system like Fleet. One approach would pack all possible editing services into a single, monolithic IDE, which could run within a single VM. It would mount the disk containing the files you want to work on, present a file browser, and let you edit away to your heart’s content.

There’s nothing wrong with that approach, and it wouldn’t be hard to build out of existing components, but it doesn’t really satisfy my sense of elegance. I’d rather find a way to plug my editing tools together like Lego bricks.

It’d be really convenient, for example, to separate the code that renders text on screen from the code that manages all the data and performs the edits. Text can be displayed in lots of different ways depending on the context (code? email? notepad? letter writing?), but the process of editing a text buffer is the same. Wouldn’t it be neat if I could write the editing engine once and just slap a bunch of different interfaces on it depending on context?

The Fleet philosophy says that every connection between components has to take the form of a wire protocol, but what kind of wire protocol would represent a text editor? That really isn’t the sort of thing client/server processes typically do!

It occurred to me, however, that unix is full of command-line apps which accept commands typed in through a serial connection, producing output as text. There is an ancient program called ‘ed’, part of Unix since the 60s, whose user interface is basically a little line-oriented command language. What if we just redefined its interface as a wire protocol? A text-editing interface program would become a bridge, with one end connected to an “edit buffer service” and the other connected to a “terminal display service”.

This would allow multiplexing: one could have an arsenal of tiny, single-purpose editing tools which do their work by sending commands to an edit-buffer service. No need to keep reimplementing the edit buffer in every tool – just send some ed commands down the wire.

The `ed` program was designed to edit text files, but considering its command language as a wire protocol, what we’re looking at in the abstract is simply an array of text chunks. There’s no reason the actual bits on disk have to be nothing more than a flat text file: one could implement a different edit-buffer service for each different kind of file format, allowing one to mix and match editor interfaces and buffer services.

We can take it further. `ed` commands consist of a line reference, an identifying char, and optional parameters if the command needs them. What if we could extend the line reference syntax and used the same protocol to manipulate multidimensional data?

The syntax currently makes no use of the colon character ‘:’, so I suggest that the editor wire protocol could be extended by allowing a sequence of indexes delimited by colons:

Current: ‘X’\n

2D extension:
:‘X’\n

One could thus write a generic table viewer which would speak this protocol, then plug it into an edit-buffer service representing a CSV spreadsheet file or an NCSA server log file. And of course there’s no reason you couldn’t continue stacking dimensions arbitrarily if you wanted an edit service backed by JSON or some other hierarchical format.

It might be worthwhile to define a read-only subset of the protocol, since some tools will be content to view data, and it would be useful to develop buffer services which present a common interface for exploring structured data even if it’s not practical to perform edits.

System programming is fun: introducing FLEET

I couldn’t sleep the other night so I spent a few hours coding up the foundation of a kernel for this new exokernel-style operating system concept I’ve been talking about, which I’ve decided to call ‘fleet’. (Trindle was the microkernel idea, now dead.) It’s a lot of fun – it feels a lot like working on an embedded device, except the board just happens to have been designed by lunatics. I feel satisfied with my progress; the kernel boots the machine, configures memory and interrupts, spews log messages to the serial port, and enumerates the devices on the PCI bus.

Since I’m treating the PC as an embedded device dedicated to a single application, this “rump kernel” is really more like a new flavor of the C runtime library than a traditional kernel. I don’t have to worry about paging, memory protection, or user/supervisor mode switches, and most of the usual concurrency problems just disappear. An application which needed those services could link them in as libraries, but I’ll worry about that later.

Once upon a time, when the world was young and people were still trying to figure out what you could do with a computer network, people tried to build abstractions that would represent remote services as though they were local ones. “Remote procedure call” was the concept of the day, and this really took off in the early days of OOP: the idea was that you’d have local proxy objects which transparently communicated with remote ones, and you’d just call methods and get property values and everything would be shuttled back and forth automatically.

This just plain doesn’t work, because the semantics are totally different. You simply can’t make the fundamental constraints of concurrency, latency, and asynchrony disappear just by throwing a lot of threads around.

Modern interfaces are focused not on procedure calls, but on data blobs. Instead of making lots of granular, modal, stateful requests, machines communicate by serializing big blobs of data and streaming them back and forth at each other. This emphasizes bandwidth over latency, and focusing on large transactions rather than small interactions simplifies the problem of concurrent changes to remote state.

My plan is to take this idea out of the network and apply it inside a single PC. The rise of multicore computing has demonstrated that the traditional approaches don’t even scale within a single machine, once that machine is full of asynchronous processes competing for shared resources! In the ‘fleet’ world, rather than trying to represent remote resources with local proxies, we’ll represent local resources as though they were remote. There will be no DLLs and no system calls: the system API will be a folder full of wire protocol and data format specifications.

This solves the problem of network transparency from the opposite direction: since programs will already be communicating with local services through some network datastream interface, remote services will look exactly the same, except for the higher latency and lower reliability.

I believe that this approach will substantially improve the security picture, since the absence of any shared memory or common filesystem limits the damage a single program can do to the rest of the machine should it become compromised. Hypervisors seem to be holding up well in practice. Of course there’s nothing which would prevent a single ‘fleet’ process from spawning its own subprocesses and reintroducing all those concerns – the fleet shell would be perfectly happy to run linux as a subprocess, for that matter – but it’ll be easier to use the hypervisor interface and spawn “sub”-processes as independent virtual machines.

Requiring each program to include drivers for every possible hardware device would be madness, and slow madness since device emulation is tricky and expensive. These programs are never going to be run on bare metal anyway, so I’m going to ignore all legacy PC devices and define the ‘fleet’ system interface as consisting solely of virtio devices. These devices all have a simple, standardized IO interface, so it should be no problem to build drivers for six or eight of them into my kernel-library. I’ll offer an efficient low-level I/O API for nonblocking DMA transfers. All the clunky, synchronous, blocking C APIs can be implemented on top of that.

Looking at this system from above, it’s clear that making this fleet of VMs do useful work is going to involve a lot of datastream routing. I’m still working on the details, but I’m thinking that each program will have to include a compiled-in manifest describing the connections it wants to make and receive and the protocols it wants to use with them. Fixed connections like ‘stdin’, ‘stdout’ can be represented as serial ports, while other traffic can be specified using IP port numbers.

I have no idea how far I’ll get with all this, but I’m back in my old stomping grounds with all this low-level hackery and having a great time at it, so I’ll probably stick with it long enough to build a proof of concept. Something that boots into a shell where you can manipulate a filesystem and pipe data between programs, with a little monitor that lets you see what all the VMs are doing – that should be fun.

September 27, 2015

Deep Playa 2015

Well, that was a fun weekend, out in the trees near Sedro-Woolley. This was apparently the fourth year of the Deep Playa campout and it looked to be around 300 people this time. There were interesting art projects, fun activities, decent music, and overall a happy burnery festival vibe despite the cold damp weather.

AJ and I camped out in our big truck, as is becoming usual, and while it really needs a heater, at least it’s insulated and we had a generator powering the electric blanket. We also hung a big tarp off the side of the truck and made a shaded area where we could set up the propane camp fire – and lo there was much gathering and enjoying on Saturday night, as everyone was pretty much clustered up around one fire or another.

I brought the small version of my sound system and set it up by our camp, renegade-style. Two 15″ subs and two Mackie 450 tops – it was more sound than we needed, honestly, and I had a great time rocking the neighborhood with it. I played an electroswing set on Friday afternoon, and three psytrance sets at various other times when the mood struck me. I also got to play glitch-hop on the big main stage sound system Saturday night – it was a little challenging, perhaps due to the cold, but it went well anyway and I’m glad I did it.

Tomorrow it’ll be time to unpack; tonight I’m making an early night of it.

Next Page »