Source Code of the Day

From shutdown.c, part of Darwin's system_cmds project:

#ifdef __APPLE__

void log_and_exec_reboot_or_halt(void);

#else

void die_you_gravy_sucking_pig_dog(void);

#endif

Probably not the cause of our "WARNING: couldn't lock kext manager for reboot: (ipc/send) invalid destination port" problems, but still good to know.

More evidence that C++ is the work of the Devil

libstdc++ is the project providing Darwin's standard C++ library. (You can't criticise the naming, anyway.) The binary root (.tar.gz) is 36.7Mb. Decompressed, it's 196Mb. Thinned down to only its i386 version this falls to 176Mb. 5.6Mb of this is the libraries. Yes, that's almost 170Mb of headers and docs.

I've produced a special libstdcxx_libs root for use in the code Darwin installation I'm working on. You can thank me later.

Darwin Server

You probably don't want to read about today's debugging exploits — how I managed to track down a couple of un-implemented methods using the power of the printf(), how I finally twigged that I'd only need the two OS X frameworks to get gdb working in PureDarwinXmas, and how I eventually discovered that most of dscl's problems were caused by the memory PureFoundation was allocating for objects not being zeroed — so I though I'd just quickly tap out a few lines about where I see Darwin going.

(Of course, these won't include any of my super-secret plans. For those you'll have to wait a little while longer.)

I think you can guess from this post's title the direction I think that Darwin should be headed. It's not that I've got anything against it becoming a GUIed desktop system, it's just that I don't think there will be much demand for it. Remember that we already have a class-leading desktop OS built on top of Darwin. This leaves us in an awkward position: do we go down the Unix/Linux route, slap on the Gnome desktop and get lost in the crowd? Or do we try and emulate OS X — possibly by using code from Cocotron — with predictable sub-par results? Personally, I'm happy to let others decide. My interest lies elsewhere.

I think the idea of a version of Darwin which can be installed on (the way driver support is going, a narrow sub-set of) generic server PCs is a very compelling prospect, and may offer a feasible alternative to Linux in the web server space. A good number of web site designers and web app developers already use Macs as their primary machines. And while they run all the same software as your typical Linux machine (Apache, PHP, Ruby, Python, [insert your favourite web stack here]), I know from bitter experience that there are enough subtle gotchas to make deployment from OS X, at least on the first try — shall we say — interesting.

One question which is always hanging in the air in situations like this is "How does this mesh with Apple's plans?" — accompanied by the unspoken "And how quickly will they shut us down if they don't approve?" I won't go so far as to argue that a commoditised Darwin server would be good for Apple and therefore they should embrace it with open funding, but I will say that they shouldn't feel threatened by it. An open server built on Darwin is likely to appeal to sectors such as discount web hosting — the typical $9.99 per month virtual private servers on shared boxes type of arrangement — where XServes or even OS X Server under virtualisation were never going to be an option.

I think the PureDarwin boys are getting ready to put out another build, which will mark the next step on the road towards this bright future of Darwin on everything.

PureFoundation: Exceptions

(I've decided that this blog is going to become the unofficial PureFoundation development blog, in the hope that (a) it might attract a little interest — and who knows, maybe a contributor or two — to the project, and (b) I can get some of why I'm coding what and how I'm coding down before I forget about it.)

The last couple of days have been spent chasing down bugs in PureFoundation. The next couple of days will probably be spent in the same way. The number and stupidity of them is getting depressing, and the fact that I don't have access to a working version of gdb, or even a version of DTrace which will trace Objective-C method calls, doesn't help. Yes, I know that real men debug using printf()s, but this is getting ridiculous.

My current target is dscl. We've just got to the point where the DirectoryService daemon will run, so now we wouldn't mind using it to add a few users. (Yes, I know real men always run as root...) dscl is exactly the kind of project which spurred the creation of PureFoundation: it's both completely vital and written in Obj-C. Given the general paucity of admin tools available to us (and I'm really going to have to kick-start a Darwin Admin Tools project one of these days with the aim of cloning Apple's systemsetup and networksetup), I can't see PureDarwin getting anywhere without it.

My single aide was that I had access to the dscl source. Now, I honestly don't know if it compiles. I didn't try. darwinbuild has been enough of a frustration recently that I didn't even want to try. The moments of "why the hell am I wasting my time on this?" introspection are getting further apart again, but I'm still easily discouraged. If it had built I could have filled it full of more printf()s, which may have helped. Instead, all I could do was tag Foundation and see which library calls it made, following along as best I could with the source code. (My groovy new two monitor setup — using an old 15" CRT I found lying around — really helped here. If I'd had to do all this plus constantly shuffle windows about...)

The deep pessimism I've developed over the last 30+ years led me down a few blind alleys. Chiefly, the fact that dscl uses exceptions to propagate error conditions in exactly the way you aren't meant to, coupled with the fact I was sure logging into the local directory domain would fail, led me to believe that PureFoundation's exception handling was broken. Cue today's diversion.

Exception handling is one of those things which seems magical until you start digging into it. It's part language feature, part OS trickery. But it turns out that, at least in the default objc4 runtime version, it's fairly simple. Those @try and @catch blocks are converted into simple function calls by the compiler. These functions are defined in the file objc-exception.m, and basically just manipulates a linked-list of exception handlers and compiler-produced contexts. The comments on the code plainly state that this is place-holder code until Foundation starts up and provides its own, which is enlightening but ultimately unhelpful.

I've bored you enough without going into the details of the quick solution I hacked together. You can see for yourself by examining NSException.m. It uses thread-local storage to manage the chains of exception handlers, which I think is a neater and more elegant way of doing it, which also has the benefit of being thread-safe. Also added are both an ultimate default uncaught exception handler, and implementations of NSGetUncaughtExceptionHandler() and NSSetUncaughtExceptionHandler().

The punch line is that it turns out that no exceptions were being thrown by dscl, the local domain was being successfully accessed, and the first of many bus errors was in fact caused by my retarded double CFRelease()ing of a CFTimeZone type in my shiny new NSLog(). Which is why I really need an extra pair of eyeballs on the project. Any takers?

Extreme Open Source

I've been hacking (in the neckbeard sense of the word) Darwin for a couple of months solidly now, and have decided to christen the experience "Extreme Open Source". What's the difference between this and normal open source? It's the difference between "Sports" and "Extreme Sports". Basically, the difference can be summed-up as "why the hell would you want to do that when there are plenty of equally exciting but exponentially safer alternatives?" I think insanity plays a part.

It would be easy for me to type some stuff here about the bloody shocking state of the Darwin source code, but I've slowly come to the realisation that that isn't the problem. The source is, in fact, of quite excellent quality. The problem is it simply isn't designed to compile and run separately from the proprietary parts of OS X. And it's not just the Foundation-and-Objective-C using components, either. I've spent the past fortnight writing implementations of two function groups — CFNotificationCenter and CFFileDescriptor — which are absent from CFLite but relied upon by (the really rather vital) DirectoryService daemon.

darwinbuild — the chroot environment designed for compiling the separate projects — can be an exercise in frustration. When it works it's an absolute dream: packages are downloaded, dependencies met and code neatly assembled and arranged. But when it doesn't you're left pretty much helpless. I'm currently bashing my head (and not for the first time) against the fact that one particular tool (copystrings) is written in Ruby and yet there's no working Ruby install in the chroot and all my attempts to install one have failed. And without copystrings — which does exactly what its name suggests and so shouldn't be the most complicated application ever written — certain builds fail before they even really begin. In this case I'm trying to build a version of the DHCP-marshling IPConfiguration which doesn't rely on Apple's closed-source 80211 code. Working, pure-source networking is one broken script away.

So there you have Extreme Open Source: it's stressful, time consuming, and while you're working on it there's always this nagging suspicion at the back of your mind that as soon as you reach the finish line Apple's going to turn around and say, "Actually... I think we're going to close source all this stuff."

Foundation on Darwin (attempt #2)

Well that was... painless(ish).

Following the lead from GNUStep, I've had some success using a CFMutableDictionary to store retain counts keyed to object addresses. So far, so good. (GNUStep actually uses NSMapTables. These have only just been introduced in 10.5. I may move to them some day... but I need a working retain/release system before I can try to implement them.) Anyway, my NSObject subclassing test — the one described below — passed, so I'm happy.

Now on to CF-bridged class clusters. Looking into how to achieve these, all those NSCFStrings suddenly start to make sense.

Foundation on Darwin (attempt #1)

The PureDarwin project seems to be coming along in leaps and bounds, recently releasing the PureDarwin Xmas VMWare image. Since this gives me a working Darwin install to play with — and since I've had some spare time this week — I thought I'd take a look at what it would take to get a binary compatible Foundation working on Darwin (a project which I've tentatively entitled "PureFoundation"). By 'binary compatible' I mean compile on OS X and run on Darwin. Since Apple gives us the Objective-C 2.0 runtime, the AutoZone Garbage Collector, and CFLite, how hard could it be?

As I'm sure you're aware, many CoreFoundation (henceforth CF) objects are toll-free bridged to their Foundation counterparts. To most intents and purposes they are identical structures. Since CFLite implements reference counting (and the documentation and source code even hints that it plays nicely with the Garbage Collector) I hit upon what I thought would be a simple approach: rather than having a mix of pure objective-C and bridged classes, I'd create a CF class which I could bridge to NSObject, meaning that all the "PureFoundation" objects inherited CF's memory management.

Long(-ish) story short, I got this working. More or less. I could compile a simple Foundation.framework in XCode and copy it across to the Darwin VM; compile a simple Cocoa command line app, also in XCode, which created an NSObject, sent it -retain and -release, and reported it's retainCount. The same binary ran in an identical fashion under both OS X and Darwin. At this point I was about ready to call up the Nobel people and suggest they started a computing prize just for me. But...

CF objects are almost identical to objective-C structures. They're 8 (or 12 under 64-bit) byte structures, the first 4 (or 8) bytes of which are the isa pointer used to identify whether the object is an objective-C class and if so which one. The remaining 4 bytes holds other info, such as their CFTypeID and the lower 16 bits of the retain count. Since NSObject only allocates a single 4 (or 8) byte isa pointer, I had to pad the NSObject in my "PureFoundation" framework with another 4 dummy bytes, meaning that while the proper NSObject on OS X took up 4 bytes, my NSObject on (32-bit) Darwin took 8.

Now, I had (rather over optimistically) hoped that the dynamism of the objective-C runtime stretched to defining Classes and assigning ivars. So that if you defined a class which extended NSObject by adding extra variables, the runtime — rather than the compiler — would allocated them after whichever already existed. The runtime certainly provides functions for doing so. Unfortunately, this isn't the case. Testing an NSObject subclass with one extra NSUInteger ivar, compiled in XCode, showed it took up 8 bytes on both OS X (correctly) and Darwin (aargh!). Under "PureFoundation", retain and release worked fine immediately after the CF object was allocated, but once the -init ran the reference counting area of memory got stomped on.

So it's back to the drawing board. Next stop is the GNUStep source code, to see how they implement release/retain. The GNUStep source was always going to come into play at some point, so I guess this isn't too big a problem. It also looks like there's no getting away from patching CFLite to reinstate the bridging functions which were turned into no-ops by Apple before they released it. Which means setting up a darwinbuild environment and coding in Nano. Oh, well.

This Sark Business

I don't usually let myself get wound up by stuff I read about in the news, but this Sark business has me pretty mad. If you've never heard of Sark, go read about it in Wikipedia. The island and its (former) way of government is living proof that the UK is old. This country wasn't created, but instead evolved over millennia, along the way accreting administrative cruft like the Chief Pleas. Titles like Seigneur and Seneschal are beautifully evocative, and it really didn't seem like the pseudo-feudal system was hurting the island's 600 inhabitants.

The hypocrisy of current events is simply stunning. The Barclay Brothers may claim to want democracy, but in reality they only wanted to place themselves as feudal lords over the island. When their nominees lost the popular vote, they chose to shutter all their concerns on the island in a fit of pique. I'd call it childish if it wasn't so far beyond that. It's downright criminal. The islanders wouldn't let the Barclays take over — probably with the intention of installing a new tax regime more favourable to themselves — and so they are being punished. Notice in the BBC article where the Barclay's spokesman says that they have no need to sell the properties. Rather than risking someone else moving in and giving these people their jobs back, they will board-up the hotels and other businesses and let them rot in order to spite the islanders.

I live just across the Channel from Sark, and I've got half a mind to grab a boat from the harbour and stage my own one-man invasion of Brecqhou — the Barclays' Bond-villan-esuqe island fortress — to see how they'd like it.

I'm Looking for a Performer

I'm looking for a performer. A comedian or actor. Although not actively. Not quite yet. Just passively. Keeping my eyes peeled. I mention it here just on the off chance someone stumbles across it.

For some reason, I've found myself watching a lot of Daily Show and Colbert Report recently. Probably the US elections. The formula's pretty simple: five minutes of jokes, then a (usually toe-curling) interview. The BBC's Late Edition was a fairly accurate copy, albeit way more hit-and-miss. And now I'd like to try my own on the internet.

Of course, I wouldn't dream of sticking myself in front of the camera. Apart from always feeling uncomfortable, I simple haven't got the talent. Mind you, I see myself writing the thing and I certainly don't appear to have the talent for that. I've started another blog at hereisthenews.tumblr.com with the intention of posting short snappy jokes. So far, nothing. But give it time.

Anyway, this post is really more for me, to try and spur me into action. Although if anyone else reads this and is interested, drop me a message.

Sarah Palin Prank Called

Apparently it was Russell Brand claiming responsibility for Bristol's sprog.