Shine: Your Mac Indie Business in a .zip

Last week I released SimCap, my first OS X application. This entailed — in addition to the whole 'designing and coding and testing' part — setting up the infrastructure necessary to sell it. Of course, I did all this at the very last minute, with SimCap sat there ready to go and the absolute minimum of advanced planning. That minimum planning was the decision to use Shine.

Shine is an application management dashboard, written by Tyler Hall and designed around the basic workflow of your average Mac Indie developer: taking payments via PayPal, generating licences with AquaticPrime, and pushing updates using Sparkle. I first heard about Shine on The MDN Show (epsiode 11) (which was — sorry, Scotty — the first episode I'd listened to in a long time). One of the things about iPhone development is that Apple does take care of most of the business side of things for you. While Shine doesn't exactly sit you down with a hot cuppa and tell you not to worry your pretty little head about all those nasty numbers, it certainly makes things easier, especially if you're starting out from scratch. Assemble components to fulfil the same roles Shine does would probably taken a few days.

I'm afraid that this won't be a complete Shine "how to" guide, but instead a collection of my thoughts on the process of setting the dashboard up, scribbled down while the experience is still fresh in my mind.

Firstly, did you notice that "especially if you're starting out from scratch" bit above? I mention it because Shine is rather inflexible in what it does. You can replace the AquaticPrime portion with your on licence generator, but otherwise the workflow is pretty-much locked down. Payment info comes in via PayPal's IPN service, so if you use a different payment processor you'll either need to write your own handler or look for another solution. Likewise, updates are stored in and downloaded from Amazon S3, without, say, the option to store them on your local server. In this last case, I decided that the ease of using Shine would probably outweigh the cost of using S3. (And I think I was right: this week's thousand downloads have cost me about 20 cents.)

Shine is basically a set of PHP scripts (available from the link above, either directly via Git or in the archive format of your choosing). Installation is just a matter of copying them to your server. The config file allows you to configure the exact same code for both your test and production servers. Documentation is minimal, basically amounting to the comments in the source code — but then this is aimed at Indie developers, who can probably be expected to feel comfortable diving into the code and figuring things out that way.

But just in case you aren't, or you've stumbled across this post looking for help... You build the MySQL database shine needs by running the .sql file from the archive using MySQL'a source command. You'll then have to create your own entry in the users table. There's an entry for using encrypted passwords in the config file, and if you set it, don't forget to also provide a string of random characters as salt. I ended up inserting random echo statements into the hashing function (on my test server) to see what value I should enter into the database as a password, but I'm almost certain there's a simpler way of doing it.

I can't personally say that installing Shine on my production server was a breeze, but none of that was Shine's fault. The code is neatly modular, which means that you may only encounter a problem with your server configuration the first time you go to use a particular feature. For instance, the first time I tried to test the PayPal IPN, I discovered that Shine needed libcurl, which I didn't have installed. (What followed was a couple of days of server re-imaging and mod_php re-compiling — but I'm sure that your mileage will vary. Especially if you have a grown-up looking after your servers for you.) As far as I can tell, the not-really-at-all-exotic-although-some-Linux-distros-seem-to-think-so modules you'll need are libcurl (for PayPal IPN), libjson (for everything(?)), and bcmath (for AquaticPrime — and this can only be added to PHP at compile time... sigh). A little pre-flight test script to check for these and warn of any potential nasty surprises might be an idea (and I guess if someone who knows their way around PHP doesn't write it then I'll have a go...).

Once you get this far, another little gotcha is that Shine will only process IPN messages with "Completed" status. Which makes perfect sense and is how you want it to behave, but can easily catch you out. If, like me, you're selling in a different currency to that of your bank account, you'll want to throw the "accept and convert" switch (it's under "Profile > Payment Receiving Preferences") to have transactions automatically completed. And while you're there, you'll also probably want to disable accepting eCheques, since transactions using them also won't complete straight away.

I've dwelt on the cons of using Shine because a post saying "use Shine, it's simply amazing" would be a little dull. But, honestly, I really can't recommend it enough. It has made the business side of launching an OS X application about as painless as you could hope. (And I haven't even mentioned it's integration with Tyler's OpenFeedback framework.) There are a couple of extra features which I'd like to see — some indication of the sample size used to calculate your Sparkle stats (did I mention Shine will also collect statistics from Sparkle?), for instance, and maybe auto-response and notification e-mails for OpenFeedback submissions — but otherwise Shine does everything it needs to so you don't have to. Get it and use it: it honestly is that much of a no-brainer.

Apple Does Love Me!

I'm sorry if I ever doubted you guys.

Is the App Store Broken?

You could say that these are Interesting Times to be an iPhone developer — although you could easily have said the same thing at any point since the announcement of the SDK. It seems that time and again we get weeks like this, where the number of "App Store Evil" stories in the press suddenly swells and it looks like matters are coming to a head. There have certainly been some happenings recently. The more public events you've doubtlessly heard of from Gruber, so I'll start with what's been going on behind the iPhone Dev Centre log in.

Trouble in Store

If you're a frequent browser of the App Store you may — or more likely, may not — have noticed something a little funny about the listings; in particular the "Recent" lists. Apple has introduced a new updates policy for the App Store. It used to be that whenever an update to an app was released it would be bumped back up to the top of the relevant list. This is no more. Now this list only shows newly released v1.0 applications. To say that this change is unpopular among developers would be an understatement.

For many developers, frequent small updates — and their accompanying bump in position with attendant spike in sales — was their main (or only) business plan. (I can certainly attest that the only times Lexical has seen any number of sales were after initial release and after the 1.1.) Cue much wailing, gnashing of teeth and cries of "sod this, I'm off to Android", interspersed with pragmatic — and somewhat exasperated — messages which basically boil down to: "Oh FFS, you can't rely on Apple to do all your marketing for you, why don't you try a little advertising, I hear it works wonders in every other single business ever."

As a side effect of these changes — or, more likely, of cracking the App Store open and tinkering with its workings — we get the messed-up listings. Apps released last year are suddenly reappearing at the top of lists, accompanied by other apps which won't be released until next year. On the plus side, you no longer have to manually adjust your release date once your app has been approved, but at this moment that seems like a very small comfort. It's been over a week now and things finally seem to be settling down. Presumably it's taken so long because those whose job it is to fix it — and who made the mistake in the first place — have been nailed up in the lobby of their building on Infinite Loop — alongside the rotting corpses of the old MobileMe team — as a warning to others.

(Playing armchair App Store manager for a moment, I'd suggest a two-track approval process for updates. When the developer submitted their update they'd check one of two options: "Bug fix" or "Feature Update". Bug fixes would be fast-tracked — let's say there's a semi-guarentee of under five days — but would not receive a list position bump. Feature Updates would be subject to the same fourteen-days-if-you're-lucky process as new apps — and as we get by default now — but would benefit from top-of-the-list exposure. But since I do not, have not and probably never will run a international application download service it really doesn't matter two hoots what I think should happen.)

What's in a Name?

Meanwhile, TUAW reports that a number of developers have been sticking the word "edge" into their app names as a protest against the actions of the owner of the trademark "Edge", who has been busy enforcing it — with Apple's help — in the App Store. I suppose it's good to see that, despite their world collapsing around them, iPhone developers can find the time to have a little silly fun. The word "edge" isn't the only one to be proscribed in this way. "Memory" is also now verbotten, although it seems on Jeff Lamarche has reported on this. So should we take this as another sign that the App Store is broken? No. In this case, Apple is responding to a legal request in the only way it can. Whether or not that request is fair is not their decision.

I'm Going Home and I'm Taking My Toys with Me

We've had a couple of high profile hissy-fits in recent days. I'll include that Facebook guy under "high profile", even though I'd never heard of him before. But we shouldn't be too surprised by his departure. If he works for Facebook then he's going to be rather limited in the type of apps he's gets to work on (hint: it will mostly be Facebook clients). The Facebook iPhone app has been around long enough that it probably does everything it needs to, so there's no point wasting your rock star developer on maintaining it. Might as well get yourself some free publicity with a spectacular walkout and then hand the day-to-day upkeep to some lacky.

And then we have Mike Ash and Rogue Amoeba. The Aitfoil Speakers Touch saga is being held up as the latest great failing of the review process and of how the rule reviews enforce are arbitrary, obscure and inconsistent. (Although at least no-one seems to be blaming this on "out-sourcing" and "off-shoring" any more. Those terms rank alongside "UK-only call centres" for making me want to stab people in the face.) But just for the sake of being contrary, I'm going to disagree. I think the — admittedly unspoken — rule is very clear: Apple is asking iPhone developers, "Please do not let users think that your application is our problem".

The Airfoil problem boils down to this: Rouge Amoeba want it to show (a) a Mac OS X application icon, and (b) an image of a Macintosh computer; Apple don't. Rogue Amoeba's first argument is that neither icon nor image are distributed with the app, but are instead accessed using public API calls on the host machine. Doubtless true, but still beside the point. If I try to release XXX iHardcore iPorn I shouldn't be surprised if it's rejected, no matter how loudly I complain that the content isn't included in the app but is downloaded using public APIs. A high proportion of the time, the icon which Airfoil displays will be from an Apple product. 100% of the time it will display one of Apple's images of a Mac. When an application does something 100% of the time arguments about how it does it are nothing more than weasel words.

The second argument is that the first version of Airfoil did exactly the same thing and was both accepted and remained available for the whole time. Which is where — and I'm just speculating wildly here, but then so are you, so shut up — the unspoken "Please do not let users think that your application is our problem" rule comes in. Maybe the person whose job it is to trawl through the support logs found an abnormal number of requests which went something like: "I'm having terrible problems with your iPhone application X and if you don't fix it right away I'm telling my lawyer" "But sir or madam, X is not an Apple application" "Oh no you don't fool me that easily, it has your a picture of [random Apple product] in it so it must be yours". Airfoil need not even have been the product mentioned. We already know that you cannot use images of the iPhone in iPhone apps. Images of any Apple product seems like a natural extension of this. And since we've all seen the App Store evolve over time, it's not an enormous leap to assume that this new tightening of the policy was instigated sometime between the release of the original Airfoil and the submission of the update.

The iPhone and iPod Touch are not Macs, and not only from a technical point of view. They are consumer electronic items in a way that computers never really have been. As such, they have a different user base (although admittedly with a lot of crossover). Let's try a little experiment. Find a non-Apple-devotee iPhone user. Take their iPhone and pull up the home screen. Now ask them to identify which of the apps listed there come from Apple and which don't. Product support costs money. Yes, Apple makes great margins, but they don't do it by paying their staff to explain time and again to customers why the problems caused by that app showing a picture of a Mac and the iTunes logo isn't actually anything to do with them. (After all, Apple is more than capable of causing its own bad publicity, thank you very much.)

Let me try a different metaphor. You may be familiar with the "advertising feature" which you occasionally find in magazines and newspapers: an advert formatted to look like editorial. Now, I'm willing to bet that your average journalist hates these more than an over-zealous expenses auditor. Because you may have the smartest readership in the world, but when they're curled up on the sofa with your publication, defences down, flicking away, there's a good chance that a least a few of them may be tricked into thinking that it's actually you or your colleagues who are recommending that they install Super-Dooper Virus Smasher 2012. I'm sure that Apple views apps that appropriate their content in exactly the same light.

And... I think that's more than enough for now.

Plaques + SimCap = A Busy Week

It's like the tired old joke about buses: I wait months (no, years) to release any applications, and then suddenly along come two at once.

First, on — I think — Monday, late, my second iPhone application Plaques, eventually limped into the App Store. Where it was promptly buried on page 4 by a slew (correct collective noun) of near-identical travel guides. Since then its sold a couple of copies but generally not done very well.

And then today we have SimCap, my first proper grown-up OS X app. It's a deceptively simple little application which does some eye-wateringly nasty stuff with window buffers and uses an NSOperationsQueue and everything. (In my spare time I've been trawling StackOverflow for questions about NSOperations being either delayed or cancelled, so I can chime in and suggest the poster checks for typos, because it sounds like they've accidentally sub-classed NHSOperation... this is actually a very funny Cocoa coder joke.) I'm really very happy with the way this has turned out.

So all in all a pretty busy week. I know, it's really amazing the lengths I'll go to to avoid my NaNoWriMo obligations.

Why the Hell am I Still Developing for the iPhone?

iPhone development — or in particular, the App Store — continues to appear to be an unpleasant place to be. First up there's the Tweetie 2 debacle (covered here by Dan Moren, who's one of my favourite Mac journalists, but who seems to care a little bit too much about currying favour with the Mac dev community). I've got nothing to add to the upgrade pricing fuss — if the developer can get people to pay then good luck to them — but I will just pull out this from Gruber:

I’ve been using it for a few weeks, and there’s a ton of new stuff (all of it copiously detailed by Brichter), but the persistence is the one that means the most to me. The effect is that you can leave Tweetie at any point, use another app, then go back to Tweetie, and it’s almost as though you never left. Feels like switching, rather than quitting/relaunching.

Seriously? Adding stored state — the main iPhone design pattern Apple pushes, and the single method we as developers have to compensate users for the lack of multitasking — finally adding this kind of stored state is heralded as the second coming, rather than at last making up for a major long-standing flaw? And yet AT&T cops shit for only now rolling out MMS.

And then we hear the sad story of how the App Store is broken because Iconfactory's Ramp Champ has not been selling well. Ignoring the fact that "not selling well" means the kinds of numbers I'd cheerfully give my left nut for, let's dig a little deeper and see what's up.

The lack of store front exposure combined with a sporadic 3G crashing bug conspired to keep Ramp Champ down for the count.

So here we have (2) they released a faulty product, but that isn't important because (1) Apple didn't give them front page billing. I mean, for fuck's sake, this is inexcusable. This is the fucking Iconfactory we're talking about. Don't you know who they are? Every time I select the wrong history link in Safari and get sent to the main iPhone dev page, there's the little blue Twitterific bird. Apple wants to start spending some of their advertising budget educating iPhone owners as to exactly which gods are developing for their phone. Ignoring them in this way is simply not on.

We also have this from "Talos" in the comments, which I think just about sums up most of the problems with the world today:

We (Iconfactory) spend at least 3840 hours (8 designers for 7 months) on art and design of Ramp Champ. DS Media Labs working on the programming side on and off for 7 months.

Coding as a afterthought. Yeah, what could possibly go wrong.

Flashforward

Kudos to Jack Davenport for being the only Brit actor working in American TV today able to resist the urge to do a silly accent.

A Quick Twitter Tip

Not aimed at anyone in particular: If you're going to tell me you're too busy to test the app I've been coding for you, you might like to think about not wiring your game achievements and YouTube favourites into your Twitter stream. M'kay?

Lexical: the App Store Control Group

I hope you're sitting down, boys and girls, because what I'm about to type will probably rock the foundation of your entire world: it turns out that the iPhone SDK and a few hours of your spare time are NOT a license to print money. I know, shocking isn't it. Sure, deep down we've all known that the good times have been over for a while — I personally realised it was finished when I saw an article about it in the Sunday Times Magazine — but this may still come as a shock to some people, such as my little friends here.

It's been just over a calendar month since Lexical, my first iPhone app, slipped quietly into the App Store. I wrote it in a few weeks after the project I'd been working on for the previous few months ran spectacularly into a brick wall. I really just wanted to prove to myself that I could actually finish and ship a product.

The chart above shows sales of Lexical for these first five weeks of its commercial life. Week one is particularly impressive (comparatively speaking, of course) given that Lexical appeared on a Thursday and so only represents four days of sales. Its 29 sales are more than the total for the following weeks (14, 4, 4 again and 2 copies, respectively) and simply go to re-underline the importance of the 'New Apps' lists in driving sales.

If you followed the link above, you would have probably drawn your own conclusions about why Lexical hasn't been selling very well. The app icon is dull (or as I like to think of it, elegantly simple), there is only a single line description, and the screenshot is particularly uninspiring (although to be fair that captures pretty well the game's visuals). Coupled with this goes the fact that I made no attempt at promoting it elsewhere.

So I guess the take-home is that, yes, App Store success and the riches and fame which go with it are still possible, it's just that, like everything else worth achieving in life, it now takes some serious commitment and effort to make it. Maybe I should have had the random tile-sorting code generate a swear word each time. Rejections are still great marketing tools. Maybe I should push out incremental update after update to keep it near the top of its category list. Whatever, I'm off to work on another couple of iPhone apps right now. Maybe the next one will sell a million copies.

Introducing Lexical

Lexical, my first iPhone application, has just been accepted into the App Store. (You can find it here.)

Lexical is an incredibly simple word puzzle: tap to swap letter tiles around, then swipe to highlight words. It was written in about 14 days (after my previous attempt at an iPhone app ran into an unexpected brick wall), and took about as long to get approved.

Actually, following the recent problems with App Store rejections, I was kind of hoping that I'd get the same treatment. From a publicity point of view it's solid marketing gold. Given that dictionaries are now being rejected for containing dirty words, I was thinking of having a few expletives 'accidentally' appear on each board. (Mind you, I don't do any checking when the board's randomly generated, so there's still a chance that will happen every now and again — probably just as someone's grandmother fires up their first game, knowing my luck...)

We'd only have to call it 'DAMP'

Since, in a fit of exuberant irony, my crash reporter clone is causing more crashes than it reports, I thought I'd give it a rest for a little while and instead look at getting some untested applications running instead. I thought that maybe having a working web app stack might be of some use to someone, so decided to look at the Apache-MySQL-PHP trio.

Binary roots are available on Mac OS Forge for Apache and MySQL, which I should have taken as a warning. They both installed without too much trouble: Apache needed expat, which was available but uninstalled, and MySQL just needed a little post-installation configuration, creating the default database and so on. So far, so good.

There was also a binary root of PHP4 available, but I thought, no, we'll go for the almost latest PHP5 in the apache_mod_php project. Of course, the absence of a binary root means that it wouldn't build for the boys at Apple, but then they don't usually try that hard.

Four hours later and I'm about ready to smash something. Trying to coax darwinbuild into building something it doesn't want to is one of the most frustrating experiences its ever been my misfortune to undertake. A custom build .plist got me over one set of problems — it was trying to build all four OS X architectures and threw a fit when it hit a library which had been stripped down to just i386 — but right now it's refusing to see a set of header files which are absolutely definitely there, and I have no idea what to try next, short of hacking the source code about, which I really didn't want to have to do.

Tomorrow I'll waste some more time seeing if I can get a tool chain running in Darwin itself. We then may be able to get non-Apple source packages to compile natively... although I'm not going to hold my breath.

I honestly can't wait until Snow Leopard comes out and we have to start again from scratch.

Nothing to (Crash) Report

I wish I could write the kind of informative technical blog post I've spent the past couple of days Googling for. Instead I seem to spend all of my time trudging along in other people's footsteps. And all of my open source efforts appear to be focused on re-inventing Apple's wheel.

Case in point: I'm currently trying to implement a crash reporter-like daemon for Darwin. A couple of launchd-spawned daemons expect to find it waiting for them, and scrawl their displeasure at its absence across the console, which is a pain as I've just got a proper getty-based login prompt up and running. Crash reporter is, of course, closed source, so enter my pale imitation. The power of cut-and-paste quickly got me the appropriate .plists, which shut the nagging daemons right up. Until one of them crashed, meaning I might as well make the fake crash reporter do something useful and report the crash.

So I'm now working on code which combines interacting with launchd — to retrieve the ports it's set up in advance for the daemon — and Mach exceptions — by which the actual crash is reported. If there are two less-well-documented parts of OS X/Darwin, well, I doubt anyone outside of Apple has heard of them. The launchd side of things I managed to get working using code from one of Darwin's launchd-using parts (I think it was xinit). Mach exceptions — and in particular, the difference between exc_server() and mach_exc_server(), and why one refuses to work while the other is hiding from the linker — however, are still a mystery to me.

I guess a trip to the darwin-dev mailing list on Monday is in order.

Source Code of the Day

From shutdown.c, part of Darwin's system_cmds project:

#ifdef __APPLE__

void log_and_exec_reboot_or_halt(void);

#else

void die_you_gravy_sucking_pig_dog(void);

#endif

Probably not the cause of our "WARNING: couldn't lock kext manager for reboot: (ipc/send) invalid destination port" problems, but still good to know.

More evidence that C++ is the work of the Devil

libstdc++ is the project providing Darwin's standard C++ library. (You can't criticise the naming, anyway.) The binary root (.tar.gz) is 36.7Mb. Decompressed, it's 196Mb. Thinned down to only its i386 version this falls to 176Mb. 5.6Mb of this is the libraries. Yes, that's almost 170Mb of headers and docs.

I've produced a special libstdcxx_libs root for use in the code Darwin installation I'm working on. You can thank me later.

Darwin Server

You probably don't want to read about today's debugging exploits — how I managed to track down a couple of un-implemented methods using the power of the printf(), how I finally twigged that I'd only need the two OS X frameworks to get gdb working in PureDarwinXmas, and how I eventually discovered that most of dscl's problems were caused by the memory PureFoundation was allocating for objects not being zeroed — so I though I'd just quickly tap out a few lines about where I see Darwin going.

(Of course, these won't include any of my super-secret plans. For those you'll have to wait a little while longer.)

I think you can guess from this post's title the direction I think that Darwin should be headed. It's not that I've got anything against it becoming a GUIed desktop system, it's just that I don't think there will be much demand for it. Remember that we already have a class-leading desktop OS built on top of Darwin. This leaves us in an awkward position: do we go down the Unix/Linux route, slap on the Gnome desktop and get lost in the crowd? Or do we try and emulate OS X — possibly by using code from Cocotron — with predictable sub-par results? Personally, I'm happy to let others decide. My interest lies elsewhere.

I think the idea of a version of Darwin which can be installed on (the way driver support is going, a narrow sub-set of) generic server PCs is a very compelling prospect, and may offer a feasible alternative to Linux in the web server space. A good number of web site designers and web app developers already use Macs as their primary machines. And while they run all the same software as your typical Linux machine (Apache, PHP, Ruby, Python, [insert your favourite web stack here]), I know from bitter experience that there are enough subtle gotchas to make deployment from OS X, at least on the first try — shall we say — interesting.

One question which is always hanging in the air in situations like this is "How does this mesh with Apple's plans?" — accompanied by the unspoken "And how quickly will they shut us down if they don't approve?" I won't go so far as to argue that a commoditised Darwin server would be good for Apple and therefore they should embrace it with open funding, but I will say that they shouldn't feel threatened by it. An open server built on Darwin is likely to appeal to sectors such as discount web hosting — the typical $9.99 per month virtual private servers on shared boxes type of arrangement — where XServes or even OS X Server under virtualisation were never going to be an option.

I think the PureDarwin boys are getting ready to put out another build, which will mark the next step on the road towards this bright future of Darwin on everything.

PureFoundation: Exceptions

(I've decided that this blog is going to become the unofficial PureFoundation development blog, in the hope that (a) it might attract a little interest — and who knows, maybe a contributor or two — to the project, and (b) I can get some of why I'm coding what and how I'm coding down before I forget about it.)

The last couple of days have been spent chasing down bugs in PureFoundation. The next couple of days will probably be spent in the same way. The number and stupidity of them is getting depressing, and the fact that I don't have access to a working version of gdb, or even a version of DTrace which will trace Objective-C method calls, doesn't help. Yes, I know that real men debug using printf()s, but this is getting ridiculous.

My current target is dscl. We've just got to the point where the DirectoryService daemon will run, so now we wouldn't mind using it to add a few users. (Yes, I know real men always run as root...) dscl is exactly the kind of project which spurred the creation of PureFoundation: it's both completely vital and written in Obj-C. Given the general paucity of admin tools available to us (and I'm really going to have to kick-start a Darwin Admin Tools project one of these days with the aim of cloning Apple's systemsetup and networksetup), I can't see PureDarwin getting anywhere without it.

My single aide was that I had access to the dscl source. Now, I honestly don't know if it compiles. I didn't try. darwinbuild has been enough of a frustration recently that I didn't even want to try. The moments of "why the hell am I wasting my time on this?" introspection are getting further apart again, but I'm still easily discouraged. If it had built I could have filled it full of more printf()s, which may have helped. Instead, all I could do was tag Foundation and see which library calls it made, following along as best I could with the source code. (My groovy new two monitor setup — using an old 15" CRT I found lying around — really helped here. If I'd had to do all this plus constantly shuffle windows about...)

The deep pessimism I've developed over the last 30+ years led me down a few blind alleys. Chiefly, the fact that dscl uses exceptions to propagate error conditions in exactly the way you aren't meant to, coupled with the fact I was sure logging into the local directory domain would fail, led me to believe that PureFoundation's exception handling was broken. Cue today's diversion.

Exception handling is one of those things which seems magical until you start digging into it. It's part language feature, part OS trickery. But it turns out that, at least in the default objc4 runtime version, it's fairly simple. Those @try and @catch blocks are converted into simple function calls by the compiler. These functions are defined in the file objc-exception.m, and basically just manipulates a linked-list of exception handlers and compiler-produced contexts. The comments on the code plainly state that this is place-holder code until Foundation starts up and provides its own, which is enlightening but ultimately unhelpful.

I've bored you enough without going into the details of the quick solution I hacked together. You can see for yourself by examining NSException.m. It uses thread-local storage to manage the chains of exception handlers, which I think is a neater and more elegant way of doing it, which also has the benefit of being thread-safe. Also added are both an ultimate default uncaught exception handler, and implementations of NSGetUncaughtExceptionHandler() and NSSetUncaughtExceptionHandler().

The punch line is that it turns out that no exceptions were being thrown by dscl, the local domain was being successfully accessed, and the first of many bus errors was in fact caused by my retarded double CFRelease()ing of a CFTimeZone type in my shiny new NSLog(). Which is why I really need an extra pair of eyeballs on the project. Any takers?

Extreme Open Source

I've been hacking (in the neckbeard sense of the word) Darwin for a couple of months solidly now, and have decided to christen the experience "Extreme Open Source". What's the difference between this and normal open source? It's the difference between "Sports" and "Extreme Sports". Basically, the difference can be summed-up as "why the hell would you want to do that when there are plenty of equally exciting but exponentially safer alternatives?" I think insanity plays a part.

It would be easy for me to type some stuff here about the bloody shocking state of the Darwin source code, but I've slowly come to the realisation that that isn't the problem. The source is, in fact, of quite excellent quality. The problem is it simply isn't designed to compile and run separately from the proprietary parts of OS X. And it's not just the Foundation-and-Objective-C using components, either. I've spent the past fortnight writing implementations of two function groups — CFNotificationCenter and CFFileDescriptor — which are absent from CFLite but relied upon by (the really rather vital) DirectoryService daemon.

darwinbuild — the chroot environment designed for compiling the separate projects — can be an exercise in frustration. When it works it's an absolute dream: packages are downloaded, dependencies met and code neatly assembled and arranged. But when it doesn't you're left pretty much helpless. I'm currently bashing my head (and not for the first time) against the fact that one particular tool (copystrings) is written in Ruby and yet there's no working Ruby install in the chroot and all my attempts to install one have failed. And without copystrings — which does exactly what its name suggests and so shouldn't be the most complicated application ever written — certain builds fail before they even really begin. In this case I'm trying to build a version of the DHCP-marshling IPConfiguration which doesn't rely on Apple's closed-source 80211 code. Working, pure-source networking is one broken script away.

So there you have Extreme Open Source: it's stressful, time consuming, and while you're working on it there's always this nagging suspicion at the back of your mind that as soon as you reach the finish line Apple's going to turn around and say, "Actually... I think we're going to close source all this stuff."

Foundation on Darwin (attempt #2)

Well that was... painless(ish).

Following the lead from GNUStep, I've had some success using a CFMutableDictionary to store retain counts keyed to object addresses. So far, so good. (GNUStep actually uses NSMapTables. These have only just been introduced in 10.5. I may move to them some day... but I need a working retain/release system before I can try to implement them.) Anyway, my NSObject subclassing test — the one described below — passed, so I'm happy.

Now on to CF-bridged class clusters. Looking into how to achieve these, all those NSCFStrings suddenly start to make sense.

Foundation on Darwin (attempt #1)

The PureDarwin project seems to be coming along in leaps and bounds, recently releasing the PureDarwin Xmas VMWare image. Since this gives me a working Darwin install to play with — and since I've had some spare time this week — I thought I'd take a look at what it would take to get a binary compatible Foundation working on Darwin (a project which I've tentatively entitled "PureFoundation"). By 'binary compatible' I mean compile on OS X and run on Darwin. Since Apple gives us the Objective-C 2.0 runtime, the AutoZone Garbage Collector, and CFLite, how hard could it be?

As I'm sure you're aware, many CoreFoundation (henceforth CF) objects are toll-free bridged to their Foundation counterparts. To most intents and purposes they are identical structures. Since CFLite implements reference counting (and the documentation and source code even hints that it plays nicely with the Garbage Collector) I hit upon what I thought would be a simple approach: rather than having a mix of pure objective-C and bridged classes, I'd create a CF class which I could bridge to NSObject, meaning that all the "PureFoundation" objects inherited CF's memory management.

Long(-ish) story short, I got this working. More or less. I could compile a simple Foundation.framework in XCode and copy it across to the Darwin VM; compile a simple Cocoa command line app, also in XCode, which created an NSObject, sent it -retain and -release, and reported it's retainCount. The same binary ran in an identical fashion under both OS X and Darwin. At this point I was about ready to call up the Nobel people and suggest they started a computing prize just for me. But...

CF objects are almost identical to objective-C structures. They're 8 (or 12 under 64-bit) byte structures, the first 4 (or 8) bytes of which are the isa pointer used to identify whether the object is an objective-C class and if so which one. The remaining 4 bytes holds other info, such as their CFTypeID and the lower 16 bits of the retain count. Since NSObject only allocates a single 4 (or 8) byte isa pointer, I had to pad the NSObject in my "PureFoundation" framework with another 4 dummy bytes, meaning that while the proper NSObject on OS X took up 4 bytes, my NSObject on (32-bit) Darwin took 8.

Now, I had (rather over optimistically) hoped that the dynamism of the objective-C runtime stretched to defining Classes and assigning ivars. So that if you defined a class which extended NSObject by adding extra variables, the runtime — rather than the compiler — would allocated them after whichever already existed. The runtime certainly provides functions for doing so. Unfortunately, this isn't the case. Testing an NSObject subclass with one extra NSUInteger ivar, compiled in XCode, showed it took up 8 bytes on both OS X (correctly) and Darwin (aargh!). Under "PureFoundation", retain and release worked fine immediately after the CF object was allocated, but once the -init ran the reference counting area of memory got stomped on.

So it's back to the drawing board. Next stop is the GNUStep source code, to see how they implement release/retain. The GNUStep source was always going to come into play at some point, so I guess this isn't too big a problem. It also looks like there's no getting away from patching CFLite to reinstate the bridging functions which were turned into no-ops by Apple before they released it. Which means setting up a darwinbuild environment and coding in Nano. Oh, well.