Ragged Clown

It's just a shadow you're seeing that he's chasing…


My life in 100 words or less

Footscray. Council estate. Fancy grammar school. Parents divorce. First in class. Run away to sea. Engineering apprenticeship. Girlfriend. Falklands patrol. Girlfriend. Sonar engineer. Polaris submarine in Scotland.  Patrol North Atlantic. Midshipman. BRNC Dartmouth. Quit. Backpack (California, Tahiti, Moorea, Bora Bora, Huahine, Raitaia, hitchhike Sydney->Cairns->Darwin, Bali, Lombok, Sumbawa, Flores, Komodo, Singapore, Malasia, Thailand). East End. Poor. Girlfriend. Read The C Programming Language. City of London. Girlfriend dies. Malta. Girlfriend. Marry in Jamaica. Malta (again). Plymouth Hoe. Manhattan. Child. Wall Street. Silicon Valley. Startup (failed). Child. Startup (failed). Startup (failed). Portland (and back). Startup. Dog. Now.

Future. Success. Riches. South of France. Happy ever after. The end.


In Which Sturgeon’s Law Explains Everything

In 1951 Theodore Sturgeon was giving a talk about science fiction when someone in the audience noted that “90% of science fiction writing was crap”. Sturgeon shot back that “90% of everything is crap”.

This observation came to be known as Sturgeon’s Law.

Using the same standards that categorize 90% of science fiction as trash, crud, or crap, it can be argued that 90% of film, literature, consumer goods, etc. are crap. In other words, the claim (or fact) that 90% of science fiction is crap is ultimately uninformative, because science fiction conforms to the same trends of quality as all other artforms.

– Theodore Sturgeon

Sturgeon’s law is just as valuable when you are thinking about professionals as the old joke about doctors illustrates.

Q: What do you call a doctor who graduates at the bottom of his class?

A: Doctor

After years of pondering, I have come to believe that Sturgeon’s Law is more about the 10% that isn’t crap than the 90% that is. Forgetting the exact ratios for a moment, in every artform and every sporting contest, in every profession and every human endeavour, there is a distribution of quality in which a minority stands out as much better than the rest. We all know teachers who work that little bit harder to make their lessons interesting or the pharmacist who goes the extra mile to make sure that you understand your prescription. In my own profession—software engineering—only about 10% of engineers ever read a book about their craft once they leave college. This leads me to what I think is a more profound corollary to Sturgeon’s Law.

If you are a software engineer at the top of your game, you probably read books, attend conferences, study relentlessly to enhance your skills and engage in endless discussion on how to improve the state of the art of software craftsmanship.  Average software engineers do not do this but you are probably surrounded by the rare few who seek to be the best they can be. This is where the less obvious aspect of Sturgeon’s Law casts its insidious spell. When you meet people outside of your elite circle, they are more likely to be average than elite.

Most engineering managers have a similar disinclination to better themselves. The best of them are very good but most of them are not the best. Too many are merely average. When the very best of the software engineering profession looks at the competence of software engineering management, they sigh a little because most managers are not very good.

Oddly enough, the very best of software engineering managers have a symmetrical view of the engineers they manage. Most engineers are not very good either. This isn’t just true of software engineers and their managers; it’s true of QA engineers, product managers, and designers too. Most QA engineers think that most programmers suck at testing. It’s true. Most of them do. But most QA engineers suck at it too. To the best QA engineers, the average programmers seem like uncaring barbarians. And vice versa.

I should emphasize at this point that I am not suggesting that some people are better than others at everything. Even very great software engineers might suck at gardening or astronomy.

This corollary to Sturgeon’s Law pertains across so many domains where the best of folks surround themselves with other folks who are motivated to excel. The average person outside their circle contrasts poorly as a result. Consider a Kalenjin long distance runner from the Rift Valley in Kenya. He probably encounters other great runners almost every day of his life. Most of the people he runs with, eats with and loves with are also great runners. If our Kalenjin got on a plane to Helsinki, he’d probably be disappointed to find that most Finns are not great long-distance runners. Most of them are just average. Truth be told, most of the Kalenjin are probably average too but our guy doesn’t encounter them very often. He hangs with the elite when he is at home. Away from home he has less opportunity to be so picky. It was only when he went to Finland that he encountered so many people who were not elite runners. Finland has elite runners too of course (I fondly recall cheering for Lasse Virén at the Montreal Olympics) but you are not going to bump into them at the airport as you might when you are at the Kenyan School for Elite Runners.

It’s helpful at this point to remember that individuals are not statistics (The median is not the message in the memorable words of Stephen Gould). If you wanted to recruit a team of long-distance runners it wouldn’t be a great idea to fly off to the Rift Valley and round up the first 5 runners you came across. You’d probably have a very average running team. You’d be much better off choosing great runners wherever they hail from. Furthermore, if you had to choose between a Finnish runner and a Kenyan runner, there is no need to check their birth certificate or the colour of their skin when you decide which one should join your elite running team. You can actually race them and choose the one that runs the fastest. When you are dealing with individuals, you should concern yourself with their individual qualities, not with some arbitrary statistical correlation however accurate that may be. It might be true that the average Kenyan runs faster than the average Finn but—so what? You will never be in a situation where you have to choose between average individuals without some other evidence to inform your choice.

The insidious nature of statistical racism is magnified by confirmation bias. Once you have decided that Kenyans run faster than Finns, it’s all too easy to reinforce your prejudice by noticing all the data points that confirm your bias—Hey! There goes another fat, slow and lazy Finn!—and to overlook the occasions that contradict your instincts. There is a long, unfortunate history of people doing exactly this.

There’s a whole garden of isms that wither under Sturgeon’s steely gaze. My dentist (Hi, Dr Bobba!) has a lovely cartoon on the wall with a caption that says something like “Women will never make good dentists. Their wrists are too weak!”.

Put yourself in the perspective of a Victorian gentleman who just happens to be a very good dentist. All your friends are very good dentists. They all have a certain background in dentistry and very strong wrists. You probably have a quite distinct image of what a proper dentist looks like. The average woman of your acquaintance probably seems very un-dentist-like. She is probably very uninterested in dentistry and has very dainty wrists. If you had to choose a dentist based only on wrist strength, you’d be marginally better off choosing the dentist with the stronger wrists—in 1875. But in 2014, you can skip right past concerns about wrist strength and whether your dentist has the appropriate genitalia and just hire the one who is the best at dentistry. And Doctor Bobba *is* very good. Trust me on that.

More casually—in our everyday lives—we are surrounded at work by people who share a certain intellectual outlook on life. Maybe your colleagues are more interested in politics than the average citizen. Maybe your friends at the sports bar care an awful lot about the intricacies of the infield fly rule or exactly how many defenders need to be behind the ball before offside is called. They know more than the average Joe about sports and certainly more than the average wife. Does that mean that women don’t understand politics or sports? No, of course not. The median is not the message, remember? It means that the average wife—in fact, the average anyone—knows less about sports than the fanatics you hang out with at the sports bar.

Let’s try some more examples.

The average tourist who visits Paris from their friendly little town in Georgia will find most Parisians quite distant, abrupt and possibly rude. The literary Parisians that he encounters will surely conclude that tourists from Georgia know very little about French art and are quite uncultured.  If you repeat the experiment in the opposite direction and send a farmer to Atlanta from a little village in Provence I’ll wager the outcome will be identical. Repeat as necessary with Beijing, Nairobi, Melbourne and Rio.

The average kid who spent every evening of the 1970s browsing record stores for rare blues recordings is likely to be disappointed with the crap his kids listen to on The YouTube. And vice versa.

If you are really interested in US history, I bet you are disappointed with how little the kids of today know about your favourite topic. Guess what! They are disappointed in you too!

The average software developer who does not have a degree in computer science probably doesn’t know much about data structures and algorithms. Neither does the average CS graduate. Most of them slept through that class or forgot most of it the next day. More surprisingly, the average PhD is not very good at software engineering either unless they are working in their very narrow field of expertise. Of the best engineers I have ever worked with, only a few had a PhD or a Master’s degree in CS. Some had degrees in English or music and a good number had no degree at all. In fact, it’s quite amazing that many of the most famous people in software dropped out of college—or maybe that’s just my own confirmation bias playing tricks on me.

I expect the world would be a much happier place if people listened to their Uncle Sturgeon and relied on statistics and biases only when they prove useful. A statistical overview of a population can be helpful when you are deciding how to profitably market your new product or where to spend your campaign dollars or which college recruiting fair to attend. But if you are choosing an umpire for your baseball league or an anchor for your running team or a new hire for your software startup you’d do better to ignore the statistics and hire the individuals with the right skills for the job. To do otherwise is prejudice.


Songs Without Love

I was wondering what songs are about. Most of them are about love of course but what about the other ones?

Terry Gross had a dude on the other day who wrote songs about a bomb that went off his train carriage on the way to Machu Pichu. Abba won the 1974 Eurovision Song Context with a song about Waterloo. Are there any other songs about weird topics?

I haven’t done a fun science project for a while and I need to learn about the latest versions of Ruby & Rails & Elastic Search & D3 & Hicharts. I also want to dabble in some NLP stuff—sentiment analysis; classifiers; that kind of thing.

Here’s the TODO list:

* Grab every top 100 song since Al Martino in 1952.
* Grab all the lyrics to all the songs.
* Build a word cloud for each song.
* Build a word cloud for each week/year/decade.
* Do cluster analysis to find interesting topics.
* Write a classifier that can figure out what each song is about (love, war, bombs, whatever).
* Plot how that changes over time.
* Do sentiment analysis to see if songs are happy or sad.
* Plot how that changes over time.

The results will be here:


If you know Ruby (or want an excuse to learn) fork my repo and play along.


I need to get this finished by next year. The year after that at the very latest.


Moral Instincts Don’t Scale

Excellent idea that I never thought of before…

The first half of this bloggingheads video is about Paul Bloom experiments showing how very young babies have a well-developed moral sense. In the second half of the video, Paul & Bob discuss how well our moral instincts and emotions like justice, anger and empathy work in our everyday lives but how badly they scale up to interactions between nations.


No Apologies

I’ve developed a habit in recent years of getting so excited after reading the first few chapters of a book that I want to write a provisional review before I even really know what it’s about.

Two chapters into Unapolagetic, I was shouting yes! yes! yes! and couldn’t wait to capture my agreement in digital ink. I even, for the first time ever, used the highlight feature on my kindle so that I could accurately report all the many, many points that resonated with my own experience.

On one point in particular Spufford was singing my life with his words. When he described that transcendental moment when I meditate on the immensity of eternity – Spufford calls it ‘praying’ but, whatever – and I experience a little shimmer in the corner of my mind’s eye and suddenly everything makes sense and the whole universe lines up for my inspection and appreciation….  I didn’t know that other people experienced it too. I thought that was just me and I had no idea that some people called that shimmer ‘God’.

Seeing my secret thoughts in pixels, I was tremendously excited to document my assent to Spufford’s project which, as I understood it from tentatively supportive reviews by a couple of Christian bloggers that I follow, was to describe Christianity’s benefits in words that a non-Christian could identify with. I could get behind that project.

Real-life intervened though and the reviewable moment was lost. An emergency dash down to Santa Barbara later and the blogging window slammed shut. As I ploughed on through the next chapter on theodicy all my admiration fell away.

Spufford concedes that the problem of evil  is not a problem for atheists. It’s only a problem if you believe in one very specific kind of God: that is, a God who loves the world and is in full control of all the outcomes. There is no problem of evil if your God doesn’t care about children with cancer. Without love, there is no problem to be solved.

Likewise, there’s no problem with a God who cares an awful lot about you but is powerless to do much about it. That God would be lame and not much worthy of all the cathedrals that get built for him and he’d create all kinds of problems for his marketing department but he wouldn’t create any particular challenges for philosophers.

Since the God that Spufford worships is allegedly both omnipotent and benevolent, the problem of evil is a big problem indeed and Spufford absolutely demolishes all the usual excuses for why God allows evil to happen to good people. Let’s just say that if Spufford were in God’s marketing department, God would’ve fired him by now. Spufford works in mysterious ways.

I just went back to my Kindle to read over all the sections that I highlighted back when I thought  this was a great book. Spufford really is an astoundingly good writer. He wasn’t just singing my life; he was strumming my secret pain in ways that make me wonder if he isn’t in league with the NSA. I really want his project to succeed but chapters three and four make me despair for the whole liberal Christian project, cuddly new popes notwithstanding.

I’ll report back if I learn anything new.



The Priest with the Cold Hands

When Rita was in intensive care there was a moment where it seemed like it was all over and her mother told me to go find the priest.

I went down to the chapel and there was an old, old priest in there. As old as Methuselah. I was frantic and I couldn’t get the words out.

That old priest held both of my hands in his and the words he spoke were like the wisdom of the ages. They slowed the whole world down and we all made it through another day.


It’s all good

lesmisI saw Les Misérables on stage with my wife-to-be on one of our first dates and read the book soon after. I watched it again last night with my daughter after a twenty-something-year gap and I think I spotted something new about the story that I have not heard mentioned before. I even went to check the SparkNotes to see if this was some obvious theme that only I was missing but… nope. No one else noticed it but me.

When anyone mentions Les Mis, there are a few themes that immediately pop into your mind: it’s very long and tedious; it’s sentimental kitsch; the songs are awful; the story is an entirely predictable tale of redemption for Jean Valjean; the scene on the barricades is totally unnecessary and out of place.

All those things are true but I think there is something quite profound about how the characters interact with each other. All of the characters (*with one possible exception below!) are very moral according to their own particular version of morality. Of course, since it’s a liberal play/book/movie, all the ‘good guys’ subscribe to liberal conceptions of morality: everyone deserves a chance; kindness will be repaid; poor people commit crimes out of necessity.

But if that was all there was to the story, it would be a very shallow fable about a very liberal morality — we all get to cry for Fontine and boo at Javert — but I suspect that there might be more to it. We have all been missing something quite shocking.

Everyone in Les Mis behaves quite morally—according to their own understanding of right and wrong. Even the baddies.

Jean Valjean gets off to a bad start but learns the power of love and earns his redemption many times over.

The priest believes that everyone deserves a chance and gives Valjean his at great cost to himself.

Fantine gives up every sliver of self-respect to keep her daughter alive.

Javert believes in the Rule of Law above all and it matters not at all to paladin-types like Javert that Valjean has done good in his life. Valjean has broken the law and must be punished. Javert is willing to devote his whole life to bringing Valjean to justice.

The anarchists on the barricades believe the government is corrupt and should be overthrown. The soldiers who gun them down have the opposite view.

The slut-shaming women that get Fantine fired from her job believe that sexual impropriety and non-traditional families are Very Very Bad and should be punished. The Slut-shamers’ beliefs are probably shared by a majority of People Who Consider Themselves to be Morally Upstanding.

The photo that got Ashley Payne fired from her teaching job.

The photo that got Ashley Payne fired from her teaching job.

The foreman who actually sets Fantine on the road to self-destruction is following an ethical code that has held sway for most of human history and has only recently begun to retreat from the mainstream. In the glorious 50s that conservatives love to reminisce over,  most young women would have been suffered the same fate as Fantine. Even today, young teachers can be fired for inappropriate Facebook photos.

The only exception to the ‘everyone is moral’ rule is, of course, The Master of the House but I bet, if you asked him, even he would say that all those other conceptions of morality were invalid and that the only proper behaviour is to look out for your own interests.

[* OK. That last one is a stretch. The Thénardiers are quite evil]

It’s quite something that all these conceptions of morality still have their champions 150 years after Hugo wrote it all down for us. Will we ever figure out which one is correct?