Ragged Clown

It's just a shadow you're seeing that he's chasing…


May
25
2014

In Which Sturgeon’s Law Explains Everything

In 1951 Theodore Sturgeon was giving a talk about science fiction when someone in the audience noted that “90% of science fiction writing was crap”. Sturgeon shot back that “90% of everything is crap”.

This observation came to be known as Sturgeon’s Law.

Using the same standards that categorize 90% of science fiction as trash, crud, or crap, it can be argued that 90% of film, literature, consumer goods, etc. are crap. In other words, the claim (or fact) that 90% of science fiction is crap is ultimately uninformative, because science fiction conforms to the same trends of quality as all other artforms.

– Theodore Sturgeon

Sturgeon’s law is just as valuable when you are thinking about professionals as the old joke about doctors illustrates.

Q: What do you call a doctor who graduates at the bottom of his class?

A: Doctor

After years of pondering, I have come to believe that Sturgeon’s Law is more about the 10% that isn’t crap than the 90% that is. Forgetting the exact ratios for a moment, in every artform and every sporting contest, in every profession and every human endeavour, there is a distribution of quality in which a minority stands out as much better than the rest. We all know teachers who work that little bit harder to make their lessons interesting or the pharmacist who goes the extra mile to make sure that you understand your prescription. In my own profession—software engineering—only about 10% of engineers ever read a book about their craft once they leave college. This leads me to what I think is a more profound corollary to Sturgeon’s Law.

If you are a software engineer at the top of your game, you probably read books, attend conferences, study relentlessly to enhance your skills and engage in endless discussion on how to improve the state of the art of software craftsmanship.  Average software engineers do not do this but you are probably surrounded by the rare few who seek to be the best they can be. This is where the less obvious aspect of Sturgeon’s Law casts its insidious spell. When you meet people outside of your elite circle, they are more likely to be average than elite.

Most engineering managers have a similar disinclination to better themselves. The best of them are very good but most of them are not the best. Too many are merely average. When the very best of the software engineering profession looks at the competence of software engineering management, they sigh a little because most managers are not very good.

Oddly enough, the very best of software engineering managers have a symmetrical view of the engineers they manage. Most engineers are not very good either. This isn’t just true of software engineers and their managers; it’s true of QA engineers, product managers, and designers too. Most QA engineers think that most programmers suck at testing. It’s true. Most of them do. But most QA engineers suck at it too. To the best QA engineers, the average programmers seem like uncaring barbarians. And vice versa.

I should emphasize at this point that I am not suggesting that some people are better than others at everything. Even very great software engineers might suck at gardening or astronomy.

This corollary to Sturgeon’s Law pertains across so many domains where the best of folks surround themselves with other folks who are motivated to excel. The average person outside their circle contrasts poorly as a result. Consider a Kalenjin long distance runner from the Rift Valley in Kenya. He probably encounters other great runners almost every day of his life. Most of the people he runs with, eats with and loves with are also great runners. If our Kalenjin got on a plane to Helsinki, he’d probably be disappointed to find that most Finns are not great long-distance runners. Most of them are just average. Truth be told, most of the Kalenjin are probably average too but our guy doesn’t encounter them very often. He hangs with the elite when he is at home. Away from home he has less opportunity to be so picky. It was only when he went to Finland that he encountered so many people who were not elite runners. Finland has elite runners too of course (I fondly recall cheering for Lasse Virén at the Montreal Olympics) but you are not going to bump into them at the airport as you might when you are at the Kenyan School for Elite Runners.

It’s helpful at this point to remember that individuals are not statistics (The median is not the message in the memorable words of Stephen Gould). If you wanted to recruit a team of long-distance runners it wouldn’t be a great idea to fly off to the Rift Valley and round up the first 5 runners you came across. You’d probably have a very average running team. You’d be much better off choosing great runners wherever they hail from. Furthermore, if you had to choose between a Finnish runner and a Kenyan runner, there is no need to check their birth certificate or the colour of their skin when you decide which one should join your elite running team. You can actually race them and choose the one that runs the fastest. When you are dealing with individuals, you should concern yourself with their individual qualities, not with some arbitrary statistical correlation however accurate that may be. It might be true that the average Kenyan runs faster than the average Finn but—so what? You will never be in a situation where you have to choose between average individuals without some other evidence to inform your choice.

The insidious nature of statistical racism is magnified by confirmation bias. Once you have decided that Kenyans run faster than Finns, it’s all too easy to reinforce your prejudice by noticing all the data points that confirm your bias—Hey! There goes another fat, slow and lazy Finn!—and to overlook the occasions that contradict your instincts. There is a long, unfortunate history of people doing exactly this.

There’s a whole garden of isms that wither under Sturgeon’s steely gaze. My dentist (Hi, Dr Bobba!) has a lovely cartoon on the wall with a caption that says something like “Women will never make good dentists. Their wrists are too weak!”.

Put yourself in the perspective of a Victorian gentleman who just happens to be a very good dentist. All your friends are very good dentists. They all have a certain background in dentistry and very strong wrists. You probably have a quite distinct image of what a proper dentist looks like. The average woman of your acquaintance probably seems very un-dentist-like. She is probably very uninterested in dentistry and has very dainty wrists. If you had to choose a dentist based only on wrist strength, you’d be marginally better off choosing the dentist with the stronger wrists—in 1875. But in 2014, you can skip right past concerns about wrist strength and whether your dentist has the appropriate genitalia and just hire the one who is the best at dentistry. And Doctor Bobba *is* very good. Trust me on that.

More casually—in our everyday lives—we are surrounded at work by people who share a certain intellectual outlook on life. Maybe your colleagues are more interested in politics than the average citizen. Maybe your friends at the sports bar care an awful lot about the intricacies of the infield fly rule or exactly how many defenders need to be behind the ball before offside is called. They know more than the average Joe about sports and certainly more than the average wife. Does that mean that women don’t understand politics or sports? No, of course not. The median is not the message, remember? It means that the average wife—in fact, the average anyone—knows less about sports than the fanatics you hang out with at the sports bar.

Let’s try some more examples.

The average tourist who visits Paris from their friendly little town in Georgia will find most Parisians quite distant, abrupt and possibly rude. The literary Parisians that he encounters will surely conclude that tourists from Georgia know very little about French art and are quite uncultured.  If you repeat the experiment in the opposite direction and send a farmer to Atlanta from a little village in Provence I’ll wager the outcome will be identical. Repeat as necessary with Beijing, Nairobi, Melbourne and Rio.

The average kid who spent every evening of the 1970s browsing record stores for rare blues recordings is likely to be disappointed with the crap his kids listen to on The YouTube. And vice versa.

If you are really interested in US history, I bet you are disappointed with how little the kids of today know about your favourite topic. Guess what! They are disappointed in you too!

The average software developer who does not have a degree in computer science probably doesn’t know much about data structures and algorithms. Neither does the average CS graduate. Most of them slept through that class or forgot most of it the next day. More surprisingly, the average PhD is not very good at software engineering either unless they are working in their very narrow field of expertise. Of the best engineers I have ever worked with, only a few had a PhD or a Master’s degree in CS. Some had degrees in English or music and a good number had no degree at all. In fact, it’s quite amazing that many of the most famous people in software dropped out of college—or maybe that’s just my own confirmation bias playing tricks on me.

I expect the world would be a much happier place if people listened to their Uncle Sturgeon and relied on statistics and biases only when they prove useful. A statistical overview of a population can be helpful when you are deciding how to profitably market your new product or where to spend your campaign dollars or which college recruiting fair to attend. But if you are choosing an umpire for your baseball league or an anchor for your running team or a new hire for your software startup you’d do better to ignore the statistics and hire the individuals with the right skills for the job. To do otherwise is prejudice.

May
9
2014

Songs Without Love

I was wondering what songs are about. Most of them are about love of course but what about the other ones?

Terry Gross had a dude on the other day who wrote songs about a bomb that went off his train carriage on the way to Machu Pichu. Abba won the 1974 Eurovision Song Context with a song about Waterloo. Are there any other songs about weird topics?

I haven’t done a fun science project for a while and I need to learn about the latest versions of Ruby & Rails & Elastic Search & D3 & Hicharts. I also want to dabble in some NLP stuff—sentiment analysis; classifiers; that kind of thing.

Here’s the TODO list:

* Grab every top 100 song since Al Martino in 1952.
* Grab all the lyrics to all the songs.
* Build a word cloud for each song.
* Build a word cloud for each week/year/decade.
* Do cluster analysis to find interesting topics.
* Write a classifier that can figure out what each song is about (love, war, bombs, whatever).
* Plot how that changes over time.
* Do sentiment analysis to see if songs are happy or sad.
* Plot how that changes over time.

The results will be here:

http://www.songswithoutlove.com/

If you know Ruby (or want an excuse to learn) fork my repo and play along.

https://github.com/klawrence/songs-without-love

I need to get this finished by next year. The year after that at the very latest.

Nov
16
2013

Moral Instincts Don’t Scale

Excellent idea that I never thought of before…

The first half of this bloggingheads video is about Paul Bloom experiments showing how very young babies have a well-developed moral sense. In the second half of the video, Paul & Bob discuss how well our moral instincts and emotions like justice, anger and empathy work in our everyday lives but how badly they scale up to interactions between nations.

Nov
7
2013

No Apologies

I’ve developed a habit in recent years of getting so excited after reading the first few chapters of a book that I want to write a provisional review before I even really know what it’s about.

Two chapters into Unapolagetic, I was shouting yes! yes! yes! and couldn’t wait to capture my agreement in digital ink. I even, for the first time ever, used the highlight feature on my kindle so that I could accurately report all the many, many points that resonated with my own experience.

On one point in particular Spufford was singing my life with his words. When he described that transcendental moment when I meditate on the immensity of eternity – Spufford calls it ‘praying’ but, whatever – and I experience a little shimmer in the corner of my mind’s eye and suddenly everything makes sense and the whole universe lines up for my inspection and appreciation….  I didn’t know that other people experienced it too. I thought that was just me and I had no idea that some people called that shimmer ‘God’.

Seeing my secret thoughts in pixels, I was tremendously excited to document my assent to Spufford’s project which, as I understood it from tentatively supportive reviews by a couple of Christian bloggers that I follow, was to describe Christianity’s benefits in words that a non-Christian could identify with. I could get behind that project.

Real-life intervened though and the reviewable moment was lost. An emergency dash down to Santa Barbara later and the blogging window slammed shut. As I ploughed on through the next chapter on theodicy all my admiration fell away.

Spufford concedes that the problem of evil  is not a problem for atheists. It’s only a problem if you believe in one very specific kind of God: that is, a God who loves the world and is in full control of all the outcomes. There is no problem of evil if your God doesn’t care about children with cancer. Without love, there is no problem to be solved.

Likewise, there’s no problem with a God who cares an awful lot about you but is powerless to do much about it. That God would be lame and not much worthy of all the cathedrals that get built for him and he’d create all kinds of problems for his marketing department but he wouldn’t create any particular challenges for philosophers.

Since the God that Spufford worships is allegedly both omnipotent and benevolent, the problem of evil is a big problem indeed and Spufford absolutely demolishes all the usual excuses for why God allows evil to happen to good people. Let’s just say that if Spufford were in God’s marketing department, God would’ve fired him by now. Spufford works in mysterious ways.

I just went back to my Kindle to read over all the sections that I highlighted back when I thought  this was a great book. Spufford really is an astoundingly good writer. He wasn’t just singing my life; he was strumming my secret pain in ways that make me wonder if he isn’t in league with the NSA. I really want his project to succeed but chapters three and four make me despair for the whole liberal Christian project, cuddly new popes notwithstanding.

I’ll report back if I learn anything new.

 

Oct
24
2013

The Priest with the Cold Hands

When Rita was in intensive care there was a moment where it seemed like it was all over and her mother told me to go find the priest.

I went down to the chapel and there was an old, old priest in there. As old as Methuselah. I was frantic and I couldn’t get the words out.

That old priest held both of my hands in his and the words he spoke were like the wisdom of the ages. They slowed the whole world down and we all made it through another day.

Oct
22
2013

It’s all good

lesmisI saw Les Misérables on stage with my wife-to-be on one of our first dates and read the book soon after. I watched it again last night with my daughter after a twenty-something-year gap and I think I spotted something new about the story that I have not heard mentioned before. I even went to check the SparkNotes to see if this was some obvious theme that only I was missing but… nope. No one else noticed it but me.

When anyone mentions Les Mis, there are a few themes that immediately pop into your mind: it’s very long and tedious; it’s sentimental kitsch; the songs are awful; the story is an entirely predictable tale of redemption for Jean Valjean; the scene on the barricades is totally unnecessary and out of place.

All those things are true but I think there is something quite profound about how the characters interact with each other. All of the characters (*with one possible exception below!) are very moral according to their own particular version of morality. Of course, since it’s a liberal play/book/movie, all the ‘good guys’ subscribe to liberal conceptions of morality: everyone deserves a chance; kindness will be repaid; poor people commit crimes out of necessity.

But if that was all there was to the story, it would be a very shallow fable about a very liberal morality — we all get to cry for Fontine and boo at Javert — but I suspect that there might be more to it. We have all been missing something quite shocking.

Everyone in Les Mis behaves quite morally—according to their own understanding of right and wrong. Even the baddies.

Jean Valjean gets off to a bad start but learns the power of love and earns his redemption many times over.

The priest believes that everyone deserves a chance and gives Valjean his at great cost to himself.

Fantine gives up every sliver of self-respect to keep her daughter alive.

Javert believes in the Rule of Law above all and it matters not at all to paladin-types like Javert that Valjean has done good in his life. Valjean has broken the law and must be punished. Javert is willing to devote his whole life to bringing Valjean to justice.

The anarchists on the barricades believe the government is corrupt and should be overthrown. The soldiers who gun them down have the opposite view.

The slut-shaming women that get Fantine fired from her job believe that sexual impropriety and non-traditional families are Very Very Bad and should be punished. The Slut-shamers’ beliefs are probably shared by a majority of People Who Consider Themselves to be Morally Upstanding.

The photo that got Ashley Payne fired from her teaching job.

The photo that got Ashley Payne fired from her teaching job.

The foreman who actually sets Fantine on the road to self-destruction is following an ethical code that has held sway for most of human history and has only recently begun to retreat from the mainstream. In the glorious 50s that conservatives love to reminisce over,  most young women would have been suffered the same fate as Fantine. Even today, young teachers can be fired for inappropriate Facebook photos.

The only exception to the ‘everyone is moral’ rule is, of course, The Master of the House but I bet, if you asked him, even he would say that all those other conceptions of morality were invalid and that the only proper behaviour is to look out for your own interests.

[* OK. That last one is a stretch. The Thénardiers are quite evil]

It’s quite something that all these conceptions of morality still have their champions 150 years after Hugo wrote it all down for us. Will we ever figure out which one is correct?

 

 

 

Jul
12
2013

Fire! Fire! Fire!

Hearing the reports and interviews about the San Francisco plane crash brought back a long lost memory.

HMS Revenge

I had just turned 21 when I joined the nuclear submarine HMS Revenge. I was one of two petty officers responsible for all the sonar systems on board. The other dude, Mark, was a year or two older than me. As you can probably imagine, sonar systems are pretty important to a submarine so it was a big responsibility. But no one on a sub has just one job. I had four. My main watch-keeping position was OPSO.

If you have ever seen a movie of a submarine in action - wait!…. here’s one!… - you may have noticed that there are five key roles in the control room of a sub and one more further back in the engine room.

This is a recruitment film deviously crafted to tempt disaffected but smart young men to run away from home and become engineers in the Royal Navy [they got me! - ed] Skip to 17:10 to see a submarine that may very well be HMS Revenge. Fun fact: The sub on the inside is different from the one on the outside!

The officer of the watch (OOW) – or the captain during tricky manoeuvres - is in charge of the whole boat and makes all the tactical decisions and the other four folks report to the OOW. The after-planesman is responsible for steering and keeping depth, a responsibility he shares with The Panel watch-keeper (open main vents, sir!) who continually monitors the submarine’s trim, opening and closing valves to keep the boat level. Back aft, the stokers take care of the main engines and generators and, of course, the nuclear reactor. Last, and very definitely not least, there is OPSO. Me.

When a sub is underwater – which is most of the time – it is almost completely blind. That thing you hear in movies with the Ping! Blip! of the sonar every 10 seconds does not actually happen these days. Soviet submarines used active sonar but NATO boats didn’t (if you hear a sonar ping in the middle of the North Atlantic, it’s probably from a Russian boat or a surface ship) so the only way to know what was out there was to listen very carefully.

HMS Revenge at dawn

HMS Revenge at dawn

 

Most of the listening was done by 16 year old boys with headphones and they would report everything they heard – carpenter fish; snapping shrimp; underwater pigs; other ships – and (if it was a ship) a guess at their speed from the sound of its propellers. We had no range information at all. We could guess the distance of a ship based on how much its bearing changed but even that was of no use if the ship was coming straight at us. If a ship had cut its engines, we wouldn’t even know it was there. OPSO’s job was to keep track of all the targets reported by the sonar department -including all the whales and dolphins and fishing boats that would start their engines, motor for ten minutes then cut their engines – and recommend a safe course to the OOW. Oh and – this at the height of the cold war remember – listen out for enemy submarines. So, while OPSO was my job #2, and actually took most of my working hours, job #3 was the most glamorous.

Polaris Missile Launched from HMS Revenge

Polaris Missile Launched from HMS Revenge

When a ballistic missile sub is called upon to fire its weapons of armageddon (weapons of mass destruction does not quite capture the extent of the horror) the submarine makes a lot of noise, instantly announcing its presence to every enemy submarine for hundreds of miles and its role changes instantly from dealer of destruction to recipient of it and the boat needs to defend itself to survive. My job #3 was, as torpedo guider, to kill those other submarines before they killed us. Lots of video game practice helped honed my torpedo-guiding skills.

If job #3 was most glamorous, job #4 was the most terrifying as I was in the attack squad of the firefighting team. Fires happen surprisingly often on ships and submarines and they are quite dangerous, what with all the hydraulics and the fuel and the explosives everywhere. We’d be called out to fight a fire at least once a week and maybe once a month it’d be a proper scary fire.

One big difference between a ship and a submarine is that on a ship – at least, on the destroyer I served on – when there is a fire, only the designated fire-fighting team has to react to it. Everyone else just carries on eating their dinner or cleaning the bathrooms or whatever. On a submarine, fires are a much bigger deal and the whole crew joins in the fun of fighting the fire.

There are two main sections to the firefighting team. One team dresses up in  fearnought suits – big woolly suits that keep you toasty warm even when you are not walking into a fire – ready to do the main work of fighting the fire with the main hose and a waterwall. The attack team just grabs a flimsy little mask and a fire extinguisher and runs into the blaze while the main team is suiting up.

I remember my first big, super-scary fire in an auxiliary machinery room (AMR) filled with diesel generators. We had already had several little fires since I joined the boat, but they were little affairs and extinguished quickly. There was always a kind of anti-climax after the excitement of the  Fire! Fire! Fire! on the Tannoy and the scrambling to grab a mask and get to the fire first, when the first dude on the scene was able to put out the fire straight away. This one was different though and it was clearly going to be a big deal.

At the Fire! Fire! Fire! alarm, I ran as usual to grab my mask from the pile but, as I reached for the very last mask and steeled myself for the battle ahead, a burly stoker PO named Mitch, 15 years my senior, put his hand on the same mask. He looked me in the eye and said “I think you’d better let me have that, son”. I didn’t argue and he ran into the burning AMR leaving me mask-less and safely away from the flames. No one died that day and the fire was extinguished without too much drama but, ever since then, I have had a healthy respect for people who run into burning buildings for a living. That memory came back to me this morning when I read the interviews of the first responders to the plane crash at San Francisco Airport over the weekend.

From KQED,

Firefighters said they encountered smoke, leaking jet fuel and passengers coming down on chutes when they arrived. Lt. Christine Emmons said Monday at the news conference that she and her partner ran up a chute into the plane and found four passengers trapped in the back. The conditions in the plane were changing rapidly, with the fire coming down on rescuers and the smoke thickening as the trapped passengers were pulled out to safety, she said.

Lt. Christine Emmons

Lt. Christine Emmons

and

Lt. Dave Monteverdi, who had also run up the chute, said someone had to be extricated after one of the bulkheads fell on top of him. When the firefighters entered the plane, they were surprised to find San Francisco police Officer Jim Cunningham already there, not wearing any protective equipment. Several of the public safety officers who spoke gave him a special shout-out for heroism. Police Lt. Gaetano Caltagirone also entered the plane, following Cunningham. “I couldn’t let him go inside the plane and just be there by himself,” he said.

People who fight fires for a living are amazing but I have a special regard for whose who, like flight attendants Lee Yoon-hye and Kim Ji-yeon have firefighting as job #4 – after handing out ginger ale, picking up your trash and getting unruly passengers to please turn off their electronic equipment now.

From Yahoo News

One flight attendant, Kim Ji-yeon, 30, put a scared and injured elementary schoolboy on her back and slid down a slide, said Lee, in the first comments by a crew member since the crash of the Boeing 777. A pilot helped another injured flight attendant off the plane after the passengers escaped.

Jiyeon Kim

Jiyeon Kim

Lee herself worked to put out fires and usher passengers to safety despite a broken tailbone that kept her standing throughout a news briefing with mostly South Korean reporters at a San Francisco hotel. She said she didn’t know how badly she was hurt until a doctor at a San Francisco hospital later treated her. When Lee saw that the plane was burning after the crash, she was calm. “I was only thinking that I should put it out quickly. I didn’t have time to feel that this fire was going to hurt me,” she said. Lee said she was the last person off the plane and that she tried to approach the back of the aircraft before she left to double-check that no one was left inside. But when she moved to the back of the plane, a cloud of black, toxic smoke made it impossible. “It looked like the ceiling had fallen down,” she said. http://news.yahoo.com/asiana-attendant-describes-dramatic-evacuation-101658097.html

Brave people, firefighters. Especially when they are flight attendants too.

Burning Plane at SFO

Burning Plane at SFO