politics – Ragged Clown

A More Social Software Engineer

That Google memo business has got me thinking.

The author got himself into trouble for a bunch of reasons (too much logos and not enough pathos for a start) but I thought the core of his argument, restated to be more positive, could be appealing to many of the people who condemned it.

I’m going where angels fear to tread. Wish me luck!

Diversity is good for software companies. It helps us make better products.

I work at a little software startup just up the road from Google. Men are a minority at my company and I like it that way. It makes for a more pleasant work environment. I’ve worked on too many all-male teams and they tend [stereotype alert!] to be much more competitive and less collaborative. With women around, the men try — but sometimes fail — to be on their best behaviour.

Less selfishly, I think that our diversity makes us a better team. The women [stereotype alert!] really do see things that the men don’t see (and vice versa).

Men and Women are Different

Although there are differences in interests and traits between men and women at the group level, we should judge individuals as individuals, not as members of their group.

I don’t think either point is controversial, though the second might be hard to accept if your blood pressure is already high from reading the first. There’s a long history of people using statistics like this for ill purposes but I hope and expect that we can avoid the mistakes of the past with a little goodwill and a generous reading of the evidence.

There’s no need to posit genetic explanations for the differences. Cultural history can explain it adequately well.

There is chronic sex discrimination in the hiring process.

I have no doubt about this and we should certainly fix it. But bias in the hiring process and discrimination in the workplace are inadequate explanations for the gender imbalance in software engineering.

Creating awareness of implicit bias during the interview process is certainly helpful but it’s not enough to fix the whole of the problem. If we want to get more women into software development (we do! we do!), we need to look further. We need to consider other solutions too.

Vive la différence!

Men are taller than women on average but, if I need tall people for a job, I would be foolish to go out to hire a bunch of men. I can just measure the candidates. If they are tall enough, I can hire them without considering their gender.

The analogy doesn’t quite work because an objective quality like height is easy to measure. Subjective qualities of the kind we might consider when hiring a software engineer can be harder to untangle from our biases and our stereotypes but we can try.

Once we’ve decided that our candidate is good enough (or tall enough) for the job, it’s not helpful to consider whether they have the right configuration of sex chromosomes (individuals, not groups remember?). But maybe it’s useful to think about group differences during the interview processes itself?

The evidence from psychology seems to suggest that women and men are quite different, on average, on certain traits like neuroticism and agreeableness even if there is considerable overlap between the sexes. Could those differences make the interview process more treacherous (on average) for women? I think it does, especially when you consider that most of the interviewers are men.

I don’t know of any work that looks at agreeableness as a liability in interviews but there was an interesting experiment recently at interview.io that tried to understand why women are less likely to succeed in interviews.

Make interviews more social?

interview.io is a website that gives software engineers an opportunity to practice interviewing in a safe environment. They noticed large differences in outcomes between male and female candidates.

we’ve amassed data from thousands of technical interviews, and in this blog, we routinely share some of the surprising stuff we’ve learned. In this post, I’ll talk about what happened when we built real-time voice masking to investigate the magnitude of bias against women in technical interviews. In short, we made men sound like women and women sound like men and looked at how that affected their interview performance. We also looked at what happened when women did poorly in interviews, how drastically that differed from men’s behavior, and why that difference matters for the thorny issue of the gender gap in tech.

http://blog.interviewing.io/we-built-voice-modulation-to-mask-gender-in-technical-interviews-heres-what-happened/

Here’s what they knew before the experiment:

Specifically, men were getting advanced to the next round 1.4 times more often than women.

They designed the experiment expecting to confirm a bias against female interviews but…

Contrary to what we expected (and probably contrary to what you expected as well!), masking gender had no effect on interview performance with respect to any of the scoring criteria (would advance to next round, technical ability, problem solving ability).

They didn’t really come up with a satisfactory explanation for the failure to uncover bias but an author can speculate…

Anecdotally, it seemed like women were leaving the platform a lot more often than men. So I ran the numbers.

What I learned was pretty shocking. As it happens, women leave interviewing.io roughly 7 times as often as men after they do badly in an interview.

If women drop out so much more during the interview process, might we see the same behaviour throughout a woman’s fledgling career in software engineering?

The author speculates some more…

If that’s true, then we need 3 times as many women studying computer science than men to get to the same number in our pipelines. Note that that’s 3 times more than men, not 3 times more than there are now. … [snip]… to get to pipeline parity, we actually have to increase the number of women studying computer science by an entire order of magnitude.

Maybe the author of the Google memo was on to something when he wondered if neuroticism was significant when we look at the differences in the number of male versus female engineers? Just not in the way you thought.

Armed with this new information, perhaps we can change the way we interview engineers?

I’ve interviewed and been interviewed perhaps hundreds of times over my career. I enjoy being interviewed. But I can honestly say that my interview at Google was the least pleasant that I have ever endured. The interviewers were completely uninterested in my hopes and dreams and were actively hostile when I tried to engage them in conversation.

Maybe Google has got better at interviewing since then. If they haven’t, maybe they could try? No need to lower the bar on quality. They could start by acknowledging that there’s more to being a great software engineer than solving Big O problems.

Maybe we can make software engineering more social?

The author of the interviewing.io study suggested that we can’t fix the gender imbalance by better interviewing alone. Is there anything about software engineering itself that is off-putting to women?

Let’s first acknowledge that the predominance of men off-putting to women. Being the odd one out on a team is a high cultural hurdle to clear before we even think about differences in traits and interests. The evidence from other disciplines suggests that the hurdle is not insurmountable though.

Women now progress to PhDs as often as men in science, maths…

…and engineering and women now dominate health professions and life sciences.

If the life sciences can do it, maybe software engineering can too.

Software engineering has a poor image in the outside world. Most people think of a software engineer as a lonely introvert locked away in a a cubicle and typing code into a computer all day. Maybe Google is like that, but I haven’t worked in such an environment for over fifteen years.

Since I discovered Kent Beck’s crazy way of making software, I’ve come to value interactions and collaboration much more than the old process of turning specifications into code. I spend most of my day collaborating with other people.

If young women knew that software engineering was highly collaborative, might they be more inclined to give it a try?

Isn’t that just what Damore argued in his memo?

I read four or five rebuttals of the Google memo before I read the actual memo itself and I was quite surprised, when I finally read it, to find that the memo was neither as crazy nor as hostile to women as I had been led to believe.

Rather than refuting his points, most of the rebuttals were reiterating what Damore himself had said in his memo.

Now, granted, my argument is not exactly the same as Damore’s but it’s not far off. Damore would’ve done better to have saved his rant about liberal bias at Google for another day. And the rant about extra support for women and minorities would have sounded better if he had shouted it at the clouds.

He got a lot of things right though.

The psychologists at the Heterodox Academy looked at the research.

In conclusion, based on the meta-analyses we reviewed above, Damore seems to be correct that there are “population level differences in distributions” of traits that are likely to be relevant for understanding gender gaps at Google and other tech firms. The differences are much larger and more consistent for traits related to interest and enjoyment, rather than ability.

…

Population differences in interest may be part of the explanation for why there are fewer women in the applicant pool, but the women who choose to enter the pool are just as capable as the larger number of men in the pool. This conclusion does not deny that various forms of bias, harassment, and discouragement exist and contribute to outcome disparities, nor does it imply that the differences in interest are biologically fixed and cannot be changed in future generations.

…

If our three conclusions are correct then Damore was drawing attention to empirical findings that seem to have been previously unknown or ignored at Google, and which might be helpful to the company as it tries to improve its diversity policies and outcomes.

The Heterodox Academy

Most of the first round of criticism missed his point entirely. Gizmodo called it an anti-diversity screed .

Gizmodo:

Damore:

I value diversity and inclusion, am not denying that sexism exists, and don’t endorse using stereotypes. When addressing the gap in representation in the population, we need to look at population level differences in distributions. If we can’t have an honest discussion about this, then we can never truly solve the problem.

Conor Friedersdorf at The Atlantic summarizes the misleading coverage well.

To me, the Google memo is an outlier—I cannot remember the last time so many outlets and observers mischaracterized so many aspects of a text everyone possessed.

Casually perusing “anti-diversity” headlines without reading the memo might mislead readers into thinking a Google employee had assigned a negative value to gender diversity, when in fact he assigned a positive value to gender diversity, but objected to some ways it was being pursued and tradeoffs others would make to maximize it.

https://www.theatlantic.com/politics/archive/2017/08/the-most-common-error-in-coverage-of-the-google-memo/536181/

Sabine Hossenfelder at Nautilus wonders about the wider implications of his memo.

Damore’s memo strikes me as a pamphlet produced by a well-meaning, but also utterly clueless, young white man. He didn’t deserve to get fired for this. He deserved maybe a slap on the too-quickly typing fingers. But in his world, asking for discussion is apparently enough to get fired.

…

If you remove biology from Damore’s notion of “population level differences”, his critique is still nearly as powerful. And his question is still valid: “If we can’t have an honest discussion about this, then we can never truly solve the problem.”

…

Damore was fired, basically, for making a well-meant, if amateurish, attempt at institutional design, based on woefully incomplete information he picked from published research studies. But however imperfect his attempt, he was fired, in short, for thinking on his own. And what example does that set?

http://nautil.us/blog/outraged-by-the-google-diversity-memo-i-want-you-to-think-about-it

Bring your daughters to work. And keep them there.

After the last round of “Silicon Valley Is Sexist” outrage, I asked each of the very smart, very capable women of my acquaintance why they did not become software engineers. They all answered with some variant of “I can’t think of anything less appealing than the idea of working with a machine all day” or “I would hate to work in an environment where I was the only woman”.

I also asked all my male friends why their daughters were not interested in computer science. They all sighed and said “I just can’t get her interested”.

Only 18% of CS graduates are women. If we want to attract more women into software engineering (and we should – it’s a fun job; very social and pays well!), we have to find a way to get our daughters interested. We are not gonna fix the whole industry by making interviewers less biased.

To the Left!

I’ve always struggled with the words progressive and liberal.

In America at least, liberal seems to mean so many different things to different people that it doesn’t seem to mean anything at all. I have no idea what progressive means at all.

Interfluidity has a handy-dandy chart that attempts to sort lefties into useful categories.

While I am sympathetic to, for example, the goals of Black Lives Matter, and mindful of the fact that, in my friend’s memorable phrase, “It’s harder to move through the world” if you don’t have a particular genital configuration or skin colour, I’ve long believed that identity politics is poisonous to social discourse and is at least partially responsible for Trump, UKIP and worse.

Universalists want group identity to become less salient and consequential, and so resist tactics that highlight difference in order to promote intragroup solidarity and to sow open conflict with other groups. Identity-centered activists view solidarity and conflict as the best and perhaps only way to overcome identity-distributed oppression. To a universalist, tactics like “no platforming” sow precisely the sort of divisions we ought to be working to overcome. To an identity-centered activist, “no platforming” an apologist for racism or sexual violence is just winning..

Put me down firmly for universalist and allow me to claim Dr King for my side.

I have a dream that my four little children will one day live in a nation where they will not be judged by the color of their skin, but by the content of their character.

The other axis is more difficult to place myself on.

“Classical liberals” and the people sometimes referred to as neoliberals don’t challenge the existence of large, consequential differences between rich and poor. They seek to remedy what is oppressive in economic stratification by putting a humane floor beneath the consequences of being sorted downwards, and by working to ensure that the sorting is “fair”. They tend to promote equality of opportunity and emphasize education as a solution.

I have long believed that runaway inequality is the biggest threat to American democracy. I believe that the game is rigged and that a handful of millionaires and billionaires have hijacked the apparatus of government and found a way to pretend that 47% of the population are moochers and parasites.

According to the chart’s economic axis, my fear of runaway inequality is incompatible with my deep-seated understanding that meritocracy and rewarding success are vital to a flourishing society. I don’t see them as in opposition to each other.

Strategically, I am a Classical Liberal but, practically, I think we need to reign in the power-grabbing plutocrats and find ways to ensure that Warren Buffet pays more tax than his secretary.

The author of interfluidity seems to struggle with this too.

The diagram above summarizes the differences as I see them among US liberal-to-left factions. Note that these are questions of more or less, not absolutes. I’d place myself in the “universalist left”, for example, but I do believe that some degree of economic stratification is legitimate and necessary, under economists’ usual rationale of preserving incentives to produce. I just think that the degree of economic stratification that currently prevails is way, way, way, way, way past the point where benefits of sharp incentives to produce are undone by even sharper incentives to cheat and outweighed by destructive social fragmentation.

As far as I can tell, the author’s self-diagnosis is the same as mine, but he wants put himself in the bottom left quadrant. I’m pretty sure I’m up in the top left.

Levi Stubbs Tears

I’ve been a Billy Bragg fan since I saw him at the Portsmouth Guildhall in ’86 and I’ve terrorized my family and friends by singing his songs ever since.

My best Billy Bragg memories include teaching Train Train to Dylan when we lived in New York. He was only a year old and he used to finish the chorus for me “Train! Train!…Hurry bring my baby back again!”.

I was cornered by an Old Etonian and an Old Harrovian for singing There is Power in a Factory in the showers at BRNC Dartmouth. They’d never met a Labour supporter before.

The night before I was due to see Billy at the Guildhall, Rob and I saw him being arrested on the Nine O’Clock News for breaking into the nuclear base at Greenham Common. Billy was released in time though and he was brilliant. Ted Hawkins was his warm up act and I bought all of Ted’s albums too.

Levi Stubb’s Tears is one my favourites and I’ve wanted to learn it since I got my first guitar. This is the first song I’ve recorded in a couple of years and I’m a bit rusty. Was fun to play though.

I decided that the song needed a bit of a video to go with it so I spent a very pleasant couple of hours wandering through Flickr’s Creative Commons gallery looking for images and cramming them together in iMovie (iMovie sucks).

Government payroll exploded under Obama

Actually, no it didn’t. The federal government under Obama employs fewer people than at any time in the last fifty years.

Does that surprise you?

Bonus question: why does the federal government grow so much under Republican presidents?

This one’s for Stan (2)

I had a very pleasant chat with the guys on my football team after the game today. After the third beer the topic turned to politics and I made some claims that my teammates thought were outrageous. I promised to back them up.

First claim: Net Migration from Mexico under Obama is Zero

The specific point that Stan made was that under President Obama, immigration from Mexico was out of control. I said that net migration from Mexico under President Obama was zero.

Here’s the data (from Politifact):

The chart shows that the number of Mexicans living in California has stayed more or less the same under President Obama. I make no claims as to whether or not that’s a good thing or a bad thing. But it’s a fact.

Second Claim: The number of deportations under President Obama has gone up.

Again, I’m not expressing an opinion on whether deportations are a good thing or a bad thing. I was merely countering the claim that President Obama’s was not enforcing the immigration laws.

Here’s the data (from Pew Research):

Third Claim: Deficits are more likely to go down under Democratic presidents

My conservative friends were under the impression that the debt increases more under Democratic presidents. I’m on dodgier ground with this one because of recent history and a lot of depends-how-you-measure-it but I will valiantly do my best to defend the claim.

Here’s the data. This first chart is from The Atlantic.

The first thing to notice is that the debt shot up under President Reagan. The next thing to notice is the massive drop under Clinton. The situation under Obama is complicated and I’ll get to that in a minute.

Reagan proved that deficits don’t matter. — Dick Cheney

There’s a different way to look at the numbers though that is kinder to Republicans. Let’s look at that first.

Here’s the per annum increase.

That Reagan jump still stands out and the Clinton drop does too. But this chart accentuates the increase under Obama that really troubles Stan. What’s up with that?

Let’s drill into the Obama years. Here’s the Wall Street Journal.

Now, we all agree that that first year was horrendous but, as we all remember, we were in the aftermath of the word financial crisis in 70 years. The numbers have improved a bit since though.

The main cause of the Obama deficit was the reduction in tax revenues caused by the recession.

deficit=revenues – spending

As before, I’m making no comment about whether or not the government should increase spending during a recession, but the data says that spending increased more, percentage-wise, under President Bush than it did under President Obama.

Here’s a Forbes article (Obama’s a Big Spender…Just Like His Predecessors) that shows some data on spending. Again, check out the massive rise in spending under Reagan and the massive drop under Clinton.

You probably also didn’t fail to notice the massive rise under President George W. Bush and the absolutely enormous leap in 2009. Who gets the blame for that?

Another Forbes article (Who Is The Smallest Government Spender Since Eisenhower? Would You Believe It’s Barack Obama?) says that the budget in the first year of a presidency should be credited to the previous guy.

I think that’s going a bit too far but it does shine a bit of light on the complexity.

In summary, the data clearly shows that the deficit (and spending!) increased more under Reagan, Bush I and Bush II than it did under Clinton or Carter. Obama’s record stands or falls on whether you think the 2008/9 financial meltdown was Bush’s fault or Obama’s fault but let’s not pretend that it didn’t happen at all.

Here’s Factcheck.org with the final word,

Federal outlays for the current fiscal year are estimated to be $3.65 trillion, according to figures released in March along with the president’s proposed budget for fiscal 2015. We calculate that to be just 10.1 percent higher than fiscal 2009 spending levels he inherited from George W. Bush. (Fiscal 2009 began nearly four months before Obama took office, and spending levels were mainly set by Bush. As we’ve detailed elsewhere, we attribute — at most –$203 billion of actual FY 2009 outlays to increases signed by Obama.)

That 10.1 percent increase in spending is below the nearly 12 percent rate of inflation, as measured by the Consumer Price Index since Obama took office.

Obama’s increase in federal spending over five years contrasts dramatically with the 78 percent increase during Bush’s eight years.

A couple of other points came up that weren’t points of contention, but I’ll support them any way.

Claim: The number of uninsured has fallen under Obamacare.

Here’s a Gallup poll reported in Forbes (After Obamacare: Number of Uninsured Hits Five-Year Low):

And here’s similar data from the CDC showing the 16 million people who now have insurance who didn’t before:

Now, I think Obamacare is shit. But it’s better than what came before. Stan and I differed on whether it’s a tragedy that 9% of Americans still don’t have health care but we did agree that the USA spends too much on health care. Here’s how much too much (Health Costs: How the U.S. Compares With Other Countries on KQED).

First of all, note that the US spends more on healthcare than everyone else. Second of all, note that the US Government spends more on healthcare than everyone else (except Norway and the Netherlands)! But it doesn’t cover all the people! And does the USA get better healthcare for all this money? [SPOILER ALERT: no].

Stan said that if I could produce data to support my claims he’d reconsider his political outlook. I doubt I’ve changed his mind very much on the deficit (though it is certainly true that the deficit tends to go up more under Republicans) but I hope he will concede the facts on immigration and deportation at our next discussion 🙂

Stan?

No Dirty Tricks

My son made me watch the Democratic debate this week (a first!) and I’m glad I did. What a contrast to the Republican debate!

I love to read the reviews of debates like this and, more than any other time, I was shocked at how the media interpreted the result compared to my understanding.

My analysis first and then I’ll do a round-up of what the pundits are saying.

I don’t think debates like this are a zero-sum game, especially at this stage in the race. Each participant has different goals and it’s possible for more than one candidate to achieve their objectives. For example,

Hillary Clinton is the front-runner by an overwhelming margin. She is unpopular within her own party and there’s a sense that she is the inevitable candidate because she is the inevitable candidate. She needed to do well enough to assure her supporters that she really is the candidate that she thinks she is; well enough to convince Biden that he would be wasting his time to join the race; and to show just enough charm and deference to Democrats further to her left that she is not a complete robot and/or tool of the plutocracy. Hillary achieved her goals.

Bernie Sanders needed to excite his supporters and to demonstrate that he is not the crazy socialist that everyone says he is. He also needed to make a good introduction to the people who are seeing him for the first time. I’m sure he knows as well as anyone that he is a long-shot but if he can move the Overton Window to the left, he is a winner. Bernie achieved his goals.

Martin O’Malley probably doesn’t expect to become president either but he didn’t disgrace himself. I doubt that he moved the needle on his campaign to be president but he probably made it a little bit more likely that he’ll get a post in the next Clinton administration. He’s probably content with that.

Lincoln Chafee and Jim Webb should just stay home for the next debate. They lost.

The day after the debate, the mainstream pundits proclaimed with one voice that Hillary was the overwhelming winner and that the other candidates were a disgrace.

Hillary Clinton won. She won because she’s a strong debater. She won because Bernie Sanders is not. She won because the first Democratic presidential debate focused on liberal policies — and not her email scandal or character.” — Ron Fournier, The National Journal

“Hillary Clinton won because all of her opponents are terrible.” — Ryan Lizza, The New Yorker

Clinton demonstrated that she was, by far, the best presidential candidate onstage. Indeed, she may have been the only person onstage actually running for president. […snip…] But none of them waged the kind of frontal assault that would be required to dislodge a front-runner who commands Clinton’s breadth of institutional support. — Jonathan Chait, New York Magazine

The commentators on the right actually thought that the whole debate was a setup, staged to make Hillary look presidential.

The most irritating criticism though came from those voices of the center-right who have been telling us for years that there’s no virtue left in politics and that everyone just wants to get in a cheap shot or score points with sound-bites. When—finally!—there’s a debate where politicians talk about the issues, they’re all aghast at the lack of character assassination.

Here’s David Brooks on the Newshour.

…the one advantage the Republicans have is they actually, a bunch of them want to be president. On the Democratic side, only one person wants to be president. That’s Hillary Clinton.

and

But the other factor is, the Republicans are actually arguing and fighting with each other. And what I saw up there was Hillary Clinton performing extremely well, and four other guys lying down and let her, letting her have the nomination. It’s like Bernie Sanders held up the white flag of surrender when he refused to really go after her on the character and moral issue, which is his only way in. — David Brooks, The Newshour

Really, David? The only way for Bernie Sanders, a democratic socialist, to distinguish himself from the centrist front-runner is on moral issues?

Ross Douthat was even more nakedly critical of Bernie’s hesitance to get down and dirty.

by declining to attack Clinton on the issues where she’s actually most vulnerable — issues that have driven down her numbers with swing voters these last few months, issues that make her a weaker-than-expected general-election bet — Sanders is preserving his present popularity at the expense of any possibility of actually knocking off the frontrunner, actually beating her at the polls.—Ross Douthat, NY Times

Does it not occur to them that Bernie has no interest in winning a dirty tricks campaign? Even if he doesn’t win, he can shift the debate more significantly by staying true to his principles.

Rare among the pundits, Michelle Goldberg saw this.

His statement made him look like a mensch and a man of principle, ensuring that the debate remained a surprisingly substantive exchange on the issues he cares about most, rather a GOP-style pro-wrestling match. He actually seemed less interested in taking down the front-runner than in elevating his own ideas. That’s hugely rare in a politician. Clinton supporters who want a more progressive America have reason to be grateful for her strongest challenger. — Michelle Goldberg, Slate

AlterNet points out how much Bernie’s performance did for him in the polls.

Bernie Won All the Focus Groups & Online Polls, So Why Is the Media Saying Hillary Won the Debate? Bernie Sanders by all objective measures “won” the debate. Hands down. I don’t say this as a personal analysis of the debate; the very idea of “winning” a debate is silly to me. I say this because based on the only relatively objective metric we have, online polls and focus groups, he did win. And it’s not even close. —Adam Johnson, AlterNet

Bernie did pretty well in GooglePoints too.

In the end though, I have to agree with Matt Yglesias.

Clinton, in short, isn’t a flawless candidate, nor did she deliver a flawless debate performance. But she doesn’t need to be flawless to win — she just needs to be better than the opposition. And against a relatively weak field, she dominated. — Matt Yglesias, Vox

In Which Sturgeon’s Law Explains Everything

In 1951 Theodore Sturgeon was giving a talk about science fiction when someone in the audience noted that “90% of science fiction writing was crap”. Sturgeon shot back that “90% of everything is crap”.

This observation came to be known as Sturgeon’s Law.

Using the same standards that categorize 90% of science fiction as trash, crud, or crap, it can be argued that 90% of film, literature, consumer goods, etc. are crap. In other words, the claim (or fact) that 90% of science fiction is crap is ultimately uninformative, because science fiction conforms to the same trends of quality as all other artforms.

— Theodore Sturgeon

Sturgeon’s law is just as valuable when you are thinking about professionals as the old joke about doctors illustrates.

Q: What do you call a doctor who graduates at the bottom of his class?

A: Doctor

After years of pondering, I have come to believe that Sturgeon’s Law is more about the 10% that isn’t crap than the 90% that is. Forgetting the exact ratios for a moment, in every artform and every sporting contest, in every profession and every human endeavour, there is a distribution of quality in which a minority stands out as much better than the rest. We all know teachers who work that little bit harder to make their lessons interesting or the pharmacist who goes the extra mile to make sure that you understand your prescription. In my own profession—software engineering—only about 10% of engineers ever read a book about their craft once they leave college. This leads me to what I think is a more profound corollary to Sturgeon’s Law.

If you are a software engineer at the top of your game, you probably read books, attend conferences, study relentlessly to enhance your skills and engage in endless discussion on how to improve the state of the art of software craftsmanship. Average software engineers do not do this but you are probably surrounded by the rare few who seek to be the best they can be. This is where the less obvious aspect of Sturgeon’s Law casts its insidious spell. When you meet people outside of your elite circle, they are more likely to be average than elite.

Most engineering managers have a similar disinclination to better themselves. The best of them are very good but most of them are not the best. Too many are merely average. When the very best of the software engineering profession looks at the competence of software engineering management, they sigh a little because most managers are not very good.

Oddly enough, the very best of software engineering managers have a symmetrical view of the engineers they manage. Most engineers are not very good either. This isn’t just true of software engineers and their managers; it’s true of QA engineers, product managers, and designers too. Most QA engineers think that most programmers suck at testing. It’s true. Most of them do. But most QA engineers suck at it too. To the best QA engineers, the average programmers seem like uncaring barbarians. And vice versa.

I should emphasize at this point that I am not suggesting that some people are better than others at everything. Even very great software engineers might suck at gardening or astronomy.

This corollary to Sturgeon’s Law pertains across so many domains where the best of folks surround themselves with other folks who are motivated to excel. The average person outside their circle contrasts poorly as a result. Consider a Kalenjin long distance runner from the Rift Valley in Kenya. He probably encounters other great runners almost every day of his life. Most of the people he runs with, eats with and loves with are also great runners. If our Kalenjin got on a plane to Helsinki, he’d probably be disappointed to find that most Finns are not great long-distance runners. Most of them are just average. Truth be told, most of the Kalenjin are probably average too but our guy doesn’t encounter them very often. He hangs with the elite when he is at home. Away from home he has less opportunity to be so picky. It was only when he went to Finland that he encountered so many people who were not elite runners. Finland has elite runners too of course (I fondly recall cheering for Lasse Virén at the Montreal Olympics) but you are not going to bump into them at the airport as you might when you are at the Kenyan School for Elite Runners.

It’s helpful at this point to remember that individuals are not statistics (The median is not the message in the memorable words of Stephen Gould). If you wanted to recruit a team of long-distance runners it wouldn’t be a great idea to fly off to the Rift Valley and round up the first 5 runners you came across. You’d probably have a very average running team. You’d be much better off choosing great runners wherever they hail from. Furthermore, if you had to choose between a Finnish runner and a Kenyan runner, there is no need to check their birth certificate or the colour of their skin when you decide which one should join your elite running team. You can actually race them and choose the one that runs the fastest. When you are dealing with individuals, you should concern yourself with their individual qualities, not with some arbitrary statistical correlation however accurate that may be. It might be true that the average Kenyan runs faster than the average Finn but—so what? You will never be in a situation where you have to choose between average individuals without some other evidence to inform your choice.

The insidious nature of statistical racism is magnified by confirmation bias. Once you have decided that Kenyans run faster than Finns, it’s all too easy to reinforce your prejudice by noticing all the data points that confirm your bias—Hey! There goes another fat, slow and lazy Finn!—and to overlook the occasions that contradict your instincts. There is a long, unfortunate history of people doing exactly this.

There’s a whole garden of isms that wither under Sturgeon’s steely gaze. My dentist (Hi, Dr Bobba!) has a lovely cartoon on the wall with a caption that says something like “Women will never make good dentists. Their wrists are too weak!”.

Put yourself in the perspective of a Victorian gentleman who just happens to be a very good dentist. All your friends are very good dentists. They all have a certain background in dentistry and very strong wrists. You probably have a quite distinct image of what a proper dentist looks like. The average woman of your acquaintance probably seems very un-dentist-like. She is probably very uninterested in dentistry and has very dainty wrists. If you had to choose a dentist based only on wrist strength, you’d be marginally better off choosing the dentist with the stronger wrists—in 1875. But in 2014, you can skip right past concerns about wrist strength and whether your dentist has the appropriate genitalia and just hire the one who is the best at dentistry. And Doctor Bobba *is* very good. Trust me on that.

More casually—in our everyday lives—we are surrounded at work by people who share a certain intellectual outlook on life. Maybe your colleagues are more interested in politics than the average citizen. Maybe your friends at the sports bar care an awful lot about the intricacies of the infield fly rule or exactly how many defenders need to be behind the ball before offside is called. They know more than the average Joe about sports and certainly more than the average wife. Does that mean that women don’t understand politics or sports? No, of course not. The median is not the message, remember? It means that the average wife—in fact, the average anyone—knows less about sports than the fanatics you hang out with at the sports bar.

Let’s try some more examples.

The average tourist who visits Paris from their friendly little town in Georgia will find most Parisians quite distant, abrupt and possibly rude. The literary Parisians that he encounters will surely conclude that tourists from Georgia know very little about French art and are quite uncultured. If you repeat the experiment in the opposite direction and send a farmer to Atlanta from a little village in Provence I’ll wager the outcome will be identical. Repeat as necessary with Beijing, Nairobi, Melbourne and Rio.

The average kid who spent every evening of the 1970s browsing record stores for rare blues recordings is likely to be disappointed with the crap his kids listen to on The YouTube. And vice versa.

If you are really interested in US history, I bet you are disappointed with how little the kids of today know about your favourite topic. Guess what! They are disappointed in you too!

The average software developer who does not have a degree in computer science probably doesn’t know much about data structures and algorithms. Neither does the average CS graduate. Most of them slept through that class or forgot most of it the next day. More surprisingly, the average PhD is not very good at software engineering either unless they are working in their very narrow field of expertise. Of the best engineers I have ever worked with, only a few had a PhD or a Master’s degree in CS. Some had degrees in English or music and a good number had no degree at all. In fact, it’s quite amazing that many of the most famous people in software dropped out of college—or maybe that’s just my own confirmation bias playing tricks on me.

I expect the world would be a much happier place if people listened to their Uncle Sturgeon and relied on statistics and biases only when they prove useful. A statistical overview of a population can be helpful when you are deciding how to profitably market your new product or where to spend your campaign dollars or which college recruiting fair to attend. But if you are choosing an umpire for your baseball league or an anchor for your running team or a new hire for your software startup you’d do better to ignore the statistics and hire the individuals with the right skills for the job. To do otherwise is prejudice.

It’s not what you think

A Harvard economist asked 5000 people how they thought wealth was distributed throughout the United States and then asked how it should be distributed.

If wealth were distributed the way that most people think it is distributed, I think the country would be in pretty good shape. Watch the video to see how the wealth is actually distributed.

Acceptable Prejudice

Usual disclaimer: discrimination against atheists is pretty tame compared to the discrimination that blacks and jews and gays have historically faced. It’s not like atheists were ever persecuted or excluded from public office [er, you sure? -ed].

This bloggingheads.tv vialog makes the claim that discrimination against atheists is different from other kinds of discrimination in that prejudice against atheists is still seen as acceptable or even desirable whereas, while there is still an awful lot of discrimination against jews, blacks, gays, moslems, fat people and the disabled, the balance of public opinion has passed a tipping point and polite society will condemn it rather than nodding in agreement.

This, from today’s Guardian, illustrates the point nicely.

Conservative anti-gay prejudice was under scrutiny again on Friday after the Welsh secretary, David Jones, was forced to backtrack on an assertion that gay couples “clearly” cannot provide a “warm and safe environment” in which to raise children.

The important part of the story is not that Jones is a homophobe. It’s that he recognizes that it is unacceptable in 2013 to be a homophobe and that he is obliged to circumscribe his prejudices to try to make them acceptable and to walk back any comments that betray what he really thinks.

He even summons his invisible gay friends to vouch for his good faith.

“I regard marriage as an institution that has developed over many centuries, essentially for the provision of a warm and safe environment for the upbringing of children, which is clearly something that two same-sex partners can’t do.

“Which is not to say that I’m in any sense opposed to stable and committed same-sex partnerships.”

He did not believe he was homophobic, insisting he had “people in my life who are important to me who are gay”.

This is big news. Jones’s real crime is to be about 10 years behind public opinion. Back in 2003 it was obvious to most right-thinking people that the gays couldn’t be trusted to bring up children. Now that everyone knows a gay couple who are doing a fine job as parents, those ancient attitudes seem silly. By contrast, it’s still acceptable to think that atheists are morally inferior or that we need to protect children from them.

Here’s the vialog.

Unfortunately, as long as atheists remain a disparate group with few interests in common (ie. forever. Ricky Gervais says atheists are like a group of people whose hobby is not-skiing) this state of affairs is likely to continue.

Red states, blue states. Makers and takers.

Nate Silver’s column today was about how the tech industry overwhelmingly supported the Dems in the recent election. Obama won 84% of the vote in San Francisco and almost as much in Silicon Valley and the rest of the Bay Area.

Even more striking was the proportion of donations from tech companies that went to the Dems – not just in Silicon Valley but across the nation. The following table shows where donations from employees of the ten most admired tech companies went.

Ron Paul received more money from Google employees than Romney did.

Nate Silver focussed on the possible link between donations by techies and the failure of Romney’s data systems, presumably because all the competent techies were working for Obama. It’s far more interesting to me to consider what this pattern of donations says about Romney’s musings on who are the makers and who are takers and how Obama bought his votes with bribes to an underclass of moochers.

Mitt Romney told his top donors Wednesday that his loss toPresident Obama was a disappointing result that neither he or his top aides had expected, but said he believed his team ran a “superb” campaign with “no drama,” and attributed his rival’s victory to “the gifts” the administration had given to blacks, Hispanics and young voters during Obama’s first term.

ThinkProgress

I often wonder how the Republicans explain away Silicon Valley’s and New York’s overwhelming support for the Democrats. Does it interfere with their they-want-to-punish-success explanation that the well-remunerated employees of the most successful companies in the country actually prefer their political opponents? Surely not all of those tech employees are minorities voting for bigger handouts or young women voting for free contraceptives? What does it say about the class warfare narrative when 44% of people earning more than $250,000, voted for the party that wants to raise their taxes? How do they deal with the cognitive dissonance?

While we are on the topic of cognitive dissonance, have the conservatives not noticed that federal spending overwhelmingly flows from blue states to red states?

According to The Economist,

New York transferred over $950 billion to the rest of America’s fiscal union from 1990 to 2009. But relative to the size of its economy, Delaware made the biggest contribution, equivalent to more than twice its 2009 GDP.

Do they not know that marriages are more stable in blue states? Or that rates of drug abuse, teen pregnancy, obesity, drunk driving, armed assaults and poverty are higher in red states than blue states (not to mention European states)? Remind me, which is the party of individual responsibility?

Jeffrey Frankel, Professor of Economics at Harvard (from whom I purloined many of my red state/blue state facts), thinks that the folks who eat most heartily from the federal trough have a gap in their awareness of which groups are the freeloaders and that their politicians help to widen that gap.

The people who suffer the biggest gap between their perceived and actual share of the federal pie are likely to be getting a disproportionate share and yet to believe the opposite. If they believe that others are getting more than they themselves are, they are more likely to buy into the angry belief that other social groups are freeriding on society, and the ideology that government spending is wasteful and needs to be cut back, without realising that this includes the benefits they themselves receive. Perhaps these people are more likely to vote for Republican politicians, who tell them what they want to hear.

I have often wondered, in an odd echo of Romney’s 47% speech, whether the federal system with blue states subsidizing the red states, provides cover for the failings of conservative policies. If the federal government stayed out of healthcare, education, social policies and social security like the tenth amendment says it should, the blue states could experiment with universal healthcare, gay marriage and gun control and the red states could ban abortion, teach abstinence only sex-ed and do away with social security without messing it up for the rest of us.

In the laboratory of democracy, which petri dish would you rather live in?