A More Social Software Engineer

That Google memo business has got me thinking.

The author got himself into trouble for a bunch of reasons (too much logos and not enough pathos for a start) but I thought the core of his argument, restated to be more positive, could be appealing to many of the people who condemned it.

I’m going where angels fear to tread. Wish me luck!

Diversity is good for software companies. It helps us make better products.

I work at a little software startup just up the road from Google. Men are a minority at my company and I like it that way. It makes for a more pleasant work environment. I’ve worked on too many all-male teams and they tend [stereotype alert!] to be much more competitive and less collaborative. With women around, the men try — but sometimes fail — to be on their best behaviour.

Less selfishly, I think that our diversity makes us a better team. The women [stereotype alert!] really do see things that the men don’t see (and vice versa).

Men and Women are Different

Although there are differences in interests and traits between men and women at the group level, we should judge individuals as individuals, not as members of their group.

I don’t think either point is controversial, though the second might be hard to accept if your blood pressure is already high from reading the first. There’s a long history of people using statistics like this for ill purposes but I hope and expect that we can avoid the mistakes of the past with a little goodwill and a generous reading of the evidence.

There’s no need to posit genetic explanations for the differences. Cultural history can explain it adequately well.

There is chronic sex discrimination in the hiring process.

I have no doubt about this and we should certainly fix it. But bias in the hiring process and discrimination in the workplace are inadequate explanations for the gender imbalance in software engineering.

Creating awareness of implicit bias during the interview process is certainly helpful but it’s not enough to fix the whole of the problem. If we want to get more women into software development (we do! we do!), we need to look further.  We need to consider other solutions too.

Vive la différence!

Men are taller than women on average but, if I need tall people for a job, I would be foolish to go out to hire a bunch of men. I can just measure the candidates. If they are tall enough, I can hire them without considering their gender.

The analogy doesn’t quite work because an objective quality like height is easy to measure. Subjective qualities of the kind we might consider when hiring a software engineer can be harder to untangle from our biases and our stereotypes but we can try.

Once we’ve decided that our candidate is good enough (or tall enough) for the job, it’s not helpful to consider whether they have the right configuration of sex chromosomes (individuals, not groups remember?). But maybe it’s useful to think about group differences during the interview processes itself?

The evidence from psychology seems to suggest that women and men are quite different, on average, on certain traits like neuroticism and agreeableness even if there is considerable overlap between the sexes. Could those differences make the interview process more treacherous (on average) for women? I think it does, especially when you consider that most of the interviewers are men.

I don’t know of any work that looks at agreeableness as a liability in interviews but there was an interesting experiment recently at interview.io that tried to understand why women are less likely to succeed in interviews.

Make interviews more social?

interview.io is a website that gives software engineers an opportunity to practice interviewing in a safe environment. They noticed large differences in outcomes between male and female candidates.

we’ve amassed data from thousands of technical interviews, and in this blog, we routinely share some of the surprising stuff we’ve learned. In this post, I’ll talk about what happened when we built real-time voice masking to investigate the magnitude of bias against women in technical interviews. In short, we made men sound like women and women sound like men and looked at how that affected their interview performance. We also looked at what happened when women did poorly in interviews, how drastically that differed from men’s behavior, and why that difference matters for the thorny issue of the gender gap in tech.

http://blog.interviewing.io/we-built-voice-modulation-to-mask-gender-in-technical-interviews-heres-what-happened/

Here’s what they knew before the experiment:

Specifically, men were getting advanced to the next round 1.4 times more often than women.

They designed the experiment expecting to confirm a bias against female interviews but…

Contrary to what we expected (and probably contrary to what you expected as well!), masking gender had no effect on interview performance with respect to any of the scoring criteria (would advance to next round, technical ability, problem solving ability).

They didn’t really come up with a satisfactory explanation for the failure to uncover bias but an author can speculate…

Anecdotally, it seemed like women were leaving the platform a lot more often than men. So I ran the numbers.

What I learned was pretty shocking. As it happens, women leave interviewing.io roughly 7 times as often as men after they do badly in an interview.

If women drop out so much more during the interview process, might we see the same behaviour throughout a woman’s fledgling career in software engineering?

The author speculates some more…

If that’s true, then we need 3 times as many women studying computer science than men to get to the same number in our pipelines. Note that that’s 3 times more than men, not 3 times more than there are now.  … [snip]… to get to pipeline parity, we actually have to increase the number of women studying computer science by an entire order of magnitude.

Maybe the author of the Google memo was on to something when he wondered if neuroticism was significant when we look at the differences in the number of male versus female engineers? Just not in the way you thought.

Armed with this new information, perhaps we can change the way we interview engineers?

I’ve interviewed and been interviewed perhaps hundreds of times over my career. I enjoy being interviewed. But I can honestly say that my interview at Google was the least pleasant that I have ever endured. The interviewers were completely uninterested in my hopes and dreams and were actively hostile when I tried to engage them in conversation.

Maybe Google has got better at interviewing since then. If they haven’t, maybe they could try? No need to lower the bar on quality. They could start by acknowledging that there’s more to being a great software engineer than solving Big O problems.

Maybe we can make software engineering more social?

The author of the interviewing.io study suggested that we can’t fix the gender imbalance by better interviewing alone. Is there anything about software engineering itself that is off-putting to women?

Let’s first acknowledge that the predominance of men off-putting to women. Being the odd one out on a team is a high cultural hurdle to clear before we even think about differences in traits and interests. The evidence from other disciplines suggests that the hurdle is not insurmountable though.

Women now progress to PhDs as often as men in science, maths…

…and engineering and women now dominate health professions and life sciences.

If the life sciences can do it, maybe software engineering can too.

Software engineering has a poor image in the outside world. Most people think of a software engineer as a lonely introvert locked away in a a cubicle and typing code into a computer all day. Maybe Google is like that, but I haven’t worked in such an environment for over fifteen years.

Since I discovered Kent Beck’s crazy way of making software, I’ve come to value interactions and collaboration much more than the old process of turning specifications into code. I spend most of my day collaborating with other people.

If young women knew that software engineering was highly collaborative, might they be more inclined to give it a try?

Isn’t that just what Damore argued in his memo?

I read four or five rebuttals of the Google memo before I read the actual memo itself and I was quite surprised, when I finally read it, to find that the memo was neither as crazy nor as hostile to women as I had been led to believe.

Rather than refuting his points, most of the rebuttals were reiterating what Damore himself had said in his memo.

Now, granted, my argument is not exactly the same as Damore’s but it’s not far off. Damore would’ve done better to have saved his rant about liberal bias at Google for another day. And the rant about extra support for women and minorities would have sounded better if he had shouted it at the clouds.

He got a lot of things right though.

The psychologists at the Heterodox Academy looked at the research.

In conclusion, based on the meta-analyses we reviewed above, Damore seems to be correct that there are “population level differences in distributions” of traits that are likely to be relevant for understanding gender gaps at Google and other tech firms. The differences are much larger and more consistent for traits related to interest and enjoyment, rather than ability.

Population differences in interest may be part of the explanation for why there are fewer women in the applicant pool, but the women who choose to enter the pool are just as capable as the larger number of men in the pool. This conclusion does not deny that various forms of bias, harassment, and discouragement exist and contribute to outcome disparities, nor does it imply that the differences in interest are biologically fixed and cannot be changed in future generations.

If our three conclusions are correct then Damore was drawing attention to empirical findings that seem to have been previously unknown or ignored at Google, and which might be helpful to the company as it tries to improve its diversity policies and outcomes.

The Heterodox Academy

Most of the first round of criticism missed his point entirely. Gizmodo called it an anti-diversity screed .

Gizmodo:

Damore:

I value diversity and inclusion, am not denying that sexism exists, and don’t endorse using stereotypes. When addressing the gap in representation in the population, we need to look at population level differences in distributions. If we can’t have an honest discussion about this, then we can never truly solve the problem.

Conor Friedersdorf at The Atlantic summarizes the misleading coverage well.

To me, the Google memo is an outlier—I cannot remember the last time so many outlets and observers mischaracterized so many aspects of a text everyone possessed.

Casually perusing “anti-diversity” headlines without reading the memo might mislead readers into thinking a Google employee had assigned a negative value to gender diversity, when in fact he assigned a positive value to gender diversity, but objected to some ways it was being pursued and tradeoffs others would make to maximize it.

https://www.theatlantic.com/politics/archive/2017/08/the-most-common-error-in-coverage-of-the-google-memo/536181/

Sabine Hossenfelder at Nautilus wonders about the wider implications of his memo.

Damore’s memo strikes me as a pamphlet produced by a well-meaning, but also utterly clueless, young white man. He didn’t deserve to get fired for this. He deserved maybe a slap on the too-quickly typing fingers. But in his world, asking for discussion is apparently enough to get fired.

If you remove biology from Damore’s notion of “population level differences”, his critique is still nearly as powerful. And his question is still valid: “If we can’t have an honest discussion about this, then we can never truly solve the problem.”

Damore was fired, basically, for making a well-meant, if amateurish, attempt at institutional design, based on woefully incomplete information he picked from published research studies. But however imperfect his attempt, he was fired, in short, for thinking on his own. And what example does that set?

http://nautil.us/blog/outraged-by-the-google-diversity-memo-i-want-you-to-think-about-it

 

Bring your daughters to work. And keep them there.

After the last round of “Silicon Valley Is Sexist” outrage, I asked each of the very smart, very capable women of my acquaintance why they did not become software engineers. They all answered with some variant of “I can’t think of anything less appealing than the idea of working with a machine all day” or “I would hate to work in an environment where I was the only woman”.

I also asked all my male friends why their daughters were not interested in computer science. They all sighed and said “I just can’t get her interested”.

Only 18% of CS graduates are women. If we want to attract more women into software engineering (and we should – it’s a fun job; very social and pays well!), we have to find a way to get our daughters interested. We are not gonna fix the whole industry by making interviewers less biased.

 

 

In Which Sturgeon’s Law Explains Everything

In 1951 Theodore Sturgeon was giving a talk about science fiction when someone in the audience noted that “90% of science fiction writing was crap”. Sturgeon shot back that “90% of everything is crap”.

This observation came to be known as Sturgeon’s Law.

Using the same standards that categorize 90% of science fiction as trash, crud, or crap, it can be argued that 90% of film, literature, consumer goods, etc. are crap. In other words, the claim (or fact) that 90% of science fiction is crap is ultimately uninformative, because science fiction conforms to the same trends of quality as all other artforms.

— Theodore Sturgeon

Sturgeon’s law is just as valuable when you are thinking about professionals as the old joke about doctors illustrates.

Q: What do you call a doctor who graduates at the bottom of his class?

A: Doctor

After years of pondering, I have come to believe that Sturgeon’s Law is more about the 10% that isn’t crap than the 90% that is. Forgetting the exact ratios for a moment, in every artform and every sporting contest, in every profession and every human endeavour, there is a distribution of quality in which a minority stands out as much better than the rest. We all know teachers who work that little bit harder to make their lessons interesting or the pharmacist who goes the extra mile to make sure that you understand your prescription. In my own profession—software engineering—only about 10% of engineers ever read a book about their craft once they leave college. This leads me to what I think is a more profound corollary to Sturgeon’s Law.

If you are a software engineer at the top of your game, you probably read books, attend conferences, study relentlessly to enhance your skills and engage in endless discussion on how to improve the state of the art of software craftsmanship.  Average software engineers do not do this but you are probably surrounded by the rare few who seek to be the best they can be. This is where the less obvious aspect of Sturgeon’s Law casts its insidious spell. When you meet people outside of your elite circle, they are more likely to be average than elite.

Most engineering managers have a similar disinclination to better themselves. The best of them are very good but most of them are not the best. Too many are merely average. When the very best of the software engineering profession looks at the competence of software engineering management, they sigh a little because most managers are not very good.

Oddly enough, the very best of software engineering managers have a symmetrical view of the engineers they manage. Most engineers are not very good either. This isn’t just true of software engineers and their managers; it’s true of QA engineers, product managers, and designers too. Most QA engineers think that most programmers suck at testing. It’s true. Most of them do. But most QA engineers suck at it too. To the best QA engineers, the average programmers seem like uncaring barbarians. And vice versa.

I should emphasize at this point that I am not suggesting that some people are better than others at everything. Even very great software engineers might suck at gardening or astronomy.

This corollary to Sturgeon’s Law pertains across so many domains where the best of folks surround themselves with other folks who are motivated to excel. The average person outside their circle contrasts poorly as a result. Consider a Kalenjin long distance runner from the Rift Valley in Kenya. He probably encounters other great runners almost every day of his life. Most of the people he runs with, eats with and loves with are also great runners. If our Kalenjin got on a plane to Helsinki, he’d probably be disappointed to find that most Finns are not great long-distance runners. Most of them are just average. Truth be told, most of the Kalenjin are probably average too but our guy doesn’t encounter them very often. He hangs with the elite when he is at home. Away from home he has less opportunity to be so picky. It was only when he went to Finland that he encountered so many people who were not elite runners. Finland has elite runners too of course (I fondly recall cheering for Lasse Virén at the Montreal Olympics) but you are not going to bump into them at the airport as you might when you are at the Kenyan School for Elite Runners.

It’s helpful at this point to remember that individuals are not statistics (The median is not the message in the memorable words of Stephen Gould). If you wanted to recruit a team of long-distance runners it wouldn’t be a great idea to fly off to the Rift Valley and round up the first 5 runners you came across. You’d probably have a very average running team. You’d be much better off choosing great runners wherever they hail from. Furthermore, if you had to choose between a Finnish runner and a Kenyan runner, there is no need to check their birth certificate or the colour of their skin when you decide which one should join your elite running team. You can actually race them and choose the one that runs the fastest. When you are dealing with individuals, you should concern yourself with their individual qualities, not with some arbitrary statistical correlation however accurate that may be. It might be true that the average Kenyan runs faster than the average Finn but—so what? You will never be in a situation where you have to choose between average individuals without some other evidence to inform your choice.

The insidious nature of statistical racism is magnified by confirmation bias. Once you have decided that Kenyans run faster than Finns, it’s all too easy to reinforce your prejudice by noticing all the data points that confirm your bias—Hey! There goes another fat, slow and lazy Finn!—and to overlook the occasions that contradict your instincts. There is a long, unfortunate history of people doing exactly this.

There’s a whole garden of isms that wither under Sturgeon’s steely gaze. My dentist (Hi, Dr Bobba!) has a lovely cartoon on the wall with a caption that says something like “Women will never make good dentists. Their wrists are too weak!”.

Put yourself in the perspective of a Victorian gentleman who just happens to be a very good dentist. All your friends are very good dentists. They all have a certain background in dentistry and very strong wrists. You probably have a quite distinct image of what a proper dentist looks like. The average woman of your acquaintance probably seems very un-dentist-like. She is probably very uninterested in dentistry and has very dainty wrists. If you had to choose a dentist based only on wrist strength, you’d be marginally better off choosing the dentist with the stronger wrists—in 1875. But in 2014, you can skip right past concerns about wrist strength and whether your dentist has the appropriate genitalia and just hire the one who is the best at dentistry. And Doctor Bobba *is* very good. Trust me on that.

More casually—in our everyday lives—we are surrounded at work by people who share a certain intellectual outlook on life. Maybe your colleagues are more interested in politics than the average citizen. Maybe your friends at the sports bar care an awful lot about the intricacies of the infield fly rule or exactly how many defenders need to be behind the ball before offside is called. They know more than the average Joe about sports and certainly more than the average wife. Does that mean that women don’t understand politics or sports? No, of course not. The median is not the message, remember? It means that the average wife—in fact, the average anyone—knows less about sports than the fanatics you hang out with at the sports bar.

Let’s try some more examples.

The average tourist who visits Paris from their friendly little town in Georgia will find most Parisians quite distant, abrupt and possibly rude. The literary Parisians that he encounters will surely conclude that tourists from Georgia know very little about French art and are quite uncultured.  If you repeat the experiment in the opposite direction and send a farmer to Atlanta from a little village in Provence I’ll wager the outcome will be identical. Repeat as necessary with Beijing, Nairobi, Melbourne and Rio.

The average kid who spent every evening of the 1970s browsing record stores for rare blues recordings is likely to be disappointed with the crap his kids listen to on The YouTube. And vice versa.

If you are really interested in US history, I bet you are disappointed with how little the kids of today know about your favourite topic. Guess what! They are disappointed in you too!

The average software developer who does not have a degree in computer science probably doesn’t know much about data structures and algorithms. Neither does the average CS graduate. Most of them slept through that class or forgot most of it the next day. More surprisingly, the average PhD is not very good at software engineering either unless they are working in their very narrow field of expertise. Of the best engineers I have ever worked with, only a few had a PhD or a Master’s degree in CS. Some had degrees in English or music and a good number had no degree at all. In fact, it’s quite amazing that many of the most famous people in software dropped out of college—or maybe that’s just my own confirmation bias playing tricks on me.

I expect the world would be a much happier place if people listened to their Uncle Sturgeon and relied on statistics and biases only when they prove useful. A statistical overview of a population can be helpful when you are deciding how to profitably market your new product or where to spend your campaign dollars or which college recruiting fair to attend. But if you are choosing an umpire for your baseball league or an anchor for your running team or a new hire for your software startup you’d do better to ignore the statistics and hire the individuals with the right skills for the job. To do otherwise is prejudice.

Songs Without Love

I was wondering what songs are about. Most of them are about love of course but what about the other ones?

Terry Gross had a dude on the other day who wrote songs about a bomb that went off his train carriage on the way to Machu Pichu. Abba won the 1974 Eurovision Song Context with a song about Waterloo. Are there any other songs about weird topics?

I haven’t done a fun science project for a while and I need to learn about the latest versions of Ruby & Rails & Elastic Search & D3 & Hicharts. I also want to dabble in some NLP stuff—sentiment analysis; classifiers; that kind of thing.

Here’s the TODO list:

* Grab every top 100 song since Al Martino in 1952.
* Grab all the lyrics to all the songs.
* Build a word cloud for each song.
* Build a word cloud for each week/year/decade.
* Do cluster analysis to find interesting topics.
* Write a classifier that can figure out what each song is about (love, war, bombs, whatever).
* Plot how that changes over time.
* Do sentiment analysis to see if songs are happy or sad.
* Plot how that changes over time.

The results will be here:

http://www.songswithoutlove.com/

If you know Ruby (or want an excuse to learn) fork my repo and play along.

https://github.com/klawrence/songs-without-love

I need to get this finished by next year. The year after that at the very latest.

What’s your trick, Python?

The folks who wrote Pragmatic Programming recommend that you learn a new language frequently because, with each language, you’ll learn a new trick or a new way of thinking about programming that you never thought of before. When you go back to your old language you’ll take your new trick with you. Last year, I learned Objective C.

There is a lot to hate about Objective C. When I first started learning it, I felt like I had been time-warped back to 1987 along with some aliens from the planet Zarg but, over the last year, the language has improved so dramatically and so many of the rough edges have been smoothed that I could almost recommend it.

It’s still an ugly language, of course. The moments when you are confronted with bits of C in the middle of your Objective C method are like discovering that your ice cream topping is cod-liver oil.

It’s verbose too. The libraries feel like they were designed by colonial administrators in early-nineteenth century India. But, with automatic reference counting (ARC), it is no longer daunting to programmers who have forgotten how to alloc and dealloc.

Objective C has a couple of nice tricks though. My favourite is the fact that nil is an object and you can call its methods. In most languages, this would explode (or at least start a small fire):

collection = nil;
for (int i = 0; i < collection.length; i++) {
  id item = [collection objectAt: i];
  [item doSomething];
}

but it’s perfectly natural in Objective C. You can happily call methods on nil, it will return nil or 0 so you can just get on with your work without dodging NullPointerExceptions at every turn.

I like Objective C’s syntax for calling methods too, strange as it is. There is something heart-warming about the way that the method name wraps itself around the arguments so that in,

[object populate: collection
        fromFile: filename]

the method name is actually populate:fromFile:. It feels more comfortable than named arguments, in my humble opinion, and the way Xcode wraps the method call and aligns the colons makes it easy to read. If only the method names weren’t designed by colonial civil servants who mistook verbosity for clarity, it would be pleasant even. The names in the Cocoa libraries have that odd do the needful feel about them, like the authors learned grammar in a faraway country, probably one with steam trains, punkah wallahs and government forms in triplicate and it’s hard to love a language that doesn’t have a syntax for accessing array elements.

Ruby is the biggest trickster of them all. My only complaint about that language is that sometimes – especially in Rails – the whole language feels like one big trick. Every time I come back to it, I am constantly saying – “Wow! You can do that? That is awesome! Wait! How does that work again?”.

Ruby taught me blocks:

collection.each { |item|  item.do_something }

Sure, every language has blocks or lambdas these days, but there is just something very soothing about the simplicity of Ruby’s syntax that puts me at ease. In C#, I have to concentrate really hard to get the syntax right and, in Objective C, I doubt there is anyone in the world who remembers how to make a callback without looking it up online. I like to imagine that there was one primordial Objective C block written in a prototype at One Infinite Loop in 1994 and it has been copy-pasted ever since.

The trait I like most about Ruby is its humanity. If it seems like you can do something, you can. All these expressions work and do exactly what you might expect:

2.years.ago
3.times { print 'Ho! ' }
Date.today + 5.days
[1..100].each do |number|
  puts "#{number} is even." if number.even?
end

If only the Objective C folks would glance at the Ruby libraries and learn that terse does not have to be obscure and that verbosity is not intrinsically a good thing. Just ask the COBOL people.

A couple of years ago there was a debate online about the relative benefits of adding methods to objects to make a programmer’s life easier. The proposition was that such methods result in bloat which makes the API harder to learn but, really, how can you seriously argue that this:

if( array.length > 0 )
  element = array[array.length-1];

is more humane than this:

element = array.last

Meta-programming takes the Ruby language into the astroplane where the angels live and foolish mortals tread carefully. Here’s a builder for generating an xml file:

xml.slimmers do
  @slimmers.each do |slimmer|
    xml.slimmer do
      xml.name slimmer.first_name
    end
  end
end

And here’s the code for parsing some xml (OK, it’s not meta-programming but it is neat and tidy):

xml = File.read('posts.xml')
parser = XML::Parser.new
doc = parser.parse xml
doc.find('//posts/post').each do |post|
  puts post['title']
end

In Objective C, that would be over 7 million lines of code.

C# learned all of Java’s tricks and smoothed away its rough edges. It added lots of little tricks of its own to make it at least 9% better than Java. But its big, new trick is LINQ.

LINQ is essentially a functional language rammed right in the middle of a curly-braced imperative language. Once you get the hang of it, it’s amazing. I never did get the hang of it though and wrote all of my LINQ by typing it out in longhand and then clicking the helpful green squigglies that cause Resharper to turn this:

public IList<Album> FindAlbumsToGiveAway(IList<Album> albums)
{
  var badAlbums = new List<Album>();

  foreach (Album album in albums)
  {
    if (album.Genre == "Country")
    {
      badAlbums.Add(album);
    }
  }
  return badAlbums;
}

into this:

public IList<Album> FindAlbumsToGiveAway(IList<Album> albums)
{
  return albums.Where(album => album.Genre == "Country").ToList();
}

or, more ambitiously, into:

public IList<Album> FindAlbumsToGiveAway(IList<Album> albums)
{
  return from album in albums
         where album.Genre == "Country";
         select album
}

if I was in a functional mood (example stolen shamelessly from Alvin Ashcraft).

To achieve its lofty status of 9% better than Java, C# has had to add about 83% more syntax and therein lies its downfall. There is no way that one person can fit all that syntax into their brain unless he dedicates a lifetime to learning it, and why would anyone do that when there are so many finer languages to learn?

Less syntax is more, et cetera paribus, and this:

frequency = {}

is nicer than this:

Dictionary<string, int> frequency = new Dictionary<string, int>();

which brings us to Python, the language where whitespace is syntax.

At first blush, significant whitespace is Python’s big trick. There’s no need to add loop delimiters; just indent correctly – and you were going to do that anyway, right? – and Python will know what you mean. Once you get used to it, indenting loops is just so easy and obvious that you wonder a) why all the other languages didn’t copy it years ago and b) if Python has a better trick for me to learn.

Since Ruby, I am no longer impressed by parallel assignment,

a,b = 2,3

or generators,

def fib():
     a, b = 1, 1
     while True:
         yield a
         a, b = b, a + b 

sequence = fib() 

sequence.next()
>>> 1 

sequence.next()
>>> 1 

sequence.next()
>>> 2

or default values for arguments,

def f(a, b=100):
  return a + b

f(2)
>>> 102

or the myriad other ways that Ruby and Python are more pleasant to use than Java or C# (OK. I am a still a little bit impressed by generators).

List comprehension is a nice little trick,

numbers = range(1..100)
squares = [x*x for x in numbers]

but it’s not dramatically better than Ruby’s collect method,

numbers = 1..100
squares =  numbers.collect { |x| x*x }

or C#’s,

var numbers = Enumerable.Range(4, 3);
var squares = numbers.Select(x => x * x);

(OK, it’s a lot better than C#’s)

In fact, Python is so similar to Ruby that I feel forced to compare based on æsthetic terms alone and, æsthetically Python loses big time. If Guido and Matz were cousins, Guido would be the awkward, bookish cousin who is perfectly happy typing underbar underbar init underbar underbar open paren self close paren colon instead of initialize. Python has a strong mark of the geek about it.

Python also throws a lot of exceptions and you can barely shake a stick without causing a ShakenStickException. I mean, honestly, what is exceptional about getting something from a hash without checking to see if it’s in the hash first? Even Java gets that right, for Gosling’s sake!

Python’s inclination to hurl exceptions at the slightest provocation has cured me of the last traces of a youthful folly that said you should write the happiest of happy paths inline and put the rarer cases in exception handlers. Exceptions are nasty things and shouldn’t be tossed around lightly and guard clauses are not much better. The PragProgs (again) have a coding kata that requires you to minimize the number of boundary conditions in the implementation of a linked list. It’s a fine aspiration and finessing boundary conditions seems to result in less complexity and complexity is where the bugs hide.

The one Python feature that I haven’t seen anywhere else is the tuple. They are said to be magnificent and the distinction between

[1,2,3]

and

(1,2,3)

is allegedly profound but so far the significance escapes me. I’d be delighted if a commenter would help me understand or point me to some other feature that would make them choose Python over Ruby.

All this harsh buzz over Python might make you wonder why I would be foolish enough to decide to choose Python rather than Ruby at my new gig. The answer is that there is a specific library, nltk, I needed to use.

The natural language toolkit does cool stuff like this:

text = 'Mary had a little lamb. Its fleece was white as snow.'
sentences = nltk.sent_tokenize(text)
>>> ['Mary had a little lamb.', 'Its fleece was white as snow.']

which is harder than it looks. Once you have your sentences, you can find the words and, teleporting back to 6th grade language arts (assuming you grew up in America) or first year Latin (if you didn’t) you can analyse the parts of speech with:

words = [nltk.word_tokenize(sentence) for sentence in sentences]
>>> [
  ['Mary', 'had', 'a', 'little', 'lamb', '.'],
  ['Its', 'fleece', 'was', 'white', 'as', 'snow', '.']
]
parts_of_speech = nltk.pos_tag(words[0])
>>> [('Mary', 'NNP'), ('had', 'VBD'),
    ('a', 'DT'), ('little', 'RB'), ('lamb', 'NN'), ('.', '.')]

Hmmm. I think Mr Hickey would’ve gone with adjective rather than adverb for ‘little’ there. So would I. Anyhoo…

Once you have your parts of speech, you can diagram the sentence automatically (ssshhhh. Don’t tell your middle school kids):

That’s gotta be handy for something, right?

Now that we are stuck with Python, we get to wrestle with Django which is like Rails but brought to you by the same people that thought def __init__(self): was a good idea. I’m sure it’ll be great when it catches up with the state of the art but, Dudes! A separate language for templating!? I’m already learning a new language? You’re gonna make me learn another one for generating HTML? Didn’t you learn anything from JSP?

I think the folks who decided that separate languages for templates are descended from the folks who thought separate drinking fountains were a good idea. Is it really easier for designer folks to type

<ul>
{% for slimmer in slimmers %}
    <li>{{ slimmer.name|lower }}</li>
{% endfor %}
</ul>

than

<ul>
{% for slimmer in slimmers %}
    <li>{{ slimmer.name.lower() }}</li>
{% endfor %}
</ul>

Suddenly that significant whitespace business doesn’t seem so clever, does it? But, seriously, separate is rarely equal when it comes to template languages and the soft bigotry of low expectations hurts those it aims to help.

Template languages are the one area where the microsofties are ahead of the game with their Razor template syntax. It reduces the number of angle brackets and other unwanted syntax by 83%. Guaranteed!

<ul>
@foreach (var slimmer in slimmers)
{
   <li>@slimmer.name.ToLower();</li>
}
</ul>

How, you might wonder, if you know all these languages, are you supposed to keep all the various syntaxes straight? The plain answer is… I don’t. I immediately forget everything I knew about the previous language about two weeks after I stopped using it. That makes for embarrassing interviews when they ask me a Java question and, despite having 12 years of Java on my resume, I can’t remember how to construct and initialize a List, or is that a Vector? Or an ArrayList? One of them, anyway.

Fortunately for the forgetful among us, there is JetBrains. Even more fortunately, they have just released a brilliant Python IDE, PyCharm, to go along with the also brilliant, RubyMine and IntelliJ. They also have the brilliant Resharper for the microsofties but you have to use it inside the not-quite-so-brilliant Visual Studio and they don’t get along entirely well together. They both enjoy a lot of memory consumption for a start.

PyCharm amazes me a little bit every day despite my 10 years of being amazed by JetBrains. The type inference system is, frankly, spooky. PyCharm knows the type of a variable that I merely whispered to a colleague the day before and knows all its methods and parameters, what it likes to have for lunch and its taste in science fiction. It handles renaming and more sophisticated refactorings even better than Resharper and it doesn’t even have .NET’s type system to help it along.

So. Python.

To summarize:

  • It’s not quite Ruby.
  • It’s jolly excellent at text mining.
  • It’s a lot nicer than C# (except in html templates) or Java.
  • PyCharm. Oh yeah.

I’m happy with our choice so far but ask me again when I get good enough to stop needing to refer to my cheat sheet every ten minutes. I might have a more informed opinion.

Proudly Powered by Wordress

I was bored with my wordpress theme and Stu’s fresh look made me decide it was time for a refresh. This is my third theme and I wanted to go right back to basics this time rather than copy an existing theme.

I started from the most basic theme template I could find – Starkers – and converted it to use all new html5 tags.  Starkers has no CSS, so I was starting from scratch.

Here, for posterity, are my three themes side by side.

Best part of the whole exercise? I have confirmed once and for all that PHP is absolutely the nastiest programming language I have ever come across. I don’t get why it is so popular at all. Debugging wordpress is like doing a surreal jigsaw puzzle where you are looking for a brightly coloured machine tool to match the giraffe. If there is an organizing principle, I couldn’t find it. It seems completely random whether it grabs markup from a template or spits it out from a function or a widget or a plugin. It’s amazing that WordPress is so good.

I’m not quite done yet. I have a few weird tags left to style. I want to do something with responsive design and I want to do something special for ipad and iphone. When I am done with all that, I might make it work on IE ( < 9.0 ). Google Analytics says I get hardly any visitors with IE (72% of visitor time on my blog comes from macs and ipads!) but my mum has IE so I either need to make it work or fly to England to install Firefox for her. That's probably the cheapest option to be honest. PS. If those side-by-side images are still aligned vertically when you read this, it's because I haven't figured out how to style the image gallery yet. I didn't even know wordpress had a gallery plugin until just now. PPS. If anyone needs a site built in wordpress - find someone else.

I’m at my peak

The Hacker’s Diet gives us two tools for bringing wayward eating under control. The first one is the little red dot that says your weight is trending upwards.

As you can see, i’ve been ignoring the red dots for too long now. When you ignore red dots, your daily deficit turns into a daily excess. A daily excess of 212 calories is equivalent to a (large) beer every day which is about what I have been having.

Luckily for me, today was the day I hit my peak weight and, I pledge, I will be back to a deficit within two weeks. Within a few short months I will weigh less than I have fror 20 years.

You watch!

My Second Rant About Playlists

Rhapsody’s service has been spotty recently so I thought I’d try Spotify to see if it is all it is cracked up to be. And the verdict is…

…it’s OK.

But like every other music app I have ever tried it doesn’t do one the thing I want a music player to do.

I want to listen to music that I like.

Spotify excels at playlist management but I don’t want to manage playlists. I want to listen to music.

Here’s a reminder of why playlists suck:

Playlists are very seductive at first. You think Oh yes. I’ll build me a playlist with all my favourite songs. But then, after the third time you play it. You start thinking Oh man! This again!? I’m gonna build me another playlist. Then I’ll have two.

Before you know it, you have hundreds of playlists called things like Early English Folk (I) and Early English Folk (II) and you are spending all your time managing your playlists which, by the way, is exactly what the people who make the playlist managers want you to be doing.

I had a go at writing my own Rhapsody client a couple of years back but dropped that when a) Rhapsody started suing everyone who used their API to build apps and b) Rhapsody made an iPhone client that didn’t suck.

My dream didn’t die though and I am still in the market for an app that will play music I like. Here’s how it will work.

I search for some music, say, Gogol Bordello and play some “Top Tracks”.

I hit the button “Play more stuff like this” and it’ll find some Firewater or The Pogues.

After a while, I get bored with gypsy punk and play some Mediæval Bæbes instead. When it gets to Gaudete, I’ll tell it to play more like this and it’ll drift over into some Steeleye Span or some Fairport Convention.

When I use the app a few days later, I won’t need to tell it what to play because it will already know what I like. But, if I want to hear 23 versions of John Barleycorn I can do that too (this is where Pandora falls short).

Is there an app out there like this? Maybe someone has done something on top of the Spotify API? Don’t make me write it myself!

Hacker’s Diet

The Hacker's Diet Online

I’ve been following John Walker’s Hacker’s Diet for about nine months now. I love the simplicity of it.

“Anyone can control their weight. It’s a simple matter of balancing calories.” – John Walker

Mr Walker’s book is a fun, simple read. If you are technically inclined and you want to lose weight, you should certainly read The Hacker’s Diet. The book introduces a model of the human body as a rubber bag full, mostly, with water. Every day stuff goes in and stuff comes out, but the rubber bag always obeys the laws of physics.

If you eat more calories than you burn, you gain weight at a rate of a 3500 kCal/lb. Simply count the calories you eat and subtract the calories you burn and the resulting number will tell you how quickly you will gain (or, hopefully, lose) weight.

But I think it’s even simpler than that.

If you weigh yourself every day, you can quickly figure out whether you are gaining weight. If your weight goes up, you are eating too much. Eat less.

Ok, ok. It’s not quite that simple. Your weight can vary by a couple of pounds each day as you retain water (or, as John Walker delicately puts it, solids); but if you plot the moving average you get a surprisingly stable trend line. From that trend, you can figure out your daily excess (or deficit) and decide to eat more (or less) accordingly.

People who work in software development use a planning technique called yesterday’s weather. The idea is based on weather forecasting. Imagine a computer that monitored the humidity and the temperature and pressure and a thousand other variables and used it to predict the weather with an accuracy of 82%.

It turns out that, if you just predict that today’s weather will be the same as yesterday’s, you’ll be correct about 70% of the time. That’s close enough for most purposes and it saves you a really expensive computer.

In my case, the prediction that I’ll eat about the same number of calories today as I did yesterday saves me lots of tedious calorie counting. With a little practice, I got quite good at knowing whether I was eating too much or too little and adjusting my intake accordingly.

Here’s my trend since last October:

You can see from the chart that I lost weight pretty steadily for several months. My daily deficit held steady at about 250 cal/day for most of that time. 250 calories is about a bagel a day and represents the loss of a pound every two weeks. Since I don’t really like bagels anyway (or french fries, or bread, or candy) it was easy to stop eating them. I hit my target weight about a month ago and, since then, my weight has crept up a little (I don’t like bagels but I do enjoy beer).

The little red dots are a warning sign that I might be eating too much (notable red dots: The Captain’s Table Dinner in January and Piratefest in July) and the prominent red 74 says that I need to have half a pint less beer at Quiz Night.

Like any good hacker, I decided that I didn’t like any of the weight trackers out there so I wrote my own for my iphone (it was also an excuse to learn Objective C). I might decide to stick the app on the app store one day but, for now, it’s just a bit of fun that I am sharing with some friends. Ping me if you want to play along too.

Deal with this

I’m trying to learn Objective C at the moment so I can write my first iPhone app (this time next year, we’ll be millionaires).

Objective C is just about the ugliest language that I have ever come across (and I have coded in COBOL and Ada).

That makes it all the more frustrating to read the trailer for the new version of Ruby.

  suits = %w{ C D H S }
  ranks = [ *’2′..’10’, *%w{ J Q K A } ]
  deck = suits
  .product(ranks)
  .map(&:join)
 
  hands = deck
  .shuffle
  .each_slice(13)
 
  puts hands.first.join(“, “)

Could it be any simpler?

(I don’t know what the ampersand thing does either).

It’s hard to believe that my beautiful iPad is full of nasty Objective C.