Perfect Example

Here’s a perfect example of how the computer field is broken:

In a post at Coding Horror, based on earlier posts at Imran on Tech and Raganwald, the author parrots what the others state, that programmers can’t program. With lots of exclamation points.

Why make such a breathtakingly grandiose claim? Because of what happens in interviews. It would seem that the originator of this newest fooflah created a series a tests given during the interview process and found:

After a fair bit of trial and error I’ve discovered that people who struggle to code don’t just struggle on big problems, or even smallish problems (i.e. write a implementation of a linked list). They struggle with tiny problems.

So I set out to develop questions that can identify this kind of developer and came up with a class of questions I call “FizzBuzz Questions” named after a game children often play (or are made to play) in schools in the UK. An example of a Fizz-Buzz question is the following:

Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.

Most good programmers should be able to write out on paper a program which does this in a under a couple of minutes. Want to know something scary? The majority of comp sci graduates can’t. I’ve also seen self-proclaimed senior programmers take more than 10-15 minutes to write a solution.

Jeff Atwood of Coding Horror also goes on to quote others who run into the same problems: interviewees can’t seem to do even the simplest coding tasks during interviews. These gentlemen completely ignore the environment and focus on the grossest of generalities:

Programmers can’t program.

Here’s a clue for you: I don’t do well in programming tasks during interviews, and I’ve love someone to come into my comments and tell me I can’t program based on this event. No, I’ve only faked it while working for Nike, Intel, Boeing, John Hancock, Lawrence Livermore, and through 14 or so books–not to mention 6 years of online tech weblogging.

In fact, you’ll find a lot of people who don’t necessarily do well when it comes to programming tasks or other complex testing during job interviews. Why? Because the part of your brain that manages complex problem solving tasks is the first that’s more or less scrambled in high stress situations. The more stress, the more scrambled. The more stressed we are, the more our natural defensive mechanisms take over, and the less energy focused into higher cognitive processes.

Why do you think that NASA, the military, and other organizations training people for high risks jobs spend so much time in simulation? Because they want the tasks to be so ingrained that in a stress situation, the people’s responses are almost automatic.

If you add the potential for embarrassment on to the strong desire to do well, the need to get the job, toss in a panel of arrogant twats sitting around a table looking directly at you while you do your little tests, and you have the makings of an environment that almost guarantees the elimination of many fine candidates.

Who does well in these kinds of testing situations? Good testers, the supremely self-confident and equally, typically arrogant, and the people who don’t care: none of which is necessarily the best candidate.

The whole purpose of tests such as these are not to determine if a person has programming capability–how can one stupid test determine this? What these tests do, though, is add to the self-consequence of the person doing the interview.

“I can do this, but all these people can’t. Therefore, I’m so much better.”

It’s also a lazy interview technique, which shows that HR associated with the company doesn’t give a crap about the IT department.

Some justify such tests with, “We need people who can do well in stress situations.” Bilge water.

The stress one goes through when one is an outsider faced with a bank of insiders, is completely different than the stress an individual goes through when they’re part of a team trying to fix a problem or roll out a product. Comparing the two is ludicrous, and nothing more than a demonstration of completely two-dimensional thinking: one form of stress is completely the same as another. My god, no wonder we’ve had few tech innovations lately if this is demonstrative of leadership in IT.

Having candidates bring in samples of code and having the interviewer and interview team review such, and question why decisions were or weren’t made is an excellent way of getting insight into the person’s problem solving skills, without the trained dog and pony show. Asking a person what approach they would use in a situation is superior to doing a random memory test on keywords. Providing applications and having the person provide their own critique is an amazingly effective way of getting insight, not only into their problem solving skills, but also into their personality. If they point out errors but do so in a thoughtful manner, it’s a heck better than doing so in as scathing a manner as possible.

Looking at past applications or effort is another effective approach. New programmers with no job experience can provide pointers to open source applications; experienced people who have worked in an NDA situation can provide pointers for discussions and work online: heck, Google the person’s name–that will tell the interviewer much more about the person than a silly programming test.

That primitive techniques such as the abysmally stupid “FizzBuzz” approach are used shows that companies are still missing out on good people because they have idiots doing most of the interviews.

And making the leap between how people do on interviews into such grand claims that programmers can’t program demonstrates that idiocy travels up the food chain.

You know what’s especially humorous? All the people who solve the test questions in the comments. What possible reason would a person have to do such a thing? It’s completely irrelevant to the environment in which these so-called tests are given. This no more shows that these people can program, then it shows that the other people can’t.

The lack of logic in this whole thread is amazing.

What’s less funny, though, is the slavish adherence to Joel Spolky’s elitist crap. Joel runs a smallish computer company with limited products: what the hell makes him the definitive answer on these topics? Perhaps the people should spend less time making pronouncements, and more time developing independent thinking skills.

Many of the comments in the Coding Horror post do mention these concerns, and provide other effective approaches to interview. If the people who create these tests will actually read these responses, some good will have come from the discussion.

I have found, though, that people who write these kinds of tests aren’t always willing to considering other options. The other approaches just aren’t ‘clever’.

This entry was posted in Stuff, Technology. Bookmark the permalink.

32 Responses to Perfect Example

  1. Ethan says:

    Back when I performed interviews, I developed a fairly decent technique: My applicants were already working internally, and were interested in a change. I assigned them a fairly simple task: Write out a process document explaining how to pour a glass of milk. I emphasized that I was not looking for a letter perfect carbon copy of what my existing team routinely produced, although that material was available on the corporate intranet for those savvy folks who wanted to make a better impression. What I did want, and occasionally received, was a document that gave an inkling that the applicant a) had the excellent written communication skills as per claim on resume, b) could follow instructions, c) could think through the variables associated with pouring milk (which milk? where?), and most importantly, d) could meet deadlines. I gave 5 business days to complete this fairly simple assignment, which could be hacked together in all but the most screwed-up of work environments. Plus they could be completed elsewhere, without the stress of me looking over their shoulders. The people who gave a rip completed it, and those that didn’t, didn’t. This may sound unfairly broad, but I assure you, follow-up discussions with the applicants validated this.

    Conversely, back when I was a temp in the Chicago area, for the time that I was, the end of the line came when the agency sent me to a local factory. The interviewer wanted to watch me thread nuts onto bolts, put a round block in a round hole, verify that my shoes were on the correct feet, etc. I knew that if I went through with the interview process I’d be royally screwed. I walked out, and doubled my efforts at finding a “real” job.

    I can see the value in testing an applicant’s mettle, but within reason. I agree that the aforementioned “FizzBuzz” test really doesn’t prove that an applicant has the necessary skill set for the position. I’m not a “techy” guy, nor do I conduct hiring interviews at the present time, but I think I’d be more interested in evaluating past performance (there are many ways to do this without having to break NDAs, etc) than offering goofy tests. But that’s just me.

  2. Bud Gibson says:

    I agree with the testing based on past projects approach. Research shows that the best predictor of a person’s future performance is their past performance. Interviewing, particularly stress interviewing has proven much less effective.

  3. Scott says:

    Another blog post points out that even though everyone reading the blog post who decided to try and write out the solution had no stress and plenty of time to read the “spec” and write out the code. Most of them STILL didn’t satisfy the requirements.

    http://www.haacked.com/archive/2007/02/27/Why_Cant_Programmers._Read.aspx

    I’m generally less interested in finding out a person technical merit than I am in finding out if this is the kind of person I can work with (or work for) from day to day. I commented on another blog, “Where are the pithy tests that will tell me if this person will call my client ‘assface’ during a meeting?” Programming question, actual questions about code, I usually pull out if I have doubts about the persons technical qualifications or suspect they exaggerated more than usual on their resume. I’ve pulled examples out of my real life experience and asked people questions about how they would solve it during interviews to gain insight into their problem solving process. Not the “How would you move Mt. Fuji” types of questions, but “How would you normalize this table?” kind of stuff. Again, usually if I’m unsure of their actual qualifications for the job.

  4. Scott Reynen says:

    When I interviewed for my current day job, my first non-freelance development job, I was surprised there was no technical interview at all and I wasn’t shown their development tools until I asked to see them. At the time I was running an entirely open-source website, so they could see plenty of code I’d written, but still, they had no idea how I wrote it.

    Now we’re hiring what will be my coworker, and I’ve pushed for a technical interview. I don’t expect anyone to write a flawless script on paper in a few minutes, but I see a lot of value in sitting someone down in front of the development environment they’d be using on the job, giving them the resources they’d have on the job (e.g. the ability to search the web or ask others for help), and seeing how they would approach actually doing the job.

    I think you’re right that it’s a challenge to make an interview more like an actual work environment, but I don’t think it’s impossible, and when it happens, it seems like the perfect interview. What better way to know if I want to work with someone than to try working with her (and vice versa)?

  5. Ethan says:

    “I agree with the testing based on past projects approach. Research shows that the best predictor of a person’s future performance is their past performance.”

    Not to be all horn-tooty, but to clarify part of my previous comment, the “assignment” that I used to give out allowed people with the aptitude for the job to shine, even if they lacked formal on the job experience. Things can get really insular (no!) if selections are based solely on past performance. But thinking in terms of interviewing for a “tech” position, I would either like to evaluate past work to judge competence, or if that is not possible, I’d come up with a similar take-home project. Especially if the job entails a multi-stage interview process. Otherwise I echo Scott’s comments to the effect that attitude/team harmony is a big X-factor, to me.

  6. Never underestimate the stupidity of _____________

  7. Charles says:

    I think the true horror here is that a programmer is adopting the Human Resources mindset. HR people believe their true calling is to weed out bad people, once they’re gone from the pool of candidates, the remainder must be qualified by default. This is not the way to find highly qualified candidates, it’s the way to find the least-worst candidates.

  8. Who does well in these kinds of testing situations? Good testers, the supremely self-confident and equally, typically arrogant, and the people who don’t care: none of which is necessarily the best candidate.

    This is the only part of your post that I disagree with. It’s entirely possible not to have undesirable traits and yet not get so nervous during an interview that they one loses one’s higher cognitive functions.

    I agree that the most effective approach for judging pure ability is to have the interviewee bring in a limited-size sample of their work, a piece of code roughly 100 lines long, say. You can ask a lot of interesting questions about that; and particularly questions beyond the examining the code itself. Why did they choose this piece of code? How many samples did they weigh against each other? How hard was it to decide? What made it hard or easy? Examining the code itself and asking questions about it gives you some insight to their aptitude and mindset; examining the choice of code gives you a insight into their own view of their own aptitude and mindset.

  9. shelleyp says:

    Good comments, all, including Mr. Braithwaite’s posted response. I did want to say that you’re right, Aristotle, and my arrogant dismissal of those who do well is just as harmful as the arrogant dismissal of those who don’t do well.

  10. Tom Clancy says:

    That’s a heck of a lot of text without any attempt to look at the other side of things. How do you go about interviewing people to join a programming team? What seems like arrogance to you might be hard-won self defense techniques.

  11. shelleyp says:

    Tom, I’ve been in the industry 22 years. I have never not looked at the ‘other side of things’. I’ve also been involved in many interviews, as well as being leader of teams made up of people who I interviewed.

    I, and others in this comment thread, give alternative approaches other than this ‘test’ — which ultimately measures a person’s ability to do well on tests more than anything else.

    I would suggest that the people using these ‘self-defense’ mechanisms need to rethink exactly what they’re looking for in new employees. Frankly, I can learn more about a person in a half hour conversation than in all the tests in the world.

  12. I’ve seen the exact opposite — people who are totally relaxed, confident, and talk like a senior salesperson even though they’re applying for a technical job. Sometimes I get this strange feeling — the answers, although they come back at me very smoothly, seem somewhat strange. Going into more detail, and using a simple test as an example, exposes this way more often than I would like it to happen.

    It’s very easy to distinguish errors made because of stress and those made because of total ignorance. Or are you seriously suggesting you’d forget about loops and division and simple if-else constructs in all programming languages you know because of stress?

    If so, you’d not be a good candidate, too — just for different reasons.

  13. shelleyp says:

    But what does it show that they can do a simple loop or division?

    There were a good half dozen suggestions given here other than a test and beyond just a chat. Anyone of these will show not only that they can code, but give insight into thought processes, attitudes, interests.

    You have to interact with the person being interviewed. When you sit them at a computer and watch them code, there is no interaction. That’s a way of having the computer make the choice for you, because you don’t want to go out on the limb and make it for yourself.

    Just the of code and application they bring in, as a sample, tells you an enormous amount about the person. Sitting down and talking about it will show you if it’s bull, or if they did the app, or if they know what they’re talking about. But it’s in an ‘interest’ and non-confrontational manner. This then helps you determine, could you work with this person as a team mate?

    You should never have a person come in for an interview, when they’re not being interviewed. Time in front of a computer? That’s a waste of everyone’s time.

    If you want code proof, then ask that they provide some form of certification. Most field types now have this. That way they can do the ‘code part’ separate from the interview.

  14. Steve says:

    I think you missed a key point behind the FizzBuzz problem. It’s so easy that even a decent developer who’s HORRIBLE under this kind of pressure, should be able to do it.

    This FizzBuzz problem isn’t supposed to tell you how well the person can code. It’s only supposed to be an indicator whether the person who can code the most basic of programs.

    If you want to rant about how more more complex tests are poor indicators for rating developers, you have my ear. But to specifically suggest that the FizzBuzz problem is a poor tool for weeding out incompetent developers? Please.

  15. shelleyp says:

    Steve, did you happen to read what I wrote in the post or in comments?

    If it’s so simple anyone can do it, what sort of filter is it?

    If someone doubts the candidate so much they think they can’t even do a simple programming job, why interview them?

    Any one of the other options I, and others, mentioned would work much more effectively. The more I hear about “FizzBuzz” the more I disdain it’s use.

    Eventually, it could be a good way of weeding out employers: who wants to work for a company who uses such as an interview technique?

  16. ARJ says:

    Shelley, I completely agree with you. I had one of those myself for the current job, and I almost stuffed it, simply because of the way the question was put to me. It started out, “Assign two integers to two variables.” Yay, easy. I picked 0 and 1. This ended up making the rest of the question a lot harder: “Swap the two values of the variables without introducing a new variable.” I basically had to start from scratch with larger numbers to work out how to do it. It was a very stressful situation, and I only made it through because my rampant perfectionism and fussiness wouldn’t let me drop it, even after the interviewer was suggesting we move on. The bottom line being, I didn’t have all the information about the problem I had to solve, or I might have taken a different approach. I think my determination to finish did me more good than my success at solving the problem, and that’s easily demonstrated in many other ways.

    My partner just interviewed at Google after being contacted by a recruiter. He did swimmingly on the high level questions. The coding questions, not so great. When he produced sample code that he had worked on, the interviewers were more impressed. Still didn’t get the job, but at least we don’t have to decide now whether we want to pack up & move to Mountain View. ;-)

    Artificial coding questions in interviews don’t really reflect what development work you need to do day to day. Just like in the age old school complaint of “When am I ever going to use long division in real life?” I have never once needed to swap two variables without introducing a third in any paid development work I’ve done.

  17. Bobbie F says:

    There is something to this, but I really have think that if someone who claims experience can’t so the FuzzBuzz program in a language they are familiar with, something is wrong beyond stress.

    And yes, totally unqualified candidates can and do slip through purely “talking” interviews.

  18. With regards to #14:

    Nobody (I think claimed) that being able to do FizzBuzz proves someone can program.
    Not being able to code it means someone *can not*.

    That’s the whole point of it — filter out those who cannot even do something trivial like this, which you might otherwise only find out way too late.

  19. Danny says:

    If I was set a problem like that in an interview, and the interviewers didn’t like the fact I pulled up the relevant language tutorial in a browser to remind myself how the % operator works (which I can never remember), that’d be a good indicator I wouldn’t want the job.

    Not sure I’ve ever had a programming task in an interview, but I do remember a question from one for a sysops job: “what file attributes does Netware support?”. I probably should have known, but managed a fair guess. The interview was conducted by a college’s Head of IT and Head of Management IT. A while afterwards I asked the Head of IT if they knew the answer: “No way!”.

    Same place, later I had to come up with a test for tech support candidates. There it was important to make sure the person could still work with some intimidating twat leaning over their shoulder…

  20. Paul says:

    While I agree that it’s a loathsome problem and a pretty lame question, I’ve adopted the question myself.

    Some environments you don’t get to choose your source of candidates. Where do you get your candidates from, Shelly? I work for a multinational bank where we are required to use specific vendors and specific recruiting firms that have benn vetted by the bank (specifically the HR department). They don’t know how to properly check out a recruiting firm, so they sit on the paperwork for 6 months and hope the request goes away (which it usually does when the position is filled). If it doesn’t, then they just approve it. So usually the only shops approved are those who have some sort of buddy relationship with someone in the bank.

    Now, there’s a shortage of programmers, especially in the financial world. We’ve hired people who have openly admitted that their firms counsel interviewees, provide mock interview sessions, and train them to answer questions appropriately. They really don’t care what kind of quality applicants they send us. If one makes it through the cracks, they get paid. If a bad one makes it through the cracks, it’s our fault for not interviewing them properly.

    Now anyone who is halfway smart knows how most of their past projects worked, even if they were mostly on the fringes. And most people looking for a job don’t currently have one. So the candidates themselves are trying to aggrandize their roles, responsibilities and talents.

    Last, to make matters worse, once they’re hired it takes weeks if not months to get them removed. During that time, we don’t have an opening for another position and everyone else has to pick up the dead wood’s slack.

    Now I have no say in any of this process. I’m not a manager and I have no aspirations to be a manager (I like to actually code too much). So i will never have any say in the above policies. I’m just asked to make sure people can do the technical aspects of the job.

    Without asking the basic questions like FizzBuzz, I’m left with picking the guy who can bluff the best. Since I started asking questions like it first, I haven’t had to sit through people trying to tell me that XML, databases and FTP are really just the same thing with different names (I sh*t you not, that’s the kind of drivel we deal with)

  21. shelleyp says:

    ARJ, Google came to mind when I wrote this. I actually had a discussion with a Google HR person on this very issue and how Google’s hiring procedures guarantee a homogeneous environment — not healthy. Google may be successful now, but it has not been having many truly innovative releases of technology in the last year or so.

    Paul, many candidates for jobs are people looking to make a change, not poor desperate losers down to their last dollar. Recruiting companies are not something I will work with, primarily because many don’t care if there’s a good match.

    And dead wood is just as likely to know how to do FizzBuzz, as not.

    What I don’t understand is people are saying that no, FizzBuzz really doesn’t test programming that much, but its a way of determining if a person can program at all. Do all of you really get so many people who don’t know how to write a line of code?

    Any one of the half dozen or so alternatives listed here is going to show if a person can’t tell the difference between FTP and XML (though I wonder how you coached your question to get this answer — again, you all might want to consider that your interview techniques are, perhaps, the reason ‘dead wood’ is getting through).

    You’re defending to the death your right to FizzBuzz. But none of the reasons make sense — any of the alternatives given would demonstrate the problems you can’t afford, without introducing interview anxiety and its effect. More than that, they’d also allow you to get a better idea if the person has good problem solving skills that go beyond just code, as well determining if they have a mind set compatible for the job.

    When you’re not talking with people, you’re not interviewing.

    Still, its your spot, your jobs, and if you want to FizzBuzz, FizzBuzz with pride.

  22. Jeff Atwood says:

    > Many of the comments in the Coding Horror post do mention these concerns, and provide other effective approaches to interview. If the people who create these tests will actually read these responses, some good will have come from the discussion.

    That’s one of the dirty little secrets of blogging: the discussion in the comments are often better than the original entry.

    But it’s wonderful either way.

    Sadly, the people who could benefit most from reading it are the ones most likely never to read a blog in the first place.

    > Do all of you really get so many people who don’t know how to write a line of code?

    Apparently so– that’s the problem. No barriers to entry and lots of demand. This is the problem I wanted people to discuss when I posted the original entry, but they were too busy posting comments with FizzBuzz coded in Haskell and Scheme..

  23. Jeff Atwood says:

    > If it’s so simple anyone can do it, what sort of filter is it?

    1) It’s not meant to be a challenge. It’s akin to asking an applicant for a position driving trucks if they know where the gas pedal is and how the gearshift works. And yet many people who apply for programming jobs cannot complete this very simple test. It’s shocking, I agree, but look at all the evidence. This is what happens.

    2) It’s a basic screening question that lets you weed out candidates early, before an on-site interview. Why waste everyone’s time?

  24. shelleyp says:

    This is the problem I wanted people to discuss when I posted the original entry, but they were too busy posting comments with FizzBuzz coded in Haskell and Scheme.

    Best laugh I had all day.

  25. This is the problem I wanted people to discuss when I posted the original entry, but they were too busy posting comments with FizzBuzz coded in Haskell and Scheme.

    Haskell? Scheme? A real programmer can write Fizzbuzz with composable functions that.. ah.. err…

    Boy do I have egg on my face. Thanks, Jeff and Shelley!

  26. shelleyp says:

    Scrambled or with a lovely hollandaise sauce? Sorry Reg, it’s Saturday. Saturday’s my day to be flippant.

    I was waiting on a SNOBOL implementation myself.

    Well, once we worked our way through the Haskell and Scheme, I thought the discussion was rather good. Win/win for all.

  27. Haacked says:

    > Do all of you really get so many people who don’t know how to write a line of code?

    You’d be surprised. One thing you learn, don’t post job postings on Monster.com or the ilk. I’ve made that mistake before as a manager (corporate policy so not really *my* mistake).

    I had people calling in who couldn’t speak a lick of English. I’m not talking about heavy accent. My mother is Korean with an accent the NSA couldn’t figure out. Accent is not a problem. Inability to answer basic questions about yourself is a problem.

    Also, I’ve had situations where I’ve received over 100 resumes. All with the same buzzwords. I can’t have a half-hour conversation with every one of them.

  28. Haacked says:

    But I should follow up and say I’ve also been on the receiving end of a FizzBuzz test. It sucks when not administered well. Definitely. I would rather have candidates email me their submission before granting a longer interview, as it can be an effective screening tool.

    Yeah, it sucks if you miss someone good who doesn’t test well. But a false negative is better than a false positive.

  29. Shelley says:

    I’d think the amount of time to review what they code down on paper or computer could be just as time consuming.

    One solution to much of this is to require certification. If your candidates must be Sun certified Java developers, than the testing part is done ahead of time, and separate from the interview. It’s not fool proof, but it will do the ‘tech filter’ portion, and without people having to code during an interview.

    Of course, not all skills have certification tests.

  30. Haacked says:

    Requiring certification seems like an even worse solution. Are you certified? Many of the best developers I know felt it wasn’t worth their time. Maybe it’s just an issue in the Microsoft world though.

    One way around the time to review code, and this takes a little more up front prep, is to send the user a “broken” console app with clear instruction on what it should do. Ask them to email it back. Run the app. If it prints out the correct value, they get a more in depth interview.

  31. Scott says:

    “One solution to much of this is to require certification. ”

    Save me from the graduates of the week long MCSD/E boot camps. The Java and Oracle certifications WERE mostly worth the paper they are printed on last time I checked. But I’ve never hired a Java or Oracle developer so I can’t speak from experience. But even then, certifications mostly just test your ability to take a test. Much like undergraduate college degrees just prove that you can be trained and graduate degrees prove the training took hold.