Coding tests as an interview smell

Recently I had an opportunity to interview with a company that I really respected. Normally I wouldn’t consider taking any kind of role or gig (as I run a small business), but the problem space was particularly interesting and the company seemed progressive and unique.

But then came the coding test. And I bombed. Hard.

It might surprise people that an engineer of my caliber would “fail” a coding test. After all, I have extensive contributions to open-source software, have written applications for major corporations, written books on software development, and I’ve been working in software development for 24+ years.

But execution on a coding test isn’t about skill – it’s about how you perform on a test.

The problem with standardized coding tests

A standardized coding test doesn’t measure the ability of a programmer to program. It measures their ability to test well.

My test was given on HackerRank, a site that purports to evaluate developers on their skill, and even “rank” them. You log in, use their IDE, answer problems against a set of tests, and when you’re satisfied you submit your work. The test I took had a two-hour limit.

There’s an extensive list of problems here:

• You’re using an unfamiliar IDE without the benefit of standard debuggers.
• The tests are hidden from your view, meaning that you can’t see the data inputs being used to test your code; you only know that your test passed or failed.
• The test is timed, creating an artificial deadline for potentially challenging problems.
• You only have the input of the prompt, without the ability to ask clarifying questions or seek additional information.
• The test logs uses of other tabs or programs, meaning that if you refer to documentation (as I often do), you’re penalized. Most developers I know haven’t memorized the entire PHP code base.
• The test utilizes AI to determine your rank, and to evaluate your performance, which is problematic and fodder for another article entirely.

As you can see, there are numerous problems with this type of coding assessment.

“But why did YOU fail?”

The short answer as to why I failed: I am not built to take or pass standardized tests.

The longer answer: as someone with ADHD and testing anxiety, I already feel extreme pressure when taking a test. This leads to challenges with execution, recall, and memory. It’s hard to get “in the zone” when there’s a ticking clock in the corner that will ultimately expire and end your testing session. You’ve always got to be cognizant of the time, or you might run out of it.

In contrast, when developing software taking as much time as you need is a help, not a hinderance. While perfection may be the enemy of the good and shipping matters, it’s also critical that developers take all the time required to understand the specification, develop the code, test the code and ship it. This is the environment that I’m used to working in, and in fact, every developer should expect at work.

Despite my disability and anxiety, there weren’t any accommodations made on the exam. This likely violates United States law (and is also potentially a topic for another post), but most certainly put me at a disadvantage over other applicants who might test better or not have ADHD.

A better path

I am firmly convinced that developers shouldn’t be subjected to coding tests as a prerequisite for getting a gig. But there is a better path forward; in fact, there are several.

Body of work

For a developer with an extensive body of work, ask to see some of it. Evaluate that work. Do a code review. See how they work – do they write tests? Do they use SOLID or your preferred framework well? Do they understand basic programming concepts?

Some developers will argue that they can’t share their work because it’s under NDA or an employment agreement. I understand this concern – and I don’t think that anyone should be required to contribute to open-source just to get a job. For these developers, a body of work may not be available. But this isn’t the only way to evaluate developers.

Pair programming

Taylor Otwell of Laravel mentioned on Twitter that pair programming was one of the ways that he could tell if someone was good at programming. I think the idea has merit.

There are two ways to do this: either you “drive” or you let the candidate. Leave the decision up to them.

If the candidate drives, you have to take into account nerves and typos. People often program worse when nervous or when typing in front of people (fat fingers when watched is a real thing!). However, you can see (and suggest) whether or not the candidate has some of the skills that you value. And you can see how they think, which is far more valuable in a role than the code they actually produce.

If they have you drive, you can see how they think in the suggestions they make. Are they assertive? Are they open to your ideas? Are they open to discussing and debating? Do they catch and report obvious typos, logic errors, etc? Don’t intentionally make mistakes to “test” the candidate, but do expect that the candidate will highlight issues that you may miss, or have different ideas than you.

In both cases, it helps to have a clear outline of what you’re looking for and communicate this to the candidate. Timid candidates may not be as assertive as you might expect or desire; they may be anxious, and this will lead to less-than-optimal outcomes. But establishing expectations beforehand can definitively help with whether or not the exercise goes well.

Strategic interview questions

Even if a candidate lacks a solid body of work or completely bombs a pair programming exercise, I still recommend having a cursory interview. Why? Because the candidate may still be a good fit.

This might seem counterintuitive, but some of the best hires I’ve ever made have been people that performed the worst on standardized assessments.

Here’s my standard list of interview questions and the reasons I ask them. These interview questions are strategic, and offer insight into a candidate’s history, along with helping to understand their skill level. Here’s some questions I ask.

What do you like about PHP? What would you change about PHP?

Skilled developers will be able to tell you their favorite and least-favorite things about a programming language that they are intimately familiar with. The answer doesn’t matter; the ability to answer the question is the important thing, provided the candidate provides a valid reason for their answers.

Tell me what you love about <framework> and what you’d change.

Same reasoning as above: if you know a framework well, you’re going to have loves and gripes.

Let’s do a design exercise. <Insert realistic problem here>

I ask candidates to do an architecture exercise with me that mirrors the realistic work they’ll be doing. No whiteboarding, no coding. Just talking through a solution. Why? Because the ability to think outweighs the ability to code every single time. I’m evaluating for whether they’ll ask questions, propose solutions, make adjustments based on new information, and hold a concept in their head while we discuss.

What’s your favorite API?

Again, testing how a candidate thinks and what they love to work with. Candidates will have worked with an API in the past, and they should have strong feelings about one or two that they’ve used.

Language-specific questions

I also typically ask questions about the language they’re going to be developing in. For example, I’ll ask about latest features in versions of PHP that have been released in the last two years, as well as about tooling. If they don’t have a strong opinion on frameworks, testing libraries, static analysis, the namespace operator in PHP, or other topics, they may either not have a good grasp of the overall language or they may not have strong opinions about programming. Either way, strong but flexible opinions are important to me.

Final thoughts

Coding tests are not really a useful tool for developing an understanding of a candidate’s ability – it’s a useful tool for determining if a candidate can develop software in an unfamiliar IDE without typical tooling in a fixed time window. You want to understand the candidate’s abilities, not the testability of any candidate.

My final thought is this: you always want to interview qualified candidates. Even if they bomb the pair programming, even if they don’t have a body of work, even if they stutter or stammer or struggle. Be gracious. Be patient. Understand that anxiety is real, and that candidates who are desperate for a job in this economy and climate will likely be terribly nervous.

“But this takes enormous amounts of time!” Yes. It does. And if you’re posting a job ad you owe it to the applicants to take that time. After all, they took the time to apply, to write a cover letter and customize their resume and read your job description. You owe it to them to interview qualified candidates and make a human determination.

Coding tests are used to weed out candidates quickly, and yet deeply flawed.. It’s disrespectful to candidates, potentially violates the law, and most certainly doesn’t ensure you get top programmers – just top test-takers.