Andrew Rondeau

Software Developer, Audio Enthusiast

Note: The article made it to the front page of hacker news! Lots of great discussion there.


The software engineering field has controversy over using whiteboard coding in interviews, versus candidates doing take-home coding assignments. This is a false dichotomy, the problem is bad interviews.

Quincy Larson, in his post Why is hiring broken? It starts at the whiteboard, gives some examples of bad whiteboard questions. In his article, he slams the practice as either a form of hazing, or he gives examples of horrible questions that rely on the candidate memorizing arcane algorithms.

Yet, the bad examples that Larson lists are the opposite of what Joel Spolsky recommends in Smart and Gets Things Done. The only thing I can conclude is that Larson is really complaining about bad interviews, not bad technique.

In this post I focus on the tradeoffs between take-home coding questions, and live coding exercises, such as whiteboard coding.

Examples of Bad Take-Home Questions

After I wrote the first draft of this post, I actually walked away from a job opportunity that asked me to do a horrible take home exercise. Here's the assignment, summarized to respect the company's privacy:

This should take a few hours. Create a web-based message board. Use C# and ASP.Net Core, with Angular and Material design. Use a database. When someone adds a message, update all other open browsers in real time. To make it easy for you, you don't need to implement authentication.

This isn't the first time I've walked away from such a situation. Why did I walk away? In both cases, it was the same exact reason.

The exercise was supposed to take "a few hours" according to the hiring manager, but I'm extremely rusty with ASP. I haven't done anything real time since before WebSocket, nor have I used Angular or Material design. I anticipate that such a project would take one of their engineers more than "a few hours" to finish, and take me at least a day to come up to speed with all the various APIs and patterns.

At this point, we've only had one phone screen. The company hasn't invested much time in me, but they expected me to invest a lot of time in them. I hadn't met other members of the team, and I've never seen the product.

I wondered, what if I put a lot of time into this and make a novice mistake? Will some junior engineer reject me? Is this a sign that someone in a leadership role has unreasonable expectations? What if, at the end of this process, I just don't want the job?

Take-home assignments should never have learning curves. There are reasons when it's okay to request an alphabet soup of technologies, which I explain a little later.

Another situation where I've walked away was the "fix a bug in our open source project" task. Onboarding on a complicated piece of source code can take a few days. It can be longer if the source code uses a language or idiom that I'm unfamiliar with.

"Fix a bug in our open source project" always feels like hazing to me, because I just can't compete fairly with someone who's already worked in that codebase. I always worry that I'll put a lot of time into the take-home and not get an interview.

In another case, a hiring manager emailed me a candidates' take-home coding assignment and told me that I needed to interview the candidate, in-person, later that day. The take home question was fine, but the candidate's code was awful! We should have rejected the candidate instead of flying the candidate to the office. Even worse, I suspected that one of my colleagues was rude to the candidate when reviewing the code.

An Example of a Bad Whiteboarding Question

I once interviewed at a company where a young engineer gave me a simple question, which I quickly answered with a straightforward solution. The engineer then spent the rest of the time berating me for not including an insignificant micro-optimization.

The interviewer was just as rude as my colleague was when he reviewed the bad take-home assignment. I remember telling the recruiter that I didn't think I'd like the company.

In another case, I interviewed at a video streaming company where the whiteboard question required that I know a textbook algorithm that I would never use directly. In reality, the question was asked poorly. If they said, "here's the problem and here's the algorithm you need to solve it," it would have been a great question.

But, more importantly, a bad interview can happen with either a take-home assignment or a whiteboard assignment.

The Problem is Bad Interviews

It's always important to remember that an interview is a two-way street. It's just as important to sell the job to the candidate as the candidate must sell themselves to you. Normally, it's extremely hard to find software engineers qualified to show up for an interview, so running bad interviews means that the best candidates will just walk away.

Larson, in his blog post, explains a more disturbing trend:

Whiteboard interviews are akin to a hazing ritual — a rite of passage that one must endure before joining an organization — because everyone else there did. “I went through a battery of whiteboard interviews when I got here, so why shouldn’t this person?”

I suspect companies who haze potential employees have bigger problems, the first of which is that they aren't coaching their interviewers in how to perform a good interview!

No one ever coached me in how to run a good interview. Instead, I had to take the initiative to read "Smart and Gets Things Done." When I've worked for large tech companies, the HR departments knew less about how to interview a software engineer than we, the developers running the interviews, did.

What About Interview Websites? How About The Candidate Brings Their Laptop

Sites like https://leetcode.com/ and https://coderpad.io/ offer ways to facilitate live and take-home coding questions.

I think these sites are great, especially when they provide a wealth of examples of questions to base an interview on. But, just using a tool won't magically make someone a better interviewer.

I think the biggest value of these tools is that they respect the candidate's time. Before asking a candidate to spend time traveling to your office, you can do the first part of the interview without the candidate investing travel time. This is critical if the candidate needs to fly for the interview, but also respectful for candidates on the other side of town.

I also think a great technique is to ask the candidate to bring a laptop to an on-site interview and then watch the candidate code on-screen. As a candidate, I once plugged my laptop into a projector and fired up my IDE. I personally found it easier than scribbling on a whiteboard.

Tradeoffs Between Whiteboarding vs Take-Home questions

When a company relies on take-home questions, it's low-risk for them to send anyone their take-home question, in contrast, an in-person whiteboard question requires a more time from the company's engineers.

But, a company can send out its take-home interview question indiscriminately and waste candidates' time. (This is why I walk away from time-consuming take-homes, or take-homes that require an alphabet soup of technologies.)

It's possible to cheat on a take-home, because a candidate can send it to a stronger programmer, or sit with another programmer. Cheating on a live coding interview is harder.

But, many engineers prefer to code without an audience and feel more comfortable coding in private. The take-home questions often give bigger insight into the candidate's style and ability to organize a program.

Take-home questions give little insight into how a candidate behaves in the office, or has a technical discussion. They also give little insight into a candidate's intuition, because you only see the finished product, not their working process.

With a whiteboard question, or other live coding techniques, it's easy to see a candidate's biases. These biases can expose someone who's going to struggle in your stack, or just someone who you won't get along with. They also can expose when a candidate has poor "common sense" for the kind of work you're doing.

Another advantage of whiteboard questions is that it's easy to tune them on the fly. You can see where a candidate stumbles, and you can infer a bad question based on a candidate's reactions.

It's much harder to tell when a take-home question is bad. Some candidates might want to impress you and spend an unreasonable time on the question. Other candidates might ghost. (Ghosting means the candidate just stops responding to emails and phone calls.) The feedback loop just isn't there.

As a candidate, when I get a bad whiteboarding question, it's easy for me to just muddle through it. Depending on circumstances, I'll either decline the job or discuss the odd question with someone else in the process. But, when I get a bad take-home question, I'm stuck between either ghosting, (not responding at all,) or providing feedback. I've done both, and I'm not sure which is a better approach. The one case I gave feedback, the hiring manager got defensive, which confirmed my decision to walk away.

So, take-home versus whiteboard (or another live coding technique) isn't a slam-dunk decision. It's more about knowing what's critical and weighing tradeoffs. You can also just let the candidate decide for themselves, but then you have the overhead of planning two different kinds of interviews.

How To Plan a Good Interview Question

In my experience, the purpose of a technical challenge is more than just determining competence. It's also how the interviewer verifies that a candidate has good comprehension of tasks, and can conduct themselves well during a technical discussion. Furthermore, these discussions ensure that the candidate can get along with the team.

A good question requires careful planning.

  1. Focus on 2-4 very high level programming concepts that are critical for the job.
  2. Have an emotional understanding of what "common sense" is for the job. (What kind of intuition should a candidate have?)
  3. Accept that you can't onboard someone in your entire stack in order to screen them
  4. What are specific biases you need to screen out?
  5. Are some concepts better screened in a live exercise or in a take-home assignment?
  6. Make sure that you're screening in people who will succeed in your questions.

Specifically, if you're trying to hire generalists, accept that new hires will have a learning curve. You won't be able to test such candidates on details of your stack because they probably haven't worked with it before.

In contrast, if you're trying to hire someone who can work in your stack from day one, it's okay for questions to require knowledge in your stack. You just need to make sure to only screen candidates who already know your stack. This is useful when screening contractors for short jobs.

Translation: Only ask someone questions about frameworks and libraries if you require direct experience with those technologies. Otherwise, If you're interviewing someone with Postgres experience for a job with Oracle, don't ask Oracle-specific questions or assign Oracle-specific tasks.

You Can Ask the Candidate to Present Code they Wrote.

One recent interview completely avoided a take-home question or a live coding exercise. Instead, they asked me to present some code of my choice. I spent some time going through my github repository finding some interesting work I've done over the past decade, and then we reviewed a fitbit watch face. Ironically, I probably spent 2-3 hours preparing; both in choosing the code to show, and then reviewing the watchface code as I hadn't looked at it in over a year.

The thing that I really like about this technique is that it's reusable. If another company asks me to present code, I'll just propose the fitbit watchface and it'll be fresh in my memory. Saves me time too!

What Makes a Good Take-Home Question

A good take-home question should take 2-3 hours.

It's important to avoid planning fallacy where you think something is simple, but then what you thought should be 2-3 hours really is 15 hours.

What's critical is to avoid pushing a learning curve on a candidate. If your take-home require specific libraries, languages, frameworks, ect, then you should discuss it with the candidate before you send the question. Any framework or tool that you ask the candidate to use should be on the candidate's resume.

Dave Rolsky, in A Technical Hiring Process Revisited, says:

When you come up with a homework exercise you must have at least two (and preferably more) of your existing staff do it, including some people who were not involved in formulating the exercise.

If the homework is in a problem domain that relates to what your company does (and it should be) then you should expect in house to staff to finish it in half the time it will take a candidate.

In my experience, the screenings that I like are "use any language, libraries, and frameworks" and then run in a shell; or verified with unit tests; or present a very basic HTTP API. You can also direct the candidate to use a language that they claim significant experience in.

My favorite assignment was for me to feel my way through a maze that was provided incrementally though a REST API. Another one had me write a RAM database and tests.

When I was desktop client architect at Syncplicity, I occasionally used a coding challenge that had candidates build a document by loading multiple XML documents from a web server. We primarily screened candidates with C# experience for a job in C#, so we asked the candidate to use C#. But, when we interviewed a Mac developer with Objective C experience, we asked the candidate to use Objective C; and if we interviewed someone who needed to learn C# on the job, we'd let the candidate use their most comfortable language. We assumed that candidates could load documents over HTTP and work with XML. Internally, we had a few engineers make sure they could do the exercise in about an hour or so.

We did not require any specific library or framework, but we did suggest two of .Net's built in XML parsers as a hint.

Working on the Syncplicity desktop client often required working with poorly documented APIs, so we deliberately kept a little bit of ambiguity in the instructions. My original version of the challenge was slightly more ambiguous, and I think "more fun" for people who can work with poorly documented APIs.

How To Plan a Good Whiteboard Question

A good whiteboard question requires careful planning. The interviewer, or team, should be able to plan a 1-hour interview in about 1-2 hours, but can re-use the interview for most candidates, and even different (but similar) positions.

  1. Focus on 2-4 very high level programming concepts that are critical for the job.
  2. Have an emotional understanding of what "common sense" is for the job.
  3. Plan coding questions that rely on 10-20 lines of very basic code.
  4. Plan the discussion: How do you give instructions? (What's written and what's verbal?) Assume you're going to have to explain all needed APIs and algorithms.
  5. Plan hints
  6. Anticipate biases that indicate a poor match
  7. Decide if these are true whiteboarding questions, or if the candidate will bring a laptop, or if you'll do these in teleconference with an online coding tool

I personally like to include a question that tests a candidate's listening skills, where the task just requires following instructions. These aren't trick questions, instead, they're meant to ensure that the candidate can participate in design discussions and comprehend assigned tasks.

I also include a very straightforward question where I provide an API and have the candidate transform data from one format to another format. These primarily filter out candidates who don't get certain basic concepts; but I also use these to filter out candidates who's fundamental biases would mean they wouldn't fit in with our team, or demand that we refactor everything because we didn't use their framework or design pattern of choice.

As a candidate, the thing that I love about live coding exercises is that they give me a real good feel for what it's like to work with someone, and what the general team dynamics are. In the case when the interviewer slammed me for not microoptimizing my code, I realized I didn't want to work there. It was more due to their attitude than the desire to microoptimize.

What if it's important that a candidate demonstrate using an algorithm? Actually, that makes a great live coding question. Just make sure that the question involves explaining the algorithm instead of expecting the candidate to know it ahead of time. Or, before the interview, suggest that the candidate review the Wikipedia page about the algorithm.

In Summary: It's about planning a good interview

When I'm a job candidate, I really don't care if a coding question is whiteboard or take-home; nor do I care if it's truly on a whiteboard, or I need to bring my laptop, or we do a teleconference with an interview website. I like the style where I can present some code that I've written, but it's certainly not required for me.

Some candidates have preferences, and if you're in a situation where you want to please everyone, the best thing to do is let the candidate choose between some form of live coding, take home, or presenting existing code. Assuming you don't give the candidate the choice, just look at the tradeoffs and pick what's more important to you.

What's important is that you research how to plan good questions and make sure that everyone involved in the process is also trained in running a good interview.

Take HomeLive (Whiteboarding)
Some candidates feel less pressureOther candidates want team interaction
Can evaluate more complicated codeCan evaluate collaboration and behavioral style
May not get feedback on issues with the questionCan get feedback on the question from body language, tone of voice, and reactions
Candidate can use the tools and environment where they are comfortableCoding on a whiteboard or in an online tool is about discussion
Candidate has little time pressureCandidate has time pressure
You evaluate the finished product as a wholeInsight into the candidate's thought process
All hints have to sent ahead of timeYou can give hints on a case-by-case basis
Can not adjust the question as you goCan adjust the question as you go
Easy to send a take-home to a "maybe" candidateCompany has to invest extra time for each "maybe" candidate
Can waste a "maybe" candidate's timeCompany has to choose wisely before inviting a "maybe" candidate

Highlights:

  • Make sure your sound works within 3 minutes. If there are problems, switch to your phone immediately
  • Your background is more important than what you wear
  • A good computer often sounds better than a cheap headset. What's important is that you're in a quiet space.
  • Use good listening skills when performing tasks
  • If you have a thick accent, hearing issues, or certain learning disabilities, discuss these ahead of time. Otherwise, you may perform poorly over teleconference when you would do well face-to-face
  • Don't cheat

With many offices closed due to the COVID-19 pandemic, a lot of interviews are moving to teleconference. I've interviewed many candidates through teleconference. Here is what I expected, as a software engineer, interviewing global software engineering candidates.

Other industries, locations, and interviewers, will have different expectations than I do. Finally, take all career advice with a grain of salt and use your best judgement.

Demonstrate that you can use teleconference software

On my team, we conducted all of our meetings through teleconference. This was necessary because we were a global team. It also meant that basic proficiency with teleconference software was a requirement for the job.

Frequently, candidates had difficulties with sound. In these situations, I would always suggest that the candidate call in with their phone. Still, many candidates were inconsiderate and forced me to wait while they debugged their computer.

In a teleconference, it is unprofessional to make someone wait longer than three minutes while you fix sound. Almost all teleconference software will call you on the phone if you have problems. Never make an interviewer wait while you try to fix your sound.

Some people suggest doing a test-run with teleconference software the night before. One staffing firm that we worked with started their teleconferences early so the candidate could fix technical problems before I joined the call.

Another infrequent mistake was a candidate joining the teleconference from their phone, and not having a computer ready. These candidates appeared unprepared. When an interview is by teleconference, assume that you will have to share your screen and perform a basic task, just like on a face-to-face interview.

Don't get obsessed with a headset

A good computer, like a Macbook, works extremely well without a headset. In my experience, the Macbook's speakers and microphone work better than headsets.

Furthermore, a problem with USB and Bluetooth headsets is that the teleconference software now needs to pick which speakers and microphone to use, which can be confusing, especially when a bluetooth headset shows up as “BT 543XYZ.”

In my opinion, the best way to do a teleconference is with a quality laptop, in a quiet room, without a headset.

Pay close attention to what's behind you

Software engineering is a casual field, so I never paid close attention to what candidates wore. What I did pay attention to was what was behind the candidate: Paint colors, furniture, cabinets, views out the window.

Candidates in a video teleconference interview should sit at a desk, with the camera aimed so their face is centered in the camera. Make sure the background is something you'd be comfortable showing a guest.

I've seen advice that candidates should stand for teleconference interviews, or wear a suit. I never saw candidates do that, and I personally think it would be awkward.

Anticipate and discuss teleconference format difficulties ahead of time (Like hearing loss)

Teleconferences are much harder for non-native language speakers, people with heavy accents, people with hearing difficulties, and people with some learning challenges. Someone who might be perfectly fine working in a face-to-face situation may do poorly on a teleconference for any variety of reasons.

In our case, where we worked through teleconference daily, the ability to handle teleconference was a basic requirement. In some cases, we wondered if a candidate who performed poorly would make a great employee in a face-to-face situation.

During the COVID-19 pandemic, if you're interviewing for a job that's going to be primarily face-to-face, and the interview is through teleconference; and you have some kind of an issue where the teleconference will work against you, it's best to discuss this with HR before the interview. You might need to make other arrangements so you can "put your best foot forward." The last thing you want to do is lose a golden opportunity because you have a little bit of hearing loss that normally isn't an issue when you're with someone face-to-face.

Use Your Best Listening Skills

When I asked candidates to perform small coding tasks in a teleconference, I would bend over backwards to explain the task:

  • I'd show a written description of the task in the teleconference
  • I'd go through the task point-by-point
  • I'd give the candidate plenty of opportunity for questions
  • I'd paste the task into the chat window

Some candidates would just get stuck in a circular pattern of asking the same clarifications over and over again, as if they couldn't understand the premise of the question. These were the "I can explain it to you [the candidate], but I can't understand it for you [the candidate]" situations.

In most of these cases, the candidate demonstrated exceptionally poor listening skills. I never knew if these candidates were truly capable of doing the job in a face-to-face situation, but merely had trouble understanding me through the teleconference. It didn't matter, in this case, because the job required daily interaction through teleconference.

Now, you might wonder why I didn't give someone who needed a little extra attention a chance. I always did. To put it differently, there's only so many hours in the day. I can either work with one engineer who needs extensive handholding, or eight engineers who can work independently. Demonstrate that you don't need extensive handholding.

Finally: Don't Cheat

When I interviewed a candidate, I tried to fairly assess their ability to do the job.

Once, a candidate gave me a memorized, well-rehearsed answer to one of my questions. Their answer was almost sing-song, as if making it through my interview was a mere formality, or a speech they had to memorize in order to pass a class at school.

I then made up a short question on the spot and the candidate gave me the "this wasn't supposed to be on the test" tone of voice. Then the candidate couldn't answer the question, either because they panicked or just didn't know the answer.

I suspected that someone covertly recorded an interview with me and then sent it to their buddy. We rejected the candidate.

It's not worth cheating: Hires who do well on the interview, but poorly on the job, don't last long. You don't want to be the person on the team who can't do their part.