How To Build A Great Coding Assessment
In this guide, we'll explore general concepts, goals, and tips for outlining a great assessment for any Qualified Candidate. Last month we outlined the difference between an assessment and challenge. This month we are narrowing in on how to create a great Assessment.
Great assessments tend to have some similar features. Here is a guide to help you build a sturdy framework for an effective assessment.
Some assessments are intended for Prescreening, where we simply want to filter out poor performers quickly. Others are intended for deeper exploration of a candidate's skills. A third type is intended for live interviews, where the interviewer is working with the candidate directly.
- Prescreens should be short and fairly simple. They shouldn't be solvable simply with a Google search, because we actually would prefer to fail the majority of weaker candidates.
- Take Home assessments are usually longer, and designed to test domain-specific knowledge deeply. They should still contain a strong set of automated testing, but it's OK to rely on some manual reviewing of the solution for final assessing.
- Live Interview tests are not as common. While any assessment can be used in a live interview, a good live interview would purposefully be built to be a little more open-ended. Traditional algorithm-based assessments do well in live interviews, because there's more back-and-forth between the interviewer and the candidate.
Plan Around an Overall Testing Goal
Before you begin writing code, it helps to come up with an overall plan. Rather than try to build a test from the bottom up, look at the overall assessment from the top-down. This will help ensure the assessment is doing a good job at testing its intended goal - without missing key functionality or getting overly complex.
Some examples would be:
- Test a specific framework or technology (e.g., AngularJS or Ruby on Rails)
- Test toward a specific skill level (e.g., Junior vs Senior)
- Test for general knowledge vs domain specific knowledge
Determine What Concepts Should be Tested
Once you have your overall theme and goals narrowed down, then you should create a list of all the facets that need to be tested. You want to make sure your final assessment does a decent job of testing these features, and doesn't make assumptions or forget to include challenges & tests for key concepts.
A general flow that might work is:
- Write down a decent sized list of everything that could be tested.
- Organize the concepts
- Some concepts will be tested without any extra effort (core to the framework or language being tested)
- Some concepts will need to be called out specifically, because they are either edge cases, or good at highlighting specific skills
- Group similar concepts together when possible, so you can reduce the number of steps
- Reduce the set to something reasonable
- Aim for around a dozen strong concepts, which can reasonably be grouped into 3–4 challenges
Now you should have a mental model of what you need to test, which can help come up with a theme to build a "testing story" around.
Make It Feel Real
The best assessments make the candidate feel like they are actually accomplishing something. Even if the end result is only a partial product, it's better to have built something that feels like it could be the basis for a real application, than just to work with generic terms.
Some examples:
- A media library with different types of items in the catalog
- An online store of some kind
- A virtual game lobby
Things to avoid:
- Using generic terms like Foo or Bar to describe objects
Overused generic programs can be lame too, such as:
- A ToDo list
- A Chat Room
- A Twitter-like app
If you want to use these as a base, try to make sure there's enough to it to make it interesting beyond the generic "example program" level.
Aim For a Comfortable Amount of Time
In most cases, the overall assessment should be something that can be completed within a couple of hours max. It depends on the type of assessment you are building, as well. Prescreens usually take between 20 minutes and an hour, while take-home's are usually solvable in 1 to 3 hours.
Just remember to put yourself in the candidate's shoes. While they are applying for a job, they might not have the time to spend an entire day solving complex tests if they don't feel reasonably confident that they can get the job.
Try a Challenge Series
In some cases, an assessment can be broken up into a series of challenges where each builds upon the previous. This can create a more satisfying experience for the candidate, as they aren't forced to start each step from scratch.
This isn't possible for every type of assessment, but when it is, it can enable testing much deeper knowledge, too, as the candidate will need to be able to work with their own code and refactor or expand upon it as the specification grows.
In sum, we can say creating a great assessment takes a bit of planning before you start to put it all together. We're here to help you do that. Want to know how to create a great challenge? Stay tuned for next week's Product Blog.
