How To Create An Effective Coding Challenge

In this guide, we’ll explore general concepts, goals, and tips for creating challenges for the Qualified Candidate Assess system. If you do not know what an Assessment is or how to build one be sure to read over our prior Product Blog Posts.

Creating Great Challenges

If you’ve gone through the previous article, you should have a general set of testing goals in mind. Now it’s time to break it up into one or more challenges that can be used to effectively evaluate the candidate.

Challenge testing goals

For the benefit of the reviewer, each challenge should have a couple of specific testing goals in mind. Using your list of concepts for the overall assessment, you can now try to determine up to four separate challenges that each test a handful of those concepts.

You’ll want to make sure each core concept is covered by at least one challenge. This will help you know where to emphasize the tests as you define the challenges further.

Provide Clear, Precise Instructions

Good instructions shouldn’t be a book, but they should be long enough to cover everything needed for a programmer to understand the goal of the tests. They should be easy for the candidate to skim through on the first pass, while having detailed rules where you’d expect them. Whenever possible, provide specific examples.

Instructions are written in Markdown, which provides a lot of tools for formatting text. Use it judiciously, especially section headers, lists, code delimiters (both inline and blocks), bold for highlighting important terms, etc.

Multiple Language Code Blocks

The instructions renderer has the ability to render specific code sections based on the currently selected language. If you are using multiple languages, and have code examples, make sure to include the examples in each language, like so:

On Naming & Formatting

It’s easy to simply call everything Challenge or solution(), but using proper naming helps the candidate avoid simple mistakes. Make sure you treat the challenge as if it's real, production code—just as you expect of the candidate.

This also means that all your examples are properly formatted, so the reviewer can see if the candidate tried to follow the example code formatting.

Provide Setup Code

In almost all instances, you should have some setup code. If you have a specific method, make sure you include the template for that method. If you are building up something more complex, make sure you provide the basic framework for the candidate.

This helps reduce the time it takes to get started significantly. When only provided a blank editor and instructions, the candidate has a lot more work to get started than necessary.

Cover Testing Goals

Once you have a basic challenge built, you need to make sure you have a strong set of tests for that challenge.

Every challenge should have:

• Thorough coverage of the core concepts. Test different inputs and outputs, with any edge cases you think are important.
• Proper test design. Make sure you use the testing frameworks test grouping, such as describe and it blocks.
• Prefer more individual tests over multiple tests within a block. This gives a more refined score, by providing more overall “points”.
• Use detailed assertions. Whenever possible, make sure to provide both the expected and actual output.
• Whenever possible, make sure tests describe the input and expected output. This is especially important for more complex data, as reviewing (for example) two complex JSON blobs is very tedious.
• Tests should build up in complexity. It’s easier for the candidate to solve the simple cases and then build on edge case handling, allowing them to focus on building up the solution, rather than having to get everything right at once.
• If practical, include a block of randomized tests. Randomized tests help reduce the risk of cheating by simply returning the expected answer based on the input order. Note: these should never replace the standard tests.

Other good testing hints:

Use the output to render more complex data in helpful ways. For example, you can generate graphics using <canvas> or render input and output data in ways that can help the candidate find mistakes. Be smart about tests — you can write methods to help reduce repetition in your tests. You can use the Preloaded section to lock down the environment, or to provide preloaded code or data to the candidate before their code runs.

Provide Candidate Tests

Another easy thing to ignore is providing sample tests for the candidate. These are critical, because without some sample tests, it’s often much harder for the candidate to know how to begin setting up those tests.

Usually you’ll write the full test suite first, then copy them over the sample and remove a portion of them. However, make sure to provide enough sample testing coverage that the candidate can expand upon the setup if they need.

Help Interpret the Results

As the creator of the challenge, you’ll have the best insight into the goals of the challenge. Once you have the overall challenge complete, make sure you write up a guide for interpreting the results, under the Purpose field.

This should explain things like:

• What were the overall goals? Be specific in what was being tested, and how.
• What are some things to look out for from stand out candidates?
• What red flags should there be from poor candidates?

You can avoid overly obvious details (code quality, comments, documentation, etc), unless those are part of the specific testing goals.

Have more questions? Reach out to us at [email protected]

‍