Skip to content Skip to footer

How CodeSignal saves time by generating test cases automatically

As an engineer creating an interview question, you may spend a lot of time thinking up the perfect question and its optimal solution. But you can’t stop there: if you want to score your question automatically, you need to come up with test cases—and those test cases should not be easy to guess or hardcode.

At CodeSignal, we are always trying to save you time and make it easier for you to create high-quality questions. So, to make generating test cases simple, we added the ability to generate Random Tests automatically. All you have to do is write the author solution to your programming question, and we handle the rest. In this article, we’ll explain how it works.

It’s like algebra…

When you’re building an interview question, there are several variables that form the overall equation. First, there’s the “author” or “reference” solution. This represents how a successful candidate would solve the task. It’s important to include this to know what the solution looks like (and whether it’s even possible!). Then there are the test cases, which specify the expected output for the given input.

But there’s a bit of redundancy here: do you really need to supply the test case inputs, output, and author solution? As with algebra, we can remove one variable and figure it out from the other pieces. After all, if you know that the author solution is correct, then you can give it any input to derive the output. 

We can extend this idea and solve for the other variables, too. When solving a task, candidates can query the author solution with custom test inputs and see the generated output, which can help them understand the boundaries of the problem. Similarly, task authors can run their author solution against known test inputs and outputs to verify that the solution is correct.

Finally, what if you have neither the inputs nor the outputs? We can go a step further and generate inputs randomly and then derive the outputs from the author solution.

Let’s examine how this works in more detail within CodeSignal’s Single Function Tasks.

Automatically generating test cases

Imagine you’re thinking of a task, such as asking candidates to take two numeric strings, add them together, and return the answer as a string. You’ve written the author solution, and now it’s time to create your test cases. 

You might come up with a few examples that you know you want to test, such as inputs that are negative or include leading 0s. If you’ve already written the author solution, when you go to create your test cases, you can simply provide these inputs and click Autogenerate Output. That way, you don’t have to worry about writing the output yourself (and potentially making a mistake). 

What if you want to add even more test cases for scoring, without coming up with the inputs manually? In that case, you can generate Random Tests. CodeSignal will create a random input, run it through your author solution, and derive the output. As we’ll discuss next, you can restrict the parameters of the randomization—such as how long the inputs should be and whether they should exclude certain values.

Parameters for generating random inputs

In order to generate random tests, CodeSignal asks for parameters for randomly generating each input in your task. Since coding tasks sometimes need very specific test cases, we designed these parameters to provide flexibility and control. 

For integers, it’s fairly straightforward: we accept a range for restricting the output. For floating point numbers, the number of decimal places is configurable. For booleans, which can only be true or false by definition, it’s possible to configure the probability that the value is true. 

Characters and strings are interesting. We built parameters that resemble what you might see in a password generation tool—such as specifying which types of characters to include, or the length—along with some extra features that are useful to have for programming tasks. For example, we provide a way to generate only palindromes (since they frequently come up in interview questions). We let interviewers whitelist characters, as well. If you’re interviewing a computational biologist, for instance, you might want to generate a long, random DNA sequence by only including the characters A, C, G, and T.

We also support generating random collections, like matrices, arrays, and linked lists. Collections add another layer to the logic: we randomize the individual elements within each collection and then also randomize the collections using separate parameters. Here’s an example of the parameters for generating a random matrix of booleans:

Conclusion

With features like Autogenerate Output and Random Tests, you can quickly build interview questions and easily include as many hidden test cases as you want. Randomization helps you create test cases that are impossible to hardcode and are free from human bias in making up test case inputs. 

Looking ahead, we’re considering how we can extend this functionality to support randomly generating more types of collections, such as trees, or for different task types, such as SQL.

Do these kinds of problems sound interesting? Check out our careers page to learn how you can join CodeSignal and help build the #1 tech hiring platform.