CodeSignal’s question creation UI empowers you to measure the specific skills that you have in mind for a role. If you can’t find exactly what you’re looking for in CodeSignal’s library of over 4,000 programming questions, you can design your own question, share it with your team, and use it across the entire platform. But what should you keep in mind when designing a custom question?
Recently, our Measuring Dev Skills series discussed custom questions with Michael Newman, Director of Product & Engineering at CodeSignal (watch part 1 and part 2). Below we’ve summarized some key tips for creating custom questions that will help you measure the specific skills you want to hire for.
1. Use the right question type
When you click on Create Task from your Tasks view in CodeSignal, you’ll get to pick from 7 different task types.
Let’s discuss a few of the most common options. A single-function task is a good starting point. Here, you’ll describe the problem and provide test cases in the form of input and output values. The candidate’s job is to write a function that satisfies the requirements by passing all the test cases. Single-function tasks are very flexible because the same task can work for a variety of different programming languages.
Sometimes you don’t want or need test cases. In a live interview where you have the luxury of working directly with the candidate, you might prefer to simply base your evaluation on your interactions and observations. Even with CodeSignal Test, you might want to provide a more open-ended question—e.g. ”Add polish to this UI”—and have your team score submissions manually. Our free-coding and free-frontend tasks are designed to be extremely versatile and don’t use test cases. Instead, the output can be evaluated in the built-in terminal and mobile/web emulator (for free-frontend tasks).
SQL and filesystem tasks let you ask a range of even more targeted and complex questions; we’ll cover these in more detail in a future Measuring Dev Skills installment.
2. Check “enable network access” to evaluate API skills
For some types of questions, you might want to have logic that requests data from an external service—for example, making an API request, downloading a file, or using curl. To make this possible, you can check a box to enable network access in your task Settings.
With network access enabled, you could ask the candidate to make a request from the Google Maps API or load a tweet from Twitter’s API and process it in some way. This can help you measure the real-world skills involved in working with external providers and communicating across systems.
3. Set timeouts to challenge candidates on optimization
For most task types, you can set timeouts to place a limit on how long the candidate’s solution can take to run. You can even make the timeout different across different languages.
Depending on your question, you might want to require the candidate to build a solution that doesn’t just satisfy the test cases, but is also efficient. To enforce that the solution is, say, O(nm) and not O(n^m), use timeouts. And if you expect the candidate to have a certain level of optimization knowledge in a given language, you might choose to set language-specific timeouts (e.g. making the runtime for C a bit stricter).
4. Think about which test cases to show as samples
“When it comes to building tasks that are pleasant to solve, I think it’s really important to get the sample test strategy right,” Michael said. “Every test case is a chance for you to communicate an expectation to the developer solving your task.”
While our default behavior is to show half of your test cases and hide the rest, you will often want to override this by defining which tests (and how many) you want to be visible. It can be helpful to ease a candidate into the complexity of the problem by choosing sample test cases that move from simple to challenging. You might want to show tests that check edge cases—or hide them. The choice is yours.
5. Make frontend tasks more realistic with libraries
For frontend developers, it’s important to assess their knowledge of the latest JS and CSS libraries. You can turn any free-frontend task into a React task (or your framework of choice) using the Manage Libraries feature.
Just start typing any library name into the search bar, which is powered by CDNJS. If you have a specific link you want to embed, you can easily pull that in too. Since it’s important to install dependencies in the right sequence, you can drag to reorder the libraries directly in the UI.
6. Duplicate and modify existing tasks for the perfect fit
Did you see a task in CodeSignal’s library that you liked, but you wished it had a slight modification? Simply click into the task, click on the More menu in the bottom-left corner, and select Duplicate. This will create a copy that you can modify however you like.
You might also want to duplicate and then modify tasks to isolate specific skills that you want to assess for. For example, you might create a task that asks candidates to implement a calculator app and add some layout elements. By forking this task with Duplicate, you can easily break it out into distinct chunks: one task that tests skills around implementing the functionality, and another one that assesses frontend design ability.
We hope you find these tips helpful as you think about designing custom tasks for your candidates in CodeSignal. And if you haven’t had a chance to try CodeSignal for yourself, simply request a demo.