Mention the phrase “coding test” to an engineer, and the response might be… less than enthusiastic. For candidates, coding tests for interviews can seem random, unfair, and a waste of time. For companies, coding tests can seem equally random, unfair, and a waste of time—although for entirely different reasons.
If you’re skeptical of coding tests, we don’t blame you: there are a lot of ways for them to go wrong. However, we’ve seen many coding test examples from top tech companies that show that when done right, they can reduce bias, improve diversity, and leave candidates with a much better impression than the average phone screen. Plus, using some kind of technical screen or pre-screen skills assessment is one of the best ways to scale your tech hiring in a hyper-competitive market.
In this blog post, we’ll show you how to design a coding test that works. But first, let’s take a look at why coding tests can so often fail for candidates and companies.
How Coding Tests Go Wrong for Candidates
When interviewing at almost all of the largest and most competitive tech companies today, the first step is to take a coding test. However, you just have to look at platforms like Reddit, LeetCode, and Glassdoor to find engineers encouraging their peers to refuse coding tests from certain companies or certain online coding test platforms. That’s how easy it is for coding tests to leave candidates with a terrible impression of your company.
When these assessments go wrong, it’s usually for one or more of the following reasons:
- The online coding test platform was hard to use. A platform that’s glitchy, or an integrated development environment (IDE) that’s uncomfortable or lacking the expected features, means that engineers will need to spend time trying to figure out how to take the test rather than staying focused on the questions. As a result, candidates can feel like they weren’t given a fair shot during the assessment.
- The questions were poorly written. From the ordering of questions to the phrasing and content itself, writing interview questions is an art. Questions should build in complexity and not require too much context switching for the candidate. When professionals aren’t involved to develop these questions, the writing might suffer from a lack of clarity or use culturally-specific language and examples. This can be alienating and confusing, especially for people from different backgrounds and with different levels of English language ability.
- The questions were unrealistic. Coding tests often ask generic algorithmic questions because it’s easy to grade these questions in any language, and variations are relatively simple to generate. But these questions are more closely related to academic theory than they are to software engineering. So, experienced and senior-level candidates end up frustrated, feeling that they can only pass the test if they spend a lot of time studying up on concepts that have little bearing on their real jobs.
- The test was too long. Some coding tests are unnecessarily long. Interviewing is already a lengthy and stressful process. If candidates have an existing job and family commitments and they have to pass what feels like a college exam just to talk to someone at the company, that’s a lot to ask.
- There was no actionable feedback. The results of coding tests for interviews are often sent via a form email that doesn’t tell the candidate what they could have done better. Plus, there’s no way for the candidate to have a dialogue about how the test went. The interview can end up feeling very impersonal as a result.
How Coding Tests Go Wrong For Companies
Coding tests aren’t only problematic for candidates, they can also cause issues for engineering organizations when not implemented thoughtfully. Here are some of the top complaints:
- Questions get leaked. It’s extremely common for coding questions to be leaked on platforms like LeetCode and Blind. It’s hard or impossible to know if the candidate has seen the question somewhere else already.
- Tests are time-consuming for engineers to maintain. To keep a steady stream of questions and variations (and replace questions when they’re inevitably leaked), engineers need to maintain different versions of the coding test and ensure these versions are fair and comparable. This requires a massive amount of work and takes time away from product development.
- Candidates can pass even if they lack necessary skills. A good coding test should evaluate the baseline skills that someone needs for the job; the rest of the interview can be spent digging deeper into these skills and understanding what it would be like to work with the person. However, too often coding tests are a noisy filter because the test questions are not tailored to the skills needed for the role. So, the engineering team ends up spending time at the onsite stage with candidates who aren’t qualified.
- There’s no validation. Depending on the online coding test platform that’s used, there may be no way to verify that the test-taker’s identity matches the candidate, or that they haven’t plagiarized their answers.
Doing Skills Evaluation the Right Way
At CodeSignal we spend a lot of time thinking about the ways coding tests for hiring can go wrong, because we’ve seen proof that there are massive benefits for candidates and companies if we can get coding tests right. Engineering teams can scale faster and spend less time interviewing unqualified candidates, while candidates can be considered on the basis of their skills, not the keywords on their resumes.
Here are two strategies that teams can implement to feel confident that they’re getting these benefits—and to make sure candidates never refuse coding tests from your organization!
Skills Evaluation Frameworks
An assessment is only as good as its questions. If the questions are poorly written or not related to the job, the candidate suffers. If they’re hard to develop, easily leaked, or not measuring the right skills, the company suffers.
Today, top engineering organizations are turning to skills evaluation frameworks to overcome these problems. Serving as the “blueprints” for a coding assessment, a skills evaluation framework describes the particular types of questions needed to measure a candidate’s readiness for a given role. Frameworks are usually created and maintained by technical interview and assessment vendors that employ IO psychologists and assessment design engineers. This is a team of experts whose job is to design, test, and validate job-specific frameworks and create hundreds or even thousands of question variations to avoid plagiarism.
Job-relevant pre-screens
One of the common complaints candidates have is that coding tests ask questions that seem to have nothing to do with the job they’re applying for. CodeSignal Pre-Screen uses validated, role-relevant assessments that evaluate the skills that matter for each role. This helps hiring teams make the right hire, faster, and provides candidates with a positive candidate experience by allowing them to demonstrate their job-relevant skills in realistic coding environment.
Conclusion
Coding tests really can work—when they’re accurate, fair, and offer a comfortable development environment. The benefits for candidates include being able to demonstrate their programming skills in a natural manner, without the pressure of needing to narrate their thoughts to the interviewer on the other end of the screen. And, based on the online coding test platform use, they may be able to take a test once and then reuse their results when applying to many different companies.
Perhaps most importantly, this means they don’t have to worry about having the perfect resume: they can score an onsite interview based on their demonstrated programming ability. There’s a massive upside for recruiters and hiring teams too, who are seeking to hire more diverse candidates who may not have the traditional software engineering background (perhaps they’re even self-taught!), but who are prepared to do the job on day one. Implementing structured assessments at the top of the funnel also drastically reduces the hours that teams spend on hiring engineers.
Want to see coding test examples that deliver great results for candidates and companies alike? Request a demo of CodeSignal today.