If you’ve taken any of CodeSignal’s Certified Coding Assessments, you’ve interacted with the work of our team of Assessment Design Engineers. The same is true if you’ve used one of the questions in CodeSignal’s Question Library during an interview or assessment. Assessment Design Engineers are responsible for coming up with interesting programming challenges to assess a wide range of developer skills.
This is a pretty special role, requiring creativity, a broad knowledge of software, and an eagerness to learn and explore. Assessment Design Engineers only exist at a few companies in the world, so we decided to shed some light on the work our team does.
What Assessment Design Engineers do
Our team creates practical tasks that challenge engineers in specific skills-based areas like backend, frontend, DevOps, analytics, data science, and mobile development. Our work is focused on developing coding questions for the Skills Evaluation Frameworks that power our Pre-Screen and Tech Screen products. We also create questions our customers can use in our Interview product and support the CodeSignal Engineering team in implementing new product features.
Develop content for Skills Evaluation Frameworks
One of the most challenging parts of our work is building and maintaining our Skills Evaluation Frameworks, which are pre-built, role-relevant assessments that are built by my team and validated by CodeSignal’s in-house Industrial-Organizational Psychologists. As our assessments are an industry standard for programmers, we have to be very careful to ensure that our questions are balanced, unbiased, and protected from plagiarism. We develop thousands of question variations to allow for dynamic question rotation in our framework-based assessments, which reduces the likelihood of cheating on the assessment. We are continually improving our existing frameworks, and plan to add more of them to standardize other job areas as well.
Designing a Skills Evaluation Framework is not a fast process. We are constantly brainstorming, monitoring, and testing assessments within our team and externally. As part of CodeSignal’s Skills Evaluation Lab, we collaborate closely with our team of IO Psychologists to get an academic perspective on the types of questions we should be asking. We also work with CodeSignal’s product team to make sure that our IDE and platform can support these questions.
By serving as a bridge between the research and the product, Assessment Design Engineers have a key impact on CodeSignal’s roadmap. For example, the idea for supporting project-based questions that resemble a real-life coding project originated during our assessment design work, and is now an essential part of the CodeSignal platform.
Continuously improve our question library for CodeSignal customers
Assessment Design Engineers are responsible for maintaining and growing a rich library of questions (over 4,000 total) that customers can pull from for our Interview product. While customers can design their own coding questions, it’s our team’s job to make sure that they can also choose from a wide variety of high-quality questions across many different topics. We focus on the kinds of practical, job-related coding questions that help our customers reduce bias in their interview process.
When we’re designing questions for the library, we have two end-users: the hiring manager or interviewer who is going to use this question, and the candidate who is going to be asked to solve it. To create a good experience for both kinds of users, we have a bit of a Goldilocks problem on our hands. The question has to be hard, but not too hard. It should be interesting, but not a reading comprehension test. It can’t be something that the candidate has seen a million times, but it shouldn’t be coming from left field. And it has to take just the right amount of time to complete. To read more about how we think through some of these design questions, check out this article.
Help implement product features
Lastly, Assessment Design Engineers contribute to the CodeSignal platform and product by sometimes implementing features related to building our assessments and questions. We also implement support for new languages and tools, and work on the improving internal content-related tools we use every day.
One product feature we worked on was developing a fast way to import solutions for our questions in our Question Library. Now, the customer can click a button to auto-generate the answer for their custom test. Recently, we’ve been working on improving the current way of running and managing libraries for JVM languages like Java, Kotlin, Groovy, and Scala.
The skillset needed to be an Assessment Design Engineer
The role of an Assessment Design Engineer is quite unique. One of the key skills is the ability to understand a wide variety of software engineering roles. For example, what are the skills you need to have as a frontend engineer? How would we design a fair, high-quality assessment for those skills?
Assessment Design Engineers also need to be able to quickly get up to speed on new technologies. The world of software engineering changes constantly, and there are so many languages and frameworks that it’s impossible to know them all. Every day, we’re creating content for new technologies that no one on the team is an expert in. Even when you’ve never worked with the tool, framework, or technology you’re assessing, you have to be able to design great content for it. Assessment Design Engineers have to love learning new things in a short amount of time. A strong foundation in software best practices is a big help, too.
Think you might be interested in the Assessment Design Engineer role at CodeSignal? We’re growing our team and looking for creative and kind team-players to join us in building the #1 tech assessment platform. Take a look at our Careers page for more information.
Dmitry Filippov manages CodeSignal’s Assessment Design Engineering team. His main focus is planning, communication, and collaboration across teams.