What makes an effective assessment for tech hiring? You may think it’s one that uses questions that are highly specific to a coding language, or that are unique to your company’s product and market. While job-relevant questions are important, the most critical measure of effectiveness is the outcome. Strong assessments should enable you to accurately identify the candidates with the right skills for the role. To do that, you need to use assessments that are validated.
This blog post will unpack what “valid” means in the context of pre-hire assessments by walking through four types of validation or validity involved in developing an effective assessment: internal content validation, external content validation, criterion validation, and face validity. We’ll also dive into how CodeSignal applies these methods to validate the Certified Evaluations that power our technical interview and assessment solutions.
Introduction to assessment validation
Validation, in the context of assessments, refers to the process of collecting evidence to evaluate how well an assessment measures what it is supposed to measure. A coding assessment that is valid, then, is one that has been shown to effectively measure the coding skills and knowledge that it is intended to.
The 4 types of validity
To evaluate the validity of your tech hiring assessments, you need to perform internal and external content validation, as well as criterion validation. It’s also important to consider face validity when using assessments in your hiring process. Let’s dive deeper into what these terms mean.
1. Internal content validation
Internal content validation asks: do the coding questions you use measure the right skills? This type of validation is all about making sure that the assessment tool accurately and comprehensively measures the knowledge, skills, abilities, and other characteristics (KSAOs) that it’s intended to assess. This is done by carefully examining the content and design of the assessment to ensure it captures the requirements of the job.
To achieve internal content validation, assessment developers follow a few key steps:
- Job analysis: To start, a comprehensive job analysis is conducted to identify the critical knowledge, skills, abilities, and other characteristics (KSAOs) required for the role. This often involves interviews with subject matter experts, observation of job tasks, or even a review of existing job descriptions.
- Assessment development: Based on the information gathered during the job analysis, the assessment developers create questions that align with the identified job requirements.
- Question review: Before the assessment goes live, subject matter experts (SMES) review assessment questions and refine them to ensure they are accurate, relevant, and free of potential biases.
- Pilot testing: The assessment is then administered to a small sample of individuals, who are ideally SMEs. This allows the assessment developers to collect data on the assessment’s performance and make any necessary adjustments.
- Ongoing monitoring and evaluation: Assessment researchers need to keep evaluating content validity as industry trends and needs shift. They also need to monitor opportunities for cheating, which reduces the validity of an assessment.
At CodeSignal, the Assessment Research arm of our Talent Science team conducts internal content validation of our Certified Evaluations by partnering with SMEs to identify core skills of the target role and developing questions accordingly—a research process that takes approximately 2,800 hours per Certified Evaluation. They then pilot the evaluation with SMEs to further validate the content within the product and to gauge the candidate experience.
2. External content validation
External content validation addresses to what extent your coding questions—or assessments as a whole—are relevant to the role and the company you’re hiring for. An assessment for a front-end engineering role, for instance, should include coding questions that ask candidates to render UX designs, handle dynamic interactions, and work with an API, among other key front-end development skills.
At CodeSignal, the external content validation process includes conducting a job analysis with our customers. When you partner with the Talent Science team at CodeSignal to conduct a local job analysis, your team will receive data-driven recommendations on cut scores and Certified Evaluation content to further optimize for effectiveness, fairness, and candidate experience.
3. Criterion validation
Criterion validation—and more specifically, predictive validity—examines the relationship between a candidate’s score on a pre-hire assessment and their later job performance. If a candidate scores highly on a pre-hire assessment, for instance, they should perform well in the role they’re hired for, assuming there is strong predictive validity.
Criterion validation like predictive validity, while time-intensive, establishes additional legal defensibility for pre-hire assessments by ensuring their job-relevance.
CodeSignal conducts criterion validation in partnership with our customers using performance data from your organization. This ensures that our evaluations are empowering your team to make the right hires who will excel on the job.
4. Face validity
Face validity refers to the candidate’s impression of the job-relevance of the coding question, assessment, or assessment UI—it’s how relevant a question seems “on its face.” While face validity is the weakest form of assessment content validation (in terms of understanding the assessment’s effectiveness), it can affect candidates’ perceptions of your hiring process and, in turn, their likelihood to accept an offer. So, it’s important to develop coding questions and assessments that are face valid in addition to undergoing content and criterion validation.
Here at CodeSignal, we develop assessments that act like a flight simulator for developers and are both validated (in terms of predicting job performance) and highly realistic, face valid experiences for candidates.
Are your assessments truly validated?
In this blog post, we’ve seen that when it comes to pre-hire assessments, the concepts of “valid” and “validated” are more complex than they may seem. When a vendor claims their technical assessments are validated, ask: have they undergone a rigorous validation process? Or are they just face valid?
CodeSignal’s Certified Evaluations undergo a rigorous validation process that includes internal and external content validation, criterion validation to ensure alignment to job performance, and consideration for face validity. This process is carried out by our one-of-a-kind Talent Science team, made up of scientists and engineers setting the industry standard for fair skills evaluation and development.Â
If you’d like to learn more about introducing our Certified Evaluations into your technical hiring process, schedule a discovery call with one of our assessment experts today.