For recruiters getting started with using technical assessments, the question can appear deceptively simple: which score should we use as the cut-off for candidates we want to move forward?
If you’re using CodeSignal’s General Coding Framework, for example, you could choose to narrow your pool of applicants to those who score above 450, 500, or even 550. Easy, right? Not so fast, say the test design experts at CodeSignal.
Drawing on our team’s research-backed practices, here are the top 5 things your team should keep in mind when setting a technical bar for your assessments.
1. Does your company regularly perform job analyses?
Job analysis, according to the Society for Human Resource Management (SHRM), is “the process of studying a job to determine which activities and responsibilities it includes, its relative importance to other jobs, the qualifications necessary for performance of the job and the conditions under which the work is performed.” It’s also the best place to start for developing and making use of technical assessments.
Here at CodeSignal, we use what is called a Combination Job Analysis Method to conduct a job analysis. This method “combines” two other methods: one that looks at the duties of the job, and another that looks at the knowledge, skills, abilities, and other person-related characteristics (KSAOs) that an employee needs to succeed in the role.
2. Do you carry out validation studies of your assessments?
Put simply, validation is the process of gathering evidence to see how well an assessment measures what it is supposed to measure. In the context of hiring assessments, a test can be considered valid if the way you interpret candidates’ scores is supported by evidence.
To evaluate the validity of your technical assessments, you need to perform both internal and external content validation. Internal content validation asks: do the coding tasks you use measure the right skills? External content validation, on the other hand, addresses whether individual tasks or tests are relevant to the job you’re hiring for.
3. Do you conduct adverse impact analyses for your assessments?
To ensure compliance with the EEOC and other regulatory standards, companies often conduct adverse impact analyses of their assessments to ensure that they do not unintentionally discriminate against protected classes: age, sex, and race, among others. How you set the bar for your assessment will have significant implications for how the assessment performs in an adverse impact analysis.
We recommend working with assessment research experts to perform these important analyses of your assessments and navigate the complex terrain of EEOC compliance.
How you set the bar for your assessment will have significant implications for how the assessment performs in an adverse impact analysis.
4. Do you have an assessment research team?
While you may have a team of rockstar engineers who can serve as subject matter experts (SMEs) for developing technical assessments, bringing in assessment researchers is crucial to creating effective assessments and setting the bar at an appropriate level.
These researchers will ideally have formal training in a field like Industrial-Organizational Psychology, which takes a scientific approach to studying how people behave in the workplace. I-O Psychologists will possess specialized knowledge in job analysis, assessment validation, and adverse impact analysis.
5. Can they coordinate effectively with SMEs?
Lastly, a key component that teams often lack is an assessment research team that can work effectively with engineering teams—the SMEs when it comes to technical assessments. Tasking your assessment team with conducting a job analysis, for instance, won’t be of any use if your engineering team doesn’t know how to translate the research findings into appropriate content for a technical assessment.
At CodeSignal, our assessment research and engineering teams work together closely in an iterative process to develop, validate, and continuously improve the tasks and assessments we develop.
Learn more
Want to learn more about creating validated and research-backed technical assessment for your organization? CodeSignal’s team of assessment experts would love to talk with you. Schedule a call with us today.