For as long as there have been hiring assessments, there have been candidates who attempt to game the system. Before they had easy access to generative AI (GenAI) tools like ChatGPT, candidates for technical roles might copy and paste someone else’s code from an online forum, or have someone take the assessment or interview in their place. Now, candidates are attempting to use GenAI and other tools to cheat on coding challenges.
At CodeSignal, we’ve developed and refined techniques over the past 10 years to detect and mitigate cheating on our technical assessments, including using GenAI. Here’s how we do it:
Suspicion Score
Suspicion Score is how we at CodeSignal flag potential instances of cheating to our customers. Whether a candidate tries to copy a solution from LeetCode or asks ChatGPT to solve a problem for them, our Suspicion Score is a robust solution to flag potential integrity issues.
Suspicion Score combines multiple factors into one model to assess whether a candidate’s submission represents their own work. These include:
Similarity score
CodeSignal compares candidates’ code to all submissions on our platform and searches the web for similar solutions posted online. Hiring teams can then view a candidate’s code next to a similar solution in a line-by-line diff view.
Pattern detection
Our Suspicion Score dissects each solution to identify patterns across the millions of interviews and assessments conducted through our platform. We then flag candidate submissions that include patterns that may indicate the use of GenAI or other unauthorized resources.
Telemetry
We use telemetry from our advanced IDE to reliably identify activities correlated with use of AI coding assistance, such as unusual typing patterns. This allows us to evaluate if a candidate’s behavior in the IDE resembles that of someone writing and debugging code on their own.
Paste events
CodeSignal tracks when candidates copy and paste code into our IDE from another window, and what code was copied. Sometimes, these copy-paste events may not indicate cheating—for instance, if a candidate prefers to write code in their own IDE—so we flag them for hiring team review.
With our one-of-a-kind Suspicion Score, we synthesize these factors to assign an overall trust level to each result, which gives companies confidence in their candidates’ results while flagging those that require further review.
Proctoring
At CodeSignal, we offer full-service proctoring for our assessments, where a real human reviews a recording of the assessment session to monitor for suspicious activity. We also use rigorous verification of government-issued IDs, as well as full audio, video, and screen recording, to ensure a candidate is who they say they are and does not receive assistance from unauthorized outside resources.
Here’s how our proctoring process works:
Step 1: Candidate takes a proctored assessment. Prior to starting the assessment, the candidate is prompted to share their camera & microphone, screen, and a valid, government-issued photo ID. During the evaluation, CodeSignal records the candidate’s video, audio, and computer screen.
Step 2: CodeSignal reviews the recording. After the candidate completes the assessment, CodeSignal’s team of proctoring specialists reviews the recording for rule violations. If no rule violations or anomalies are detected, CodeSignal verifies the result.
Step 3: Hiring company notified of decision. Lastly, CodeSignal notifies the hiring company of the proctoring decision and receives an explanation for any rejected submissions. If the evaluation result is not verified, the candidate can retake the assessment after a cooldown period.
Dynamic question rotation
We use dynamic question rotation in our Certified Assessments to minimize the likelihood that any two candidates see the same questions. Our unique framework-based approach to creating questions enables us to create variations at scale, without compromising validity or introducing new levels of difficulty.
Leak Sweep
CodeSignal has a copyright on all of the content on our platform. Leak Sweep, our proprietary technology, searches for leaked questions and flags them to our team. Armed with this information, we can replace leaked questions and request DMCA takedowns—these require LeetCode, Stack Overflow, or any other forum to remove our questions from their site.
Want to learn more?
As GenAI technology continues to evolve, so do our methods of detecting and mitigating its use in coding assessments on our platform. If you’d like to learn more about the cheating prevention techniques described here, we’re happy to talk—set up a time to chat with one of our experts.