Updated July 2023
What generative AI means for technical hiring, now and in the future
ChatGPT, OpenAI’s artificial intelligence-powered chatbot, has inspired both awe and fear in the tech industry and beyond. Since its public launch in November 2022, ChatGPT has proven to be adept at drafting college essays, analyzing data, and even writing code—with the latter having obvious implications for technical hiring.
Working in partnership with OpenAI, CodeSignal’s data and engineering teams have analyzed the impact of ChatGPT on our live and asynchronous technical assessments, now and in the future. In this blog post, we’ll address a few of the most common questions about technical hiring and AI-powered tools like ChatGPT:
-
- What is ChatGPT, and how good is it at writing code?
-
- Is it possible to tell if a candidate used ChatGPT or another generative AI solution to solve a CodeSignal question?
-
- What’s the risk of a candidate using ChatGPT to cheat on a CodeSignal assessment?
-
- How does CodeSignal make use of generative AI to enhance our evaluations?
Overview of ChatGPT as a code assist tool
ChatGPT, short for Chat Generative Pre-trained Transformer, is a browser-based chatbot that excels at mimicking human speech. It works a bit like a search engine—but rather than producing a list of search results in response to a query, it generates an original and coherently worded (though often factually incorrect) answer.
If prompted, it can also generate and improve code in a number of popular coding languages, including Python, JavaScript, and C++. Researchers recently tested its skill as a debugger, and found ChatGPT to be on par with or even beat out other automated code repair tools.
The power of ChatGPT to do complex tasks (like writing and debugging code) lies in the innovative training of its AI model, which included billions of data points and human trainers. In fact, CodeSignal worked in close collaboration with OpenAI as they developed and trained the GPT-3 and Codex models that power ChatGPT. Some of our anonymized coding data and questions have even been used (with our consent!) in the training of ChatGPT models.
Understandably, some engineer hiring teams are concerned about how ChatGPT and similar technologies will impact their technical evaluations. Are candidates using ChatGPT to cheat on their assessments? And if so, can it be detected?
Here’s what we at CodeSignal have found based on analysis of thousands of assessment results on our platform.
There are clear tells of ChatGPT-generated code
First, the use of generative AI tools, including ChatGPT, often leaves footprints of detectable indicators that CodeSignal currently tracks using our proprietary technology. We communicate these to company users with our Suspicion Score. The Suspicion Score flags potentially suspicious activity from a variety of sources, including AI-aided plagiarism, previous candidate submissions, and more, by:
- Dissecting each solution to identify patterns across the millions of interviews and assessments conducted on the CodeSignal platform
- Looking beyond code to reliably identify activities correlated with the use of AI coding assistance or unauthorized resources
- Synthesizing these factors to assign a trust level to each result, giving you confidence in your candidates’ results while flagging those that require further review
We also recommend companies use proctoring in their assessments, which provides an additional layer of protection against cheating via ChatGPT.
ChatGPT-based cheating is rare on CodeSignal assessments—and when it happens, it’s ineffective
Additionally, we’ve found that ChatGPT is very rarely used by candidates to copy and paste their way through the hiring process. CodeSignal has analyzed the activity logs for all of our Certified Evaluations to identify where a coding assist tool may have been used. We are confident that a low percentage of CodeSignal assessments have ChatGPT usage. Where proctoring was enabled, the vast majority of these instances were caught during our review process and the results were marked as unverified.
Moreover, among the assessment sessions flagged, the vast majority of candidates did not score high enough to pass the assessment. From our extensive testing of ChatGPT on our Certified Evaluations, we’ve found that in 91% of cases, ChatGPT only makes it possible to earn up to half of all available points. In other words: the chances that a candidate can use ChatGPT to answer questions and pass a CodeSignal Certified Evaluation are extremely low.
The future of technical hiring must embrace, rather than shun, AI tools
At CodeSignal, we believe that AI-assisted coding solutions like ChatGPT and Github Copilot will eventually become the norm in software development. We developed an AI-powered coding assistant, Cosmo, that is built into our IDE.
With Cosmo, candidates can get instant support for navigating the CodeSignal platform—and take advantage of the AI revolution, just as they would on the job, with the ability to write code, debug solutions, and more. Plus, the candidate’s conversation with Cosmo can be recorded and made available to you, providing insight into how well the candidate can leverage generative AI when coding.
Looking to the future, we plan to regularly adapt our Certified Evaluations to continue mimicking what real-world developers do in their respective job categories.
We believe ChatGPT is here to stay and will become a tool that software developers and engineers use in their everyday work. The future of technical hiring requires learning how to embrace this tool and innovate new ways of evaluating core engineering skills that incorporate ChatGPT and other AI resources.