Updated April 2024
What generative AI means for technical hiring, now and in the future
ChatGPT, OpenAI’s artificial intelligence-powered chatbot, has inspired both awe and fear in the tech industry and beyond. Since its public launch in November 2022, ChatGPT has proven to be adept at drafting college essays, analyzing data, and even writing code—with the latter having obvious implications for technical hiring.
Working in partnership with OpenAI, CodeSignal’s data and engineering teams have analyzed the impact of ChatGPT on our live and asynchronous technical assessments, now and in the future. In this blog post, we’ll address a few of the most common questions about technical hiring and AI-powered tools like ChatGPT:
-
- What is ChatGPT, and how good is it at writing code?
-
- Is it possible to tell if a candidate used ChatGPT or another generative AI solution to solve a CodeSignal question?
-
- How does CodeSignal make use of generative AI to enhance our evaluations?
Overview of ChatGPT as a code assist tool
ChatGPT, short for Chat Generative Pre-trained Transformer, is a browser-based chatbot that excels at mimicking human speech. It works a bit like a search engine—but rather than producing a list of search results in response to a query, it generates an original and coherently worded (though sometimes factually incorrect) answer.
If prompted, it can also generate and improve code in a number of popular coding languages, including Python, JavaScript, and C++. Researchers recently tested its skill as a debugger, and found ChatGPT to be on par with or even beat out other automated code repair tools.
The power of ChatGPT to do complex tasks (like writing and debugging code) lies in the innovative training of its AI model, which included billions of data points and human trainers. In fact, CodeSignal worked in close collaboration with OpenAI as they developed and trained the GPT-3 and Codex models that power ChatGPT. Some of our anonymized coding data and questions have even been used (with our consent!) in the training of ChatGPT models.
Understandably, some engineer hiring teams are concerned about how ChatGPT and similar technologies will impact their technical evaluations. Are candidates using ChatGPT to cheat on their assessments? And if so, can it be detected?
Here’s what we at CodeSignal have done to address the use of AI tools in our evaluations.
There are clear tells of ChatGPT-generated code
The use of generative AI tools, including ChatGPT, often leaves footprints of detectable indicators that CodeSignal currently tracks using our proprietary technology. We communicate these to company users with our Suspicion Score.
The Suspicion Score flags potentially suspicious activity from a variety of sources, including AI-aided submissions, previous candidate submissions, and more, by:
- Dissecting each solution to identify patterns across the millions of interviews and assessments conducted on the CodeSignal platform
- Looking beyond code to reliably identify activities correlated with the use of AI coding assistance or unauthorized resources (for instance, copying and pasting large blocks of code into CodeSignal IDE)
- Synthesizing these factors to assign a trust level to each result, giving you confidence in your candidates’ results while flagging those that require further review
We also recommend companies use proctoring in their assessments, which provides an additional layer of protection against cheating via ChatGPT.
The future of technical hiring must embrace, rather than shun, AI tools
At CodeSignal, we believe that AI-assisted coding solutions like ChatGPT and Github Copilot will eventually become the norm in software development. We developed an AI-powered coding assistant, Cosmo, that is built into our IDE.
With Cosmo, candidates can get instant support for navigating the CodeSignal platform—and take advantage of the AI revolution, just as they would on the job, with the ability to write code, debug solutions, and more. Plus, the candidate’s conversation with Cosmo can be recorded and made available to you, providing insight into how well the candidate can leverage generative AI when coding.
Looking to the future, we plan to regularly adapt our Certified Evaluations to continue mimicking what real-world developers do in their respective job categories.
We believe ChatGPT is here to stay and will become a tool that software developers and engineers use in their everyday work. The future of technical hiring requires learning how to embrace this tool and innovate new ways of evaluating core engineering skills that incorporate ChatGPT and other AI resources.