DeepMind has created a system called AlphaCode that writes computer programs at a competitive
level.
DeepMind is the subsidiary of Google's parent company Alphabet.
AlphaCode achieved an estimated rank within the top 54% of participants in programming
competitions by solving new problems that require a combination of critical thinking,
logic, algorithms, coding, and natural language understanding.
AlphaCode uses transformer-based language models to generate code at an unprecedented
scale, and then smartly filters to a small set of promising programs.
DeepMind researchers validated the performance using competitions hosted on Codeforces, a
popular platform which hosts regular competitions that attract tens of thousands of participants
from around the world who come to test their coding skills.
The researchers selected for evaluation 10 recent contests, each newer than their training
data.
AlphaCode placed at about the level of the median competitor, marking the first time
an AI code generation system has reached a competitive level of performance in programming
competitions.
Competitive programming is a popular and challenging activity; hundreds of thousands of programmers
participate in coding competitions to gain experience and showcase their skills in fun
and collaborative ways.
During competitions, participants receive a series of long problem descriptions and
a few hours to write programs to solve them.
Typical problems include finding ways to place roads and buildings within certain constraints,
or creating strategies to win custom board games.
Participants are then ranked mainly based on how many problems they solve.
Companies use these competitions as recruiting tools and similar types of problems are common
in hiring processes for software engineers.
The problem-solving abilities required to excel at these competitions are beyond the
capabilities of existing AI systems.
However, by combining advances in large-scale transformer models (that have recently shown
promising abilities to generate code) with large-scale sampling and filtering, DeepMind
researchers made significant progress in the number of problems they can solve.
They pre-train their model on selected public GitHub code and fine-tune it on their relatively
small competitive programming dataset.
At evaluation time, they create a massive amount of C++ and Python programs for each
problem, orders of magnitude larger than previous work.
Then they filter, cluster, and rerank those solutions to a small set of 10 candidate programs
that they submit for external assessment.
This automated system replaces competitors’ trial-and-error process of debugging, compiling,
passing tests, and eventually submitting.
With the permission of Codeforces, DeepMind evaluated AlphaCode by simulating participation
in 10 recent contests.
The impressive work of the competitive programming community has created a domain where it’s
not possible to solve problems through shortcuts like duplicating solutions seen before or
trying out every potentially related algorithm.
Instead, the model must create novel and interesting solutions.
Overall, AlphaCode placed at approximately the level of the median competitor.
Although far from winning competitions, this result represents a substantial leap in AI
problem-solving capabilities and these results will inspire the competitive programming community.
AlphaCode ranked within the top 54% in real-world programming competitions, an advancement that
demonstrates the potential of deep learning models for tasks that require critical thinking.
These models elegantly leverage modern machine learning to express solutions to problems
as code, circling back to the symbolic reasoning root of AI from decades ago.
And this is only a start.