the study
Author issued on December 8, 2022
Alpha Code Team
Note: This blog was first released on February 2, 2022. On December 8, 2022, a minor update of text was performed to reflect this following the publication of a science paper.
Solve new problems and set new milestones with competitive programming
Creating an unexpected problem solution is a second nature in human intelligence. This is the result of critical thinking notified by experience. The machine learning community has made great progress in the generation and understanding of text data, but the problem solving progress is limited to relatively simple mathematics and programming issues, and the acquisition and copy of existing solutions. Limited to copy.
As part of the DeepMind mission to solve intelligence, we have created a system called Alphacode, which writes computer programs at a competitive level. AlphaCode has achieved an estimated rank within the top 54 % of the participants in programming competition by solving new problems that require a combination of critical thinking, logic, algorithm, coding, and natural languages.
This paper on the cover of science uses a transformer -based language model to generate code on an unprecedented scale and smartly filter a small set of promising programs.
We verified the performance using the competition hosted by Code Force. This is a popular platform that holds regular competitions that attract tens of thousands of participants from all over the world to test coding skills. We chose to evaluate the new 10 recent contests rather than training data. ALPHACODE has recorded a mark when the AI code generation system has reached a competitive level performance in programming competitions at the median level of competitors.
We have released a competitive programming issue and solution datasets for other people to build based on the results. This includes a wide range of tests to confirm that the programs that pass these tests are correct. I hope this benchmark will lead to problem solving and further innovation of code generation.
The problem was from CodeForces and the solution was generated by Alpha Code.
Competitive programming is a popular and challenging activity. Hundreds of thousands of programmers participate in coding competitions, gain experience, and introduce their skills in a fun and cooperative way. During the competition, the participants will explain a series of long questions and receive a few hours to create a program to solve them.
Typical issues include finding a way to place roads and buildings in specific restrictions and creating a strategy to win custom board games. After that, the participants are ranked mainly based on the number of problems to be solved. Companies use these competitions as recruitment tools, and the same type of problem is common in the recruitment process of software engineers.
“”
You can safely say that the result of Alpha Code has exceeded my expectations. I was skeptical, even in simple competitive issues, not only implementing algorithms (and this is the most difficult part). ALPHACODE managed to perform at the promising new competitors. I can’t wait for what’s first!
Founder of Mike Mirzayanov, Codeforces
The problem -solving ability required for these competitions exceeds the abilities of the existing AI system. However, the combination of large -scale transformer models (recent promising abilities to generate code) and large -scale sampling and filtering advances have made great progress in the number of solutions. Train the model in advance with the selected public GitHub code, and make fine adjustments with relatively small competitive programming datasets.
At the time of evaluation, create a huge amount of C ++ and Python programs for each problem. This is a few digits than the previous work. Next, filter, cluster, and reconfirm these solutions in a small set of 10 candidate programs submitted for external evaluation. This automated system is replaced by the debugging of competitors, compilation, passing tests, and the trial and error process of competitors for the final transmission.
Alpha cord was evaluated by simulating the recent 10 contests with the permission of Codeforces. A domain was created by impressive work in a competitive programming community. This domain has created a domain that cannot solve problems through shortcuts, such as duplicating the solutions found earlier or trying out all related algorithms. Instead, the model needs to create a novel and interesting solution.
Overall, Alpha Cord was placed at almost the median level of competitors. Although it is far from competition, this result indicates a significant leap in AI’s problem -solving ability, hoping that our results will affect the competitive programming community.
“”
It is very difficult to solve competitive programming issues, requiring both excellent coding skills and human creativity. I was very impressed that Alphacode could make progress in this field, and I was excited to see how to create code using statement understanding, lead a random search and create a solution.
PETR Mitrichev, Software Engineer, Google & World-Class competitive programmer
In order for artificial intelligence to support mankind, our system needs to develop problem -solving skills. ALPHACODE was ranked in the top 54 % in the real world programming competition. This is a progress that indicates the possibility of a deep learning model in a task that requires critical thinking. These models have been used elegantly to express modern machine learning as a code to express the solution to the problem, and have returned to the AI symbolic route for decades. And this is just the beginning.
In search of code generation, there is room for improvement, the programmer improves productivity, and hinders more exciting ideas to open the field to those who are not currently writing code. We hope that this survey will continue to be a tool to strengthen programming and to approach the problem solving AI.
Display AlphaCode solutions and explore the model on Alphacode.deepmind.com
