Teaching Language Models to Self-Debug Code

Code generated by a language model often runs but produces the wrong result, and fixing it usually depends on a human reviewing the output or on an external error message pointing at the problem. A paper titled “Teaching Large Language Models to Self-Debug,” by Xinyun Chen, Maxwell Lin, Nathanael Schaerli, and Denny Zhou, proposes a method that lets a model correct its own generated code through few-shot prompting. The work is available at https://arxiv.org/abs/2304.05128 and was first submitted in April 2023.

Rubber Duck Debugging for Models

The central idea is what the authors call “rubber duck debugging.” Rather than relying on a human to flag mistakes, the model is prompted to investigate its own output by examining execution results and explaining the generated code in plain language. By describing line by line what the code is supposed to do, the model can surface the discrepancy between its intent and the actual behavior, then propose a fix.

A key point the authors stress is that this happens without human feedback on whether the code is correct and, in some settings, without an error message. The model reasons about its own work and decides what to change. This makes the approach a prompting technique applied to an existing model rather than a separate tool or a retraining step.

How It Was Evaluated

The method was tested on three code generation benchmarks, each covering a different task. Spider is a text-to-SQL benchmark, where a natural language question is turned into a database query. TransCoder covers code translation, specifically converting C++ programs into Python. MBPP is a text-to-Python benchmark made up of basic programming problems described in natural language.

These benchmarks differ in an important way. TransCoder and MBPP come with unit tests, so the model can run its code and observe whether it passes. Spider does not include unit tests, so the model has to rely on explaining its own code rather than on test results. That contrast lets the paper show how self-debugging behaves both when execution feedback is available and when it is not.

Reported Results

The authors report state-of-the-art performance across the benchmarks. On Spider, where no unit tests are available, having the model explain its code consistently improved the baseline by 2 to 3 percent, and improved accuracy on the hardest problems by 9 percent. On TransCoder and MBPP, where unit tests exist, the approach improved baseline accuracy by up to 12 percent.

The paper also points to a gain in sample efficiency. By reusing failed predictions and the feedback drawn from them, self-debugging was able to match or beat baselines that generate more than ten times as many candidate programs. In other words, instead of sampling many attempts and hoping one works, the model can refine a smaller number of attempts and reach comparable results.

Why the Approach Matters

Self-debugging reframes code generation as an iterative process rather than a single shot. The model produces code, inspects it, explains it, and revises it, much as a programmer would talk through a problem to find the flaw. Because the technique is driven by prompting and the model’s own reasoning, it does not require a new model or external correctness labels during use. For tasks like text-to-SQL, code translation, and short Python programs, the paper presents evidence that this loop produces more reliable output than generating code once and accepting it as final.