On 18 October the economists Sebastian Galiani, Federico A. López and Raul A. Sosa published a VoxEU column summarising their NBER working paper, which measures how often AI writes correct econometric code in Stata, R and Python across 40 tasks. When an AI agent runs, checks and revises its own code, accuracy rises from 81.4% to 99.2% for Claude Sonnet 4.6 and from 77.3% to 95.1% for GPT-5.6 Luna. The errors that remain all come from code that runs to completion and produces full output with wrong numbers.

The authors note that code written by people has never been free of errors either. An audit of 110 studies recently published in leading economics and political science journals found coding errors in about a quarter of them, and such errors can be present even when the posted code reproduces the published numbers exactly.

How the study was done

The benchmark covers linear regression, difference-in-differences, instrumental variables, regression discontinuity, discrete choice, survival analysis and time series. Each task states the method and comes with a simulated dataset and a reference output computed in advance. A run succeeds when the numbers it reports match the reference within a tolerance.

Task cards in a neat grid, each with an icon for a different econometric method, with three software environment windows behind them
The study tested AI agents across econometric methods and software environments (illustration)

Every task was run in Stata, R and Python, with and without an example of a related task in the prompt, and under three degrees of agency:

  • Chatbot: the model writes one script and never sees it run.
  • Self-repair: the researchers run the script and, if it fails or lacks a required value, return the output to the model for revision, up to three times.
  • Constrained agent: the model has only a command line and a prompt that confines it to the task. It runs and revises its code as often as it chooses, is instructed to verify its calculation with an independent check, and decides when it is finished.

Each combination was repeated 10 times with two models, Claude Sonnet 4.6 in Claude Code and GPT-5.6 Luna in Codex, for 14,400 runs in total.

Accuracy rises when the model runs its own code

Success rates by degree of agency:

An assistant walks a loop of writing, running and checking code, beside a rising bar
Letting the model run and revise its own code raised accuracy (illustration)
  • Claude Sonnet 4.6: 81.4% as a chatbot, 93.5% with self-repair, 99.2% as a constrained agent.
  • GPT-5.6 Luna: 77.3%, 90.3% and 95.1%.

Self-repair, with at most three revisions, already achieves 68% of the agent's improvement over the chatbot for Claude Sonnet 4.6 and 73% for GPT-5.6 Luna.

Once the model can run its code, the prompt and the software matter less. Under the chatbot, an example in the prompt raises success by 5.8 percentage points for Claude Sonnet 4.6 and 7.3 points for GPT-5.6 Luna; under the agent it adds 0.7 and 1.5 points. The software shows the same pattern: under the chatbot with no example, Stata trails Python by 25.5 and 30.8 points; under the agent the gap falls to 1.3 and 6.8 points. The authors point out that Stata is the most widely used software in empirical economics.

Every remaining error is silent

The authors divide failures into two kinds:

Two terminal windows: the left shows a warning, the right shows a complete table with one cell quietly wrong
Loud failures are easy to spot; the remaining errors raise no warning (illustration)
  • Loud failures: the script does not run or returns no complete output, so the researcher sees an error message or a missing result.
  • Silent failures: the script runs and returns its output, but at least one reported number is wrong, and nothing in the output flags it.

Pooled over the two models, loud failures fall from 15.8 per 100 runs under the chatbot to zero under the agent, and silent failures fall from 4.9 to 2.9. Total failures fall from 20.7 to 2.9 per 100 runs, and every failure that remains is silent.

The fall in silent failures comes almost entirely from Claude Sonnet 4.6, from 4.7 to 0.8 per 100 runs, against 5.1 to 4.9 for GPT-5.6 Luna. More than two-thirds of the silent failures whose scripts the authors read come from a coding error in an otherwise correct method, such as a wrong index or a wrong term in a formula. Under the agent, the task with the most failures is shift-share instrumental variables, where every failing run returns all its estimates but reports a standard error outside the tolerance.

The paper also notes a limit of the scoring: a run that uses the wrong estimator but reports numbers inside the tolerance is counted as a success, so the reported rates may be higher than the true ones.

Cost and time

At API list prices, the mean cost of a Claude Sonnet 4.6 run rises from 4.56 cents as a chatbot to 14.09 cents as an agent, a factor of 3.1. For GPT-5.6 Luna the factor is 5.0, but an agent run still costs less than one cent on average. The median agent run takes 5.8 times as long as the median chatbot run for Claude Sonnet 4.6 and 3.4 times as long for GPT-5.6 Luna.

For Claude Sonnet 4.6, each extra correct result from the agent costs about $0.54. At research-assistant wages of $15 to $50 an hour, that pays for 0.6 to 2.2 minutes of an assistant's time.

What the authors recommend

The authors find that, on average, the degree of agency matters more than the prompt or the software. Letting the model run and revise its code removes the errors that are cheapest to detect, but it removes only about two-fifths of the silent errors that the output does not flag. The rest must be found by the researcher, who checks whether the model implemented the requested specification and whether its numbers are right. Rerunning the code does not reveal such errors, since code can reproduce the same numbers and still be wrong.

A researcher checks a results table with a magnifying glass and ruler while a second laptop recomputes the same figure
Researchers still have to check whether the numbers are right (illustration)

Researchers who move from chat windows to coding agents meet fewer errors, and a larger share of them are silent. The authors conclude that agents run and revise code at little cost, while judging whether the numbers are right remains the researcher's job.

Taiwan's guidelines for public-sector AI use

Taiwan's Executive Yuan issued its Guidelines for the Use of Generative AI by the Executive Yuan and Its Subordinate Agencies in 2023. Point 2 says that information produced by generative AI must be given an objective and professional final judgement on its risks by the responsible civil servant. Point 5 says agencies must not fully trust generative AI output and must not use unconfirmed output directly as the basis of an administrative act or as the sole basis of a policy decision. Point 6 requires appropriate disclosure when generative AI is used as an aid. Under Point 9, state-owned enterprises, public schools, administrative corporations and government-funded foundations may also apply the guidelines.

This study shows that "confirming" AI-assisted statistical analysis needs concrete steps: code that runs and numbers that look plausible do not mean the result is correct.

Possible directions for Taiwan

  • Test on data with known answers first: research teams and agencies can follow the study's design, preparing simulated data and reference outputs for the methods they use most, setting a tolerance, and testing AI-written code before applying it to real data.
  • Aim checklists at silent errors: check that the estimator and specification match what was requested, paying particular attention to how standard errors are computed, variable indexing and each term in a formula, and recompute key numbers independently in another software package or with another method.
  • Keep reviewable records of public-sector analysis: under Point 5 of the guidelines, AI-produced figures must be confirmed before they enter policy documents. Code, run logs, and model and software versions can be kept together, with a note on which analyses used AI, in line with Point 6.
  • Adjust research and teaching together: when students and research assistants use AI, understanding and verifying the code can be part of the requirement, following the principle in How Students Should Use AI. Research institutions turning papers and methods into AI agents can pair them with this kind of verification; see Paper2Agent and knowledge management.

Sources

  • Sebastian Galiani, Federico A. López and Raul A. Sosa, "Letting AI agents write econometric code", VoxEU (CEPR), 18 October 2026: https://cepr.org/voxeu/columns/letting-ai-agents-write-econometric-code
  • Sebastian Galiani, Federico Ariel López and Raul A. Sosa, "AI Agents and Prompt Engineering in Econometric Coding", NBER Working Paper 35588, August 2026 (revised October 2026): https://www.nber.org/papers/w35588
  • National Science and Technology Council, 行政院及所屬機關（構）使用生成式AI參考指引 (Guidelines for the Use of Generative AI by the Executive Yuan and Its Subordinate Agencies): https://www.nstc.gov.tw/folksonomy/list/c79bf57b-dc94-4aff-8d14-3262b5559cfc?l=ch
  • Central News Agency, 政院通過公部門生成式AI參考指引 禁用於高機密文件, 31 August 2023: https://www.cna.com.tw/news/aipl/202308310120.aspx