First systematic benchmarking of LLMs on Stata

[Khaled Eltokhy]

It is no secret that many researchers and RAs have copy-pasted code from ChatGPT to Stata. It is also true that those same researchers often spend hours debugging "hallucinated" commands or syntax that simply doesn't run.

I created Stata Bench to measure which models are actually reliable for our work.

It is the first such benchmark systematically testing current models on real-world Stata tasks to see which ones generate runnable, accurate code.

The project covers:

• A Leaderboard: Comparing pass rates on data cleaning and analysis tasks.
• Fine-tuning: I am training a smaller model specifically on Stata syntax to reduce hallucinations.

You can see the results here: www.khaledeltokhy.com/benchmarks

[Leonardo Guizzetti]

This is interesting. Have you published the entire suite of benchmarks somewhere?

[Khaled Eltokhy]

This is interesting. Have you published the entire suite of benchmarks somewhere?

I am in the process of organizing the problems themselves in a publishable format. Otherwise I have only shared samples on my website just to show what type of problems exist.

[Leonardo Guizzetti]

Thanks, please post back when they are ready. In look forward to reading it.

[Khaled Eltokhy]

Thank you for the interest! I definitely will soon!

[Khaled Eltokhy]

Thanks, please post back when they are ready. In look forward to reading it.

As promised, I have updated the benchmark and included the latest GPT-5.2 model, and added a library of the tasks posted. Once any overlooked issues are found, I will fix and rerun on all 20 models.


You can see on statabench.com

[Leonardo Guizzetti]

Thank your for posting back. I'm adding this to my Christmas reading list.

[Khaled Eltokhy]

Since the creation of this was largely automated, there is bound to be some errors in the dataset. I like to think that they apply equally to all models so they are not an issue (for example I caught a task where it would always pass no matter what - not ideal, but doesn't change relative rankings). So, I am happy to receive any feedback of that kind to incorporate for an eventual V3 (these things get quite expensive to run ha).

I appreciate your interest truly!