Are any of the Stata commands for a staggered-treatment difference-in-differences efficient with large microdata?

[Ryan Sandler]

I do a lot of my work with a large dataset that follows several million individuals on a monthly basis for as much as 20 years. Often I want to estimate the impact of some kind of shock or event on the individuals. These are individual-level shocks that potentially can have a different treatment time for each individual. The natural thing to do is to estimate a difference-in-differences/event study specification. Since this is a setting with treatment occurring at different times, best practice is now to estimate one of the many staggered DiD specifications out there.

However, I've found that Stata's new-ish built-in commands hdidregress/xthdidregress are unworkably slow on data like mine. I haven't tried all of the user written packages (such as csdid, wooldid, did_multiplegt etc.), but the ones I have tried are also unworkably slow (like, a week+ of runtime on a fast cloud server to run one regression). The various packages seem to be designed with a state/year or maybe county/year panel in mind, with cross-sectional units in the thousands at most, and work poorly with large data. Perhaps it's just a feature of the underlying methods that they don't scale well.

So, my question: Are there any packages, built-in or user-written, that have reasonable run-times for a staggered difference-in-differences estimation with data that has millions of cross-sectional units, and 100+ time periods with treatments potentially occurring in every calendar time period?

[Jeff Wooldridge]

One of the practically useful results from my 2021 working paper and 2025 Empirical Economics paper is that it is sufficient to control for cohort dummies rather than full fixed effects. That equivalence was mostly ignored, except by others in your situation using millions of cross-sectional observations. The result is that, if you have a balanced panel and time-constant controls, controlling for the cohort dummies -- what I call dq, d(q+1), ..., dT, where q is the first treatment time and T is the last period -- is equivalent to putting in a full set of individual fixed effects. I might have 100 cohorts. But putting in 100 dummies is computationally much more efficient than either putting in millions of dummies, or doing the withing transformation of millions of observations. In fact, when I derived the original estimator, I showed consistency of the imputation estimator that uses cohort dummies. That's then the same as running one regression, which is then the same as running one regression with full FEs.

Stata's official command, xthdidregress twfe, implements my approach using unit FEs -- as the name suggests. The user-written command I like best, jwdid, does the same. But you can easily run the single regression yourself and safe a lot on computationally time. It is a bit tedious with many cohorts and covariates, but, in the end, it is just a regression on many fewer dummy variables. I have a shared Dropbox where I show how it's done in simple cases. By using loops and globals it's not too hard to handle complicated cases.

[Ryan Sandler]

Thanks Jeff. So, if I follow your paper correctly, for a leads-and-lags type event study the specification would be interacting the treatment time/event dummies with treatment cohort dummies, and then either aggregating the results or presenting cohort results separately? I'm not sure that would be feasible in my setting, where generally every calendar time period will have at least one treatment cohort. For instance, in an application I'm working on right now I'm following a group of individuals in monthly data over about 4 years, and want to estimate event study coefficients up to 6 months before and 18 months after a shock occurs. Some shocks occur every month. A fully saturated model would require roughly 24*48 coefficients around 1,000 in all. Not technically impossible with modern Stata, but unwieldy to be sure. Am I missing something?

[Jeff Wooldridge]

With a very large cross section you can afford to be flexible. And, just to be clear, approaches such as Callaway-Sant'Anna are doing something very similar inside the black box. Having said that, you might very well want to impose common effects by exposure time. With 48 different exposure times -- actually, 47, with the reference period being just before the intervention -- that's 47 treatment dummy variables. And, conveniently, you only need to control for 24 cohort dummies -- not millions of unit-specific dummies. I'm attaching a simplified version of this approach that I use for teaching purposes. Go to the very end, and you'll see the equivalence between putting in cohort dummies and using full unit fixed effects. So you can do the usual leads and lags event study more efficiently.

[Daniele Girardi]

I do a lot of my work with a large dataset that follows several million individuals on a monthly basis for as much as 20 years. Often I want to estimate the impact of some kind of shock or event on the individuals. These are individual-level shocks that potentially can have a different treatment time for each individual. The natural thing to do is to estimate a difference-in-differences/event study specification. Since this is a setting with treatment occurring at different times, best practice is now to estimate one of the many staggered DiD specifications out there.

However, I've found that Stata's new-ish built-in commands hdidregress/xthdidregress are unworkably slow on data like mine. I haven't tried all of the user written packages (such as csdid, wooldid, did_multiplegt etc.), but the ones I have tried are also unworkably slow (like, a week+ of runtime on a fast cloud server to run one regression). The various packages seem to be designed with a state/year or maybe county/year panel in mind, with cross-sectional units in the thousands at most, and work poorly with large data. Perhaps it's just a feature of the underlying methods that they don't scale well.

So, my question: Are there any packages, built-in or user-written, that have reasonable run-times for a staggered difference-in-differences estimation with data that has millions of cross-sectional units, and 100+ time periods with treatments potentially occurring in every calendar time period?

Hi Ryan, have you tried our -lpdid- command? We built it to be pretty fast also in very large datasets, so it might be a good fit.

[Ryan Sandler]

I haven't tried -lpdid- as yet (cybersecurity policy in my organization makes getting new packages challenging and I haven't tackled that one yet), but the LP-did method is definitely promising. I've spent the last few days staring at your lpdid code on RePEc and it's been super helpful (thank you for contributing the package!). Right now I'm working on DIY version for my application, and hopefully we can get the package in the near future.

[Daniele Girardi]

In addition to the -lpdid- code on Repec, I have posted a few STATA (and R) example codes for "manual" implementation at this repository: https://github.com/danielegirardi/lpdid/ They are exactly meant to help people do DIY implementations, so I hope they can be of help.