Clustering in csdid: Why is clustering by macroarea inappropriate?

[Adriano Ruggiero]

Dear all,

I'm currently estimating a staggered Difference-in-Differences model using the csdid command in Stata, following the Callaway and Sant’Anna (2021) framework. My units of observation are hospitals (org_id), observed over multiple years (years), and some receive treatment in different years (first_treatment).

Initially, I ran the following command:

csdid outcome, ivar(org_id) time(year) cluster(macroarea) gvar(first_treatment) method(drimp) notyet

In this specification, macroarea refers to 4 broad geographical areas in Italy: North, Center, South, and Islands.

However, I received the following critique from a reviewer:

"Clustering by macroarea rather than by ASL or hospital is incorrect. Clustering should be done at the level at which the treatment is assigned. Using macroarea artificially improves the results because you are defining the variation in the standard errors at too aggregate a level."

My understanding is that this critique refers to the fact that:

1. The treatment is assigned at the hospital (org_id) level.
2. Clustering at too high a level (macroarea) underestimates standard errors, potentially leading to over-rejection of the null hypothesis.

Indeed, when trying to switch to cluster(org_id), I encountered the error:

org_id may not be both target and by()

And I understand this stems from the fact that ivar() and cluster() are the same variable in combination with method(drimp) (and wboot). My questions are:

• Is it possible that the reviewer made a mistake and the methodology used is correct?
• Is there a workaround that lets me specify a robust clustering scheme that respects the level of treatment assignment?

I’d really appreciate any clarification or references on best practices in this situation. Thank you in advance!

Best regards,
Adriano

[Jeff Wooldridge]

As a statistical matter, clustering when there are only four clusters should be avoided because the validity of the standard errors relies on having sufficiently many clusters. You can often get away with 30, maybe even 20, clusters if you have some treated units in enough clusters. With G = 4 clusters the standard errors are likely downward biased — probably what the reviewer is alluding to. Was the policy assigned at the region level? Seems unlikely given it’s a staggered rollout.

[FernandoRios]

Also
if using cadid with panel, results automatically cluster by ivar

[Adriano Ruggiero]

Thank you very much for your reply Jeff Wooldridge. The policy was introduced at the national level, but the implementation varied across hospitals over time. I also tried clustering at the regional level (Italy has 20 regions), which would give me more clusters than macroareas. However, I encountered a different but equally important issue: when clustering at the regional level, the parallel trends assumption no longer holds, for this reason I opted for clustering with macroareas.

[Adriano Ruggiero]

Yes, I had the same impression, FernandoRios , that's why I believe my approach may not be entirely inappropriate after all.

[Jeff Wooldridge]

If the implementation is at the discretion of the hospitals then I would cluster at the hospital level. Making the violation of PT insignificant by clustering at an essentially arbitrary higher level, which cannot be supported statistically, won't convince anyone. I would instead include region x year or macroarea x year fixed effects. This allows unrestricted trends by, say, region.

BTW, this is easy to do in jwdid. Just include the region dummies among x, and Fernando's impressive code does the rest.