"When Should You Adjust Standard Errors for Clustering?"

[Ariel Karlinsky]

I have read the above mentioned paper by Abadie, Athey, Imbens & Wooldridge - and I have a simple question: I have annual (~10 years) US county level data and a county level treatment. What (if any) level should I cluster on? I ask this, since the sampling itself is not clustered (as far as I understand the term), there's no randomization - all (mainland) US counties are sampled and all data are at the county level...

[Phil Bromiley]

I would think it depends on where you believe the variances will change. This may depend on your dv. If it is something that varies by the scale of the county, then you would expect the error variance to vary by county. Number of x in a county year would almost certainly have an error variance that changed with county. x/population might not have an error variance that varied by county (but it might - large counties may have a lower variance since small counties could have a small sample problem). Clustered standard errors by county would also account for the possibility of serial correlation.

[Dimitriy V. Masterov]

The way I read this paper is that there are two reasons for clustering standard errors:

1) a sampling design reason,
2) an experimental design reason

The first happens because you sampled data from a population using clustered sampling, and want to say something about the broader population. This does not apply to your case, since you have all the counties.

The second happens because the assignment mechanism for some causal treatment of interest is clustered. This also does not apply, since you don't have sub-county data.

I think that implies that you don't need to cluster, unless you have repeated observations on each county in your analysis.

[Ariel Karlinsky]

The dataset is county-year panel, so each county is observed each year for about 10 years.

In a sense there's also a "deep" question here, are US counties "the population" or are they a sample from the population which is the entire world? Also are US counties in my time frame the population or a sample from all time periods, before and after?

Maybe Jeff Wooldridge can shed some more light on this

[Dimitriy V. Masterov]

Here's how I understand the deep issue about SEs. The standard errors reflect uncertainty about where and when and how the treatment happens. We don't have access to the population of all the possible ways for the treated counties to be selected over all possible time periods. We only have one sample from the super-population of potential outcomes.

Sometimes one is interested in extrapolating about the effect from the actual treatment that actually happened to other places and times where it did not, so SEs are meaningful in that case. But there are occasions where you don't need to extrapolate, say in a legal case where you want to estimate the damages in of particular event and all you care about the internal validity of that estimate and its significance, so clustering is not necessary and robust errors suffice. So the answer depends on your analytical goals.

To bring it back to clustering, you may want to do it if you want to extrapolate since you have sampled from a super-population of potential outcomes. If your counties change treatment in the ten years, you don't have the second experimental design issue. A second reason to cluster is if you are not aggregating the annual data to county level, so you have dependent data. You may also want to cluster at higher level, say state, if you believe that errors might be correlated because of shared state-level shocks.

What I find troubling is that this goes against the grain of the advice to not cluster in an experimental setting with individual-level treatment, since there are many ways to roll out the treatment.

[Clyde Schechter]

There is also a purely statistical way to think about these questions. The uncertainty that the standard error is understood to express is the uncertainty attributable to sampling variation. (As such it is really an underestimate of uncertainty as it ignores other sources of uncertainty.) Indeed, fundamental to frequentist statistics is the notion that we consider a hypothetical universe of replications of our study and the standard error of any estimator we calculate is supposed to represent the standard deviation of the sampling distribution for that statistic.

So if the nature of your study is such that you would always sample these same counties, then variation outside those counties should not be captured in the standard error calculation, and you should treat those units as a census (and if I interpret Abadie et al. correctly you would not cluster at this level). But if a replication of your study would involve some other (possibly overlapping) set of countries, then variation among counties outside the ones in your study should be captured (and you would cluster at this level, or perhaps at a higher level counting it, depending on the sampling design.)

It is harder to apply this concept to time periods because generally speaking the time periods in our study are restricted to those for which data is available: the future is excluded and often much of the past is as well. Yet we would still like to think of our conclusions as being, if not entirely timeless, at least generalizable to the recent past and near future.

These considerations are sometimes difficult to apply in real-world situations, but I think that even if we don't always come up with an answer, it is important when planning a study to consider which things would be allowed to vary in a replication of the study and which would be replicated exactly. (This is also a consideration in choosing between fixed and random effects designs, by the way.)

[Jeff Wooldridge]

Sorry, I haven't been on in awhile, and I'm bad about checking my gmail account. This is a clear case for clustering at the county level because that's the level at which the treatment is applied. When you have the entire population of counties, the uncertainty is generated by the policy assignment and the potential outcomes across different assignments.

I hope this helps.

[Frederick Guy]

Thank you, Jeff Wooldridge. But one more question: in Abadie et al (2017), you propose a cluster adjustment which differs from the Liang-Zeger adjustment. Do you know if anybody has implemented your new adjustment, in Stata?