Checking Clustered Standard Errors / p-value

[Siew Sanz Ng]

Hello! I am new to STATA and would appreciate some guidance in this.

I am running a clustered regression with just 1 dependent variable (access_base) and 1 independent variable (treatment_pooled). The cluster variable is time_afternoon and I am also including fixed effects using daily_strata_base

In the STATA results, coefficient is -0.308, standard error is 0.093, and there are 9563 observations. The t-statistic is -3.30 and so, I am guessing that there should be a low p-value. But the p-value is very high at 0.187 and the code/output is as below:


reg access_base treatment_pooled i.daily_strata_base, vce(cluster time_afternoon)

Linear regression Number of obs = 9,563
F(0, 1) = .
Prob > F = .
R-squared = 0.1383
Root MSE = 3.6333

(Std. err. adjusted for 2 clusters in time_afternoon_base)
-----------------------------------------------------------------------------------
| Robust
access_base | Coefficient std. err. t P>|t| [95% conf. interval]
------------------+----------------------------------------------------------------
treatment_pooled | -.3077829 .0932648 -3.30 0.187 -1.492825 .8772587


I am wondering how do I check if the standard errors and p-value is calculated correctly? Thank you very much!

[Clyde Schechter]

It is all calculated correctly. The problem is with your use of clustered standard errors. You have only two clusters--so clustered standard errors are not valid. While there is no simple rule of thumb how many clusters are needed, you shouldn't even think about them with fewer than 15 clusters, and most people would recommend a much larger number like 50 or even 100.

Putting that misuse of clustering aside, and imagining that it was OK, when you have n clusters, the degrees of freedom for your t-statistic is not based on the number of observations in the estimation sample. It is, instead, the n-1; in your case that's 2-1 = 1. (You can also see this in the header of the output, where it shows F(0, 1) = .; you have only 1 df.) The p-value associated with t = -3.30 with 1 df is, indeed, 0.187 (to 3 decimal places).

[Rajdeep Chaudhuri]

Hi Clyde,
I was just wondering, when we have 2 clusters, is there any way to get a clustered standard error? For example, in a canonical 2x2 DiD model, where I have 1 treatment and 1 control group, then I should cluster at the treatment level, but I have only 2 such groups. How should we go about in estimating the DiD then?

[Clyde Schechter]

The generic analysis for a 2 group DID where all groups undergo the intervention simultaneously is:

regress outcome i.treatment##i.pre_post

where treatment is a dichotomous variable indicating treatment vs control group and pre_post is a dichotomous variable indicating after vs before intervention.
If the units of observation are measured repeatedly in the pre or post periods, so that the data set is panel data, and if we call the variable that defines the individual panels, panel, then it may be appropriate to also add, -vce(cluster panel)- if you have sufficiently many of them. In no event would you use -vce(cluster treatment)-. If you have only a single treatment panel and a single control panel, then you would not use clustered standard errors at all, just use ordinary standard errors. (I don't recommend doing DID analysis at all when there is just a single treatment panel and a single control panel, but if you were going to do it, that is how.)

[Carlo Lazzaro]

Rajdeep:
for an interesting paper on clustering, see Cameron_Miller_Cluster_Robust_October152013.pdf

[Rajdeep Chaudhuri]

Thanks Clyde.
If I understand correctly, suppose there is a treatment that is given to a state A, and not to another state B and C. Each state has districts under it A1, A2, A3 and B1 B2, and C1 C2 C3 C4( I mean the number of districts under each state could be different) So now I have a treatment variable which takes 1 for A and 0 for B and C
Now if I do

Code:

reg outcome i.treatment ##i.pre_post

Then should I do -vce(cluster district) or -vce(cluster state) or no clustering at all?

[Clyde Schechter]

Given the small number of states (3) and districts (9) I would not cluster at all.