insufficient observations to compute bootstrap error in the case of logit diff-in-diff estimation, clusters present

[Ilinca Balasa]

Hi everyone. I had a question related to bypassing this following error: insufficient observations to compute bootstrap standard errors; no results will be saved
r(2000);

One of the previous posts on this particular issue recommended the use of the -nodrop- option, but even if I use that I still get the same result. It is worth also mentioning that my data is clustered into 33 clusters of varying sizes. When I ran my logit regression with bootstraps but did not specify any clusters, this seemed to work, but I do think it is very important that I am somewhat able to include clusters since they represent geographical departments. Anyways, here is my code:

. logit sec_comp younger_post younger_cohort hh_in hh_memb urban male i.father_educ_clean i.mot
> her_educ_clean i.resi_dept, vce(bootstrap, cluster(resi_dept))
(running logit on estimation sample)

Bootstrap replications (50): xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx done
x: Error occurred when bootstrap executed logit.
insufficient observations to compute bootstrap standard errors; no results will be saved
r(2000);

. logit sec_comp younger_post younger_cohort hh_in hh_memb urban male i.father_educ_clean i.mot
> her_educ_clean i.resi_dept, vce(bootstrap, cluster(resi_dept) nodrop)
(running logit on estimation sample)

Bootstrap replications (50): xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx done
x: Error occurred when bootstrap executed logit.
insufficient observations to compute bootstrap standard errors; no results will be saved
r(2000);

. logit sec_comp younger_post younger_cohort hh_in hh_memb urban male i.father_educ_clean i.mot
> her_educ_clean i.resi_dept, vce(bootstrap)
(running logit on estimation sample)

Bootstrap replications (50): x..x.xx..10.x..xx.xx20x..x..xx.30.x...x..xxxx....x .x50 done
x: Error occurred when bootstrap executed logit.

Logistic regression Number of obs = 13,626
Replications = 29
Wald chi2(28) = .
Prob > chi2 = .
Log likelihood = -8050.9903 Pseudo R2 = 0.1453

--------------------------------------------------------------------------------------
| Observed Bootstrap Normal-based
sec_comp | coefficient std. err. z P>|z| [95% conf. interval]
---------------------+----------------------------------------------------------------
younger_post | .3225229 .077061 4.19 0.000 .171486 .4735598
younger_cohort | -.4392233 .0397712 -11.04 0.000 -.5171733 -.3612732
hh_in | .2429134 .0251573 9.66 0.000 .1936061 .2922208
hh_memb | -.1829674 .0132483 -13.81 0.000 -.2089336 -.1570011
urban | .8106458 .0309223 26.22 0.000 .7500391 .8712525
male | -.3513826 .0287996 -12.20 0.000 -.4078288 -.2949363
|
father_educ_clean |
Primary partial | .356788 .0615166 5.80 0.000 .2362177 .4773583
Primary complete | .637722 .0790806 8.06 0.000 .4827269 .7927171
Secondary partial | .5674029 .105006 5.40 0.000 .3615949 .7732109
Secondary complete | 1.057469 .099011 10.68 0.000 .8634114 1.251528
Technical partial | 1.060628 .6810672 1.56 0.119 -.2742392 2.395495
Technical complete | 2.418008 .6098454 3.96 0.000 1.222733 3.613283
University partial | .7927246 .6430799 1.23 0.218 -.4676889 2.053138
University complete | 1.505658 .2085458 7.22 0.000 1.096916 1.9144
|
mother_educ_clean |
Primary partial | .4339611 .0744824 5.83 0.000 .2879784 .5799439
Primary complete | .747999 .0784476 9.54 0.000 .5942446 .9017534
Secondary partial | .7592562 .0895665 8.48 0.000 .5837091 .9348033
Secondary complete | 1.411128 .1046602 13.48 0.000 1.205998 1.616258
Technical partial | 1.820885 .5980049 3.04 0.002 .6488167 2.992953
Technical complete | 2.013862 .4171813 4.83 0.000 1.196202 2.831523
University partial | 1.964985 .4704994 4.18 0.000 1.042823 2.887147
University complete | 1.898383 .3639793 5.22 0.000 1.184997 2.611769
|
resi_dept |
8 | .1572113 .1650349 0.95 0.341 -.1662513 .4806738
11 | .7938662 .2126067 3.73 0.000 .3771647 1.210568
13 | .389389 .1567788 2.48 0.013 .0821083 .6966697
15 | .3757984 .181136 2.07 0.038 .0207783 .7308184
17 | -.1046539 .1881843 -0.56 0.578 -.4734884 .2641806
18 | -.3566946 .1198382 -2.98 0.003 -.5915732 -.121816
19 | .4435121 .1621094 2.74 0.006 .1257835 .7612406
20 | .3630252 .1690425 2.15 0.032 .0317081 .6943424
23 | .4098738 .1618056 2.53 0.011 .0927406 .7270069
25 | .3270911 .2090632 1.56 0.118 -.0826653 .7368474
27 | .3476945 .1922003 1.81 0.070 -.0290112 .7244001
41 | -.0317402 .1632329 -0.19 0.846 -.3516709 .2881904
44 | .0176467 .1699824 0.10 0.917 -.3155128 .3508061
47 | .0612929 .1475288 0.42 0.678 -.2278582 .350444
50 | .00478 .1735762 0.03 0.978 -.335423 .3449831
52 | -.0025456 .172667 -0.01 0.988 -.3409667 .3358756
54 | .1462255 .1539708 0.95 0.342 -.1555517 .4480026
63 | -.1020115 .1970536 -0.52 0.605 -.4882294 .2842065
66 | -.0157981 .1752316 -0.09 0.928 -.3592458 .3276496
68 | .5321754 .1943338 2.74 0.006 .1512882 .9130626
70 | .5118829 .1501395 3.41 0.001 .2176149 .8061509
73 | .1748099 .1417761 1.23 0.218 -.1030661 .4526859
76 | .0392129 .1940722 0.20 0.840 -.3411616 .4195875
81 | .0511738 .1564848 0.33 0.744 -.2555309 .3578784
85 | .1769258 .181233 0.98 0.329 -.1782844 .5321359
86 | .3925925 .1701294 2.31 0.021 .059145 .72604
88 | .9558104 .4958017 1.93 0.054 -.015943 1.927564
91 | .5041503 .1676043 3.01 0.003 .175652 .8326487
94 | -.6275042 .2490393 -2.52 0.012 -1.115612 -.1393962
95 | .0485914 .1893 0.26 0.797 -.3224299 .4196126
97 | .172662 .1497298 1.15 0.249 -.120803 .466127
99 | -.0859054 .1775655 -0.48 0.629 -.4339274 .2621166
|
_cons | -3.769914 .4051683 -9.30 0.000 -4.564029 -2.975799
--------------------------------------------------------------------------------------
Note: One or more parameters could not be estimated in 21 bootstrap replicates;
standard-error estimates include only complete replications.

Please let me know what you think. Apologies if this is badly formatted, I am new to the forum and in a bit of a time crunch.

[Clyde Schechter]

Well, you can see from all the x's in the output that the logit-command is simply not executing on the bootstrap samples, except in the final attempt, and even there it does so less than half the time. The list of reasons this could be happening is quite long. Go back to your original version, with clustering on resi_dept, but add the -noisily- suboption inside the -vce()- option. (i.e. -vce(bootstrap, cluster(resi_dept) noisily)-.)

When that runs, the actual runs of -logit- on the bootstrap samples will produce output, most importantly, you will see the error messages that explain why -logit- is failing on the bootstrap samples. Then you can focus your efforts on resolving whatever problem(s) that uncovers.

[Ilinca Balasa]

Well, you can see from all the x's in the output that the logit-command is simply not executing on the bootstrap samples, except in the final attempt, and even there it does so less than half the time.

Thank you for your prompt reply, I did not know that that is what those x's meant! I reran my specification with the noisy version instead and for quite a few of the tries I got this following error message: "collinearity in replicate sample is not the same as the full sample, posting missing values". I looked it up but barely found anything. If this means what I think it does, it doesn't immediately raise red flags because I guess that some with enough repetitions of re-sampling this is bound to happen at some point. I am worried though that the vce(bootstrap) automatically runs 50 simulations which does not seem all that much to me. Would it help at all if I only ran boottest with my variable/s of interest specifying the clusters? I did this earlier and it seemed to be fine. If not, would it be worth including in results at all? Thank you again.

[Clyde Schechter]

...it doesn't immediately raise red flags because I guess that some with enough repetitions of re-sampling this is bound to happen at some point.

But it should raise red flags when 50 out of 50 reps fail! Yes, you can tolerate a small number of failed replications, but when most, or even all, of the replications fail, that says you are doing something wrong. In this case, I suggest you try changing the -vce()- specification to -vce(bootstrap, cluster(resi_dept) idcluster(newvarname)-, replacing newvarname with a valid variable name that is not already used in your data set. I think that will solve your problem.

[Ilinca Balasa]

I suggest you try changing the -vce()- specification to -vce(bootstrap, cluster(resi_dept) idcluster(newvarname)-,

It did not unfortunately, still same results and error with all reps failing. Really unsure what to do.

[Felix Bittmann]

Without the actual data it can be hard to tell what goes wrong. I suggest you start with a very simple model, something like:

logit sec_comp younger_post, vce(bootstrap)

And then start adding more variables. Maybe the cluster is also the issue. As soon as a change suddenly produces the Xs, you can be confident that the problem might be there and dig deeper.