[Hanyi Wang]

Ok I see, thanks a lot!

[Tom Coupe]

I am trying to replicate a paper - the paper doesn't specify which commands have been used - should boottest and cgmwildboot give the same result if one imposes the null and sets the seed to be the same?

[Sonal Agarwal]

Dear David,

I and a couple of others are facing an issue with the boottest command. We are using Stata/MP 14.1 and trying to run the following commands:

eststo clear

global controls CH_F##CH_BirthOrd CH_Sib M_Age M_Ed M_Wt4Ht HH_Mus HH_Hin HH_SCST HH_Urban HH_Mem HH_Wealth HH_Fam

eststo xi: reghdfe NN_D Treat_E##FB_f##CH_F Treat_P Treat_S $controls [pw=perweight] if kidbirthy>1985, a(i.district##c.kidbirthyr) cluster(stateid)
boottest 1.Treat_E#1.FB_f#1.CH_F, reps (9999) nograph noci bootcluster (stateid)
return list
estadd scalar p_boot = r(p)

These commands return the following error "factor variables and time-series operators are not allowed" when trying to run boottest. But when someone else tried on their computer, it worked for them. They use the same state version and ran the same commands as these. We're really unsure what the issue might be? There has been no post resolving this kind of an error. It would be really helpful if you could help us out.

Thanks,

Sonal Agarwal

[Stephen Jenkins]

Your call to -esto- includes a call to "xi". Why? Factor variables were available in version 14. What happens if you simply omit it?

Also check whether your command works without the -esto- prefix. If it does, that will tell you (I think) that your problem is somehow the combination of -esto- and "xi"

[Elly Field]

Are there any additional resources on implementing boottest after a gsem model? I have a moderate number of clusters (22) and am trying to figure out if boottest is an appropriate approach rather than clustering the standard errors with vce().

Here's my current code. Should I be including robust standard errors as well? Or just testing the coefficients using bootttest?

 gsem (charters_chng <- lag_nhwhite_5to9 $nhoodcovar_00 i.distnum) ///
(magnets_chng <- lag_nhwhite_5to9 $nhoodcovar_00 i.distnum) ///
(privates_chng <- lag_nhwhite_5to9 $nhoodcovar_00 i.distnum) ///
(nh_chng_10 <- $numbchng nhwhite_5to9_1990 $nhoodcovar_chng i.distnum) ///
(sch_chng_10 <- nh_chng_10 $numbchng white_school_1990 $nhoodcovar_chng i.distnum) ///
, covstructure(e._En, diagonal)

boottest [charters_chng]lag_nhwhite_5to9 , bootcluster(distnum)

Thanks for any help!

[David Roodman]

If you think errors are correlated within each distnum-defined cluster, you should probably add cluster(distnum) to the gsem command line. If you don't then you should add it to the boottest command line.

You don't need the bootcluster() option. That only becomes relevant when the standard errors are multiway clustered, and you have to choose one grouping for the bootstrapping process.

Because gsem models are in general nonlinear, boottest will only run the score bootstrap, which is discussed in the paper.

[Chinmay Korgaonkar]

Hello David Roodman and everyone,

I request advice regarding the use of score bootstrap-t test for the nonlinear models in boottest command.

I have a highly right-skewed positive valued outcome variable with 0.41% of valid zero observations. I use Poisson pseudo-maximum likelihood estimation. Because of small number of heterogeneous clusters, I obtain score bootstrap p values from boottest.

While the cluster-robust standard errors indicate statistical insignificance, the score bootstrap p values suggest statistical significance at 5% significance level. I see this pattern in all four specifications for the variable that I estimate.

See one example.

 
. poisson cash_in_hand_bank_real did i.state_cd i.year i.NIC_2008 age age2 i.rural_urban_cd [p
> weight=mult], vce(cluster NIC_2008)
note: you are responsible for interpretation of noncount dep. variable.

Iteration 0:   log pseudolikelihood = -6.499e+11  
Iteration 1:   log pseudolikelihood = -6.453e+11  
Iteration 2:   log pseudolikelihood = -6.452e+11  
Iteration 3:   log pseudolikelihood = -6.452e+11  
Iteration 4:   log pseudolikelihood = -6.452e+11  
Iteration 5:   log pseudolikelihood = -6.452e+11  

Poisson regression                                      Number of obs = 18,736
                                                        Wald chi2(22) =      .
Log pseudolikelihood = -6.452e+11                       Prob > chi2   =      .

                                         (Std. err. adjusted for 24 clusters in NIC_2008)
-----------------------------------------------------------------------------------------
                        |               Robust
 cash_in_hand_bank_real | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
------------------------+----------------------------------------------------------------
                    did |  -.0345498   .1429369    -0.24   0.809     -.314701    .2456014
                        |
  -----
-----------------------------------------------------------------------------------------

. boottest did, cluster(NIC_2008) reps(9999) weighttype(rademacher) nograph

Overriding estimator's cluster/robust settings with cluster(NIC_2008)

Re-running regression with null imposed.

note: you are responsible for interpretation of noncount dep. variable.

Iteration 0:   log pseudolikelihood = -6.501e+11  
Iteration 1:   log pseudolikelihood = -6.453e+11  
Iteration 2:   log pseudolikelihood = -6.452e+11  
Iteration 3:   log pseudolikelihood = -6.452e+11  
Iteration 4:   log pseudolikelihood = -6.452e+11  
Iteration 5:   log pseudolikelihood = -6.452e+11  

Poisson regression                                  Number of obs =     18,736
                                                    Wald chi2(22) = 1213856.30
Log pseudolikelihood = -6.452e+11                   Prob > chi2   =     0.0000

 ( 1)  [cash_in_hand_bank_real]did = 0
                                         (Std. err. adjusted for 24 clusters in NIC_2008)
-----------------------------------------------------------------------------------------
                        |               Robust
 cash_in_hand_bank_real | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
------------------------+----------------------------------------------------------------
                    did |          0  (omitted)
                        |
------
-----------------------------------------------------------------------------------------

Score bootstrap-t, null imposed, 9999 replications, Wald test, clustering by NIC_2008, bootstr
> ap clustering by NIC_2008, Rademacher weights:
  did

                               z =    -2.6161
                        Prob>|z| =     0.0092

Roodman and co-authors, in Para 6.2 of their paper, advice caution while using the score bootstrap. Accordingly, as a robustness check, I drop 0.41 % zeroes in the variable, estimate a log-linear model, and obtain restricted wild cluster bootstrap-t p values. Here, the p values denote statistical insignificance.

See the example.

. generate cash_in_hand_bank_real_ln = .
(18,817 missing values generated)

. replace cash_in_hand_bank_real_ln = log(cash_in_hand_bank_real) if cash_in_hand_bank_real >
> 0 & cash_in_hand_bank_real < .
(18,664 real changes made)

. reghdfe cash_in_hand_bank_real_ln did i.state_cd i.year age age2 i.rural_urban_cd [pweight=m
> ult], absorb(NIC_2008) vce(cluster NIC_2008)
(MWFE estimator converged in 1 iterations)
warning: missing F statistic; dropped variables due to collinearity or too few clusters

HDFE Linear regression                            Number of obs   =     18,664
Absorbing 1 HDFE group                            F(  38,     23) =          .
Statistics robust to heteroskedasticity           Prob > F        =          .
                                                  R-squared       =     0.1535
                                                  Adj R-squared   =     0.1507
                                                  Within R-sq.    =     0.0764
Number of clusters (NIC_2008) =         24        Root MSE        =     2.0800

                                         (Std. err. adjusted for 24 clusters in NIC_2008)
-----------------------------------------------------------------------------------------
                        |               Robust
cash_in_hand_bank_rea~n | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
------------------------+----------------------------------------------------------------
                    did |   .0679519   .1073844     0.63   0.533    -.1541896    .2900935
                        |
----
-----------------------------------------------------------------------------------------

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
    NIC_2008 |        24          24           0    *|
-----------------------------------------------------+
* = FE nested within cluster; treated as redundant for DoF computation

. boottest did, cluster(NIC_2008) reps(9999) weighttype(rademacher) nograph

Overriding estimator's cluster/robust settings with cluster(NIC_2008)

Wild bootstrap-t, null imposed, 9999 replications, Wald test, clustering by NIC_2008, bootstra
> p clustering by NIC_2008, Rademacher weights:
  did

                           t(23) =     0.6328
                        Prob>|t| =     0.5768

95% confidence set for null hypothesis expression: [−.1486, .3701]

Thank you very much.

[David Roodman]

I'm not sure what to say about this. Two different estimation results are producing different results. I notice that in the poisson regression, the "Wald chi2(22)" is missing. Do you understand what that is about?

It is good that you are being careful with this result.

That paper mentioned the case of Pitt and Khandker (1998). In reanalyzing that, I found it useful to do a non-parametric bootstrap, because that varies the data set and passes each data set fully through the non-linearity of the estimator. Something like:

bs, cluster(NIC_2008): poisson cash_in_hand_bank_real did ...

[Chinmay Korgaonkar]

Thank you very much David Roodman

On the missing Wald chi-square, I have 24 clusters. My understanding is that the missing model-level Wald statistic means that the estimated VCE is not of sufficient rank to perform the model test. The specification includes did, state fixed effects, year fixed effects, industry fixed effects, and covariates. So the overall model test involves more coefficients that would be constrained to 0, than 24. I consider this as a limitation for the model-level joint test, rather than a problem with the coefficient-level estimate for did.

I tried the non-parametric cluster bootstrap using the bootstrap prefix, but Stata returned “weights not allowed” because my Poisson specification includes pweights. I will try some way out or run an unweighted version as a diagnostic check.

Thank you once again.