Bootstrapping standard errors moves the p-value of a variable from 0.000 to 1.000

[Hakan Gunduz]

Hi all,

To overcome the generated regressor problem, I perform bootstrapping to correct the standard errors by using the following code:

xtprobit y1 x1 x2 x3, vce(bs)

However, bootstrapping moves the p-value of one of the variables from 0.000 to 1.000. What could be the reason for this? Here is the two outputs with without -vce(bs)- option:

xtprobit AcquirerStatus LeverageDeficit Size Markettobook Profitability, vce(cluster CompanyNo)

Calculating robust standard errors:

Random-effects probit regression                Number of obs     =      2,547
Group variable: CompanyNo                       Number of groups  =        248

Random effects u_i ~ Gaussian                   Obs per group:
                                                              min =          1
                                                              avg =       10.3
                                                              max =         23

Integration method: mvaghermite                 Integration pts.  =         12

                                                Wald chi2(4)      =      67.27
Log pseudolikelihood  = -668.40718              Prob > chi2       =     0.0000

                               (Std. Err. adjusted for 248 clusters in CompanyNo)
---------------------------------------------------------------------------------
                |               Robust
 AcquirerStatus |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
----------------+----------------------------------------------------------------
LeverageDeficit |  -.4867099   .2280932    -2.13   0.033    -.9337643   -.0396555
           Size |   .2556389    .091086     2.81   0.005     .0771137    .4341641
   Markettobook |  -.0000113   1.82e-06    -6.20   0.000    -.0000148   -7.70e-06
  Profitability |     .14997   .3930542     0.38   0.703    -.6204021    .9203421
          _cons |  -3.788979   .7845155    -4.83   0.000    -5.326601   -2.251357
----------------+----------------------------------------------------------------
       /lnsig2u |  -1.676469   .3295443                     -2.322364   -1.030574
----------------+----------------------------------------------------------------
        sigma_u |   .4324734   .0712596                      .3131159    .5973292
            rho |   .1575636   .0437428                      .0892877    .2629729
---------------------------------------------------------------------------------

. xtprobit AcquirerStatus LeverageDeficit Size Markettobook Profitability, vce(bs)

Random-effects probit regression                Number of obs     =      2,547
Group variable: CompanyNo                       Number of groups  =        248

Random effects u_i ~ Gaussian                   Obs per group:
                                                              min =          1
                                                              avg =       10.3
                                                              max =         23

Integration method: mvaghermite                 Integration pts.  =         12

                                                Wald chi2(4)      =      22.16
Log likelihood  = -668.40718                    Prob > chi2       =     0.0002

                                (Replications based on 248 clusters in CompanyNo)
---------------------------------------------------------------------------------
                |   Observed   Bootstrap                         Normal-based
 AcquirerStatus |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
----------------+----------------------------------------------------------------
LeverageDeficit |  -.4867099   .2300963    -2.12   0.034    -.9376904   -.0357294
           Size |   .2556389   .0833188     3.07   0.002     .0923371    .4189407
   Markettobook |  -.0000113    .086467    -0.00   1.000    -.1694835     .169461
  Profitability |     .14997   .3159822     0.47   0.635    -.4693438    .7692838
          _cons |  -3.788979   .7239035    -5.23   0.000    -5.207804   -2.370154
----------------+----------------------------------------------------------------
       /lnsig2u |  -1.676469   .3515036                     -2.365403   -.9875343
----------------+----------------------------------------------------------------
        sigma_u |   .4324734    .076008                      .3064497    .6103229
            rho |   .1575636   .0466577                      .0858492    .2713994
---------------------------------------------------------------------------------
LR test of rho=0: chibar2(01) = 22.76                  Prob >= chibar2 = 0.000

As you can see, the p-value of the Markettobook variable jumps to 1.000 from 0.000.

Thanks for the comments in advance!

[Joro Kolev]

This is not a correct bootstrap scheme for a generated regressor.

For a generated regressor, you need to bootstrap both stages, the first stage in which you generate the regressor, and the second stage where you estimate your regression.

[Hakan Gunduz]

Thanks for the response Joro. What do you think about this, why do you think I am getting this error?:

 program define prg1
  1. xtset CompanyNo Year
  2. xtreg MarketLeveraget Tangibilityt1 Profitabilityt1 MarkettoBookRatiot1 OperatingLeveraget1 AssetRiskt1
>  DividendPayingStatust1 Sizelogt1Adjusted i.Year, fe
  3. capture drop new_var
  4. predict new_var, resid
  5. gen leverage_deficit = MarketLeveraget - new_var
  6. xtprobit AcquirerStatus leverage_deficit Size Markettobook Profitability
  7. exit
  8. end

. bootstrap, reps(50): prg1
(running prg1 on estimation sample)

Bootstrap replications (50)
----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx    50
insufficient observations to compute bootstrap standard errors
no results will be saved
r(2000);

Thanks again!

[Joro Kolev]

This sounds more like it, you re wrapping both stages in the bootstrap loop, so this is right.

I think the error here comes from line 5., gen leverage_deficit. On first round this would work, on second round you already have it generated, so Stata encounters an error.

Try replacing line 3., by

capture drop new_var leverage_deficit

and see whether it goes through.

[Hakan Gunduz]

Dear Joro, I did what you said and also ran the -xtset- command before starting the -prg1- and it worked.

However, I am still getting the p-value of one of my variables exact 1.000. In this circumstance, I'll have to acknowledge that the variable is highly insignificant, right?

[Joro Kolev]

I think we are overlooking something, Hakan... A p-value of exactly 1 does not sound right.

Why dont you check -dataex- and post a subsample of your data, so that we and other people can investigate the issue. Maybe somebody will figure out what is wrong.

[Hakan Gunduz]

Joro, I've just summarised the problematic variable and realised that I made a typo in one observation. It's all sorted out now. Thank you very much for your help.

[Joro Kolev]

You are welcome, Hakan ! I am glad that it all got resolved.