error when combining bootstrap and rdrobust

[Sebastien Willis]

I am using the -rdrobust- command from SSC in Stata 14. Because I use the same data to identify the cutoff as I do to estimate the treatment effect at the cutoff, I believe I need to calculate bootstrap standard errors. So I use the following code:

bootstrap, seed(1234) strata(treat): rdrobust D_deu d1

After appearing to complete the 50 replications, this fails with error code r(100), and the message "something required". I think this is a problem with the bootstrap command, since

rdrobust D_deu d1

works just fine.

The following recreates a subset of my data which can be used to reproduce the error message. Apologies for the length, I couldn't get the bootstrap replications to work on a smaller dataset, given the data requirements for -rdrobust-.

clear
input float(D_deu d1 treat)
    -31.25  -6.108034 0
  492.3077  -76.74717 0
        75 -14.789054 0
         0 -32.758766 0
  57.14286  -98.55405 0
        25  -96.47466 0
         0 -16.376535 0
         0   -77.2933 0
         0  -76.74717 0
         0   -65.1313 0
 133.33333  -46.05086 0
 -113.9535 -28.859293 0
         0  -.7615094 0
  45.12195 -25.486624 0
-31.566265   -82.0192 0
      4200 -23.205584 0
  92.30769  -48.83578 0
         0  -.7615094 0
 -13.72549  -62.69519 0
  5.555555 -16.277328 0
 35.135136  -26.15659 0
 33.333332  -68.41383 0
         0  -24.36622 0
  21.21212  -76.74717 0
         0  -53.70616 0
 -7.692307 -64.816376 0
 -5.220883   -55.5936 0
   -21.875 -33.200542 0
         0 -10.314776 0
-27.173914 -12.376018 0
-17.948717 -10.164627 0
         0  -.7615094 0
  63.04348 -28.859293 0
 14.117647  -79.10925 0
-8.1081085  -93.93493 0
         0 -14.789054 0
-4.1666665  -86.90412 0
       575  -98.55405 0
 33.333332  -47.28803 0
  38.88889  -83.05103 0
-24.489796 -21.670074 0
         0  -.7615094 0
 -53.33333 -28.859293 0
  4.918033 -18.833086 0
 -94.44444 -28.859293 0
 11.538462 -1.9219055 0
  23.52941   -44.9101 0
  30.68182  -20.94117 0
 -3.738318  -25.60391 0
         0  -53.70616 0
         0  -20.23643 0
         0  -88.61089 0
 23.255814  -43.00917 0
         0 -15.137874 0
-17.647058  -60.31364 0
         0  -53.70616 0
         0  -.7615094 0
 -58.62069 -28.859293 0
  38.36478   -22.0476 0
  62.22222  -74.02868 0
   -84.375  -21.87715 0
         0  -.7615094 0
 -59.03614  -60.22797 0
-11.538462  -97.54356 0
         0 -15.188906 0
         0  -84.16108 0
        15  -68.31211 0
 70.588234  -78.23978 0
      1000  -4.159112 0
 -7.843137  -.3043476 0
 -24.05063  -82.30303 0
        30 -3.9696884 0
 -24.13793 -17.075993 0
    -43.75  -60.31364 0
         0  -59.96104 0
-13.286714  -76.54393 0
         0  -87.29086 0
         0  -88.61089 0
  64.28571  -5.318596 0
  5.574913  -62.94139 0
         0  -.7615094 0
  57.89474 -12.547684 0
       300  -88.61089 0
         0  -.7615094 0
  -83.1579  -63.18082 0
 4.5454545  -75.14979 0
 12.962963   -15.4574 0
         0 -2.1633766 0
 194.44444  -48.11048 0
  5.401662  -33.64421 0
 157.14285   48.88162 1
 16.666666  26.696264 1
-13.541667  .12398243 1
   11.3834   29.05054 1
         0    62.5428 1
-25.714285  16.978521 1
 1.1824324    82.6171 1
     -37.5   4.567808 1
         0   7.152679 1
  5.660378   37.45851 1
  12.658228 15.665567 1
         0  40.03896 1
 -5.031446  21.98519 1
 -9.090909   9.98574 1
 3.3101046 18.210032 1
 133.84616 27.646225 1
 11.197183  40.40638 1
 -7.594937  1.809189 1
  19.80676  50.56956 1
         0  46.29384 1
       100  99.23849 1
  27.77778 14.802475 1
         0  28.92784 1
  1.818182  20.02496 1
-19.200523 19.119425 1
 15.384615  5.304115 1
 -10.25641  44.13516 1
         0  25.42702 1
  6.382979  53.52538 1
      17.5  32.29603 1
         0  53.94914 1
   47.4359  6.937422 1
     137.5  21.56346 1
 3.5353534  45.50165 1
         0  86.32065 1
  27.27273  46.34938 1
  1.388889  63.05215 1
 34.328358  7.412704 1
  35.48387 17.621897 1
-19.444445 20.546654 1
        40 15.621315 1
 152.63158   27.1009 1
 -7.934509   39.7476 1
  17.34104  73.81273 1
         0  65.90516 1
 11.320755  8.936054 1
-14.285714   28.8086 1
       100  99.23849 1
         0 13.081375 1
 1.7857143 1.8246727 1
       -16  3.022457 1
-1.6949153 32.890602 1
  3.333333  .8119431 1
 19.220057 24.009365 1
 33.333332  25.11009 1
         0 35.210945 1
        50  68.14443 1
 -6.862745 14.146095 1
-11.764706  41.72894 1
         0  14.37284 1
-24.489796  21.14464 1
 15.748032   15.4096 1
 -6.666667  7.694391 1
        20  44.97065 1
 -3.333333   59.4189 1
-15.789474  42.05805 1
         0  65.90516 1
 -8.333333 2.4194946 1
  45.45454 14.247694 1
  333.3333  4.553528 1
  7.692307  .4761276 1
  4.255319  48.41209 1
 -2.097902  43.06796 1
    21.875   9.50779 1
      27.5  75.52855 1
  -1.66205  25.68177 1
 -8.064516  5.749283 1
-1.6393442 10.196568 1
  59.57447  55.79282 1
-2.0408163  65.21242 1
  64.51613  39.36708 1
   8.99654  6.521555 1
        24 18.329926 1
       480  50.81244 1
 2.3809524 37.213326 1
  41.30435  62.56785 1
        35  25.61133 1
  162.6087 1.1947174 1
         0  41.12532 1
         0  57.78244 1
end

[Rich Goldstein]

I am not familiar with -rdrobust-; however, I downloaded a copy and tried with no bootstrapping on your data sample above:

. rdrobust D_deu d1

Sharp RD estimates using local polynomial regression.

      Cutoff c = 0 | Left of c  Right of c            Number of obs =        180
-------------------+----------------------            BW type       =      mserd
     Number of obs |        90          90            Kernel        = Triangular
Eff. Number of obs |        46          59            VCE method    =         NN
    Order est. (p) |         1           1
    Order bias (q) |         2           2
       BW est. (h) |    40.417      40.417
       BW bias (b) |    40.919      40.919
         rho (h/b) |     0.988       0.988

Outcome: D_deu. Running variable: d1.
--------------------------------------------------------------------------------
            Method |   Coef.    Std. Err.    z     P>|z|    [95% Conf. Interval]
-------------------+------------------------------------------------------------
      Conventional | -7.7647     61.303   -0.1267  0.899   -127.917      112.388
            Robust |     -          -     0.2193   0.826   -101.663       127.28
--------------------------------------------------------------------------------

So, what are you trying to bootstrap since there is no coefficient or SE?

note that you have not told bootstrap what to do so it will, I believe, default to a coefficient - but you have none for your "robust" row; I think that the command is confused and you need to supply some expression immediately after the "bootstrap"; see

help bootstrap

[Sebastien Willis]

The -rdrobust- command implements different estimators of the treatment effect in a regression discontinuity design, the full details and references are here.

Thanks for pointing out that there is no coefficient reported for the Robust method. The following code appears to successfully calculate the RDD estimator by three different methods

. rdrobust D_deu d1, all

Sharp RD estimates using local polynomial regression.

      Cutoff c = 0 | Left of c  Right of c            Number of obs =        180
-------------------+----------------------            BW type       =      mserd
     Number of obs |        90          90            Kernel        = Triangular
Eff. Number of obs |        46          59            VCE method    =         NN
    Order est. (p) |         1           1
    Order bias (q) |         2           2
       BW est. (h) |    40.417      40.417
       BW bias (b) |    40.919      40.919
         rho (h/b) |     0.988       0.988

Outcome: D_deu. Running variable: d1.
--------------------------------------------------------------------------------
            Method |   Coef.    Std. Err.    z     P>|z|    [95% Conf. Interval]
-------------------+------------------------------------------------------------
      Conventional | -7.7647     61.303   -0.1267  0.899   -127.917      112.388
    Bias-corrected |  12.809     61.303   0.2089   0.834   -107.344      132.961
            Robust |  12.809     58.405   0.2193   0.826   -101.663       127.28
--------------------------------------------------------------------------------

However the following command

bootstrap, seed(1234) strata(treat): rdrobust D_deu d1, all

Still return the error code r(100).

I have looked again at the documentation for -bootstrap-. The coefficient estimates are stored in the e(b) object, however I don't know how to access the entries in this object within the bootstrap command, as my following efforts show:

. bootstrap b = e(b) , seed(1234) strata(treat): rdrobust D_deu d1, all
(running rdrobust on estimation sample)
matrix operators that return matrices not allowed in this context
error in expression: e(b)
r(509);

. bootstrap scalar b = e(b)[1, 1] , seed(1234) strata(treat): rdrobust D_deu d1, all
(running rdrobust on estimation sample)
scalar not found
error in expression: scalar
r(111);

. bootstrap b = e(b)[1, 1] , seed(1234) strata(treat): rdrobust D_deu d1, all
(running rdrobust on estimation sample)
matrix operators that return matrices not allowed in this context
error in expression: e(b)

. bootstrap b = el(e(b), 1, 1) , seed(1234) strata(treat): rdrobust D_deu d1, all
(running rdrobust on estimation sample)
matrix operators that return matrices not allowed in this context
error in expression: el(e(b), 1, 1)
r(509);

[Rich Goldstein]

see

help exp_list

to see what is allowed there

[Sebastien Willis]

Thanks. I have looked over -help exp_list-. Since -rdrobust- does not create a _b variable, but only stores the estimated coefficients in e(b), I don't see how to fit the output of -rdrobust- into the syntax allowed by -exp_list-. I have tried various combinations, and typically I get an error r(509) -- does this mean that -exp- cannot return a matrix? Furthermore, does this mean that -bootstrap- can't be combined with -rdrobust-?

. bootstrap treat = (e(b)) , seed(1234) strata(treat): rdrobust D_deu d1, all
(running rdrobust on estimation sample)
matrix operators that return matrices not allowed in this context
error in expression: e(b)
r(509);

[Steve Samuels]

bootstrap (_b[Conventional]) (_b[Bias-corrected]) ( _b[Robust])  , seed(38755) strata(treat) reps(5): rdrobust D_deu d1, all
estat bootstrap

See the User's Guide Section 13.5 "Accessing coefficients and standard errors"

You'll need at least 1,000 replicates, I'd guess. See, e.g.
https://www.stata.com/support/faqs/s...es-guidelines/

[Sebastien Willis]

Thanks Steve, that syntax work for me, so I guess the question ends here.

Having said that, as a follow-up I don't understand why

. bootstrap (_b), seed(38755) strata(treat) reps(5): rdrobust D_deu d1, all

returns an error ("_b not found", r(111)), when the following works

bootstrap (_b[Conventional]) (_b[Bias-corrected]) ( _b[Robust])  , seed(38755) strata(treat) reps(5): rdrobust D_deu d1, all
estat bootstrap

Perhaps this is a questions best addressed to the authors of the -rdrobust- command. Thanks both of you for your help.

Sébastien

[Bram Hogendoorn]

I have the same problem. Bootstrapping my own program with all coefficients specified explicitly works, whereas leaving them out or indicating them with _b gives me an error.