"collect stars" does not use estimated p-value

[Richard Thomas Boylan]

When using rdrobust, I cannot seem to get the "stars" corresponding to the right p-value. Here is an example.


clear all
set obs 1000
gen low = _n < 500
set seed 1004
gen x = runiform(0,1)
gen w = runiform(0,1)
gen z = runiform(0,1)

gen y1 = 0.1*(x > 0.5) + w
gen y2 = 0.05*(x > 0.5) + w



local model 1
foreach outcome in y1 y2 {
foreach low in 0 1 {
quietly rdrobust `outcome' x if low==`low', c(0.5)
collect get e(), tag(outcome[`outcome'] wt[`low'] model[`model'])
collect get controls = "N", tag(outcome[`outcome'] wt[`low'] model[`model'])
local model = `model' + 1
quietly rdrobust `outcome' x if low==`low', c(0.5) covs(z)
collect get e(), tag(outcome[`outcome'] wt[`low'] model[`model'])
collect get controls = "Y", tag(outcome[`outcome'] wt[`low'] model[`model'])
local model = `model' + 1
}
}
collect layout (colname[RD_Estimate]#result[_r_b _r_se] result[controls N]) (outcome#wt#model)


collect stars _r_p .001 "***" .01 "**" .05 "*" .1 "+", attach(_r_b)

collect note "*** p<.001; ** p<.01; * p<.05; + p<.1."

collect preview

Which produces

----------------------------------------------------------------------------------------
| y1 y1 y1 y1 y2 y2 y2 y2
| 0 0 1 1 0 0 1 1
| 1 2 3 4 5 6 7 8
--------------+-------------------------------------------------------------------------
RD_Estimate |
Coefficient | .1432747 .1393473 .1531677* .151766* .0932747 .0893473 .1031677 .101766
Std. error | .0980967 .0967733 .0754744 .0757233 .0980967 .0967733 .0754744 .0757233
controls | N Y N Y N Y N Y
N | 501 501 499 499 501 501 499 499

So, the third coefficient (0.1531677) is significant at the 5% significance level.

However, this coefficient should only be significant at the 10% confidence level as can be checked by estimating

rdrobust y1 x if low==1, c(0.5)

[Jeff Pitblado (StataCorp)]

Show the p-values in your layout to see that the ones you specify in the call to collect stars are not the same as the ones in the output.

While not mentioned in the help file, the scalar posted to e() that appears to contain reported p-value is e(pv_rb). This p-value appears to correspond with the documented scalar results e(tau_cl) and e(se_tau_cl). The matrices e(b) and e(V) are not mention in the help file, so using _r_b, _r_se, _r_p might not be what you really want.

In the following, I highlight my modifications to your code in blue.

clear all

set obs 1000
gen low = _n < 500
set seed 1004
gen x = runiform(0,1)
gen w = runiform(0,1)
gen z = runiform(0,1)

gen y1 = 0.1*(x > 0.5) + w
gen y2 = 0.05*(x > 0.5) + w



local model 1
foreach outcome in y1 y2 {
  foreach low in 0 1 {
    rdrobust `outcome' x if low==`low', c(0.5)
    collect get e(), tag(outcome[`outcome'] wt[`low'] model[`model'])
    collect get controls = "N", tag(outcome[`outcome'] wt[`low'] model[`model'])
    local model = `model' + 1
    rdrobust `outcome' x if low==`low', c(0.5) covs(z)
    collect get e(), tag(outcome[`outcome'] wt[`low'] model[`model'])
    collect get controls = "Y", tag(outcome[`outcome'] wt[`low'] model[`model'])
    local model = `model' + 1
  }
}
collect addtags colname[RD_Estimate], fortags(result[tau_cl se_tau_cl pv_rb])
collect label levels result tau_cl "Coefficient"
collect label levels result se_tau_cl "Std. error"
collect layout (colname[RD_Estimate]#result[tau_cl se_tau_cl pv_rb] result[controls N]) (outcome#wt#model)


collect stars pv_rb .001 "***" .01 "**" .05 "*" .1 "+", attach(tau_cl)

collect note "*** p<.001; ** p<.01; * p<.05; + p<.1."

collect preview

Here is the resulting table.

----------------------------------------------------------------------------------------
              |       y1       y1        y1       y1       y2       y2       y2       y2
              |        0        0         1        1        0        0        1        1
              |        1        2         3        4        5        6        7        8
--------------+-------------------------------------------------------------------------
RD_Estimate   |
  Coefficient | .1432747 .1393473 .1531677+ .151766+ .0932747 .0893473 .1031677  .101766
  Std. error  | .0980967 .0967733  .0754744 .0757233 .0980967 .0967733 .0754744 .0757233
  pv_rb       |  .225399  .259521  .0553153 .0576439 .4296819  .486823 .1754033 .1806261
controls      |        N        Y         N        Y        N        Y        N        Y
N             |      501      501       499      499      501      501      499      499
----------------------------------------------------------------------------------------
*** p<.001; ** p<.01; * p<.05; + p<.1.

Remove pv_rb from the layout once you are satisfied that the collect stars labels are correct.

[Richard Thomas Boylan]

Thank you so much Jeff!