psmatch2 unmatched t-stat ≠ naive ttest

[Joseph Chrestien]

Dear Statalisters,

I am playing around with psmatch2 (treatment effects propensity score matching) and I have got the output 1 below — so far, so good.

Now I did not quite understand what the "Unmatched t-stat" line could stand for, and I came across this excellent statalist post from Ariel Linden, saying

The first line indicates what the unmatched/unadjusted values look like
(ie., comparison of all the treated to all of the untreated).

I understand this as a naive t-test. As you can imagine, I instantly and eagerly checked it up, but I found the t-test gave quite a different t-stat (output 2). Uh oh.

The Question : Does anybody have a specific idea of what the first line t-stat (red font) in the output stands for ?

Thanks a ton, great day to all,
Joseph

Output 1

. psmatch2 treatvar covariate1, kernel out(outvar)

(Probit blabla omitted)

----------------------------------------------------------------------------------------
        Variable     Sample |    Treated     Controls   Difference         S.E.   T-stat
----------------------------+-----------------------------------------------------------
          outvar  Unmatched | 9.23717948   9.90951236  -.672332875   .365221621    -1.84   <-- ?
                        ATT | 9.23717948   9.81980187  -.582622388    .34449081    -1.69
----------------------------+-----------------------------------------------------------

Output 2

. ttest outvar, by(treatvar)

Two-sample t test with equal variances
------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
       0 |     827    9.859381    .1282484    3.688116     9.60765    10.11111
       1 |     127    9.360564    .3010767    3.392962    8.764742    9.956386
---------+--------------------------------------------------------------------
combined |     954    9.792977    .1182522    3.652443    9.560912    10.02504
---------+--------------------------------------------------------------------
    diff |             .498817    .3479066               -.1839354    1.181569
------------------------------------------------------------------------------
    diff = mean(0) - mean(1)                                      t =   1.4338   <-- != ± 1.84. Why ?
Ho: diff = 0                                     degrees of freedom =      952

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.9240         Pr(|T| > |t|) = 0.1520          Pr(T > t) = 0.0760

[Joseph Chrestien]

Sorry for spamming the list, I'm afraid I've mistaken, and I've found the answer. The first line "unmatched" t-stat is indeed the "naive t-test".

The two following procedures do indeed seem to yield the same t-stat :

. sysuse auto, clear
(1978 Automobile Data)

. gen treatvar=(rep78>2) // kind of stupid thing to do, just to test it out

. psmatch2 treatvar price, kernel out(mpg)       // kids, do not try this at home

(Probit blabla)
----------------------------------------------------------------------------------------
        Variable     Sample |    Treated     Controls   Difference         S.E.   T-stat
----------------------------+-----------------------------------------------------------
             mpg  Unmatched |  21.578125         19.5     2.078125   1.96570032     1.06
                        ATT |  21.578125   19.5565921    2.0215329   1.39610137     1.45
----------------------------+-----------------------------------------------------------
Note: S.E. does not take into account that the propensity score is estimated.

(blabla)


. ttest mpg, by(treatvar)

Two-sample t test with equal variances
------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
       0 |      10        19.5    1.166667    3.689324    16.86082    22.13918
       1 |      64    21.57813    .7525765    6.020612    20.07422    23.08203
---------+--------------------------------------------------------------------
combined |      74     21.2973    .6725511    5.785503     19.9569    22.63769
---------+--------------------------------------------------------------------
    diff |           -2.078125      1.9657               -5.996677    1.840427
------------------------------------------------------------------------------
    diff = mean(0) - mean(1)                                      t =  -1.0572  <-- OK, same t-stat.
Ho: diff = 0                                     degrees of freedom =       72

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.1470         Pr(|T| > |t|) = 0.2940          Pr(T > t) = 0.8530




.
. // same thing below
. psmatch2 treatvar price if price>5000, kernel out(mpg)

Probit blabla
----------------------------------------------------------------------------------------
        Variable     Sample |    Treated     Controls   Difference         S.E.   T-stat
----------------------------+-----------------------------------------------------------
             mpg  Unmatched | 19.2121212        17.25   1.96212121   2.90572694     0.68
                        ATT | 19.2121212   17.2615755   1.95054571    1.9650559     0.99
----------------------------+-----------------------------------------------------------
Note: S.E. does not take into account that the propensity score is estimated.

blabla


. ttest mpg if price>5000, by(treatvar)

Two-sample t test with equal variances
------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
       0 |       4       17.25    1.701715     3.40343    11.83438    22.66562
       1 |      33    19.21212    .9825731    5.644453    17.21069    21.21356
---------+--------------------------------------------------------------------
combined |      37          19    .8954339    5.446712    17.18398    20.81602
---------+--------------------------------------------------------------------
    diff |           -1.962121    2.905727               -7.861061    3.936818
------------------------------------------------------------------------------
    diff = mean(0) - mean(1)                                      t =  -0.6753  <-- OK, same t-stat.
Ho: diff = 0                                     degrees of freedom =       35

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.2520         Pr(|T| > |t|) = 0.5039          Pr(T > t) = 0.7480

[Anzony Quispe]

Sorry for interrupting your thread. I have a question How should I construct p-values from t-stat?. After reviewing scalars from the regression I found SE and t-value, but I would like to get p-value from the regression. I know that *p-value = (2 * ttail( df_r, t-stat ))*, but I would like to know how can i get df_r for the ATT.

  ----------------------------------------------------------------------------------------         Variable     Sample |    Treated     Controls   Difference         S.E.   T-stat ----------------------------+-----------------------------------------------------------              mpg  Unmatched | 19.2121212        17.25   1.96212121   2.90572694     0.68                         ATT | 19.2121212   17.2615755   1.95054571    1.9650559     0.99 ----------------------------+-----------------------------------------------------------

[Anzony Quispe]

In my previous question, I refer to the first table of @Joseph Chrestien.

[David Radwin]

Do you mean this?

. display  2 * ttail(r(df_t), r(t))
1.4960524

Your standard errors won't have the Abadie-Imbens adjustment for estimated propensity scores, though.

Abadie A., & Imbens, G. (2006). Large sample properties of matching estimators for average treatment effects. Econometrica 74(1), 235-267.