The same data while different results between Stata 18 and Stata 17, which one is correct?

[Fred Lee]

* Example generated by -dataex-. For more info, type help dataex
clear
input float(dv iv) byte(cv1 cv2 cv3 cv4 industry) float(serial id)
 4.833333       .125 0 49 2 1 11 1  1
4.6666665 -.07500005 0 49 2 1 11 2  1
        5 -.07500005 0 49 2 1 11 3  1
        4 -.07500005 0 49 2 1 11 4  1
4.3333335 -.07500005 0 49 2 1 11 5  1
        4 -.07500005 0 49 2 1 11 6  1
        4  .32500005 0 49 2 1 11 7  1
        4 -.07500005 0 49 2 1 11 8  1
        4 .025000095 1 59 2 3  9 1  5
 3.833333 -.17499995 1 59 2 3  9 2  5
        4 -.17499995 1 59 2 3  9 3  5
      3.5 .025000095 1 59 2 3  9 4  5
 3.666667   .4250001 1 59 2 3  9 5  5
4.1666665  .22500002 1 59 2 3  9 6  5
        4 -.17499995 1 59 2 3  9 7  5
        4 -.17499995 1 59 2 3  9 8  5
        4         .7 0 40 5 2  4 1  6
 4.833333   .3000002 0 40 5 2  4 2  6
4.1666665  .10000014 0 40 5 2  4 3  6
4.1666665 -.29999995 0 40 5 2  4 4  6
        4 -.29999995 0 40 5 2  4 5  6
        4 -.29999995 0 40 5 2  4 6  6
4.1666665  .10000014 0 40 5 2  4 7  6
4.3333335 -.29999995 0 40 5 2  4 8  6
 3.833333  -.0999999 0 41 4 2  9 1  7
4.6666665  -.0999999 0 41 4 2  9 2  7
        4  -.0999999 0 41 4 2  9 3  7
        4  -.0999999 0 41 4 2  9 4  7
        4   .3000002 0 41 4 2  9 5  7
        4  .10000014 0 41 4 2  9 6  7
        4  .10000014 0 41 4 2  9 7  7
        4  -.0999999 0 41 4 2  9 8  7
        .          . 1 36 4 0 11 1  8
        .          . 1 36 4 0 11 2  8
 4.833333   .0333333 1 36 4 0 11 3  8
        5  -.3666668 1 36 4 0 11 4  8
 4.833333   .0333333 1 36 4 0 11 5  8
        5 -.16666675 1 36 4 0 11 6  8
 4.833333   .4333332 1 36 4 0 11 7  8
        5   .0333333 1 36 4 0 11 8  8
        4  .20000005 1 35 4 0 11 1 10
        4  .20000005 1 35 4 0 11 2 10
        4  .20000005 1 35 4 0 11 3 10
        4          0 1 35 4 0 11 4 10
        4  .20000005 1 35 4 0 11 5 10
        4        -.8 1 35 4 0 11 6 10
        4          0 1 35 4 0 11 7 10
        4          0 1 35 4 0 11 8 10
 3.333333         .8 0 46 4 0  9 1 11
        4 -.20000005 0 46 4 0  9 2 11
        4 -.20000005 0 46 4 0  9 3 11
        4 -.20000005 0 46 4 0  9 4 11
        4          0 0 46 4 0  9 5 11
        4 -.20000005 0 46 4 0  9 6 11
        4          0 0 46 4 0  9 7 11
        4          0 0 46 4 0  9 8 11
        5       -.75 0 30 2 1  4 1 12
        5 -.54999995 0 30 2 1  4 2 12
        5        .25 0 30 2 1  4 3 12
        5       -.75 0 30 2 1  4 4 12
        5  -.3499999 0 30 2 1  4 5 12
        5   .6500001 0 30 2 1  4 6 12
        5        .25 0 30 2 1  4 7 12
        5       1.25 0 30 2 1  4 8 12
 3.833333          0 0 50 3 4  3 1 13
 3.833333          0 0 50 3 4  3 2 13
 3.666667          0 0 50 3 4  3 3 13
 3.833333          0 0 50 3 4  3 4 13
        4          0 0 50 3 4  3 5 13
        .          . 0 50 3 4  3 6 13
        .          . 0 50 3 4  3 7 13
        4          0 0 50 3 4  3 8 13
        4   .2285714 1 36 4 1  5 1 14
4.3333335 -.17142844 1 36 4 1  5 2 14
        .          . 1 36 4 1  5 3 14
        5   .4285715 1 36 4 1  5 4 14
 3.833333 .028571606 1 36 4 1  5 5 14
 4.833333 -.17142844 1 36 4 1  5 6 14
        5 -.17142844 1 36 4 1  5 7 14
4.1666665 -.17142844 1 36 4 1  5 8 14
end

xtset id serial
vcemway mixed dv cv1 cv2 cv3 cv4 cl.iv || id:  cl.iv ,cluster(id ind) cov(exc)

The results for Stata 18.0:

The results for Stata 17.0:


So which one is correct? In my specific example, the IV can be significnat using 17 while not significant using 18

[Nick Cox]

You need to explain where vcemway comes from and to confirm you're using the same version in both cases.

[Fred Lee]

Yes, I use the same vcemway, using "ssc install vcemway,replace"

[Carlo Lazzaro]

Fred:
is the way you clustered the standard errors the same in both specifications?

[Fred Lee]

Fred:
is the way you clustered the standard errors the same in both specifications?

Yes, I run the same code above in Stata 18 and 17, however, found different results

[daniel klein]

Make sure your Stata 17 is up to date, i.e.

Revision 21 May 2024

I cannot replicate the problem with an up-to-data Stata 17. I get the same results that you report (unfortunately in a screenshot; code delimiters would have been the better choice) for Stata 18 with a fully updated Stata 17, regardless of the version of vcemway. (SSC, SJ-19-4, SJ-20-2).

[Fred Lee]

Make sure your Stata 17 is up to date, i.e.

Revision 21 May 2024

I cannot replicate the problem with an up-to-data Stata 17. I get the same results that you report (unfortunately in a screenshot; code delimiters would have been the better choice) for Stata 18 with a fully updated Stata 17, regardless of the version of vcemway. (SSC, SJ-19-4, SJ-20-2).

The Stata 17 I used is the 17.0 (the first release version), not the latest. I will check then.

[daniel klein]

So here is the relevant output from the data and code in #1 in fully updated Stata 17 BE on Windows 10.

. vcemway mixed dv cv1 cv2 cv3 cv4 cl.iv || id:  cl.iv ,cluster(id ind) cov(exc)

Mixed-effects regression                        Number of obs     =         63
Group variable: id                              Number of groups  =         10
                                                Obs per group:
                                                              min =          4
                                                              avg =        6.3
                                                              max =          7
                                                Wald chi2(5)      =    3276.16
Log pseudolikelihood = -8.2887124               Prob > chi2       =     0.0000

                               (Std. err. adjusted for clustering on id industry)
---------------------------------------------------------------------------------
                |               Robust
             dv | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
----------------+----------------------------------------------------------------
            cv1 |   .0353136   .1678608     0.21   0.833    -.2936875    .3643146
            cv2 |  -.0401416    .003488   -11.51   0.000     -.046978   -.0333052
            cv3 |   -.189919   .0205624    -9.24   0.000    -.2302206   -.1496174
            cv4 |  -.0402977   .0381902    -1.06   0.291    -.1151492    .0345538
                |
             iv |
            L1. |   .0722754   .1336986     0.54   0.589    -.1897691    .3343199
                |
          _cons |   6.648578    .141548    46.97   0.000     6.371149    6.926008
---------------------------------------------------------------------------------

------------------------------------------------------------------------------
                             |               Robust           
  Random-effects parameters  |   Estimate   std. err.     [95% conf. interval]
-----------------------------+------------------------------------------------
id: Exchangeable             |
             var(L.iv _cons) |   .0440987   .0355243      .0090935    .2138562
             cov(L.iv,_cons) |   .0299759   .0178623     -.0050336    .0649854
-----------------------------+------------------------------------------------
               var(Residual) |    .055367   .0179254      .0293541     .104432
------------------------------------------------------------------------------
Notes:
    Std. Err. adjusted for 2-way clustering on id industry
      Number of clusters in id           = 10
      Number of clusters in industry     = 5

    Stata's default small-cluster correction factors have been applied. See vcemway for detail.

    chi2() and Prob > chi2 above only account for one-way clustering on id.
      Use test to compute chi2() and Prob > chi2 that account for 2-way clustering.

    The initial variance-covariance matrix, e(V_raw), was not positive semi-definite.
      The final matrix, e(V), was computed by replacing negative eigenvalues with 0s.

. 
. about

Stata/BE 17.0 for Windows (64-bit x86-64)
Revision 21 May 2024
Copyright 1985-2021 StataCorp LLC

Total physical memory:       16.00 GB
Available physical memory:    8.21 GB

Stata license: Single-user  perpetual
Serial number: (output omitted)
  Licensed to: (output omitted)

. type c:\ado\plus\v\vcemway.ado
// Ariel Gu ([email protected]) and Hong Il Yoo ([email protected]): 12 June 2022.
// M-Way Clustered Standard Errors
// v1.0.4 (12 June 2022) 
(output omitted)

[Fred Lee]

Correct, this will not happen on the latest version. It was happed on Revision 2021

[Richard Williams]

When encountering strange errors, the first thing I often do is run update and adoupdate. If I'm lucky the problem has already been fixed and if nothing else I have the most up to date versions of everything.

I've occasionally had students come in running Stata 13 that they apparently got from their grandmothers or maybe an older student. I like to run their code on my machine because the problem may have been fixed 10 years ago.

[Fred Lee]

I want to rediscuss the issue as I found the output e(V) of vcemway cannot be correctly displayed. Originally found here:https://github.com/benjann/estout/issues/68

* Example generated by -dataex-. For more info, type help dataex
clear
input float(DV gpDV IV id serial) byte industry
 4.944445   .54861116     .15625  1 1 11
4.2222223  -.17361116    -.09375  1 2 11
 4.944445   .54861116    -.09375  1 3 11
4.3333335      -.0625    -.09375  1 4 11
4.7222223    .3263888    -.09375  1 5 11
        4   -.3958335    -.09375  1 6 11
        4   -.3958335     .40625  1 7 11
        4   -.3958335    -.09375  1 8 11
 4.111111      -.4375    1.46875  2 1  6
4.1666665   -.3819447     .71875  2 2  6
4.5555553   .00694418    -.53125  2 3  6
 4.777778    .2291665    -.03125  2 4  6
        4  -.54861116   -1.03125  2 5  6
 4.777778    .2291665     .21875  2 6  6
        5    .4513888    -.28125  2 7  6
        5    .4513888    -.53125  2 8  6
        4 -.013888836    -.09375  3 1  6
        4 -.013888836     .15625  3 2  6
        4 -.013888836    -.09375  3 3  6
 4.111111   .09722233    -.09375  3 4  6
        4 -.013888836    -.09375  3 5  6
        4 -.013888836     .15625  3 6  6
        4 -.013888836    -.09375  3 7  6
        4 -.013888836     .15625  3 8  6
 4.111111 -.007936478 .035714388  4 1  5
4.1666665   .04761887  -.9642856  4 2  5
4.0555553   -.0634923   .2857144  4 3  5
        4  -.11904764   .2857144  4 4  5
4.2222223   .10317469   .7857144  4 5  5
4.4444447     .325397 .035714388  4 6  5
        .           .          .  4 7  5
 3.833333   -.2857144  -.4642856  4 8  5
4.5555553     .583333     .03125  5 1  9
        4   .02777767    -.21875  5 2  9
 4.388889    .4166665    -.21875  5 3  9
 3.722222        -.25     .03125  5 4  9
 3.611111   -.3611112     .53125  5 5  9
 3.777778  -.19444466     .28125  5 6  9
 3.722222        -.25    -.21875  5 7  9
        4   .02777767    -.21875  5 8  9
 4.111111   -.0833335     .65625  6 1  4
4.6666665   .47222185     .40625  6 2  4
4.2222223   .02777767     .15625  6 3  4
 4.111111   -.0833335    -.34375  6 4  4
 4.111111   -.0833335    -.34375  6 5  4
4.0555553  -.13888931    -.34375  6 6  4
 4.111111   -.0833335     .15625  6 7  4
4.1666665  -.02777815    -.34375  6 8  4
4.1666665       .0625      -.125  7 1  9
4.4444447   .34027815      -.125  7 2  9
4.2222223   .11805582      -.125  7 3  9
        4   -.1041665      -.125  7 4  9
        4   -.1041665       .375  7 5  9
        4   -.1041665       .125  7 6  9
        4   -.1041665       .125  7 7  9
        4   -.1041665      -.125  7 8  9
        .           .          .  8 1 11
        .           .          .  8 2 11
4.6666665  -.11111116       .125  8 3 11
4.6666665  -.11111116      -.375  8 4 11
4.7222223  -.05555534       .125  8 5 11
        5   .22222233      -.375  8 6 11
 4.944445     .166667       .375  8 7 11
4.6666665  -.11111116       .125  8 8 11
 3.888889    .3680556          0  9 1 11
        4   .47916675        .25  9 2 11
 1.388889  -2.1319444       -.25  9 3 11
 3.666667   .14583349        .25  9 4 11
 3.666667   .14583349          0  9 5 11
 3.888889    .3680556       -.25  9 6 11
 3.777778   .25694442          0  9 7 11
 3.888889    .3680556          0  9 8 11
        4           0     .21875 10 1 11
        4           0     .21875 10 2 11
        4           0     .21875 10 3 11
        4           0    -.03125 10 4 11
        4           0     .21875 10 5 11
        4           0    -.78125 10 6 11
        4           0    -.03125 10 7 11
        4           0    -.03125 10 8 11
end

eststo clear
xtset id serial
eststo: vcemway mixed DV l.gpDV cl.IV i.ind || id: l.gpDV  cl.IV ,cluster(id ind) cov(exc)
eststo:  mixed DV l.gpDV cl.IV i.ind || id: l.gpDV  cl.IV ,cluster(id ) cov(exc)
esttab,p

 eststo: vcemway mixed DV l.gpDV cl.IV i.ind || id: l.gpDV  cl.IV ,cluster(id ind) cov(exc)

Mixed-effects regression                                Number of obs    =  66
Group variable: id                                      Number of groups =  10
                                                        Obs per group:
                                                                     min =   5
                                                                     avg = 6.6
                                                                     max =   7
                                                        Wald chi2(5)     =   .
Log pseudolikelihood = -37.400465                       Prob > chi2      =   .

                                      (Std. err. adjusted for multiway clustering)
----------------------------------------------------------------------------------
                 |               Robust
              DV | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
-----------------+----------------------------------------------------------------
            gpDV |
             L1. |   .2133617   .0857786     2.49   0.013     .0452387    .3814846
                 |
              IV |
             L1. |  -.2010575   .1409616    -1.43   0.154    -.4773371     .075222
                 |
        industry |
              5  |  -.0501896   .0153042    -3.28   0.001    -.0801853   -.0201938
              6  |   .3333027   .2263672     1.47   0.141    -.1103689    .7769743
              9  |  -.0574494   .0601336    -0.96   0.339    -.1753091    .0604104
             11  |   .0747133   .2123752     0.35   0.725    -.3415345    .4909611
                 |
           _cons |   4.072868   .0407487    99.95   0.000     3.993002    4.152734
----------------------------------------------------------------------------------

------------------------------------------------------------------------------
                             |               Robust           
  Random-effects parameters  |   Estimate   std. err.     [95% conf. interval]
-----------------------------+------------------------------------------------
id: Exchangeable             |
      var(L.gpDV L.IV _cons) |   .0765541    .030033       .035484    .1651598
      cov(L.gpDV L.IV _cons) |   .0650035   .0278915      .0103372    .1196698
-----------------------------+------------------------------------------------
               var(Residual) |    .136608   .0671699      .0521125    .3581052
------------------------------------------------------------------------------
Notes:
    Std. Err. adjusted for 2-way clustering on id industry
      Number of clusters in id           = 10
      Number of clusters in industry     = 5

    Stata's default small-cluster correction factors have been applied. See vcemway for detail.

    The initial variance-covariance matrix, e(V_raw), was not positive semi-definite.
      The final matrix, e(V), was computed by replacing negative eigenvalues with 0s.
(est1 stored)

. eststo:  mixed DV l.gpDV cl.IV i.ind || id: l.gpDV  cl.IV ,cluster(id ) cov(exc)

Performing EM optimization ...

Performing gradient-based optimization: 
Iteration 0:  Log pseudolikelihood =  -37.48527  
Iteration 1:  Log pseudolikelihood =  -37.40084  
Iteration 2:  Log pseudolikelihood = -37.400466  
Iteration 3:  Log pseudolikelihood = -37.400465  

Computing standard errors ...

Mixed-effects regression                                Number of obs    =  66
Group variable: id                                      Number of groups =  10
                                                        Obs per group:
                                                                     min =   5
                                                                     avg = 6.6
                                                                     max =   7
                                                        Wald chi2(5)     =   .
Log pseudolikelihood = -37.400465                       Prob > chi2      =   .

                                        (Std. err. adjusted for 10 clusters in id)
----------------------------------------------------------------------------------
                 |               Robust
              DV | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
-----------------+----------------------------------------------------------------
            gpDV |
             L1. |   .2133617   .0857786     2.49   0.013     .0452387    .3814846
                 |
              IV |
             L1. |  -.2010575   .1409615    -1.43   0.154    -.4773371     .075222
                 |
        industry |
              5  |  -.0501896   .0153042    -3.28   0.001    -.0801854   -.0201938
              6  |   .3333027   .2263672     1.47   0.141    -.1103689    .7769743
              9  |  -.0574494   .0601336    -0.96   0.339    -.1753091    .0604104
             11  |   .0747133   .2123752     0.35   0.725    -.3415345    .4909611
                 |
           _cons |   4.072868   .0407487    99.95   0.000     3.993002    4.152734
----------------------------------------------------------------------------------

------------------------------------------------------------------------------
                             |               Robust           
  Random-effects parameters  |   Estimate   std. err.     [95% conf. interval]
-----------------------------+------------------------------------------------
id: Exchangeable             |
      var(L.gpDV L.IV _cons) |   .0765541   .0391404       .028104    .2085299
      cov(L.gpDV L.IV _cons) |   .0650035   .0479264     -.0289305    .1589376
-----------------------------+------------------------------------------------
               var(Residual) |   .1366081   .0828409      .0416195    .4483899
------------------------------------------------------------------------------
(est2 stored)

The two models should have different p values, and actually different in e(V). However, the displyed results are wrongly the same.

[Fred Lee]

So maybe Stata 17 is correct while Stata 18 is wrong.

[daniel klein]

The issue discussed in this thread seems to arise because the community-contributed command vcemway relies on mixed, typed without arguments, replaying e(V). mixed used to do that but no longer does.

More specifically, mixed without arguments used to replay results from e(V); it now sometimes re-displays results from hidden e() matrices. More specifically, mixed calls _xtmixed_display in which the internal subroutine DiEstTable calls _coef_table. At least until Stata 16.1 _xtmixed_display (*! version 1.4.0 10jan2020) did not explicitly specify bmat() and vmat(); it now typically does and obtains previous results from (hidden) matrices other than e(V).

You should notify the authors of vcemway.

Whether the described changed internal behavior of mixed should have been put under version control is a question for StataCorp.