Opinion on the Brant test and the assumption of proportional odds.

[Ivan Privalko]

Hi all,

I wanted to ask about the assumption of proportional odds and the Brant test. I have two questions; First, is a model useless after the assumptions is violated? Specifically, if theoretically significant variables do not break the assumption, but general controls do break the assumptions, is the model invalid? Secondly, can limiting the number of categories in the dependent variable help the model? Since the dependent variable has seven categories, I wonder if limiting them down to a simple "disagree, neutral, agree" would limit the cross over between categories.

I have some output below, showing an ordinal logistic regression predicting "satisfaction with hours worked". The dependent variable has seven categories. Theoretically I'm interested most in the variable titled "M_event", which contains six types of job mobility.

 tab M_even

                         Mobility event |      Freq.     Percent        Cum.
----------------------------------------+-----------------------------------
                Same job, same employed |     86,014       79.04       79.04
            Changed employer- voluntary |      8,193        7.53       86.56
          Changed employer- involuntary |      2,231        2.05       88.61
         Changed employer- other reason |      3,562        3.27       91.89
  Changed job, kept employer- voluntary |      6,440        5.92       97.80
Changed job, kept employer- involuntary |        500        0.46       98.26
      Changed job, kept employer- other |      1,890        1.74      100.00
----------------------------------------+-----------------------------------
                                  Total |    108,830      100.00

The model basically predicts lots of theoretically relevant things, but the brant test suggests that it's broken the PO/PL assumption, including the all important M_event factor variable.

. ologit jbsat7 ljbsat7 jbsat ib0.skill i.sex ///
> ib1.perm  i.M_e  i.qfedhi age i.child ///
> ///
> wave , cluster(pid) nolog

Ordered logistic regression                       Number of obs   =      76133
                                                  Wald chi2(28)   =   15614.92
                                                  Prob > chi2     =     0.0000
Log pseudolikelihood =  -104261.9                 Pseudo R2       =     0.1577

                                (Std. Err. adjusted for 12317 clusters in pid)
------------------------------------------------------------------------------
             |               Robust
      jbsat7 |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     ljbsat7 |   .6102763   .0089629    68.09   0.000     .5927093    .6278434
       jbsat |   .7793823   .0107557    72.46   0.000     .7583015    .8004631
             |
       skill |
    skilled  |  -.2096002   .0185973   -11.27   0.000    -.2460503   -.1731502
             |
         sex |
     female  |   .1365385   .0200529     6.81   0.000     .0972354    .1758415
             |
   permanent |
  Temporary  |   .2296856    .049656     4.63   0.000     .1323616    .3270097
             |
     M_event |
Changed e..  |   .3293026   .0300587    10.96   0.000     .2703887    .3882165
Changed e..  |   .1077459   .0592723     1.82   0.069    -.0084258    .2239175
Changed e..  |   .1650086   .0461811     3.57   0.000     .0744953     .255522
Changed j..  |  -.0747845   .0285645    -2.62   0.009    -.1307698   -.0187991
Changed j..  |  -.0164938   .1036868    -0.16   0.874    -.2197162    .1867287
Changed j..  |    .095854    .052989     1.81   0.070    -.0080025    .1997105
             |
      qfedhi |
first deg..  |     .03263   .0653356     0.50   0.617    -.0954254    .1606854
teaching qf  |   .0117025   .0924879     0.13   0.899    -.1695705    .1929754
other hig..  |   .2257928   .0621483     3.63   0.000     .1039843    .3476012
 nursing qf  |   .2550195   .0915002     2.79   0.005     .0756824    .4343567
gce a lev..  |    .313028   .0643728     4.86   0.000     .1868595    .4391964
gce o lev..  |   .2851908   .0638083     4.47   0.000     .1601289    .4102528
commercia..  |   .3856074   .0973005     3.96   0.000     .1949019     .576313
cse grade..  |   .3057054   .0715325     4.27   0.000     .1655043    .4459064
apprentic~p  |   .3938594   .1300082     3.03   0.002      .139048    .6486708
   other qf  |   .1288678   .1398562     0.92   0.357    -.1452453    .4029808
      no qf  |   .3585312   .0674823     5.31   0.000     .2262682    .4907942
still at ..  |  -.0831313    .287263    -0.29   0.772    -.6461565    .4798938
             |
         age |   .0046013   .0008295     5.55   0.000     .0029755    .0062271
             |
       child |
          1  |  -.0291667   .0211463    -1.38   0.168    -.0706128    .0122794
          2  |   .0350747   .0229193     1.53   0.126    -.0098463    .0799958
          3  |  -.0689519   .0389601    -1.77   0.077    -.1453123    .0074086
             |
        wave |   .0008363   .0016951     0.49   0.622     -.002486    .0041586
-------------+----------------------------------------------------------------
       /cut1 |   2.471169   .0918639                      2.291119    2.651219
       /cut2 |   3.741284   .0880944                      3.568622    3.913946
       /cut3 |   5.467899   .0903708                      5.290775    5.645022
       /cut4 |    6.34045   .0915465                      6.161022    6.519878
       /cut5 |   7.771034   .0948209                      7.585189    7.956879
       /cut6 |   10.20031   .0992505                      10.00578    10.39483
------------------------------------------------------------------------------

. 
end of do-file

. do "C:\Users\ADMINI~1.ADM\AppData\Local\Temp\STD10000000.tmp"

. brant

Brant test of parallel regression assumption

              |       chi2     p>chi2      df
 -------------+------------------------------
          All |    2458.61      0.000     140
 -------------+------------------------------
      ljbsat7 |      31.42      0.000       5
        jbsat |     908.64      0.000       5
      1.skill |       7.65      0.177       5
        2.sex |     253.72      0.000       5
  0.permanent |       4.80      0.441       5
    2.M_event |      22.96      0.000       5
    3.M_event |      23.08      0.000       5
    4.M_event |      31.84      0.000       5
    5.M_event |       5.96      0.310       5
    6.M_event |      12.75      0.026       5
    7.M_event |       0.83      0.975       5
     2.qfedhi |       2.69      0.748       5
     3.qfedhi |      10.94      0.053       5
     4.qfedhi |      23.46      0.000       5
     5.qfedhi |       8.59      0.127       5
     6.qfedhi |      28.36      0.000       5
     7.qfedhi |      31.59      0.000       5
     8.qfedhi |      14.86      0.011       5
     9.qfedhi |      20.11      0.001       5
    10.qfedhi |       9.52      0.090       5
    11.qfedhi |      21.60      0.001       5
    12.qfedhi |      48.98      0.000       5
    13.qfedhi |       8.18      0.147       5
          age |     134.66      0.000       5
      1.child |      17.99      0.003       5
      2.child |      12.87      0.025       5
      3.child |       7.54      0.183       5
         wave |     302.19      0.000       5

A significant test statistic provides evidence that the parallel
regression assumption has been violated.

Looking at the results of the Brant test for M_event, I noticed that the first three categories break the assumption, but the last three don't. The first three categories capture instance where individual leave their employer, the last three categories look at instances where people change jobs with the same employer. I think theoretically, leaving an employer constitutes a greater intervention than changing jobs with the same employer, and that might be the reason why the assumption is broken as leaving a firm increases your likelihoods of crossing several categories of satisfaction, whereas changing jobs with the same employer may just improve one or two boundaries.

I recode the dependent variable into a three category variable which just captures "disagree, neutral, and agree". I run the ordinal logistic regression again.

. ologit jbsat7mini ljbsat7mini jbsat ib0.skill i.sex ///
> ib1.perm  i.M_e  i.qfedhi age i.child ///
> wave , cluster(pid) nolog

Ordered logistic regression                       Number of obs   =      76133
                                                  Wald chi2(28)   =   11534.15
                                                  Prob > chi2     =     0.0000
Log pseudolikelihood = -44463.114                 Pseudo R2       =     0.2009

                                (Std. Err. adjusted for 12317 clusters in pid)
------------------------------------------------------------------------------
             |               Robust
  jbsat7mini |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
 ljbsat7mini |   .9830918    .015206    64.65   0.000     .9532885    1.012895
       jbsat |    .677014   .0092879    72.89   0.000       .65881    .6952179
             |
       skill |
    skilled  |  -.2097881   .0254309    -8.25   0.000    -.2596318   -.1599445
             |
         sex |
     female  |   .1335524   .0274013     4.87   0.000     .0798468     .187258
             |
   permanent |
  Temporary  |   .1602662   .0689351     2.32   0.020     .0251559    .2953766
             |
     M_event |
Changed e..  |   .2084667   .0419358     4.97   0.000     .1262741    .2906593
Changed e..  |  -.0005936   .0761938    -0.01   0.994    -.1499307    .1487436
Changed e..  |   .0600634   .0607583     0.99   0.323    -.0590208    .1791475
Changed j..  |  -.0998616   .0386601    -2.58   0.010     -.175634   -.0240893
Changed j..  |   .2207144   .1614364     1.37   0.172    -.0956951     .537124
Changed j..  |   .1026248   .0714939     1.44   0.151    -.0375006    .2427503
             |
      qfedhi |
first deg..  |   .0254113   .0796304     0.32   0.750    -.1306614     .181484
teaching qf  |   .0416455   .1210283     0.34   0.731    -.1955655    .2788566
other hig..  |    .328927   .0763287     4.31   0.000     .1793256    .4785284
 nursing qf  |   .3529244   .1223951     2.88   0.004     .1130345    .5928144
gce a lev..  |   .4723285   .0802419     5.89   0.000     .3150572    .6295997
gce o lev..  |   .4268029   .0791424     5.39   0.000     .2716867    .5819191
commercia..  |   .4118384    .120123     3.43   0.001     .1764017    .6472751
cse grade..  |   .4287832   .0931371     4.60   0.000     .2462379    .6113285
apprentic~p  |   .3439612   .1689428     2.04   0.042     .0128395    .6750829
   other qf  |   .2192476   .1771439     1.24   0.216     -.127948    .5664432
      no qf  |     .40117   .0831582     4.82   0.000     .2381829     .564157
still at ..  |   -.158634   .2564787    -0.62   0.536    -.6613231    .3440551
             |
         age |  -.0002758   .0011466    -0.24   0.810     -.002523    .0019715
             |
       child |
          1  |  -.0200702   .0296272    -0.68   0.498    -.0781385    .0379981
          2  |   .0165892   .0318352     0.52   0.602    -.0458067    .0789851
          3  |  -.1034556   .0521423    -1.98   0.047    -.2056526   -.0012586
             |
        wave |   .0196952   .0022694     8.68   0.000     .0152472    .0241433
-------------+----------------------------------------------------------------
       /cut1 |   4.484099   .1027336                      4.282745    4.685454
       /cut2 |   5.326011   .1035509                      5.123055    5.528967
------------------------------------------------------------------------------

. 
end of do-file

. do "C:\Users\ADMINI~1.ADM\AppData\Local\Temp\STD10000000.tmp"

The basic effects of M_event remain the same, with some differences in significance. However, the Brant test looks different.

. brant

Brant test of parallel regression assumption

              |       chi2     p>chi2      df
 -------------+------------------------------
          All |     746.32      0.000      28
 -------------+------------------------------
  ljbsat7mini |      36.81      0.000       1
        jbsat |     220.93      0.000       1
      1.skill |       0.12      0.724       1
        2.sex |     207.98      0.000       1
  0.permanent |       0.79      0.373       1
    2.M_event |       0.55      0.457       1
    3.M_event |       1.33      0.249       1
    4.M_event |       0.85      0.356       1
    5.M_event |       0.07      0.796       1
    6.M_event |       1.25      0.264       1
    7.M_event |       0.36      0.549       1
     2.qfedhi |       0.02      0.894       1
     3.qfedhi |       0.99      0.321       1
     4.qfedhi |       0.87      0.352       1
     5.qfedhi |       1.10      0.295       1
     6.qfedhi |       1.77      0.183       1
     7.qfedhi |       5.15      0.023       1
     8.qfedhi |       0.27      0.606       1
     9.qfedhi |       2.93      0.087       1
    10.qfedhi |       0.56      0.453       1
    11.qfedhi |       8.91      0.003       1
    12.qfedhi |       1.25      0.263       1
    13.qfedhi |       0.33      0.563       1
          age |      52.54      0.000       1
      1.child |       0.79      0.375       1
      2.child |       8.07      0.004       1
      3.child |       3.98      0.046       1
         wave |      18.29      0.000       1

A significant test statistic provides evidence that the parallel
regression assumption has been violated.

. 
end of do-file

The mobility events no longer violate the proportional odds assumption. Would I get away with this, or am I ignoring wider issues with the data?

[Maarten Buis]

You have 76,000 observations, so you should not be surprised that you find statistically significant results. In such datasets the more important question, especially when it comes to model choice, is whether the differences you find are substantively significant. This means just looking at the coefficients of the unconstrained model and judging whether the differences are important or not. You can get a fair approximation of the unconstrained coefficients by adding the detail option to brant. If you find some substantively significant differences, than you can relax the proportional odds assumptions for just those variables with Rich William's gologit2 (available from SSC).

[Ivan Privalko]

Thanks for your answer Martin,

The dataset is really large, I'm worried about this myself. Specificaly because it's a panel which I'm estimating with cross-sectional techniques, but I'll move on to FE later. Anyway.

Am I right in thinking I should compare the estimated coefficients from binary logits?

. brant, detail

Estimated coefficients from binary logits

--------------------------------------------------------------------------------
    Variable |  y_gt_1     y_gt_2     y_gt_3     y_gt_4     y_gt_5     y_gt_6   
-------------+------------------------------------------------------------------
     ljbsat7 |    0.548      0.566      0.601      0.583      0.559      0.587  
             |    29.35      46.46      76.26      81.44      79.08      47.52  
       jbsat |    0.766      0.722      0.591      0.692      0.757      1.124  
             |    39.83      57.74      70.10      85.56      84.22      62.28  
             |
       skill |
    skilled  |   -0.085     -0.160     -0.214     -0.206     -0.221     -0.239  
             |    -1.26      -3.77      -8.31      -9.30     -11.40      -8.65  
             |
         sex |
     female  |   -0.019      0.022     -0.105      0.176      0.127      0.281  
             |    -0.29       0.52      -4.15       8.15       6.87      11.03  
             |
   permanent |
  Temporary  |   -0.108     -0.007      0.108      0.168      0.273      0.191  
             |    -0.57      -0.05       1.34       2.39       4.59       2.61  
             |
     M_event |
Changed e..  |   -0.014      0.055      0.258      0.280      0.355      0.447  
             |    -0.10       0.66       5.14       6.69      10.22      10.44  
Changed e..  |   -0.513     -0.202     -0.019      0.027      0.063      0.346  
             |    -2.69      -1.49      -0.21       0.34       0.93       3.82  
Changed e..  |   -0.557     -0.189      0.070      0.112      0.196      0.338  
             |    -3.70      -1.77       1.00       1.84       3.70       4.89  
Changed j..  |   -0.189     -0.151     -0.087     -0.090     -0.025     -0.106  
             |    -1.43      -1.88      -1.84      -2.24      -0.71      -2.13  
Changed j..  |   -0.237      0.308      0.081      0.240     -0.107     -0.405  
             |    -0.64       1.07       0.46       1.53      -0.79      -1.85  
Changed j..  |    0.097      0.117      0.068      0.122      0.100      0.138  
             |     0.46       0.86       0.83       1.70       1.58       1.54  
             |
      qfedhi |
first deg..  |   -0.035     -0.087      0.052      0.040      0.067      0.033  
             |    -0.19      -0.80       0.84       0.71       1.28       0.39  
teaching qf  |   -0.434     -0.188      0.091      0.137      0.112     -0.161  
             |    -1.92      -1.27       1.01       1.67       1.49      -1.43  
other hig..  |    0.118      0.101      0.413      0.350      0.153      0.155  
             |     0.67       0.96       6.90       6.55       3.07       1.94  
 nursing qf  |    1.269      0.360      0.450      0.330      0.194      0.080  
             |     2.35       1.53       3.62       3.09       2.12       0.65  
gce a lev..  |    0.404      0.362      0.582      0.494      0.233      0.189  
             |     2.05       3.08       8.79       8.48       4.38       2.25  
gce o lev..  |    0.208      0.258      0.560      0.431      0.205      0.176  
             |     1.13       2.31       8.85       7.69       3.98       2.16  
commercia..  |   -0.261      0.031      0.425      0.439      0.242      0.458  
             |    -0.94       0.17       4.02       4.71       3.08       4.45  
cse grade..  |    0.182      0.209      0.567      0.431      0.196      0.228  
             |     0.77       1.43       6.58       5.86       2.99       2.34  
apprentic~p  |   -0.048     -0.082      0.351      0.378      0.494      0.373  
             |    -0.16      -0.42       2.69       3.22       4.62       2.66  
   other qf  |   -0.804     -0.460      0.148      0.413      0.164      0.163  
             |    -2.44      -1.92       0.90       2.70       1.24       0.92  
      no qf  |   -0.418     -0.227      0.425      0.446      0.262      0.377  
             |    -2.25      -1.97       6.23       7.33       4.74       4.50  
still at ..  |   -1.433     -0.651     -0.003     -0.228      0.175      0.213  
             |    -2.35      -1.28      -0.01      -0.67       0.53       0.39  
             |
         age |   -0.001      0.001     -0.007      0.000      0.006      0.007  
             |    -0.17       0.37      -5.92       0.12       7.35       6.61  
             |
       child |
          1  |    0.023      0.092     -0.036     -0.013     -0.072      0.025  
             |     0.26       1.66      -1.11      -0.49      -3.05       0.78  
          2  |   -0.142     -0.011     -0.033      0.030      0.041      0.078  
             |    -1.62      -0.20      -0.97       1.05       1.66       2.38  
          3  |   -0.165     -0.200     -0.174     -0.091     -0.082      0.000  
             |    -1.10      -2.21      -3.16      -1.92      -2.00       0.01  
             |
     jbsize1 |
          1  |   -0.226     -0.208     -0.189     -0.135     -0.076      0.031  
             |    -2.60      -3.84      -5.86      -4.92      -3.22       0.95  
          2  |   -0.167     -0.104     -0.040     -0.019     -0.030     -0.065  
             |    -1.74      -1.71      -1.08      -0.61      -1.12      -1.67  
             |
        wave |    0.047      0.025      0.015      0.025      0.008     -0.026  
             |     6.92       5.69       5.62      11.21       4.23     -10.23  
       _cons |   -1.895     -2.995     -4.106     -5.889     -7.415    -12.037  
             |    -7.66     -19.01     -41.97     -65.20     -84.00     -78.31  
--------------------------------------------------------------------------------
                                                                     legend: b/t

[Maarten Buis]

A large sample size is in itself not something to worry about; it just means that statistical tests are less useful (though strictly speaking still correct). The panel structure is something to worry about.

To answer your question: yes. So female would be a variable that changes noticebly.