Is it valid to include country fixed and time fixed effects in multi level mixed model with -melogit-,-meqrlogit- or -xtmelogit- command alongwith random effects?
-
Login or Register
- Log in with
- You are not logged in. You can browse but not post. Login or Register by clicking 'Login or Register' at the top-right of this page. For more information on Statalist, see the FAQ.
-
-
Yes and no. If you want to interpret your results as limited to the particular countries and times that are in your data set, then it is OK. If, however, you want to interpret your results as generalizing to all countries and all times, it is not. For the latter type of inference, you should add the countries and times as random effects in the model with the nesting relationships appropriate to your study design.Comment
-
Thank you Clyde for your reply. Which are the values of an overall Level 1 variation, Level 2 variation , Proportion Level 1 variation explained, from the output of -melogit- ? It would be great if you show an output to highlight these values.
Comment
-
I don't know what you mean by "overall Level 1 variation, Level 2 variation , Proportion Level 1 variation explained." I think you are thinking of quantities that make sense in the context of linear regression of a continuous outcome but are not applicable in the context of ordinal regression.Comment
-
Dear Clyde,
I believe my issue is very similar to that of Maliha's but not identical thus I was hoping you could assist me with it.
I have cross-sectional longitudinal data employing a random effects model with three levels: individuals nested within country-years clustered with countries (as specified by the likes of Schmidt-Catran and Fairbrother 2016). For the purposes of the example below I wish to calculate the predictive margins for y=1 - that is, the probability that one is supportive of the European Union - for each of the six age cohorts that I theorise exist across central and eastern European member states across the period of 2004-2015 utilised by my study.
Each cohort can be thought of as generations; generally, the theory is that each successive cohort is more supportive of the EU than the previous cohort. My descriptive aggregate analysis appears to indicate the validity of this relationship (I tried to find a way to show the images but could not find a way to do so other than attaching them which, as you stated previously in this thread, is undesirable - I couldn't find it in the FAQs either; I note what you said previously in the thread about images but I did not understand it fully so if you could point me in the direction of where I could understand this in more detail that'd be wonderful).
Yet, in direct contrast to my descriptive aggregate analysis, my regression results appear to convey precisely the opposite. With the reference category for cohorts the oldest cohort - the generation that came to age (16 years old) in the aftermath of the Second World War, my statistical model clearly indicates that, broadly speaking and contrary to my expectations and the descriptive aggregate analysis, successive cohorts have not become more supportive of the EU than those that came before it. The headline result being that the most recent cohort that came of age (16) during the period of their country's accession to the EU (EU Cohort) is evidently much more Eurosceptic than the WWII cohort (reference category); a negative coefficient indicates less support for the EU contrasted to the reference category.
Code:melogit EU_Goodmembership Gender Age i.Education4 i.Occupation Centerism i.CEECohort i.year || Country: || cyear:, /// Iteration log omitted Mixed-effects logistic regression Number of obs = 107,893 ------------------------------------------------------------- | No. of Observations per Group Group Variable | Groups Minimum Average Maximum ----------------+-------------------------------------------- Country | 10 8,408 10,789.3 13,365 cyear | 90 517 1,198.8 2,599 ------------------------------------------------------------- Integration method: mvaghermite Integration pts. = 7 Wald chi2(27) = 552.82 Log likelihood = -34359.284 Prob > chi2 = 0.0000 ---------------------------------------------------------------------------------------------------------------- EU_Goodmembership | Coef. Std. Err. z P>|z| [95% Conf. Interval] -----------------------------------------------+---------------------------------------------------------------- Gender | .0958534 .0209913 4.57 0.000 .0547113 .1369955 Age | -.0077781 .0021742 -3.58 0.000 -.0120395 -.0035168 | Education4 | 16-19 years | .177717 .0286765 6.20 0.000 .1215121 .2339219 20+ years | .4348324 .0344087 12.64 0.000 .3673925 .5022723 Still Studying | .0970692 .3519061 0.28 0.783 -.5926541 .7867925 No full time education | -.1857 .1669297 -1.11 0.266 -.5128763 .1414762 | Occupation | Managers (coded 10 to 12 in V432) | .0708984 .0572618 1.24 0.216 -.0413328 .1831295 Other white collars (coded 13 or 14 in V432) | -.0850771 .0513118 -1.66 0.097 -.1856464 .0154922 Manual workers (coded 15 to 18 in V432) | -.1472843 .0486853 -3.03 0.002 -.2427057 -.051863 House persons (coded 1 in V432) | -.2099896 .0705756 -2.98 0.003 -.3483152 -.0716641 Unemployed (coded 3 in V432) | -.3104216 .0552211 -5.62 0.000 -.418653 -.2021902 Retired (coded 4 in V432) | -.1744436 .0542082 -3.22 0.001 -.2806897 -.0681976 Students (coded 2 in V432) | .2277265 .3547057 0.64 0.521 -.4674838 .9229369 | Centerism | .1047876 .0210369 4.98 0.000 .063556 .1460192 | CEECohort | Exclusive Nat. Id. Cohort | -.0972863 .043503 -2.24 0.025 -.1825507 -.0120219 Pre-Democracy Cohort | -.2197141 .0669892 -3.28 0.001 -.3510105 -.0884176 Elementary Democracy Cohort | -.2594702 .0989821 -2.62 0.009 -.4534716 -.0654688 Mature Democracy Cohort | -.2181035 .110254 -1.98 0.048 -.4341974 -.0020095 EU Cohort | -.361624 .1344347 -2.69 0.007 -.6251112 -.0981368 | year | 2005 | -.1971077 .127506 -1.55 0.122 -.447015 .0527995 2006 | -.0330447 .1322088 -0.25 0.803 -.2921692 .2260798 2007 | -.03179 .1284018 -0.25 0.804 -.2834529 .2198729 2008 | -.2108615 .1278694 -1.65 0.099 -.4614808 .0397579 2009 | -.2983127 .126828 -2.35 0.019 -.546891 -.0497345 2010 | -.3772642 .1307442 -2.89 0.004 -.6335181 -.1210103 2013 | -.3558175 .1313589 -2.71 0.007 -.6132761 -.0983588 2015 | -.4296779 .1315059 -3.27 0.001 -.6874248 -.171931 | _cons | 2.736926 .2106566 12.99 0.000 2.324046 3.149805 -----------------------------------------------+---------------------------------------------------------------- Country | var(_cons)| .0995162 .0485897 .0382197 .2591197 -----------------------------------------------+---------------------------------------------------------------- Country>cyear | var(_cons)| .0717233 .0131501 .0500723 .1027363 ---------------------------------------------------------------------------------------------------------------- LR test vs. logistic model: chi2(2) = 1826.78 Prob > chi2 = 0.0000
Quite simply, I wish to calculate the predictive margins for each of these cohorts at the occurrence of each year of my dataset. Thus I used the margins command:
When I follow this through with the marginsplot command, the relationship described by the coefficients is reversed as can be seen by the values of the margins. Indeed, although the levels of support envinced by the cohorts decrease across the years, the WWII cohort persists as the most Eurosceptic and the Euro cohort the least throughout the period; thus, the proposed hypothesis of each successive cohort being more supportive of the EU than the last cohort is confirmed. My question is: how is this relationship the polar opposite of that identified by the coefficients in the regression model? I realise the coefficients are not directly interpretable as they would be in a linear model and that they should be taken as odds ratios to provide meaningful interpretation yet even in doing so the relationship is still the same with each successive cohort having an odds ratio < 1.00 when contrasted to the WWII cohort. I also note that because of the nature of my model I should I have estimated the margins with derivatives using the command:Code:margins, at(year=(1(1)9)) over (CEECohort) Predictive margins Number of obs = 107,893 Model VCE : OIM Expression : Marginal predicted mean, predict() over : CEECohort 1._at : 1.CEECohort year = 1 2.CEECohort year = 1 3.CEECohort year = 1 4.CEECohort year = 1 5.CEECohort year = 1 6.CEECohort year = 1 2._at : 1.CEECohort year = 2 2.CEECohort year = 2 3.CEECohort year = 2 4.CEECohort year = 2 5.CEECohort year = 2 6.CEECohort year = 2 \\\ *Continues sequentially until year 9 (2015) ------------------------------------------------------------------------------------------------ | Delta-method | Margin Std. Err. z P>|z| [95% Conf. Interval] -------------------------------+---------------------------------------------------------------- _at#CEECohort | 1#WWII Cohort | .9043577 .0117522 76.95 0.000 .8813238 .9273915 1#Exclusive Nat. Id. Cohort | .9078451 .0113353 80.09 0.000 .8856282 .930062 1#Pre-Democracy Cohort | .9150386 .0104679 87.41 0.000 .8945219 .9355553 1#Elementary Democracy Cohort | .9217715 .009989 92.28 0.000 .9021935 .9413495 1#Mature Democracy Cohort | .9307186 .0088544 105.11 0.000 .9133642 .9480729 1#EU Cohort | .9299433 .0092226 100.83 0.000 .9118674 .9480192 2#WWII Cohort | .8862858 .0135769 65.28 0.000 .8596755 .9128962 2#Exclusive Nat. Id. Cohort | .8903472 .0131028 67.95 0.000 .8646661 .9160283 2#Pre-Democracy Cohort | .8987534 .0121327 74.08 0.000 .8749737 .922533 2#Elementary Democracy Cohort | .9066176 .0116196 78.02 0.000 .8838435 .9293917 2#Mature Democracy Cohort | .9171214 .0103356 88.73 0.000 .8968639 .9373788 2#EU Cohort | .9161988 .01076 85.15 0.000 .8951095 .9372881 3#WWII Cohort | .901515 .0123733 72.86 0.000 .8772639 .9257661 3#Exclusive Nat. Id. Cohort | .9050945 .0119203 75.93 0.000 .881731 .9284579 3#Pre-Democracy Cohort | .9124818 .0110133 82.85 0.000 .8908961 .9340675 3#Elementary Democracy Cohort | .9193956 .0104971 87.59 0.000 .8988216 .9399696 3#Mature Democracy Cohort | .9285905 .0093141 99.70 0.000 .9103353 .9468457 3#EU Cohort | .9277921 .0096695 95.95 0.000 .9088402 .9467439 4#WWII Cohort | .9016243 .0120501 74.82 0.000 .8780065 .925242 4#Exclusive Nat. Id. Cohort | .9052002 .0115953 78.07 0.000 .8824738 .9279266 4#Pre-Democracy Cohort | .9125801 .0107096 85.21 0.000 .8915896 .9335706 4#Elementary Democracy Cohort | .919487 .0102265 89.91 0.000 .8994433 .9395306 4#Mature Democracy Cohort | .9286724 .0090662 102.43 0.000 .9109029 .9464418 4#EU Cohort | .9278748 .009414 98.56 0.000 .9094236 .946326 5#WWII Cohort | .8849218 .0137399 64.41 0.000 .8579921 .9118515 5#Exclusive Nat. Id. Cohort | .8890254 .0132293 67.20 0.000 .8630965 .9149544 5#Pre-Democracy Cohort | .8975212 .0122466 73.29 0.000 .8735183 .9215241 5#Elementary Democracy Cohort | .905469 .0117342 77.16 0.000 .8824703 .9284677 5#Mature Democracy Cohort | .9160885 .0104427 87.73 0.000 .8956211 .9365558 5#EU Cohort | .9151548 .0108295 84.51 0.000 .8939293 .9363803 6#WWII Cohort | .8759193 .01454 60.24 0.000 .8474214 .9044171 6#Exclusive Nat. Id. Cohort | .880298 .0139894 62.93 0.000 .8528792 .9077168 6#Pre-Democracy Cohort | .8893787 .0129702 68.57 0.000 .8639575 .9147999 6#Elementary Democracy Cohort | .8978718 .0124525 72.10 0.000 .8734654 .9222783 6#Mature Democracy Cohort | .9092485 .011105 81.88 0.000 .887483 .9310139 6#EU Cohort | .9082414 .0115105 78.91 0.000 .8856813 .9308015 7#WWII Cohort | .8672903 .0157813 54.96 0.000 .8363594 .8982212 7#Exclusive Nat. Id. Cohort | .8719266 .0151955 57.38 0.000 .842144 .9017093 7#Pre-Democracy Cohort | .8815571 .0141018 62.51 0.000 .8539181 .9091961 7#Elementary Democracy Cohort | .8905626 .0135404 65.77 0.000 .8640239 .9171012 7#Mature Democracy Cohort | .9026544 .0120987 74.61 0.000 .8789413 .9263675 7#EU Cohort | .9015769 .0125048 72.10 0.000 .877068 .9260858 8#WWII Cohort | .8696823 .0156012 55.74 0.000 .8391045 .9002601 8#Exclusive Nat. Id. Cohort | .8742478 .0149969 58.30 0.000 .8448545 .9036412 8#Pre-Democracy Cohort | .8837269 .0139013 63.57 0.000 .8564808 .9109731 8#Elementary Democracy Cohort | .8925914 .0133381 66.92 0.000 .8664492 .9187336 8#Mature Democracy Cohort | .904486 .0119227 75.86 0.000 .881118 .927854 8#EU Cohort | .9034281 .0122798 73.57 0.000 .8793601 .927496 9#WWII Cohort | .8612914 .0164077 52.49 0.000 .8291329 .8934499 9#Exclusive Nat. Id. Cohort | .8661033 .0157748 54.90 0.000 .8351852 .8970214 9#Pre-Democracy Cohort | .8761097 .0146475 59.81 0.000 .8474012 .9048182 9#Elementary Democracy Cohort | .8854653 .0140849 62.87 0.000 .8578594 .9130713 9#Mature Democracy Cohort | .8980483 .0126064 71.24 0.000 .8733401 .9227564 9#EU Cohort | .8969219 .0129686 69.16 0.000 .8715038 .9223399 ------------------------------------------------------------------------------------------------
(I wasn't too sure of the differences at first in doing it with the derrivative syntax hence why I wrongly estimated the margins with my initial command.)Code:margins, dydx(CEECohort) at(year=(1(1)9))
I have the margins command rerunning with the latter command but it takes many many hours for the margins to be obtained. I presage that the same pattern will hold however.
In short: I cannot understand why the margins are the opposite of the regression coefficients. Obviously the odds ratios of a particular coefficient is the odds ratio of that coefficient holding all other variables constant, and margins takes the average marginal effects for the predictors in the model for each cohort at each time point. Despite this, however, I cannot understand why the relationship between the regression coefficients and the margins would be reversed. Is this because the regression coefficients do not take into account the random effects and the margins explicitly includes the random effects in its estimations? Rerunning the margins command (again, incorrectly without the derivative option but I've no reason to believe that the direction of the relationship would be reversed) to obtain just the fixed components of the model is congruent with the regression coefficients and is the antithesis of the margins command specified above to obtain both fixed and random effects as shown below :
I would greatly appreciate any assistance and advice that you could offer, thank you for your time.Code:margins CEECohort, at(year=(1(1)9)) predict(mu fixedonly) vsquish Predictive margins Number of obs = 107,893 Model VCE : OIM Expression : Predicted mean, fixed portion only, predict(mu fixedonly) 1._at : year = 1 2._at : year = 2 3._at : year = 3 4._at : year = 4 5._at : year = 5 6._at : year = 6 7._at : year = 7 8._at : year = 8 9._at : year = 9 ------------------------------------------------------------------------------------------------ | Delta-method | Margin Std. Err. z P>|z| [95% Conf. Interval] -------------------------------+---------------------------------------------------------------- _at#CEECohort | 1#WWII Cohort | .9327901 .0089621 104.08 0.000 .9152248 .9503555 1#Exclusive Nat. Id. Cohort | .9264647 .0093494 99.09 0.000 .9081402 .9447891 1#Pre-Democracy Cohort | .9177319 .0102259 89.75 0.000 .8976896 .9377742 1#Elementary Democracy Cohort | .9147001 .0113208 80.80 0.000 .8925117 .9368885 1#Mature Democracy Cohort | .9178526 .0112347 81.70 0.000 .8958329 .9398723 1#EU Cohort | .9064439 .0138727 65.34 0.000 .8792539 .933634 2#WWII Cohort | .9194121 .0105863 86.85 0.000 .8986633 .9401609 2#Exclusive Nat. Id. Cohort | .9119541 .0109745 83.10 0.000 .8904445 .9334637 2#Pre-Democracy Cohort | .9016925 .0119077 75.72 0.000 .8783539 .9250311 2#Elementary Democracy Cohort | .8981392 .0131328 68.39 0.000 .8723995 .923879 2#Mature Democracy Cohort | .9018341 .0130292 69.22 0.000 .8762973 .9273709 2#EU Cohort | .8884873 .0160046 55.51 0.000 .8571188 .9198558 3#WWII Cohort | .9307 .0095264 97.70 0.000 .9120286 .9493714 3#Exclusive Nat. Id. Cohort | .9241949 .009902 93.33 0.000 .9047873 .9436025 3#Pre-Democracy Cohort | .9152189 .0107611 85.05 0.000 .8941275 .9363102 3#Elementary Democracy Cohort | .9121039 .0118055 77.26 0.000 .8889656 .9352422 3#Mature Democracy Cohort | .9153429 .0116753 78.40 0.000 .8924597 .9382262 3#EU Cohort | .9036246 .0143205 63.10 0.000 .875557 .9316921 4#WWII Cohort | .9307805 .0093412 99.64 0.000 .9124721 .9490888 4#Exclusive Nat. Id. Cohort | .9242822 .0096626 95.66 0.000 .9053439 .9432206 4#Pre-Democracy Cohort | .9153155 .0104588 87.52 0.000 .8948166 .9358144 4#Elementary Democracy Cohort | .9122037 .0114828 79.44 0.000 .8896979 .9347096 4#Mature Democracy Cohort | .9154395 .0113482 80.67 0.000 .8931975 .9376815 4#EU Cohort | .903733 .0139481 64.79 0.000 .8763953 .9310707 5#WWII Cohort | .9183936 .0108754 84.45 0.000 .8970783 .9397089 5#Exclusive Nat. Id. Cohort | .9108511 .0111874 81.42 0.000 .8889241 .932778 5#Pre-Democracy Cohort | .9004758 .0120154 74.94 0.000 .876926 .9240256 5#Elementary Democracy Cohort | .8968839 .01314 68.26 0.000 .87113 .9226378 5#Mature Democracy Cohort | .900619 .0129847 69.36 0.000 .8751695 .9260684 5#EU Cohort | .8871289 .0158628 55.92 0.000 .8560383 .9182195 6#WWII Cohort | .9116411 .0116658 78.15 0.000 .8887765 .9345057 6#Exclusive Nat. Id. Cohort | .9035441 .01194 75.67 0.000 .8801422 .9269461 6#Pre-Democracy Cohort | .8924249 .0127498 70.00 0.000 .8674358 .9174141 6#Elementary Democracy Cohort | .8885805 .0139064 63.90 0.000 .8613244 .9158366 6#Mature Democracy Cohort | .8925782 .0137408 64.96 0.000 .8656468 .9195096 6#EU Cohort | .8781528 .0167406 52.46 0.000 .8453418 .9109638 7#WWII Cohort | .9051188 .0127786 70.83 0.000 .8800731 .9301645 7#Exclusive Nat. Id. Cohort | .8964961 .013071 68.59 0.000 .8708774 .9221148 7#Pre-Democracy Cohort | .8846742 .0138981 63.65 0.000 .8574345 .911914 7#Elementary Democracy Cohort | .880592 .0150655 58.45 0.000 .8510642 .9101198 7#Mature Democracy Cohort | .884837 .0148565 59.56 0.000 .8557189 .9139552 7#EU Cohort | .8695325 .0179563 48.42 0.000 .8343388 .9047261 8#WWII Cohort | .9069317 .0127597 71.08 0.000 .8819232 .9319403 8#Exclusive Nat. Id. Cohort | .8984542 .0129787 69.23 0.000 .8730165 .9238919 8#Pre-Democracy Cohort | .8868261 .0136705 64.87 0.000 .8600323 .9136198 8#Elementary Democracy Cohort | .8828094 .0146885 60.10 0.000 .8540205 .9115982 8#Mature Democracy Cohort | .8869863 .0144258 61.49 0.000 .8587122 .9152604 8#EU Cohort | .8719237 .0173454 50.27 0.000 .8379274 .90592 9#WWII Cohort | .9005556 .0136454 66.00 0.000 .8738112 .9273 9#Exclusive Nat. Id. Cohort | .8915709 .0138103 64.56 0.000 .8645032 .9186386 9#Pre-Democracy Cohort | .8792666 .0144325 60.92 0.000 .8509793 .9075538 9#Elementary Democracy Cohort | .8750215 .0154155 56.76 0.000 .8448076 .9052354 9#Mature Democracy Cohort | .8794359 .0150951 58.26 0.000 .84985 .9090218 9#EU Cohort | .8635303 .0180622 47.81 0.000 .828129 .8989316 ------------------------------------------------------------------------------------------------
Have a good weekend.
References: Schmidt-Catran, A. and Fairbrother, M. (2016) "The Random Effects in Multilevel Models: Getting Them Wrong and Getting Them Right", European Sociolgical Review, 32 (1): pp.23-38Comment
-
Wait, stop the presses. I think your model is mis-specified. So I'd like to raise a number of questions about it before attempting to interpret these results.
1. What is the relationship between cyear and year? I'm guessing that they are the same variable, but you are trying to include year as both a random and fixed effect. You can't actually have it both ways. You might get the model to converge (as you did) specifying it both ways, but the results will make no sense. Or is cyear something else?
2. Why are you modeling year as a random effect at all? I'm not saying it's wrong to do that, but it's unusual.
3. Assuming year truly should be a random effect, I think it's completely implausible that it is nested within country. I would expect it to be crossed with country, or nearly so. I see you have given a reference in support of doing that, but I do not have access to it, and it really doesn't make sense to me at all.
Note: questions 2 and 3 do not apply if cyear is something substantially different from year.
4. Including age, period (year) and cohort all at the same time in a model is a recipe for non-convergence due to the model being unidentified. I suspect you "got away with it" because your cohorts are not single-year birth cohorts but span several years whereas age and period are single years. That gets it by, but remember that you still have a close to exact linear relationship among these three variables, so I wouldn't put much stock in the estimates of any of them.
5. I'm not sure I understand how your year variable works. Is it coded 1 through 9 with value labels such as 2005 to 2015 attached? Or does it actually take on the values 2005 to 2015 (with gaps at 2011 and 2014, and with maybe 2004 as a reference level)?
If we can clear these issues up, then we can try to proceed from there.
Comment
-
Dear Clyde,
Thank you very much for your response. I do not believe that my model is misspecified but appreciate your queries nonetheless; I've addressed your points below which will hopefully clear things up.
1. & 2. cyear is country-year, not year; my apologies if I led you to believe the opposite. So the random effects model specification is: Level 1: Individuals; Level 2: Country-years; Level 3: Country. In short, Schmidt-Catran and Fairbrother (2016) demonstrated through a series of monte carlo simulations that a failure to include random effects for all relevant levels at which data is clustered in multilevel random effects models induces downward bias into the standard errors. In essence, with cross-sectional longitudinal data, (individuals are clustered within) country-years are clustered within countries i.e. observations of individuals within Slovakia in 2004 and individuals in Romania in 2005 are clustered within the countries of Slovakia and Romania respectively.
3. Just for the sake of (fun) information: if one was assessing variables at the level of years within their study - i.e. the period of study is 1980-2015 and one wishes to ascertain the effects of the Great Recession that began in 2008 on individuals then the Great Recession, for example, is a year variable as it affects all individuals across all countries in a given year - then year would be included at the highest level of the random effects parameter and it would be cross-classified with countries as you rightly suspect (again, as demonstrated by Schmidt-Catran and Fairbrother, 2016). A sufficient number of contextual level units, that is, a large number of years in this example, would be needed to ensure sufficient dfs for reliable estimates of the effects of year-level variables, such as the effect of the Great Recession, on individuals to be drawn.
4. Very interesting, I was not aware of that, thank you. I had no problems for this model but did for a similar cohort model for the EU15. These cohorts were theorised by other authors and I am now theorising cohorts for the CEE member states; I had issues obtaining starting values for the EU15 so I had to specify my command as:.Code:melogit EU_Goodmembership Gender Age i.Education4 i.Occupation Centerism i.CohortOver86 i.year || Country: || cyear:, evaltype(gf0)
5. You are indeed correct: year is coded as the values 1 to 9 with value labels attached for each year. So years 1 - 7 represent the years 2004-2010, year 8: 2013, and year 9: 2015.
I hope that this clears things up for you. Thank you again for your assistance.Comment
-
Well, even with the understanding of what cyear is, I remain skeptical of this model. For one thing, the i.year variables are clearly colinear with the cyear level random effects because each i.year is always constant within any cyear. While having them in the model in this way doesn't lead to a detection of colinearity, it does make the model unidentified. Of course, if you need to capture idiosyncratic period effects like the Great Recession, then you can't just rely on random effects modeled as a normal distribution, so I can understand including a small number of specific year or era indicators like 2008 (or maybe 2008-2010 or something like that) for the Great Recession. But the wholesale inclusion of i.year along with cyear: at the next level up leaves the model in a precarious state. If you were to write out the model as an equation with subscripts i, c, and t for individual, country, and time (year), you would say that the yeart indicators are redundant in the presence of the vct random intercepts. So these different terms are "competing" to capture variation in time and the allocation between them is indeterminate. The particular form of the likelihood for the multilevel logit model imposes a way of identifying the model, but it is arbitrary. And it still remains a problem that you are including age, period and cohort all together, when these are also colinear. So I still think this model is extremely problematic from a conceptual perspective. I think if this were my project, I would remove i.year from the model. This would solve one problem. If there is a need to specifically represent a few years because of special "shocks" that occurred then, I would put back indicators just for those specific years. I'm not sure how I would deal with the age-period-cohort problem: that requires commitment to a specific theory of the underlying processes, and I have no expertise in this area.
But leaving all that aside, let me turn to the paradoxical findings of negative coefficients for the cohort indicators but predicted values moving in the opposite direction. I believe the problem is that your first -margins- command does not actually calculate the adjusted predicted values. You, incorrectly, used the -over()- option there. That does not give you adjusted predicted values. It gives you average values restricted to the observations with the corresponding values of cohort. This is, in a sense, the antithesis of adjustment, because your calculations for any given cohort are only calculated for those observations within that cohort, which implies only those values of the other predictors that are observed in that cohort. So, at the very least, you are failing to properly adjust for age, and I suspect you are also for education (as I imagine it, too, has different distributions in the different cohorts) by using the -over()- option.
Your second -margins- command, which has the cohort variable listed in the -varlist- of the -margins- command is the correct syntax for getting adjusted predicted values. And as you see it does not produce paradoxical results. The -over()- option in -margins- really is of limited usefulness, and, in particular, it is not useful for getting adjusted average outcomes or average marginal effects.Comment
-
Dear Clyde,
Those are some excellent insights, many thanks. I wished to obtain the marginal effects across the period of 2004-2015 but I realise now that that may not be possible as it results in the model being unidentifable if I include i.year. I understand its relevance in this context but is there any way to specifically confirm that the model is unidentifiable due to high multicollinearity?
In addition, does it seem perplexing to you that the relationship presented by the regression coefficients and the correct margins command is the opposite of that presented by the descriptive aggregate analysis? I've rarely come across a statistical relationship that's the precise opposite of that presented by the descriptive aggregate analysis.
And just to confirm, is there any difference in the estimation results obtained between the:
and:Code:margins, dydx(CEECohort) at(year=(1(1)9))
commands? That is, moving the derivative of CEE cohort to after the comma from before the comma.Code:margins dydx(CEECohort), at(year=(1(1)9))
Cheers again for all your help.Last edited by Ryan Bain; 01 Oct 2016, 19:39.Comment
-
If you move the dydx(CEECohort) before the comma you will get a syntax error.
The fact that the descriptive aggregate analysis moves the opposite way of the "correct" margins command (i.e. the version that actually fully adjusts for the other variables in the model) is not surprising. It has a name: Simpson's paradox. If you Google that term you'll find lots written about it. (I recommend the Wikipedia article on the topic.) It's a very well known phenomenon. This is just an unusual way to stumble over it.
Well, let me put it in somewhat more nuanced terms. The conceptual model is unidentifiable due to high multicollinearity. The statistical model imposes strong assumptions (logistic distribution for the bottom level error terms, normal distributions for the higher levels) which do identify the model, or at least identify it enough so that your calculations did converge. But that means that the identifiability of your model rests entirely on those strong parametric assumptions. It isn't necessarily a fatal problem: it just makes your results less robust to violation of those assumptions than they might otherwise be.I understand its relevance in this context but is there any way to specifically confirm that the model is unidentifiable due to high multicollinearity?Last edited by Clyde Schechter; 01 Oct 2016, 21:13.Comment
-
Dear Clyde,
Many thanks for your feedback. I'd never come across Simpson's paradox before but I'll certainly look into it, thank you.
The results are broadly identical with the omission of year fixed effects (except for minor changes in the coefficients); therefore, can I ask: does this mean that the statistical model is more robust than the unidentifiable conceptual model - that includes the year fixed effects - otherwise suggests? Notwithstanding, would it make any difference to the identifiability of the statistical or conceptual model if year was included as a continuous variable as opposed to being included in the form of fixed effects?
Have a great day,Comment
-
I'm not sure what you're asking here. I do think the model without i.year is more conceptually sound and robust, and less reliant on the specific distributional assumptions of the statistical analysis than the model that includes it. But to empirically demonstrate that would require fitting both models to a range of data sets to see which produces more stable results.The results are broadly identical with the omission of year fixed effects (except for minor changes in the coefficients); therefore, can I ask: does this mean that the statistical model is more robust than the unidentifiable conceptual model - that includes the year fixed effects - otherwise suggests?
As for adding year as a continuous variable, that doesn't create any identification problems, and it even breaks the inherent unidentifiability of the age-period-cohort model. Of course, it does that by imposing another constraint: that the trend in log-odds outcome over time is linear, with some random walk superimposed by the random effects. Now, in my field, epidemiology, that kind of assumption is often quite reasonable and realistic, especially if the epoch under study isn't terribly long. I couldn't say if the same is true in your area of study.
Comment
-
Clyde,
Apologies for my incredibly late reply. I did type a response to thank you last month but I missed pressing the send button. I loaded this webpage up when clearing through my internet history today and saw that the text hadn't be posted.
Thank you very much Clyde, that's very insightful and has given me a few things to ponder. Thank you again for all your assistance on this matter.
Kind regards,Comment
-
Hello!
I am running a multilevel model, but I am not sure how I should interpret the confidence interval of the independent variables at the second level. Some of them are between 0 and .
Does it mean they do not have a significant effect at the second level?
------------------------------------------------------------------------------
diabetes | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
weight | -.000239 .0001229 -1.95 0.052 -.0004799 1.80e-06
health | -.122913 .0188429 -6.52 0.000 -.1598443 -.0859816
depression | .1788211 .0469478 3.81 0.000 .0868051 .2708372
exercise | -.2549471 .0443097 -5.75 0.000 -.3417925 -.1681017
smoke | -.3297773 .0677661 -4.87 0.000 -.4625963 -.1969582
alcohol | .1646265 .0833558 1.97 0.048 .0012521 .3280009
sleep | .0458308 .010589 4.33 0.000 .0250768 .0665848
parents | .5979495 .0345157 17.32 0.000 .5302999 .665599
hyp | .9216214 .0432706 21.30 0.000 .8368125 1.00643
_cons | -1.649032 .1192447 -13.83 0.000 -1.882747 -1.415317
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
comm_size: Independent |
sd(parents) | 3.04e-07 .0377492 0 .
sd(_cons) | .0967326 .0443607 .039375 .2376435
------------------------------------------------------------------------------
LR test vs. logistic model: chi2(2) = 4.89 Prob > chi2 = 0.0867
Comment
Comment