Hi all
I have a problem with results that compare marginal effects under different conditions. Simply put, I think both "test" command and "mlincom" command are generating dubious results.
My question is whether they are reliable and if not, what is wrong with the tests I conducted.
Here is what I did:
I ran a three-way interaction effect of the variables growthpr, s, and income using "xtlogit" and estimated marginal effects of growthpr while fixing s and income at certain values of interest using "margins, dxdy(x1) at(x2=.. x3=...)".growthpr and s are continuous variables, and income is discrete and runs from 1 to 5.
The results are shown below:
Based on the margins results and the graph produced by "marginsplot, x(s) plot(income)" following the margins command, it seems that both income groups (1:the poorest and 5: the richest) show similar patterns such that as "s" increases the effect of "growthpr" diminishes, and the effect disappears when "s" becomes larger than 2 for the rich (income=5) and 2.5 for the poor (income=1). Thus, it seems that the effect of "growthpr" reduces as "s" grows, and the effect is apparent among both the poor and the rich.
In order to statistically show that s reduces marginal effects of growthpr, I calculated the first difference of marginal effects for each income group.
For that, I used both "test" command following the margins estimation as well as "mlincom" command in the spost13 package. Both generate the same results with the same p-value for each test although the test statistics they report are different: “test” reports Chi-2 and “mlincom” generates linear combination of estimates from margins. The results of "mlincom" are shown below:
Based on the first difference estimation, for the rich (income=5), the marginal effect of "growthpr" when "s" takes the highest value (=3.5) is statistically different from that at all levels of "s", suggesting that there is a moderating effect of "s" among the richest income group.
However, for the poor (income=1), the marginal effects of "growthpr" at all levels of "s" are not statistically distinguishable from one another, suggesting that there is no moderating effect of "s" among the poor.
And this conclusion is quite different from what the graph shows. Based on the graph and the actual gap in numbers, it is nonsensical to say that the marginal effects # 14 (= -.0038948) and #16 (= -.0078576) for the rich at s=3 and 3.5, respectively, are statistically different, while the marginal effects #1 (=.0250461) and #15 (=.0012408) for the poor at s=0 and 3.5, respectively, are not different from each other, especially given that their standard errors are not very different in size.
Does the mlincom results make sense...?
It would make sense if the marginal effects for both income groups to be statistically different when I compare them at very different levels of s, but their differences become insignificant as I compare marginal effects at similar levels of s; and the gaps to be slightly more significant for the rich than for the poor, which is not what the test results show right now...
I had problems in using "test" command to compare marginal effects previously. It sometimes seems to work in a weird way. So, that makes me think that there is something wrong with the test results here as well. I wonder if everybody agrees that there is something wrong. I also want to know how to fix this and compare the marginal effects correctly.
Any comments and advice will be highly appreciated. Thank you in advance!
I have a problem with results that compare marginal effects under different conditions. Simply put, I think both "test" command and "mlincom" command are generating dubious results.
My question is whether they are reliable and if not, what is wrong with the tests I conducted.
Here is what I did:
I ran a three-way interaction effect of the variables growthpr, s, and income using "xtlogit" and estimated marginal effects of growthpr while fixing s and income at certain values of interest using "margins, dxdy(x1) at(x2=.. x3=...)".growthpr and s are continuous variables, and income is discrete and runs from 1 to 5.
The results are shown below:
Code:
. xtlogit votech c.growthpr##c.s##c.income educ soph pidch female married unemployed, nolog
note: 8400.country omitted because of collinearity
Random-effects logistic regression Number of obs = 72210
Group variable: election1 Number of groups = 77
Random effects u_i ~ Gaussian Obs per group: min = 128
avg = 937.8
max = 3339
Integration method: mvaghermite Integration points = 12
Wald chi2(52) = 15043.21
Log likelihood = -30646.39 Prob > chi2 = 0.0000
------------------------------------------------------------------------------------------
votech | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------------------+----------------------------------------------------------------
growthpr | .1442353 .0596027 2.42 0.016 .0274161 .2610544
s | -.4836696 .1520272 -3.18 0.001 -.7816374 -.1857017
|
c.growthpr#c.s | -.0314627 .0332276 -0.95 0.344 -.0965875 .0336621
|
income | -.0405134 .022002 -1.84 0.066 -.0836366 .0026097
|
c.growthpr#c.income | .0088782 .0061788 1.44 0.151 -.0032321 .0209884
|
c.s#c.income | .0414441 .0136099 3.05 0.002 .0147692 .0681191
|
c.growthpr#c.s#c.income | -.0085141 .003867 -2.20 0.028 -.0160934 -.0009349
|
educ | -.066387 .0070612 -9.40 0.000 -.0802267 -.0525474
soph | -.0872549 .0118639 -7.35 0.000 -.1105077 -.0640021
pidch | 3.603269 .0296561 121.50 0.000 3.545145 3.661394
female | .0767834 .021088 3.64 0.000 .0354517 .1181151
married | .0909665 .0246408 3.69 0.000 .0426713 .1392616
unemployed | -.0806562 .0522876 -1.54 0.123 -.1831381 .0218257
|
_cons | -1.478319 .4122553 -3.59 0.000 -2.286325 -.6703135
-------------------------+----------------------------------------------------------------
/lnsig2u | -2.049376 .1757649 -2.393868 -1.704883
-------------------------+----------------------------------------------------------------
sigma_u | .3589085 .0315418 .302119 .4263727
rho | .0376798 .0063732 .0269956 .052365
------------------------------------------------------------------------------------------
Likelihood-ratio test of rho=0: chibar2(01) = 928.71 Prob >= chibar2 = 0.000
. margins, dydx(growthpr) at(s=(0(.5)3.5) income=(1 5)) predict(pu0) post
Average marginal effects Number of obs = 72210
Model VCE : OIM
Expression : Pr(votech=1 assuming u_i=0), predict(pu0)
dy/dx w.r.t. : growthpr
1._at : s = 0
income = 1
2._at : s = 0
income = 5
3._at : s = .5
income = 1
4._at : s = .5
income = 5
5._at : s = 1
income = 1
6._at : s = 1
income = 5
7._at : s = 1.5
income = 1
8._at : s = 1.5
income = 5
9._at : s = 2
income = 1
10._at : s = 2
income = 5
11._at : s = 2.5
income = 1
12._at : s = 2.5
income = 5
13._at : s = 3
income = 1
14._at : s = 3
income = 5
15._at : s = 3.5
income = 1
16._at : s = 3.5
income = 5
------------------------------------------------------------------------------
| Delta-method
| dy/dx Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
growthpr |
_at |
1 | .0250461 .0098778 2.54 0.011 .0056858 .0444063
2 | .0291296 .0094184 3.09 0.002 .01067 .0475892
3 | .0204341 .0070597 2.89 0.004 .0065974 .0342708
4 | .0226016 .0068806 3.28 0.001 .0091159 .0360872
5 | .0160909 .0045529 3.53 0.000 .0071674 .0250144
6 | .0163199 .0045498 3.59 0.000 .0074024 .0252374
7 | .0121725 .0029369 4.14 0.000 .0064163 .0179287
8 | .0104831 .0030103 3.48 0.000 .004583 .0163831
9 | .0087574 .00289 3.03 0.002 .0030931 .0144216
10 | .0051963 .0029983 1.73 0.083 -.0006803 .0110729
11 | .0058407 .0038104 1.53 0.125 -.0016276 .013309
12 | .0004317 .0040198 0.11 0.914 -.0074469 .0083103
13 | .003362 .0048688 0.69 0.490 -.0061807 .0129048
14 | -.0038948 .0052978 -0.74 0.462 -.0142783 .0064887
15 | .0012408 .005857 0.21 0.832 -.0102386 .0127203
16 | -.0078576 .0065926 -1.19 0.233 -.0207789 .0050638
------------------------------------------------------------------------------
Based on the margins results and the graph produced by "marginsplot, x(s) plot(income)" following the margins command, it seems that both income groups (1:the poorest and 5: the richest) show similar patterns such that as "s" increases the effect of "growthpr" diminishes, and the effect disappears when "s" becomes larger than 2 for the rich (income=5) and 2.5 for the poor (income=1). Thus, it seems that the effect of "growthpr" reduces as "s" grows, and the effect is apparent among both the poor and the rich.
In order to statistically show that s reduces marginal effects of growthpr, I calculated the first difference of marginal effects for each income group.
For that, I used both "test" command following the margins estimation as well as "mlincom" command in the spost13 package. Both generate the same results with the same p-value for each test although the test statistics they report are different: “test” reports Chi-2 and “mlincom” generates linear combination of estimates from margins. The results of "mlincom" are shown below:
Code:
. qui mlincom 1-15, add rowname(Poor1_15)
. qui mlincom 3-15, add rowname(Poor3_15)
. qui mlincom 5-15, add rowname(Poor5_15)
. qui mlincom 7-15, add rowname(Poor7_15)
. qui mlincom 9-15, add rowname(Poor9_15)
. qui mlincom 11-15, add rowname(Poor11_15)
. qui mlincom 13-15, add rowname(Poor13_15)
. qui mlincom 2-16, add rowname(Rich2_16)
. qui mlincom 4-16, add rowname(Rich4_16)
. qui mlincom 6-16, add rowname(Rich6_16)
. qui mlincom 8-16, add rowname(Rich8_16)
. qui mlincom 10-16, add rowname(Rich10_16)
. qui mlincom 12-16, add rowname(Rich12_16)
. mlincom 14-16, add rowname(Rich14_16)
| lincom pvalue ll ul
-------------+----------------------------------------
Poor1_15 | 0.024 0.107 -0.005 0.053
Poor3_15 | 0.019 0.103 -0.004 0.042
Poor5_15 | 0.015 0.095 -0.003 0.032
Poor7_15 | 0.011 0.085 -0.002 0.023
Poor9_15 | 0.008 0.076 -0.001 0.016
Poor11_15 | 0.005 0.070 -0.000 0.010
Poor13_15 | 0.002 0.068 -0.000 0.004
Rich2_16 | 0.037 0.012 0.008 0.066
Rich4_16 | 0.030 0.012 0.007 0.054
Rich6_16 | 0.024 0.011 0.006 0.043
Rich8_16 | 0.018 0.009 0.005 0.032
Rich10_16 | 0.013 0.008 0.003 0.023
Rich12_16 | 0.008 0.008 0.002 0.014
Rich14_16 | 0.004 0.008 0.001 0.007
However, for the poor (income=1), the marginal effects of "growthpr" at all levels of "s" are not statistically distinguishable from one another, suggesting that there is no moderating effect of "s" among the poor.
And this conclusion is quite different from what the graph shows. Based on the graph and the actual gap in numbers, it is nonsensical to say that the marginal effects # 14 (= -.0038948) and #16 (= -.0078576) for the rich at s=3 and 3.5, respectively, are statistically different, while the marginal effects #1 (=.0250461) and #15 (=.0012408) for the poor at s=0 and 3.5, respectively, are not different from each other, especially given that their standard errors are not very different in size.
Does the mlincom results make sense...?
It would make sense if the marginal effects for both income groups to be statistically different when I compare them at very different levels of s, but their differences become insignificant as I compare marginal effects at similar levels of s; and the gaps to be slightly more significant for the rich than for the poor, which is not what the test results show right now...
I had problems in using "test" command to compare marginal effects previously. It sometimes seems to work in a weird way. So, that makes me think that there is something wrong with the test results here as well. I wonder if everybody agrees that there is something wrong. I also want to know how to fix this and compare the marginal effects correctly.
Any comments and advice will be highly appreciated. Thank you in advance!
Comment