Significant predictive margins VS insignificant average marginal effects

[Joonho Kim]

Hello,

I am doing my research in political science. I included binary*binary interaction term in my model, and I ran margin and average margin.
First, here is my estimation result including interaction terms. I did probit model.

Probit regression Number of obs = 446
Wald chi2(15) = 146.64
Prob > chi2 = 0.0000
Log pseudolikelihood = -118.81356 Pseudo R2 = 0.4567

(Std. Err. adjusted for 179 clusters in name)
--------------------------------------------------------------------------------------
| Robust
vote | Coef. Std. Err. z P>|z| [95% Conf. Interval]
---------------------+----------------------------------------------------------------
1.compete | .6167187 .3708698 1.66 0.096 -.1101727 1.34361
1.conservative | .7567618 .5191474 1.46 0.145 -.2607484 1.774272
|
conservative#compete |
1 1 | -.8386206 .483335 -1.74 0.083 -1.78594 .1086986
|
1.unsafe | -.6638056 .3788255 -1.75 0.080 -1.40629 .0786787
|
conservative#unsafe |
1 1 | -.2375732 .5869731 -0.40 0.686 -1.388019 .9128729
|
ido | .2424258 .0802953 3.02 0.003 .0850499 .3998016
seniority | -.0638672 .1477812 -0.43 0.666 -.3535131 .2257787
leggender | -.3944897 .7005009 -0.56 0.573 -1.767446 .9784669
age | .3408219 .1482007 2.30 0.021 .0503539 .6312899
edu | -.1078646 .1406759 -0.77 0.443 -.3835842 .1678551
securitylaw | .3757814 .2143941 1.75 0.080 -.0444233 .7959861
northaid | .9259885 .2543168 3.64 0.000 .4275368 1.42444
diplosecu | .3236994 .1933323 1.67 0.094 -.055225 .7026237
initialvote | -1.602286 .2539444 -6.31 0.000 -2.100008 -1.104564
addedtroops | -1.237338 .2566488 -4.82 0.000 -1.740361 -.734316
_cons | -3.624946 1.054334 -3.44 0.001 -5.691402 -1.558489
--------------------------------------------------------------------------------------

Dependent variable is vote (nay: 0, yay:1)
Conservative (non-conservative party member:0, conservative party member:1)
compete (non-competitive district:0, competitive district:1)

My hypothesis is that conservative party members are less likely vote for yay in a competitive district.

Here is the result of margin command, margin conservative#compete


Predictive margins Number of obs = 446
Model VCE : Robust

Expression : Pr(vote), predict()

--------------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
---------------------+----------------------------------------------------------------
conservative#compete |
0 0 | .746918 .0430746 17.34 0.000 .6624933 .8313426
0 1 | .8395636 .0271827 30.89 0.000 .7862865 .8928407
1 0 | .8323398 .0315363 26.39 0.000 .7705299 .8941497
1 1 | .8018074 .0372661 21.52 0.000 .7287672 .8748476
--------------------------------------------------------------------------------------


Here is the average marginal effect, margins, dydx(conservative) at(compete=(0 1)) vsquish

Average marginal effects Number of obs = 446
Model VCE : Robust

Expression : Pr(vote), predict()
dy/dx w.r.t. : 1.conservative
1._at : compete = 0
2._at : compete = 1

------------------------------------------------------------------------------
| Delta-method
| dy/dx Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.conserva~e |
_at |
1 | .0854218 .0516731 1.65 0.098 -.0158556 .1866993
2 | -.0377562 .050136 -0.75 0.451 -.136021 .0605085
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

As you can see, the difference found in the second average marginal effect, -0.377562, is statistically insignificant. In this case, should I abandon my hypothesis that conservative party members are less likely to vote for yay in a competitive district? I am confused about the difference between significance in predictive margins (first margin table) and insignificance in average marginal effect (second margin table).

It would be appreciated if I can answer some comments on my analysis.

Thank you

[Clyde Schechter]

I don't have time to write a longer explanation but:

1. The statistical significance of predictive margins is usually of no importance in any analysis. You could contrive one in which they mattered, but it would be an arbitrary exercise. In real world data analysis, the statistical significance of predictive margins is meaningless and should be ignored.

2. To the extent that statistical significance is a useful concept, marginal effects are an area where it can be constructively applied. In my practice, I never rely on statistical significance, largely because I reject the framework of hypothesis testing for most applications anyway. But if you believe that hypothesis testing and statistical significance as a way of testing hypotheses are applicable to your work, the statistical significance of your marginal effect is a key result. Whether you should reject your hypothesis in the face of lack of statistical significance, however, would still depend on issues of statistical power, validity of your model, reliability of your measures, and the other usual considerations.

[Clyde Schechter]

Back from my meeting. I have more time to elaborate.

Re #1. The predictive margins, after probit, are the predicted probabilities of a yay vote among each of the four combinations of conservative party member and competitive district. For example, among the non-conservative party members in non-competitive districts, the predicted probability of a yay vote, adjusted for everything in the model, is .75 (to two decimal places), with a 95% CI of .66 to .83). The p-value in that row is a test of the null hypothesis that the predicted probability is zero. But of course, it is not among your research goals to test the null hypothesis that the non-conservatives in non-competitive districts have a zero probability of voting yay. That isn't even a straw man hypothesis--it's just ridiculous. So the p-values in that table are meaningless: they test hypotheses that nobody would ever be interested in testing, and should be ignored. The reason, I think, they are there, is that there are occasional circumstances where a zero predicted outcome (not a probability, to be sure, but say the outcome of a linear regression) might be a plausible null hypothesis to test. Such circumstances are unusual and, as I said in #2, mostly artificial. Anyway, when in doubt, ignore the p-values in a predictive margins table as they are almost always useless.

Re #2. These are marginal effects. They are the estimates of the expected difference in probability of a yay vote from a conservative compared to a non-conservative, and the first row gives this difference in the non-competitive districts, whereas the second gives this difference in the competitive districts. While in my own research practice I generally do not like dichotomizing results into "difference vs no difference" or testing null hypotheses that I think of as straw men, in this instance, while I think a null hypothesis that conservatives and non-conservatives do not vote at all differently is a bit far-fetched, I wouldn't go so far as to call it preposterous. And certainly many, probably most, researchers would be quite comfortable testing that null hypothesis. (I would just report the conservative vs. non-conservative differences themselves along with their 95% CI's and then present some commentary on whether I thought these were large or small relative to other effects in the model. But, I digress.) So assuming you want to do null hypothesis significance testing, the p-values in this table are the ones to look at. In this case, the p-values are greater than the conventional .05 significance level, and for the competitive districts the p-value isn't even in the ballpark of .05. So you would not reject your null hypothesis.

Now, whenever you have a non-significant result, you have to consider the possible explanations. The null hypothesis being true is just one of them. Other possibilities include lack of statistical power. A sample size of 179 clusters, while not tiny, is only midling size. Ideally, when you planned this study you did power calculations and can approach this question. Other issues to consider are whether your sample is perhaps biased in some way, whether the probit model you have constructed is a reasonable specification of the data generating process, and whether your measurements are sufficiently precise. Regarding the latter, I imagine that yay or nay votes are drawn from voting records that are more or less unimpeachable. And if conservative refers specifically to membership in one political party and is assessed from voting rolls, then it, too should be fine. But classifying a district as competitive or not may be less cut and dried. It's not clear to me what most of your other variables are, but you should think about each of them as a potential source of noise in the model.

[Joonho Kim]

Dear Clyde Schechter,

Thank you you for your kind response. Yes, I also read a recent debate on the "too much obsession about P value." As a beginner of empirical analysis, I would like to lean more about validity of research in general. However, as a learner of statistical analysis I am curious why there is a P value difference between predictive margin and average marginal effect. Before I post my question, I read your past posting on the interpretation on the difference of marginal effect
(https://www.statalist.org/forums/for...and-difference).

In your past posting, you indicated that the difference in margin table can be found in the second margin table (dy / dx). Of course, in the past posting, second average marginal effect's P >[z] was 0.000.

In my case, difference from 0 (non compete) to 1 (compete) in the first margin table is 0.03053... with P > [z] is 0.000 (95% CI .7287672 - .8748476). And, the difference between those is shown in the first line of the second margins table: 0.0377... but P > [z] is 0.451 (95% CI -.136021 -.0605085). If so, does it mean that other factors affect the calculation of difference of marginal effect in the second dy /dx table? I guess there might be possible that insufficient number of observation, multicollinearity, and etc. affected in this process, which leads to P> [z] 0.451. Does it valid analysis of my result?

Thank you
Hwalmin

[Joonho Kim]

Oh, you have gave detailed comment. I have to read. Thank you.