margins - are p-values ever useful?

[R Allan Reese]

Apologies if this is a faq; search found 500 threads about margins and I'm not reading that many! My general point is that margins fits predicted values at levels of factors but appears to then give the "significance" testing each value against zero. This seems to me either pointless or nonsensical. Maybe if you standardized the response variable before fitting you would have a level that was canonically zero, so why test that? The Stata online video about margins predicts blood pressure for male and female; I'm rather glad my BP is "significantly" above zero. Would nopvalues be the more sensible default for the output?

[Rich Goldstein]

in the situation you appear to be discussing, I agree; however, there are other situations, esp. those using either the "contrast" or the "pwcompare" option, when p-value are as meaningful as p-values ever are; further, the p-values may be of interest when using either the "at()" or "within" options; it may be possible to set up the command so that by default the p-values are not shown but they are automatically shown when certain options are chosen; whether this is actually possible, and, more important, whether it is worth the trouble are, however, other questions; I think I prefer StataCorp to spend their time on other things and I can just live with the current situation

[Richard Williams]

Testing whether something is different from 0 can be pointless but testing, say, where the estimated value for men is different from the value for women may be good.

If you don't care about p values, nose will get rid of them and can substantially speed execution.

[R Allan Reese]

Maybe I need to point out I've fitted a model (in this case a glm) which gives the parameters with p-values for each effect. I use margins to apply the model and give the group means and CIs. In the PDF documentation there are numerous examples showing these (as I'm now convinced) nonsense p-values, and no discussion until this: (p1416 - Manipulability of tests - I've bolded)
"We would prefer if the test against zero produced by margins, df(.) was equal to the
test produced by margins, expression(predict(xb)^2). But alas, they produce different results.
The first produces z = 12.93, and the second produces z = 12.57.
The difference is not much in our example, but behind the scenes, we worked to make it small.
We subtracted 65 from y so that the experiment would be for a case where it might be reasonable that
you would be testing against 0. One does not typically test whether the mean income in the United
States is zero or whether the mean blood pressure of live patients is zero. Had we left y as it was
originally, we would have obtained z = 190 and z = 96. We did not want to show that comparison
to you first because the mean of y is so far from 0 that you probably would never be testing it. The
corresponding difference in is tiny.

[Richard Williams]

There are lots of p-values and hypotheses you are not interested in. For example, you are rarely interested in whether the intercept = 0. Or whether the mean income is 0.

But, it is easy enough to test other hypotheses of interest. Test whether the mean income of two countries is equal. If previous research showed the mean score on a variable 5 years ago was 3.7, test whether it is 3.7 today. If you have 5 religions, see whether their mean values differ. Test whether the effects of X1 and X2 are equal.

Yes, the hypothesis that something = 0 is often of no interest. But, you can test hypotheses that are of interest by using test, lincom, pwcompare, etc. Stata doesn't know what the interesting and relevant hypotheses are, but you can tell it what is interesting and worth testing.

In the case of margins, with a categorical independent variable, you may rarely care if the adjusted predictions for any given group is 0 (but that is the test margins presents). But, you will often be interested in the marginal effects (which you can get with dydx, or with options like pwcompare) are significant, e.g. are Catholic scores different from Protestant scores?

So, I don't think this is a valid criticism of margins (or most other commands for that matter). It tests what it tests, and if that isn't what you are interested in it is easy enough to test what you do want.

[Leonardo Guizzetti]

Maybe I need to point out I've fitted a model (in this case a glm) which gives the parameters with p-values for each effect. I use margins to apply the model and give the group means and CIs.

As a trivial example, you could have fit your model to give you group means, instead of the more common reference-coded contrasts.

glm y ibn.group , nocons // you might have fit this ...
glm y i.group            // ... instead of this

Then you would need -margins- (or -test- or -lincom-) to give you estimate of of the between-group contrast. this demonstrates the need for those commands, including p-values, if that's what you are interested in.

So, I don't think this is a valid criticism of margins (or most other commands for that matter). It tests what it tests, and if that isn't what you are interested in it is easy enough to test what you do want.

I agree. There are many valid criticisms one may lob at p-values, but this is not one of them. Once a software goes as far as computing the parameter and its variance, then it's not much more work to derive the p-value and show the user. I'd rather have all output reported and then I can choose what is relevant to interpret and report, rather than have to do extra work to try to get these values out later.

[R Allan Reese]

Thanks for comment Leonardo but don't think that helps. It's a Poisson model so the effects are not the means but have to be back-transformed. I think the point has been raised and discussed. With all the help from the list, I believe my best solution is:
glm y i.factor . fam(pois)
margins factor, pwcomp(group pveffects) // This shows the group mean counts, differences and p-values for paired differences. BIG THANKS TO ALL.

[Leonardo Guizzetti]

Thanks for comment Leonardo but don't think that helps. It's a Poisson model so the effects are not the means but have to be back-transformed.

We could not have known the details you did not share with us, such as the context or specific commands used. Nevertheless, my generic exam still works to illustrate a trivial example where p-values from margins may be useful.

sysuse auto
glm price i.foreign , fam(poisson) link(log) nolog
margins i.foreign
margins, dydx(foreign)
glm price ibn.foreign , nocons fam(poisson) link(log) // identical results with different model specification
margins i.foreign
margins, dydx(foreign)