Conflicting results with margins command

[John Larsson]

Hello,
I am doing a logistic regression with a binary indicator. The regression results show a statistically significant result (p

[John Larsson]

The above message appeared to be truncated. This a logistic regression with a single binary predictor. My regression results show a statistically significant result and executing "margins.x" also shows a confidence interval that does not include zero. However, "marginsiI.x" shows two confidence intervals that just overlap. How should one interpret these apparently conflicting results?

[Clyde Schechter]

First, the output of -margins- consists of different statistics than the output of -logistic-.

With -logistic- you are getting an odds ratio, and the significance test is on the null hypothesis that the odds ratio = 1. With -margins- you are getting predicted probabilities of the outcome in each category of x, and the significance tests are of the hypothesis that the predicted probability is zero. Since it is rarely if ever reasonable to even entertain a null hypothesis that a predicted probability is zero, the p-values in this output are not really useful and should just be ignored. Similarly the confidence intervals may extend below zero, but a negative probability of outcome is also not possible (nor, at the other extreme, above 1.) So these confidence intervals are also not terribly useful if they include or come close to including 0 or 1. (They are based on normal-theory approximations that break down in these situations.)

Next, even assuming, however, that your confidence intervals are not close to zero or one and can be taken at face value, you cannot make inferences about a significant difference in outcome probability between the levels of x based on them. It is always true that when the confidence intervals do not overlap, the difference is significant; but the converse is not necessarily true. That's because the widths of the separate confidence intervals are determined by the separate standard errors of the outcome in each group, whereas the test of significant differences between groups is based on the standard error of the difference, which can be smaller or larger. This is a widely discussed phenomenon, and if you Google "confidence interval overlap significant" you will find a broad array of relevant papers ranging from very elementary to fairly complicated that you can select from to read more about this.

[John Larsson]

Thanks for the clarification. I have a follow up on a related issue.

The original logistic regression is done under the svy suite, with strata/cluster and Taylor series linearization for the standard errors. In following up with the margins command thereafter, there is no option for vce(linearized). Is the appropriate option vce(unconditional)?

[Clyde Schechter]

From page 1360, in the -margins- section of the [R] manual:

vce(unconditional) specifies that the covariates that are not fixed be treated in a way that
accounts for their having been sampled. The VCE is estimated using the linearization method.
This method allows for heteroskedasticity or other violations of distributional assumptions
and allows for correlation among the observations in the same manner as vce(robust) and
vce(cluster : : : ), which may have been specified with the estimation command. This method
also accounts for complex survey designs if the data are svyset. See Obtaining margins with
survey data and representative samples. When you use complex survey data, this method
requires that the linearized variance estimation method be used for the model. See [SVY] svy
postestimation for an example of margins with replication-based methods.
nose suppresses calculation of the VCE and standard errors. See Requirements for model