Margins p-value vs model coef p-value difference?

[Brad Anderson]

I’m estimating a mixed effects Poisson regression model for an RCT and using the margins command to estimate the marginal effect of treatment as a mean rate. Using robust standard errors the p-value for the model coefficient is not equal to the p-value for the marginal effect as estimated by the margins command with default delta standard errors (they are close). This inconsistency persists if I use the vce(unconditional) option with of the margins command. Ad If I estimate the model with oim standard errors, the p-value for the marginal effect of treatment is again slightly different than the p-value for the model coefficient; it’s also different than the p-value for the marginal effect estimated when using vce(robust) for the model. I don’t understand what’s generating the inconsistency in z-statistics and p-values. Thanks.

[Clyde Schechter]

What's generating the "inconsistency" is that these are two different things and they are not supposed to be the same. The only error here is your expectation that they will be the same.

The coefficient in the Poisson regression is an estimate of the logarithm of the ratio of the expected outcome associated with the treated state vs the untreated state, estimated under the constraint that this ratio applies uniformly for al values of all study variables. The marginal effect, however, estimates the expected difference in the outcome variable itself between the treated and untreated, averaged over the observed joint distribution of all the model variables. So the two statistics are in different metrics, the second is highly dependent on the joint distribution of all model variables whereas the first is not, and there is also a non-linear (logarithmic) transformation between them. So there is no reason to think they will have the same p-value.

[Fei Wang]

Clyde accurately distinguished the original coefficient and marginal effect. I'll show more math details and give some examples.

Assume the original coefficient of x in the population is beta, and it's estimator is b. Under the null hypothesis of beta = 0, the z-statistic = b / se(b), asymptotically follows the standard normal distribution based on which p-values are calculated.

For commonly-used non-linear models, like logit, probit, and poisson, the marginal effect of x on the expected y is essentially a function of b. For example, in poisson, the expected y conditional on x = exp(b*x), and the marginal effect of x on expected y = d[exp(b*x)] / dx = exp(b*x)*b, which is a non-linear function of b. Of course, it's also a function of other covariates and estimators -- let's ignore them for now to simplify the expression.

Assume the non-linear function mentioned above is h(b) -- an estimator of the marginal effect, and its population value is h(beta) -- the true marginal effect in the population. The delta method basically says, under the null hypothesis of h(beta) = 0, the z-statistic, shown in the marginal effect table, is equal to h(b) / [se(b)*h'(beta0)], where h'(.) is the partial derivative of h(.) with respect to beta, and beta0 is a value of beta such that h(beta0) = 0. Under the null, this z-statistic also asymptotically follows the standard normal distribution and we can derive p-values from there.

The inconsistency of z-statistics and p-values between original coefficient and marginal effect is essentially from the fact that, "b / se(b)" in general is NOT identical to "h(b) / [se(b)*h'(beta0)]". When are they close to or far from each other? The Taylor series of h(b) at the value of beta0 can be expressed as: h(b) = h(beta0) + h'(beta0)*(b - beta0) + higher order terms = h'(beta0)*(b - beta0) + higher order terms. At first, we ignore the higher order terms. Then, the z-statistic for marginal effect, h(b) / [se(b)*h'(beta0)], becomes (b - beta0) / se(b). So we can get the first necessary condition for the similarity of the two z-statistics: Condition 1. beta0 = 0, where beta0 is a value of beta such that h(beta0) = 0. As shown above, for poisson, h(beta) = exp(beta*x)*beta, and beta = 0 exactly makes h(beta) = 0 as well. Condition 1 satisfies. Actually, this condition also holds for logit, probit, and many other common models.

Then why is there still inconsistency? Because there are higher order terms in the Taylor series. Higher order terms = h''(beta0) * (b - beta0)^2 / 2! + h'''(beta0) * (b - beta0)^3 / 3! + ... = h''(0) * b^2 / 2! + h'''(0) * b^3 / 3! + ... The second equality holds if Condition 1 holds. Smaller the higher order terms, closer the two z-statistics will be. -- If b is exactly 0, then the higher order terms would truly disappear. So we have a second necessary condition: Condition 2. b is sufficiently close to 0.

A simple conclusion: When conditions 1 and 2 both hold, the two z-statistics and p-values would be close. In an extreme case, when b = 0 exactly, the two z-statistics would be identical. As condition 1 usually holds, then condition 2 is the key. Below is an example.

.  webuse dollhill3, clear
(Doll and Hill (1966))

.  poisson deaths smokes i.agecat, exposure(pyears)

Iteration 0:   log likelihood = -33.823284  
Iteration 1:   log likelihood = -33.600471  
Iteration 2:   log likelihood = -33.600153  
Iteration 3:   log likelihood = -33.600153  

Poisson regression                              Number of obs     =         10
                                                LR chi2(5)        =     922.93
                                                Prob > chi2       =     0.0000
Log likelihood = -33.600153                     Pseudo R2         =     0.9321

------------------------------------------------------------------------------
      deaths |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      smokes |   .3545356   .1073741     3.30   0.001     .1440862     .564985
             |
      agecat |
      45-54  |   1.484007   .1951034     7.61   0.000     1.101611    1.866403
      55-64  |   2.627505   .1837273    14.30   0.000     2.267406    2.987604
      65-74  |   3.350493   .1847992    18.13   0.000     2.988293    3.712693
      75-84  |   3.700096   .1922195    19.25   0.000     3.323353     4.07684
             |
       _cons |  -7.919326   .1917618   -41.30   0.000    -8.295172   -7.543479
  ln(pyears) |          1  (exposure)
------------------------------------------------------------------------------

.  margins, dydx(_all)

Average marginal effects                        Number of obs     =         10
Model VCE    : OIM

Expression   : Predicted number of events, predict()
dy/dx w.r.t. : smokes 2.agecat 3.agecat 4.agecat 5.agecat

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      smokes |   25.91656   7.907363     3.28   0.001     10.41841     41.4147
             |
      agecat |
      45-54  |   30.01352   3.908222     7.68   0.000     22.35355     37.6735
      55-64  |   112.9864   8.121876    13.91   0.000     97.06782     128.905
      65-74  |   242.1508    17.2516    14.04   0.000     208.3383    275.9633
      75-84  |   347.1751   30.90388    11.23   0.000     286.6046    407.7456
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

As the original coefficient of "smokes" (b) is relatively small, the z-statistic and p-value from the margins are close to those in the original table. But as b grows up (like coefficients for "agecat"), the two z-statistics are more and more far away from each other.