Wald test

[Ioannis Siskos]

Dear Stata users,
Apologies if this has been answered before, but I haven't been able to find an answer with common keywords.

I have two count models with the same configuration (only dependent variables are different) as follows:

Model A: nbreg depvar_a x1 x2 ... x7
Model B nbreg depvar_b x1 x2 ...x7

I am performing an equality test on the individual coefficients, as
test [modelA=modelB], common mtest

In one of the tests e.g for x4, the p-value is <0.05 which indicates that I cannot reject the null hypothesis that the coefficients are equal.
The x4 coefficient of ModelA is significant whereas the x4 coefficient of ModelB is not. Does this affect the equality test conclusion?
Thank you in advance for your kind help.

[Clyde Schechter]

The x4 coefficient of ModelA is significant whereas the x4 coefficient of ModelB is not. Does this affect the equality test conclusion?

No it does not. The difference between statistically significant and not statistically significant is not, itself, statistically significant!!

I know, it's counterintuitive. So let's explain it. The problem is that statistical significance doesn't mean what many people think it means. A non-statistically significant result does not mean that there is no effect, or that the effect (or coefficient) is zero. What it means is that the precision with which the data enabled you to estimate the effect, and the magnitude of that estimate, are such that you cannot with adequate confidence exclude the possibility that the effect is zero. It might or might not be. So, I could have two coefficients that have exactly the same value, let's say both come out exactly 1.0. Now one of them might be estimated very precisely, say with a 95% CI from 0.99 to 1.01, because the data make that possible (say because this dependent variable is easy to measure with a high degree of accuracy and reliability, and I have a reasonably large sample size). The other coefficient, also 1.0, might be estimated rather imprecisely, say with a 95% CI from (-1 to + 3) because the outcome variable this time is noisy, or maybe this data set is just really small. Clearly no statistical test will tell you to reject the hypothesis that these coefficients are equal. But the first is "statistically significant" and the second one is not.

I advise my students to always first study all of the output other than the p-values. Once they have fully understood the meanings of their coefficients, predicted outcomes, marginal effects, whatever, and their associated confidence intervals, then, if they have nothing better to do with their time, they can turn their attention to the significance tests. Once you understand what everything else is telling you, it usually becomes clear that the p-values have nothing additional to offer, except, as in this situation, possibly some confusion. Yes, there are occasional contexts where the p-value is in fact the key result of the analysis. But in real world research such situations are actually uncommon. Focus on your coefficients, predicted outcomes, etc., and the precision with which you have estimated them, and you will usually find that there is nothing useful to be gained by talking about "statistical significance."

[Ioannis Siskos]

Dear Clyde,
Many thanks for your comprehensive answer. It was the kind of confirmation I was looking for.
Ioannis

[Richard Williams]

Another related example: In group comparisons, it may be tempting to say that a variable has an effect on whites but not on blacks, because the effect for whites is statistically significant but it isn't for blacks. But, that can be highly deceptive; the differences might just reflect the fact that there are far more whites than blacks in the sample, hence it is easier to get statistically significant effects for whites. A formal statistical test may show that the estimated effects for whites and blacks do not significantly differ.

[Ioannis Siskos]

Just to make sure that I got it right: Let's say that one variable is significant for whites, but not for blacks. Let's also say that the equality of coefficients test gives a p value<0.05 i.e we cannot reject the hypothesis that the coefficient estimates of this variable are equal both for whites and blacks.
The conclusion is that we cannot support the argument that this variable has a stronger (or weaker - depending on the sign of the coeff) effect on whites compared to blacks, due to the Wald test. Is my understanding right?
Thank you in advance.

[Ioannis Siskos]

Also, if we have the opposite:
Equality test rejects the hypothesis that some coefficients are equal for blacks and whites. Lets say that among these, one coefficient is significant for whites, but not for blacks. I assume that it is fair to argue that this variable has a different effect to whites compared to blacks.
However what happens to the strength of the effect? The coeff is significant for whites, but for blacks it may not be significant because of the smaller sample. Is there any argument to be made in this regard (i.e the strength of the effect on either)
Thanks in advance

[Clyde Schechter]

Re #5, yes that is correct.

Re #6 assuming that you want to rely on hypothesis tests of the equality of coefficients, if the equality test rejects the hypothesis that the coefficients are equal for the blacks and white populations, then you would conclude that the effects of those variables are different among blacks and whites. This would be true whether the black and white coefficients are individually statistically significant or not. The individual statistical significance of the things being compared in an equality test is irrelevant to whether or not they can be declared different after that equality test.


You do not describe the subject matter or context of your question, and it is possible that if I knew the details I would advise differently. But for most situations, I would say that it is best to ignore statistical significance and p-values and focus instead on the magnitudes of the effects you are finding and the confidence intervals around those magnitudes. It is also important to interpret those effect sizes in terms of their practical real-world importance. Once you understand what the model is telling you about those things, the p-values will have little or nothing to add to the story. Well, except, as here, confusion.

[Eric de Souza]

There is a nice example in the paper by Gelman and Stern (I don't remember the exact reference) that illustrates the point made by Clyde in #2:

Two teams of researchers each study independently of each other the effects of two different treatments on individuals, each team analysing one treatment. A regression analysis of the effect of the first treatment yields a coefficient of 25 with a standard error of 10. A similar analysis of the effect of the second treatment yields a coefficient of 10 with a standard error of 10. On comparing their results, the researchers conclude that since the effect of the first treatment is significantly different from zero (25/10 = 2.5), whereas the effect of the second treatment is not (10/10 = 1), only the first treatment should be implemented. Do you agree?
(Hint: keep in mind that the variance of the sum of two random variables is equal to the sum of the variances of the two random variables plus twice the covariance between the two random variables; and that the variance of the difference between two random variables is equal to the sum of the variances of the two random variables minus twice the covariance between the two random variables.)
Added on edit: the hint is not in their paper, nor is the example given as an exercise.