testing moderation in a panel regression model, but R-squared barely changed

[Jimmy Wang]

I am comparing two nested fixed-effects models, which are shown as follows

y_it= b₀ + b₁ x_it + b₂ (x_it)² +..., (1)

and

y_it= b₀ + b₁ x_it + b₂ (x_it)² + b₃ z_itx1_it + b₄ z_it(x1_it)² + b₅ z_it+... (2)

where z_it is the moderation variable of interest and x has been mean-centered.

Question 1: my estimates of b₃ and b₄ in (2) are significant. However, the within R-squared value only increased marginally in Model (2), and the increase is not statistically significant. How should I interpret the outcome, as some references seem to suggest that a significant increase in R-squared is important in a moderation analysis?

Question 2: I observed that the b₁ and b₂ coefficients of Model (1) are quite different from those in Model (2). I thought this is not indicative of a problem since these are two different models. Yet I got comments from a colleague, who said that, given that the R-squared of these two models are not significantly different , this result shows that the significance of b₃ and b₄ in Model (2) are likely to be falsely driven by multicollinearity. But later I computed vif for Model (2), and the scores didn't suggest multicollinearity problems with the augmented variables in model (2). Values of the standard errors also look quite reasonable. Should I worry about multicollinearity in this case?

Thank you all in advance. Any comments and thoughts are appreciated.

[Clyde Schechter]

Question 1: my estimates of b₃ and b₄ in (2) are significant.

That means nothing. Assuming you want to stay within the null hypothesis significance testing framework (which I do not recommend, but that's another discussion for another day) you need to do a joint test of b3 and b4. The separate significance of each tells you nothing useful.

How should I interpret the outcome, as some references seem to suggest that a significant increase in R-squared is important in a moderation analysis?

Well, what was the R² in the non-interaction model? If it was already high, there may not be much room for it to increase. Also, were you careful to assure that model (1) was estimated on the same set of observations as model (2). Model (2) has additional variables (the z_it and the interaction terms) so missing data on those variables could cause the estimation sample to shrink, and then you do not have an apples-to-apples comparison of the two models. And if your sample is large, you can have statistically significant effects that are too small to noticeably budge R² in any case. Whether such effects are meaningful is a different question and certainly open to discussion on a case-by-case basis, but since you are, for better or worse, talking about statistical significance, R²is not a metric of interest. Without seeing the actual outputs it's really not possible to say anything more specific here.

I thought this is not indicative of a problem since these are two different models.

It isn't. Your thinking on this is correct. In particular, in model (1) b1 and b2 are estimates of the coefficients of a one-size fits all quadratic relationship between x and y. But in model (2) there is no overall quadratic relationship between x and y: there is a different quadratic relationship for each value of z, and the coefficients b1 and b2 here only represent the coefficients that apply when z = 0. (And if z = 0 does not occur in the data then b1 and b2 have no meaning at all.)

Yet I got comments from a colleague, who said that, given that the R-squared of these two models are not significantly different , this result shows that the significance of b₃ and b₄ in Model (2) are likely to be falsely driven by multicollinearity.

With all due respect, either you misunderstood your colleague's comments or that colleague does not know what he/she is talking about. One possible source of misunderstanding: you could look at an F-test of the difference in R² between model 2 and a model that omitted the zx and zx² interaction terms (but not z itself) as one way to jointly test the significance of b3 and b4. But that is not the same as contrasting the R² in model (2) with that of model (1)--the R² difference itself would be different, as would the degrees of freedom for the test. Even conditioning on that misunderstanding, multicollinearity would not falsely drive significance of the findings--if anything it would work in the opposite direction.

But later I computed vif for Model (2), and the scores didn't suggest multicollinearity problems with the augmented variables in model (2). Values of the standard errors also look quite reasonable. Should I worry about multicollinearity in this case?

You've already wasted more time on multicolinearity than you should have by doing a useless vif. You should never worry about multicollinearity. It's a bogus issue. See Arthur Goldberger's A Course in Econometrics for an entertaining and spot-on critique of the very concept.

[Jimmy Wang]

Dear Clyde,

Many wholehearted thanks to your detailed comments! Really helpful!

I have a few more questions that I hope you can shed light on.

That means nothing. Assuming you want to stay within the null hypothesis significance testing framework (which I do not recommend, but that's another discussion for another day) you need to do a joint test of b3 and b4. The separate significance of each tells you nothing useful.


Well, what was the R² in the non-interaction model? If it was already high, there may not be much room for it to increase. Also, were you careful to assure that model (1) was estimated on the same set of observations as model (2). Model (2) has additional variables (the z_it and the interaction terms) so missing data on those variables could cause the estimation sample to shrink, and then you do not have an apples-to-apples comparison of the two models. And if your sample is large, you can have statistically significant effects that are too small to noticeably budge R² in any case. Whether such effects are meaningful is a different question and certainly open to discussion on a case-by-case basis, but since you are, for better or worse, talking about statistical significance, R²is not a metric of interest. Without seeing the actual outputs it's really not possible to say anything more specific here.

The R² of these models (including the test model with z but without its interactions) are around 0.15. The panel is rather short and has around 400 people and average 4 observations per individual. The sample size doesn't change with or without z in the model.

It isn't. Your thinking on this is correct. In particular, in model (1) b1 and b2 are estimates of the coefficients of a one-size fits all quadratic relationship between x and y. But in model (2) there is no overall quadratic relationship between x and y: there is a different quadratic relationship for each value of z, and the coefficients b1 and b2 here only represent the coefficients that apply when z = 0. (And if z = 0 does not occur in the data then b1 and b2 have no meaning at all.)

With all due respect, either you misunderstood your colleague's comments or that colleague does not know what he/she is talking about. One possible source of misunderstanding: you could look at an F-test of the difference in R² between model 2 and a model that omitted the zx and zx² interaction terms (but not z itself) as one way to jointly test the significance of b3 and b4.

Just now I followed your suggestion and ran the test model without the two interactions terms with z. The change in R² was still very small (<1%) compared with model (2) and presumably insignificant as well. However, I tried a joint test for b3 and b4 and the F test statistic is significant at 5%.

In your opinion, which test can be considered more conclusive about the moderation effect? Your opening comments seem to be not positive about the joint test of the coefficients in this context.

But that is not the same as contrasting the R² in model (2) with that of model (1)--the R² difference itself would be different, as would the degrees of freedom for the test. Even conditioning on that misunderstanding, multicollinearity would not falsely drive significance of the findings--if anything it would work in the opposite direction.


You've already wasted more time on multicolinearity than you should have by doing a useless vif. You should never worry about multicollinearity. It's a bogus issue. See Arthur Goldberger's A Course in Econometrics for an entertaining and spot-on critique of the very concept.

Thank you for the reference. I will definitely read thru this book.

[Clyde Schechter]

In your opinion, which test can be considered more conclusive about the moderation effect? Your opening comments seem to be not positive about the joint test of the coefficients in this context.

If you are working in the context of null hypothesis significance testing, the joint test of b3 and b4 is definitive.

My reservations about it are just an instance of my general reservations about using significance testing at all in this context. I generally tend to think about this kind of work as predictive modeling, and so I am more inclined to look at the extent to which the interaction terms actually change the predicted outcomes or predicted marginal effects over the range of values of the x's and ignore p-values. But the pros and cons of p-values vs predictive modeling is a long, philosophical one, for which I do not have time today.

[Jimmy Wang]

Thank you very much for the explanation!