effect sizes and multilevel models

[CJ Wilson]

Hi all,

I am running the below mixed effects model.

 mixed outcome indeps || _all: R.rater || _all: R.target || _all: R.ph

A colleague questioned whether effect sizes should be reported, since a particular journal likes them (however, I think the journal may tend to have a lot of analysis using ANOVA where I think it is more important to report). I’m trying to determine whether or not to report these. For my model, all of the variables relate to the research questions, and also coefficients are already being reported in addition to p-values. The examples I’ve seen of people calculating effect sizes seem to relate to one key variable of interest, for example the effects of an intervention in an ANOVA study, whereas all of the variables here relate to the research questions.

My question:

• Under what circumstances is it appropriate or necessary to report effect sizes in mixed effects models in addition to reporting coefficients and p-values? Is this really just a good idea these days that should be done for every model and all variables in the model?

My sense is that the coefficients themselves are already a measure of effect size so should be sufficient, unless there is a specific reason to calculate them.

Sorry if this post is too broad and vague. Thanks in advance for any comments.

[Phil Bromiley]

You didn't get a quick answer. You'll increase your chances of a useful answer by following the FAQ on asking questions - provide Stata code in code delimiters, readable Stata output, and sample data using dataex.

This is partially a matter of taste and norms that vary by discipline. Parameters do not give a good indication of effect size (since the variance of the x's varies). For non-linear models, parameters can be very hard to interpret - this is why we use the margins command so much. As Clyde has noted on this listserve, normalizing the x's may not be desirable if the x's have a meaningful scale. Personally, in regression type models, I want to know the difference in predicted y for a given change in each x of direct research interest. This would not necessarily include all the control variables.

That said, in a paper with a ton of results, this may not be feasible for all the x's but I still would like some idea of effect size.

[daniel klein]

I agree with Phil in that reporting effect size measures seems mostly to be a matter of taste and tradition in different disciplines. To me, effect size measures (and standardizing, in general) are often not useful and might sometimes be misleading. I have not yet really understood how effect size measures are supposed to add anythig to our understanding of the (ideally causal) relationships between variables. Here is a very simple example.

Suppose there is an outcome, y, and there are two factors, x, and z, which cause this outcome (yet do not determine it, completely). Suppose further, that both x and z are measured (without error) on the same scale and that both x and z have the exact same effect on y. Suppose, that this effect is 2.5, that is, a one unit increase in x or z causes a 2.5 unit increase in y. Further, suppose that there are unmeasured causes of y, say e, that are uncorrelated with both x and z. However, x and z might be correlated. Now, assume that the standard deviation of z is twice that of x. Here is the setup

clear
set seed 42
matrix C = 1, 0.3, 0\ 0.3, 1, 0\ 0, 0, 1
corr2data x z e , n(1000) sd(1, 2, 8) corr(C)
summarize
correlate x z e
generate y = 2.5*x+2.5*z + e

The output

(output omitted)
. summarize

    Variable |        Obs        Mean    Std. Dev.       Min        Max
-------------+---------------------------------------------------------
           x |      1,000    1.59e-08           1  -3.591861   3.448143
           z |      1,000    2.86e-08           2   -5.79394     6.8119
           e |      1,000   -4.23e-08           8  -24.20594   25.14735

. correlate x z e
(obs=1,000)

             |        x        z        e
-------------+---------------------------
           x |   1.0000
           z |   0.3000   1.0000
           e |  -0.0000   0.0000   1.0000
(output omitted)

Now, since we have created the perfect world, where all the assumptions of the linear regression model are satisfied, we can actually estimate the (causal) effect of both x and z on y using a simple linear regression model.

. regress y x z

      Source |       SS           df       MS      Number of obs   =     1,000
-------------+----------------------------------   F(2, 997)       =    301.83
       Model |  38711.2499         2   19355.625   Prob > F        =    0.0000
    Residual |  63936.0001       997  64.1283852   R-squared       =    0.3771
-------------+----------------------------------   Adj R-squared   =    0.3759
       Total |   102647.25       999      102.75   Root MSE        =     8.008

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
           x |        2.5   .2655961     9.41   0.000     1.978808    3.021192
           z |        2.5   .1327981    18.83   0.000     2.239404    2.760596
       _cons |  -4.93e-08   .2532358    -0.00   1.000    -.4969364    .4969363
------------------------------------------------------------------------------

As expected, we recover the true parameters for both x and z. We can also readily interpret the results. Looking at effect sizes, however,

. estat esize

Effect sizes for linear models

-------------------------------------------------------------------
             Source |   Eta-Squared     df     [95% Conf. Interval]
--------------------+----------------------------------------------
              Model |    .377129         2     .3322452    .4176864
                    |
                  x |   .0816143         1     .0520662     .115357
                  z |   .2622478         1     .2181312    .3052079
-------------------------------------------------------------------

would give the impression that z is "practically" more significant than x. I argue that this result is much more likely to be misunderstood/misleading than the (unstandardized) regression coefficients. And, since this happens in the best of all worlds, I wonder in which situations such measures are really useful. At best, it seems, they do no harm.

Best
Daniel

[Weiwen Ng]

An effect size measure related to eta-squared is f-squared, and you can calculate f-squared in a mixed model, instructions courtesy of UCLA (it appears that a SAS statistician suggested the technique first, and UCLA adapted it).

In response to Daniel: perhaps I'm reading your example wrong, but it looks like you generated data where a one-unit change in x and z are associated with a 2.5-unit change in y. x and z both have mean 0. However, x has a SD of 1, and z has an SD of 2. So, if z varies quite a bit more than x, it does make sense to me that z should account for more variability in y, which eta-squared and f-squared would both capture. Am I missing something?

[daniel klein]

Weiwen, thanks for joining in. You have perfectly summarized my example, and you are not missing anything. It seems that I have not nearly spelled out the details enough.

Let me start by stating that I do not doubt that effect size measures do measure what they are supposed to measure, mathematically. My critique is that effect size measures might not measure what they are supposed to measure in a substantive sense: practical significance, importance, etc. The latter terms are not from the field of math or statistics, and they do not have a clear, universal definition. These terms convey a normative standpoint. Importance can only be judged against the background of specific goals.

As Paul Allison has put it, there are two major goals of regression models: prediction and causal analysis. I tend to think of these two as y-centered (prediction) vs. x-centered (causation). Most of the research that I am confronted with is more x-centered (e.g., testing theories and hypothesis). I argue that when your goals are x-centered, effect size measures are pretty meaningless and might even lead to wrong conclusions. This is what my example is supposed to demonstrate. Effect size measures might help when your goals are y-centered (e.g., to select a parsimonious model).

Best
Daniel