Calculating effect size for coefficients in multiple linear regression,.

[David Speed]

Hi Everyone,

I am using the General Social Survey from NORC. I have an N of ~1200 people. I am using linear regression to determine if atheists and theists differ in terms of nihilism (nih_ath). As you can see, the results for the variable negath (0=theists; 1=atheists) are non-significant, but I want to discuss the observed effect size. There was a group imbalance (approximately 9 theists for every 1 atheist), which has implications for power that I would like to address preemptively.

------------------------------------------------------------------------------
             |             Linearized
     nih_ath |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      negath |   .1877569   .1159446     1.62   0.108    -.0419951    .4175089
         sex |  -.1471211   .0613612    -2.40   0.018    -.2687124   -.0255297
         age |  -.0012937   .0024676    -0.52   0.601    -.0061835     .003596
       drac2 |  -.0386779   .0971669    -0.40   0.691    -.2312206    .1538648
       drac3 |   .4211109    .171517     2.46   0.016     .0812384    .7609833
       dmar2 |   -.007733   .1130517    -0.07   0.946    -.2317524    .2162864
       dmar3 |  -.1266924   .0924548    -1.37   0.173    -.3098977    .0565129
       dmar4 |    .092398   .1612822     0.57   0.568    -.2271935    .4119895
       dmar5 |   .0825658   .0852277     0.97   0.335    -.0863185    .2514501
        ded2 |  -.5666068   .1159979    -4.88   0.000    -.7964643   -.3367493
        ded3 |  -.5428579   .1286772    -4.22   0.000    -.7978404   -.2878754
        ded4 |  -.5687932   .1207139    -4.71   0.000    -.8079958   -.3295906
        ded5 |  -.6803206   .1411244    -4.82   0.000    -.9599679   -.4006732
         inc |  -.0288876   .0074534    -3.88   0.000    -.0436571   -.0141181
       _cons |   .8739862   .2451671     3.56   0.001     .3881712    1.359801
------------------------------------------------------------------------------

Prior to running the analyses, I standardized the outcome variable; the coefficients of the model can be interpreted as relating to changes in SD. My decision to standardize the outcome variable was because I wanted the coefficients to be interpretable as approximations of Cohen's d. The approximations appear to be pretty decent, but I'd rather use a precise approach.

How do you calculate the effect size for group differences, within a multiple linear regression?

[Marcos Almeida]

To start, being a survey, you may wish to deal with svy prefix and the situation prompts to a different approach, instead of getting, say, omega-squared and eta-squared.

Second, it seems you created dummies. Perhaps you could use factor notation.

Third, I don't know whether nihilism is a continuous variable and follows linearity issues (basically, concerning the residuals).

Fourth, with Cohen's d we'd need just a group (here, predictor) with two levels, but you have several predictors.

These things being remarked and considered, you may use the omega-squared as well as the eta-squared.

To get them, just type:

estat esize
estat esize, omega

Hopefully that helps.

[David Speed]

Hi Marcos,

Thank-you for your prompt response.

1. Sorry for the confusion, I did use the "svy" prefix.
2. I used a hierarchical model which doesn't support factor notation.
3. For the purposes of the study nihilism is being treated as a continuous (in this case, interval) variable.
4. Yes, I understand that Cohen's d is used for two groups, I want to determine the effect size of being a theist vs. an atheist when those covariates have been entered. In other words, how much do atheists and theists differ in terms of nihilism once other factors are controlled for?
5. svy: doesn't support esize commands. Also, the effect size indicators address the model's ability to account for variance, not the differences between groups. These aren't equivalent or really even close given the size discrepancy in groups.

Cheers,

David.

[Carlo Lazzaro]

David:
as per the outcome of your regression, you cannot rule out that there's no difference in -nihilism- between theists and atheists when adjusted for theb remainng predictors (the width of the 95% CI support this statement better than the non-significant p-value).

[David Speed]

Hi Carlo,

Thanks for your feedback. I haven't ruled out that there's no difference, in fact I suspect that there is no difference. This is why I want to discuss effect size so there is more nuance on what statistical significance (or lack thereof) means. When using large datasets getting statistical significance is not uncommon, but it will often happen because the power is so high and the findings are actually trivial.

What are your thoughts?

Cheers,

David.

[Carlo Lazzaro]

David:
I agree with you about the easiness of finding statistical significance (sometimes without praictical significance, though) when the dataset is large.
That said, I would still consider the 95% CI width informative enough for communicating the absence of evidence on any difference in -nihilism- between theists and atheists.
As an aside, I would also act on Marcos' helpful advice to use -fvvarlist- notation for categorical variables; you may also want to te test dummies joint statistical significance via -testparm-.

[David Speed]

Hi Carlo,

If the results had been statistically significant, would you know of any way to describe the effect size of the negath group? I agree that it's somewhat a moot point in this presented case, but I have a similar problem with other models where it's difficult to describe the magnitude of the difference between groups in multiple regression.

Cheers,

David.

[Carlo Lazzaro]

David:
Marcos pointed you out to the right codes (please see #2).
Otherwise, you may interact -group- with the variable(s) you're interested in following -fvvarlist- notation.

[David Speed]

Hi Carlo,

esize commands don't work in "svy" and their indication for effect size is in the model, not the group. This is problematic because of the group size imbalance will "mess with" model effect sizes. This is why I was looking for an approach that would be equivalent to Cohen's d (or Hedge's g) but would usable in the context of a multiple regression. Essentially, I want to be able to compare atheists and theists' coefficients and translate this into an effect size. Standardizing the outcome variable means that the coefficients approximate Hedge's g closely (from what I can tell), but I wanted a more precise way of determining effect size.

Interactions wouldn't really work in this case because I'm really only interested in the intercept differences for atheists and theists not slope differences.

Cheers,

David.

[Phil Bromiley]

Alternatively, you could talk about effect size as the change in the predicted value for a given change in an x. This also works for dummy variables. The margins command gives you the predicted values, and you can easily calculate the difference with macros and test if the change is statistically significant as well (if you want). Some use a metric that is essentially a standard deviation change in x associates with a z standard deviation change in predicted y. I prefer meaningful changes in x and y (e.g., moving x from 10th to 90th percentile increases predicted y by z).

[AungSoe Htet]

Dear David,

Do you find out the way? If so, please share us. I have the same problem. I want to calculate effect sizes for different proportions ( also means), with 'svy' and direct standardization.

best regards,

aung soe htet

[Kathleen Hupfeld]

To echo the previous comment, please do share if you ever resolved this David.

I am having a similar issue-- I want to estimate effect sizes after running a glm with survey data (e.g., svy: glm ___) and cannot figure out a straightforward way to do this in Stata.

[David Speed]

Sorry for the much-delayed response. I was never able to really find a perfect solution. I ended up approximating the effect size using Hedges' g. Let me know if you come up with a better solution!