Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Calculating effect size for coefficients in multiple linear regression,.

    Hi Everyone,

    I am using the General Social Survey from NORC. I have an N of ~1200 people. I am using linear regression to determine if atheists and theists differ in terms of nihilism (nih_ath). As you can see, the results for the variable negath (0=theists; 1=atheists) are non-significant, but I want to discuss the observed effect size. There was a group imbalance (approximately 9 theists for every 1 atheist), which has implications for power that I would like to address preemptively.

    Code:
    ------------------------------------------------------------------------------
                 |             Linearized
         nih_ath |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
    -------------+----------------------------------------------------------------
          negath |   .1877569   .1159446     1.62   0.108    -.0419951    .4175089
             sex |  -.1471211   .0613612    -2.40   0.018    -.2687124   -.0255297
             age |  -.0012937   .0024676    -0.52   0.601    -.0061835     .003596
           drac2 |  -.0386779   .0971669    -0.40   0.691    -.2312206    .1538648
           drac3 |   .4211109    .171517     2.46   0.016     .0812384    .7609833
           dmar2 |   -.007733   .1130517    -0.07   0.946    -.2317524    .2162864
           dmar3 |  -.1266924   .0924548    -1.37   0.173    -.3098977    .0565129
           dmar4 |    .092398   .1612822     0.57   0.568    -.2271935    .4119895
           dmar5 |   .0825658   .0852277     0.97   0.335    -.0863185    .2514501
            ded2 |  -.5666068   .1159979    -4.88   0.000    -.7964643   -.3367493
            ded3 |  -.5428579   .1286772    -4.22   0.000    -.7978404   -.2878754
            ded4 |  -.5687932   .1207139    -4.71   0.000    -.8079958   -.3295906
            ded5 |  -.6803206   .1411244    -4.82   0.000    -.9599679   -.4006732
             inc |  -.0288876   .0074534    -3.88   0.000    -.0436571   -.0141181
           _cons |   .8739862   .2451671     3.56   0.001     .3881712    1.359801
    ------------------------------------------------------------------------------


    Prior to running the analyses, I standardized the outcome variable; the coefficients of the model can be interpreted as relating to changes in SD. My decision to standardize the outcome variable was because I wanted the coefficients to be interpretable as approximations of Cohen's d. The approximations appear to be pretty decent, but I'd rather use a precise approach.

    How do you calculate the effect size for group differences, within a multiple linear regression?
    Last edited by David Speed; 03 Jul 2017, 04:45.

  • #2
    To start, being a survey, you may wish to deal with svy prefix and the situation prompts to a different approach, instead of getting, say, omega-squared and eta-squared.

    Second, it seems you created dummies. Perhaps you could use factor notation.

    Third, I don't know whether nihilism is a continuous variable and follows linearity issues (basically, concerning the residuals).

    Fourth, with Cohen's d we'd need just a group (here, predictor) with two levels, but you have several predictors.

    These things being remarked and considered, you may use the omega-squared as well as the eta-squared.

    To get them, just type:

    Code:
    estat esize
    estat esize, omega
    Hopefully that helps.
    Last edited by Marcos Almeida; 03 Jul 2017, 06:12.
    Best regards,

    Marcos

    Comment


    • #3
      Hi Marcos,

      Thank-you for your prompt response.

      1. Sorry for the confusion, I did use the "svy" prefix.
      2. I used a hierarchical model which doesn't support factor notation.
      3. For the purposes of the study nihilism is being treated as a continuous (in this case, interval) variable.
      4. Yes, I understand that Cohen's d is used for two groups, I want to determine the effect size of being a theist vs. an atheist when those covariates have been entered. In other words, how much do atheists and theists differ in terms of nihilism once other factors are controlled for?
      5. svy: doesn't support esize commands. Also, the effect size indicators address the model's ability to account for variance, not the differences between groups. These aren't equivalent or really even close given the size discrepancy in groups.

      Cheers,

      David.

      Comment


      • #4
        David:
        as per the outcome of your regression, you cannot rule out that there's no difference in -nihilism- between theists and atheists when adjusted for theb remainng predictors (the width of the 95% CI support this statement better than the non-significant p-value).
        Kind regards,
        Carlo
        (Stata 18.0 SE)

        Comment


        • #5
          Hi Carlo,

          Thanks for your feedback. I haven't ruled out that there's no difference, in fact I suspect that there is no difference. This is why I want to discuss effect size so there is more nuance on what statistical significance (or lack thereof) means. When using large datasets getting statistical significance is not uncommon, but it will often happen because the power is so high and the findings are actually trivial.

          What are your thoughts?

          Cheers,

          David.
          Last edited by David Speed; 03 Jul 2017, 06:52. Reason: Clarification.

          Comment


          • #6
            David:
            I agree with you about the easiness of finding statistical significance (sometimes without praictical significance, though) when the dataset is large.
            That said, I would still consider the 95% CI width informative enough for communicating the absence of evidence on any difference in -nihilism- between theists and atheists.
            As an aside, I would also act on Marcos' helpful advice to use -fvvarlist- notation for categorical variables; you may also want to te test dummies joint statistical significance via -testparm-.
            Kind regards,
            Carlo
            (Stata 18.0 SE)

            Comment


            • #7
              Hi Carlo,

              If the results had been statistically significant, would you know of any way to describe the effect size of the negath group? I agree that it's somewhat a moot point in this presented case, but I have a similar problem with other models where it's difficult to describe the magnitude of the difference between groups in multiple regression.

              Cheers,

              David.

              Comment


              • #8
                David:
                Marcos pointed you out to the right codes (please see #2).
                Otherwise, you may interact -group- with the variable(s) you're interested in following -fvvarlist- notation.
                Kind regards,
                Carlo
                (Stata 18.0 SE)

                Comment


                • #9
                  Hi Carlo,

                  esize commands don't work in "svy" and their indication for effect size is in the model, not the group. This is problematic because of the group size imbalance will "mess with" model effect sizes. This is why I was looking for an approach that would be equivalent to Cohen's d (or Hedge's g) but would usable in the context of a multiple regression. Essentially, I want to be able to compare atheists and theists' coefficients and translate this into an effect size. Standardizing the outcome variable means that the coefficients approximate Hedge's g closely (from what I can tell), but I wanted a more precise way of determining effect size.

                  Interactions wouldn't really work in this case because I'm really only interested in the intercept differences for atheists and theists not slope differences.

                  Cheers,

                  David.

                  Comment


                  • #10
                    Alternatively, you could talk about effect size as the change in the predicted value for a given change in an x. This also works for dummy variables. The margins command gives you the predicted values, and you can easily calculate the difference with macros and test if the change is statistically significant as well (if you want). Some use a metric that is essentially a standard deviation change in x associates with a z standard deviation change in predicted y. I prefer meaningful changes in x and y (e.g., moving x from 10th to 90th percentile increases predicted y by z).

                    Comment


                    • #11
                      Dear David,

                      Do you find out the way? If so, please share us. I have the same problem. I want to calculate effect sizes for different proportions ( also means), with 'svy' and direct standardization.

                      best regards,

                      aung soe htet

                      Comment


                      • #12
                        To echo the previous comment, please do share if you ever resolved this David.

                        I am having a similar issue-- I want to estimate effect sizes after running a glm with survey data (e.g., svy: glm ___) and cannot figure out a straightforward way to do this in Stata.

                        Comment


                        • #13
                          Sorry for the much-delayed response. I was never able to really find a perfect solution. I ended up approximating the effect size using Hedges' g. Let me know if you come up with a better solution!

                          Comment

                          Working...
                          X