Effect sizes with blank or zero confidence intervals

[Vincent Jones]

I'm new to Stata. Can anyone explain why the 90% confidence interval for a significant factor or interaction with a "large" effect size has a lower bound that is calculated to be either 0 or is blank? The test is rANOVA.

Also, are the effect sizes calculated in Stata the partial variants of eta and omega squared? I had a disagreement between SAS and Stata in terms of omega squared and its CIs (SAS calculates the partial and semipartial).

[Anders Alexandersson]

Which ANOVA command do you refer to -- the user-written ranova from SSC (1998), or the official anova or some other ANOVA command? For those who use SAS, please also state which SAS command you refer to. Please show an example.

[Vincent Jones]

I did not us ranova, just anova. Syntax for Stata and the output is below.

   
15 . anova IPI Dose / Subject|Dose interval Dose#interval, repeated(interval)
      
                               Number of obs =        270    R-squared     =  0.9162
                               Root MSE      =    12.0722    Adj R-squared =  0.8873
      
                        Source | Partial SS         df         MS        F    Prob>F
                 --------------+----------------------------------------------------
                         Model |  318843.51         69   4620.9205     31.71  0.0000
                               |
                          Dose |  5331.2535          3   1777.0845      2.33  0.0851
                  Subject|Dose |  38060.529         50   761.21059  
                 --------------+----------------------------------------------------
                      interval |  250960.99          4   62740.247    430.50  0.0000
                 Dose#interval |  4993.2366         12   416.10305      2.86  0.0012
                               |
                      Residual |  29147.569        200   145.73785  
                 --------------+----------------------------------------------------
                         Total |  347991.08        269   1293.6471  
      
      
      Between-subjects error term:  Subject|Dose
                           Levels:  54        (50 df)
           Lowest b.s.e. variable:  Subject
           Covariance pooled over:  Dose      (for repeated variable)
      
      Repeated variable: interval
                                                Huynh-Feldt epsilon        =  0.4986
                                                Greenhouse-Geisser epsilon =  0.4541
                                                Box's conservative epsilon =  0.2500
      
                                                  ------------ Prob > F ------------
                        Source |     df      F    Regular    H-F      G-G      Box
                 --------------+----------------------------------------------------
                      interval |      4   430.50   0.0000   0.0000   0.0000   0.0000
                 Dose#interval |     12     2.86   0.0012   0.0132   0.0165   0.0464
                      Residual |    200
                 -------------------------------------------------------------------

20 . estat esize, level(90)
      
      Effect sizes for linear models
      
      -------------------------------------------------------------------
                   Source |   Eta-Squared     df     [90% Conf. Interval]
      --------------------+----------------------------------------------
                    Model |   .9162405        69     .8659409    .9033958
                          |
                     Dose |   .1228632         3            .    .2312441
             Subject|Dose |
      --------------------+----------------------------------------------
                 interval |   .8959419         4     .8736135    .9088912
            Dose#interval |   .1462542        12     .0351898    .1735148
      -------------------------------------------------------------------
      
    21 . estat esize, omega level(90)
      
      Effect sizes for linear models
      
      -------------------------------------------------------------------
                   Source | Omega-Squared     df     [90% Conf. Interval]
      --------------------+----------------------------------------------
                    Model |   .8873434        69     .8196905    .8700674
                          |
                     Dose |    .070235         3            .    .1851188
             Subject|Dose |
      --------------------+----------------------------------------------
                 interval |   .8938607         4     .8710858     .907069
            Dose#interval |   .0950295        12            0    .1239257
      -------------------------------------------------------------------

Interval consists of 5 repeated measures. There are 4 between groups (dose), 54 subjects, and total of 270 observations. I will also post the SAS code, but the important question here is why the CI for omega sq blank or zero in this case when the effect is significant. I have other examples where the blank or zero CI appears also for eta sq. I see a lot of zeros for the omega sq CIs. Let me know if you need more information.

[Anders Alexandersson]

I would check the references for estat esize to get an idea why there could be a problem in the calculation of the CI.

Some combinations of the F statistic, numerator degrees of freedom, and denominator degrees of
freedom yield confidence limits that do not contain the corresponding estimated value

The methods and formulas are under esize.

[Vincent Jones]

Thanks. The full quote from this document is:

eta2 and omega2 were developed in the context of analysis of variance. Thus, the published research on
the calculation of their confidence intervals focuses on cases where the numerator degrees of freedom
are relatively small (for example, df < 20).

Some combinations of the F statistic, numerator degrees of freedom, and denominator degrees of
freedom yield confidence limits that do not contain the corresponding estimated value for an eta2 or
omega2. This problem is most commonly observed for larger numerator degrees of freedom.

Nothing in the literature suggests alternative methods for constructing confidence intervals in such
cases; therefore, we recommend cautious interpretation of confidence intervals for eta2 and omega2 when
the numerator degrees of freedom are greater than 20.

While helpful, I am not sure if this explains the current problem. Numerator df for the model is indeed 69, which is beyond the df 20 that the tech note refers to and as warned, the CI there does not include the calculated effect size. On the other hand, Dose and Dose#interval both have numerator dfs well below 20 and the problem here is not that the effect size isn't included in the CI range, but rather that the lower bound of the CI is either zero or not calculated. Any other ideas?

[Marcos Almeida]

About differences in ANOVA concerning "default" SS types in Stata, R and SAS, there has been some discussion here: http://www.statalist.org/forums/foru...es-stata-and-r

[Vincent Jones]

As far as the discrepancy between SAS and Stata omega sq, I'm not sure it has to do with SS. SS values are very slightly off, but almost all the values, such as F, r sq, etc, agree. Eta sq and it's CIs also agree. The discrepancy is specifically with omega sq.

. anova IPI1 Dose

                         Number of obs =         54    R-squared     =  0.1813
                         Root MSE      =    27.2477    Adj R-squared =  0.1321

                  Source | Partial SS         df         MS        F    Prob>F
              -----------+----------------------------------------------------
                   Model |  8218.4157          3   2739.4719      3.69  0.0178
                         |
                    Dose |  8218.4157          3   2739.4719      3.69  0.0178
                         |
                Residual |  37121.814         50   742.43628  
              -----------+----------------------------------------------------
                   Total |   45340.23         53   855.47603

SAS

. estat esize, level(90)

Effect sizes for linear models

-------------------------------------------------------------------
             Source |   Eta-Squared     df     [90% Conf. Interval]
--------------------+----------------------------------------------
              Model |    .181261         3     .0190984    .2988667
                    |
               Dose |    .181261         3     .0190984    .2988667
-------------------------------------------------------------------



. estat esize, omega level(90)

Effect sizes for linear models

-------------------------------------------------------------------
             Source | Omega-Squared     df     [90% Conf. Interval]
--------------------+----------------------------------------------
              Model |   .1321367         3            0    .2567987
                    |
               Dose |   .1321367         3            0    .2567987
-------------------------------------------------------------------

SAS

As you can see, the problem is again with the omega sq and the CIs. Stata seems to have most issues specifically with omega sq CIs (either 0 or missing lower bound). So this makes me wonder whether there are issues with omega sq calculations in Stata. SAS also reports the same CI for both eta sq and omega sq.

SAS is only able to calculate only univariate effect sizes and does not calculate repeated measures effect sizes. This is the main reason I decided to give Stata a shot since I need the effect sizes, along with the CIs. So again, I am not sure what is going on here. But the more pressing issue are the missing CI lower bounds in Stata for both eta and omega sq.

Unfortunately, it seems that reporting effect sizes hasn't caught on, so most scientists overlook these calculations, which in turn end up being skipped or incompletely developed in statistical software.

[Vincent Jones]

Alright, as far as omega squared is concerned, it seems that Stata does not calculate omega squared, but epsilon squared. Manual calculations and the disagreement with SAS all point to that evidence. Daniel Lakens also says the same in his blog. Really, it would be good to get someone from Stata to explain these problems, both with confidence intervals and with the incorrect output for the omega squared.

[Bruce Weaver]

Alright, as far as omega squared is concerned, it seems that Stata does not calculate omega squared, but epsilon squared.

. * Q. Does -estat esize, omega- actually compute epsilon^2?
.
. clear

. use http://www.ats.ucla.edu/stat/stata/faq/crf24
(CRF Example - Kirk, 1st Edition)

. anova y b

                           Number of obs =      32     R-squared     =  0.8259
                           Root MSE      = 1.21008     Adj R-squared =  0.8072

                  Source |  Partial SS    df       MS           F     Prob > F
              -----------+----------------------------------------------------
                   Model |       194.5     3  64.8333333      44.28     0.0000
                         |
                       b |       194.5     3  64.8333333      44.28     0.0000
                         |
                Residual |          41    28  1.46428571   
              -----------+----------------------------------------------------
                   Total |       235.5    31  7.59677419   

. * Hand calculation of eta^2:
. display "eta^2 = " 194.5/235.5
eta^2 = .82590234

. estat esize // eta^2 via

Effect sizes for linear models

-------------------------------------------------------------------
             Source |   Eta-Squared     df     [95% Conf. Interval]
--------------------+----------------------------------------------
              Model |   .8259023         3     .6545609    .8756227
                    |
                  b |   .8259023         3     .6545609    .8756227
-------------------------------------------------------------------

.
. display "omega^2 = " (194.5-3*1.46428571)/(235.5+1.46428571)
omega^2 = .80226074

. display "epsilon^2 = " (194.5-3*1.46428571)/235.5
epsilon^2 = .80724901

. * Now use -estat esize-
. estat esize, omega

Effect sizes for linear models

-------------------------------------------------------------------
             Source | Omega-Squared     df     [95% Conf. Interval]
--------------------+----------------------------------------------
              Model |    .807249         3     .6175495    .8622965
                    |
                  b |    .807249         3     .6175495    .8622965
-------------------------------------------------------------------

. * Compare with result from Phil Ender's -omega2- program
. * package name:  omega2.pkg
. *         from:  http://www.ats.ucla.edu/stat/stata/ado/analysis/
. omega2


 omega squared for b = 0.8023
 fhat effect size   = 2.0142

.
. * CONCLUSIONS:  
. * 1. The result from -estat esize, omega-
. *    matches my hand calculation of epsilon^2.
. * 2. The result from Phil Ender's omega2 matches my
. *    hand calculation of omega^2.