corrcii: invalid correlation for r = 1

[Dirk Enzmann]

Nick Cox:

Using corrcii (or corrci) (from SSC) for correlation coefficients with abs(r) = 1 yields the error message "invalid correlation". This is slightly unfortunate if corrcii (or corrci) is applied to a list of correlation coefficients from any correlation matrix.

Wouldn't it be more convenient to modify the argument to atanh()in corrcii.ado (or corrci.ado) such as to subtract a tiny number from r if r = 1 (or to add if r = -1)? For example

. scalar r = 1
. corrcii 100 `=sign(r)*min(abs(r),`=1-5e-12')'

       n       r      95% confidence limits
     100      1.000      1.000     1.000

. scalar r = -1
. corrcii 100 `=sign(r)*min(abs(r),`=1-5e-12')'

       n       r      95% confidence limits
     100     -1.000     -1.000    -1.000

. // see:
.
. scalar r = -.9992
. assert r == sign(r)*min(abs(r),`=1-5e-12')

. scalar r = -1
. assert r == sign(r)*min(abs(r),`=1-5e-12')
assertion is false
r(9);

[Nick Cox]

Thanks for flagging this problem. I will think about what to do, but it won't involve an arbitrary fix.

In any case, who is this a problem for? I see that throwing out correlations with magnitude 1 is deliberate in my corrcii code.

A problem for me is that sometimes correlations of +1 or -1 are just vacuous (you correlated two variables that are exactly linearly related, such as a variable and itself) and sometimes they are just happenstance, usually in very small samples. (After all, two points distinct on two variables always imply a correlation with magnitude 1.)

I am not clear that the two cases deserve the same treatment. There is no sampling error in the first case -- the same result would be seen in any sample data -- but sampling error possibly is the entire issue in the second case.

At this moment I think the fix is just spelling out in the help why the command bounces the question back at the user.

[Dirk Enzmann]

At this moment I think the fix is just spelling out in the help why the command bounces the question back at the user.

I agree -- and the user can then think why the data are that way or how to proceed.

[Dirk Enzmann]

Bye the way: Under "Software Updates" in The Stata Journal 20(4), p. 1028 [pr0041_3: Correlation with confidence, or Fisher’s z revisited] you write

z ≡ atanhr = atanhρ + ρ/[2(n − 1)]

Shoudn't it be z ≡ atanhr = atanhρ − ρ/[2(n − 1) , instead?

[Nick Cox]

I will check this too.