Wrapper for medians failing in loop after first by var

[Girish Venkataraman]

Hello all,

I was writing a wrapper to call medians, ranges and Mann-Whitney p-values but the wrapper is failing to call the second factor variable data in the loop. I am guessing there is something wrong in my loop specification. Showing the auto data set with my code below for reproducibility. You will see that it gives me data for all p-values correctly, but fails to generate anything for the second variable foreign. I switched the order and then I get data for medians etc for foreign 0 and foreign 1 from that matrix.

sysuse auto, clear
gen byte w2 = cond(weight>3000, 1, 0)

local x gear_ratio
local f w2 foreign



// Convert p-values to appropriate formatting needed
*---------------------------------------------------
foreach v of local f {
    ranksum `x', by(`v')
    local p = `r(p)'
    
    if `p'>=0.056 {
    local p`v' = "= " + string(`p', "%3.2f") 
}
if `p'>=0.044 & `p'<0.056 {
    local p`v'  = "= " + string(`p', "%5.4f") 
}
if `p' <0.044 {
    local p`v'  = "= " + string(`p', "%4.3f") 
}
if `p' <0.001 {
    local p`v'  "< .001"
}
if `p' <0.0001 {
    local p`v'  "< .0001"
}
}


// Collect all stats below for all variables around round to one decimal
*-----------------------------------------------------------------------
foreach v of local f{
levelsof `v', local(mylev)
tabstat `x', by(`v') statistic(p25 p50 p75 min max) save
return list
}
local col 0
foreach v of local f{
    local ++col
    foreach lev of local mylev{
        local j= `lev'+1
        mat list r(Stat`j')
        matmap r(Stat`j') B`j', map(round(@, 0.1))
mat list B`j'
        local `v'lqr`lev'  `=round(B`j'[1,`col'], 0.01)'
        local `v'med`lev'  `=round(B`j'[2,`col'], 0.01)'
        local `v'uqr`lev'  `=round(B`j'[3,`col'], 0.01)'
        local `v'min`lev'  `=round(B`j'[4,`col'], 0.01)'
        local `v'max`lev'  `=round(B`j'[5,`col'], 0.01)'
    }
}

di `foreignmax0'
di `w2min1'

[Clyde Schechter]

-matmap- is a user-written program and I am not familiar with it at all. So depending on what it does, my explanation here may be wrong, but I don't think that is the case.

Looking just at:

foreach v of local f{
levelsof `v', local(mylev)
tabstat `x', by(`v') statistic(p25 p50 p75 min max) save
return list
}
local col 0

foreach v of local f{
    local ++col
    foreach lev of local mylev{
        local j= `lev'+1
        mat list r(Stat`j')
    .....

Notice that local mylev and matrices r(Stat`j') are all created in the upper loop. At the end of the upper loop, local macro mylev contains the levels of foreign. You then go into the second loop, and while you attempt to loop over the variables in local macro f, notice that you never update local macro mylev or any of the r(Stat`j') matrices, so regardless of what the current value of `v' is, you are still working with the same leftover values of `mylev' and the r(Stat`j') matrices. In fact, if we add a -display "`v'"- command right before -local ++col-, you can actually see that the loop does get executed once with `v' == gear_ratio and then with `v' == foreign. But the results produced in both iterations of the loop are identical, because `v' itself is never used in the loop, and the values of `mylev' and r(Stat`j') are just the left over values from the preceding loop and do not change.

I think you need to combine the two loops into one:

foreach v of local f{
    local col 0
    levelsof `v', local(mylev)
    tabstat `x', by(`v') statistic(p25 p50 p75 min max) save
    return list
    local ++col
    foreach lev of local mylev{
        local j= `lev'+1
        mat list r(Stat`j')
        matmap r(Stat`j') B`j', map(round(@, 0.1))
        mat list B`j'
        local `v'lqr`lev'  `=round(B`j'[1,`col'], 0.01)'
        local `v'med`lev'  `=round(B`j'[2,`col'], 0.01)'
        local `v'uqr`lev'  `=round(B`j'[3,`col'], 0.01)'
        local `v'min`lev'  `=round(B`j'[4,`col'], 0.01)'
        local `v'max`lev'  `=round(B`j'[5,`col'], 0.01)'
    }
}

[Nick Cox]

matmap was posted on SSC in 2000. It's not used now by its original author.

A larger point is that you should never try to round() with second argument 0.1 or 0.01 when the real problem calls for a display format of say %2.1f or %3.2f or any other appropriate format with rounding. Most of the time you will get away with that, but it is the wrong solution, given precision problems. See also section 4 of https://journals.sagepub.com/doi/pdf...867X1801800311 for detailed discussion.

You can use display directly or apply a format in advance.

. local toad 0.123

. local frog : display %03.2f `toad'

. local newt : display %02.1f `toad'

* output edited below 
. mac li

_newt:          0.1
_frog:          0.12
_toad:          0.123

[Girish Venkataraman]

-matmap- is a user-written program and I am not familiar with it at all. So depending on what it does, my explanation here may be wrong, but I don't think that is the case.

Looking just at:

foreach v of local f{
levelsof `v', local(mylev)
tabstat `x', by(`v') statistic(p25 p50 p75 min max) save
return list
}
local col 0

foreach v of local f{
local ++col
foreach lev of local mylev{
local j= `lev'+1
mat list r(Stat`j')
.....

Notice that local mylev and matrices r(Stat`j') are all created in the upper loop. At the end of the upper loop, local macro mylev contains the levels of foreign. You then go into the second loop, and while you attempt to loop over the variables in local macro f, notice that you never update local macro mylev or any of the r(Stat`j') matrices, so regardless of what the current value of `v' is, you are still working with the same leftover values of `mylev' and the r(Stat`j') matrices. In fact, if we add a -display "`v'"- command right before -local ++col-, you can actually see that the loop does get executed once with `v' == gear_ratio and then with `v' == foreign. But the results produced in both iterations of the loop are identical, because `v' itself is never used in the loop, and the values of `mylev' and r(Stat`j') are just the left over values from the preceding loop and do not change.

I think you need to combine the two loops into one:

foreach v of local f{
local col 0
levelsof `v', local(mylev)
tabstat `x', by(`v') statistic(p25 p50 p75 min max) save
return list
local ++col
foreach lev of local mylev{
local j= `lev'+1
mat list r(Stat`j')
matmap r(Stat`j') B`j', map(round(@, 0.1))
mat list B`j'
local `v'lqr`lev' `=round(B`j'[1,`col'], 0.01)'
local `v'med`lev' `=round(B`j'[2,`col'], 0.01)'
local `v'uqr`lev' `=round(B`j'[3,`col'], 0.01)'
local `v'min`lev' `=round(B`j'[4,`col'], 0.01)'
local `v'max`lev' `=round(B`j'[5,`col'], 0.01)'
}
}

Explanation was priceless, Clyde Schechter I needed that more than anything. Of course, the solution worked, as it always does.

[Girish Venkataraman]

matmap was posted on SSC in 2000. It's not used now by its original author.

A larger point is that you should never try to round() with second argument 0.1 or 0.01 when the real problem calls for a display format of say %2.1f or %3.2f or any other appropriate format with rounding. Most of the time you will get away with that, but it is the wrong solution, given precision problems. See also section 4 of https://journals.sagepub.com/doi/pdf...867X1801800311 for detailed discussion.

You can use display directly or apply a format in advance.

. local toad 0.123

. local frog : display %03.2f `toad'

. local newt : display %02.1f `toad'

* output edited below
. mac li

_newt: 0.1
_frog: 0.12
_toad: 0.123

Point taken, Nick Cox. I will revise all my wrappers to change only display without this rounding. I did run into an issue with the code rounding up albumin values which changed the data for quite a few values around the 3.5 cut point and it took me a while to notice the error.