Missing return values in program on second run

[Vivek Gupta]

Dear Friends,
I have written two programs
The first program saves the svy computed proportion estimation results as local macros

cap program drop clustercibysexwt
program define clustercibysexwt , rclass
    args outcome
    return clear
    qui svy linearized : proportion `outcome',  over(sex)
            mat all  = r(table)
            mat all  = all'
             return local propM : di  %4.1f  all[3,1]*100 // Cluster-adjusted  prevalence for Males
            return local propMlb  : di  %4.1f  all[3,5]*100 // Cluster-adjusted  prevalence Lower CI
            return local propMub  : di  %4.1f  all[3,6]*100 // Cluster-adjusted  prevalence Upper CI
            return local propF : di  %4.1f  all[4,1]*100 // Cluster-adjusted  prevalence for FeMales
            return local propFlb  : di  %4.1f  all[4,5]*100 // Cluster-adjusted  prevalence Lower CI
            return local propFub  : di  %4.1f  all[4,6]*100 // Cluster-adjusted  prevalence Upper CI    
      
    qui svy linearized : proportion `outcome'
            mat all  = r(table)
            mat all  = all'
            return local propAll : di  %4.1f all[2,1]*100 // Cluster-adjusted  prevalence for All
            return local propAlllb  : di  %4.1f  all[2,5]*100 // Cluster-adjusted  prevalence Lower CI
            return local propAllub  : di  %4.1f  all[2,6]*100 // Cluster-adjusted  prevalence Upper CI
            return local denom = e(N)
            return local outcome = e(varlist)
    qui count
            return local sample = r(N)
    qui count if sex==1 & `outcome'!=. //male
            return local sampleM = r(N)    
    qui count if sex==2 & `outcome'!=. //female
            return local sampleF = r(N)        
            
    // di "Cluster Adjusted CIs"
    // di  _column(1) "M = "`propM' "(`propMlb', `propMub')" _column(25)  "F =  `propF'(`propFlb', `propFub') " _column(50)  "T = `propAll'(`propAlllb', `propAllub') "
end

The second resultsByDistandIndia program iterates over subsets of the data (based on areacode2) from the full dataset and calls "clustercibysexwt" program on each subset. Then it calls the "clustercibysexwt" on entire dataset

qui svyset cluster, vce(linearized) singleunit(certainty) weight(wtRAABEnroll2Dist)

cap program drop resultsByDistandIndia
program define   resultsByDistandIndia
    args outcome
        levelsof areacode2, local(levels)
        foreach l of local levels  {  // of local levels    in 1
            qui preserve
            qui keep if areacode2==`l'    
            local DistName : label(areacode2) `l'
            local DistName = ustrregexra(`"`DistName'"', " ", "-")
            local distCode = `l'
             clustercibysexwt    `outcome'
            // return list
            di   "`DistName'" _column(17)  `r(sample)'   _column(25)  `r(denom)' _column(32)  `r(sampleM)' _column(40)  `r(sampleF)'    _column(47) "M = "r(propM) " (`r(propMlb)', `r(propMub)')" _column(70)  "F =  `r(propF)' (`r(propFlb)', `r(propFub)') " _column(95)  "T = `r(propAll)' (`r(propAlllb)', `r(propAllub)') "
            restore
        }

        qui svyset areacode2, fpc(numDists)   vce(linearized) singleunit(missing) || cluster, strata(sex) weight(wtRAABEnroll2Dist)
        clustercibysexwt    `outcome'
        di   "INDIA" _column(17)  `r(sample)'    _column(25)  `r(denom)'  _column(32)  `r(sampleM)' _column(40)  `r(sampleF)'    _column(47) "M = "r(propM) " (`r(propMlb)', `r(propMub)')" _column(70)  "F =  `r(propF)' (`r(propFlb)', `r(propFub)') " _column(95)  "T = `r(propAll)' (`r(propAlllb)', `r(propAllub)') "        
end

The command I need to run is resultsByDistandIndia mainOutputVariable
The first run generates the desired output

Jammu           2751    712    283     429    M = 27.8 (22.8, 33.3)  F =  24.6 (20.4, 29.4)   T = 26.0 (22.3, 30.1)
Bilaspur        2718    574    240     334    M = 42.5 (36.6, 48.7)  F =  36.4 (31.6, 41.6)   T = 39.2 (35.7, 42.8)
....
Thoubal         2732    325    138     187    M = 26.1 (18.3, 35.8)  F =  18.9 (13.6, 25.5)   T = 22.1 (16.9, 28.4)
East-Siang      2729    608    259     349    M = 18.0 (12.6, 24.9)  F =  23.8 (18.6, 30.0)   T = 20.9 (16.4, 26.2)
INDIA           85135   21651  8964    12687  M = 38.0 (34.7, 41.4)  F =  35.6 (32.3, 39.0)   T = 36.7 (33.6, 39.9)

The second run of the resultsByDistandIndia mainOutputVariable command however generates incomplete output for the subset

Jammu           2751    712    283     429    M = 27.8 (   .,    .)  F =  24.6 (   .,    .)   T = 26.0 (   .,    .)
Bilaspur        2718    574    240     334    M = 42.5 (   .,    .)  F =  36.4 (   .,    .)   T = 39.2 (   .,    .)
..
Thoubal         2732    325    138     187    M = 26.1 (   .,    .)  F =  18.9 (   .,    .)   T = 22.1 (   .,    .)
East-Siang      2729    608    259     349    M = 18.0 (   .,    .)  F =  23.8 (   .,    .)   T = 20.9 (   .,    .)
INDIA           85135   21651  8964    12687  M = 38.0 (34.7, 41.4)  F =  35.6 (32.3, 39.0)   T = 36.7 (33.6, 39.9)

However

I am absolutely stumped and just cannot figure out where I am going wrong. Any help would be utterly appreciated

Thanks

Vivek

[Nick Cox]

Each time you call the command, you return clear, so that would seem to be the root of your problem. There are various ways round this, such as using postfile. People who use svy a lot (not me) should have a better idea what to advise.

[Vivek Gupta]

Many Thanks Nick
The return clear was intended to clear any prior in-memory return values so as not to contaminate the next run of the program
I identified the error as also not setting the svyset properly for district level estimates "qui svyset cluster, vce(linearized) singleunit(certainty) weight(wtRAABEnroll2Dist)" from the second run of the loop.
AS a result, during the second run, the svyset of India level estimates were being used for district level estimates.
I corrected the same by bringing in the svyset for districts in the loop as below and now everything works normally

My updated working commands are

cap program drop resultsByDistandIndia
program define   resultsByDistandIndia
    args outcome
    qui    levelsof areacode2, local(levels)
        foreach l of local levels  {  // of local levels    in 1
            qui preserve
            qui keep if areacode2==`l'    
            local DistName : label(areacode2) `l'
            local DistName = ustrregexra(`"`DistName'"', " ", "-")
            local distCode = `l'
            
            qui svyset cluster, vce(linearized) singleunit(certainty) weight(wtRAABEnroll2Dist)
            clustercibysexwt    `outcome'
            di   "`DistName'" _column(17)  `r(enrollT)'   _column(25)  `r(denomT)' _column(32)  `r(denomM)' _column(40)  `r(denomF)'    _column(47) "M = "r(propM) " (`r(propMlb)', `r(propMub)')" _column(70)  "F =  `r(propF)' (`r(propFlb)', `r(propFub)') " _column(95)  "T = `r(propAll)' (`r(propAlllb)', `r(propAllub)') "
            restore
        }
        qui svyset areacode2, fpc(numDists)   vce(linearized) singleunit(missing) || cluster,  weight(wtRAABEnroll2Dist)
        clustercibysexwt    `outcome'
        di   "INDIA" _column(17)  `r(enrollT)'    _column(25)  `r(denomT)'  _column(32)  `r(denomM)' _column(40)  `r(denomF)'    _column(47) "M = "r(propM) " (`r(propMlb)', `r(propMub)')" _column(70)  "F =  `r(propF)' (`r(propFlb)', `r(propFub)') " _column(95)  "T = `r(propAll)' (`r(propAlllb)', `r(propAllub)') "        
end