How to make default options override a user-written programme?

[Chris Larkin]

Hello Statalisters,

I'm working up a programme to make a twoway bar graph with error bars, which can be run as a single command after running a regression. This is the first programme i've written, beyond very small things to assist with cleaning, and I know from others (such as coefplot) it is possible to permit all twoway options to be placed after the code. So, if my programme was called graphprog, i could write graphprog, title("An example of my graph programme", col(black) size(medsmall)). Does anyone with Stata programming experience now how I can proceed?

Chris

[Clyde Schechter]

What you are looking for is called a passthru option.

In your -syntax- statement, just add an asterisk (*) after the last of your regular options (if any). Then anything in the options part of a command invoking that program which is not covered by one of your command's specific options gets placed in local macro `options'. You can then pass those on to -graph twoway- or any other program by giving a command like -graph twoway scatter whatever whatever_else, `options'-.

[Chris Larkin]

Thank you Clyde. That looks like a good place to start; i'll see what I can pull together over the next few days. Once I have a working example (it might be a little while before I get to it), i'll post it here for community reference/interest

[Chris Larkin]

Update! If anyone is looking to do the same thing with a programme, i found these commands which pass through twoway options into your syntax.

  syntax [ using/ ] ,        ///
    [                             ///
    title(string)             ///
    ]
    
    // Twoway options
    _get_gropts , graphopts(`options') gettwoway

    
    if "`title'" != "" {
        local title `title' 
    }

My understanding is that the first bit allows you to access the twoway options generally, whereas the specific syntax definition and the logic at the end is my manual construction of the title option. Thus far I have got the latter to work but if i try adding other twoway options without defining their purpose first in syntax and logic the programme breaks.

[Nick Cox]

The title to this thread seems backwards to me. It seems that you want user-specified options to override defaults. If you really want to do the opposite, all you have to do is ignore whatever the user specifies.

I also don't understand your code here, or even its precise intent. There is no * in the syntax statement, so `options' will never be defined. In fact, even before that, if title() is the only allowed option, any other option will be illegal and syntax will issue an error message and the program will terminate.

The final block of code just copies a macro's contents back to itself and is itself legal but appears utterly pointless. This can be shown interactively:

local title "some stuff"

if "`title'" != "" {
   local title `title'
}

di "`title'"
some stuff

My main advice is to find some program as close as possible to what you want to do and look inside to see how it works.

Otherwise it may be that you are just selecting code and not showing us what is, you think, not relevant to your question. But if so, the corresponding risk is that we have no way to work out what you are not showing us. Also, as my substantive comments show, the code may be rendered inscrutable or otherwise puzzling.

Stata programming isn't trivial and I can recall from the early 1990s when I started with Stata that despite previous experience in various programming languages I didn't jump quickly between writing do-files and writing independent programs. The positive message in that is to get a do-file working that accepts arguments. It will then be easier to advise on converting it to a program. In fact some of the most eminent Stata users in the community signal privately that they typically use do-files and let others write programs.

Bars plus error bars sound like detonator plots, which are keywords to find (very negative) discussions on Statalist.

[Chris Larkin]

Hi Nick, thanks for your comment and advice. This is the first programme of any substance (modest as it is) that i've tried to put together. And I have certainly read discussions (negative) discussions around bar graphs with error bars; from a practical perspective though, the vast majority of statistics are presented as simple descriptions and i think it's unlikely we see a widespread programme of education to up-skill policy makers, journalists, NGO researchers, nor broader society that consumes statistics. As such, i'd argue a bar chart with error bars and significance stars is progress relative to just a bar chart. I digress..

I did jump the gun on posting a bit. The context of the above snippet is the below programme

    capture program drop graphprogv2
    program graphprogv2, rclass

    syntax [, *]
    // Twoway options
    _get_gropts , graphopts(`options') gettwoway

        preserve
            matrix A = r(table) //convert regression results to matrix
            
            forvalues i = 1/`= colsof(A)'{
                if `i'==`= colsof(A)'{
                        local rhs A[1,`i']
                        }
                    else{
                        local rhs A[1,`= colsof(A)']+A[1,`i']
                    }
                    if `i'==1{
                            local cmd generate
                    }
                    else{
                        local cmd replace
                    }
                            
                    `cmd' B = `rhs' in `i' //storing the coefficients in local
                    `cmd' C = B + (A[2,`i']*1.96) in `i' //storing upper CI in local
                    `cmd' D = B - (A[2,`i']*1.96) in `i' //storing lower CI in local
                    local pval`i' = A[4,`i'] //storing p-values in local
                    local end = `= colsof(A)'-1
                    local nObs = (A[7,1]+1) in `i'
            }
            
            drop if D==.
            
            gen Z = _N - _n+1 //making an index
            
            local forstars = ""
            forvalues i = 1/`end'{
                local stars = cond(`pval`i''<=0.01,`""**""',cond(`pval`i''<=0.05,`""*""',cond(`pval`i''<=0.1,`""+""', `""""')))
                if `stars'=="+"{
                    local posY = B[`i'] + B[`i']/30
                }
                else{
                    local posY = B[`i'] + B[`i']/50
                }
                local posX = Z[`i']+0.2
                local forstars = `"`forstars' text(`posY' `posX' `stars', place(e))"'
            }
            
            format B %9.1fc
            
            egen Bmax = max(B)
            local scmax round(Bmax+(0.4*Bmax), 10)

            if `scmax' - 1 >=10000{
                local interv 2000
            }
            else if `scmax' - 1 >=4000{
                local interv 1000
            }
            else if `scmax' - 1 >=1000{
                local interv 500
            }
            else if `scmax' - 1 >=600{
                local interv 100
            }
            else if `scmax' - 1 >=400{
                local interv 50
            }
            else if `scmax' - 1 >=200{
                local interv 40
            }
            else if `scmax' - 1 >=50{
                local interv 20
            }
            else if `scmax' - 1 >=10{
                local interv 10
            }
            else if `scmax' - 1 >=1{
                local interv 2
            }
            else if `scmax' -1 >=0.1{
                local interv 0.2
            }
            else if `scmax' -1 >=0.01{
                local interv 0.002
            }
        
            twoway (bar B Z, barw(0.95)) ///
                   (scatter B Z, msym(i) `forstars'), ///                    
                    ysc(r(0 "`=`scmax''") titlegap(*5)) ///
                    ylab(0(`=`interv'')"`=`scmax''", nogrid angle(0) format(%9.0fc)) ///
                    xlabel(, valuelabel noticks) ///
                    ytitle("Percentage", color(black) size(medsmall)) ///
                    title("That graph") ///
                    xtitle("") ///
                    title("", color(black) size(medsmall)) ///
                    note("  N=`nObs'" "        "  "** p<0.01, * p<0.05, + p<0.1") ///
                    legend(off) `options'                        
        restore
    end

As you can see I have now removed the logical statement for the title option, and have correctly applied the asterisk as Clyde suggested. I know it's not the prettiest bit programme (!) but it works for now.

Next steps are to take labels from the variables included in the regression and feed them to xlabels(), allow plotting of factor variables sensibly (at the moment this just plots everything), and include an option for plotting interactions.

If interested, you programme runs with the below snippet

sysuse auto, clear
reg mpg foreign rep78 length, r
graphprogv2, title("This graph") ytitle("Not percent..")

Thanks again for your comment! Much appreciated

[Nick Cox]

I'd check what you can do already with coefplot (SJ).

I think you need a version statement in your program to check for the availability of r(table) after regress.

I can't yet check this, but it includes various simplifications and style changes, and there are notes on things to do.

program graphprogv3        
    syntax [, *]
    tempname A
    matrix `A' = r(table) //convert regression results to matrix
            
    local cmd "generate"
    tempvar B C D Z
    local ncols = colsof(`A')
    forval i = 1/`ncols' {
        if `i'==`ncols' local rhs `A'[1,`i']
        else local rhs `A'[1,`ncols']+`A'[1,`i']
        `cmd' `B' = `rhs' in `i'

        `cmd' `C' = `B' + (`A'[2,`i']*1.96) in `i'
        `cmd' `D' = `B' - (`A'[2,`i']*1.96) in `i'
        local pval`i' = `A'[4,`i'] // storing p-values in local
 
        local cmd "replace"
    }
          
    local nObs = `A'[7,1] + 1  
    gen `Z' = _N - _n + 1 // making an index
            
    forval i = 1/`= `ncols' - 1'{
        local stars = cond(`pval`i'' <= 0.01, "**", ///
                      cond(`pval`i'' <= 0.05, "*",  ///
                      cond(`pval`i'' <= 0.1,  "+", "")))

        if "`stars'" == "+" local posY = `B'[`i'] + `B'[`i']/30
        else                local posY = `B'[`i'] + `B'[`i']/50
                
        local posX = `Z'[`i'] + 0.2
        local forstars `forstars' text(`posY' `posX' "`stars'", place(e))
    }      
          
    format `B' %9.1fc
    su `B', meanonly        
    local scmax = round(1.4 * `r(max)', 10)

    if `scmax' - 1 >= 10000     local interv 2000
    else if `scmax' - 1 >= 4000 local interv 1000
    else if `scmax' - 1 >= 1000 local interv 500
    else if `scmax' - 1 >= 600  local interv 100
    else if `scmax' - 1 >= 400  local interv 50
    else if `scmax' - 1 >= 200  local interv 40
    else if `scmax' - 1 >= 50   local interv 20
    else if `scmax' - 1 >= 10   local interv 10
    else if `scmax' - 1 >= 1    local interv 2
    else if `scmax' - 1 >= 0.1  local interv 0.2
    else if `scmax' - 1 >= 0.01 local interv 0.002
            
    twoway bar `B' `Z', barw(0.95)                           ///
    || scatter `B' `Z', msym(i) `forstars',                  ///                    
    ysc(r(0 `scmax') titlegap(*5))                           ///
    ylab(0(`interv')`scmax', nogrid angle(0) format(%9.0fc)) ///
    xlabel(, valuelabel noticks)                             ///
    ytitle("Percentage", color(black) size(medsmall))        ///
    title("That graph")                                      ///
    xtitle("")                                               ///
    title("", color(black) size(medsmall))                   ///
    note("  N=`nObs'" "   "  "** p<0.01, * p<0.05, + p<0.1") ///
    legend(off) `options'                        
end

/*

* = to do

* Use more informative variable and matrix names.

Making this r-class serves no point.

-preserve- and -restore- won't protect you from name clashes. Use temporary
names for matrices and variables needed temporarily.

Remove incorrect comments. What were named B and C are variables, not locals.

* Use sample size-appropriate multiplier, not 1.96.

It's surprising that this line is legal:
local nObs = (`A'[7,1]+1) in `i'
but it's certainly not needed every time around the first loop.

This line wasn't needed every time around the first loop, or at all:
local end = `= colsof(`A')'-1

Don't fire up -egen- just to get one maximum.

* The -scmax- code looks ad hoc and will not work as you seem to expect.
For example, rounding to multiples of 10 can produce 10,  0, -10 but
no other values given an argument in [-10, 10]

* You can't modify options of -twoway bar- without segregating pertinent
options.

* You don't plot the confidence intervals.

*/

[Nick Cox]

Small but crucial fixes.

 program graphprogv3       
    syntax [, *]
    tempname A
    matrix `A' = r(table) //convert regression results to matrix
           
    local cmd "generate"
    tempvar B C D Z
    local ncols = colsof(`A')
    quietly forval i = 1/`ncols' {
        if `i'==`ncols' local rhs `A'[1,`i']
        else local rhs `A'[1,`ncols']+`A'[1,`i']
        `cmd' `B' = `rhs' in `i'

        `cmd' `C' = `B' + (`A'[2,`i']*1.96) in `i'
        `cmd' `D' = `B' - (`A'[2,`i']*1.96) in `i'
        local pval`i' = `A'[4,`i'] // storing p-values in local
 
        local cmd "replace"
    }
         
    local nObs = `A'[7,1] + 1 
    gen `Z' = _N - _n + 1 // making an index
           
    forval i = 1/`= `ncols' - 1'{
        local stars = cond(`pval`i'' <= 0.01, "**", ///
                      cond(`pval`i'' <= 0.05, "*",  ///
                      cond(`pval`i'' <= 0.1,  "+", "")))

        if "`stars'" == "+" local posY = `B'[`i'] + `B'[`i']/30
        else                local posY = `B'[`i'] + `B'[`i']/50
               
        local posX = `Z'[`i'] + 0.2
        local forstars `forstars' text(`posY' `posX' "`stars'", place(e))
    }     
         
    format `B' %9.1fc
    su `B', meanonly       
    local scmax = round(1.4 * `r(max)', 10)

    if `scmax' - 1 >= 10000     local interv 2000
    else if `scmax' - 1 >= 4000 local interv 1000
    else if `scmax' - 1 >= 1000 local interv 500
    else if `scmax' - 1 >= 600  local interv 100
    else if `scmax' - 1 >= 400  local interv 50
    else if `scmax' - 1 >= 200  local interv 40
    else if `scmax' - 1 >= 50   local interv 20
    else if `scmax' - 1 >= 10   local interv 10
    else if `scmax' - 1 >= 1    local interv 2
    else if `scmax' - 1 >= 0.1  local interv 0.2
    else if `scmax' - 1 >= 0.01 local interv 0.002
           
    twoway bar `B' `Z' in 1/`ncols', barw(0.95)              ///
    || scatter `B' `Z' in 1/`ncols', msym(i) `forstars'      ///                   
    ysc(r(0 `scmax') titlegap(*5))                           ///
    ylab(0(`interv')`scmax', nogrid angle(0) format(%9.0fc)) ///
    xlabel(, valuelabel noticks)                             ///
    ytitle("Percentage", color(black) size(medsmall))        ///
    title("That graph")                                      ///
    xtitle("")                                               ///
    title("", color(black) size(medsmall))                   ///
    note("  N=`nObs'" "   "  "** p<0.01, * p<0.05, + p<0.1") ///
    legend(off) `options'                       
end

[Chris Larkin]

Hi Nick,

Thank you for your detailed post. This is immensely helpful (and the code looks much better now). See below what I have now.

As suggested, I have created more meaningful variable names, updated my comments throughout, and plotted my CIs. I have a few questions regarding your comments though if you will. I'm not sure how to action your suggestion to use a more sample-size appropriate multiplier (i.e. not 1.96); i'm also not exactly sure what you mean by not being able to modify options of -twoway bar- without segregating pertinent options.

 capture program drop graphprogv4

 program graphprogv4
    version 12.0
    syntax [, *]
    tempname A
    matrix `A' = r(table) //convert regression results to matrix
           
    local cmd "generate"
    tempvar mean upperb lowerb treat
    local ncols = colsof(`A')
    quietly forval i = 1/`ncols' {
        if `i'==`ncols' local rhs `A'[1,`i'] //for constant
        else local rhs `A'[1,`ncols']+`A'[1,`i'] //for constant + marginal effect
        
        `cmd' `mean' = `rhs' in `i'
        `cmd' `upperb' = `mean' + (`A'[2,`i']*1.96) in `i' //upper CIs
        `cmd' `lowerb' = `mean' - (`A'[2,`i']*1.96) in `i' //lower CIs
        local pval`i' = `A'[4,`i'] // storing p-values in local
 
        local cmd "replace"
    }
         
    local nObs = `A'[7,1] + 1 
    gen `treat' = _N - _n // making an index
           
    forval i = 1/`= `ncols' - 1'{
        local stars = cond(`pval`i'' <= 0.01, "**", ///
                      cond(`pval`i'' <= 0.05, "*",  ///
                      cond(`pval`i'' <= 0.1,  "+", "")))

        if "`stars'" == "+" local posY = `mean'[`i'] + `mean'[`i']/30
        else                local posY = `mean'[`i'] + `mean'[`i']/50
               
        local posX = `treat'[`i'] + 0.2
        local forstars `forstars' text(`posY' `posX' "`stars'", place(e))
    }     
         
    format `mean' %9.1fc
    su `mean', meanonly       
    local scmax = round(1.4 * `r(max)', 10)

    if `scmax' - 1 >= 10000     local interv 2000
    else if `scmax' - 1 >= 4000 local interv 1000
    else if `scmax' - 1 >= 1000 local interv 500
    else if `scmax' - 1 >= 600  local interv 100
    else if `scmax' - 1 >= 400  local interv 50
    else if `scmax' - 1 >= 200  local interv 40
    else if `scmax' - 1 >= 50   local interv 20
    else if `scmax' - 1 >= 10   local interv 10
    else if `scmax' - 1 >= 1    local interv 2
    else if `scmax' - 1 >= 0.1  local interv 0.2
    else if `scmax' - 1 >= 0.01 local interv 0.002
        
    local baropts "barw(0.75) plotregion(margin(8 8 0 0))"    
    local scatopts = "msym(i) mlabgap(12) mlab(`mean') mlabpos(6) mlabcolor(white) mlabsize(5)"    
    
    twoway bar `mean' `treat' in 1/`ncols', `baropts'                 ///
    || rcap `upperb' `lowerb' `treat' in 1/`=`ncols'-1'               ///
    || scatter `mean' `treat' in 1/`ncols', `scatopts' `forstars'      ///                                                                           
    ysc(r(0 `scmax') titlegap(*5))                                   ///
    ylab(0(`interv')`scmax', nogrid angle(0) format(%9.0fc))         ///
    xlabel(, noticks)                                                 ///
    ytitle("Percentage", color(black) size(medsmall))                ///
    title("That graph")                                              ///
    xtitle("")                                                      ///
    title("", color(black) size(medsmall))                          ///
    note("  N=`nObs'" "   "  "** p<0.01, * p<0.05, + p<0.1")        ///
    legend(off)                                                        ///
     `options'                       
end

Finally, i'm running into stumbling blocks trying to grab the variable label names for the and apply them to `treat'. This is the sort of thing i've been thinking; alas it has not worked exactly as i would like.

I first created an empty local -- local labvals = "" -- immediately after the -- matrix A = r(table) -- command. I then put -- local labvals = `"`labvals' `lab`=`i'-1''"' -- right at the end of the first forvalues loop, and I added -- Treat(string) -- to my syntax. I then added the below snippet immediate after my syntax. It stored the labels, but it's ended up just making a blank graph.

     if "`treat'" ==""{
         di as err "Must specify treatment indicator in graph command, e.g. bitgraph, treat(treatment)"
        exit 499
    }
    else if "`treat'"!=""{
        qui: levelsof `treat', local(t)
    }
        qui: levelsof `treat', local(levels)
        local lbe : value label `treat'
        foreach l of local levels {
            local lab`l' : label `lbe' `l'
        }
        di "`lab0' and `lab1'"

[Nick Cox]

I'll be travelling again soon and for that and other mundane personal reasons I don't guarantee to answer all your questions, let alone in detail.

I'm not sure how to action your suggestion to use a more sample-size appropriate multiplier (i.e. not 1.96)

You need to pick an appropriate number of degrees of freedom after the regression and then apply it.

not being able to modify options of -twoway bar- without segregating pertinent options.

You need to let the user separately specify bar options. Whatever the user currently specifies is applied only to the scatter plot command, and will be thus be ignored or at worst trigger an error. So, you could allow a collective baropts() and then let the user's choices override your default.

[Chris Larkin]

Fair enough! Enjoy your travels wherever you're off to. Thanks again for the tips - i'll keep working on my programme, and see if parts of coefplot can't help too.

Cheers