String scalar not displaying outside program

[Girish Venkataraman]

Hello all:

I decided to graduate myself from using only include files to creating a program for a standard display of Cox model HR and 95% CIs for a single dummy indicator to be used in putdocx statements. I will change the di command below to a putdocx down the line in my paper write up.

While the last scalar is evaluating correctly within the program based on my set trace on, the scalar calls and local macros defined inside the program are not surviving outside the program. I was blindly changing around quotes, compound double quotes and assignment operators in that last scalar call without success. I have chosen to use the stan3 data set so it is fully reproducible.

webuse stan3, clear

// Program coxer below-------------------------
version 18.0
cap program drop coxer
cap drop scalar callhr
program define coxer, rclass
  syntax varlist(numeric min=0 max=1) [if]
  marksample touse
    stcox `varlist' if(`touse')
    tempname a 
    mat `a' = r(table)
    matrix list `a'

// Gather HR, 95% and pvalue
*--------------------------
local b = string(`a'[1,1], "%09.1fc")
local l = string(`a'[5,1], "%09.1fc")
local u = string(`a'[6,1], "%09.1fc")
local p = `a'[4,1]
local n = e(N)

// Fix p-value formats
*---------------------
if `p'>=0.056 {
    local pvalue = "= " + string(`p', "%03.2f") 
}
if `p'>=0.044 & `p'<0.056 {
    local pvalue = "= " + string(`p', "%05.4f") 
}
if `p' <0.044 {
    local pvalue = "= " + string(`p', "%04.3f") 
}
if `p' <0.001 {
    local pvalue "< 0.001"
}
if `p' <0.0001 {
    local pvalue "< 0.0001"
}

set trace on
// Create display text for the HR
*---------------------------------
scalar callhr = "(HR = `b'; 95% CI: [`l',  `u'];P = `pvalue')"
end

// Test the program
*------------------
coxer transplant
di "This is the effect of transplant callhr and checking `b'"
di "`b'"

What this evaluates to in the display:

  - scalar callhr = "(HR = `b'; 95% CI: [`l',  `u'];P = `pvalue')"
  = scalar callhr = "(HR = 0.3; 95% CI: [0.2,  0.4];P = < 0.0001)"

. di "This is effect of transplant callhr and checking `b'"
This is effect of transplant callhr and checking 

. di "`b'"

[Mike Lacy]

1) Why is di "`b'" showing blank?

The local b is defined inside the program coxer. A local has a scope that is "local" to the program in which it is defined. When you reference the local b *outside* the program, Stata looks for a local b in the namespace defined outside your program, finds no local with such a name, and does what Stata always does with a nonexistent local, namely to treat it as blank. If you want to make the contents of the local b inside your program accessible outside your program, the standard thing to do is to put that value into the return list of your r-class program. So, somewhere inside your program and after where you create the local b, you can put a return statement:

return local b = "`b'"

Then, after you run your program, there will be an item r(b) in your return list that contains the value of b from inside your program. This item doesn't have to be called "b," but it would seem a reasonable choice of name.

2) You say "the scalar calls ...defined inside the program are not surviving... ."
I assume you mean the value of your one scalar, callhr, is not surviving. That is not true. You can do:

di callhr

after your program and the contents will show. This is because scalars, unlike locals, are not "local," and are visible outside the program. Using a scalar inside your program is not a good idea, since as you apparently know, you don't want to overwrite the value of any scalar named "callhr" defined outside the program. The proper thing to do would be to make your scalar callhr a local, and return its value.

3) Here's a more Stata-ish way to construct your program, using return statements to makes various of your formatted values accessible outside the program. I could have made the following program more compact and better structured in various ways, but for my own convenience and to avoid using some fancier features of -return-, I decided to leave it as much like what you did as I could

webuse stan3, clear
version 18.0
cap program drop coxer
program define coxer, rclass
syntax varlist(numeric min=0 max=1) [if]
marksample touse
stcox `varlist' if(`touse')
tempname a
mat `a' = r(table)
matrix list `a'
// HR, 95% and pvalue
*--------------------------
local b = string(`a'[1,1], "%09.1fc")
local l = string(`a'[5,1], "%09.1fc")
local u = string(`a'[6,1], "%09.1fc")
local p = `a'[4,1]   
local n = e(N)

// Fix p-value formats and return them
*---------------------
if `p'>=0.056 {
   local pvalue = "= " + string(`p', "%03.2f")
}
if `p'>=0.044 & `p'<0.056 {
    local pvalue = "= " + string(`p', "%05.4f")
}
if `p' <0.044 {
    local pvalue = "= " + string(`p', "%04.3f")
}
if `p' <0.001 {
    local pvalue "< 0.001"
}
if `p' <0.0001 {
    local pvalue "< 0.0001"
}

// Create display text for the HR and return it
*---------------------------------
return local callhr = "(HR = `b'; 95% CI: [`l',  `u'];P = `pvalue')"
// Return all of the other values you formatted
return local pvalue = "`pvalue'"
return local b = "`b'"
return local l = "`l'"
return local u = "`u'"
return scalar n = `n'
end

// Test the program
*------------------
coxer transplant
return list

With a program like preceding, all the values in the return list will be available to use for any purpose, included being output with -putdocx- You don't *have* to return *all* of the values I did, of course, just the ones you might want to use.


-help local-, -help scalar-, and -help return- could add more detail and depth to the preceding.

[Clyde Schechter]

It is the very essence of a local macro that it is undefined outside the program where it is created. So you cannot display `b', nor any other macro defined in program coxer once you are outside that program. That's why you are getting no response to your -display "`b'"- command.

You have several options. Of these, I think the best is to return the value of `b' in the program:

webuse stan3, clear

// Program coxer below-------------------------
version 18.0
cap program drop coxer
cap drop scalar callhr
program define coxer, rclass
  syntax varlist(numeric min=0 max=1) [if]
  marksample touse
    stcox `varlist' if(`touse')
    tempname a
    mat `a' = r(table)
    matrix list `a'

// Gather HR, 95% and pvalue
*--------------------------
local b = string(`a'[1,1], "%09.1fc")
local l = string(`a'[5,1], "%09.1fc")
local u = string(`a'[6,1], "%09.1fc")
local p = `a'[4,1]
local n = e(N)

// Fix p-value formats
*---------------------
if `p'>=0.056 {
    local pvalue = "= " + string(`p', "%03.2f")
}
if `p'>=0.044 & `p'<0.056 {
    local pvalue = "= " + string(`p', "%05.4f")
}
if `p' <0.044 {
    local pvalue = "= " + string(`p', "%04.3f")
}
if `p' <0.001 {
    local pvalue "< 0.001"
}
if `p' <0.0001 {
    local pvalue "< 0.0001"
}

set trace on
// Create display text for the HR
*---------------------------------
scalar callhr = "(HR = `b'; 95% CI: [`l',  `u'];P = `pvalue')"
return scalar b = `b'
return scalar l = `l'
return scalar u = `u'
return scalar p = `p'
return scalar n = `n'
end

// Test the program
*------------------
coxer transplant
di "This is the effect of transplant callhr and checking `b'"
di "b = `r(b)'"
di "l = `r(l)'"
di "u = `r(u)'"
di "p = `r(p)'"
di "n = `r(n)'"

Now, the limitation of this approach is that the contents of r() are volatile. That is, you need to run these -display- commands before you run any other command that leaves results behind in r(); otherwise your program coxer results will disappear. For a more permanent solution, you can just define (rather than return) scalars b, l, u, p, and n. Scalars are part of the global information space in Stata and will persist until you (or some other program) change them. This, actually, is the approach you have taken with callhr. You saved it as a scalar. And if you run -di callhr- after your program as it was originally written, you do get correct output:

. display callhr
(HR = 0.3; 95% CI: [0.2,  0.4];P = < 0.0001)

As an aside, your sequence of -if- commands in the program steps through a ladder of ranges of p that are mutually exclusive. So only one of the -if- commands will ever be satisfied. So it would be clearer style to use -if ... else if ... else if ........ else ...- instead of a sequence of -if-'s. What you did is not an error, and changing it this way will not affect program performance, but the code would be more transparent, which, in itself, is a desirable thing.

Added: Crossed with #2, which offers essentially same advice.

[Girish Venkataraman]

Thanks much Mike Lacy for pointing me to return local and its use within the program. That worked perfectly. That will significantly streamline my workflow. Thanks again for giving me an extended explanation and thoughtful comments on my code beyond just providing a solution. It is a pleasure asking for input in this forum.

[Girish Venkataraman]

It is the very essence of a local macro that it is undefined outside the program where it is created. So you cannot display `b', nor any other macro defined in program coxer once you are outside that program. That's why you are getting no response to your -display "`b'"- command.

You have several options. Of these, I think the best is to return the value of `b' in the program:

webuse stan3, clear

// Program coxer below-------------------------
version 18.0
cap program drop coxer
cap drop scalar callhr
program define coxer, rclass
syntax varlist(numeric min=0 max=1) [if]
marksample touse
stcox `varlist' if(`touse')
tempname a
mat `a' = r(table)
matrix list `a'

// Gather HR, 95% and pvalue
*--------------------------
local b = string(`a'[1,1], "%09.1fc")
local l = string(`a'[5,1], "%09.1fc")
local u = string(`a'[6,1], "%09.1fc")
local p = `a'[4,1]
local n = e(N)

// Fix p-value formats
*---------------------
if `p'>=0.056 {
local pvalue = "= " + string(`p', "%03.2f")
}
if `p'>=0.044 & `p'<0.056 {
local pvalue = "= " + string(`p', "%05.4f")
}
if `p' <0.044 {
local pvalue = "= " + string(`p', "%04.3f")
}
if `p' <0.001 {
local pvalue "< 0.001"
}
if `p' <0.0001 {
local pvalue "< 0.0001"
}

set trace on
// Create display text for the HR
*---------------------------------
scalar callhr = "(HR = `b'; 95% CI: [`l', `u'];P = `pvalue')"
return scalar b = `b'
return scalar l = `l'
return scalar u = `u'
return scalar p = `p'
return scalar n = `n'
end

// Test the program
*------------------
coxer transplant
di "This is the effect of transplant callhr and checking `b'"
di "b = `r(b)'"
di "l = `r(l)'"
di "u = `r(u)'"
di "p = `r(p)'"
di "n = `r(n)'"

Now, the limitation of this approach is that the contents of r() are volatile. That is, you need to run these -display- commands before you run any other command that leaves results behind in r(); otherwise your program coxer results will disappear. For a more permanent solution, you can just define (rather than return) scalars b, l, u, p, and n. Scalars are part of the global information space in Stata and will persist until you (or some other program) change them. This, actually, is the approach you have taken with callhr. You saved it as a scalar. And if you run -di callhr- after your program as it was originally written, you do get correct output:

. display callhr
(HR = 0.3; 95% CI: [0.2, 0.4];P = < 0.0001)

As an aside, your sequence of -if- commands in the program steps through a ladder of ranges of p that are mutually exclusive. So only one of the -if- commands will ever be satisfied. So it would be clearer style to use -if ... else if ... else if ........ else ...- instead of a sequence of -if-'s. What you did is not an error, and changing it this way will not affect program performance, but the code would be more transparent, which, in itself, is a desirable thing.

Added: Crossed with #2, which offers essentially same advice.

This came over just as I was responding to #2. It took me a while to understand that locals don't survive programs like they do in include files as I learnt from you several months ago. I will mull over what you said and work through with refining the program with these thoughts. Thanks for pointing me to that pvalue display statements codes. I will fix them. Thanks again for taking the time.

[Girish Venkataraman]

Not to belabor this further, how do I refer to/access a defined scalar (created in above program) within a putdocx statement? I don't see it in the return list though. I have a sample output that does not seem to pull the defined scalar when within quotes as part of a sentence as I would have to do in a putdocx statement. I understand return local and return scalar thanks to this thread.

When I

 
 scalar callhr = "(HR = `b'; 95% CI: [`l', `u'];P = `pvalue')"

 Test the program *------------------ coxer transplant return list di "This is the effect of transplant callhr "

[Clyde Schechter]

When you refer to callhr inside the quotes, Stata doesn't know that you are referring to the scalar callhr: it thinks you are just talking about the string made of the characters c-a-l-l-h-r. Local and global macros work differently because they are set off by `' and $, respectively. But there is no analogous way to signal to Stata that callhr inside quotes is a scalar, and not the literal word callhr. The solution is to put callhr outside the quotes:

. di "This is the effect of transplant " callhr
This is the effect of transplant (HR = 0.3; 95% CI: [0.2, 0.4];P = < 0.0001)

[Girish Venkataraman]

I see. In that case, how do I make a putdocx statement capture the evaluated version of the callhr scalar as in the instance below? For purely display purposes, #7 seems fine. If there is not an easy way, I will stick to using return-based commands inside the program so I can call return locals inside the quotes of putdocx which are evaluated in the final output.

putdocx text ("This is the effect of transplant") callhr

[Nick Cox]

Here is some technique. I don't use putdocx but I suspect the last device here could be used in your command. Just add the scalar call at the end of your existing text.

. scalar sometext = "frog toad newt"

. di sometext
frog toad newt

. di scalar(sometext)
frog toad newt

. di "`= sometext'"
frog toad newt

On the original question, see also https://journals.sagepub.com/doi/pdf...36867X20931028

[Clyde Schechter]

Re #8: I do not use -putdocx- myself, and my famliarity with it is limited. So there may be another way to do this that I'm unaware of. But what I would do is include -return scalar summary_statement = callhr- inside program coxer, and then -putdocx text ("This is the effect of transplant: `r(summary_statement)'")- in the appropriate place outside of program coxer.

[Girish Venkataraman]

The last option worked well with the double quotes, Nick Cox. Thanks for the link to your local macro publication. I will read through it. Loved your write up on loops and gettoken, both of which I keep refreshing myself with. Wish I had your command of the English language as amply evident in your write-ups.

On the matter of putdocx, I have found it pretty useful (albeit cumbersome) thus far for writing transparent manuscript analysis/results portions, not to mention the convenience of having periodically updated results without having to rewrite any code. Is there a better/efficient option for inline scripting for something like that in Stata? I don't want to go Rmarkdown or Quarto both of which I dabbled with for a bit for just collating figures and figure paneling.

[Girish Venkataraman]

Re #8: I do not use -putdocx- myself, and my famliarity with it is limited. So there may be another way to do this that I'm unaware of. But what I would do is include -return scalar summary_statement = callhr- inside program coxer, and then -putdocx text ("This is the effect of transplant: `r(summary_statement)'")- in the appropriate place outside of program coxer.

Thanks again, Clyde Schechter. I will experiment with that option too and see how it works. Good to know you don't use putdocx either. Would love to know how you collate your final analysis scripts for manuscripts, although I know this is off-topic.

[Nick Cox]

Thanks for your nice comments. In secondary school I won precisely no prizes in Mathematics (they all went IIRC to https://en.wikipedia.org/wiki/Alan_Rouse -- later sadly famous for other reasons) but I won some for English.

My papers in the Stata Journal are just about the only publications in which I have cause to show directly any Stata commands and/or output. I don't use MS Word for that at all. I do use MS Word sometimes at work because that's a standard, but for teaching with Stata I usually use TeX.

So, I don't think what I usually do is close to what you are seeking. I would start a new thread on your new question.

[Clyde Schechter]

My reason for not using -putdocx- is similar to Nick's. I simply have no need for what it does in my workflow. The research I do these days takes place in large collaborating groups, and I am never the one assembling the final paper in Word. I simply pass on my results to those who do that part. Since the advent of the -table- command and its -dtable- and -etable- refinements, I am sometimes asked to provide my results as tables in a Word docoument, and I use these commands and -collect export- for that. But I am never asked to, in effect, compose the results section in Stata, so I have no need of what -putdocx- does. If I did need it, I'm sure I would learn it and use it.

[Girish Venkataraman]

Glad to know about both your use cases. In my situation, I am nearly exclusively doing all data cleaning, analysis and result write up as much as I have trained one of my students to run some do files across Dropbox. Guess it did make sense for me to learn putdocx. Will investigate TeX although I don't need to use much mathematical notations in my work.