How do I display bootstrapped values in Stata

[Carlos Echevarria]

Dear all,

I am new to Stata and new to this forum- greetings!

I am trying to perform a bootstrap analysis of the difference in costs between two groups, and then export the 1000 bootstrap results into excel. I need to transfer this information into excel in order to insert the information to generate a CEAC which is a type of health economics analysis.

I have managed to figure out how to perform the analysis and display the results relatively easily. However, despite 2 hours reading online I am unable to display the actual 1000 bootstrap values so that I can export them. I am fairly sure that all I need is a simple command, but despite reading through several articles and look at several fora I am stuck.

Any help is greatly appreciated!

Carlos

[Carlo Lazzaro]

Carlos:
welcome to the list.
You might be interested in something along the following lines:

. sysuse auto.dta
(1978 Automobile Data)

. ttest price, by(foreign) unequal

Two-sample t test with unequal variances
------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
Domestic |      52    6072.423    429.4911    3097.104    5210.184    6934.662
 Foreign |      22    6384.682    558.9942    2621.915     5222.19    7547.174
---------+--------------------------------------------------------------------
combined |      74    6165.257    342.8719    2949.496    5481.914      6848.6
---------+--------------------------------------------------------------------
    diff |           -312.2587    704.9376               -1730.856    1106.339
------------------------------------------------------------------------------
    diff = mean(Domestic) - mean(Foreign)                         t =  -0.4430
Ho: diff = 0                     Satterthwaite's degrees of freedom =  46.4471

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.3299         Pr(|T| > |t|) = 0.6599          Pr(T > t) = 0.6701

. return list

scalars:
              r(level) =  95
                 r(sd) =  2949.49588476892
               r(sd_2) =  2621.915083190759
               r(sd_1) =  3097.104279086425
                 r(se) =  704.937643377471
                r(p_u) =  .6700743053648903
                r(p_l) =  .3299256946351096
                  r(p) =  .6598513892702191
                  r(t) =  -.4429593797298984
               r(df_t) =  46.44711321147706
               r(mu_2) =  6384.681818181818
                r(N_2) =  22
               r(mu_1) =  6072.423076923077
                r(N_1) =  52




. bootstrap (r(mu_1)-r(mu_2)), reps(1000) strata(foreign) saving(C:\Users\user\Desktop\bootstrap.dta, every(1) double replace) bca nodots : ttest price, by(foreign) unequal

Warning:  Because ttest is not an estimation command or does not set e(sample), bootstrap has no way to
          determine which observations are used in calculating the statistics and so assumes that all
          observations are used.  This means that no observations will be excluded from the resampling
          because of missing values or other reasons.

          If the assumption is not true, press Break, save the data, and drop the observations that are to be
          excluded.  Be sure that the dataset in memory contains only the relevant data.
(note: file C:\Users\user\Desktop\bootstrap.dta not found)

Bootstrap results

Number of strata   =         2                  Number of obs     =         74
                                                Replications      =      1,000

      command:  ttest price, by(foreign) unequal
        _bs_1:  r(mu_1)-r(mu_2)

------------------------------------------------------------------------------
             |   Observed   Bootstrap                         Normal-based
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       _bs_1 |  -312.2587   679.6921    -0.46   0.646    -1644.431    1019.913
------------------------------------------------------------------------------

. estat bootstrap, all

Bootstrap results
Number of strata   =          2                 Number of obs     =         74
                                                Replications      =       1000

      command:  ttest price, by(foreign) unequal
        _bs_1:  r(mu_1)-r(mu_2)

------------------------------------------------------------------------------
             |    Observed               Bootstrap
             |       Coef.       Bias    Std. Err.  [95% Conf. Interval]
-------------+----------------------------------------------------------------
       _bs_1 |  -312.25874  -9.658969   679.69212   -1644.431   1019.913   (N)
             |                                      -1682.709   978.2037   (P)
             |                                      -1681.502   978.2037  (BC)
             |                                      -1702.757   967.1469 (BCa)
------------------------------------------------------------------------------
(N)    normal confidence interval
(P)    percentile confidence interval
(BC)   bias-corrected confidence interval
(BCa)  bias-corrected and accelerated confidence interval

. use "C:\Users\user\Desktop\bootstrap.dta", clear
(bootstrap: ttest)

. export excel using "C:\Users\user\Desktop\bootstrap.xls"
file C:\Users\user\Desktop\bootstrap.xls saved

As in all likelihood you are a colleague of mine dealing with the economic evaluation of health care programmes, you should spell out what CEAC means (Cost-Effectiveness Acceptability Curve) (one of my favourite references on this topic is: https://www.ncbi.nlm.nih.gov/pubmed/11747057),as this list is not focussed on health economics. Thanks.
As a closing-out remark, if you're going to generate a CEAC, you also need the bootstrapped difference in effectiveness between the two groups, isn'it?

[Carlos Echevarria]

Hi Carlo,

Thanks very much for the information.

I am using the bootstrap for the difference in the costs and the effects (adjusting for baseline.)

I have an excel spreadsheet which will produce the Cost-Effectiveness Acceptability Curve (apologies for previous abbreviation), so I need to get the RAW DATA (i.e. the values produced by the bootstrap procedure for each) and transfer them across to produce a graph.

I think the command I need is something similar to "estat bootstrap all", but this does not quite display what I need. Just to be completely clear, it is the 1000 bootstrap values that I want to see, not the end result.

I know this is probably straight forward for those familiar with stata. (I am used to SPSS, but SPSS cannot provide the calculated bootstrap values).

Kind regards,

Carlos

[Carlo Lazzaro]

Carlos:
the previous code saves the bootstrap replications on a dedicated file (bootstrap.dta), which can be exported in Excel via the two last lines of code.

[Carlos Echevarria]

Hi Carlo,

I thought this might be the case, but I am unable to get it to work. I am performing bootstrapping separately for each variable. Here is the first variable:
bootstrap, reps (1000) sa ("name of file", replace) seed (10001): regress Outcome Allocation Therefore, I assume that I need to change the "ttest" component of the following?!
use "C:\Users\user\Desktop\bootstrap.dta", clear (bootstrap: ttest)

Thank you very much for your help!

[Carlo Lazzaro]

Carlos:
yes, probably the problem lies there.
However, to allow other listers to diagnose what the problem actually is, you would be better off with posting what you typed and what Stata gave you back (as per FAQ). Thanks.

[Carlos Echevarria]

Thanks for all your help, Carlo.

Regrettably, I am using stata remotely so I cannot copy and paste the results.

I will have a bit more of a read around what you have posted and see if I can figure it out (though any other suggestions in the meantime are welcome!)

Carlos

[Carlo Lazzaro]

Carlos:
take a look at -bootstrap- entry in Stata .pdf manual.
About the theoretical side of the matter you're after, you probably already know: https://www.ncbi.nlm.nih.gov/pubmed/10180748

[Carlos Echevarria]

Hi,

I think that the problem I am having is a technical one. I have been informed that the list of bootstrap values essentially should generate automatically. The fact that this is not happening may be because I am accessing Stata remotely.

Thanks again for the help, Carlo.

Carlos

[Carlo Lazzaro]

Carlos:
you can access the list of bootstrap replications only if you save them on dedicated file (please, see example in my previous reply). I would check whether that step has been performed.

[Carlos Echevarria]

Hi Carlo,

When I tried this on a copy of Stata with the version saved on the computer (as opposed to via a remote access system) it worked first time. All I did was click on the file following the bootstrap procedure and the replications were there visible within stata. I am still not sure why I couldn't get it to work before, but anyway, the problem is now solved!

Carlos

[Carlo Lazzaro]

Carlos:
happy with reading that you've fixed the problem.

[Elena Mvk]

Carlos:
welcome to the list.
You might be interested in something along the following lines:

As in all likelihood you are a colleague of mine dealing with the economic evaluation of health care programmes, you should spell out what CEAC means (Cost-Effectiveness Acceptability Curve) (one of my favourite references on this topic is: https://www.ncbi.nlm.nih.gov/pubmed/11747057),as this list is not focussed on health economics. Thanks.
As a closing-out remark, if you're going to generate a CEAC, you also need the bootstrapped difference in effectiveness between the two groups, isn'it?

Dear Carlo,

I am also conducting a Cost-Effectiveness Analysis of data from a randomised controlled trial, I have gotten to estimating the ICER (via a mixed multilevel model). The next step is to have my results on a Cost-Effectiveness Plane.
As I can see from your post above, I need to bootstrap the cost and effect and replicate 5,000.
1) is it possible to do it in one line of code (can I estimate 5000 ICERs, or do I need to estimate costs and effects separately) ?
2) is it possible to then plot them across the 4 quadrants?
3) is it possible to also estimate the probability of CE at different willingness to pay thresholds?
And to do all of this in Stata, or do I need to use other software?

Thank you very much.

Elena

[Pengpeng Ye]

Dear Carlo,

I am also conducting a Cost-Effectiveness Analysis of data from a randomised controlled trial, I have gotten to estimating the ICER (via a mixed multilevel model). The next step is to have my results on a Cost-Effectiveness Plane.
As I can see from your post above, I need to bootstrap the cost and effect and replicate 5,000.
1) is it possible to do it in one line of code (can I estimate 5000 ICERs, or do I need to estimate costs and effects separately) ?
2) is it possible to then plot them across the 4 quadrants?
3) is it possible to also estimate the probability of CE at different willingness to pay thresholds?
And to do all of this in Stata, or do I need to use other software?

Thank you very much.

Elena

Dear Elena，

I have met similar challenges recently and gong through all relevant posts at the Stata Forum. I did not find reply from Dr.Carlos (his replies in other questions are very helpful), and I made some codes below to deal with your first two questions based on my personal experience (not so sure whether the codes below are proper or not, only for your information). Would you share the solution with us If you had solved your questions?

Dear Dr. Carlos,

Would you mind if you could provide the solution to those questions or any comment on the code below when you see this post?

In orde to use bootstrap command to generate ICER, I made two files, one ado file and one do file.

1. For the ado file named icer.ado

// icer.ado
program define icer, rclass
    preserve
    ttest cost, by(group) unequal // intervention group V.S. control group
    local cdiff=r(mu_1)-r(mu_2) // get the cost difference from the T-test based on Carlos's reply above
    
    ttest effect, by(group) unequal
    local ediff=r(mu_1)-r(mu_2) // effectiveness difference
    
    local ratio=`cdiff' / `ediff'
    di `cdiff' " " `ediff' " " `ratio'
    
    return scalar cdf=`cdiff'
    return scalar edf=`ediff'
    return scalar r=`ratio'
    restore
end

2. For the do file

// do file
cd "" // to your directory
clear all
use *.dta,clear // using your own dataset containing cost variable and effectiveness variable for each observation in a long format form

bootstrap costdiff=r(cdf) effectdiff=r(edf) icer=r(r), reps(1000) saving(bootstrap.dta, every(1) double replace) bca nodots : icer // get the estimated results of ICER, for the 1st question

use bootstrap.dta,clear
twoway (scatter costdiff effectdiff, sort) // guess this is the cost-effectiveness plane for the 2nd question

[Carlo Lazzaro]

I totally missed this follow-up question.