Use of -simulate- command for cost benefit analysis

[Moh Hussein]

Hello, I would like to learn about how to use -simulate- command to estimate impact of different policy intervention options, using data with high uncertainty.

Can you direct me please to any available Stata Press or other publications that can provide good overview and examples of how organise data and run such analysis? My data is several Excel spreadsheets within the same workbook and contains formulas for calculation of annual costs and benefits across different time horizons and discount rates.

Thanks,
Mohamud

[Mead Over]

I am just seeing this query now. Sorry you did not receive a more timely answer. I am responding in case you are still interested or others might be led to this query and not find a response.

While Stata's forte is clearly statistical analysis, I believe Stata is a good platform for performaing cost-benefit, cost-effectiveness or return-on-investment analysis of alternative policy scenarios, especially if the time-period for the simulation is long and one wants to perfom sensititiviy analysis over a great many alternative parameter combinations.

My package -AIDSCost- uses Stata to compare alternative policy scenarios for HIV-AIDS prevention and treatment in low and middle income countries. My students in my courses at Sciences-Po, the University of the Auvergne and Georgetown's School of Foreign Service have used this package or its ancestors to analyze diverse policy scenarios in various AIDS-affected countries for the last 25 years. And I have used it in my own research. While less sophisticated than Specturm, which is the dominant specialized package in this space, -AIDSCost- is in some ways more transparent and easier to use, especially for masters-level students who are not specialized in demography or epidemiology.

My package is available from inside Stata by issuing this command:

. view net describe AIDSCost, from("http://digital.cgdev.org/doc/stata/MO/AIDS")

I doubt whether you could use my package to model any policy scenario oher than the ones for which it was designed. However, you might want to look at my package for more general lessons though, and especially for the dialogs that you can access by typing:

db aidsproj

or

db aidsdif

Here are a few lessons I've learned that might be useful in starting your own project.

1. Build the simulation model first, saving the results of each policy scenario as a -tsset- Stata data file. Then write a separate program (i.e. ado file) to perfom the cost-benefit, cost-effectiveness or return-on-investment analysis. (The package -AIDSCost- includes the program -aidsproj- to compute the scenarios until the year 2050 and the program -aidsdif- to perform the economic analysis.)
2. The program that generates your scenarios will have some kind of "production function" that converts policy choices into the outcomes that constitute benefits. Ideally this "production function" could be esitmated by a regression relating outcomes to input parameters or variables from some kind of experimental or non-experimental data generating process. In this case, Stata's -simulate- will be particulalrly useful, allowing you to exploit Stata's stochastic features. (Perhaps this is exactly what you had in mind when you posed this question.) For AIDSCost, since data to plausibly estimate the production functions did not exist, the "production functions" are non-stochastic difference equations, which Stata can easily use to project the outcome of a policy far into the future.
3. Allow the user substantial flexibility in choosing the "baseline" or "counterfactual" scenario against which all other scenarios are compared. The choice of counterfactual will, of course, have a major effect on the desirability of most investments.
4. Consider the granularity of the analysis. In my case, it is useful to model the individual country with annual data and to allow the use to project the benefits and costs for groups of countries. But for another problem, e.g. modeling Covid policy, the appropriate granularity might be the individual person and the time period might be the day or week.
5. If you want to allow for stochastically generated future scenarios, exploit Stata's statisitical strengths to design them into both the scenario generation package and the economic analysis package.
6. Integrate an approriate cost function into the scenario building model. The usual practice in dynamic cost-benefit analysis is to assume unit costs and simply multiply those units into the future, ignoring economies of scope, learning by doing, economies of scale and other issues that plausibly affect cost over time. But constructing a plausible non-linear cost model is often, in my experience, the most challenging and potentially original part of dynamic cost-benefit analysis.
7. If the benefits of your policy include saving lives, allow the user to choose whether to use cost-effectiveness analysis (counting only e.g. life-years saved) or cost-benefit analysis (which requires valuing the health benefits in dollar terms). Each serves the need of a different audience.
8. Include the capability of producing graphs of the projections (by the scenario projection model) and the comparative analysis (by the economic analysis model). Automatic production of the "Incremental Cost-Effectiveness Ratio" plot is particulalry useful for diagnosing logical problems with the scenario definitions.
9. If you intend to experiment with a large number of alternative parameter combinations, I recommend coding a Stata dialog for each of your -ado- files. The dialog will help you or other users experiment with policy combinations in real time.

If you believe that the most useful policy simulation approach would be to model the behavior of individual "agents", take a look at this presentation at the 2019 Stata sers' Conference in Berlin by Maarten Buis entitled: "Agent based models in Mata"
https://www.stata.com/meeting/germany19/

There are also two user-written program on SSC that you might find useful: -fxbcr-, -heabs-. I have not tried either of these.

Best of luck to you - or to others addressing the same kind of analytical challenge. If other Statalist participants have experience using Stata to perform dynaic or stochastic cost-benefit analysis, I would be interested to learn of their experiences on this thread.

Reference to Spectrum: https://avenirhealth.org/software-spectrum.php . Another major competitor in this space is of course Simulink.