Simulteneously executing multiple do files

[Oscar Ozfidan]

Suppose I have three do files with following commands in them
File diX.do : di "X"
File diY.do : di "Y"
File diZ.do : di "Z"
How do I execute them simultaneously in separate instances of Stata?
I created a do file as follows
do diX.do
do diY.do
do diZ.do
but it executes them in a single instance one after another.
I looked into package parallel but could not find the install package, I looked into multishell package which seemed to be geared toward running simulations, I looked into batch processing but that requires entering commands separately. I tried winexec command but that simply open the do file but does not execute it. What am I missing? I am sure there is a way.

[Nick Cox]

How would output differ in the way you want it? Stata's output is sequential, regardless of how the calculations are done.

[Oscar Ozfidan]

The win exec command I used was
foreach l in X Y Z {
winexec C:\Program Files (x86)\Stata14\StataSE-64.exe C:\Users\gfc87833\Documents\Personal\MacTahminleri \Soccer\Models\di`l'.do
}

[Oscar Ozfidan]

Basically the do files write the results to xlx and csv files. based on specs in each file. the main issue is invoking separate do files simultaneously from a single cmmand

[Oscar Ozfidan]

Hi Nick are you there? I am not sure I understand your point. Are you saying it cannot be done?

[Nick Cox]

You could open up multiple Statas and get them executing different do files. Whether that is a good idea depends on many details.

The winexec example isn't simultaneous at all. as you open Stata to call Stata three times to run three do files. That is extraordinarily puzzling code. I can't see any advantage of doing that over running a batch command from your operating system.

I haven't got more complicated than ever opening Stata and running do-files one by one, but I am of a generation who grew up on card deck input and lineprinter output hours later if you were lucky. And I don't ever deal with e.g. billions of observations and take the view that models that don't converge quickly must be poor models. .

Naturally you can nest do-files too.

[Leonardo Guizzetti]

Keep in mind that if you could parallelize execution and it made sense to do so, you still have one set of physical computing resources to be shared among your scripts. If you are trying to parallelize them because they are time-consuming and require complex calculations, you risk slowing down overall execution time as your computer is "pulled" in many competing directions.

[Oscar Ozfidan]

Nick,
First paragraph in your previous reply, check
Third paragraph, check,
Fourth paragraph, if you mean executing a do file through another it still is done through the same instance one after another,
Second paragraph, through Winexec I can launch multiple instances from a single do file but it opens the files but does not execute them. So, the trouble with winexec is getting the do files run rather than only opening.
The batch approach seems to require entering a line of command one after another for each file which is not helpful either. So, my guess is either I need a script to run multiple files in batch mode, or figure out a way to get multiple launched files to run after winexec.

[Oscar Ozfidan]

Keep in mind that if you could parallelize execution and it made sense to do so, you still have one set of physical computing resources to be shared among your scripts. If you are trying to parallelize them because they are time-consuming and require complex calculations, you risk slowing down overall execution time as your computer is "pulled" in many competing directions.

Leonardo, that is not an issue
Thi pic shows the cpu performance when running 30 files (running 3-4 hours each). I am sick of having to open 30 separate do files selecting the code and hitting the run button over and over again. And BTW this is with Stata MP 4-core running the files.

[Leonardo Guizzetti]

I don't doubt that each run of one file can take 3-4 hours, but are you sure resource sharing isn't an issue? From your picture, you have 24 cores at >98% utilization, for apparently, 30 processes of Stata MP/4. That means each Stata instance is sharing, on average, 0.8 cores?

[Oscar Ozfidan]

I don't doubt that each run of one file can take 3-4 hours, but are you sure resource sharing isn't an issue? From your picture, you have 24 cores at >98% utilization, for apparently, 30 processes of Stata MP/4. That means each Stata instance is sharing, on average, 0.8 cores?

That is during the most intensive phase of computing (large matrix operations phase) then cpu usage goes down to 35%-60%. Also, I am pretty sure MP4 does not make use of more than 4 cores per cpu. Before I upgraded to MP4 I had MP2 and most Cpu utilization I was getting with running 30 files was about 35%-50%. And when I run them on SE on a single core machine each file consumes about 33% of cpu. So each core should be able to handle 3 files at 100% of cpu.

[Oscar Ozfidan]

Problem solved! It appears I misspelled when I was trying to install parallel from ssc. But it do exist. The following code works for me. Also, in my set up I have a single main do file that takes in 8 local variables and scalars. Then I have 30 single line do files that pass those 8 parameters to that single main file with do commands and it works! Yeayy....

clear all
set more off
set trace off
parallel setclusters 3
program def myprogram
if ($pll_instance == 1) do "mydofile1.do"
else if ($pll_instance == 2) do "mydofile2.do"
else if ($pll_instance == 3) do "mydofile3.do"
end
parallel, nodata prog(myprogram): myprogram