Parallel computing with Stata/SE 13.1 and parallel package: error -198- while setting memory

[Roberto Liebscher]

Dear Statalisters,

I am working with Stata SE 13.1 on a large dataset (15 gigabyte) on a Windows 64 machine with 32gb RAM. Because of the datasets' size some tasks take a little while. To increase performance I downloaded George Vega's and Brian Quistorff's parallel from ssc (ssc install parallel). From my understanding, what this package does is to split the dataset up, performs a task on the splitted datasets and then appends the individual datasets. That's what I wanted to do. I typed:

parallel setclusters 4
parallel do "C:\Users\wwa594\Documents\Promotion\Projects\Dualholding\Work\Do-Files\dh_holdings_03_1_bowadd.do", by(managerdum)

Unfortunately, this resulted in the following error messages:

--------------------------------------------------------------------------------
Parallel Computing with Stata (by GVY)
Clusters   : 4
pll_id     : 2uctinfg15
Running at : C:\Users\wwa594\Documents\Promotion\Projects\Dualholding\Work\Do-Files
Randtype   : datetime
Waiting for the clusters to finish...
cluster 0004 Exited with error -198- while setting memory (view log)...
cluster 0001 Exited with error -198- while setting memory (view log)...
cluster 0002 Exited with error -198- while setting memory (view log)...
cluster 0003 Exited with error -198- while setting memory (view log)...
--------------------------------------------------------------------------------
Enter -parallel printlog #- to checkout logfiles.
--------------------------------------------------------------------------------

Since dividing a 15gb dataset into four datasets should not exceed the 32gb memory limit I assume that memory cannot be the problem. I also tried parallel setclusters 2 and changing the called do-file so that all it does is a simple task:

forvalues i=1/5 {
          gen newvar`i' = .
}

giving the same error again.

Does someone have a hint for me where this problem might come from? I am grateful for any advice. For completeness I copy the text from the logfile:

. parallel printlog 1, e(2uctinfg15)
--------------------------------------------------------------------------------
beginning of file -C:\Users\wwa594\AppData\Local\Temp/__pll2uctinfg15_do0001.log-
--------------------------------------------------------------------------------

. capture {
--------------------------------------------------------------------------------
Parallel computing with stata (by GVY)
--------------------------------------------------------------------------------
cmd/dofile   : "dh_holdings_03_1_bowadd.do"
pll_id       : 2uctinfg15
pll_instance : 1/4
tmpdir       : C:\Users\wwa594\AppData\Local\Temp/__pll2uctinfg15_tmpdir0001/
date-time    : 10:55:58  5 Aug 2016
seed         : X25c52044c43f462544a474abacbdd93d00040fa3
--------------------------------------------------------------------------------

. local result = _rc

. if (c(rc)) {
. cd "C:\Users\wwa594\Documents\Promotion\Projects\Dualholding\Work\Do-Files/"
C:\Users\wwa594\Documents\Promotion\Projects\Dualholding\Work\Do-Files
. mata: parallel_write_diagnosis(strofreal(c("rc")),"C:\Users\wwa594\Documents\Promotion\Projects\Dualholding\Work\Do-Files/__pll2uctinfg15
> _finito0001","while setting memory")
. clear
. exit

end of do-file
--------------------------------------------------------------------------------
  end of file -C:\Users\wwa594\AppData\Local\Temp/__pll2uctinfg15_do0001.log-  
--------------------------------------------------------------------------------

[Sergiy Radyakin]

Strange, given that you don't have to set the memory in Stata 13 at all:

. set mem 1000g
set memory ignored.
    Memory no longer needs to be set in modern Statas; memory adjustments are performed on the fly automatically.

this succeeds on my machine which has slightly less memory than requested and in the above process Stata (seems like) doesn't verify whether it will be able to allocate the indicated amount, so it can't fail on set mem. The write_diagnosis procedure will just write the error, but the key should be inspecting the _rc.

Just a guess: Try it with Stata 12 and see if the same works. One possibility could be that the code requests allocation of certain amount of memory, then checks if it was actually allocated. That should work in Stata 12. In Stata 13 allocation request will be ignored and the actual memory allocation would be smaller than requested.

Another possibility is that the problem is not memory related at all since the message is issued regardless of the value of the _rc (at least I don't see any branching there).

Best, Sergiy

[Hua Peng (StataCorp)]

Also make sure you are running 64bit Stata instead of 32bit Stata, on 32bit Stata, -set memory- will produce an error if the value is larger than 4g

. about

Stata/MP 13.1 for Windows (32-bit)
Revision 10 Mar 2016
Copyright 1985-2013 StataCorp LP

Total physical memory:     2097151 KB
Available physical memory: 2097151 KB

. set mem 3g
set memory ignored.
    Memory no longer needs to be set in modern Statas; memory adjustments are
    performed on the fly automatically.

. set mem 5g
value too large
r(912);

[Roberto Liebscher]

Thanks for your responses Sergiy Radyakin and Hua Peng (StataCorp). I did the "Stata" version check Hua suggested. It turns out it's the 64bit version I am using:

Stata/SE 13.1 for Windows (64-bit x86-64)
Revision 10 Mar 2016
Copyright 1985-2013 StataCorp LP

Total physical memory:    33435040 KB
Available physical memory: 6250772 KB

Inspired by Sergiy's idea to run the script on Stata 12 I changed the version control in the ado file from version 11.0 to version 13.1 but without any changes. I would love to try it with an older Stata version but for an unjustified reason I deinstalled Stata 12 from my notebook. I have a copy in my office but will be unable to install it before Tuesday.

However, I tried to replicate the error with a toy example in Stata 13 which was an easy to achieve task. I would be interested to know whether this script works for you guys? Does it provide any insights?

clear
sysuse auto
expand 2000000
parallel setclusters 2
sort rep78
parallel do "C:\Users\wwa594\Documents\Untitled1.do", by(rep78)

And the called script "Untitled1.do" looks like this:

levelsof rep78, local(reps)
foreach r of local reps {
    sum mpg if reps==`r'    
    gen newvar`r' == r(sum)
}

Thanks for your help.

[Dimitriy V. Masterov]

Your code in Untitled1.ado will not work as is, which you can verify by running it on its own. With my edits, it seems to work. I also show you some alternative ways to get this to work using a tiny program, which avoids the nonsense with the multiple files.

. sysuse auto, clear
(1978 Automobile Data)

. expand 20
(1,406 observations created)

. parallel setclusters 2
N Clusters: 2
Stata dir:  /Applications/Stata/StataMP.app/Contents/MacOS/StataMP

.
. /* Method 1 */
. type "~/Desktop/Untitled1.do"
levelsof rep78, local(reps)
foreach r of local reps {
    sum mpg if rep78==`r'    
    gen newvar`r' = r(sum)
}

. sort rep78

. parallel do "~/Desktop/Untitled1.do", by(rep78)
--------------------------------------------------------------------------------
Parallel Computing with Stata (by GVY)
Clusters   : 2
pll_id     : 7v7i8x2678
Running at : /Users/dmasterov/Desktop
Randtype   : datetime
Waiting for the clusters to finish...
cluster 0001 has exited without error...
cluster 0002 has exited without error...
--------------------------------------------------------------------------------
Enter -parallel printlog #- to checkout logfiles.
--------------------------------------------------------------------------------

. rename newvar* old_newvar*

.
. /* Alternatively */
. capture program drop myloop

. program define myloop
  1.         levelsof rep78, local(reps)
  2.         foreach r of local reps {
  3.                 sum mpg if rep78==`r'    
  4.                 gen newvar`r' = r(sum)
  5.         }
  6. end

.
. sort rep78

. parallel, by(rep78) programs(myloop): myloop
--------------------------------------------------------------------------------
Exporting the following program(s): myloop

myloop:
  1.         levelsof rep78, local(reps)
  2.         foreach r of local reps {
  3.                 sum mpg if rep78==`r'
  4.                 gen newvar`r' = r(sum)
  5.         }
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Parallel Computing with Stata (by GVY)
Clusters   : 2
pll_id     : et7u439i80
Running at : /Users/dmasterov/Desktop
Randtype   : datetime
Waiting for the clusters to finish...
cluster 0002 has exited without error...
cluster 0001 has exited without error...
--------------------------------------------------------------------------------
Enter -parallel printlog #- to checkout logfiles.
--------------------------------------------------------------------------------

.
. compare old_newvar1 newvar1

                                        ---------- difference ----------
                            count       minimum      average     maximum
------------------------------------------------------------------------
old_new~1=newvar1             800
                       ----------
jointly defined               800             0            0           0
jointly missing               680
                       ----------
total                        1480

. parallel clean

.
end of do-file

[Roberto Liebscher]

Thanks Dimitriy V. Masterov for this insightful example and for correcting my typos (I wrote reps==`r' instead of rep78==`r' and gen newvar`r' == r(sum) instead of gen newvar`r'= r(sum)) I also like the idea of writing a short program that is passed to the parallel function instead of refering to an extra file. With the dataset expanded 20 times is works like charm.

However, when I take the example and expand 2 000 000 times instead of 20 times I get strange results although it exits without an error. rep78 now has only four categories instead of five in the original dataset and the resulting dataset is 60 million observations short: 2 mio * 74 - 88 mio = 60 mio.

. sysuse auto, clear
(1978 Automobile Data)

. tab rep78, miss

     Repair |
Record 1978 |      Freq.     Percent        Cum.
------------+-----------------------------------
          1 |          2        2.70        2.70
          2 |          8       10.81       13.51
          3 |         30       40.54       54.05
          4 |         18       24.32       78.38
          5 |         11       14.86       93.24
          . |          5        6.76      100.00
------------+-----------------------------------
      Total |         74      100.00

. expand 2000000
(147999926 observations created)

.
. capture program drop myloop

. program define myloop
  1.   levelsof rep78, local(reps)
  2.         foreach r of local reps {
  3.                         sum mpg if rep78==`r'    
  4.                         gen newvar`r' = r(sum)
  5.   }
  6. end

.
. cd "C:\Users\wwa594\Documents"
C:\Users\wwa594\Documents

. sort rep78

. parallel setclusters 4
N Clusters: 4
Stata dir:  C:\Program Files (x86)\Stata13/StataSE-64.exe

. parallel, by(rep78) programs(myloop): myloop
--------------------------------------------------------------------------------
Exporting the following program(s): myloop

myloop:
  1.   levelsof rep78, local(reps)
  2.         foreach r of local reps {
  3.                         sum mpg if rep78==`r'
  4.                         gen newvar`r' = r(sum)
  5.   }
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Parallel Computing with Stata (by GVY)
Clusters   : 4
pll_id     : 1mdx8agr10
Running at : C:\Users\wwa594\Documents
Randtype   : datetime
Waiting for the clusters to finish...
  0
cluster 0001 has exited without error...
  0
cluster 0004 has exited without error...
  -3621
cluster 0003 has exited without error...
  -3621
cluster 0002 has exited without error...
--------------------------------------------------------------------------------
Enter -parallel printlog #- to checkout logfiles.
--------------------------------------------------------------------------------

. parallel clean

. tab rep78, miss

     Repair |
Record 1978 |      Freq.     Percent        Cum.
------------+-----------------------------------
          1 |  4,000,000        4.55        4.55
          2 | 16,000,000       18.18       22.73
          4 | 36,000,000       40.91       63.64
          5 | 22,000,000       25.00       88.64
          . | 10,000,000       11.36      100.00
------------+-----------------------------------
      Total | 88,000,000      100.00

I assume "-3621" in the routine's output is supposed to tell me something?

I also tried the same code with Stata 11 SE. Here, I received an error message saying that part of the program has been compiled under Stata 12.1 and can therefore not be run in Stata 11:

. parallel setclusters 4
(parallel_setclusters() in lparallel, compiled by Stata 12.1, is too new to be run by this version of Stata and so was ignored)
                 <istmt>:  3499  parallel_setclusters() not found
r(3499);

With my original dataset I tried to run the following code snippet:

encode manager, gen(managerdum) // Convert string into numeric variable

capture program drop myloop
program define myloop

//walk through each lead and find amount with this lead if lead is in of the lead* variables
forvalues i = 1/12 {
    gen leadamt`i' = .
    levelsof lead`i', local(leads)  
    foreach l of local leads {
        bysort managerdum obsq: egen hlpvar = total(principalUSD) if (lead1 == "`l'" | lead2 == "`l'" | lead3 == "`l'" | lead4 == "`l'" | lead5 == "`l'" | ///
         lead6 == "`l'" | lead7 == "`l'" | lead8 == "`l'" | lead9 == "`l'" | lead10 == "`l'" | lead11 == "`l'" | lead12 == "`l'") & obsdate == mindate
        bysort managerdum obsq: egen totamt = min(hlpvar)
        replace leadamt`i' = totamt if lead`i' == "`l'" & leadamt`i' == .
        drop totamt hlpvar
    }
}
end

capture parallel clean
cd "C:\Users\wwa594\Documents\"
parallel setclusters 4
sort managerdum
parallel, by(managerdum) programs(myloop): myloop

This yield the memory error again. I have got no idea what to do here. Any help is highly appreciated.

[Dimitriy V. Masterov]

Yeah, it seems to screw things up when the sample is that large. Not sure yet sure what's going wrong. Might be worth putting an error report at their GitHub.

[Roberto Liebscher]

Will do that. Thanks for your help.

[Dimitriy V. Masterov]

I installed the dev version of -parallel- from GitHub, and now the code mostly works with -expand 2000000-. It still seems to process 1 & 2 together, but now at least all the categories are accounted for in the final output.

I am still not sure what was going wrong. From looking at the log files, the threes were being processed, but were not being appended somehow.

[Jesse Wursten]

I don't really see the benefit of using parallel here. There are in my opinion two reasons to run things parallel - insufficient memory or insufficient CPU power. Running things parallel on the same machine solves neither. You do not have a memory issue (15GB < 32GB) and StataMP already runs on multiple cores. You might gain some efficiency there depending on the command, but it's unlikely to be mind-shattering.

[Sergiy Radyakin]

Jesse, imho the benefit of -parallel- was to

1. run commands that are not parallelized (see the MP report) in parallel.
2. run SE/IC in addition to MP in parallel.

Best, Sergiy

[Roberto Liebscher]

I totally agree with Sergiy Radyakin.

Thanks Dimitriy V. Masterov for pointing me to the dev version of parallel:

net install parallel, from(https://raw.github.com/gvegayon/parallel/master/) replace
mata mata mlib index

The toy example works now with expand 2000000. For the original dataset I still receive the error message. Downsizing the dataset by getting rid of unnecessary variables and observations where all of the lead* variables are missing resulted in a much smaller dataset of 3gb. Running the code on this dataset does not pose any issues. So my approach is now as follows:

• Downsize the dataset,
• Run the program on the smaller dataset,
• Merge the original dataset with the dataset resulting from the program, keepusing the new variables.

Since I am unable to replicate the error message using an example file I think I will leave it at that for the moment. Thank you all for your help. Great community here.

[Rodrigo Garcia Ayala]

Hi,
Hope you can kindly help me with this issue.
I am trying to parallelize a process which needs to be done over many stats (mean, median, etc), years, geographical areas and countries using microdata from national population census. This amounts to way to many loop iterations. However, I haven't been able to use parallel effectively. In particular, I want to take advantage of the `parallel append` feature.
I am trying to do something like the command below for a bunch of files located within a folder, under the directory labeled by the global $microdat, such that the collapsed_d program runs on each individual file.

parallel append , do("collapsed_d.do") ///
prog(collapsed_d) e("$microdata/filename_`g'.dta")

where `g' corresponds to each of the names that I want to process in parallel.

I have couple of issues where I got stuck. First, I am not sure how use the option e(). The example in the Stata helper hasn't been helpful because of how files are stored in the example.
Secondly, I am not sure this is the most efficient way to deal with the challenge. Any thoughts on this would also be highly appreciated.

Thank you in advance.
Rodrigo