Stata/MP 16.1 Processing Speed

[Omar Gondal]

Hi,
I am posting this to get some clarification on the time it takes Stata/MP 16.1 to run code. I was thinking of upgrading to Stata/MP 16.1 (2 Core) from Stata 15.1 SE, which involves a non-trivial amount of money but first obtained a trial version to test if it actually does make a difference. My computer has an i7-7Y75 Processor (1.3 GHz, 2 Core- 4 Thread), 16 GB RAM and a 256 GB SSD, and Windows 10.

I ran the code given at the end on both versions of Stata. The first time I ran it on Stata 16.1, it took ~78 seconds. However. when I reran the code, it took ~209 seconds and has not improved. I do not understand why it was so quick the first time and then started taking much longer. The same code takes ~211 second on Stata 15.1 SE.

About the code:
1) I have defined globals in a separate .do file, which I use to define paths to the datasets.
2) The datasets come from complete count US Census Data after deletion of observations and retaining relevant variables. I state the number of observations as in-line comments.
3) I loop over the following three steps three times (using the local census).

• I start with file "Census1940_Inctot.dta", rename certain variables and save as tempfile, which would allow a (M:1 merge in the next step)
• Then I load "Census_Merged_1910_1940_wide.dta", retain relevant variables and merge with the tempfile saved in the previous step.
• I save the dataset

Note: Stata was the only programme running when I recorded the time.

I would appreciate any help. Thank you.
Omar Gondal

timer clear
timer on 2
pause off
local census "1930 1920 1910"

foreach c of local census {
    use "${outputdir}/Census1940_Inctot.dta",clear   // 110,095 observations
    renvars state occ1950 race_ classwkr_ empstat_ inctot_avg_1940 , postdrop(5)
    renvars state occ1950 race classwkr empstat inctot_avg, postfix(_`c')
    
    isid stateicp_`c' occ1950_`c' race_`c' classwkr_`c' empstat_`c'
    
    tempfile Census1940_Inctot
    save    `Census1940_Inctot'    
    use "${outputdir}/Census_Merged_1910_1940_wide", clear // 3,208,571 observations
    keep *_`c'
        
    merge m:1 stateicp_`c' occ1950_`c' race_`c' classwkr_`c' empstat_`c' using `Census1940_Inctot', nogen  
        
    sa    "${outputdir}/Census`c'_Inc_Adjusted_wide.dta", replace
}

timer off 2
timer list 2

[Clyde Schechter]

I would not expect that code to show much performance improvement with MP compared to SE. The major bottleneck in that code is the reading and writing of large files--which means the bulk of the time is the mechanical response of the disk drive, the attentiveness of the operating system to Stata's requests for service, and, if the drives are over a network, the network speed. MP can't speed up any of that. Neither can a faster processor, or even entirely different software: that's all about the hardware and operating systems.

You do have two long computations in there: isid and merge both require sorting these large data sets. -sort- is 75% parallelized, so with enough cores you should see a speedup on these--but I think the time spent on those is overwhelmed by the time spent reading and writing those files.

The commands that are highly parallelized and really speed up a lot with MP are the estimation commands, of which there are none in your benchmark code.

You can find a detailed report of the impact of parallelization for most Stata commands, given the number of processors available, at https://www.stata.com/statamp/report.pdf.

[Omar Gondal]

Hi Clyde,
Many thanks for your response. I will go through the document in your post.
I was just surprised that Stata/MP 16.1 showed a marked improvement the first time I ran the code (nearly 3x as fast) and then slowed down. I just could not figure out why that happened.
In the code, I think, a majority of the time is spent running the merge and isid commands. That is probably due to the number of observations. I will try and run estimation commands next and draw a comparison.
Thank you.
Omar Gondal

[Kye Lippold]

I would recommend using gtools (https://gtools.readthedocs.io/en/latest/) and ftools (https://github.com/sergiocorreia/ftools) to speed things up, as those should substantially reduce the run time of isid (using gisid) and merge (using hashsort from gtools, or fmerge from ftools).

ssc install ftools
ssc install gtools

I believe both can take advantage of multiple cores. The gtools documentation shows the level of speedup on both IC and MP, and the numbers suggest that MP does provide a speed improvement over IC (but gtools is even better).

In terms of your code, it is somewhat odd that it runs slower the second time around. You could try putting -clear all- at the top of the script to try to get identical starting conditions. (My wild guess is that something from your script is being left in memory, maybe the tempfile, and that is slowing things down on the rerun).

That said, a good reason to upgrade to Stata 16 is to use frames--if you have the memory to load both files at once, then you could just load them as two frames and merge them with frget, which may be faster than merge (or even fmerge, although I haven't benchmarked this).