Using multiple instances of Stata on a windows Server

[Adrian Sayers]

Hi All,

I was wondering if anyone had any experience in running multiple instances of Stata on a Windows 2019 Server.


I am using an adapted version of a program I wrote a while ago called qsub, it does a similar thing to batcher, multishell, or parallel.

Basically, the program shell jobs new instances of Stata and runs them in parallel using winexec or shell.


The server i am using has 128 physical cores, split across 4 nodes with 300GB RAM.

I designed a simple stress test of drawing and adding random numbers, when i run this on one specific CPU it runs around 100% activity.

I initiate my qsub program and it starts spawning Stata's. However, after the first 10 instances have been spawned the spawning slows to a crawl and the CPU activity falls dramatically.

My original qsub program was only spawning to Node 0 and relied on windows to assign the processors and was using winexec to do the spawning.

the new qsub program uses shell and and the "start" command to distribute to different numa nodes and control the affinity of each of the CPU allocations specifically.

                       
                           if "`numa'"=="" {
                             winexec `statadir' /e /q do "`jobdir'\\`nextjobname'"
                          }
                          else {
                            affinitymask `processor' -1
                            ! start "Stata" `nodeopt' `group' `affinityopt' `s(mask)' /realtime /MIN /B  "`statadir'"   do "`jobdir'\\`nextjobname'"
                          }

The lack of performance feels like windows server is throttling the distributions

The server has fair share switched off.
I have stressed tested the server using Prime95 which searches for primes. When i run this the server spins up and all CPUs are activated to 100% and RAM usage maxes out.

I understand hyperthreading wont help because there is only 1 floating point number per CPU.

The stress test utilises

 set processors 1

so there isnt conflict or CPU switching.

I can see there was some discussion of something similar in 2020. I just wondered if it was resolved?

https://www.statalist.org/forums/for...erating-system

Alan Riley (StataCorp), Hua Peng (StataCorp) JanDitzen Oscar Ozfidan

Any help or suggestions would be greatly received.

thanks

Adrian

[Bjarte Aagnes]

Two comments: maybe setting priority to /HIGH is better. Also, does additional processes exhaust CPU and/or memory? In one project we used -sysresources- from https://github.com/wbuchanan/StataOS to start a new process only if sufficient memory and processors was available, like:

sysresources

confirm file `exefile'
confirm file `dofile'

assert $N_imputations > 0

local args \`site' \`i'

qui foreach site of numlist 1/23 {
    
    forvalues i=1(1)$N_imputations {
        
        sleep `=10*1000'
        
    
        while 1 {
            
            sysresources

            if ( r(pctfreemem) < 25 | r(cpuload) > 0.75 ) {
                
                sleep `=30*1000'            
                continue
            }
            
            else {
                    
                winexec `exefile' /e do `dofile' `args'
                
                noi di "started site `site' and imputation `i': " c(current_time)
                
                continue, break
            }
        }
    }
}

[Adrian Sayers]

Hi Bjarte,

thats handy piece of code, to throttle jobs .

I don't think its system resource. All the parameters look well within normal.

This was the output when underload.

Total committed memory is : 2.4 GiB
Free Swap Space is : 230.3 GiB
Total Available Swap Space is : 283.5 GiB
Process CPU Time is : 91078125000
Free Physical Memory Available is : 214.0 GiB
Total Physical Memory Available is : 255.6 GiB
System CPU Load is : 0.06817795378416314
Process Load is : 9.853441403530339E-4
% Free Physical Memory is : 83.70570690469283

[Adrian Sayers]

Hi,

I have been running performance reports whilst running a stress test and it seems like the page filling is growing quite quickly, but for no apparent reason as i a simply drawing random numbers.

There is nothing actually in memory, and i think the page file increase corresponds to every new Stata window that is being opened.

Does anyone know how page filling and Stata work when starting?

I guess that Stata reserves some page filing space proportional to the size of the data it can load but perhaps does this to a theoretical maximum?

Any thought would be greatly received.

thanks

Adrian

[Oscar Ozfidan]

I have not logged in for quite sometime and I see this now. I had no performance issues using windows 2019 server version (40 physical cores double threaded so really 80) and using batcher. batcher had some inherent issues I fixed for my own use and sent my code to the author but I am not sure it is all incorporated. I can send the version I fixed if you need it.

[Oscar Ozfidan]

Now this is all coming back to me so excuse the intermittent response. I was experiencing performance issues when I utilized more than about 20 instances at the same time. I think 24 is the most I used without a problem once or twice. Once I realized that I kept it below the threshold and it kept humming like bird.

[Adrian Sayers]

Hi Oscar, i think the sweet spot is somewhere between 18 -20 also .