Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Finding out available memory in a Unix system

    I am on a shared Unix server and I want to run several Stata jobs in batch mode. I have a loop that generates these batch jobs and executes them. The problem is that all the Stata batch processes are executed simultaneously, overloading the server and depriving my colleagues of much needed memory.

    I want to find out how much memory the Unix system has at a given time, then execute if there's enough memory available. I wrote this piece of code to handle that, using Stata-MP 13.1 on Unix

    Code:
    clear
    * Program to determine amount of free memory
    cap program drop freemem
    program define freemem, rclass
        preserve
        quietly {
            cap erase free.txt
            - Vmstat gives available memory
            shell (vmstat)>free.txt
            clear
            import delimited free.txt, delimiters("whitespace") rowrange(3:3) varnames(nonames)
            keep v1
            split v1
            keep v14
            li
            loc fmem = v14[1]
            loc fmem = `fmem'/1e6
            noi di "`fmem'"
            return local fmem=`fmem'
            restore
        }
    end
    
    * Test; Run regression only if memory exceeds a value
    
    loc exceed=164.7
    foreach i in a b c d{
        loc run=0
        while `run'==0 {
            freemem
            loc fm=r(fmem)
            if `fm'>`exceed' {
                di "Available mem = `fm' - clear to run"
                sysuse auto, clear
                reg price mpg
                loc run=1
            }
            else {
                di "Memory not available, waiting ..."
                pause 1000
            }
        }
    }
    is there another command to get the available memory in a Unix system within Stata?
    If it doesn't exist, I will submit this simple program to SSC. Could anyone help with a solution for my freemem program that uses file read instead of using preserve and import delimited to read the vmstat output?


    Thank you.



    Jorge Eduardo Pérez Pérez
    www.jorgeperezperez.com

  • #2
    jorge Eduardo Perez Perez it looks like there may be a Java based analog to what you were doing manually (http://docs.oracle.com/javase/8/docs...temMXBean.html). I haven't tested it yet, but it can also return processor consumption info as well. I'm trying to work out some last minute bugs parsing the Google Geocode API payload (and returning it to Stata) and will try throwing together something quick that I can send you individually to see if it generates the same results on your system.

    Comment


    • #3
      Apparently it was the Java 7 version that had better memory functions. However, there is another library called SIGAR that uses Java bindings to a bunch of C libraries to access the relevant information. I'll try experimenting with it a bit (its building in the background right now on my MacBook pro) and once I can put something together I can send you a copy privately to see if it works the same way on your system.

      Comment


      • #4
        Haven't built out how it would return results to Stata yet, but was able to throw things together fairly quickly from code snippets I found online. Let me know if this meets your needs:

        https://github.com/wbuchanan/StataOS

        ​There's an example of what the on screen output could look like and it seems like it should work across platforms.

        Comment


        • #5
          This clearly meets my needs and the portability is definitively an advantage. I don't know anything about Java though. How can I test this in my Unix server?

          In the meantime I have modified my programs to get cpu usage as well as memory, and worked out a program to run processes only if there's enough resources available in the server.
          Jorge Eduardo Pérez Pérez
          www.jorgeperezperez.com

          Comment


          • #6
            Give me a few and I'll set things up to return the numeric values in macros/scalars from Java in an r-class .ado. I did something similar to this in another program to access file system properties and it wasn't too much work.

            Comment


            • #7
              Have to run to a meeting, but the changes are already posted in the repository. Just setting up the distribution stuff.

              Comment


              • #8
                Should be able to use:

                Code:
                net inst sysresources, from("http://wbuchanan.github.io/StataOS/")
                sysresources, d
                The display option also prints the results to the console, but all of the values are returned in scalars (numeric only).

                Comment


                • #9
                  Results on Stata - MP 13.1 - 9 Dec 2015 - Windows 8 64 bit

                  Code:
                  . sysresources, d
                  java.lang.UnsupportedClassVersionError: org/paces/Stata/SysResources : Unsupported major.minor version 52.0
                          at java.lang.ClassLoader.defineClass1(Native Method)
                          at java.lang.ClassLoader.defineClass(Unknown Source)
                          at java.security.SecureClassLoader.defineClass(Unknown Source)
                          at java.net.URLClassLoader.defineClass(Unknown Source)
                          at java.net.URLClassLoader.access$100(Unknown Source)
                          at java.net.URLClassLoader$1.run(Unknown Source)
                          at java.net.URLClassLoader$1.run(Unknown Source)
                          at java.security.AccessController.doPrivileged(Native Method)
                          at java.net.URLClassLoader.findClass(Unknown Source)
                          at java.lang.ClassLoader.loadClass(Unknown Source)
                          at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
                          at java.lang.ClassLoader.loadClass(Unknown Source)
                  r(5100); t=0.56 1:00:13
                  Results on Stata - MP 13.1 - 3 Jun 2015 - Linux server

                  Code:
                  . sysresources, d
                  java.lang.NoClassDefFoundError: org/paces/Stata/SysResources
                  Caused by: java.lang.ClassNotFoundException: org.paces.Stata.SysResources
                          at java.net.URLClassLoader$1.run(Unknown Source)
                          at java.net.URLClassLoader$1.run(Unknown Source)
                          at java.security.AccessController.doPrivileged(Native Method)
                          at java.net.URLClassLoader.findClass(Unknown Source)
                          at java.lang.ClassLoader.loadClass(Unknown Source)
                          at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
                          at java.lang.ClassLoader.loadClass(Unknown Source)
                  r(5100);
                  Jorge Eduardo Pérez Pérez
                  www.jorgeperezperez.com

                  Comment


                  • #10
                    Jorge Eduardo Perez Perez Your OP asks about using -file read- instead of -import delimited- & then -preserve/restore- to parse the free.txt file. I know Billy is working with you on a Java solution that I cannot help with, but I'll mention that the -file read- solution involves seeking the line and then finding the word within the line that you want and returning it. You can get some examples from the help file (though I don't think it is very clear) for -file read- but I've made some changes to your ado-file below to illustrate pulling text from free.txt (which, btw, I change to a -tempfile- instead of writing a txt file each time).

                    The trick here is that _if_ you are going to write this to work in other non-unix systems, you'll have to customize the -shell- portion and the -file read- portion to issue the OS-specific command and to seek to the part of the file that returns free memory in each type of text file (or maybe you want the command to just be unix specific -- if so, maybe call it -freemem_unix-) . I have a Mac OS X machine with me, so I created a line that uses "vm_stat" instead in my OS environment. Finally, I have no idea of vm_stat or vmstat (unix), etc return information differently into free.txt depending on OS configuration(?) -- if so, that will muck up your approach. ((n.b., all of the points I raise in this paragraph may only underscore a good reason to use a java based implementation that Billy is working on but I don't know much about that approach -- I just wanted to give you some help with -file read-.))

                    Here's the example - note that I put a loop in there that will read every single line in the free.txt tempfile (aa) but you don't need that , you need the lines below that seek out Line 2 in the file to show free memory. This works on my computer and returns the amount of free memory when I run the command (but I'm not suggesting that it is SSC worthy yet).

                    Code:
                    clear
                    * Program to determine amount of free memory
                    cap program drop freemem
                    program define freemem, rclass
                        preserve
                        quietly {
                            
                            tempfile aa
                            cap rm `aa'
                            ** Vmstat gives available memory
                            if `"`c(os)'"' == "Unix"  shell (vmstat)>`aa'
                            if `"`c(os)'"' == "MacOSX"  shell vm_stat > `aa'
                            tempname fh
                            file open `fh' using `"`aa'"', text read
                            file read `fh' line
                            di `"`line'"'  //take a look
                    **you don't actually need this loop::
                            while r(eof)==0 {
                                            local linenum = `linenum' + 1
                                            display %4.0f  `linenum' _asis `"  `macval(line)'"'
                                            file read `fh' line
                                    }
                                             file close `fh'
                    
                        **you want line line2
                            file open `fh' using `"`aa'"', text read
                                file read `fh' line
                                di `"`line'"'
                    
                     local jj : display %4.0f 2  _asis `"  `macval(line)'"'
                     local jj `""`jj'""'    
                        di `"`jj'"'
                     local jjj: piece 13 5 of `"`jj'"'
                     noi di `"Mem free is:   `jjj'"'
                                    file close `fh'
                    
                                    
                        }
                    end
                    
                    freemem

                    prints: Mem free is: 4096


                    - Eric Booth
                    Last edited by eric_a_booth; 11 Feb 2016, 16:48.
                    Eric A. Booth | Senior Director of Research | Far Harbor | Austin TX

                    Comment


                    • #11
                      jorge Eduardo Perez Perez...strange. Which version of Java are you running? Like eric_a_booth, I'm working on OSX, but calling the program from Stata 14 MP8 with Stata set up to use the system JVM (1.8). I haven't touched my desktop at home in a bit, but I can try things from there at some point soon to see if I run into the same issue on Ubuntu and will test from work tomorrow on a Windoze box. The problem looks like it isn't finding the sysresources.jar file and/or like it isn't finding the manifest correctly. Either way I'll check on things in the morning and will give you an update if I can figure any thing out.

                      Comment


                      • #12
                        - in case it helps.... I am on Mac OS X 10.8 running Java version 1.6.0_65 (Stata 13 MP) and I am getting the same error that Jorge reported earlier.
                        Eric A. Booth | Senior Director of Research | Far Harbor | Austin TX

                        Comment


                        • #13
                          This seems like it may be a Java problem. Not sure if eric_a_booth has a newer version of Java installed on his system, but he might be able to help test by doing something like:

                          Code:
                          // This is the Java 8 version that I have Stata pointed at.  On OSX, the default path should be something like:
                          // /Applications/Stata/utilities/java/x64/jre1.6.0_65/lib/server/libjvm.dylib
                          // You can verify this by running:
                          query java
                          
                          // Prior to pointing Stata at a newer version of Java
                          set java_vmpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/server/libjvm.dylib"
                          ​This should resolve things. Since eric_a_booth and Jorge Eduardo Perez Perez are both using Stata 13, I just fired up my version of Stata 13 on OSX:

                          Code:
                          . cd ~/Desktop/Programs/Java/StataOS
                          /Users/billy/Desktop/Programs/Java/StataOS
                          
                          . query java
                          -------------------------------------------------------------------------------------------------
                              Java system information
                                  Java initialized    no
                              Advanced Java settings
                                  set java_heapinit   16m
                                  set java_heapmax    384m
                                  set java_vmpath     /Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/server/libjvm.dylib
                          
                          . run sysresources.ado
                          . sysresources, d
                          Total committed memory is : 4.4 GiB
                          Free Swap Space is : 0 B
                          Total Available Swap Space is : 0 B
                          Process CPU Time is : 2293279000
                          Free Physical Memory Available is : 8.4 GiB
                          Total Physical Memory Available is : 16.0 GiB
                          System CPU Load is : 0.0
                          Process Load is : 0.0
                          % Free Physical Memory is : 52.390360832214355
                          If I point Stata at the version of Java that was part of my version 13 distribution I can reproduce the error:

                          Code:
                          . set java_vmpath "/Applications/Stata 13/utilities/java/x64/jre1.7.0_21/lib/server/libjvm.dylib"
                          (set /Applications/Stata 13/utilities/java/x64/jre1.7.0_21/lib/server/libjvm.dylib preference rec
                          > orded)
                          
                          . query java
                          -------------------------------------------------------------------------------------------------
                              Java system information
                                  Java initialized    no
                              Advanced Java settings
                                  set java_heapinit   16m
                                  set java_heapmax    384m
                                  set java_vmpath     /Applications/Stata 13/utilities/java/x64/jre1.7.0_21/lib/server/libj
                          > vm.dylib
                          
                          . run sysresources.ado
                          . sysresources, d
                          java.lang.UnsupportedClassVersionError: org/paces/Stata/SysResources : Unsupported major.minor ve
                          > rsion 52.0
                                  at java.lang.ClassLoader.defineClass1(Native Method)
                                  at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
                                  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
                                  at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
                                  at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
                                  at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
                                  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
                                  at java.security.AccessController.doPrivileged(Native Method)
                                  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
                                  at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
                                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
                                  at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
                          r(5100);
                          This all said, I just recompiled the binary to version 1.6 of the JDK and was able to get things to work with the newer .JAR in Stata 13:

                          Code:
                          . query java
                          -------------------------------------------------------------------------------------------------
                              Java system information
                                  Java initialized    no
                              Advanced Java settings
                                  set java_heapinit   16m
                                  set java_heapmax    384m
                                  set java_vmpath     /Applications/Stata 13/utilities/java/x64/jre1.7.0_21/lib/server/libj
                          > vm.dylib
                          
                          . sysresources, d
                          Total committed memory is : 3.3 GiB
                          Free Swap Space is : 0 B
                          Total Available Swap Space is : 0 B
                          Process CPU Time is : 2749627000
                          Free Physical Memory Available is : 7.8 GiB
                          Total Physical Memory Available is : 16.0 GiB
                          System CPU Load is : 0.0
                          Process Load is : 0.0
                          % Free Physical Memory is : 48.728203773498535
                          This should hopefully take care of the issue. The .pkg file is updated so you should be able to catch it using adoupdate.

                          Comment


                          • #14
                            Just got to work and got the same error that Jorge Eduardo Perez Perez was getting on Windoze. I'll try to see what things look like if I compile things locally to see if that makes a difference at all. It's possible that there are different methods available depending on the OS. In that case I wouldn't be able to print things out neatly without a bit more work, but it shouldn't be too problematic in either case.

                            Comment


                            • #15
                              Yes - I upgraded JDK/JRE to version 8 ("1.8.0_73") on Mac OS X and ran the commands you suggested, here's the output I got:


                              Code:
                              . do "/var/folders/cn/b1f3jy4x49z0s3vx3vnrnphr0000gn/T//SD10187.000000"
                              
                              . 
                              . 
                              . 
                              .  query java
                              ----------------------------------------------------------------------------------------------
                                  Java system information
                                      Java initialized    yes
                                      Heap usage          15m
                                      Classpath           /Applications/Stata/ado/base/:/Applications/Stata/ado/base/jar/:/A
                              > pplications/Stata/ado/base/jar/libdeps-core.jar:/Applications/Stata/ado/base/jar/libstata-co
                              > re.jar:/Applications/Stata/ado/base/jar/stata-sfi.jar:./:/Users/ebooth/Library/Application S
                              > upport/Stata/ado/personal/:/Users/ebooth/Library/Application Support/Stata/ado/plus/:/Users/
                              > ebooth/Library/Application Support/Stata/ado/plus/jar/:/Users/ebooth/Library/Application Sup
                              > port/Stata/ado/plus/jar/sysresources.jar:
                              
                                  Advanced Java settings
                                      set java_heapinit   16m
                                      set java_heapmax    384m
                                      set java_vmpath     /Library/Java/JavaVirtualMachines/jdk1.8.0_73.jdk/Contents/Home/jr
                              > e/lib/server/libjvm.dylib
                              r; t=0.00 9:35:59
                              
                              .  set java_vmpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_73.jdk/Contents/Home/jre/lib/se
                              > rver/libjvm.dylib"
                              (set /Library/Java/JavaVirtualMachines/jdk1.8.0_73.jdk/Contents/Home/jre/lib/server/libjvm.dyl
                              > ib preference recorded)
                              r; t=0.00 9:35:59
                              
                              .  query java
                              ----------------------------------------------------------------------------------------------
                                  Java system information
                                      Java initialized    yes
                                      Heap usage          15m
                                      Classpath           /Applications/Stata/ado/base/:/Applications/Stata/ado/base/jar/:/A
                              > pplications/Stata/ado/base/jar/libdeps-core.jar:/Applications/Stata/ado/base/jar/libstata-co
                              > re.jar:/Applications/Stata/ado/base/jar/stata-sfi.jar:./:/Users/ebooth/Library/Application S
                              > upport/Stata/ado/personal/:/Users/ebooth/Library/Application Support/Stata/ado/plus/:/Users/
                              > ebooth/Library/Application Support/Stata/ado/plus/jar/:/Users/ebooth/Library/Application Sup
                              > port/Stata/ado/plus/jar/sysresources.jar:
                              
                                  Advanced Java settings
                                      set java_heapinit   16m
                                      set java_heapmax    384m
                                      set java_vmpath     /Library/Java/JavaVirtualMachines/jdk1.8.0_73.jdk/Contents/Home/jr
                              > e/lib/server/libjvm.dylib
                              r; t=0.00 9:35:59
                              
                              .  
                              .  
                              . ssc uninstall sysresources
                              
                              package sysresources from http://wbuchanan.github.io/StataOS
                                    'SYSRESOURCES': A Java-plugin for Stata to access system resources
                              
                              (package uninstalled)
                              r; t=0.32 9:36:00
                              
                              . **get latest version::
                              . net inst sysresources, from("http://wbuchanan.github.io/StataOS/")
                              checking sysresources consistency and verifying not already installed...
                              installing into /Users/ebooth/Library/Application Support/Stata/ado/plus/...
                              installation complete.
                              r; t=0.14 9:36:00
                              
                              . sysresources, d
                              java.lang.UnsupportedClassVersionError: org/paces/Stata/SysResources : Unsupported major.minor
                              >  version 52.0
                                      at java.lang.ClassLoader.defineClass1(Native Method)
                                      at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
                                      at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
                                      at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
                                      at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
                                      at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
                                      at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
                                      at java.security.AccessController.doPrivileged(Native Method)
                                      at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
                                      at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
                                      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
                                      at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
                              r(5100); t=0.00 9:36:00
                              
                              end of do-file
                              Eric A. Booth | Senior Director of Research | Far Harbor | Austin TX

                              Comment

                              Working...
                              X