Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Is it possible to use the Stata-python integration module inside of foreach Stata loop?

    Hello,

    I am trying to execute the code below and it is giving me an error, mainly because the Stata takes the command 'end' to end the python mode as a termination of the entire foreach loop:

    gen Alpha = .
    gen AUC = .
    local i = 0
    range alphas 0.0 1.0 20

    foreach a in alphas {

    i++

    python:

    from sklearn.naive_bayes import MultinomialNB
    from sklearn.model_selection import GridSearchCV
    from sklearn.model_selection import train_test_split
    from sklearn import metrics # import scikit-learn metrics module for accuracy calculation
    from sfi import Data
    import numpy as np
    import pandas as pd

    a = Data.get("a")

    # predict using the best value for alpha
    mnb = MultinomialNB(alpha = a, class_prior = None, fit_prior = True)

    # calculate probability of each class on the test set
    # '[:, 1]' at the end extracts the probability for each pharmacy to be under compliance
    Y_mnb_score = mnb.fit(X_train, np.ravel(Y_train)).predict_proba(X_test)[:, 1]

    # make test_compliance python variable
    test_compliance = Y_test['compliance']

    # transfer the python variables Y_mnb_score and test_compliance to STATA
    Data.setObsTotal(len(Y_mnb_score))
    Data.addVarFloat('mnbScore')
    Data.store(var = 'mnbScore', obs = None, val = Y_mnb_score)

    Data.setObsTotal(len(test_compliance))
    Data.addVarFloat('testCompliance')
    Data.store(var = 'testCompliance', obs = None, val = test_compliance)

    end // this 'end' is causing a problem

    roctab testCompliance mnbScore
    replace AUC= r(area) in `i'
    replace Alpha = `a’

    } // loop not working

    How can I use the Stata-python integration module inside of the foreach loop?

    Thank you,
    Last edited by Dominique Bourget; 12 Oct 2019, 06:37.

  • #2
    Hello,
    I resolved this issue by using 'python:' for a single line of python command, instead of using the (python: / end ) block.

    Thank you,

    Comment


    • #3
      Thank you for posting your solution. This is, at least, consistent with what is required by Stata for including a block of Mata code (headed by mata and terminated by end) within a Stata program (headed by program define and terminated by end) or foreach, forvalues and if/else blocks

      Comment


      • #4
        You could call a python script file in the loop.
        Code:
        python script pyfilename [, args(args_list) global userpaths(user_paths[, prepend])]
        The program below illustrate a difference which I don't expect; the mata code run, the python code fail:
        Code:
        capt prog drop testIT
        prog def testIT
        
        local end end
        
        forvalues i = 1/2 {
            
            mata
        
            2 + 2
        
            `end'     
        }
        
        
        forvalues i = 1/2 {
        
            python
        
            1 + 1
        
            `end'
        }
        
        end

        Comment


        • #5
          I have a problem with the "python : [python code] solution. It does not work with a "if" statement. (I am using "arcpy, which has been imported.

          e.g. in Stata

          python
          import arcpy
          ....
          end

          local n = 150
          forval i = 1/`n' {
          python : arcpy.Delete_management("[path]/_`o'.shp") if arcpy.Exists("[path]/_`o'.shp")
          }

          returns

          ">>> arcpy.Delete_management("D:/temp/taudem/exit`o'.shp") if arcpy.Exists("D:/temp/taudem/exit`o'.shp")
          File "<stdin>", line 1
          arcpy.Delete_management("D:/temp/taudem/exit154.shp") if arcpy.Exists("D:/temp/taudem/exit154.shp")
          ^
          SyntaxError: invalid syntax
          r(7102);
          "
          variations such as [Stata code]

          if arcpy.Exists("[path]/_`o'.shp"):
          arcpy.Delete_management("[path]/_`o'.shp")

          and

          python: if arcpy.Exists("D:/temp/taudem/exit`o'.shp"): arcpy.Delete_management("D:/temp/taudem/exit`o'.shp")

          also fail

          ". local o = 150
          . python : if arcpy.Exists("D:/temp/taudem/`exit`o'.shp") : arcpy.Delete_management("D:/temp/taudem/`exit`o'.shp")
          File "<stdin>", line 1
          if arcpy.Exists("D:/temp/taudem/`exit154.shp") : arcpy.Delete_management("D:/temp/taudem/`exit154.shp")
          ^
          SyntaxError: unexpected EOF while parsing
          r(7102);
          "

          ". local o = 150
          . python : if arcpy.Exists("[path]_`o'.shp") :
          File "<stdin>", line 1
          if arcpy.Exists("[path]_`o'.shp") :
          ^
          SyntaxError: unexpected EOF while parsing
          r(7102);
          "

          Thanks for any help.

          Richard

          Comment


          • #6
            Originally posted by Richard Palmer-Jones View Post
            I have a problem with the "python : [python code] solution. It does not work with a "if" statement. (I am using "arcpy, which has been imported.

            e.g. in Stata

            python
            import arcpy
            ....
            end

            local n = 150
            forval i = 1/`n' {
            python : arcpy.Delete_management("[path]/_`o'.shp") if arcpy.Exists("[path]/_`o'.shp")
            }

            returns

            ">>> arcpy.Delete_management("D:/temp/taudem/exit`o'.shp") if arcpy.Exists("D:/temp/taudem/exit`o'.shp")
            File "<stdin>", line 1
            arcpy.Delete_management("D:/temp/taudem/exit154.shp") if arcpy.Exists("D:/temp/taudem/exit154.shp")
            ^
            SyntaxError: invalid syntax
            r(7102);
            "
            variations such as [Stata code]

            if arcpy.Exists("[path]/_`o'.shp"):
            arcpy.Delete_management("[path]/_`o'.shp")

            and

            python: if arcpy.Exists("D:/temp/taudem/exit`o'.shp"): arcpy.Delete_management("D:/temp/taudem/exit`o'.shp")

            also fail

            ". local o = 150
            . python : if arcpy.Exists("D:/temp/taudem/`exit`o'.shp") : arcpy.Delete_management("D:/temp/taudem/`exit`o'.shp")
            File "<stdin>", line 1
            if arcpy.Exists("D:/temp/taudem/`exit154.shp") : arcpy.Delete_management("D:/temp/taudem/`exit154.shp")
            ^
            SyntaxError: unexpected EOF while parsing
            r(7102);
            "

            ". local o = 150
            . python : if arcpy.Exists("[path]_`o'.shp") :
            File "<stdin>", line 1
            if arcpy.Exists("[path]_`o'.shp") :
            ^
            SyntaxError: unexpected EOF while parsing
            r(7102);
            "

            Thanks for any help.

            Richard
            I haven't used Python witin STATA, but you need indentation? An indentation is recommended to be four white spaces.

            Code:
             if arcpy.Exists("D:/temp/taudem/`exit`o'.shp"):
                arcpy.Delete_management("D:/temp/taudem/`exit`o'.shp")
            EDIT: I also noticed that you have a problem here: `exit`o'.shp . You are missing an ' somewhere.
            Last edited by Karl Tjensvoll; 15 Nov 2019, 02:13.

            Comment


            • #7
              Hi Karl. Thanks for the reply.

              Well spotted. Yes, careless copying.

              But the unpaired "`" is not the problem:


              ". local o = 219

              . python : if arcpy.Exists("[path]/exit`o'.shp") :
              File "<stdin>", line 1
              if arcpy.Exists([path]/exit219.shp") :
              ^
              SyntaxError: unexpected EOF while parsing
              r(7102);

              end of do-file

              r(7102);

              . do "C:\Users\Richard Palmer-Jones\AppData\Local\Temp\statacmd.tmp"

              . local o = 219

              . python : arcpy.Delete_management("[path]/exit`o'.shp") if arcpy.Exists("[path]/exit`o'.shp")
              File "<stdin>", line 1
              arcpy.Delete_management("[path]/exit219.shp") if arcpy.Exists("[path]/exit219.shp")
              ^
              SyntaxError: unexpected EOF while parsing
              r(7102);

              . python : arcpy.Delete_management("[path]/exit219.shp") if arcpy.Exists("[path]/exit219.shp")
              File "<stdin>", line 1
              arcpy.Delete_management("[path]/exit219.shp") if arcpy.Exists("[path]/exit219.shp")
              ^
              SyntaxError: unexpected EOF while parsing
              r(7102);"

              Actually, I could run the commands as a .py script with "`o'" passed as an argument, but I have yet to master the passing of arguments to a python script.



              Comment


              • #8
                Oh, and I do have 4 spaces (indentation) in the "if.Exists(): " command.

                Comment


                • #9
                  For what it is worth, I have a rather cumbersome soultion using "lambdas":

                  . local o = 226

                  . python: fc = arcpy.Exists("D:/temp/exit`o'.shp")

                  . python: print(fc)
                  True

                  . python: ((lambda: '', lambda: arcpy.Delete_management("D:/temp/exit`o'.shp"))[fc]())
                  <Result 'true'>

                  . python: fc = arcpy.Exists("D:/temp/exit`o'.shp")

                  . python: print(fc)
                  False

                  Any improvments welcome, of course; and pointers to writinge .py scripts with argument passing.

                  Richard

                  Comment


                  • #10
                    I am a little confused, it would help if you write you code using the code wrap (the # on the bar). In this code:

                    Code:
                    if arcpy.Exists("[path]/exit`o'.shp"):
                    You get an error since there is no line of code after the if statement. You would need something like this, which prints the boolean:

                    Code:
                    if arcpy.Exists("[path]/exit`o'.shp"):
                        print(arcpy.Exists("[path]/exit`o'.shp"))
                    This code:

                    Code:
                    arcpy.Delete_management("[path]/exit219.shp") if arcpy.Exists("[path]/exit219.shp")
                    Is STATA code, in Python you would need to write:

                    Code:
                    if arcpy.Exists("[path]/exit219.shp"):
                        arcpy.Delete_management("[path]/exit219.shp")

                    Comment


                    • #11
                      This is annoying but trivial to solve. (For MATA users the solution is akin to how one can name a block of MATA code, except python uses indentation instead of brackets).

                      For example, this code:
                      Code:
                      global stop = 0
                      while $stop == 0 {
                          python:
                          print("hi")
                          end
                          global stop = 1
                      }
                      reproduces the error mentioned here:
                      Code:
                      . global stop = 0
                      
                      . while $stop == 0 {
                        2.     python:
                        3.     print("hi")
                        4.     end
                      --Break--
                      r(1);
                      
                      end of do-file
                      But simply copying-pasting the python: end block as the body of a trivial python function like here:
                      Code:
                      python:
                      def myblock():
                          print("hi")
                      end
                      
                      global stop = 0
                      while $stop == 0 {
                          python: myblock()
                          global stop = 1
                      }
                      solves the issue:
                      Code:
                      . global stop = 0
                      
                      . while $stop == 0 {
                        2.     python: myblock()
                        3.     global stop = 1
                        4. }
                      hi
                      
                      . 
                      . 
                      end of do-file
                      Best,
                      ps

                      Comment

                      Working...
                      X