Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Failure of anaconda / miniconda python in Stata 16.1 for macOS

    I have set up miniconda3 python 3.7 with various packages on macOS 10.15.3. For a project, I want to run some Python code from Stata 16.1.

    By default, Stata is running the python executable installed by homebrew, @ /usr/local/bin/python3. As I have set up python packages in the miniconda python environment, I have 'set python_exec /usr/local/miniconda3/bin/python3'. Python starts fine and works without any issues.

    Starting python from stata with the 'python'-command results in a prompt where basic operation work.

    To test the environment a bit better, I try to run 'samplepy.do' from https://www.stata.com/new-in-stata/python-integration/

    This fails with the following error message:

    Code:
    . python:
    ----------------------------------------------- python (type end to exit) ------------------------------------------
    >>> from sfi import Data
    >>> import numpy as np
    >>> from sklearn.svm import SVC
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/local/miniconda3/lib/python3.7/site-packages/sklearn/__init__.py", line 82, in <module>
        from .base import clone
      File "/usr/local/miniconda3/lib/python3.7/site-packages/sklearn/base.py", line 20, in <module>
        from .utils import _IS_32BIT
      File "/usr/local/miniconda3/lib/python3.7/site-packages/sklearn/utils/__init__.py", line 27, in <module>
        from .fixes import np_version
      File "/usr/local/miniconda3/lib/python3.7/site-packages/sklearn/utils/fixes.py", line 18, in <module>
        import scipy.stats
      File "/usr/local/miniconda3/lib/python3.7/site-packages/scipy/stats/__init__.py", line 384, in <module>
        from .stats import *
      File "/usr/local/miniconda3/lib/python3.7/site-packages/scipy/stats/stats.py", line 179, in <module>
        from scipy.spatial.distance import cdist
      File "/usr/local/miniconda3/lib/python3.7/site-packages/scipy/spatial/__init__.py", line 102, in <module>
        from ._procrustes import procrustes
      File "/usr/local/miniconda3/lib/python3.7/site-packages/scipy/spatial/_procrustes.py", line 11, in <module>
        from scipy.linalg import orthogonal_procrustes
      File "/usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/__init__.py", line 195, in <module>
        from .misc import *
      File "/usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/misc.py", line 5, in <module>
        from .blas import get_blas_funcs
      File "/usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/blas.py", line 215, in <module>
        from scipy.linalg import _fblas
    ImportError: dlopen(/usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_fblas.cpython-37m-darwin.so, 2):
    >  Symbol not found: _main
      Referenced from: /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_fblas.cpython-37m-darwin.so
      Expected in: flat namespace
     in /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_fblas.cpython-37m-darwin.so
    (24 lines skipped)
    --------------------------------------------------------------------------------------------------------------------
    r(7102);
    Apparently 'from sklearn.svm import SVC' fails from within the Stata python environment. Issuing the command in the interpreter of '/usr/local/miniconda3/bin/python3' run from the command line, does not result in an issue.

    Why is the symbol _main not found when run from within the Stata python environment?

  • #2
    I have tried to see which libraries are not loaded when trying to 'from sklearn.multiclass import OneVsRestClassifier' in python3 from the command shell vs python3 from Stata.
    The libraries which are not loaded are listed below, many miniconda3-libraries are loaded in both instances. I really have no clue why these do not load when running python from stata... apparently something gets in the way I suppose, could it be blas functions of Stata?
    Code:
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_fblas.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_flapack.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_flinalg.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_solve_toeplitz.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_decomp_update.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/cython_blas.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/cython_lapack.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/spatial/_distance_wrap.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/spatial/_hausdorff.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/special/_ufuncs.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/libgfortran.3.dylib
    /usr/local/miniconda3/lib/libquadmath.0.dylib
    /usr/local/miniconda3/lib/libgcc_s.1.dylib
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/special/_ufuncs_cxx.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/special/specfun.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/special/_comb.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/special/_ellip_harm_2.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/ndimage/_nd_image.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/ndimage/_ni_label.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/minpack2.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_trlib/_trlib.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/sparse/linalg/isolve/_iterative.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/sparse/linalg/dsolve/_superlu.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/sparse/linalg/eigen/arpack/_arpack.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_group_columns.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_lbfgsb.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/moduleTNC.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_cobyla.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_slsqp.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_minpack.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_lsq/givens_elimination.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_zeros.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_nnls.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_bglu_dense.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/optimize/_lsap_module.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/integrate/_odepack.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/integrate/_quadpack.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/integrate/vode.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/integrate/_dop.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/integrate/lsoda.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/interpolate/_fitpack.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/interpolate/dfitpack.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/interpolate/_bspl.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/interpolate/_ppoly.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/interpolate/interpnd.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/stats/_stats.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/stats/statlib.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/scipy/stats/mvn.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/sklearn/utils/_openmp_helpers.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/libomp.dylib
    /usr/local/miniconda3/lib/python3.7/site-packages/sklearn/utils/_logistic_sigmoid.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/sklearn/utils/sparsefuncs_fast.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/sklearn/preprocessing/_csr_polynomial_expansion.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/sklearn/metrics/cluster/_expected_mutual_info_fast.cpython-37m-darwin.so
    /usr/local/miniconda3/lib/python3.7/site-packages/sklearn/metrics/_pairwise_fast.cpython-37m-darwin.so
    Last edited by Jeroen Albers; 22 Feb 2020, 11:56.

    Comment


    • #3
      From Stata, what is the output if you enter the following:

      Code:
      python:
      from scipy import linalg

      Comment


      • #4
        The output after entering that command is:
        Code:
        >>> from scipy import linalg
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
          File "/usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/__init__.py", line 195, in <module>
            from .misc import *
          File "/usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/misc.py", line 5, in <module>
            from .blas import get_blas_funcs
          File "/usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/blas.py", line 215, in <module>
            from scipy.linalg import _fblas
        ImportError: dlopen(/usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_fblas.cpython-37m-darwin.so, 2):
        >  Symbol not found: _main
          Referenced from: /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_fblas.cpython-37m-darwin.so
          Expected in: flat namespace
         in /usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_fblas.cpython-37m-darwin.so
        r(7102);
        (Probably superfluous: no issues with that command from python without Stata)
        Last edited by Jeroen Albers; 22 Feb 2020, 17:10.

        Comment


        • #5
          I assumed it might have something to do with the Intel MKL lapack/blas anaconda uses by default, so I have set up another conda environment with openblas instead of the Intel libraries. This environment works.

          But, as the Intel MKL have the best performance on Intel hardware, it would be great if Stata would work with a conda environment which is set up these libraries.

          I hope this can be resolved.

          Comment


          • #6
            Unfortunately, the workaround is not working consistently.

            Comment


            • #7
              I wonder if people who run Stata 16 on macOS 10.15.3 with (ana-/mini-)conda python can reproduce this issue. If not, it might be something specific to my configuration and I could try to work out differences.

              Also, if I understand the error message 'ImportError: dlopen(/usr/local/miniconda3/lib/python3.7/site-packages/scipy/linalg/_fblas.cpython-37m-darwin.so, 2):
              > Symbol not found: _main' correctly, dynamically loading the library '_fblas.cpython-37m-darwin.so' fails because the symbol '_main' cannot be found, but according to the output of the 'nm' command, '_fblas.cpython-37m-darwin.so' should contain '_main". It is also very puzzling that this failure only occurs when running python from within Stata, when running it from the command line, everything works just fine.

              How does running python from within Stata change the loading/locating of dynamic libraries?
              Last edited by Jeroen Albers; 26 Feb 2020, 01:34.

              Comment


              • #8
                I found using packages from the conda-forge repository works. Of course this is not optimal, as use of the more standardized default conda packages and MKL has its advantages.

                The '_fblas.cpython-...-darwin.so' from conda-forge does apparently not require _main, the version in conda's main repository fails to resolve this when run from within Stata for some reason. It apparently is not easy to replicate, it might be some interaction between Stata and other software I have installed. If someone has suggestions as to how it can be resolved, I am eager to read them.

                Comment


                • #9
                  I do have the same issue with anaconda 3, Stata 16.1 and Mac 10.15.6. Has there been any progess on this? Has anyone else found a solution?

                  Regards,

                  Achim
                  http://statalasso.github.io/

                  Comment


                  • #10
                    I'm running Stata 16.1, anaconda 3.6.8, and OSX 10.15.7:

                    Code:
                    . python
                    ----------------------------------------------- python (type end to exit) -----------------
                    >>> from scipy import linalg
                    >>> end
                    -------------------------------------------------------------------------------------------
                    . python query
                    
                     Python Settings
                          set python_exec      /Users/billy/anaconda3/bin/python
                          set python_userpath  
                    
                        Python system information
                          initialized          yes
                          version              3.6.8
                          architecture         64-bit
                          library path         /Users/billy/anaconda3/lib/libpython3.6m.dylib
                    Perhaps it is something with the way your python distribution is installed?

                    Comment


                    • #11
                      Originally posted by wbuchanan View Post
                      I'm running Stata 16.1, anaconda 3.6.8, and OSX 10.15.7:

                      Code:
                      . python
                      ----------------------------------------------- python (type end to exit) -----------------
                      >>> from scipy import linalg
                      >>> end
                      -------------------------------------------------------------------------------------------
                      . python query
                      
                      Python Settings
                      set python_exec /Users/billy/anaconda3/bin/python
                      set python_userpath
                      
                      Python system information
                      initialized yes
                      version 3.6.8
                      architecture 64-bit
                      library path /Users/billy/anaconda3/lib/libpython3.6m.dylib
                      Perhaps it is something with the way your python distribution is installed?
                      @wbuchanan Hi,if type "python which numpy",Will Stata crash and exist? I am also running Stata 16.1,anaconda 3.8 and OSX 10.15.
                      Code:
                      . python query
                      ---------------------------------------------------------------------------------------------------
                          Python Settings
                            set python_exec      /Users/zhangdeliang/opt/anaconda3/bin/python
                            set python_userpath  
                      
                          Python system information
                            initialized          no
                            version              3.8.5
                            architecture         64-bit
                            library path         /Users/zhangdeliang/opt/anaconda3/lib/libpython3.8.dylib
                      But if I run "python which numpy",Stata will crash and exist.


                      Raymond
                      Best regards.

                      Raymond Zhang
                      Stata 17.0,MP

                      Comment


                      • #12
                        Raymond Zhang
                        I would guess there could be a few different issues.

                        1. What are the file permissions to /Users/zhangdeliang/opt/anaconda3/lib?
                        2. Why is your library path set to look at a single shared dynamic library file instead of the root directory where Python libraries could be found?
                        3. Even though Stata seems to be looking in that one location, perhaps the location of your anaconda installation is using a different location internally?

                        #2 I would assume will definitely be part of the problem since it is pointed at a single file and isn’t pointed at the directory that contains all of the Python libraries. #1 could also be a problem if the permissions are configured in a way that denies read permissions to the Stata process.

                        Comment


                        • #13
                          After I install anaconda, It automatically generate such a directory. How can I change the path?Can you give me some advice?
                          Best regards.

                          Raymond Zhang
                          Stata 17.0,MP

                          Comment


                          • #14
                            In fact ,if I don't run "python which numpy "or"from scipy import linalg" ,Python can work normally.
                            Code:
                            . python query
                            ---------------------------------------------------------------------------------------------------
                                Python Settings
                                  set python_exec      /Users/zhangdeliang/opt/anaconda3/bin/python
                                  set python_userpath  
                            
                                Python system information
                                  initialized          no
                                  version              3.8.5
                                  architecture         64-bit
                                  library path         /Users/zhangdeliang/opt/anaconda3/lib/libpython3.8.dylib
                            
                            . python 
                            ----------------------------------------------- python (type end to exit) -------------------------
                            >>> 100-3
                            97
                            >>> end
                            Best regards.

                            Raymond Zhang
                            Stata 17.0,MP

                            Comment


                            • #15
                              How can I set the library path?
                              Best regards.

                              Raymond Zhang
                              Stata 17.0,MP

                              Comment

                              Working...
                              X