How can Stata find Python packages in a virtual environment (OS: Linux)

[Ulrich Kohler]

Starting with Ubuntu 24, my operating system forces me to install Python "externally managed environments" into a virtual environment, and issues an error message when I try without:

 pip install beautifulsoup4
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

At the Linux shell, the process to create and activate the virtual environment is as follows:

$ python3 -m venv mypython_env
$ source mypython_env/bin/activate

Afterwards, any -pip install- installs packages to the virtual environment, and the packages can be accessed without further ado:

$ pip install beautifulsoup4
$ pip show beautifulsoup4
[...]
Location: /home/ukohler/mypython_env/lib/python3.12/site-packages
[...]

In Stata, however, the /activated/ virtual environment is not detected:

. python search
-------------------------------------------------------------------------------
 Python environments found:  
 /usr/bin/python3
 /usr/bin/python3.6
 /usr/bin/python3.6m
-------------------------------------------------------------------------------

Consequently packages installed to the virtual environment are not detected:

. python which beautifulsoup4
Python module beautifulsoup4 not found
r(601);

Also, setting -python set userpath- does not help:

. python set userpath /home/ukohler/mypython_env/lib/python3.12/site-packages
. python query
-------------------------------------------------------------------------------
 Python Settings
  set python_exec      /usr/bin/python3
  set python_userpath  /home/ukohler/mypython_env/lib/python3.12/site-packages

  Python system information
  [..]
 . python which beautifulsoup4
Python module beautifulsoup4 not found
r(601);

This is

. version
version 19.5

Is there any way to install Python packages in a virtual environment such that Stata can access them?

Many regards
Uli

[Zhao Xu (StataCorp)]

Ulrich Kohler,

On Linux, python search finds Python installations in a few directories, such as /usr/bin and /usr/local/bin, etc. It does not search through your virtual environments. If you want to use your virtual environment, you can use python set exec to set it explicitly.

The reason python which beautifulsoup4 returns not found error is due to the fact that the package beautifulsoup4 is usually to be imported as bs4 even it is pip installed as beautifulsoup4. So after you set the right executable or the right user-path, you can type python which bs4 to see its location in Stata.

You can also check in the following way:

In you terminal, type

$ source mypython_env/bin/activate
$ python

In the local Python REPL environment, if you type

>>> import beautifulsoup4

It will also output the package not found.

[Ulrich Kohler]

Zhao Xu (StataCorp) Thank you. This clarifies it. My venv works well now.