Set library path in python

Best way to set sys.path for «hot library» development in Python?

Entry points are in circus/ and grail/, as well as (possibly) each of the test/ directories, depending on how testing is implemented. I have multiple copies of this source tree present on local storage at any one point in time (corresponding to various maintenance and feature branches), so I cannot set PYTHONPATH in my shell without some pain. (I would need to remember to change it each time I switched to work on a different branch, and I am very forgetful) Instead, I have some logic that walks up the file tree, starting at the «current» file location, moving from leaf towards root, looking for branch_root_marker. Once the root directory of the current working copy is found, it adds library/ and productline/ to sys.path. I call this function from each entry-point in the system.

"""Add working copy (branch) to sys.path""" import os import sys def _setpath(): """Add working copy (branch) to sys.path""" markerfile = "branch_root_marker" path = "" if ("__file__" in locals() or globals()) and __file__ != "__main__": path = __file__ elif sys.argv and sys.argv[0] and not sys.argv[0] == "-c": path = sys.argv[0] path = os.path.realpath(os.path.expandvars(path)) while os.path.exists(path): if os.path.exists(os.path.join(path, markerfile)): break else: path = os.path.dirname(path) errormsg = " ".join(("Could not find", markerfile)) assert os.path.exists(path) and path != os.path.dirname(path), errormsg path = os.path.join(path, "src") (_, subdir_list, _) = os.walk(path).next() for subdir in subdir_list: if subdir.startswith("."): continue subdir_path = os.path.join(path, subdir) if subdir_path in sys.path: continue sys.path.append(subdir_path) _setpath() 

Currently, I need to keep a separate but identical copy of this function in each entry point. Even though it is quite a short function, I am quite distressed by how fragrantly the DRY principle is being violated by this approach, and would love to find a way to keep the sys.path modification logic in one place. Any ideas? Note:- One thing that springs to mind is to install the sys.path modifying logic into a common location that is always on PYTHONPATH. Whilst this is not a terrible idea, it means introducing an installation step that needs to be carried out each time I move to a fresh environment; another thing to remember (or, more likely, forget), so I would like to avoid this if at all possible.

Источник

How To Set Python Module Search Path To Find Modules

When you want to import a python module library in your python source code, you need first to make the python module library importable by adding the module package path in the PYTHONPATH system environment variable. You can also add the python module package path to the python module search path at runtime in python source code. This example will show you how to do it.

1. Add Python Module Package Path In System Environment Variable PYTHONPATH.

Suppose your python module is saved in folder /tmp. We will add /tmp folder in the PYTHONPATH environment variable value.

Open a terminal and go to the user home directory use cd ~ command.

$ ls -al . -rw-r--r--@ 1 zhaosong staff 1176 Apr 30 09:15 .bash_profile .

2. Display Python Library Search Path In Python Source Code.

Run into python interactive console in a terminal.

$ python3 Python 3.6.5 |Anaconda, Inc.| (default, Apr 26 2018, 08:42:37) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>

>>> import sys >>> for path in sys.path: . print(path) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa

3. Append Directory To Python Library Search Path.

1. Python sys.path.append function can append directory to the end of python library search directory.

>>> sys.path.append('/abc') >>> >>> >>> for line in sys.path: . print(line) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa /abc

4. Append Exist Module Library Directory To Python Library Search Directory.

1. Python module’s __file__ attribute returns the module file saved directory. You can append that directory to the python library search path as below.

>>> import sys, os # os.__file__ will return the os module directory. >>> sys.path.append(os.__file__) >>> >>> >>> for line in sys.path: . print(line) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa /abc /Users/zhaosong/anaconda3/lib/python3.6/os.py

5. Question & Answer.

5.1 How can I add another python source directory in the Python search path for a large python project.

1. My team just handle an old python project from another team, the python project is so large, there are a lot of python source files in the python project. Our development environment is Linux and no IDE, only in the command line. And when I run the python script with the command python abc.py it prompts the error ImportError: no module named com.test_module. And there is a lot of such kind of errors in other python scripts. All the old python project files are saved in a folder like /codebase/old_python_project. And there are a lot of subfolders in the project base folder. How can I make python search all the modules in the project folder and it’s subfolders to fix the import error? Thanks.
2. You can add your existing python project folder in the PYTHONPATH system environment variable to fix your issue. You can also use the function sys.path.append(‘/codebase/old_python_project’) in your python script source code and then can import the python modules. If you want to add all the project subfolders in the python module search path, you can reverse loop your project folder and when you reach it’s subfolder then you can call the sys.path.append() function to add the subfolder to the python module search path, you can try it.

import os, sys def reverse_add_python_module_search_path(module_dir): files_array = [] files_array = os.listdir(module_dir) for file in files_array: if os.path.isdir(file): sys.path.append(file) reverse_add_python_module_search_path(file)

Источник

The initialization of the sys.path module search path¶

A module search path is initialized when Python starts. This module search path may be accessed at sys.path .

The first entry in the module search path is the directory that contains the input script, if there is one. Otherwise, the first entry is the current directory, which is the case when executing the interactive shell, a -c command, or -m module.

The PYTHONPATH environment variable is often used to add directories to the search path. If this environment variable is found then the contents are added to the module search path.

PYTHONPATH will affect all installed Python versions/environments. Be wary of setting this in your shell profile or global environment variables. The site module offers more nuanced techniques as mentioned below.

The next items added are the directories containing standard Python modules as well as any extension module s that these modules depend on. Extension modules are .pyd files on Windows and .so files on other platforms. The directory with the platform-independent Python modules is called prefix . The directory with the extension modules is called exec_prefix .

The PYTHONHOME environment variable may be used to set the prefix and exec_prefix locations. Otherwise these directories are found by using the Python executable as a starting point and then looking for various ‘landmark’ files and directories. Note that any symbolic links are followed so the real Python executable location is used as the search starting point. The Python executable location is called home .

Once home is determined, the prefix directory is found by first looking for python majorversion minorversion .zip ( python311.zip ). On Windows the zip archive is searched for in home and on Unix the archive is expected to be in lib . Note that the expected zip archive location is added to the module search path even if the archive does not exist. If no archive was found, Python on Windows will continue the search for prefix by looking for Lib\os.py . Python on Unix will look for lib/python majorversion . minorversion /os.py ( lib/python3.11/os.py ). On Windows prefix and exec_prefix are the same, however on other platforms lib/python majorversion . minorversion /lib-dynload ( lib/python3.11/lib-dynload ) is searched for and used as an anchor for exec_prefix . On some platforms lib may be lib64 or another value, see sys.platlibdir and PYTHONPLATLIBDIR .

Once found, prefix and exec_prefix are available at sys.prefix and sys.exec_prefix respectively.

Finally, the site module is processed and site-packages directories are added to the module search path. A common way to customize the search path is to create sitecustomize or usercustomize modules as described in the site module documentation.

Certain command line options may further affect path calculations. See -E , -I , -s and -S for further details.

Virtual environments¶

If Python is run in a virtual environment (as described at Virtual Environments and Packages ) then prefix and exec_prefix are specific to the virtual environment.

If a pyvenv.cfg file is found alongside the main executable, or in the directory one level above the executable, the following variations apply:

• If home is an absolute path and PYTHONHOME is not set, this path is used instead of the path to the main executable when deducing prefix and exec_prefix .

_pth files¶

To completely override sys.path create a ._pth file with the same name as the shared library or executable ( python._pth or python311._pth ). The shared library path is always known on Windows, however it may not be available on other platforms. In the ._pth file specify one line for each path to add to sys.path . The file based on the shared library name overrides the one based on the executable, which allows paths to be restricted for any program loading the runtime if desired.

When the file exists, all registry and environment variables are ignored, isolated mode is enabled, and site is not imported unless one line in the file specifies import site . Blank paths and lines starting with # are ignored. Each path may be absolute or relative to the location of the file. Import statements other than to site are not permitted, and arbitrary code cannot be specified.

Note that .pth files (without leading underscore) will be processed normally by the site module when import site has been specified.

Embedded Python¶

If Python is embedded within another application Py_InitializeFromConfig() and the PyConfig structure can be used to initialize Python. The path specific details are described at Python Path Configuration . Alternatively the older Py_SetPath() can be used to bypass the initialization of the module search path.

Источник