Python Module Search Path
Summary: in this tutorial, you’ll learn how the module search path works in Python when you import a module into a program.
Introduction to Python module search path
When you import a module in a program:
import moduleCode language: Python (python)Python will search for the module.py file from the following sources:
- The current folder from which the program executes.
- A list of folders specified in the PYTHONPATH environment variable, if you set it before.
- An installation-dependent list of folders that you configured when you installed Python.
Python stores the resulting search path in the sys.path variable that comes from the sys module.
The following program shows the current module search path:
import sys for path in sys.path: print(path) Code language: Python (python)Here’s a sample output on Windows:
D:\Python\ C:\Program Files\Python38\python38.zip C:\Program Files\Python38\DLLs C:\Program Files\Python38\lib C:\Program Files\Python38 C:\Users\PythonTutorial\AppData\Roaming\Python\Python38\site-packages C:\Program Files\Python38\lib\site-packages Code language: Shell Session (shell)And the following is the sample output on Linux:
/Library/Frameworks/Python.framework/Versions/3.8/bin /Library/Frameworks/Python.framework/Versions/3.8/lib/python38.zip /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8 /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.8/site-packagesCode language: Shell Session (shell)To make sure Python can always find the module.py , you need to:
- Place module.py in the folder where the program will execute.
- Include the folder that contains the module.py in the PYTHONPATH environment variable. Or you can place the module.py in one of the folders included in the PYTHONPATH variable.
- Place the module.py in one of the installation-dependent folders.
Modifying the Python module search path at runtime
Python allows you to modify the module search path at runtime by modifying the sys.path variable. This allows you to store module files in any folder of your choice.
Since the sys.path is a list, you can append a search-path to it.
The following example adds the d:\modules to the search path and use the recruitment module stored in this folder:
>>> import sys >>> sys.path.append('d:\\modules\\') >>> import recruitment >>> recruitment.hire() Hire a new employee. Code language: JavaScript (javascript)Summary
- When you import a module, Python will search for the module file from the folders specified in the sys.path variable.
- Python allows you to modify the module search path by changing, adding, and removing elements from the sys.path variable.
The initialization of the sys.path module search path¶
A module search path is initialized when Python starts. This module search path may be accessed at sys.path .
The first entry in the module search path is the directory that contains the input script, if there is one. Otherwise, the first entry is the current directory, which is the case when executing the interactive shell, a -c command, or -m module.
The PYTHONPATH environment variable is often used to add directories to the search path. If this environment variable is found then the contents are added to the module search path.
PYTHONPATH will affect all installed Python versions/environments. Be wary of setting this in your shell profile or global environment variables. The site module offers more nuanced techniques as mentioned below.
The next items added are the directories containing standard Python modules as well as any extension module s that these modules depend on. Extension modules are .pyd files on Windows and .so files on other platforms. The directory with the platform-independent Python modules is called prefix . The directory with the extension modules is called exec_prefix .
The PYTHONHOME environment variable may be used to set the prefix and exec_prefix locations. Otherwise these directories are found by using the Python executable as a starting point and then looking for various ‘landmark’ files and directories. Note that any symbolic links are followed so the real Python executable location is used as the search starting point. The Python executable location is called home .
Once home is determined, the prefix directory is found by first looking for python majorversion minorversion .zip ( python311.zip ). On Windows the zip archive is searched for in home and on Unix the archive is expected to be in lib . Note that the expected zip archive location is added to the module search path even if the archive does not exist. If no archive was found, Python on Windows will continue the search for prefix by looking for Lib\os.py . Python on Unix will look for lib/python majorversion . minorversion /os.py ( lib/python3.11/os.py ). On Windows prefix and exec_prefix are the same, however on other platforms lib/python majorversion . minorversion /lib-dynload ( lib/python3.11/lib-dynload ) is searched for and used as an anchor for exec_prefix . On some platforms lib may be lib64 or another value, see sys.platlibdir and PYTHONPLATLIBDIR .
Once found, prefix and exec_prefix are available at sys.prefix and sys.exec_prefix respectively.
Finally, the site module is processed and site-packages directories are added to the module search path. A common way to customize the search path is to create sitecustomize or usercustomize modules as described in the site module documentation.
Certain command line options may further affect path calculations. See -E , -I , -s and -S for further details.
Virtual environments¶
If Python is run in a virtual environment (as described at Virtual Environments and Packages ) then prefix and exec_prefix are specific to the virtual environment.
If a pyvenv.cfg file is found alongside the main executable, or in the directory one level above the executable, the following variations apply:
- If home is an absolute path and PYTHONHOME is not set, this path is used instead of the path to the main executable when deducing prefix and exec_prefix .
_pth files¶
To completely override sys.path create a ._pth file with the same name as the shared library or executable ( python._pth or python311._pth ). The shared library path is always known on Windows, however it may not be available on other platforms. In the ._pth file specify one line for each path to add to sys.path . The file based on the shared library name overrides the one based on the executable, which allows paths to be restricted for any program loading the runtime if desired.
When the file exists, all registry and environment variables are ignored, isolated mode is enabled, and site is not imported unless one line in the file specifies import site . Blank paths and lines starting with # are ignored. Each path may be absolute or relative to the location of the file. Import statements other than to site are not permitted, and arbitrary code cannot be specified.
Note that .pth files (without leading underscore) will be processed normally by the site module when import site has been specified.
Embedded Python¶
If Python is embedded within another application Py_InitializeFromConfig() and the PyConfig structure can be used to initialize Python. The path specific details are described at Python Path Configuration . Alternatively the older Py_SetPath() can be used to bypass the initialization of the module search path.
The Module Search Path
In this lesson, you’ll learn about the module search path. Continuing with the example from the previous lesson, take a look at what happens when Python executes the following statement:
>>> import mod >>> mod.a [100, 200, 300] >>> mod.s 'Computers are useless. They can only give you answers.' When the interpreter executes the above import statement, it searches for mod.py in a list of directories assembled from the following sources:
- The directory from which the input script was run, or the current directory if the interpreter is being run interactively
- The list of directories contained in the PYTHONPATH environment variable, if it is set. (The format for PYTHONPATH is OS-dependent but should mimic the PATH environment variable.)
- An installation-dependent list of directories configured at the time Python is installed
The resulting search path is accessible in the Python variable sys.path , which is obtained from a module named sys :
>>> import sys >>> sys.path ['', '/Library/Frameworks/Python.framework/Versions/3.7/bin', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python37.zip', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages'] Note: The exact contents of sys.path are installation-dependent. The above code block will almost certainly look slightly different on your computer. The operating system used in this lesson is macOS. If you would like to see what the path structure looks like in a Windows environment, check out the original article that this course is based on.
So, to ensure that your module is found, you need to do one of the following:
- Put mod.py in the directory where the input script is located, or the current directory if interactive
- Modify the PYTHONPATH environment variable to contain the directory where mod.py is located before starting the interpreter. Or put mod.py in one of the directories already contained in the PYTHONPATH variable.
- Put mod.py in one of the installation-dependent directories, which you may or may not have write-access to, depending on the OS.
There is also one additional option: You can put the module file in any directory of your choice and then modify sys.path at run-time so that it contains that directory. For example, in this case, you could put mod.py in directory /Users/chris/ModulesAndPackages and then issue the following statements:
>>> sys.path.append(r'/Users/chris/ModulesAndPackages') >>> sys.path ['', '/Library/Frameworks/Python.framework/Versions/3.7/bin', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python37.zip', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages', '/Users/chris/ModulesAndPackages'] >>> import mod >>> mod.s 'Computers are useless. They can only give you answers.' Once you’ve imported a module, you can determine the location where it was found with the module’s __file__ attribute:
>>> import mod >>> mod.__file__ '/Users/chris/ModulesAndPackages/mod.py' >>> import re >>> re.__file__ '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/re.py' The directory portion of __file__ should be one of the directories in sys.path .