Python library path environment variable

Set LD_LIBRARY_PATH correctly

Before I have installed MCR in the docker, I have installed boost using:

$ sudo apt-get install -y libboost-all-dev 

and I was able to wrap C++ classes and call them in python. Then I have installed MCR and one of the requirement is to set LD_LIBRARY_PATH to Matlab’s library.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/MATLAB/MATLAB_Runtime/v94/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v94/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v94/sys/os/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v94/extern/bin/glnxa64 

It worked, I was able to run matlab’s python package in python but testing the C++ classes wrappers failed due to change of LD_LIBRARY_PATH . For example, I get the following error when I try to import C++ class in python:

 /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0: undefined symbol: XML_SetHashSalt 

If I unset LD_LIBRARY_PATH C++ import works but MCR fails. So the question what LD_LIBRARY_PATH should be in order to run both C++ classes and Matlab’s python package successfully?

I have tried to add path to boost libraries to LD_LIBRARY_PATH but didn’t work, so the env value would be:

/usr/lib/x86_64-linux-gnu:/usr/local/MATLAB/MATLAB_Runtime/v94/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v94/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v94/sys/os/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v94/extern/bin/glnxa64 

the dependencies of C++ library file before setting the LD_LIBRARY_PATH :

 linux-vdso.so.1 (0x00007ffcee0dc000) libpython3.4m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0 (0x00007f9d69e59000) libboost_python-py34.so.1.55.0 => /usr/lib/x86_64-linux-gnu/libboost_python-py34.so.1.55.0 (0x00007f9d69c09000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9d698fe000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9d695fd000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9d693e7000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9d6903c000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f9d68e34000) libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f9d68c0b000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9d689f0000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9d687d3000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9d685cf000) libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f9d683cc000) /lib64/ld-linux-x86-64.so.2 (0x00007f9d6a6c3000) 

After setting the LD_LIBRARY_PATH :

 linux-vdso.so.1 (0x00007ffc42e9b000) libpython3.4m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0 (0x00007fad9635b000) libboost_python-py34.so.1.55.0 => /usr/lib/x86_64-linux-gnu/libboost_python-py34.so.1.55.0 (0x00007fad9610b000) libstdc++.so.6 => /usr/local/MATLAB/MATLAB_Runtime/v94/sys/os/glnxa64/libstdc++.so.6 (0x00007fad95d8a000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fad95a89000) libgcc_s.so.1 => /usr/local/MATLAB/MATLAB_Runtime/v94/sys/os/glnxa64/libgcc_s.so.1 (0x00007fad95873000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fad954c8000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fad952c0000) libexpat.so.1 => /usr/local/MATLAB/MATLAB_Runtime/v94/bin/glnxa64/libexpat.so.1 (0x00007fad95095000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fad94e7a000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fad94c5d000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fad94a59000) libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fad94856000) /lib64/ld-linux-x86-64.so.2 (0x00007fad96bc5000) 

Источник

How To Set Python Module Search Path To Find Modules

When you want to import a python module library in your python source code, you need first to make the python module library importable by adding the module package path in the PYTHONPATH system environment variable. You can also add the python module package path to the python module search path at runtime in python source code. This example will show you how to do it.

1. Add Python Module Package Path In System Environment Variable PYTHONPATH.

Suppose your python module is saved in folder /tmp. We will add /tmp folder in the PYTHONPATH environment variable value.

Open a terminal and go to the user home directory use cd ~ command.

$ ls -al . -rw-r--r--@ 1 zhaosong staff 1176 Apr 30 09:15 .bash_profile .

2. Display Python Library Search Path In Python Source Code.

Run into python interactive console in a terminal.

$ python3 Python 3.6.5 |Anaconda, Inc.| (default, Apr 26 2018, 08:42:37) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>

>>> import sys >>> for path in sys.path: . print(path) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa

3. Append Directory To Python Library Search Path.

1. Python sys.path.append function can append directory to the end of python library search directory.

>>> sys.path.append('/abc') >>> >>> >>> for line in sys.path: . print(line) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa /abc

4. Append Exist Module Library Directory To Python Library Search Directory.

1. Python module’s __file__ attribute returns the module file saved directory. You can append that directory to the python library search path as below.

>>> import sys, os # os.__file__ will return the os module directory. >>> sys.path.append(os.__file__) >>> >>> >>> for line in sys.path: . print(line) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa /abc /Users/zhaosong/anaconda3/lib/python3.6/os.py

5. Question & Answer.

5.1 How can I add another python source directory in the Python search path for a large python project.

1. My team just handle an old python project from another team, the python project is so large, there are a lot of python source files in the python project. Our development environment is Linux and no IDE, only in the command line. And when I run the python script with the command python abc.py it prompts the error ImportError: no module named com.test_module. And there is a lot of such kind of errors in other python scripts. All the old python project files are saved in a folder like /codebase/old_python_project. And there are a lot of subfolders in the project base folder. How can I make python search all the modules in the project folder and it’s subfolders to fix the import error? Thanks.
2. You can add your existing python project folder in the PYTHONPATH system environment variable to fix your issue. You can also use the function sys.path.append(‘/codebase/old_python_project’) in your python script source code and then can import the python modules. If you want to add all the project subfolders in the python module search path, you can reverse loop your project folder and when you reach it’s subfolder then you can call the sys.path.append() function to add the subfolder to the python module search path, you can try it.

import os, sys def reverse_add_python_module_search_path(module_dir): files_array = [] files_array = os.listdir(module_dir) for file in files_array: if os.path.isdir(file): sys.path.append(file) reverse_add_python_module_search_path(file)

Источник

Set LD_LIBRARY_PATH before importing in python

My solution to this problem is to put this as the first line of a Python script (instead of the usual shebang):

exec env LD_LIBRARY_PATH=/some/path/to/lib /path/to/specific/python -x "$0" "$@" 

And here is how this works:

• with no shebang the current shell treats the file as a shell script,
• «exec» ensures that this first line is also the last command from this file executed by the shell,
• «env» is used here to set any environment variables, e.g. LD_LIBRARY_PATH,
• an exact path to Python’s interpreter can specified or «env» can find one in PATH,
• «-x» is a Python’s option which causes the first line to be ignored by the Python interpreter,
• «$0» is the script name, «$@» is substituted by positional parameters.

The solution works fine if the env is reinit

import os os.environ['LD_LIBRARY_PATH'] = os.getcwd() # or whatever path you want 

The code need to be taken in place.

Python, when gets the values of environment variables as in os.environ[‘LD_LIBRARY_PATH’] or os.environ[‘PATH’] , it copies the values, into a dictionary, from it’s parent process’s environment, generally bash (bash process’s environment get’s carried to the child process, the python running instance).

you can see this environment variable section with env command output from bash.

you can also see/read this env data from /proc//environ , by introducing an infinite loop( while 1: pass ) after modifying any environment variable.

If you see/read this variable value/data from /proc//environ after modifying it inside the python script, you would get to see that the real variable’s data doesn’t get modified, though the python script shows a modified dictionary key value, updated.

What actually happens when you modify an env variable inside python script, as in os.environ[‘LD_LIBRARY_PATH’]=’/’ , is that it just updates the value in local dictionary, which is not mapped to process’s env variable section. Hence it won’t propagate all the way back to reflect in current process’s environment, because ONLY a local dictionary was modified/updated/populated.

Hence if we want to have the new environment variable to be reflected, we should overwrite the memory image of the process with new environment variable data, using execv .

new_lib = '/' if not new_lib in os.environ['LD_LIBRARY_PATH']: os.environ['LD_LIBRARY_PATH'] += ':'+new_lib try: os.execv(sys.argv[0], sys.argv) except Exception as e: sys.exit('EXCEPTION: Failed to Execute under modified environment, '+e) import xyz #do something else 

Limitation: Ideally, python should not allow such modification of os.environ variables. But because there is no constant dictionary data type, it allows modification of the data variable. There is absolutely no use of modifying the values, as it does nothing useful to reflect in running process’s real environment, unless execv is used.

UPDATE: see the EDIT below.

import os os.environ['LD_LIBRARY_PATH'] = os.getcwd() # or whatever path you want 

This sets the LD_LIBRARY_PATH environment variable for the duration/lifetime of the execution of the current process only.

EDIT: it looks like this needs to be set before starting Python: Changing LD_LIBRARY_PATH at runtime for ctypes

So I’d suggest going with a wrapper .sh (or .py if you insist) script. Also, as @chepner pointed out, you might want to consider installing your .so files in a standard location (within the virtualenv).

See also Setting LD_LIBRARY_PATH from inside Python

Источник