Given the name of a Python package, what is the name of the module to import? [duplicate]
I didn’t find a consistent way of finding the equivalence. For some modules, it took me a lot of browsing to find it. What am I doing wrong?
2 Answers 2
Regrettably, there’s no method to the madness. The name in the package index is independent of the module name you import . Disastrously some packages share module names. If you install both, your application will break with even odds. (Ruby has this problem too)
Packaging in Python is generally dire. The root cause is that the language ships without a package manager. Ruby and Nodejs ship with full-featured package managers Gem and Npm, and have nurtured sharing communities centred around GitHub. Npm makes publishing packages as easy as installing them. Nodejs arrived 2009 and already has 14k packages. The venerable Python package index lists 24k. Ruby Gems lists 44k packages.
Fortunately, there is one decent package manager for Python, called Pip. Pip is inspired by Ruby’s Gem, but lacks some vital features (eg. listing packages, and upgrading en masse). Ironically, Pip itself is complicated to install. Installation on the popular 64-bit Windows demands building and installing two packages from source. This is a big ask for anyone new to programming.
Python’s devs are ignorant of all this frustration because they are seasoned programmers comfortable building from source, and they use Linux distributions with packaged Python modules.
Until Python ships with a package manager, thousands of developers will needlessly waste time reinventing the wheel.
Python 3 solves many problems with packaging. There aren’t any packages for Python 3.
How python deals with module and package having the same name?
I believe the package will always get loaded. You can’t work around this, as far as I know. So change either the package or the module name. Docs: http://docs.python.org/tutorial/modules.html#the-module-search-path
Hi, can you please provide a reference for your claim? I don’t see anything in the documentation link provided that says the package will always be loaded in preference to the module.
This answer is incorrect. The linked docs (for both Python 2 and 3) state that the directory of the current script is given preference in the search order. See also: python-notes.curiousefficiency.org/en/latest/python_concepts/…
While docs says package is preferred over module in my experience it’s not true, at least in python 2.7. It seems like in Python 2.7 import seeks for the module at parent (but not child) directories until it finds it, so installed packages will be checked at the end of resolving.
Actually, it is possible, by manually guiding the import machinery to use a .py file instead of directory. (This code is not well tested, but seems to work). UPDATE 2020: Note that this requires using custom import_module() function instead of normal import statement. However, with modern Python3 and its importlib , it might be possible to make the bare import statement to work the same way too. (Note that this answer shows flexibility which Python offers. It’s not an encouragement to use this in your applications. Use this only if you know what you’re doing.)
import os, imp def import_module(dir, name): """ load a module (not a package) with a given name from the specified directory """ for description in imp.get_suffixes(): (suffix, mode, type) = description if not suffix.startswith('.py'): continue abs_path = os.path.join(dir, name + suffix) if not os.path.exists(abs_path): continue fh = open(abs_path) return imp.load_module(name, fh, abs_path, (description)) import_module('.', 'foo') $ python test1.py foo package loaded $ python test2.py foo module loaded How to find «import name» of any package in Python?
I wonder if is there any reliable and consistant way to get a Python package’s «import name» / namespace. For example; Package; django-haystack
Import name; haystack or Package; ipython
Import name; IPython So far I know, PyPi doesn’t store that information that I’ve checked with PyPiXmlRpc. I also tried to automate to download the package, extract it and dig the .egg-info but some packages doesn’t have that folder at all. Any help will be appreciated and will be used for a good-manner gadget 🙂
5 Answers 5
Wheels
I know this is an old question, but wheel packages have since been invented! Since a wheel is simply a zip file that gets extracted into the lib/site-packages directory, an examination of the contents of the wheel archive can give you the top level imports.
>>> import zipfile >>> zf = zipfile.ZipFile('setuptools-35.0.2-py2.py3-none-any.whl') >>> top_level = set([x.split('/')[0] for x in zf.namelist()]) >>> # filter out the .dist-info directory >>> top_level = [x for x in top_level if not x.endswith('.dist-info')] >>> top_level ['setuptools', 'pkg_resources', 'easy_install.py'] So setuptools actually gives you three top level imports!
pip download
pip now has a download command, so you can simply run pip download setuptools (or whatever package you like) and then examine it.
Reverse look up
Unfortunately I haven’t found an easy way to go the other direction yet. That is, given the import name, what is the package name. This can be a problem if you are looking at some example code or maybe if you use Anaconda that comes with a bunch of packages pre-installed and you want to know the actual package name.
Note that what you call a package here is not a package but a distribution. A distribution can contain zero or more modules or packages. That means there is no one-to-one mapping of distributions to packages.
I’m not sure there is a way to detect what modules and packages will be installed by a distribution, other than actually installing it and introspect filesystem changes for newly added packages, modules and pth files.
I understand. Major problem is the relation between a distribution and a package/packages as I see. But there must be an answer to my question. I will find the most reliable and consistent one and let you guyz know for sure.
after a long long researching period, i end up with your answer that any Python distribution may have one or more packages such as the package itself and let’s say a test package next to as a simple example. Anyways, i was able to finish my experimental prototype (pydoc.net) and hopefully will provide some sort of API to cover this question consistently.. Thanks Wichert.
In principal, everything you need to get that information is in the setup.py that is supposed to be in every such package. That information would roughly be the union of the packages, py_modules, ext_package and ext_modules of the Distribution object. In fact, here’s a little script that mocks out distutils.core.setup just for the purpose of getting that information.
import distutils.core distutils.core._setup_stop_after = "config" _real_setup = distutils.core.setup def _fake_setup(*args, **kwargs): global dist dist = _real_setup(*args, **kwargs) distutils.core.setup = _fake_setup import sys setup_file = sys.argv[1] sys.argv[:] = sys.argv[1:] import os.path os.chdir(os.path.dirname(setup_file)) execfile(os.path.basename(setup_file)) cat = lambda *seq: sum((i for i in seq if i is not None), []) pkgs = set(package.split('.')[0] for package in cat(dist.packages, dist.py_modules, [m.name for m in cat(dist.ext_modules)], [m.name for m in cat(dist.ext_package)])) print "\n".join(pkgs) For many packages, this will work like a charm, but for a counterexample, see numpy , It breaks because numpy provides its own distutils, and I can see no obvious way around it.
Python package name conventions
Is there a package naming convention for Python like Java’s com.company.actualpackage ? Most of the time I see simple, potentially colliding package names like «web». If there is no such convention, is there a reason for it? What do you think of using the Java naming convention in the Python world?
7 Answers 7
Python has two «mantras» that cover this topic:
Explicit is better than implicit.
Namespaces are one honking great idea — let’s do more of those!
There is a convention for naming of and importing of modules that can be found in The Python Style Guide (PEP 8).
The biggest reason that there is no such convention to consistently prefix your modules names in a Java style, is because over time you end up with a lot of repetition in your code that doesn’t really need to be there.
One of the problems with Java is it forces you to repeat yourself, constantly. There’s a lot of boilerplate that goes into Java code that just isn’t necessary in Python. (Getters/setters being a prime example of that.)
Namespaces aren’t so much of a problem in Python because you are able to give modules an alias upon import. Such as:
import com.company.actualpackage as shortername So you’re not only able to create or manipulate the namespace within your programs, but are able to create your own keystroke-saving aliases as well.
The Java’s conventions also has its own drawbacks. Not every opensource package has a stable website behind it. What should a maintainer do if his website changes? Also, using this scheme package names become long and hard to remember. Finally, the name of the package should represent the purpose of the package, not its owner
The reason to use domain names is to use a name space under control of the project owner. But it would be confusing if the site differs from package names .
@deamon The issue in the answer is what do you do if you have no domain names that you control — which is true for many individuals
That is not an issue. With Maven and Gradle you always know where to get the projectX packages from. Version 1 may be org.place1, version 2 may be com.startup.projectX, etc. I do Python programming for last 2 months and I have already found at least 10 projects with names that collide, so they had to change names because of that. With tld.domain.etc schema you nicely avoid that.
If you don’t have your own domain, use the one where your project is hosted. e.g. github or the like — that’s explicitly supported for Maven. I don’t see why Python can’t adopt the same approach.
An update for anyone else who comes looking for this:
As of 2012, PEP 423 addresses this. PEP 8 touches on the topic briefly, but only to say: all lowercase or underscores.
The gist of it: pick memorable, meaningful names that aren’t already used on PyPI.
There is no Java-like naming convention for Python packages. You can of course adopt one for any package you develop yourself, but you might have to invasively edit any package you may adopt from third parties, and the «culturally alien» naming convention will probably sap the changes of your own packages to be widely adopted outside of your organization.
Technically, there would be nothing wrong with Java’s convention in Python (it would just make some from statements a tad longer, no big deal), but in practice the cultural aspects make it pretty much unfeasible.
The reason there’s normally no package hierarchy is because Python packages aren’t easily extended that way. Packages are actual directories, and though you can make packages look in multiple directories for sub-modules (by adding directories to the __path__ list of the package) it’s not convenient, and easily done wrong. As for why Python packages aren’t easily extended that way, well, that’s a design choice. Guido didn’t like deep hierarchies (and still doesn’t) and doesn’t think they’re necessary.
The convention is to pick a toplevel package name that’s obvious but unique to your project — for example, the name of the project itself. You can structure everything inside it however you want (because you are in control of it.) Splitting the package into separate bits with separate owners is a little more work, but with a few guidelines it’s possible. It’s rarely needed.
There’s nothing stopping you using that convention if you want to, but it’s not at all standard in the Python world and you’d probably get funny looks. It’s not much fun to take care of admin on packages when they’re deeply nested in com .
It may sound sloppy to someone coming from Java, but in reality it doesn’t really seem to have caused any big difficulties, even with packages as poorly-named as web.py.
The place where you often do get namespace conflicts in practice is relative imports: where code in package.module1 tries to import module2 and there’s both a package.module2 and a module2 in the standard library (which there commonly is as the stdlib is large and growing). Luckily, ambiguous relative imports are going away.