Getting corresponding module from function
works. But then I get problems if the definition modify is in a different file than the rest of the code.
4 Answers 4
import inspect def modify(fun): module = inspect.getmodule(fun) This is the same as polling the module from sys.modules using fun.__module__ . Although getmodule tries harder even if fun does not have a __module__ attribute.
You want to get the module object from its name? Look it up in the sys.modules dictionary that contains all currently loaded modules:
import sys def modify(func): module = sys.modules[func.__module__] modulename = fun.__module__ module = __import__(modulename) You do not want to do this.
- It does not work. Imagine you defined a function in module abc and then imported it in xyz . test.__module__ would be ‘abc’ when you called modify(xyz.test) . You would then know to change abc.test and you would not end up modifying xyz.test at all!
- Monkey patching should be avoided. Fooling with the global state of your program is ill-advised. Instead of doing this, why cannot you just make the new, modified thing and use it?
How can you say that it’s a bad idea when you don’t know what he is trying to achieve? Whether it’s a good idea or not really depends on the use case. Maybe he wants to replace a function from an external test script to see how the program behaves in certain situations without messing with the original code. One size does not fit all!
@Ferdinand, I offered general advice based on a broad principle, which is the best I can do with the information I have. It’s true there could be use cases where monkeypatching is the best solution, but that does not change the fact that such things should be avoided in most cases. Your example use case is one that does not at all relieve my apprehension about such an idea, because it’s highly prone to error and comes with an unexplained, unreasonable restriction. Although specific concerns applicable to every program are unique, best practices exist, and it pays to recognize them.
In my case I have to use a third party package, where the objects are tight together. Eg you want to modify the behavior of Y which is constructed in the construtor of a class X. If I want a X with a modifed Y I have to derive from X and from Y. Which introduces more dependency than just monkey patching Y. And I do not want do maintain a private fork of the third party lib.
@NoctisSkytower, That’s a very wrongheaded way to look at things. The goal should never be to answer questions, but to help people. «How» questions can be problematic at a much higher rate than what or why questions. Consider if someone came and asked, «How do I pull the trigger of a shotgun with my toe?» The best answer is, «Don’t».
python — get list of all functions in current module. inspecting current module does not work?
@abarnert It looks like I can define testfunctions at the bottom. But I thought testall would not be able to use it as it was not defined before it.
@ato: test1 has to be defined at the time it’s used, which is when you call testall , not when you define it. If you want to understand the technicalities, I can explain it in an answer.
3 Answers 3
EDIT 1: What I am trying to do is
def testall(arg): return any(f(arg) for f in testfunctions) def test1(arg): #code here # may call testall but wont call anyother test* def testall(arg): testfunctions = [obj for name,obj in inspect.getmembers(sys.modules[__name__]) if (inspect.isfunction(obj) and name.startwith('test') and name != 'testall')] return any(f(arg) for f in testfunctions) def test1(arg): #code here # may call testall but wont call anyother test* In this case, testfunctions isn’t evaluated until testall is called, so there’s no problem here—by that time, all top-level module code (including the test1 definition) will have been evaluated, so testfunctions will get all of the top-level functions. (I’m assuming here that testall or test1 is being called from an if __name__ == ‘__main__’ block at the bottom of the module, or another script is doing import tests; tests.test1(10) , or something similar.)
In fact, even if you explicitly named test1 and test2 , there would be no problem:
def testall(arg): testfunctions = ('test1',) return any(f(arg) for f in testfunctions) def test1(arg): #code here # may call testall but wont call anyother test* Again, test1 is already defined by the time you call testall , so everything is fine.
If you want to understand why this works, you have to understand the stages here.
When you import a module, or run a top-level script, the first stage is compilation (unless there’s already a cached .pyc file). The compiler doesn’t need to know what value a name has, just whether it’s local or global (or a closure cell), and it can already tell that sys and inspect and test1 are globals (because you don’t assign to them in testall or in an enclosing scope).
Next, the interpreter executes the compiled bytecode for the top-level module, in order. This includes executing the function definitions. So, testall becomes a function, then test1 becomes a function, then test2 becomes a function. (A function is really just the appropriate compiled code, with some extra stuff attached, like the global namespace it was defined in.)
Later, when you call the testall function, the interpreter executes the function. This is when the list comprehension (in the first version) or the global name lookup (in the second) happens. Since the function definitions for test1 and test2 have already been evaluated and bound to global names in the module, everything works.
What if you instead later call test1 , which calls testall ? No problem. The interpreter executes test1 , which has a call to testall , which is obviously already defined, so the interpreter calls that, and the rest is the same as in the previous paragraph.
So, what if you put a call to testall or test1 in between the test1 and test2 definitions? In that case, test2 wouldn’t have been defined yet, so it would not appear in the list (first version), or would raise a NameError (second version). But as long as you don’t do that, there’s no problem. And there’s no good reason to do so.
If you’re worried about the horrible performance cost of computing testfunctions every time you call testall … Well, first, that’s a silly worry; how many times are you going to call it? Are your functions really so fast that the time to call and filter getmembers even shows up on the radar? But if it really is a worry, just cache the value in your favorite of the usual ways—mutable default, privat global, function attribute, …:
def testall(arg, _functions_cache=[]): if not _functions_cache: _functions_cache.extend([…]) In Python, given a function name, how to get all the modules containing the function?
while can return os.path.walk, os.walk and md.walk. Or if it’s difficult to know there is a md.walk, how to get the imported os.path.walk and os.walk?
What do you mean by ‘containing’? All modules that import the object from the source module? That’s way too broad; you would have to scan your whole harddisk for any and all possible Python source files that could at some point import the module and use the function.
I do not find the question particularly unclear. The fact that there is no easy answer does not mean the question is invalid. For refactoring (for instance) it can be very useful to know which modules refer to a particular function.
So now you are looking for a string name. What about modules where walk is an entirely different function? Or a method? Or another type object.
Add linebreaks and indentation: for name, module in sys.modules.items(): if hasattr(module, ‘walk’): print name
It’s nasty to do this for all modules including those that aren’t even imported yet. Python doesn’t really support iterating over all possible modules, and it’s not clear you’d want to — some of them run startup code that could break your interpreter or take a long time to run (think test scripts). Maybe you just want to do a nice grep over all the Python files in a given tree.
2 Answers 2
OK, that’s was fun, here’s updated solution. Some Python magic, solution for Python 2.7. Any improvements are welcome. It behaves like any other import — so be careful to wrap any of your executable code in if name == «__main__» .
import inspect import pkgutil import sys def whereis_in_globals(fname): return [m for m in sys.modules.itervalues() if module_has_function(m, fname)] def whereis_in_locals(fname): modules = (__import__(module_name, globals(), locals(), [fname], -1) for _, module_name, _ in pkgutil.walk_packages(path=".")) return [m for m in modules if module_has_function(m, fname)] def module_has_function(m, fname): return hasattr(m, fname) and inspect.isfunction(getattr(m, fname)) if __name__ == "__main__": # these should never raise AttributeError for m in whereis_in_locals('walk'): m.walk for m in whereis_in_globals('walk'): m.walk Caveat: this only works if the relevant modules are imported, but +1 since it’s still a good approach.
Can you tell us more about your use case or larger goals? Getting this answered for non-imported modules is an interesting challenge, and I don’t have an answer for the way you’ve phrased it but depending on your actual use case you may be interested in knowing that may IDEs (Integrated Development Environments) can sort of do what you’re asking.
For example if you install PyCharm (available for free in community edition) you can type a full or partial function name (just a text string) and then have the IDE search the active project to see what you might be looking for. For example you could just type «walk» on a line and then (IIR) Ctrl-click on it to see what possible modules you might have matching functions. You can also search the other direction, seeing where in a project the function is used.
To search on a wide scale you might have to cheat a little and open your Python directory as a «project» but that’s easy.
Again, you haven’t provided a use case so my apologies if you are specifically looking for, e.g., an in-script way to get that result for later use in a program. If it’s something you want an answer to for you, as a developer and as you work, you might look to the tools you’re using to provide an answer.
Get function import path
How to get from object f information about import — ‘pack.mod’ I can get it using f.__module__ but if function def in module where i get this attribute ( f.__module__ ) it return ‘__main__’ . But i need real path here — ‘pack.mod’ I found this way to get this information:
inspect.getmodule(f).__file__ then i can sub start path from sys.path , replace / on . and get path like — ‘pack.mod’ But may be exist some more convenient way?
3 Answers 3
What inspect.getmodule(f) does internally, per inspect.py’s sources, is essentially sys.modules.get(object.__module__) — I wouldn’t call using that code directly «more convenient», though (beyond the «essentially» part, inspect has a lot of useful catching and correction of corner cases).
Edit: reading between the lines it seems the OP is trying to do something like
and within bla.py (which is thus being run as __main__ ) determine «what from or import statement could a different main script use to import this function from within me?».
Problem is, the question is ill-posed, because there might not be any such path usable for the purpose (nothing guarantees the current main script’s path is on sys.path when that different main script gets run later), there might be several different ones (e.g. both /foo/bar and /foo/bar/baz might be on sys.path and /foo/bar/baz/__init__.py exist, in which case from baz.bla import f and from bla import f might both work), and nothing guarantees that some other, previous sys.path item might not «preempt» the import attempt (e.g., say /foo/bar/baz is on sys.path , but before it there’s also /fee/fie/foo , and a completely unrelated file /fee/fie/foo/bla.py also exists — etc, etc).
Whatever the purpose of this kind of discovery attempt, I suggest finding an alternative architecture — e.g., one where from baz.bla import f is actually executed (as the OP says at the start of the question), so that f.__module__ is correctly set to baz.bla .