Python how to reload all modules

Python Cookbook by

Get full access to Python Cookbook and 60K+ other titles, with a free 10-day trial of O’Reilly.

There are also live events, courses curated by job role, and more.

Reloading All Loaded Modules

Problem

When you repeatedly run a test script during an interactive session, it always uses the first version of the modules you are developing, even if you made changes to the code. You need to ensure that modules are reloaded.

Solution

There are a few ways to accomplish this goal. Here’s a solution that is simple and drastic, but it may not work with integrated development environments (IDEs):

import sys sys.modules.clear( )

And here is a solution that is a bit more careful and is compatible with IDEs:

import sys if globals( ).has_key('init_modules'): # second or subsequent run: remove all but initially loaded modules for m in sys.modules.keys( ): if m not in init_modules: del(sys.modules[m]) else: # first run: find out which modules were initially loaded init_modules = sys.modules.keys( )

Discussion

When you create a Python module, you can use a test script that imports your module. But you have probably noticed that when you repeatedly run the test script inside a given interactive session, it always uses the first version of your module, even if you made changes in the code. This is because the import statement checks if the module is already in memory and does the actual importing only if this is the first time the module is used. This is an important optimization that lets you use the import statement freely, but it does get in the way in such development situations.

You can use the reload function, but this is difficult if you perform changes in a module that isn’t directly imported by your test script. One simple solution is to remove all modules from memory before running the test script. For this, two lines at the start of your test script, as shown in the first solution, suffice.

If you work with a framework that executes user code in the same interpreter as the IDE itself (such as IDLE), however, you will notice that this technique fails. This is because sys.modules.clear removes IDLE from memory, so you will have to use the second solution in that case. On the first run, the solution determines which modules are initial modules for the system (all those that are loaded at this point). On all other runs, the solution cleans up all modules whose names are not in this initial list. This, of course, relies on globals (i.e., the dictionary of this test script, seen as a module) being unchanged when this test script is run again.

See Also

Documentation on the sys standard library module, along with the reload and globals built-ins, in the Library Reference ; the section on the import statement in the Language Reference .

Get Python Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Источник

Automatically Reload Modules with %autoreload

Writing my first module in Python was a confusing experience. As it usually happens, when I was testing it in the interactive Python REPL, the first version turned out to have some bugs (the second and third ones also did 😉). That’s fine — I thought — I will just fix the module and reimport it. But, to my surprise, calling from my_module import my_function didn’t update the code! my_function still had the bug that I just fixed! I double-checked if I modified the correct file, reimported it again and still nothing. It turns out, as StackOverflow kindly explained, that you can’t just reimport a module. If you already imported a module ( import a_module ) or a function ( from a_module import a_function ) in your Python session and you try to import it again, nothing will happen. It doesn’t matter if you use the standard Python REPL or IPython.

How does importing in Python work? #

1. First, it checks if the module is already cached in the sys.module dictionary.
2. And only if it’s not there, it actually imports the module.

Which means that, if you already imported the module (or imported a different module that references this one) and you try to import it again, Python will ignore this request. You can read more about how importing works in the documentation.

So, if I can’t reimport a module, does it mean that I have to restart Python each time? Not really, that would be very inconvenient.

How to reimport a module? #

The easiest way is to quit your interactive session and start it again. It works fine if you don’t care about preserving the data that you already have in your session, like the functions that you wrote and the variables that you calculated. But usually you don’t want to restart the REPL, so there are better ways.

Since we know that the interpreter will first look for the module in the sys.modules dictionary, we can just delete our module from this dictionary. And it will work in most cases, but there are some caveats. If your module is referenced from another module, there is a chance that you still won’t be able to reimport it. So don’t do this. There is a better way.

The recommended solution is to use the importlib.reload function. This function is designed exactly for reimporting modules that have already been imported before. To reload your module, you need to run:

import importlib
importlib.reload(my_module)

So that’s how you can reimport a module in Python. And if you are not using IPython, this is where your options end. But IPython users have some other interesting solutions to this problem.

%run #

If you don’t care about actually «importing» your module and all you need is to run some functions defined in a file, you can execute that file instead. It will run all the commands as if you would copy and paste them in your IPython session. You can rerun a file as many times as you want and it will always update all the functions. Running a file in IPython is extremely easy:

%run my_file.py
# You can even skip the ".py" extension:
%run my_file

I cheated a bit when I said that this option is not available in standard Python REPL. It is, but it requires more typing:

To be honest, if I had to type all this, I might as well just use the importlib.reload instead.

All those options are great, but if you are as bad as me when it comes to writing code and you make a lot of mistakes, then it means a lot of reloading. And typing this importlib.reload / %run / exec. is annoying. Wouldn’t it be great if there was a way to automatically reload a module? Well, IPython can actually do that!

%autoreload to the rescue #

Another one of the magic methods in IPython is related to reloading modules. It’s called %autoreload. It’s not enabled by default, so you have to load it as an extension:

Now, you can turn on auto-reloading:

And each time you execute some code, IPython will reimport all the modules to make sure that you are using the latest possible versions.

There are 3 configuration options that you can set:

• %autoreload 0 — disables the auto-reloading. This is the default setting.
• %autoreload 1 — it will only auto-reload modules that were imported using the %aimport function (e.g %aimport my_module ). It’s a good option if you want to specifically auto-reload only a selected module.
• %autoreload 2 — auto-reload all the modules. Great way to make writing and testing your modules much easier.

Great, any caveats? I found 3 minor ones:

• IPython with %autoreload enabled will be slightly slower. IPython is quite smart about what to reload. It will check the modification timestamps of the modules and compare them with the time when they are imported. But this checking (and eventually reimporting of the modified modules) will still take some time. It won’t be so slow that you will feel it (unless you have modules that take seconds to import), but it will obviously run faster if you disable the auto-reloading.
• As pointed out in the documentation, %autoreload is not 100% reliable, and there might be some unexpected behaviors. I never noticed any problems, but some reddit users mentioned that it might not work correctly for the more advanced modules (with classes, etc.).
• You need to make sure that you don’t have syntax errors in your modules when you are running IPython commands. I often start writing some code in a file and, in the middle of the command, I switch to IPython to quickly test something. And when I execute some code in IPython, it will try to reimport the file that I just modified (the one with the half-written command) and throw a SyntaxError. The good thing is — after the error, you will still get the output of the command that you ran. So for me, it’s a minor annoyance, not a real problem. You can easily solve it by running two IPython sessions — one for testing the module (with %autoreload enabled) and the other for running some random commands and looking up things in the documentation.

Here is how %autoreload works in practice (this video is recorded with asciinema, and if you watch it on mobile phone, part of the final comment is cut — it says: #without autoreload, we would still see «hello !»):

So if you don’t know %autoreload yet, give it a try the next time you’re working on a module in Python!

Источник

Reload or Unimport Module in Python

1. Unimport a Module in Python
2. Reload a Module in Python

Modules allow us to store definitions of different functions and classes in a Python file, and then such files can be used in other files. The pandas , NumPy , scipy , Matplotlib are some of the most widely used modules in Python.

We can also create our own modules in Python, which can increase modularity and simplify large programs.

Unimport a Module in Python

We use the import command to load a specific module to memory in Python. We cannot unimport a module since Python stores it in the cache memory, but we can use a few commands and try to dereference these modules so that we are unable to access it during the program. These methods, however, might fail at times, so please be careful.

The first is the del command. It is used to remove a variety of objects in Python. Removing the access of a module using this command is shown below.

import module_name del module_name 

The sys.modules is a dictionary that can be viewed using the sys module and is used to store the references of a function and modules. We can remove the required module from this dictionary using the del command to remove its every reference. It is difficult to remove modules that have been referenced a lot, so one needs to be careful while using this. This method might produce unwanted results, so please be cautious.

if 'myModule' in sys.modules:  del sys.modules["myModule"] 

Reload a Module in Python

In case we have made changes to a module and wish to implement those changes without restarting the program, we can use the reload() function that will reload the required module.

The reload() function has a long history in Python. Till Python 2.7 it was a built-in function.

In Python 3.0 to Python 3.3, it was present in the imp library that was later deprecated and changed to importlib module, which contains functions for implementing the mechanisms of importing codes in files Python.

The following code shows how to use the reload() function.

import importlib reload(module_name) 

Manav is a IT Professional who has a lot of experience as a core developer in many live projects. He is an avid learner who enjoys learning new things and sharing his findings whenever possible.

Related Article — Python Module

Источник