Thread local storage in Python
I’m not sure what you’re asking—threading.local is documented, and you’ve more or less pasted the documentation below.
@Glenn I pasted the documentation in one of my answers. I quoted Alex’s solution in the other. I am simply making this content more accessible.
Imagine criticizing helpful volunteers for reformatting critical documentation as a mobile-accessible StackOverflow answer previously readable only by manually typing obfuscatory Python statements into an interactive CLI REPL (e.g., import _threading_local as tl\nhelp(tl) ).
5 Answers 5
Thread local storage is useful for instance if you have a thread worker pool and each thread needs access to its own resource, like a network or database connection. Note that the threading module uses the regular concept of threads (which have access to the process global data), but these are not too useful due to the global interpreter lock. The different multiprocessing module creates a new sub-process for each, so any global will be thread local.
threading module
import threading from threading import current_thread threadLocal = threading.local() def hi(): initialized = getattr(threadLocal, 'initialized', None) if initialized is None: print("Nice to meet you", current_thread().name) threadLocal.initialized = True else: print("Welcome back", current_thread().name) hi(); hi() Nice to meet you MainThread Welcome back MainThread One important thing that is easily overlooked: a threading.local() object only needs to be created once, not once per thread nor once per function call. The global or class level are ideal locations.
Here is why: threading.local() actually creates a new instance each time it is called (just like any factory or class call would), so calling threading.local() multiple times constantly overwrites the original object, which in all likelihood is not what one wants. When any thread accesses an existing threadLocal variable (or whatever it is called), it gets its own private view of that variable.
This won’t work as intended:
import threading from threading import current_thread def wont_work(): threadLocal = threading.local() #oops, this creates a new dict each time! initialized = getattr(threadLocal, 'initialized', None) if initialized is None: print("First time for", current_thread().name) threadLocal.initialized = True else: print("Welcome back", current_thread().name) wont_work(); wont_work() Will result in this output:
First time for MainThread First time for MainThread multiprocessing module
All global variables are thread local, since the multiprocessing module creates a new process for each thread.
Consider this example, where the processed counter is an example of thread local storage:
from multiprocessing import Pool from random import random from time import sleep import os processed=0 def f(x): sleep(random()) global processed processed += 1 print("Processed by %s: %s" % (os.getpid(), processed)) return x*x if __name__ == '__main__': pool = Pool(processes=4) print(pool.map(f, range(10))) It will output something like this:
Processed by 7636: 1 Processed by 9144: 1 Processed by 5252: 1 Processed by 7636: 2 Processed by 6248: 1 Processed by 5252: 2 Processed by 6248: 2 Processed by 9144: 2 Processed by 7636: 3 Processed by 5252: 3 [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] . of course, the thread IDs and the counts for each and order will vary from run to run.
Thread-Local Data in Python
You can use thread-local data by calling threading.local() and sharing the instance between threads.
In this tutorial you will discover how to use thread-local data storage in Python.
Need Data Unique to Each Thread
A thread is a thread of execution in a computer program.
Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create additional threads in our program in order to execute code concurrently.
Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.
Sometimes we may need to store data that is unique for each thread.
Python provides a thread-local object that can be used to store unique data for each thread. The thread-local storage provides an alternative to having to extend the threading.Thread class and define instance variables for thread-specific data.
What is thread-local data exactly?
Run your loops using all CPUs, download my FREE book to learn how.
What is Thread-Local Data?
Thread-local data storage is a mechanism in multi-threaded programming that allows data to be stored and accessed in a way that is private to each thread.
It may be called “thread-local storage“, “thread-private“, or simply “thread-local“.
Typically this involves creating an instance of a thread-local object that is shared between objects on which data is set and retrieved.
Data stored and retrieved on the thread-local object may have the same variable names across threads, meaning the same code can be used across threads, a common approach when using worker threads.
Importantly, the reads and writes to the thread-local object are unique and private at the thread-level. This means one thread may write a variable with the name “address” and another thread may read or write a variable with an identical name but it will not interact with the variable stored by the first thread.
If executing the same code and using the same thread-local instance, then each thread has its own private version of a named variable and its assigned value in the thread-local storage.
This can be useful when multiple threads need to store local data such as a partial solution or a temporary variable and need to use the same code while executing, such as the same instance of an object.
Now that we know what thread-local data is, how do we use it?
Confused by the threading module API?
Download my FREE PDF cheat sheet
How to Use Thread Local-Data
Threads can store local data via an instance of the threading.local class.
First, an instance of the local class must be created by calling the threading.local() function.