Thread-Safe Logging in Python
You can log directly from multiple threads because the logging module is thread-safe.
In this tutorial you will discover how to log safely from many threads.
Need to Log From Multiple Threads
A thread is a thread of execution in a computer program.
Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create additional threads in our program in order to execute code concurrently.
Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.
You can learn more about Python threads in the guide:
In concurrent programming, we may need to log from multiple threads in the application.
This may be for many reasons, such as:
- Different threads may perform different tasks and may encounter a range of events.
- Worker threads may encounter exceptions that need to be stored, but should not halt the program.
- Background tasks may need to report progress over the duration of the program.
What is logging, and is it safe to use from multiple threads?
Run your loops using all CPUs, download my FREE book to learn how.
What is Logging
Logging is a way of tracking events within a program.
There are many types of events that may be logged within a program, ranging in different levels of severity, such as debugging and information to warnings, errors and critical events.
The logging module provides infrastructure for logging within Python programs.
This module defines functions and classes which implement a flexible event logging system for applications and libraries.
— logging — Logging facility for Python
Logging is achieved by first configuring the log handler and then adding calls to the logging infrastructure at key points in the program.
Handler objects are responsible for dispatching the appropriate log messages (based on the log messages’ severity) to the handler’s specified destination.
— Handles, Logging HOWTO
The default handler will report log messages on the command prompt (e.g. terminal or system output stream).
Alternate handlers can be configured to write log messages to file, to a database, or to custom target locations. The handler can specify the severity of messages to store.
For example, we can log to file by calling the logging.basicConfig() function and specifying the file name and path to log to (e.g. application.log), the level of logging to capture to the file (e.g. from logging.DEBUG to logging.CRITICAL).
Events are logged via function calls based on the type of event performed, e.g. logging.debug() for debugging messages.
Logging provides a set of convenience functions for simple logging usage. These are debug(), info(), warning(), error() and critical().
— Logging HOWTO
For example, we may add information messages to application code by calling logging.info() and passing in the string details of the event.
We may also log failures, such as exceptions that are important but not critical to the program via the logger.error() or logger.exception() functions.
These function calls can then be added throughout an application to capture events of different types, and the log handler or runtime configuration can be used to filter the severity of the messages that are actually stored.
This provides a quick crash-course to Python logging, you can learn more here:
Next, let’s consider logging from multiple threads.
Confused by the threading module API?
Download my FREE PDF cheat sheet
How to Log From Multiple Threads
The logging module can be used directly from multiple threads.
The reason is because the logging module is thread-safe.
The logging module is intended to be thread-safe without any special work needing to be done by its clients.
— Thread Safety, logging — Logging facility for Python.
This is achieved using locks.
Internally, the logging module uses mutual exclusion (mutex) locks to ensure that logging handlers are protected from race conditions from multiple threads.
For example, locks are used to ensure that only one thread writes to a log file or log stream at a time. This ensures that log messages are serialized and are not corrupted.
It achieves this though using threading locks; there is one lock to serialize access to the module’s shared data, and each handler also creates a lock to serialize access to its underlying I/O.
— Thread Safety, logging — Logging facility for Python.
Therefore, once the logging module is configured for your process, you can log directly from threads.
Next, let’s look at some examples of logging from multiple threads.
Free Python Threading Course
Download my threading API cheat sheet and as a bonus you will get FREE access to my 7-day email course.
Discover how to use the Python threading module including how to create and start new threads and how to use a mutex locks and semaphores
Example of Thread-Safe Logging
We can explore how to log safely from multiple threads.
In this example, we will create five worker threads, each of which will execute a task. The task will loop five times and each iteration it will generate a random number between 0 and 1, block for that many seconds, then if the number is below a threshold, the thread will stop.
We will log whether the task was stopped as a warning message and we will log whether the task was completed successfully as an information message.
All threads will log the information message and some subset of threads will report a warning message.
Firstly, we will define a function to execute in each worker thread.
The task will take a unique integer to represent the thread, and a threshold value as arguments. The task will loop five times, generate a random number, block with a call to time.sleep() and check the value of the number to see if the task should stop.
The task() function below implements this, with warning and info level log messages.