Python function timeout windows

Adding Timeouts to Functions Using Wrapt Timeout Decorator

A short guide on how to implement timeouts when calling functions in Python

In this short guide, I will show how to use a handy timeout decorator for Python function to limit the execution times. This guide is based on an example that utilizes Python’s multiprocessing package.

Lately, I’ve been building a script using Beautiful Soup that scrapes similarly structured websites and identifies its chapters. These were 10-K annual reports on the SEC’s website which had to be split according to their 21 sections. To make the script as robust as possible, I’ve used the simple, yet effective, python-constraint module to make sure that the sections were identified in the correct order. This approach worked perfectly in most cases and took less than a second to execute. Unfortunately, for some cases, the constraint function caused the script to run for days until it went through all iterations. By analyzing the execution times of the constraint function, I came to the following conclusion:

A fix for this issue was simple thanks to the wrapt_timeout_decorator module. The great thing about this module is that it’s easy to use, works great with Python’s multiprocessing module, and has no problem running on Windows.

1. Installing the wrapt_timeout_decorator

You can install the wrapt_timeout_decorator module from PyPI using pip.

pip install wrapt_timeout_decorator

If you’re using Conda, I suggest downloading the source files from GitHub (https://github.com/bitranox/wrapt_timeout_decorator) and installing the module manually by running

within your active environment. You will need Python 3.5 or later for the module to work.

2. Using the wrapt_timeout_decorator

The following example is intended for Windows users. Linux users will have it easier using the wrapt_timeout_decorator because it’s not necessary to put the timeout function in a separate module. For more information on this, see the official documentation here. I haven’t tried this module in a Jupyter notebook, but I assume it will require a workaround similar to the multiprocessing package.

The example below shows how to apply a timeout on a function that is parallelized on multiple threads using the multiprocessing package.

2.1 Example without timeout

# main_program.pyfrom multiprocessing import Pool
import my_functionmy_list = [1,2,3,4,4,3,2,1]if __name__ == ‘__main__’: 
num_processors = 4 
p = Pool(processes = num_processors) 
p.map_async(my_function.sleep_function, my_list).get()

And here is the function my_function which is parallelized across 4 threads:

# my_function.pyfrom time import sleepdef sleep_function(num): 
sleep(num) 
print(f'The program slept for seconds.')

The output of this program will be:

The program slept for 1 seconds.
The program slept for 2 seconds.
The program slept for 3 seconds.
The program slept for 4 seconds.
The program slept for 4 seconds.
The program slept for 3 seconds.
The program slept for 2 seconds.
The program slept for 1 seconds.

The first for 4 lines will appear 1, 2, 3, and 4 seconds after executing the program respectively. Because we’re using the map_async method, the remaining 4 lines will appear all at once after another 4 seconds have passed. The first thread waits 1 second and then takes the next available element from my_list, which is 4, and waits 4 more seconds (in total 5 seconds). The second thread waits 2 seconds, then takes the next available element from my_list, which is 3, and waits 3 more seconds (again 5 seconds in total) and so on.

2.2. Example with timeout

Imagine we want to limit the time for which the sleep_function is allowed to run. Let’s say we set this timeout to 2.5 seconds. To do this, we have to make three changes to our program.

1. Apply a timeout decorator to the sleep_function.
2. Carve out the timed-out sleep_function into a separate module.
3. Change the my_function.py to execute this carved-out module.

Here is how the carved-out module of the timed-out function will look like:

# my_timeout_function.pyfrom time import sleep
from wrapt_timeout_decorator import *@timeout(2.5)
def timeout_sleep_function(num): 
sleep(num) 
print(f'The program slept for seconds.')

Using the @timeout decorator, we’ve set the limit for how long the function can run to 2.5 seconds. Apart from that, the code of the function remains unchanged.

Now we have to go back to our my_function.py module and call the newly created timeout_sleep_function from there:

# my_function.pyimport my_timeout_functiondef sleep_function(num): 
try: 
my_timeout_function.timeout_sleep_function(num) 
except: 
print('Timeout of 2.5 seconds reached!')

Because we have set a limit of 2.5 seconds for the timeout_sleep_function, the output of our main_program.py will now be:

The program slept for 1 seconds.
The program slept for 2 seconds.
Timeout of 2.5 seconds reached!
Timeout of 2.5 seconds reached!
The program slept for 1 seconds.
Timeout of 2.5 seconds reached!
The program slept for 2 seconds.
Timeout of 2.5 seconds reached!

3. Conclusion

The timeout wrapper has changed the structure of our code as follows:

1. Main program main_program.py. This initializes the multiprocessing and calls the function sleep_function from the “in-between” module my_function.py. No actual changes to the code were made.
2. The “in-between” module my_function.py. This calls the timed-out function timeout_sleep_function from the module my_timeout_function.py. Having this “in-between” module is necessary for Windows users.
3. The module my_timeout_function.py that has the actual timed-out function timeout_sleep_function.

As you can see, using the wrapt_timeout_decorator to limit execution times of functions is relatively simple. I was able to add a timeout to my 10-K annual report section separator quickly without a lot of changes to the original code. This small addition, however, caused a major reduction in execution times.

Here is a final wrap-up of the example using the wrapt_timeout_decorator module:

#----------------------------
# main_program.pyfrom multiprocessing import Pool
import my_function_timeoutmy_list = [1,2,3,4,4,3,2,1]if __name__ == '__main__': 
num_processors = 4 
p = Pool(processes = num_processors) 
p.map_async(my_function.sleep_function, my_list).get()#----------------------------
# my_function.pyimport my_timeout_functiondef sleep_function(num): 
try: 
my_timeout_function.timeout_sleep_function(num) 
except: 
print('Timeout of 2 seconds reached!')#----------------------------
# my_timeout_function.pyfrom time import sleep
from wrapt_timeout_decorator import *@timeout(2.5)
def timeout_sleep_function(num): 
sleep(num) 
print(f'The program slept for seconds.')

I hope this little guide will help you apply timeouts to functions in Python.

Источник

Demystifying Python func_timeout Module

In this article, we will be discussing the func_timeout module in Python. Let’s say a program needs to make a large function call, but it has a limited time period for the function to terminate. A practical way to limit the execution of a function call is to use func_timeout.func_timeout() from the Python func_timeout module.

About the Module

Python func_timeout module allows us to specify timeouts during existing function calls and stoppable threads. As of Python 3.10, it is NOT part of the Python standard library.

You can install it manually using PIP .

Use Case Demonstration of Python func_timeout

Let’s refer to the following sample program that implements the module.

import func_timeout def largeFunction(): while True: pass return 'PythonPool.com' def runFunction(f, max_wait, default_value): try: return func_timeout.func_timeout(max_wait, largeFunction) except func_timeout.FunctionTimedOut: pass return default_value x = runFunction(largeFunction, 5, 'Python') print(x)

Explanation

The function runFunction makes the call func_timeout.func_timeout(max_wait,largeFunction) in order to all largeFunction without arguments nor wait time ( max_wait ).

If largeFunction() fails from stopping within the time interval, an error is thrown and caught by the except block.

As largeFunction() failed to return a value within the time interval, the default value “Python” is returned.

Python timeout Function After “n” Seconds Using func_timeout

func_timeout allows us to run the given function for up to “timeout” seconds.

func_timeout(timeout, func, args=(), kwargs=None)

Any exception raised during the call will return func returns.

timeout parameter indicates the maximum number of seconds to run func before exiting.

args receives a tuple of any ordered arguments to pass to the function.

FunctionTimedOut exception is thrown upon failure to run within the given time frame.

Finally, the function will return the return value of func .

Python Function Timeout Multiprocessing

Using the multiprocessing module in Python, we can terminate a function if it breaches a time hold threshold. Refer to the following program.

from multiprocessing import Pool, TimeoutError import time def f(x): return x*x if __name__ == '__main__': with Pool(processes=4) as pool: res = pool.apply_async(f, (20,)) print(res.get(timeout=1)) . . . res = pool.apply_async(time.sleep, (10,))

Creating a Python Loop Delay without time.sleep()

timelimit = 21 while timelimit > 0 : print "%s seconds remaining" % (timelimit) timelimit = timelimit -1

We need to iterate the above loop within 20 seconds and once per each second. In this situation, we cannot use time.sleep() as it will cause the entire application to be unresponsive for a particular amount of time.

import time max = 31 start = time.time() while True: time.sleep(0.1) print( "looping. ") #Remaining time after each iteration remaining = max + start - time.time() print( "%s seconds remaining" % int(remaining)) #After final interation, break the loop if remaining 
The above implementation uses time.sleep() in order to prevent numerous print statements in the output console. It has no direct effect on the iterations.
Stopping a Python Function after a Timeout
With the help of the multiprocessing module, we can do exactly that.
import multiprocessing import time # Sample Function def myFunction(): for i in range(100): print("iteration") time.sleep(1) if __name__ == '__main__': # "Starting" myFunction as a process p = multiprocessing.Process(target=myFunction) p.start() # Waits for 10 seconds or until the function execution is over p.join(10) # If the process "p" is still alive if p.is_alive(): print("end thread") # The process is terminated p.terminate() p.join()
 
The above program allows the function to run within 10 seconds. If 10 seconds have passed, the process is automatically terminated.
 
How to Implement a StoppableThread Class in Python
 
In programming, it is precarious to terminate a thread unexpectedly. A thread may contain a vital resource to the program; closing it without proper care may cause unexpected behavior. Threads may also consist of other sub-threads which need the same attention.
 
To ensure this, we can implement an exit request flag that checks at regular intervals for the exit. The Python threading module allows us to implement this. Let’s refer to the following program
 
import threading class StoppableThread(threading.Thread): def __init__(self, *args, **kwargs): super(StoppableThread, self).__init__(*args, **kwargs) self._stop_event = threading.Event() def stop(self): self._stop_event.set() def stopped(self): return self._stop_event.is_set()
 
StoppableThread has a method stop() . As it’s called, it will exit the thread and waits to exit properly using the join() method. The thread regularly checks the exit flag.
 
FAQs on Func timeout Python
 
func_timeout works in Windows systems under the following versions of Python.
2.7, 3.4, 3.5, 3.6, 3.7 and 3.10.
 
Python func_timeout module works on all Linux and Unix-based Operating Systems under the Python versions of: 
2.7, 3.4, 3.5, 3.6, 3.7 and 3.10
 
Conclusion
 
In this article, we have reviewed the func_timeout module and how its classes allow us to control the runtime of a program. Other packages such as multiprocessing allow us to perform similar operations.
 
Источник