What is generator object python

What Are Generators in Python?

Have you ever dealt with datasets so large that your computer’s memory couldn’t handle them? Have you ever wondered if there could be a way to interrupt a function in the middle before resuming it? Here is where Python generators come into the picture.

Python generators are a way to create custom iterators. You can read more about iterators here. Before we continue, if you are not familiar with Python terminology, check our articles about Python terms for beginners and more Python terms. And if you’re not comfortable with operations on data structures in Python, you might want to try our Built-In Algorithms in Python course.

You can loop over generator objects as you would with a list. But unlike lists, generators do not store their contents in memory. A widespread use case is when you have to deal with files bigger than your machine’s memory can handle, e.g. a large dataset. Trying to open such a file would result in a MemoryError .

By using a Python generator, you can avoid such an issue. But wait! How do you define Python generators?

How to Define Generators in Python

A Python generator is very similar to a regular Python function, but we end it with yield instead of the return keyword. Let’s write a quick example using a for loop.

def regular_function(x): for i in range(x): return i*5

Once executed as regular_function(10) , this regular function will return 0, because the execution stops after the first iteration.

However, let’s write it a bit differently:

def generator(x): for i in range(x): yield i*5

The Python yield keyword indicates that we initiated a generator object; it’s here to control the flow of the Python generator. When the program reaches it, the function’s execution is paused, and the value from yield is returned.

At this point, the state of the function is saved and the function resumes its execution whenever you call one of the generator’s methods. A return statement stops the function altogether.

Next, we instantiate the generator object as g :

To execute the generator in Python, we need to use the next() method. In the following example, we include print statements to get some outputs:

>>> print(next(g)) 0 >>> print(next(g)) 5 >>> print(next(g)) 10 >>> print(next(g)) 15

While the next() method is specific to generators in Python, it is not the only way to pause or end a for loop.

Another way to define a generator in Python is to use generator comprehensions. Very similar to list comprehensions, generators comprehensions can be defined as:

gen_comp = (i*5 for i in range(10))

Compared to list comprehensions, generator comprehensions have the advantage of not building and holding the entire object in memory before iteration. Let’s compare a generator composition with a list comprehension:

list_comp = [i*5 for i in range(100000)] gen_comp = (i*5 for i in range(10000))

These expressions look very similar; the only differences are the brackets and the parenthesis. Even so, they are actually very different. Let’s have a look at their size:

>>> import sys >>> list_comp >>> print('list comprehension:', sys.getsizeof(list_comp), 'bytes') list comprehension: 87616 bytes >>> gen_comp >>> print('generator comprehension:', sys.getsizeof(gen_comp), 'bytes') generator comprehension: 112 bytes

In this case, the list object is about 782 times larger than the generator object. Therefore, if memory is an issue, you would be better off using a Python generator.

Last but not least, there is no difference between a regular generator and generator comprehensions besides the syntax. The only difference is that generator comprehensions are single-liners.

If you need to define an infinite loop for some reason, you will need to use a Python generator. While your sequence can be infinite, your computer’s memory is certainly not.

def infinity(): n = 0 while True: yield n*n n += 13

We initialize a variable n and start an infinite loop. The keyword yield will capture the initial state and imitate the action of range() ; finally, we increment n by 13. This program will continue with a for loop until we manually stop it.

In our case, by calling next() , we can manually iterate repeatedly, which is helpful to test the generator to make sure that it produces the expected output. Does this mean that the generator can continue indefinitely?

How to End Generators in Python

First, it can stop naturally. In other words, once all the values have been evaluated, the iteration will stop, and the for loop will exit.

If you use next() , you will get an explicit StopIteration exception.

Another way to end a Python generator is to use the close() method, as follows:

>>> def generator(x): . for i in range(x): . yield i*5 >>> g = generator(10) >>> print(next(g)) 0 >>> print(next(g)) 5 >>> print(g.close()) None

The close() method will throw a GeneratorExit at the yield value and stop the execution of the generator. It can be handy to control the flow of an infinite generator.

Closing Thoughts on Generators in Python

In this article, we learned about generators in Python. We discovered how they could be helpful to deal with memory-intensive computations and how they can provide us with more flexibility over our functions (e.g. when testing an output).

I encourage you to explore this article’s examples further and to check out the Python documentation for more information. Our Python Basics courses can also help new programmers get hands-on coding experience. No previous IT knowledge is required.

Last but not least, do not forget to check our other articles on LearnPython.com. Happy learning!

Источник

Python Generators

In Python, a generator is a function that returns an iterator that produces a sequence of values when iterated over.

Generators are useful when we want to produce a large sequence of values, but we don’t want to store all of them in memory at once.

Create Python Generator

In Python, similar to defining a normal function, we can define a generator function using the def keyword, but instead of the return statement we use the yield statement.

def generator_name(arg): # statements yield something

Here, the yield keyword is used to produce a value from the generator.

When the generator function is called, it does not execute the function body immediately. Instead, it returns a generator object that can be iterated over to produce the values.

Example: Python Generator

Here’s an example of a generator function that produces a sequence of numbers,

def my_generator(n): # initialize counter value = 0 # loop until counter is less than n while value < n: # produce the current value of the counter yield value # increment the counter value += 1 # iterate over the generator object produced by my_generator for value in my_generator(3): # print each value produced by generator print(value)

In the above example, the my_generator() generator function takes an integer n as an argument and produces a sequence of numbers from 0 to n-1 .

The yield keyword is used to produce a value from the generator and pause the generator function's execution until the next value is requested.

The for loop iterates over the generator object produced by my_generator() , and the print statement prints each value produced by the generator.

We can also create a generator object from the generator function by calling the function like we would any other function as,

generator = my_range(3) print(next(generator)) # 0 print(next(generator)) # 1 print(next(generator)) # 2

Python Generator Expression

In Python, a generator expression is a concise way to create a generator object.

It is similar to a list comprehension, but instead of creating a list, it creates a generator object that can be iterated over to produce the values in the generator.

Generator Expression Syntax

A generator expression has the following syntax,

(expression for item in iterable)

Here, expression is a value that will be returned for each item in the iterable .

The generator expression creates a generator object that produces the values of expression for each item in the iterable , one at a time, when iterated over.

Example 2: Python Generator Expression

# create the generator object squares_generator = (i * i for i in range(5)) # iterate over the generator and print the values for i in squares_generator: print(i)

Here, we have created the generator object that will produce the squares of the numbers 0 through 4 when iterated over.

And then, to iterate over the generator and get the values, we have used the for loop.

Use of Python Generators

There are several reasons that make generators a powerful implementation.

1. Easy to Implement

Generators can be implemented in a clear and concise way as compared to their iterator class counterpart. Following is an example to implement a sequence of power of 2 using an iterator class.

class PowTwo: def __init__(self, max=0): self.n = 0 self.max = max def __iter__(self): return self def __next__(self): if self.n > self.max: raise StopIteration result = 2 ** self.n self.n += 1 return result

The above program was lengthy and confusing. Now, let's do the same using a generator function.

def PowTwoGen(max=0): n = 0 while n < max: yield 2 ** n n += 1

Since generators keep track of details automatically, the implementation was concise and much cleaner.

2. Memory Efficient

A normal function to return a sequence will create the entire sequence in memory before returning the result. This is an overkill, if the number of items in the sequence is very large.

Generator implementation of such sequences is memory friendly and is preferred since it only produces one item at a time.

3. Represent Infinite Stream

Generators are excellent mediums to represent an infinite stream of data. Infinite streams cannot be stored in memory, and since generators produce only one item at a time, they can represent an infinite stream of data.

The following generator function can generate all the even numbers (at least in theory).

def all_even(): n = 0 while True: yield n n += 2

4. Pipelining Generators

Multiple generators can be used to pipeline a series of operations. This is best illustrated using an example.

Suppose we have a generator that produces the numbers in the Fibonacci series. And we have another generator for squaring numbers.

If we want to find out the sum of squares of numbers in the Fibonacci series, we can do it in the following way by pipelining the output of generator functions together.

def fibonacci_numbers(nums): x, y = 0, 1 for _ in range(nums): x, y = y, x+y yield x def square(nums): for num in nums: yield num**2 print(sum(square(fibonacci_numbers(10)))) # Output: 4895

This pipelining is efficient and easy to read (and yes, a lot cooler!).

Table of Contents

Источник