Introduction to Generators

Lesson Introduction

Welcome! Today, we will learn about an exciting Python feature called generators . Generators help us write code that uses memory efficiently, which is crucial when handling large amounts of data. By the end of this lesson, you'll understand what generators are and how to create and use them in Python. You’ll learn how they can make your code more efficient, especially when working with large datasets.

Why Understand Generators?

Imagine you have a vast book. You will read one sentence at a time rather than holding the whole book in your head at once. Generators in Python work similarly: they let us handle large data collections one item at a time.

Concept of Generators

Generators are special Python functions that return one item at a time, which helps process large datasets efficiently. Unlike normal functions that return a list of items and hold all items in memory at once, generators yield items one by one, using less memory and processing time. Here's a quick comparison: A normal function returns all items at once (like a big bag of candies). A generator yields one item at a time (giving one candy at a time).

Example: Using a Normal Function to Return a List

Let's look at a normal function that returns a list of numbers from 1 to 5. Python def normal_function(): return [1, 2, 3, 4, 5] def main(): numbers = normal_function() for number in numbers: print(number) # Output: 1, 2, 3, 4, 5 if __name__ == "__main__": main() This function returns a list of five numbers. Now, let's see how generators do it differently.

Example of a Simple Generator

Here is a simple_generator function: Python def simple_generator(): yield 1 yield 2 yield 3 yield 4 yield 5 The simple_generator function uses the yield keyword. Unlike return, which exits the function, yield pauses the function and saves its state. The yield keyword is what makes a function a generator. Here’s what happens when a function containing yield is called: Creates a Generator Object : Instead of running the function, it returns a generator object. Pausing and Resuming : When the generator's__next__() method is called (e.g., via next() function or a for loop), the function runs until it hits yield . Saves State and Returns Value : The function pauses at yield, saves its current state (local variables, execution point), and returns the yielded value. Resumes from Last State : When__next__() is called again, it resumes right after the last yield statement, with all its variables intact.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

To run the generator, we initialize it and iterate through it using a for loop.

Python

def simple_generator():
    yield 1
    yield 2
    yield 3
    yield 4
    yield 5

# Initializing the generator
generator = simple_generator()

# Using a for loop to go through each item
for number in generator:
    print(number)  # Output: 1, 2, 3, 4, 5

def simple_generator():
    yield 1
    yield 2
    yield 3
    yield 4
    yield 5

# Initializing the generator
generator = simple_generator()

# Using a for loop to go through each item
for number in generator:
    print(number)  # Output: 1, 2, 3, 4, 5

When the loop runs: the generator yields 1, the loop prints 1, and the generator pauses. The loop runs again; the generator yields 2, and the loop prints 2, and so on.

An alternative way of executing the generator is the next method.

Python

def simple_generator():
    yield 1
    yield 2
    yield 3
    yield 4
    yield 5

# Initializing the generator
generator = simple_generator()

# Using next to go through generator's items:
print(next(generator))  # 1
print(next(generator))  # 2
print(next(generator))  # 3
print(next(generator))  # 4
print(next(generator))  # 5

def simple_generator():
    yield 1
    yield 2
    yield 3
    yield 4
    yield 5

# Initializing the generator
generator = simple_generator()

# Using next to go through generator's items:
print(next(generator))  # 1
print(next(generator))  # 2
print(next(generator))  # 3
print(next(generator))  # 4
print(next(generator))  # 5

Each time we call next(generator), the __next__() method of the generator object is executed, which yields the next item.

Generators manage large datasets without using much memory. For example, a generator can efficiently read a huge log file line-by-line rather than loading the whole file into memory.

Let's look at a generator function that reads a large file line by line. Before we dive in, let's recall that the open function is used to open a file and returns a file object. It is typically used in a with statement for proper resource management, ensuring the file is closed after its block of code is executed.

Python

def read_large_file(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            yield line

# Using the generator
for line in read_large_file('big_log.txt'):
    print(line.strip())  # Output: Each line of the file, stripped of leading/trailing whitespace

def read_large_file(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            yield line

# Using the generator
for line in read_large_file('big_log.txt'):
    print(line.strip())  # Output: Each line of the file, stripped of leading/trailing whitespace

There will be a fun practice later that includes creating a log file generator with an actual log file to work with.

Sometimes, you may want to create a generator on the fly without a full function definition. Python provides a shorthand for creating simple generators called generator expressions.

A generator expression looks very similar to a list comprehension but uses parentheses instead of square brackets.

Here is an example:

Python

# List comprehension
numbers_list = [x * x for x in range(5)]
print(numbers_list)  # Output: [0, 1, 4, 9, 16]

# Generator expression
numbers_generator = (x * x for x in range(5))

# Using the generator
for number in numbers_generator:
    print(number)  # Output: 0, 1, 4, 9, 16

# List comprehension
numbers_list = [x * x for x in range(5)]
print(numbers_list)  # Output: [0, 1, 4, 9, 16]

# Generator expression
numbers_generator = (x * x for x in range(5))

# Using the generator
for number in numbers_generator:
    print(number)  # Output: 0, 1, 4, 9, 16

Generator expressions are a compact and memory-efficient way to generate sequences on the fly.

Great job! You've learned a lot about generators today. Let’s recap:

Generators are special functions that yield items one at a time.
They use the yield keyword to pause and resume their state.
Generators are memory-efficient and can improve code performance, especially with large datasets.
Generator expressions provide a compact way to create generators.

Now, it's time to put what you have learned into practice. You will create and use your own generators in different scenarios. This hands-on practice will help you solidify your understanding of how generators can make your code more efficient and powerful. Good luck!

Lesson Introduction

Why Understand Generators?

Concept of Generators

Example: Using a Normal Function to Return a List

Example of a Simple Generator

Join the 1M+ learners on CodeSignal

Executing the Generator: a Loop

Executing the Generator: the next Method

Real-Life Applications of Generators

Introducing Generator Expressions

Lesson Summary