In Python, an iterable object is any object you can loop through using a for loop or while loop.
Common examples of iterables are:
For example, you can loop through a list of numbers:
numbers = [1, 2, 3] for number in numbers: print(number)
1 2 3
You can use a similar approach to loop through the characters of a string:
word = "Testing" for character in word: print(character)
T e s t i n g
This comprehensive guide teaches you everything you need to know about iterables, iterators, and generators in Python. After reading this guide, you understand:
- What the yield keyword really means
- What makes an object iterable
- What makes a function iterator
- What is a generator
- How a for loop works behind the scenes
And much more.
Introduction: Iterables and Iterators in Python
An iterable object means it implements the
__iter__ method under the hood. This method returns an iterator that can be used to loop through the iterable.
Let’s take a look at an example of a traditional non-iterable object. Here is a simple Python class called
class Course: participants = ["Alice", "Bob", "Charlie"]
Let’s create a
Course object of that class:
course = Course()
Now, let’s try to loop through the
Course object and try to print each participant on the course:
for student in course: print(student)
Traceback (most recent call last): File "example.py", line 7, in <module> for student in course: TypeError: 'Course' object is not iterable
Of course, this piece of code fails. The error says it all—The
Course is not iterable.
Trying to loop through this
course object is meaningless. Python has no idea what you’re trying to do.
To make the above code work, you need to make the
Course class iterable. To convert the
Course class into an iterable, implement the two special methods:
Here is the updated version of the
class Course: participants = ["Alice", "Bob", "Charlie"] def __iter__(self): return iter(self.participants) def __next__(self): while True: try: value = next(self) except StopIteration: break return value
Now you can loop the
Course objects with the previously failing for loop syntax:
course = Course() for student in course: print(student)
Alice Bob Charlie
At this point, you probably have no idea what the above code does. No worries! That’s what you will be focusing on for the rest of this guide. The above example is just an introduction to what you can do with iterables and iterators in Python.
To understand how this code works, you need to better understand iterators and iterables. Furthermore, you need to learn what the special methods
__next__ actually do.
Iterables and Iterators in Python
So, an iterable object in Python is something you can loop through using a for (or while) loop.
But what makes an object iterable in the first place?
To qualify as an iterable, the object has to implement the
__iter__() method under the hood.
Let’s think about some common iterable types in Python. For example, a list is clearly iterable as you can loop through it using a for loop syntax, right?
Now, let’s see what special methods a
list object implements behind the scenes. To do this, call the
dir() function on a list:
numbers = [1, 2, 3, 4, 5] print(dir(numbers))
Here is the output:
This output is a list of all the methods that
list objects implement under the hood. You can see the list has the
__iter__() method. This verifies that a Python list is indeed iterable.
__iter__() method is important because it’s what makes looping through a list possible.
For a for loop to work, it calls the
__iter__() method of a list. This method returns an iterator. The loop then uses this iterator object to step through all the values.
But what on earth is an iterator then?
An iterator is an object with a state. An iterator object remembers where it is during an iteration. Iterators also know how to get the next value in the collection. They do this by using the
__next__() method, which by the way is a method every iterator needs to have.
Let’s continue with the numbers list example and grab the iterator of the list:
numbers = [1, 2, 3, 4, 5] iter_numbers = iter(numbers)
(By the way, calling the
iter(numbers) is the same as calling
Now, let’s call the
dir() function on the iterator object to see what methods it has:
__next__() method that I talked about. This is what characterizes an iterator. Without this method, the iterator would not work.
Woah… So much new information!
Let’s recap what you have learned thus far.
- A Python list is iterable. In other words, you can call a for loop on it. To be iterable, a list implements the
__iter__()method under the hood.
__iter__()method returns an iterator object. The for loop uses the iterator object to actually step through the list values.
- To qualify as an iterator, the iterator must implement the
__next__()method for obtaining the next value in the list.
Now, let’s continue exploring the numbers list object. Let’s call the
__next__() method on the
numbers iterator a bunch of times to see what happens:
>>> numbers = [1, 2, 3, 4, 5] # specify a numbers list >>> iter_numbers = iter(numbers) # get the iterator of the list >>> next(iter_numbers) # get the next value of the list using the iterator 1 >>> next(iter_numbers) 2 >>> next(iter_numbers) 3 >>> next(iter_numbers) 4 >>> next(iter_numbers) 5
(By the way, calling the
next(numbers) is the same as calling
next(iter_numbers) always returns the next number in the
numbers list. But how on earth is this possible?
This is possible because the iterator object has an internal state. It remembers where it left off when
__next__() was called last time. So when the
__next__() method is called again, the iterator object knows what value comes next and delivers it to the caller.
As you can see from the above example, the last value you received was 5. This means you have reached the end of the numbers list.
To see what happens if you continue calling the
next() function, let’s call it once more:
>>> next(iter_numbers) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration >>>
Because there are no more values to access on the list, a
StopIteration exception is thrown. At this point, the iterator is exhausted.
The cool part of the above example is you just did what a for loop does under the hood!
When you call a for loop on an iterable, such as a list:
- It calls the
__iter__()method on a list to retrieve an iterator object.
- Then it calls the
__next__()method until there are no values left.
- When there are no values left, a
StopIterationexception is thrown. The for loop handles the exception for you. Thus you never see it.
Let’s simulate the behavior of a for loop again. This time, let’s make it less manual and use a while loop instead:
numbers = [1, 2, 3, 4, 5] # Get the iterator from numbers list iter_numbers = iter(numbers) # Start retrieving the next values indefinitely while True: try: # Try to get the next value from the iterator and print it number = next(iter_numbers) print(number) # If the iterator has no more values, escape the loop except StopIteration: break
1 2 3 4 5
The above code construct works exactly the same way as a for…in loop operates on a list.
Awesome! Now you understand what are iterables and iterators in Python. Also, you now know how a for loop truly works when iterating over a list.
Next, let’s take a look at how you can implement custom iterables and iterators.
How To Create Iterators and Iterables
There is nothing special about Python’s native iterables, such as a list or tuple. You can convert any custom classes to iterables too!
To do this, you need to implement the
__next__() methods in your class.
I’m sure you’re familiar with the built-in
range() function in Python. You can use it like this:
for i in range(4): print(i)
0 1 2 3
To demonstrate iterables and iterators, let’s create a custom implementation of
range(). Here is a class
RangeValues that mimics the behavior of the
class RangeValues: def __init__(self, start_value, end_value): self.current_value = start_value self.end_value = end_value def __iter__(self): return self def __next__(self): if self.current_value >= self.end_value: raise StopIteration value = self.current_value self.current_value += 1 return value
Now, let’s go through this implementation line by line to understand what’s happening.
__init__()method makes it possible to initialize a
RangeValuesobject with start and end values. For example:
__iter__()method makes the class iterable. In other words, it is possible to call
for i in RangeValues(0,10). This method returns an iterator. In this case, it returns the class itself, because the class is an iterator as it implements the
These lines specify how the iterator behaves when someone calls a for loop on the
__next__()method is responsible for going through the values from start to end. It raises a
StopIterationexception when it reaches the end of the range.
- If the iterator hasn’t reached the end yet, it continues returning the current value (and increments it for the next round).
Now that you understand how the code works, let’s finally test the
for i in RangeValues(1,5): print(i)
1 2 3 4
As you can see, this function now works just like the
range() function in Python.
Building an example iterable/iterator like this is a great way to get a better understanding of how the iterables work in Python.
Make sure to tweak the code and see what happens. Also, feel free to come up with your own iterable concepts and put them to test. If you only read this article without experimenting with the code, chances are you won’t understand the concepts well enough.
Now, let’s move on to generators that offer a more readable way to write iterators.
Generators—Readable Iterators in Python
If you take a look at the above example of
RangeValues class, you see it’s daunting to read.
Luckily, Python provides you with generators to remedy this problem.
A generator is an iterator whose implementation is easier to read. The readability advantage stems from the fact that a generator lets you omit the implementation of
Because a generator is also an iterator, it doesn’t return a single value. Instead, it yields (delivers) values one at a time. Besides, the generator keeps track of the state of the iteration to know what value to deliver next.
For example, let’s turn the
RangeValues class from the earlier example into a generator:
def range_values(start, end): current = start while current < end: yield current current += 1
Let’s test the function:
for i in range_values(0,5): print(i)
0 1 2 3 4
range_values works exactly like the
RangeValues class but the implementation is way cleaner.
First of all, you don’t need to specify a class. Instead, you can use a generator function like the above. Then, you don’t need to implement the
How Does ‘Yield’ Work?
In the previous example, you turned iterable into a generator to make it more readable.
But in the above example, you saw a new keyword yield. If you have never seen it before, there is no way for you to tell how it works.
When you write a generator function, you don’t return values. This is because, as you might recall, a generator only knows the current value and how to get the next value. Thus, a generator doesn’t store values in memory. This is what makes generators memory efficient.
To create a generator, you cannot return values. Instead, you need to yield them.
When Python encounters the yield keyword, it delivers a value and pauses the execution until the next call.
For example, if you have a huge text file with a lot of words, you cannot store the words in a Python list. This is because there is not enough memory for you to do that. To iterate over the words in the file, you cannot store them in memory.
This is where a generator comes in handy.
A generator picks the first word, yields it to you, and moves to the next one. It does this until there are no words in the list. This way, you don’t need to store the words in your Python program to go through them.
Make sure to read Yield vs Return to get a better understanding.
Infinite Stream of Elements with Generators
Iterators and generators only care about the current value and how to get the next one. It’s thus possible to create an infinite stream of values because you don’t need to store them anywhere.
Let’s create an infinite iterator that produces all the numbers after a starting value. Let’s use a generator function to keep it readable:
def infinite_values(start): current = start while True: yield current current += 1
(If you want to see how this could be done with a class, here it is.)
This iterator produces values from
start to infinity.
Let’s run it. (Warning: An infinite loop):
infinite_nums = infinite_values(0) for num in infinite_nums: print(num)
0 1 2 3 4 5 . . .
Syntactically it looks as if the
infinite_nums really was an infinite list of numbers after
In reality, it’s nothing but an iterator that stores the current value and knows how to get the next one.
Today you learned what iterable means in Python.
An iterable is something that can be looped over in Python.
On a low level, an iterable is an object that implements the
__iter__() method which returns an iterator.
An iterator is an object with a state. It remembers where it’s at during an iteration. To qualify as an iterator, an object must implement the
__next__() method for obtaining the next value in the iteration process.
Thanks for reading. Happy coding!