Have you ever had to loop over a large pile of data -
a list so large that loading it all into memory would crash your workstation?
Or perhaps you were searching for a specific element from an infinite list of possibilities.
In the past, you may have abandoned the "for-loop" and written a hacky solution.
Well hack no more,
because there is a better way to handle these scenarios.
I'm referring to generators.
And I am now going to talk about .. generators.
A generator is a function that acts like an iterator.
As the name suggests,
a generator "generates" the elements you loop over.
Think of a generator as an "on demand" iterable object.
By contrast, typical iterators loop over existing collections that are stored in memory.
Because generators provide "on demand" objects to loop over,
you can get tremendous savings in memory.
Let us see some examples - both easy and challenging.
Imagine writing a function that has 3 return lines.
Place your hand on your chin and ask yourself:
"Self, if I call this function, will I get an error?"
"Is it possible the function will actually return something?"
We will now answer your rhetorical questions.
Run..
An exception is NOT raised and the function simply returns the integer 1.
Once a function executes a return statement, it exits happily in the knowledge it has done its job.
Next, let us write a similar function except we use "yield" instead of "return."
What do you think will happen when we call this function?
Let us find out.
Run..
It does not return a number.
Instead, it returns a generator object.
Since a generator is a type of iterator, we should be able to loop over this.
So let us loop over this..
Run..
Here is a slightly more interesting example.
We will write a function that yields each lower-case letter in the English alphabet.
Instead of typing the 26 letters, we will use the string module.
This gives you convenient access to common sets of characters, including the lowercase letters.
Now we can loop over this generator.
Run..
Let us take a moment to discuss these new ideas.
A function that uses "yield" instead of "return" is called a "generator function", or simply a "generator."
In many languages, like C++, "yield" statements are used by threads to return control to another process.
So we can see why Python generators use the "yield" keyword.
When a generator calls yield, it is momentarily passing control back to the code looping over the generator values.
A way to visualize a generator function is to imagine a game of ping-pong.
In a for-loop, you pass control to the generator, which yields a value back to you.
This process goes back-and-forth, back-and-forth until the generator has run out of yields.
We will now demonstrate the full power of generators ..
by writing a generator function that yields all the prime numbers.
Other than 2, all prime numbers are odd integers.
So we will handle the case of 2 separately.
The generator function will yield 2, and then store it in a list of prime numbers.
This list will serve as a cache of the prime numbers.
Next, we need to loop over all positive, odd integers starting with 3.
To do this, we will import the "itertools" module.
Inside that module is a function called "count."
You pass in the starting value, and how much to increase by in each step.
This for loop will never end..
The algorithm we will use is not efficient, but it is clear.
First, we will assume 'n' is prime.
Then, we test to see if it is divisible by any of the prime numbers in our cache.
If it is, then it is not prime, and we can stop checking possible divisors.
But if we finish checking all the cached primes, then we have found the next prime.
So we cache the value, and yield the value back to the iteration.
We can now loop over the prime numbers.
Once we have passed 100, we stop the loop.
If we do not stop the loop, it will continue forever.
Run..
I had no doubts this would work on the first try.
I would be most interested to hear your suggestions on how to speed up this generator.
Another, more compact way to create a generator is with generator expressions.
Before discussing this technique, I would like to share with you the generator for making a Socratica video.
Video production is a very.. time.. consuming.. activity..
In fact, Socratica does not outsource a single step of our process..
First, the script.
Many channels outsource script writing, but not Socratica.
We're here, typing away on our keyboards.
A lot of channels outsource the filming of their videos, but not Socratica.
We personally film each video using a real camera and a real microphone.
You are probably asking: who edits your videos?
I'm glad you asked - we do.
Using computers...
We want to make videos more quickly, but we need to hire human help.
And to hire human help, we need *your* help, human.
One way to help is Patreon.
If 1% of subscribers gave $1 a month for 1 year, we could hire 1 human to help Socratica.
There is a very compact way to create a generator - Generator Expressions.
These are very similar to list comprehensions with one difference: you use parentheses instead of square brackets.
For example, this generator expression will generate the squares of all the positive integers.
Not just some, but ALL.
As proof, I will loop over this generator and print the squares.
Run..
I forgot to mention one thing.
This will only end when the last star burns out, or we run out of memory - whichever happens first.
I will give us a break.
This time, I will stop the generator directly rather than breaking out of the loop.
Once the value is above 500, I will call the close() method on the generator.
Run again..
That is pleasantly more finite.
One may ask: "Why not call these *tuple* comprehensions?"
"After all, they look like list comprehensions except with tuple notation."
One may answer: "Because a list comprehension returns a list, but a tuple expression does not return a tuple.."
We can see what object type is returned by printing it.
It is a generator.
The 'squares' generator will never end.
If time and space were endless, our loop would continue forever.
And what is the cost of this infinite generator?
We can find out by printing the size of the 'squares' generator in bytes.
To do so, import the 'sys' module.
Next, print the size of the 'squares' generator.
Run..
112 bytes.
If you were to use a list comprehension, it would require an infinite number of bytes.
We saw two ways to create a generator.
One: write a function and use 'yield' instead of 'return.'
Two: use a generator expression, which looks like a list comprehension with parentheses.
I wonder if it is possible to generate a generator that generates generators?
I yield this problem to you.