Lecture #16: Iterators, Generators An Iterator Confusion The - PDF document

Lecture #16: Iterators, Generators An Iterator Confusion • The distinction between iterators (things with a method) next and iterables (things from which the iter function can construct an iterator) can be confusing, and sometimes downright incovenient. • Suppose that backwards(L) returns an iterator object that returns the values in list L from last to first: class backwards: def init (self, L): self. L = L, self. k = len(L) - 1 def next (self): if self. k < 0: raise StopIteration else: self. k -= 1; return self. L[self. k + 1] • The following won’t work [why not?]: for x in backwards(L): print(x) Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 1 Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 2 An Iterator Convention Using getitem for Iterables • Problem is that for expects an iterable , but a backwards is a pure • When confronted with a type that does not implement iter , but iterator . does have a getitem , the iter function creates an iterator. • This is awkward, so the usual fix is always to define iterator objects • This in itself is an example of generic programming! to have a trivial iter method on them: • Conceptually: class backwards: class GetitemIterator: def init (self, L): def init (self, anIterable): self. L = L, self. k = len(L) - 1 """An iterator over ANITERABLE, which must implement getitem . This iterator returns ANITERABLE[0], ANITERABLE[1], ... up def iter (self): to and not including the first index that causes an return self # Now I am my own iterator IndexError or StopIteration.""" def next (self): ... • Iterators returned by Python library methods and other standard language constructs obey this convention. def next (self): ? Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 3 Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 4 Using getitem for Iterables (II) Problem: Reconstruct the range class A possible implementation: • Want Range(1, 10) to give us something that behaves like a Python range, so that class GetitemIterator: for x in Range(1, 10): def init (self, anIterable): print(x) """An iterator over ANITERABLE, which must implement getitem . This iterator returns ANITERABLE[0], ANITERABLE[1], ... up prints 1–9. to and not including the first index that causes an class Range: IndexError or StopIteration.""" ??? self. iterable = anIterable self. nextIndex = 0 def next (self): try: v = self. iterable[self. nextIndex] self. nextIndex += 1 return v except IndexError: raise StopIteration Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 5 Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 6

Reconstructing Range (I) Reconstructing Range (II) class Range: class Range: def init (self, first, end, step=1): def init (self, first, end, step=1): assert step != 0 assert step != 0 ?? self. first, self. end, self. step = first, end, step def getitem (self, k): def getitem (self, k): ?? ?? def iter (self): def iter (self): return ?? ?? Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 7 Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 8 Reconstructing Range (III) Reconstructing Range (IV) class Range: class Range: def init (self, first, end, step=1): def init (self, first, end, step=1): assert step != 0 assert step != 0 self. first, self. end, self. step = first, end, step self. first, self. end, self. step = first, end, step def getitem (self, k): def getitem (self, k): if k < 0: if k < 0: k += self. len if 0 <= k < self. len: if 0 <= k < self. len: return self. first + k * self. step else: return raise IndexError else: def iter (self): def iter (self): Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 9 Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 10 Reconstructing Range (V) Discussion class Range: • An iterator represents a kind of “deconstruction” of a loop. def init (self, first, end, step=1): • Instead of writing a loop such as assert step != 0 self. first, self. end, self. step = first, end, step x = 0 # Initialize iterator object, iterobj while x < N: # iterobj. next , part 1 def getitem (self, k): Do something using x if k < 0: x += 1 # iterobj. next , part 2 k += self. len if 0 <= k < self. len: • . . . we break it up as suggested by the comments. return self. first + k * self. step • In some cases (e.g., iterators on trees), the result can be rather else: clumsy. raise IndexError • Python provides a different, and generally clearer way to build these def iter (self): iterator objects: as generators . return GetitemIterator(self) Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 11 Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 12

Generators Generator Example: Alterative Implementation of GetitemIterator • For a generator, one writes a function that produces in sequence all >>> def GetitemIterator(iterable): the desired values by means of yield statements. ... k = 0 • When such a function is called, it executes up to, but not including, ... while True: the first yield and returns a generator object , which is a kind of ... try: iterator. ... yield iterable[k] ... k += 1 • Trivial example: ... except IndexError: ... return >>> def pairGen(x, y): >>> iterobj = GetitemIterator([1, 3, 7]) ... """A generator that yields X and then Y.""" >>> iterobj. next () ... yield x 1 ... yield y >>> iterobj. next () >>> oneTwo = pairGen(1, 2) 3 >>> oneTwo >>> for x in GetitemIterator([1, 3, 7]): print(x, end=" ") <generator object pairGen ...> 1 3 7 >>> oneTwo. next () 1 >>> oneTwo. next () 2 >>> oneTwo. next () Traceback ... StopIteration Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 13 Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 14 RList Revisited Linked Lists: Using the Iterator • Previously, we introduced rlists —recursive lists, aka linked lists . • The iterator that Python creates from is useful inter- getitem nally: • Here’s a partial version in class form: def len (self): class Link: c = 0 empty = () for in self: c += 1 def init (self, first, rest=Link.empty): return c self. first, self. rest = first, rest def str (self): def getitem (self, i): from io import StringIO if i < 0: # Negative indices count from the end. r = StringIO() # A kind of file that builds a string in memory i += len(self) print("(", file=r, end="") p = self # Actually, could use self in place of p. sep = "" while p is not empty and i > 0: for p in self: # This creates an iterator that uses getitem . p, i = p. rest, i - 1 print(sep + repr(p), file=r, end="") if p is empty: sep = ", " raise IndexError print(")", file=r, end="") return p. first return r.getvalue() Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 15 Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 16 Linked Lists: Fixing Performance Iterating Over Trees • Unfortunately, the automatic use of to create an itera- • Writing an iterator for a tree is tricky and leads to a rather complex getitem implementation. tor like this hides a performance problem. • But with a generator, it’s pretty easy: • We have to redo the work to get to the next list item on each iter- ation. def preorderLabels(T): """Generate the labels of tree T in preorder (i.e., first the node • It would be better in this case to create a specialized iterator. label, then the preorder labels of the branches.)""" class Link: yield label(T) ... for child in branches(T): def iter (self): for label in preorderLabels(child): p = self yield label while p is not Link.empty: • A recursive generator! yield p. first p = p. next • We can use for on preorderLabels(child) because Python makes all its generators into iterables, following the convention that iter- ators should implement a trivial method. iter Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 17 Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 18

Facilitating Recursive Generators • The loop in this last generator comes up with some frequency: for label in preorderLabels(child): yield label • We call the result of preorderLabels(child) a subiterator , • There is a shorthand for this loop over a subiterator: def preorderLabels(T): """Generate the labels of tree T in preorder (i.e., first the node label, then the preorder labels of the branches.)""" yield label(T) for child in branches(T): yield from preorderLabels(child) Last modified: Wed Mar 1 15:52:20 2017 CS61A: Lecture #16 19