Composing Lazy Pipelines

In our previous lesson, we learned how to use Symbol.iterator and the yield keyword to create custom iterables. We saw how a generator function allows us to pause and resume code, producing values only when we ask for them. This concept is known as lazy evaluation. While creating a single generator like a range is useful, the real power of generators is realized when we start connecting them.

In this lesson, we will learn how to build a pipeline of generators. You can think of this as a factory assembly line where each station performs one specific task, such as filtering out defective parts or painting a product. By the end of this lesson, you will be able to process massive amounts of data — even infinite sequences — without exhausting your computer's memory.

Most of us are accustomed to using array methods like .map() and .filter(). These are known as eager methods because they process the entire collection immediately. When you call .filter() on an array of one million items, JavaScript creates a brand-new array in memory to hold the results. If you then call .map() on that result, it creates yet another array. This can become very slow and consume significant memory if the dataset is large.

Another major limitation of standard arrays is that they must have a finite size. You cannot have an infinite array in JavaScript because your computer would run out of RAM trying to store it. However, because generators only produce one value at a time, we can actually work with sequences that never end. To do this effectively, we need to stop relying on built-in array methods and start building our own generator-based versions.

To build a lazy pipeline, we need utility functions that can accept an iterable and return a new iterator. Let's look at how we can recreate map and filter using the function* syntax. These functions do not create arrays; instead, they act as "wrappers" around an existing data source.

function* map(iter, fn) { 
  for (const v of iter) {
    yield fn(v); 
  }
}

function* filter(iter, fn) { 
  for (const v of iter) {
    if (fn(v)) {
      yield v;
    }
  }
}

The map generator takes an existing iterable and a transformation function. When you request a value from map, it pulls a single value from the source iterable, runs the transformation function, and yields the result. The filter generator works similarly, but it uses an if statement to check a condition. If the condition is true, it yields the value; if not, it simply moves to the next item in the source without yielding anything. Because these are generators, no work happens until you actually start looping over them.