Composing Lazy Pipelines

In our previous lesson, we learned how to use Symbol.iterator and the yield keyword to create custom iterables. We saw how a generator function allows us to pause and resume code, producing values only when we ask for them. This concept is known as lazy evaluation. While creating a single generator like a range is useful, the real power of generators is realized when we start connecting them.

In this lesson, we will learn how to build a pipeline of generators. You can think of this as a factory assembly line where each station performs one specific task, such as filtering out defective parts or painting a product. By the end of this lesson, you will be able to process massive amounts of data — even infinite sequences — without exhausting your computer's memory.

Most of us are accustomed to using array methods like .map() and .filter(). These are known as eager methods because they process the entire collection immediately. When you call .filter() on an array of one million items, JavaScript creates a brand-new array in memory to hold the results. If you then call .map() on that result, it creates yet another array. This can become very slow and consume significant memory if the dataset is large.

Another major limitation of standard arrays is that they must have a finite size. You cannot have an infinite array in JavaScript because your computer would run out of RAM trying to store it. However, because generators only produce one value at a time, we can actually work with sequences that never end. To do this effectively, we need to stop relying on built-in array methods and start building our own generator-based versions.

To build a lazy pipeline, we need utility functions that can accept an iterable and return a new iterator. Let's look at how we can recreate map and filter using the function* syntax. These functions do not create arrays; instead, they act as "wrappers" around an existing data source.

function* map(iter, fn) { 
  for (const v of iter) {
    yield fn(v); 
  }
}

function* filter(iter, fn) { 
  for (const v of iter) {
    if (fn(v)) {
      yield v;
    }
  }
}

The map generator takes an existing iterable and a transformation function. When you request a value from map, it pulls a single value from the source iterable, runs the transformation function, and yields the result. The filter generator works similarly, but it uses an if statement to check a condition. If the condition is true, it yields the value; if not, it simply moves to the next item in the source without yielding anything. Because these are generators, no work happens until you actually start looping over them.

Now that we have our building blocks, we can chain them together to create a pipeline. In this example, we will find the squares of the first five even numbers. We will combine range, filter, and map into a single flow of data.

// Lazy pipeline — nothing runs until we pull values out.
const evenSquares = map(
  filter(range(1, Infinity), (n) => n % 2 === 0),
  (n) => n * n
);

console.log("first 5 even squares:", [...take(evenSquares, 5)]);

Output:

first 5 even squares: [ 4, 16, 36, 64, 100 ]

Notice how we nest the functions inside each other. The range feeds into filter, which feeds into map, which finally feeds into take. Even though we are starting with an infinite range, the code runs instantly and uses almost no memory. This is because JavaScript is only calculating the specific numbers needed to fill that final array of five items. The rest of the infinite sequence is never even generated.

In this lesson, we moved beyond basic iteration to build complex, lazy data pipelines. We learned that generators allow us to avoid the memory overhead of large arrays by processing data one piece at a time. We also saw how we can safely handle infinite sequences by using a take utility to limit our results.

You now have the tools to build map, filter, and other custom utilities that work with any iterable object. In the upcoming practice exercises, you will implement these utilities yourself and build your own pipelines. Let's head over to the exercises to start building!