Deterministic Readable Streams
Generating Deterministic Readable Streams
Hello, stream builder! 🌊 In the previous lesson, the sample route forwarded a fresh file Readable through an awaited HTTP pipeline(), while the inspection route consumed another fresh source with async iteration. This lesson keeps the fresh-source rule, but the source now generates deterministic NDJSON observations lazily as a consumer advances.
The completed flow is:
GET /api/lab/generated?records=N defaults to ten records, accepts values from 1 through 10_000, and returns JSON 400 INVALID_QUERY errors for invalid controls. The generated records are deterministic: the same count always produces the same sequence.
Prior knowledge: existing stream boundaries (optional reference)
Units 1 and 2 established fresh file sources, HTTP forwarding, and direct Readable inspection. Keep this recap collapsed while focusing on lazy generated sources and validation.
Lazy deterministic sources are useful for synthetic test datasets, repeatable load fixtures, exports, event feeds, and pagination adapters. Generating records only as they are consumed avoids eagerly building large payloads, while bounded request controls cap the work one request can ask the server to perform.
As with the file route, this generated route streams its output without calculating or advertising a representation length. Stream delivery and whether a total length is known are separate design choices.
Start with Fresh Sources and Observation Boundaries
The existing file source is a factory. Each call creates a new consumable stream instance.
backend/src/streams/file-source.ts keeps file-stream construction independent from routes.
- The factory receives
samplePathrather than owning configuration. createReadStream()returns a newReadStreameach time, so separate operations do not try to reuse a consumed source.- It creates a stream without first loading the complete file into application memory.
backend/src/routes/sample.ts forwards one fresh source into the HTTP response.
- The route creates the source inside the handler, making it request-specific.
pipeline()forwards source values intoServerResponseand completes the response when the source ends.- Awaiting the pipeline keeps stream completion and failures connected to the request lifecycle.
- The route does not call
response.end()afterward becausepipeline()owns destination completion.
The inspection route consumes a different fresh source directly instead of forwarding its body.
backend/src/routes/inspect.ts composes source creation, iteration, and JSON response generation.
- This route also creates a fresh stream, even though it uses the same fixture path.
inspectReadable()consumes the source beforesendJson()sends the completed metadata.- The route returns
{ source, chunks, bytes }, not the already-consumed fixture body. - This is a different observation boundary from the browser-facing sample route.
backend/src/streams/inspect-readable.ts counts values yielded directly by an async iterable.
AsyncIterable<unknown>describes the capability required by the helper without coupling it to filesystem streams.for await...ofwaits for source values and retains only numeric counters.Buffer.byteLength()measures encoded string bytes, whileUint8Array.byteLengthmeasures binary chunks.- Source-readable chunks, HTTP chunks, and browser-reader chunks do not need to have matching boundaries. Complete byte totals are the meaningful comparison.
