Hello, stream builder! 🌊 This backend already exposes GET /api/lab/generated?records=N; the goal is to make its deterministic NDJSON source respond to readable-side demand rather than eagerly producing every record.
A successful generated request returns newline-delimited JSON records with sequential IDs, predictable temperatures, and rotating sites. Deterministic output makes the behavior testable, but it does not mean eager output. A pull-driven readable produces until its internal readable queue declines another chunk, then resumes from the correct next record when Node.js asks again.
The response should be 200, use an NDJSON content type, include x-stream-lab-mode: pull-readable, and contain ten records. The provided Run generated stream control makes the same backend request and displays its NDJSON response.
HTTP or browser chunks are byte observations. They do not prove that one received chunk came from one push() call, or that one chunk contains exactly one NDJSON record. Newlines define the record boundaries.
Established HTTP request, query, and application-error reference
Follow the Generated Request
The generated route is registered in backend/src/http/router.ts. This branch identifies the endpoint and passes the original request URL to the route handler.
TypeScript
if (pathname === GENERATED_PATH) { return sendGenerated(response, request.url); }
GENERATED_PATH is /api/lab/generated, and this router accepts the route only for GET requests.
The router delegates rather than creating observations itself. Record generation belongs to the readable source.
Passing request.url preserves the query string so the generated route can validate the optional records parameter.
Validate the Record Count Before Streaming
backend/src/http/query.ts converts an optional query value into a trusted integer before the source receives it.
TypeScript
export interface IntegerParamOptions { defaultValue: number; min: number; max: number;}export const integerParam = ( url: URL, key: string, options: IntegerParamOptions,): number => { const raw = url.searchParams.get(key); if (raw === null) return options.defaultValue; const normalized = raw.trim(); if (!/^-?\d+$/.test(normalized)) { throw new HttpError(400, "INVALID_QUERY", `${key} must be an integer.`); } const parsed = Number(normalized); if ( !Number.isSafeInteger(parsed) || parsed < options.min || parsed > options.max ) { throw new HttpError( 400, "INVALID_QUERY", `${key} must be between ${options.min} and ${options.max}.`, ); } return parsed;};
The generated route supplies a default of 10 records and accepts inclusive values from 1 through 10_000.
When records is absent, raw === null returns the configured default, so /api/lab/generated produces ten records.
The regular expression rejects decimal values, empty strings, and non-numeric text before conversion.
Invalid values throw HttpError with status 400, keeping raw HTTP query parsing outside the readable source.
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Preserve the Observation Contract
backend/src/domain/observation.ts defines the object created for each generated index.
The generated source produces Observation values with id, temperatureC, and site.
TemperatureObservation adds a Fahrenheit field for a separate transformation capability; it is not part of generated source output.
An observation object is not yet NDJSON. The source creates the wire representation by serializing the object and appending \n.
Stable fields let tests assert exact IDs at the boundaries of a requested range.
Connect HTTP to a Fresh Source
backend/src/routes/generated.ts owns HTTP concerns: query validation, success headers, and connecting a new readable source to the response.
import type { ServerResponse } from "node:http";import { pipeline } from "node:stream/promises";import { integerParam } from "../http/query.js";import { createObservationSource } from "../streams/observation-source.js";export const sendGenerated = async ( response: ServerResponse, requestUrl: string | undefined,): Promise<void> => { const url = new URL(requestUrl ?? "/api/lab/generated", "http://localhost"); const records = integerParam(url, "records", { defaultValue: 10, min: 1, max: 10_000, }); response.writeHead(200, { "content-type": "application/x-ndjson; charset=utf-8", "x-stream-lab-mode": "pull-readable", "x-stream-lab-records": String(records), }); await pipeline(createObservationSource({ records }), response);};
integerParam runs before response.writeHead(200, ...), so invalid requests can still receive a structured error response rather than a committed success status.
The content type declares NDJSON, x-stream-lab-mode documents the source strategy, and x-stream-lab-records reports the validated count as a header string.
pipeline() connects the readable source to the writable HTTP response and awaits completion.
createObservationSource({ records }) runs for each request, creating independent production state. The route deliberately does not pass a highWaterMark; that is a source-test option, not an HTTP API option.
Why Header Ordering Matters
The request boundary in backend/src/app.ts handles errors that escape from routes.
Before response headers are sent, an HttpError can become a response with its intended status and error details.
After successful headers are sent, the server cannot replace them with a different error response, so it destroys the response instead.
This is why generated-query validation must finish before writeHead(200, ...).
This error path does not control normal readable completion; push(null) handles normal end-of-stream behavior.
Build the Deterministic Pull-Driven Source
The central implementation is backend/src/streams/observation-source.ts. Its first section imports Readable and keeps deterministic record creation separate from stream control.
records is required because a source needs a known endpoint for production.
nextIndex belongs to one ObservationReadable instance and always identifies the next record that has not yet been produced.
highWaterMark is optional and exists for focused source tests that need a constrained readable queue.
When no override is supplied, super({}) leaves Node.js’s normal readable buffering behavior in place. A generated HTTP request does not expose this setting.
The final section of backend/src/streams/observation-source.ts implements demand-driven production and exports the factory used by the route.
_read() responds to readable-side demand. It must not blindly generate every record regardless of available queue capacity.
Each loop iteration creates one observation, serializes one NDJSON line, and pushes that text with UTF-8 encoding.
nextIndex advances before the push() result is examined. Even when push() returns false, that record has already entered the readable side, so the next _read() must start with the following index.
A false result means production stops immediately. When every requested record has been produced, push(null) signals end-of-stream.
The factory returns a fresh ObservationReadable, preventing progress from one source instance from leaking into another request.
Model Readable-Side Demand
A pull-driven source follows this sequence:
Node.js needs more readable-side data and invokes _read().
The source creates the observation at nextIndex.
The source advances nextIndex immediately.
The source pushes one serialized NDJSON line.
If push() returns false, the source returns without producing another record.
After the consumer makes room, Node.js can invoke _read() again.
The source resumes from its stored index and eventually pushes null after the final record.
Continuing after false would ignore readable-side backpressure and overproduce into the queue.
High-Water Marks Are Thresholds
A high-water mark is a threshold, not a hard capacity limit. Node accepts the chunk passed to push() even when that chunk reaches or exceeds the threshold, so the queue can be larger than its configured high-water mark.
const accepted = this.push(line); // a 20-byte line can raise a 60-byte queue above a 64-byte HWM// accepted is false, but the line is already queued
false controls future production: return from _read() without pushing another line, then resume only when Node asks again.
Prove Resume Behavior with Source Tests
backend/test/observation-source.test.ts tests the source without HTTP. The helper creates a constrained source and drains it through async iteration.
import assert from "node:assert/strict";import test from "node:test";import { createObservationSource } from "../src/streams/observation-source.js";const collectText = async (records: number, highWaterMark: number) => { const source = createObservationSource({ records, highWaterMark }); let body = ""; let readCalls = 0; let pressureStops = 0; let pushesAfterPressure = 0; let pressureSeenInRead = false; const read = source._read.bind(source); source._read = () => { readCalls += 1; pressureSeenInRead = false; read(); }; const push = source.push.bind(source); source.push = ((chunk: string | null, encoding?: BufferEncoding) => { if (chunk !== null && pressureSeenInRead) { pushesAfterPressure += 1; } const accepted = push(chunk, encoding); if (chunk !== null && !accepted) { pressureStops += 1; pressureSeenInRead = true; } return accepted; }) as typeof source.push; for await (const chunk of source) { body += chunk.toString(); } return { source, body, readCalls, pressureStops, pushesAfterPressure };};
createObservationSource({ records, highWaterMark }) gives the test direct control over the number of records and the readable queue threshold.
The temporary _read() and push() overrides in this helper are test-only instrumentation; production sources should not replace stream methods at runtime.
for await...of consumes chunks until the source ends, without assuming that one iterated chunk equals one NDJSON line.
The helper concatenates complete text before a test splits it on newlines and parses each JSON record.
Whole-body collection is intentionally limited to these small source-level tests; production routes keep data in a streaming pipeline.
Returning both source and body allows a test to inspect configured buffering as well as emitted output.
The next portion of backend/test/observation-source.test.ts checks a 25-record stream under a small readable buffer.
test("pull-driven source produces every record under a small buffer", async () => { const { source, body, readCalls, pressureStops, pushesAfterPressure } = await collectText(25, 64); const rows = body.trim().split("\n").map((line) => JSON.parse(line)); assert.equal(source.readableHighWaterMark, 64); assert.ok(pressureStops > 0); assert.ok(readCalls > 1); assert.equal(pushesAfterPressure, 0); assert.equal(rows.length, 25); assert.equal(rows[0].id, "generated-000001"); assert.equal(rows[24].id, "generated-000025");});
The readableHighWaterMark assertion verifies that the explicit source option reached Node’s Readable constructor.
trim() removes the final NDJSON newline before splitting the completed text into records.
Checking rows.length proves that the source produced all 25 requested records.
The first and final ID assertions verify the one-based, six-digit padded ID contract at both boundaries.
The final test in backend/test/observation-source.test.ts detects the duplicate-on-resume bug that a count-only assertion could miss.
test("zero duplicate ids are introduced when reads resume", async () => { const { body } = await collectText(100, 48); const ids = body .trim() .split("\n") .map((line) => (JSON.parse(line) as { id: string }).id); assert.equal(ids.length, 100); assert.equal(new Set(ids).size, 100);});
A 48 high-water mark deliberately constrains the source’s readable queue.
The first assertion catches truncation by requiring 100 emitted IDs.
A Set preserves only unique values, so its size must also be 100.
Together, the assertions catch a source that repeats one record while skipping another, even if it still produces 100 lines.
Lock In the HTTP Contract
backend/test/app.test.ts creates an isolated server for each HTTP test and closes it even if an assertion fails.
import assert from "node:assert/strict";import { once } from "node:events";import { readFile } from "node:fs/promises";import type { AddressInfo } from "node:net";import test from "node:test";import { createApp } from "../src/app.js";import type { DatabaseClient } from "../src/db/client.js";const unavailableDatabase: DatabaseClient = { state: () => "degraded", ping: async () => "degraded", query: async () => { throw new Error("database unavailable"); }, close: async () => undefined,};const withServer = async ( run: (baseUrl: string) => Promise<void>,): Promise<void> => { const server = createApp({ database: unavailableDatabase, samplePath: new URL("../data/sample.ndjson", import.meta.url).pathname, startedAt: Date.now() - 12, }); server.listen(0, "127.0.0.1"); await once(server, "listening"); try { await run(`http://127.0.0.1:${(server.address() as AddressInfo).port}`); } finally { server.close(); await once(server, "close"); }};
The database stub satisfies the application dependency without requiring these endpoint tests to contact a database.
Listening on port 0 lets the operating system choose an available local port.
The callback receives the actual base URL, so each test uses ordinary HTTP fetch requests.
The finally block ensures the server closes after both successful and failing assertions.
The next portion of backend/test/app.test.ts preserves the sample regression test and verifies the public generated-endpoint contract.
This test confirms that the filtered transform route remains available after generated-source work.
It verifies a successful response and one NDJSON line for this deterministic request.
It is an HTTP regression check, not part of the generated source’s _read() implementation.
Keeping it alongside the generated test protects cumulative backend behavior.
Keep the Responsibilities Separate
The completed design divides responsibilities cleanly:
The route validates records, writes HTTP metadata, creates a fresh source, and pipelines it to the response.
ObservationReadable owns deterministic record production, its readable queue, and its resume position.
push() returning false tells the source to stop producing immediately.
nextIndex advances before that result is checked, preventing duplicate records when Node.js later requests more data.
Direct source tests prove constrained-buffer resume safety; HTTP tests protect the externally visible response contract.
This lesson focuses on producer-side readable backpressure: _read(), readable queue capacity, push() acceptance, and correct resumption. Writable pressure, drain, slow destinations, and pressure metrics are outside this scope.