Propagating Pipeline Failures Through a Terminal JSON Endpoint
Hello, pipeline investigator! 🧪 In the last lesson, one live HTTP operation owned a shared abort signal so its paced NDJSON pipeline could stop safely on disconnect, deadline, or shutdown. This lesson uses the same promise-based pipeline idea for a different terminal condition: an internal object-mode stage deliberately fails at an exact boundary, the pipeline settles at an internal destination, and only then can the server choose one JSON diagnostic.
The completed endpoint is:
text
GET /api/lab/failure?records=20&failAfter=4
Unlike the live route, this endpoint does not stream NDJSON to the client. It runs a complete internal pipeline and returns one JSON document only after that pipeline has completed or failed.
That response timing is essential. Once an HTTP response has started sending NDJSON bytes, the server cannot safely replace partial stream output with a JSON error body. Here, the pipeline ends at a private writable destination, so no response has been committed while the server determines the terminal result.
Follow the Byte and Object Boundaries
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
The failure belongs between parsing and stringification, where the pipeline handles validated observation objects:
In backend/src/streams/observation-source.ts, this region creates deterministic generated observations and serializes each one as newline-delimited JSON.
JSON.stringify(observation) converts an in-memory observation into serialized JSON, and \n makes that value one NDJSON row.
The source emits serialized stream data rather than Observation objects. A later stage must parse and validate those rows before object-mode code can use them.
records determines how many generated observations exist. If failAfter equals records, the source can end before another observation arrives to trigger the failure boundary.
When push() returns false, _read() stops producing until Node requests more data.
backend/src/streams/ndjson-parser.ts converts incoming chunks into validated Observation objects. Its parsing helper rejects malformed JSON and values that do not satisfy the observation contract.
const observationFromLine = (line: string): Observation => { const value: unknown = JSON.parse(line); if (typeof value !== "object" || value === null || Array.isArray(value)) { throw new Error("Each NDJSON row must be an object."); } const row = value as Record<string, unknown>; if ( typeof row.id !== "string" || row.id.length === 0 || typeof row.site !== "string" || row.site.length === 0 || typeof row.temperatureC !== "number" || !Number.isFinite(row.temperatureC) ) { throw new Error("Each NDJSON row must be a valid observation."); } return { id: row.id, site: row.site, temperatureC: row.temperatureC, };};
JSON.parse() produces unknown, because valid JSON is not automatically a valid application observation.
The first condition rejects primitive values, null, and arrays. The second verifies the required id, site, and finite temperatureC fields.
The returned value has the Observation shape consumed by the fault transform.
A parser error is an ordinary pipeline failure, not the expected injected failure diagnostic.
The rest of backend/src/streams/ndjson-parser.ts handles chunk boundaries and reports parsing errors through the transform callback.
readableObjectMode: true changes the parser’s readable side so downstream stages receive JavaScript objects instead of byte-oriented chunks.
A chunk can contain several rows or end in the middle of one. remainder preserves an incomplete final segment until a later chunk completes it.
_flush() handles a final non-empty row when the source ends without a trailing newline.
Calling callback(error) causes a connected promise-based pipeline() to reject. That error must remain distinct from InjectedStreamError.
After the object-mode section, backend/src/streams/ndjson-stringifier.ts converts values back into NDJSON-compatible output.
import { Transform, type TransformCallback } from "node:stream";import type { TemperatureObservation } from "../domain/observation.js";export class NdjsonStringifier extends Transform { constructor() { super({ writableObjectMode: true }); } override _transform( observation: TemperatureObservation, _encoding: BufferEncoding, callback: TransformCallback, ): void { callback(null, `${JSON.stringify(observation)}\n`); }}export const createNdjsonStringifier = (): NdjsonStringifier => new NdjsonStringifier();
writableObjectMode: true allows this transform to receive objects from the preceding stage.
The callback’s second argument becomes the readable output: a serialized NDJSON line.
The stringifier stays in the internal experiment even though its output is discarded. This proves the full byte-to-object-to-byte flow is connected.
The fault boundary occurs before this stage, so the observation that triggers failure is never stringified.
Create an Exact Failure Boundary
backend/src/streams/fault-transform.ts is the object-mode stage that defines the deterministic failure behavior.
import { Transform, type TransformCallback } from "node:stream";import type { Observation } from "../domain/observation.js";export class InjectedStreamError extends Error { constructor(public readonly processedRecords: number) { super(`Injected stream failure after ${processedRecords} records.`); this.name = "InjectedStreamError"; }}export class FaultTransform extends Transform { private processedRecords = 0; constructor(private readonly failAfter: number) { super({ writableObjectMode: true, readableObjectMode: true }); } override _transform( observation: Observation, _encoding: BufferEncoding, callback: TransformCallback, ): void { if (this.processedRecords >= this.failAfter) { callback(new InjectedStreamError(this.processedRecords)); return; } this.processedRecords += 1; callback(null, observation); }}export const createFaultTransform = (failAfter: number): FaultTransform => new FaultTransform(failAfter);
InjectedStreamError is the dedicated expected-error type. Its processedRecords property carries structured diagnostic data, while its name and message identify the error clearly.
The transform enables object mode on both sides because it receives parsed Observation values and forwards those same objects unchanged.
processedRecords means records already forwarded. The comparison happens before incrementing and forwarding the current observation.
callback(new InjectedStreamError(...)) fails the transform without passing the boundary observation downstream. callback(null, observation) forwards the original observation in the successful path.
The boundary is exact:
failAfter = 4records 1–4: increment and forwardrecord 5: processedRecords is 4, so fail
With failAfter: 0, the first observation reaches the failure branch immediately. Nothing is forwarded.
Using >= expresses the intended invariant: once the forwarding limit has been reached, no later observation may be forwarded.
Give the Pipeline a Private Destination
A complete pipeline needs a writable endpoint. The failure experiment should not retain NDJSON output or send it to the HTTP response, so backend/src/streams/dev-null-writable.ts supplies a byte-oriented discard destination.
await pipeline(...) is the terminal boundary. The completed result is returned only after all stages, including DevNullWritable, complete successfully.
The stage order preserves the representation boundaries: NDJSON source, parsed objects, object-mode fault transform, NDJSON stringifier, and byte-oriented destination.
Only InjectedStreamError becomes the stable failed result. Its exact forwarded-record count is preserved.
Source, parser, destination, and unrelated transform errors are rethrown unchanged rather than being mislabeled as injected failures.
The second call completes because the source ends after four forwarded observations. No fifth observation arrives to trigger the transform’s failure branch.
Validate Dependent Request Controls Before Starting Work
Send JSON Only After the Experiment Settles
backend/src/routes/failure.ts owns the request contract and sends the final JSON result.
The fallback path allows new URL() to parse an absent request URL while keeping this route’s path contract explicit.
records is resolved before failAfter, so the second validation can use max: records.
The absent-parameter default is clamped to records, so ?records=3 uses failAfter: 3 and completes normally; explicitly supplying failAfter=4 still returns 400.
records=3&failAfter=4 throws during validation and does not start the pipeline.
sendJson() receives the settled result of await runFailureExperiment(...), so the route commits exactly one JSON response only after internal work has completed or reached the expected failure boundary.
The application boundary converts pre-response HttpError values into the established structured error response. This route therefore validates before calling runFailureExperiment(), while no success response has been written.
Register the Endpoint as Known and Dispatchable
The router needs two additions:
/api/lab/failure must be known so non-GET requests receive 405 METHOD_NOT_ALLOWED.
A valid GET request must dispatch to runFailureProbe().
The first source-order region of backend/src/http/router.ts imports the route handler, defines the path, and adds it to knownPaths.
import type { IncomingMessage, ServerResponse } from "node:http";import { API_PATHS } from "@stream-lab/shared";import type { DatabaseClient } from "../db/client.js";import { sendError, sendJson } from "./response.js";import { health } from "../routes/health.js";import { labConfig } from "../routes/lab-config.js";import { sendSample } from "../routes/sample.js";import { datasets } from "../routes/datasets.js";import { inspectSample } from "../routes/inspect.js";import { sendGenerated } from "../routes/generated.js";import { sendParseSummary } from "../routes/parse-summary.js";import { sendTransformed } from "../routes/transform.js";import { runBackpressure } from "../routes/backpressure.js";import { sendLive } from "../routes/live.js";import { runFailureProbe } from "../routes/failure.js";import { exportGenerated } from "../routes/export.js";export interface RouterDependencies { database: DatabaseClient; samplePath: string; startedAt: number;}const INSPECT_PATH = "/api/lab/inspect";const GENERATED_PATH = "/api/lab/generated";const SUMMARY_PATH = "/api/lab/summary";const TRANSFORM_PATH = "/api/lab/transform";const BACKPRESSURE_PATH = "/api/lab/backpressure";const LIVE_PATH = "/api/lab/live";const FAILURE_PATH = "/api/lab/failure";const knownPaths = new Set<string>([ ...Object.values(API_PATHS), INSPECT_PATH, GENERATED_PATH, SUMMARY_PATH, TRANSFORM_PATH, BACKPRESSURE_PATH, LIVE_PATH, FAILURE_PATH,]);
runFailureProbe joins the router’s existing backend handlers.
FAILURE_PATH follows the local path-constant convention used by the other lab routes.
Adding the path to knownPaths affects method handling, not GET dispatch by itself.
Existing route constants and handlers remain unchanged.
The method gate in backend/src/http/router.ts runs before any GET branch.
export const route = async ( request: IncomingMessage, response: ServerResponse, deps: RouterDependencies,): Promise<void> => { const pathname = new URL(request.url ?? "/", "http://localhost").pathname; if (request.method !== "GET") { if (knownPaths.has(pathname)) { sendError( response, 405, "METHOD_NOT_ALLOWED", "Only GET is supported by this endpoint.", { allow: "GET" }, ); } else { sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", ); } return; }
The router extracts only the pathname. Query validation remains the responsibility of runFailureProbe().
A non-GET request to a known path, including /api/lab/failure, receives the established 405 response and Allow: GET header.
Unknown paths still receive 404.
The early return prevents non-GET requests from reaching any GET route handler.
The final source-order region preserves existing GET branches and adds the failure-route dispatch before the unchanged fallback.
if (pathname === API_PATHS.health) return sendJson(response, 200, await health(deps.database, deps.startedAt)); if (pathname === API_PATHS.labConfig) return sendJson(response, 200, labConfig()); if (pathname === API_PATHS.sample) return sendSample(response, deps.samplePath); if (pathname === API_PATHS.datasets) return datasets(response, deps.database); if (pathname === INSPECT_PATH) return inspectSample(response, deps.samplePath); if (pathname === GENERATED_PATH) return sendGenerated(response, request.url); if (pathname === SUMMARY_PATH) return sendParseSummary(response, deps.samplePath); if (pathname === TRANSFORM_PATH) return sendTransformed(response, deps.samplePath, request.url); if (pathname === BACKPRESSURE_PATH) return runBackpressure(response, request.url); if (pathname === LIVE_PATH) return sendLive(response, request.url); if (pathname === FAILURE_PATH) return runFailureProbe(response, request.url); if (pathname === EXPORT_PATH) return exportGenerated(response, request.url); sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", );};
The failure branch passes the response and original request URL to the route handler, matching the project’s query-aware route pattern.
GET /api/lab/failure now reaches validation and the awaited internal experiment.
Existing route branches are preserved rather than replaced.
The final 404 remains necessary for unmatched GET paths.
Observe the Terminal Diagnostic
An injected failure reaches the configured boundary and returns one JSON result:
A request such as ?records=3&failAfter=4 is invalid because the resolved records value becomes the maximum legal value for failAfter. The route returns the established 400 response before pipeline work begins.
The provided Run failure experiment helper sends this backend request with records and failAfter and displays the terminal JSON result.
Recap
This endpoint works because each layer has one focused responsibility:
The generated source emits deterministic NDJSON-compatible output.
The parser converts serialized chunks into validated object-mode observations.
FaultTransform forwards exactly failAfter observations, then fails before forwarding another one.
InjectedStreamError identifies the one expected pipeline failure that becomes a diagnostic result.
DevNullWritable gives the internal byte-oriented pipeline a terminal destination without retaining output.
runFailureExperiment() awaits the real pipeline, returns a typed terminal result for the injected error, and rethrows unrelated failures.
runFailureProbe() validates dependent controls before work, awaits the result, and sends one JSON response.
The router registers /api/lab/failure both for 405 handling and valid GET dispatch.
Keep the central model in view: a terminal JSON diagnostic is safe because the internal pipeline settles before the HTTP response begins.
The failure route accepts two integer query controls:
records: default 20, inclusive range 1 through 5_000
failAfter: default Math.min(5, records), inclusive range 0 through the resolved records value
backend/src/http/query.ts provides the established integer parser used by the route.
TypeScript
export interface IntegerParamOptions { defaultValue: number; min: number; max: number;}export const integerParam = ( url: URL, name: string, options: IntegerParamOptions,): number => { const raw = url.searchParams.get(name); if (raw === null) return options.defaultValue; if (!/^-?\d+$/.test(raw)) { throw new HttpError( 400, "INVALID_QUERY_PARAMETER", `${name} must be an integer between ${options.min} and ${options.max}.`, ); } const value = Number(raw); if ( !Number.isSafeInteger(value) || value < options.min || value > options.max ) { throw new HttpError( 400, "INVALID_QUERY_PARAMETER", `${name} must be an integer between ${options.min} and ${options.max}.`, ); } return value;};
An absent parameter uses defaultValue; supplied invalid input is never silently replaced.
The regular expression requires integer text, and Number.isSafeInteger() rejects values JavaScript cannot safely represent as integers.
The bounds are inclusive, which makes failAfter=0 and failAfter=records valid values.
Invalid values throw HttpError before the route starts the internal experiment.
The failAfter maximum depends on the validated records value. That makes validation order meaningful: resolve records first, then use it as the maximum for failAfter.