Hello, pipeline conductor! 🌊 In the previous lesson, you built the unfiltered flow where file bytes became validated Observation objects, a Transform added Fahrenheit values, and GET /api/lab/transform streamed all three fixture records as NDJSON.
Now the established byte-to-object-to-byte boundaries remain intact while the endpoint gains configurable filtering. The completed route validates optional controls before it commits a successful response, filters trusted objects before Fahrenheit mapping, and streams only the surviving records.
A request can combine both controls:
text
GET /api/lab/transform?site=harbor&minTemperatureC=18
A valid filter may also match no observations. For example, site=ridge&minTemperatureC=30 returns 200 OK with an empty body. That is successful stream output, not an error.
The fixed browser helper can invoke this backend request and display matching NDJSON, an empty successful response, or a backend error result.
Trace the Configurable Stream Boundary
The completed flow separates HTTP decisions from record decisions:
Query controls must be validated before 200 NDJSON headers are written. A validation error still needs the established JSON error response.
Filtering belongs after parsing because it needs trusted site and temperatureC fields. It belongs before Fahrenheit mapping so rejected records are never transformed.
The established NdjsonParser contract is unchanged: bytes enter and validated Observation objects leave. The new filter starts at that trusted object boundary and emits zero or one unchanged observation per input. Placing it before the mapper avoids transforming rejected records.
Validate Optional Query Controls
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Filter Trusted Objects in Object Mode
The filter receives validated Observation objects from the parser and produces either zero or one unchanged object for each input. It does not parse URLs, map temperatures, serialize NDJSON, or collect arrays.
backend/src/streams/observation-filter.ts defines the optional criteria and configures object mode on both sides of the Transform.
import { Transform, type TransformCallback } from "node:stream";import type { Observation } from "../domain/observation.js";export interface ObservationFilterOptions { site?: string; minTemperatureC?: number;}export class ObservationFilter extends Transform { constructor(private readonly options: ObservationFilterOptions) { super({ writableObjectMode: true, readableObjectMode: true }); }
Optional properties allow filtering by site, by minimum temperature, by both, or by neither.
writableObjectMode: true accepts Observation values from the parser.
readableObjectMode: true emits matching Observation values to the Fahrenheit mapper.
Each filter instance holds its own configured criteria, so a pipeline needs a fresh instance.
The remainder of backend/src/streams/observation-filter.ts evaluates supplied criteria one record at a time.
A defined site criterion drops observations whose site differs.
A defined minimum temperature drops observations below the threshold; equality passes because the comparison is <.
callback() completes processing without output, which removes that observation from the stream.
callback(null, observation) forwards a matching original object unchanged, and the factory creates a fresh configured Transform.
When both options are supplied, both must pass. For site=harbor&minTemperatureC=18, only the harbor fixture record at 18.4 continues. With no options, neither rejection condition applies, so all trusted observations pass through.
Map Only Surviving Observations
backend/src/streams/temperature-transform.ts receives observations that have already survived filtering. Its responsibility is mapping, not filtering.
The mapper is object mode on both sides because it transforms one JavaScript object into another.
...value preserves id, temperatureC, and site.
temperatureF is added only for observations that reached this stage.
18.4 Celsius becomes 65.1 Fahrenheit after rounding to one decimal place.
Filtering before mapping ensures that rejected records never receive a Fahrenheit value.
Centralize Pipeline Composition
The route owns HTTP validation, status, and response headers; the reusable pipeline owns source-to-destination stream composition. Extracting composition lets the same stages write to any byte-oriented Writable, not just an HTTP response. The caller supplies the destination, while the pipeline creates fresh source and Transform instances.
backend/src/streams/observation-pipeline.ts imports each stage and defines the composition options.
import type { Writable } from "node:stream";import { pipeline } from "node:stream/promises";import { createSampleReadStream } from "./file-source.js";import { createNdjsonParser } from "./ndjson-parser.js";import { createNdjsonStringifier } from "./ndjson-stringifier.js";import { createObservationFilter, type ObservationFilterOptions,} from "./observation-filter.js";import { createTemperatureTransform } from "./temperature-transform.js";export interface ObservationPipelineOptions extends ObservationFilterOptions { samplePath: string; destination: Writable;}
ObservationPipelineOptions contains stream concerns only: source path, filter criteria, and destination. It contains no request URL, HTTP status, or response headers.
A ServerResponse satisfies Writable, so the route can delegate without coupling this module to HTTP.
The rest of backend/src/streams/observation-pipeline.ts creates every stage for one request and awaits the complete pipeline.
writableObjectMode: true accepts mapped objects from the Fahrenheit Transform.
The readable side remains byte-oriented because readableObjectMode is not enabled.
Each output value is JSON followed by exactly one newline, which is the NDJSON record boundary.
If the filter emits no objects, the stringifier receives no values and produces an empty response body.
Commit HTTP Success Only After Validation
backend/src/routes/transform.ts owns request validation, the allowed-site rule, successful headers, and delegation to the reusable pipeline.
import type { ServerResponse } from "node:http";import { HttpError } from "../http/errors.js";import { optionalNumberParam, optionalStringParam,} from "../http/query.js";import { streamTransformedObservations } from "../streams/observation-pipeline.js";const SITES = new Set(["harbor", "orchard", "ridge"]);
Query helpers validate generic syntax and normalization.
SITES adds route-specific business validation: a valid string is not automatically a supported site.
The route imports the reusable pipeline instead of rebuilding stream stages locally.
The main body of backend/src/routes/transform.ts validates every supplied control before writing a successful response.
export const sendTransformed = async ( response: ServerResponse, samplePath: string, requestUrl: string | undefined,): Promise<void> => { const url = new URL(requestUrl ?? "/api/lab/transform", "http://localhost"); const site = optionalStringParam(url, "site"); const minTemperatureC = optionalNumberParam(url, "minTemperatureC", { min: -100, max: 100, }); if (site !== undefined && !SITES.has(site)) { throw new HttpError( 400, "INVALID_QUERY", "site must be harbor, orchard, or ridge.", ); } response.writeHead(200, { "content-type": "application/x-ndjson; charset=utf-8", "x-stream-lab-mode": "filtered-transform-pipeline", });
new URL(...) provides a stable base URL while retaining the incoming query string.
minTemperatureC accepts finite values from -100 through 100, including both endpoints.
A supplied unsupported site throws HttpError before the response has committed 200 NDJSON headers.
Once headers are written, the response representation is NDJSON and cannot safely switch to a JSON validation error.
The final part of backend/src/routes/transform.ts delegates byte streaming to the reusable pipeline.
samplePath lets the pipeline create a fresh file source.
destination: response connects final NDJSON bytes to the HTTP response.
Only defined criteria are forwarded to the filter.
The route does not call response.end() manually because pipeline(...) completes its destination.
A request with valid nonmatching criteria still reaches this point, writes 200 NDJSON headers, and completes with an empty body.
Preserve Routing While Forwarding the Full URL
The router dispatches by pathname, which intentionally excludes the query string. The transform route still needs the original URL to read optional controls.
backend/src/http/router.ts imports the transform route and defines its path with the other known route paths.
import type { IncomingMessage, ServerResponse } from "node:http";import { API_PATHS } from "@stream-lab/shared";import type { DatabaseClient } from "../db/client.js";import { sendError, sendJson } from "./response.js";import { health } from "../routes/health.js";import { labConfig } from "../routes/lab-config.js";import { sendSample } from "../routes/sample.js";import { datasets } from "../routes/datasets.js";import { inspectSample } from "../routes/inspect.js";import { sendGenerated } from "../routes/generated.js";import { sendParseSummary } from "../routes/parse-summary.js";import { sendTransformed } from "../routes/transform.js";export interface RouterDependencies { database: DatabaseClient; samplePath: string; startedAt: number;}const INSPECT_PATH = "/api/lab/inspect";const GENERATED_PATH = "/api/lab/generated";const SUMMARY_PATH = "/api/lab/summary";const TRANSFORM_PATH = "/api/lab/transform";
sendTransformed receives the full original URL and the sample path.
TRANSFORM_PATH keeps known-route recognition and GET dispatch consistent.
The router remains a dispatcher; it does not parse transform controls itself.
The next portion of backend/src/http/router.ts preserves GET-only behavior for all known paths.
const knownPaths = new Set<string>([ ...Object.values(API_PATHS), INSPECT_PATH, GENERATED_PATH, SUMMARY_PATH, TRANSFORM_PATH,]);export const route = async ( request: IncomingMessage, response: ServerResponse, deps: RouterDependencies,): Promise<void> => { const pathname = new URL(request.url ?? "/", "http://localhost").pathname; if (request.method !== "GET") { if (knownPaths.has(pathname)) { sendError( response, 405, "METHOD_NOT_ALLOWED", "Only GET is supported by this endpoint.", { allow: "GET" }, ); } else { sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", ); } return; }
pathname makes /api/lab/transform and /api/lab/transform?site=harbor select the same route.
Including TRANSFORM_PATH in knownPaths preserves 405 METHOD_NOT_ALLOWED for non-GET requests to this known endpoint.
The early return keeps unsupported methods out of GET dispatch.
The final portion of backend/src/http/router.ts forwards request.url to handlers that need query parsing.
if (pathname === API_PATHS.health) return sendJson(response, 200, await health(deps.database, deps.startedAt)); if (pathname === API_PATHS.labConfig) return sendJson(response, 200, labConfig()); if (pathname === API_PATHS.sample) return sendSample(response, deps.samplePath); if (pathname === API_PATHS.datasets) return datasets(response, deps.database); if (pathname === INSPECT_PATH) return inspectSample(response, deps.samplePath); if (pathname === GENERATED_PATH) return sendGenerated(response, request.url); if (pathname === SUMMARY_PATH) return sendParseSummary(response, deps.samplePath); if (pathname === TRANSFORM_PATH) return sendTransformed(response, deps.samplePath, request.url); sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", );};
Generated rows already receive request.url for their records control.
Transform requests now follow the same pattern without moving transform-specific parsing into the router.
The first omits the control and should produce undefined. The second supplies a blank value and must fail validation.
backend/src/http/query.ts keeps integerParam() unchanged for generated-record counts: integer syntax, safe-integer checks, defaults, and bounds still apply. The new optional helpers return undefined for omitted controls, normalized values for valid supplied controls, and 400 INVALID_QUERY for invalid supplied controls.
The next part of backend/src/http/query.ts parses an optional finite number within caller-provided inclusive bounds.
export const optionalNumberParam = ( url: URL, key: string, options: { min: number; max: number },): number | undefined => { const raw = url.searchParams.get(key); if (raw === null) return undefined; const normalized = raw.trim(); const parsed = Number(normalized); if ( normalized.length === 0 || !Number.isFinite(parsed) || parsed < options.min || parsed > options.max ) { throw new HttpError( 400, "INVALID_QUERY", `${key} must be a number between ${options.min} and ${options.max}.`, ); } return parsed;};
raw === null is the only absent case, so it is the only case that returns undefined.
Trimming permits meaningful supplied values such as " 18 ".
normalized.length === 0 is essential because Number("") and Number(" ") would otherwise become 0.
Number.isFinite(...) rejects NaN, Infinity, and -Infinity; the range checks then enforce the route’s allowed bounds.
The final helper in backend/src/http/query.ts applies the same absent-versus-invalid distinction to text.
export const optionalStringParam = ( url: URL, key: string, maxLength = 64,): string | undefined => { const raw = url.searchParams.get(key); if (raw === null) return undefined; const normalized = raw.trim(); if (normalized.length === 0 || normalized.length > maxLength) { throw new HttpError( 400, "INVALID_QUERY", `${key} must contain between 1 and ${maxLength} characters.`, ); } return normalized;};
Absent text returns undefined, while blank or whitespace-only supplied text is invalid.
The helper returns trimmed text, so callers do not need to repeat normalization.
The default maximum length prevents unexpectedly large controls from being accepted.
Both optional helpers throw the same typed HttpError(400, "INVALID_QUERY", ...) contract.
backend/src/routes/generated.ts remains a direct caller of the integer-only helper.
import type { ServerResponse } from "node:http";import { HttpError } from "../http/errors.js";import { optionalNumberParam, optionalStringParam,} from "../http/query.js";import { streamTransformedObservations } from "../streams/observation-pipeline.js";const SITES = new Set(["harbor", "orchard", "ridge"]);
TypeScript
export const sendTransformed = async ( response: ServerResponse, samplePath: string, requestUrl: string | undefined,): Promise<void> => { const url = new URL(requestUrl ?? "/api/lab/transform", "http://localhost"); const site = optionalStringParam(url, "site"); const minTemperatureC = optionalNumberParam(url, "minTemperatureC", { min: -100, max: 100, }); if (site !== undefined && !SITES.has(site)) { throw new HttpError( 400, "INVALID_QUERY", "site must be harbor, orchard, or ridge.", ); } response.writeHead(200, { "content-type": "application/x-ndjson; charset=utf-8", "x-stream-lab-mode": "filtered-transform-pipeline", });
These regression checks preserve the sample bytes and generated endpoint. The new contracts are matching output, valid empty output, and validation before streaming.
The matching-filter test verifies the exact transformed harbor observation.