Hello, pipeline builder! 🌡️ In the previous lesson, you built the parser boundary that accepts arbitrary file byte chunks, preserves split UTF-8 text and unfinished NDJSON lines, and emits only validated Observation objects. This lesson starts at that trusted object boundary: transform each observation, serialize it back to NDJSON bytes, and stream the result through GET /api/lab/transform.
The completed pipeline has five stages:
A file Readable emits byte-oriented chunks.
The NDJSON parser turns bytes into validated Observation objects.
A temperature Transform maps each observation to a FahrenheitObservation.
An NDJSON stringifier turns each mapped object into byte-oriented output.
The HTTP response receives those NDJSON bytes.
Keep these units separate:
Byte chunks are transport units from files and HTTP streams.
Newlines define NDJSON record boundaries.
Objects are the validated application records used by the mapper.
Browser-observed response chunks do not prove where NDJSON records begin or end.
The browser panel’s Run transform pipeline control can show the completed response: three newline-delimited JSON records that preserve Celsius data and add Fahrenheit values.
Start at the Byte-to-Object Boundary
backend/src/streams/file-source.ts creates the first stage in the flow: a fresh file-backed byte stream.
TypeScript
import { createReadStream, type ReadStream } from "node:fs";export const createSampleReadStream = (samplePath: string): ReadStream => createReadStream(samplePath);
createReadStream(samplePath) produces a Node.js ReadStream whose chunks are file bytes.
This source does not understand JSON, temperatures, or newline-delimited records.
A file stream is consumed as it is read, so the factory returns a new source for every pipeline invocation.
A single source chunk can contain part of a record, one record, several records, or part of a UTF-8 character.
NdjsonParser is the completed boundary from Unit 1: bytes or strings enter, complete NDJSON records are parsed and validated, and trusted Observation objects leave in readable object mode. It preserves split UTF-8 characters and unfinished lines and handles a final record at EOF.
The mapper begins only after this boundary. It does not need to decode bytes, locate newlines, parse JSON, or revalidate raw source data.
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Define the Transformed Record Contract
A source Observation describes a parsed fixture row. A transformed record retains every source field and adds temperatureF.
backend/src/domain/observation.ts defines both contracts while deliberately keeping source validation unchanged.
Observation remains the source contract: non-empty id, finite numeric temperatureC, and non-empty site.
FahrenheitObservation extends Observation, so it reuses all required source fields without repeating them.
temperatureF: number is required only after the mapping stage has derived it.
isObservation(...) stays an Observation guard because incoming NDJSON records are validated before Fahrenheit data exists.
A raw fixture row with temperatureC: 18.4 is valid before it has temperatureF. The mapper is responsible for producing the extended shape with temperatureF: 65.1.
Map One Object to One Object
The temperature mapper receives validated Observation objects and emits FahrenheitObservation objects. Because both writable and readable sides handle objects, both sides need object mode.
backend/src/streams/temperature-transform.ts implements that object-to-object Transform.
writableObjectMode: true allows _transform() to receive an Observation from the parser.
readableObjectMode: true allows the Transform to emit a FahrenheitObservation to the next stage.
toFahrenheit(...) applies temperatureC * 1.8 + 32, then rounds to one decimal place. For example, 18.4 becomes 65.1.
...value preserves id, temperatureC, and site, while temperatureF adds the derived field.
callback(null, transformed) emits exactly one output object for the current input object. No array of records is collected.
createTemperatureTransform() returns a fresh Transform instance for one pipeline invocation.
The mapper’s output remains in object mode. A normal ServerResponse is a byte-oriented destination, so one more boundary is required before sending the response.
Serialize Objects as NDJSON Bytes
NDJSON is a sequence of JSON values separated by newlines, not a JSON array. The stringifier accepts mapped objects, serializes each one, and returns to normal byte-mode output.
backend/src/streams/ndjson-stringifier.ts defines that object-to-byte Transform.
writableObjectMode: true lets the stringifier receive mapped objects.
The constructor does not enable readableObjectMode, so the readable side remains suitable for normal byte-oriented destinations.
JSON.stringify(...) must produce a string before the Transform appends framing; undefined is rejected because it is not valid JSON text.
The stringifier reports serialization failures through its callback instead of emitting non-NDJSON text.
The newline is NDJSON framing. Without it, adjacent serialized values would run together instead of forming separate records.
If JSON.stringify(...) throws, the error is passed to the Transform callback. Non-Error thrown values are normalized to Error("NDJSON serialization failed").
The factory creates an independent stringifier for each request.
The two middle stages have distinct responsibilities:
The temperature mapper changes the record’s shape while keeping object-mode output.
The stringifier changes the record’s representation from an object to NDJSON output.
Each input observation continues forward independently, one record at a time.
Compose the Transform Route
backend/src/routes/transform.ts connects the byte source, parser, mapper, stringifier, and HTTP response in data-flow order.
import type { ServerResponse } from "node:http";import { pipeline } from "node:stream/promises";import { createSampleReadStream } from "../streams/file-source.js";import { createNdjsonParser } from "../streams/ndjson-parser.js";import { createNdjsonStringifier } from "../streams/ndjson-stringifier.js";import { createTemperatureTransform } from "../streams/temperature-transform.js";export const sendTransformed = async ( response: ServerResponse, samplePath: string,): Promise<void> => { response.writeHead(200, { "content-type": "application/x-ndjson; charset=utf-8", "x-stream-lab-mode": "transform-pipeline", }); await pipeline( createSampleReadStream(samplePath), createNdjsonParser(), createTemperatureTransform(), createNdjsonStringifier(), response, );};
writeHead(200, ...) declares the NDJSON media type and the transform-specific mode header before body streaming begins.
Promise-based pipeline(...) resolves when the complete flow succeeds and rejects if a stage fails.
The pipeline order expresses the mode transitions: bytes → validated objects → mapped objects → NDJSON bytes → HTTP response.
Each factory creates fresh state for this request rather than sharing consumed streams or Transform state.
The route does not collect records into an array and does not manually call response.end(...); pipeline(...) manages completion of the destination.
createNdjsonStringifier() must appear immediately before response, because the response expects byte-oriented data rather than JavaScript objects.
Register the GET Endpoint
The router must recognize the transform path for method handling and dispatch matching GET requests to sendTransformed.
backend/src/http/router.ts imports the route handler and defines the transform path alongside the other lab-specific paths.
import type { IncomingMessage, ServerResponse } from "node:http";import { API_PATHS } from "@stream-lab/shared";import type { DatabaseClient } from "../db/client.js";import { sendError, sendJson } from "./response.js";import { health } from "../routes/health.js";import { labConfig } from "../routes/lab-config.js";import { sendSample } from "../routes/sample.js";import { datasets } from "../routes/datasets.js";import { inspectSample } from "../routes/inspect.js";import { sendGenerated } from "../routes/generated.js";import { sendParseSummary } from "../routes/parse-summary.js";import { sendTransformed } from "../routes/transform.js";export interface RouterDependencies { database: DatabaseClient; samplePath: string; startedAt: number;}const INSPECT_PATH = "/api/lab/inspect";const GENERATED_PATH = "/api/lab/generated";const SUMMARY_PATH = "/api/lab/summary";const TRANSFORM_PATH = "/api/lab/transform";
sendTransformed receives the Node ServerResponse and fixture path supplied by the router.
TRANSFORM_PATH gives route recognition and GET dispatch one shared path value.
samplePath is already part of RouterDependencies, so the transform route can create a fresh file source.
The transform path follows the existing lab-specific route pattern rather than changing shared API path constants.
The known-path set and method check preserve the router’s distinction between unsupported methods and unknown paths.
const knownPaths = new Set<string>([ ...Object.values(API_PATHS), INSPECT_PATH, GENERATED_PATH, SUMMARY_PATH, TRANSFORM_PATH,]);export const route = async ( request: IncomingMessage, response: ServerResponse, deps: RouterDependencies,): Promise<void> => { const pathname = new URL(request.url ?? "/", "http://localhost").pathname; if (request.method !== "GET") { if (knownPaths.has(pathname)) { sendError( response, 405, "METHOD_NOT_ALLOWED", "Only GET is supported by this endpoint.", { allow: "GET" }, ); } else { sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", ); } return; }
Adding TRANSFORM_PATH to knownPaths makes POST /api/lab/transform a known endpoint with an unsupported method.
That known non-GET request receives 405 METHOD_NOT_ALLOWED and Allow: GET.
A path not present in knownPaths still receives 404 NOT_FOUND.
pathname excludes query-string content before route matching.
The early return ensures non-GET requests never reach the GET dispatches.
The rest of backend/src/http/router.ts retains existing GET routes and adds the transform branch before the unknown-path fallback.
if (pathname === API_PATHS.health) return sendJson(response, 200, await health(deps.database, deps.startedAt)); if (pathname === API_PATHS.labConfig) return sendJson(response, 200, labConfig()); if (pathname === API_PATHS.sample) return sendSample(response, deps.samplePath); if (pathname === API_PATHS.datasets) return datasets(response, deps.database); if (pathname === INSPECT_PATH) return inspectSample(response, deps.samplePath); if (pathname === GENERATED_PATH) return sendGenerated(response, request.url); if (pathname === SUMMARY_PATH) return sendParseSummary(response, deps.samplePath); if (pathname === TRANSFORM_PATH) return sendTransformed(response, deps.samplePath); sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", );};
The transform branch passes deps.samplePath to the route, which then creates its own file stream.
The transform dispatch occurs before the fallback, so a matching GET request does not become a 404.
Returning sendTransformed(...) keeps the router’s asynchronous request flow active until the awaited stream pipeline completes.
Verify the Public HTTP Contract
The transform route is verified through the real HTTP application rather than by testing only isolated stream methods.
Use the existing withServer(...) harness in backend/test/app.test.ts to exercise the real router, stream stages, headers, and response body. The fixture path, database double, and server lifecycle remain unchanged.
The existing test keeps sample streaming and summary behavior available.
The transform endpoint does not interfere with unrelated lab route behavior.
The transform-specific test remains focused on its public NDJSON contract.
Observe the Completed Response
The fixture has three source observations. GET /api/lab/transform returns three NDJSON records, each preserving its Celsius fields and adding a Fahrenheit field.