Hello, stream builder! 🌊 Stream Lab already streams the sample NDJSON fixture from a fresh file Readable, but delivering bytes is not the same as delivering application records. This lesson completes the byte-to-object boundary: it preserves partial UTF-8 text and partial NDJSON lines, validates complete values, and emits trusted Observation objects for incremental consumers.
The fixture contains three newline-delimited records and 177 bytes. Those facts do not mean a file stream or browser response will produce three chunks. Chunk boundaries are transport observations; newline boundaries define NDJSON records.
NDJSON is common in log processing, event streams, bulk ingestion, database exports, and large API responses because each value can travel independently. Incremental parsing lets a consumer validate and process one record at a time instead of loading the complete input into memory.
Start with byte-oriented streams
backend/src/streams/file-source.ts creates a new file-backed source each time the application needs to read the fixture.
TypeScript
import { createReadStream, type ReadStream } from "node:fs";export const createSampleReadStream = (samplePath: string): ReadStream => createReadStream(samplePath);
createReadStream(samplePath) returns a Node.js ReadStream, which produces file data as byte-oriented chunks.
A stream is consumed as it is read, so sample delivery, inspection, and parsing each need their own fresh source.
This factory knows nothing about JSON, newlines, or observations. It only opens the file as a Readable.
backend/src/routes/sample.ts pipes a fresh source directly to the HTTP response.
TypeScript
import type { ServerResponse } from "node:http";import { pipeline } from "node:stream/promises";import { createSampleReadStream } from "../streams/file-source.js";export const sendSample = async ( response: ServerResponse, samplePath: string,): Promise<void> => { response.writeHead(200, { "content-type": "application/x-ndjson; charset=utf-8", "x-stream-lab-mode": "file-stream", }); await pipeline(createSampleReadStream(samplePath), response);};
The application/x-ndjson; charset=utf-8 media type describes the body format, but it does not force chunks to align with lines.
pipeline(...) connects the file Readable to the writable HTTP response and waits for completion or failure.
This route transports NDJSON bytes. It does not call JSON.parse and does not emit objects.
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Inspect chunks without confusing them with records
backend/src/routes/inspect.ts
import type { ServerResponse } from "node:http";import { sendJson } from "../http/response.js";import { createSampleReadStream } from "../streams/file-source.js";import { inspectReadable } from "../streams/inspect-readable.js";export const inspectSample = async ( response: ServerResponse, samplePath: string,): Promise<void> => { const inspection = await inspectReadable(createSampleReadStream(samplePath)); sendJson(response, 200, { source: "sample-file", ...inspection, });};
createSampleReadStream(samplePath) creates an independent source instead of reusing the stream sent by the sample route.
inspectReadable(...) measures its own read of the fixture rather than observing the HTTP response.
The response includes the fixed source label and the measured chunk and byte totals.
backend/src/streams/inspect-readable.ts shows what this inspection means.
export interface ReadableInspection { chunks: number; bytes: number;}const chunkByteLength = (chunk: unknown): number => { if (typeof chunk === "string") return Buffer.byteLength(chunk); if (chunk instanceof Uint8Array) return chunk.byteLength; throw new TypeError("Expected stream chunks to be strings or byte arrays.");};export const inspectReadable = async ( source: AsyncIterable<unknown>,): Promise<ReadableInspection> => { let chunks = 0; let bytes = 0; for await (const chunk of source) { chunks += 1; bytes += chunkByteLength(chunk); } return { chunks, bytes };};
The input is AsyncIterable<unknown>, so chunkByteLength verifies that each chunk is either text or a byte array before measuring it.
for await...of consumes chunks as they arrive. It has no knowledge of JSON syntax or newline delimiters.
The fixture total is 177 bytes, while chunks describes one particular read of that fixture.
Browser-observed chunks, server file-stream chunks, and NDJSON record counts can all differ without changing the response body.
JSON.parse confirms JSON syntax, not application validity. Parsed input begins as unknown and must pass a runtime contract before downstream code can trust fields such as temperatureC.
backend/src/domain/observation.ts defines that contract.
Observation describes the values later object-mode consumers may rely on: a non-empty id, a finite numeric temperatureC, and a non-empty site.
The first guard rejects null and primitive values before fields are inspected through Record<string, unknown>.
Number.isFinite(...) rejects NaN, Infinity, and -Infinity, even though all have the JavaScript type "number".
The return type value is Observation is a type predicate. When the function returns true, TypeScript narrows the value to Observation.
A line can therefore be valid JSON but still fail the observation contract—for example, if it contains "temperatureC": "warm".
Build the byte-to-object Transform
A Transform has writable and readable sides. This parser accepts Buffer or string chunks on its writable side, then emits validated Observation values on its readable side.
backend/src/streams/ndjson-parser.ts begins by establishing state that must survive across incoming chunks.
import { StringDecoder } from "node:string_decoder";import { Transform, type TransformCallback } from "node:stream";import { isObservation, type Observation,} from "../domain/observation.js";export class NdjsonParser extends Transform { private readonly decoder = new StringDecoder("utf8"); private pending = ""; private lineNumber = 0; constructor() { super({ readableObjectMode: true }); }
StringDecoder("utf8") preserves incomplete multibyte UTF-8 sequences between Buffer chunks.
pending retains decoded text until a newline completes a record.
lineNumber counts physical NDJSON lines, including blank lines that are ignored as records.
readableObjectMode: true applies only to the parser output. Writable object mode remains disabled because input remains byte-oriented.
decodeStrings: false deliberately preserves string writes for the explicit string branch; without it, Node normally converts string writes to Buffer values before _transform() runs.
Decode chunks incrementally
backend/src/streams/ndjson-parser.ts uses _transform() to append incoming text and emit only complete newline-delimited records.
decoder.end() signals that no more bytes will arrive and returns any final decoded text held by StringDecoder.
The final pending text is trimmed so whitespace-only endings do not create records.
Clearing pending prevents stale state from remaining after EOF.
A valid final record is emitted even if input ends after } instead of }\n.
Frame, parse, and validate complete lines
backend/src/streams/ndjson-parser.ts finds newline boundaries, preserves incomplete suffixes, and distinguishes JSON failures from domain failures.
private emitCompleteLines(): void { let newlineIndex = this.pending.indexOf("\n"); while (newlineIndex >= 0) { this.lineNumber += 1; const line = this.pending.slice(0, newlineIndex).trim(); this.pending = this.pending.slice(newlineIndex + 1); if (line.length > 0) { this.push(this.parseLine(line)); } newlineIndex = this.pending.indexOf("\n"); } } private parseLine(line: string): Observation { let value: unknown; try { value = JSON.parse(line) as unknown; } catch { throw new Error(`Invalid JSON on NDJSON line ${this.lineNumber}.`); } if (!isObservation(value)) { throw new Error( `Invalid observation on NDJSON line ${this.lineNumber}.`, ); } return value; }}
emitCompleteLines() repeatedly removes text through the next "\n", leaving an incomplete final suffix in pending.
Blank or whitespace-only lines are ignored because only non-empty text reaches parseLine(...).
The counter advances for every newline-delimited physical line, including ignored blank lines, and for a non-empty unterminated final line. parseLine(...) uses that counter without incrementing it.
JSON.parse(...) failures become Invalid JSON on NDJSON line N., while values rejected by isObservation(...) become Invalid observation on NDJSON line N..
this.push(...) sends only validated Observation objects to the readable side.
The parser factory creates fresh state for each independent stream.
export const createNdjsonParser = (): NdjsonParser => new NdjsonParser();
Each parser receives its own decoder, pending text, and line-number counter.
A parser instance belongs to one input stream and should not be shared between requests.
Prove arbitrary chunk handling
The collect(chunks) helper in backend/test/ndjson-parser.test.ts pipes controlled input chunks into a fresh parser and collects its object-mode output for assertions. Its implementation remains in the project files; these tests deliberately choose inconvenient input boundaries.
The split-line test proves that newline framing, not chunk boundaries, determines records.
test("parses records even when lines cross arbitrary chunks", async () => { const records = await collect([ '{"id":"a","temperatureC":18.4,', '"site":"harbor"}\n{"id":"b","tempera', 'tureC":19.1,"site":"orchard"}\n', ]); assert.deepEqual(records, [ { id: "a", temperatureC: 18.4, site: "harbor" }, { id: "b", temperatureC: 19.1, site: "orchard" }, ]);});
The first record crosses two chunks, so parsing chunks independently would fail.
The second chunk completes one record and begins the next.
The two emitted objects correspond to newline delimiters, not to the three input chunks.
The EOF test verifies that _flush() preserves an unterminated final record.
test("flushes a final record without a newline", async () => { const records = await collect([ '{"id":"final","temperatureC":17.8,"site":"ridge"}', ]); assert.deepEqual(records, [ { id: "final", temperatureC: 17.8, site: "ridge" }, ]);});
No newline is available while the only chunk is being transformed.
EOF triggers _flush(), which processes the remaining pending text.
The valid record is emitted rather than silently discarded.
The final tests keep syntax and domain failures separate.
test("rejects malformed JSON with the failing line number", async () => { await assert.rejects( collect([ '{"id":"a","temperatureC":18.4,"site":"harbor"}\n', "not-json\n", ]), /Invalid JSON on NDJSON line 2/, );});test("reports physical line numbers when blank lines are ignored", async () => { await assert.rejects( collect(["\n", " \n", "not-json\n"]), /Invalid JSON on NDJSON line 3/, );});test("rejects invalid observations with the failing line number", async () => { await assert.rejects( collect(['{"id":"a","temperatureC":"warm","site":"harbor"}\n']), /Invalid observation on NDJSON line 1/, );});
The malformed JSON test reaches a valid first record before failing on physical line 2; the blank-line test fails on physical line 3.
"not-json" cannot be parsed, so the parser reports an invalid JSON error.
"temperatureC":"warm" is valid JSON but fails the Observation contract.
Failures identify physical NDJSON lines rather than non-empty record counts or source chunk positions.
Consume parsed records incrementally
backend/src/routes/parse-summary.ts creates a fresh byte source and parser for each summary request.
import type { ServerResponse } from "node:http";import type { Observation } from "../domain/observation.js";import { sendJson } from "../http/response.js";import { createSampleReadStream } from "../streams/file-source.js";import { createNdjsonParser } from "../streams/ndjson-parser.js";export const sendParseSummary = async ( response: ServerResponse, samplePath: string,): Promise<void> => { const source = createSampleReadStream(samplePath); const parser = createNdjsonParser(); source.once("error", (error) => parser.destroy(error)); parser.once("error", () => source.destroy()); source.pipe(parser);
The source and parser are created per request, giving each request independent file-read, decoder, pending-text, and line-number state.
A source failure destroys the parser with the original error.
A parser failure destroys the source so the file is not read unnecessarily after downstream parsing has failed.
source.pipe(parser) connects byte-oriented file input to the parser’s writable side.
The same route aggregates emitted records one at a time.
let records = 0; let minTemperatureC = Number.POSITIVE_INFINITY; let maxTemperatureC = Number.NEGATIVE_INFINITY; const sites = new Set<string>(); for await (const value of parser) { const observation = value as Observation; records += 1; minTemperatureC = Math.min(minTemperatureC, observation.temperatureC); maxTemperatureC = Math.max(maxTemperatureC, observation.temperatureC); sites.add(observation.site); } sendJson(response, 200, { records, minTemperatureC: records === 0 ? null : minTemperatureC, maxTemperatureC: records === 0 ? null : maxTemperatureC, sites: [...sites].sort(), });};
for await...of processes one parser output record at a time without collecting all observations into an array.
Positive and negative infinity provide useful initial extrema, but the response converts them to null when no records were emitted.
Set<string> keeps site names distinct as data arrives.
[...sites].sort() produces a deterministic alphabetically sorted response array.
Accessing temperatureC and site is safe at this boundary because the parser only pushes values accepted by isObservation.
backend/src/http/router.ts adds the summary route to both path recognition and GET dispatch.
import type { IncomingMessage, ServerResponse } from "node:http";import { API_PATHS } from "@stream-lab/shared";import type { DatabaseClient } from "../db/client.js";import { datasets } from "../routes/datasets.js";import { sendGenerated } from "../routes/generated.js";import { health } from "../routes/health.js";import { inspectSample } from "../routes/inspect.js";import { labConfig } from "../routes/lab-config.js";import { sendParseSummary } from "../routes/parse-summary.js";import { sendSample } from "../routes/sample.js";import { sendError, sendJson } from "./response.js";export interface RouterDependencies { database: DatabaseClient; samplePath: string; startedAt: number;}const INSPECT_PATH = "/api/lab/inspect";const GENERATED_PATH = "/api/lab/generated";const SUMMARY_PATH = "/api/lab/summary";const knownPaths = new Set<string>([ ...Object.values(API_PATHS), INSPECT_PATH, GENERATED_PATH, SUMMARY_PATH,]);
SUMMARY_PATH has one named value used for both method recognition and GET dispatch.
Including the summary route in knownPaths is required for an unsupported method to return 405.
Existing shared API paths and lab-specific routes remain recognized.
The router checks method support before dispatching GET routes.
export const route = async ( request: IncomingMessage, response: ServerResponse, deps: RouterDependencies,): Promise<void> => { const pathname = new URL(request.url ?? "/", "http://localhost").pathname; if (request.method !== "GET") { if (knownPaths.has(pathname)) { sendError( response, 405, "METHOD_NOT_ALLOWED", "Only GET is supported by this endpoint.", { allow: "GET" }, ); } else { sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", ); } return; }
pathname excludes query-string content while preserving the request path.
Known routes with unsupported methods return 405 METHOD_NOT_ALLOWED and an Allow: GET header.
Unknown paths return 404 NOT_FOUND, even for non-GET requests.
The early return prevents unsupported methods from reaching GET handlers.
The existing dispatches remain intact, followed by the summary branch.
if (pathname === API_PATHS.health) { return sendJson(response, 200, await health(deps.database, deps.startedAt)); } if (pathname === API_PATHS.labConfig) { return sendJson(response, 200, labConfig()); } if (pathname === API_PATHS.sample) { return sendSample(response, deps.samplePath); } if (pathname === API_PATHS.datasets) { return datasets(response, deps.database); } if (pathname === INSPECT_PATH) { return inspectSample(response, deps.samplePath); } if (pathname === GENERATED_PATH) { return sendGenerated(response, request.url); } if (pathname === SUMMARY_PATH) { return sendParseSummary(response, deps.samplePath); } sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", );};
The sample route remains byte-oriented, while the summary route is record-oriented.
sendParseSummary(...) receives the same fixture path dependency as the sample and inspection routes.
Existing generated, inspection, health, dataset, and unknown-route behavior remains unchanged.
An unmatched GET path still receives the existing JSON 404 response.
Verify the complete HTTP flow
backend/test/app.test.ts uses the application factory and an unavailable database double to exercise the real HTTP router and routes.
The full server harness remains available in the project files rather than being repeated in the lesson.
Focus here on the public assertions: fixture bytes remain unchanged, summary values are record-oriented, and known routes retain their explicit method behavior.
The integration snippets below show only those route-level contracts.
Existing behavior remains covered for the file stream, inspection route, and generated endpoint.
POST /api/lab/summary returns 405 because the path is known but only supports GET.
See the completed boundary
The fixed helper UI can display the summary through Run parse summary:
Records:3
Minimum:17.8 °C
Maximum:19.1 °C
Sites:harbor, orchard, ridge
That result is fundamentally different from browser and server chunk measurements. The browser probe observes HTTP response chunks; server inspection consumes chunks from a separate fresh file Readable, not from the HTTP transport. The summary consumes validated object-mode records.
Recap
The completed backend boundary has a clear contract:
File Readable streams produce byte-oriented chunks, not NDJSON records.
StringDecoder("utf8") preserves multibyte characters split across Buffer chunks.
Pending text retains incomplete lines until newlines complete them.
_flush() handles decoder output and a valid final record without a trailing newline.
isObservation distinguishes valid JSON syntax from valid application values.
NdjsonParser emits only validated Observation objects in readable object mode.
The summary route aggregates records incrementally without collecting all observations or confusing record counts with byte and chunk measurements.
Production scope
This fixture-focused parser does not cap pending record text or pause emission when readable-side push() signals backpressure. For untrusted or large inputs, use a production parser with a maximum record size and explicit readable-side backpressure handling; Units 2 and 3 reuse this intentionally simplified boundary.
test("flushes a final record without a newline", async () => { const records = await collect([ '{"id":"final","temperatureC":17.8,"site":"ridge"}', ]); assert.deepEqual(records, [ { id: "final", temperatureC: 17.8, site: "ridge" }, ]);});
TypeScript
test("rejects malformed JSON with the failing line number", async () => { await assert.rejects( collect([ '{"id":"a","temperatureC":18.4,"site":"harbor"}\n', "not-json\n", ]), /Invalid JSON on NDJSON line 2/, );});test("reports physical line numbers when blank lines are ignored", async () => { await assert.rejects( collect(["\n", " \n", "not-json\n"]), /Invalid JSON on NDJSON line 3/, );});test("rejects invalid observations with the failing line number", async () => { await assert.rejects( collect(['{"id":"a","temperatureC":"warm","site":"harbor"}\n']), /Invalid observation on NDJSON line 1/, );});
TypeScript
import type { ServerResponse } from "node:http";import type { Observation } from "../domain/observation.js";import { sendJson } from "../http/response.js";import { createSampleReadStream } from "../streams/file-source.js";import { createNdjsonParser } from "../streams/ndjson-parser.js";export const sendParseSummary = async ( response: ServerResponse, samplePath: string,): Promise<void> => { const source = createSampleReadStream(samplePath); const parser = createNdjsonParser(); source.once("error", (error) => parser.destroy(error)); parser.once("error", () => source.destroy()); source.pipe(parser);
TypeScript
let records = 0; let minTemperatureC = Number.POSITIVE_INFINITY; let maxTemperatureC = Number.NEGATIVE_INFINITY; const sites = new Set<string>(); for await (const value of parser) { const observation = value as Observation; records += 1; minTemperatureC = Math.min(minTemperatureC, observation.temperatureC); maxTemperatureC = Math.max(maxTemperatureC, observation.temperatureC); sites.add(observation.site); } sendJson(response, 200, { records, minTemperatureC: records === 0 ? null : minTemperatureC, maxTemperatureC: records === 0 ? null : maxTemperatureC, sites: [...sites].sort(), });};