Hello, export engineer! 📦 In the previous lesson, an internal pipeline settled at a private discard writable before the server sent one terminal JSON diagnostic. This lesson preserves that boundary, but gives the pipeline a real filesystem destination: bytes first enter a private .part file, and only a completed export is promoted to its final .ndjson filename.
The completed endpoint is:
text
GET /api/lab/export?records=3
It does not stream generated records to the client. It creates a real NDJSON file beneath /tmp/codesignal-stream-lab/exports, waits for the export to complete, then returns JSON metadata about the completed file.
A final filename must mean “this export completed,” not merely “a writable started receiving bytes.”
Guarantee scope: A same-directory rename provides atomic filename visibility
during normal process execution: ordinary readers do not observe streamed partial
bytes under the final filename. This lesson does not claim power-loss or crash
durability; that requires synchronizing the file and its containing directory
with additional fsync() work.
Start with a Private Terminal Boundary
The existing failure route demonstrates the response-timing rule that makes the export route safe. backend/src/routes/failure.ts validates request controls, awaits the internal experiment, and only then sends JSON.
records is validated first because it determines the largest legal failAfter value. For example, records=3&failAfter=4 is rejected before pipeline work begins.
await runFailureExperiment(records, failAfter) is the terminal boundary for this route. The response is not sent while the stream operation is still running.
sendJson() receives one settled result rather than stream bytes. That avoids trying to replace partially written NDJSON with an error body.
The experiment’s destination in backend/src/streams/dev-null-writable.ts is private. It accepts bytes without retaining or exposing them.
DevNullWritable is a byte-oriented Writable, so it can receive the serialized NDJSON from the final stringifier stage.
_chunk is intentionally unused. The destination accepts bytes but neither stores them nor writes them to the HTTP response.
Calling callback() without an error immediately acknowledges each write.
backend/src/streams/failure-experiment.ts connects the generated source to that private destination and translates the expected injected failure into a structured result.
The .part suffix is not cosmetic. It prevents incomplete bytes from appearing under the name that represents a completed export.
Write Through a Temporary File
backend/src/streams/atomic-file-export.ts is the reusable filesystem boundary. It accepts any Node Readable, writes it to a temporary path, promotes that file after stream completion, and returns metadata about the promoted file.
import { createWriteStream } from "node:fs";import { mkdir, rename, rm, stat } from "node:fs/promises";import { dirname } from "node:path";import type { Readable } from "node:stream";import { pipeline } from "node:stream/promises";export interface AtomicExportResult { finalPath: string; bytes: number;}
Readable is imported as a type, keeping the helper independent of routes and generated-source details.
createWriteStream provides the real byte-oriented file destination.
mkdir, rename, rm, and stat prepare the destination, promote output, remove temporary artifacts, and inspect completed metadata.
AtomicExportResult deliberately returns the internal final path and byte count. A route decides which parts are safe to expose publicly.
The function implements the temporary-write, promotion, metadata, and cleanup sequence.
partialPath is a sibling of finalPath. The temporary file stays in the same directory but cannot be mistaken for the completed export.
mkdir(dirname(finalPath), { recursive: true }) prepares a missing parent directory. rm(partialPath, { force: true }) removes a stale temporary artifact without failing when it does not exist.
createWriteStream(partialPath, { flags: "wx" }) creates an exclusive byte-oriented destination. The "wx" flags prevent silently opening an already-created temporary file.
await pipeline(...) is the stream lifecycle boundary. Only a resolved pipeline reaches rename(partialPath, finalPath).
stat(finalPath) runs after promotion, so bytes comes from the completed final file rather than the temporary path.
The catch applies to failures after preparation, including pipeline, rename, and stat failures. It removes the temporary path and rethrows the caught failure after successful cleanup.
Directory creation and stale temporary-file removal happen before the try block. This implementation does not claim to clean up after an unsuccessful initial mkdir() or initial stale-file removal.
The required order is:
source → .part writable → awaited pipeline → rename to final path → stat final path → return result
A successful stream alone is not enough to report success. Promotion and final metadata lookup must also complete.
Prove Promotion and Cleanup Directly
The helper is intentionally independent of HTTP. backend/test/atomic-file-export.test.ts can therefore test the filesystem contract using controlled readable sources and isolated temporary directories.
The imports and missing() helper establish the test tools.
import assert from "node:assert/strict";import { access, mkdtemp, readFile, rm } from "node:fs/promises";import { tmpdir } from "node:os";import { join } from "node:path";import { Readable } from "node:stream";import test from "node:test";import { writeStreamAtomically } from "../src/streams/atomic-file-export.js";const missing = async (path: string): Promise<boolean> => { try { await access(path); return false; } catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return true; throw error; }};
mkdtemp() creates a unique directory for each test, preventing collisions with repository files, previous test runs, or other tests.
Readable.from(...) creates controlled readable sources without involving routes or HTTP responses.
missing() returns false when access() succeeds and true when the path cannot be accessed. In these tests, failed access represents the expected missing-file condition.
The tests import the real helper instead of recreating its filesystem behavior.
The success test verifies exact completed bytes, final metadata, and removal of the temporary sibling.
integerParam() runs before UUID generation, source creation, or filesystem work. Invalid records values reach the established structured 400 behavior before export work starts.
The requested filename has the form <uuid>.ndjson; the helper derives its own <uuid>.ndjson.part sibling.
await writeStreamAtomically(...) means "completed" represents a real promoted file with final-file metadata.
The public response contains exactly status, fileName, bytes, and records.
basename(result.finalPath) prevents the internal directory path from appearing in the response. File contents and .part paths remain private.
The fixed Create export helper sends GET /api/lab/export and displays this terminal metadata. It does not expose the file path or provide file browsing or downloading.
The file is still real inside the practice environment. A returned filename can be observed with:
ls -lh /tmp/codesignal-stream-lab/exportscat /tmp/codesignal-stream-lab/exports/<returned-fileName>
These are direct filesystem observations, not HTTP endpoint features.
Register the Export Endpoint Without Changing Router Rules
backend/src/http/router.ts distinguishes known routes for method handling from GET route dispatch. The imports, constants, and knownPaths set include the export path.
import type { IncomingMessage, ServerResponse } from "node:http";import { API_PATHS } from "@stream-lab/shared";import type { DatabaseClient } from "../db/client.js";import { sendError, sendJson } from "./response.js";import { health } from "../routes/health.js";import { labConfig } from "../routes/lab-config.js";import { sendSample } from "../routes/sample.js";import { datasets } from "../routes/datasets.js";import { inspectSample } from "../routes/inspect.js";import { sendGenerated } from "../routes/generated.js";import { sendParseSummary } from "../routes/parse-summary.js";import { sendTransformed } from "../routes/transform.js";import { runBackpressure } from "../routes/backpressure.js";import { sendLive } from "../routes/live.js";import { runFailureProbe } from "../routes/failure.js";export interface RouterDependencies { database: DatabaseClient; samplePath: string; startedAt: number;}const INSPECT_PATH = "/api/lab/inspect";const GENERATED_PATH = "/api/lab/generated";const SUMMARY_PATH = "/api/lab/summary";const TRANSFORM_PATH = "/api/lab/transform";const BACKPRESSURE_PATH = "/api/lab/backpressure";const LIVE_PATH = "/api/lab/live";const FAILURE_PATH = "/api/lab/failure";const EXPORT_PATH = "/api/lab/export";const knownPaths = new Set<string>([ ...Object.values(API_PATHS), INSPECT_PATH, GENERATED_PATH, SUMMARY_PATH, TRANSFORM_PATH, BACKPRESSURE_PATH, LIVE_PATH, FAILURE_PATH, EXPORT_PATH,]);
EXPORT_PATH follows the existing local-constant convention for lab routes.
Adding the path to knownPaths affects unsupported-method handling; it does not dispatch GET requests by itself.
Existing route imports, constants, and paths remain intact.
The router’s method gate runs before any route-specific query validation or export work.
export const route = async ( request: IncomingMessage, response: ServerResponse, deps: RouterDependencies,): Promise<void> => { const pathname = new URL(request.url ?? "/", "http://localhost").pathname; if (request.method !== "GET") { if (knownPaths.has(pathname)) { sendError( response, 405, "METHOD_NOT_ALLOWED", "Only GET is supported by this endpoint.", { allow: "GET" }, ); } else { sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", ); } return; }
The router extracts pathname, not the query string. Query parsing belongs to the selected route.
A non-GET request to /api/lab/export is now a request to a known path.
That request receives the existing 405 METHOD_NOT_ALLOWED response with Allow: GET.
A non-GET request to an unknown path still receives 404.
The remaining GET branches preserve existing behavior and add export dispatch before the unchanged unknown-route fallback.
if (pathname === API_PATHS.health) return sendJson(response, 200, await health(deps.database, deps.startedAt)); if (pathname === API_PATHS.labConfig) return sendJson(response, 200, labConfig()); if (pathname === API_PATHS.sample) return sendSample(response, deps.samplePath); if (pathname === API_PATHS.datasets) return datasets(response, deps.database); if (pathname === INSPECT_PATH) return inspectSample(response, deps.samplePath); if (pathname === GENERATED_PATH) return sendGenerated(response, request.url); if (pathname === SUMMARY_PATH) return sendParseSummary(response, deps.samplePath); if (pathname === TRANSFORM_PATH) return sendTransformed(response, deps.samplePath, request.url); if (pathname === BACKPRESSURE_PATH) return runBackpressure(response, request.url); if (pathname === LIVE_PATH) return sendLive(response, request.url); if (pathname === FAILURE_PATH) return runFailureProbe(response, request.url); sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", );};
Existing GET branches remain in place.
GET /api/lab/export?records=3 calls the imported exportGenerated handler with the response and original request URL.
Passing request.url allows the export route to parse records with the established query helper.
The final 404 remains necessary for unmatched GET paths.
Adding only the GET branch would not make non-GET export requests return 405. Adding only EXPORT_PATH to knownPaths would not make valid GET export requests reachable. Both router responsibilities are required.
Observe the Completed Export Contract
A request to GET /api/lab/export?records=3 follows this backend flow:
The router confirms the request is GET and dispatches /api/lab/export.
The route validates records with default 100 and inclusive bounds from 1 through 10_000.
The route creates a UUID-backed .ndjson final path beneath /tmp/codesignal-stream-lab/exports.
The generated NDJSON source writes into the helper’s private sibling .part file.
The helper awaits pipeline(), renames the temporary file to the final .ndjson path, and reads the completed byte count.
The export route uses the project’s established integer-query parser. backend/src/http/query.ts defines its defaulting, validation, and error behavior.
TypeScript
export interface IntegerParamOptions { defaultValue: number; min: number; max: number;}export const integerParam = ( url: URL, name: string, options: IntegerParamOptions,): number => { const raw = url.searchParams.get(name); if (raw === null) return options.defaultValue; if (!/^-?\d+$/.test(raw)) { throw new HttpError( 400, "INVALID_QUERY_PARAMETER", `${name} must be an integer between ${options.min} and ${options.max}.`, ); } const value = Number(raw); if ( !Number.isSafeInteger(value) || value < options.min || value > options.max ) { throw new HttpError( 400, "INVALID_QUERY_PARAMETER", `${name} must be an integer between ${options.min} and ${options.max}.`, ); } return value;};
An absent parameter uses defaultValue, but an invalid supplied value is never silently replaced by that default.
The regular expression requires integer text before conversion.
Number.isSafeInteger() rejects values JavaScript cannot represent safely as integers.
min and max are inclusive. For the export route, both records=1 and records=10_000 are valid.
Invalid values throw the established HttpError before filesystem work begins.
The export route receives generated bytes from the createObservationSource() factory in backend/src/streams/observation-source.ts.