Streaming Files Through HTTP
Replacing the Buffered Baseline with a File Readable
Welcome to Stream Lab! 🌊
Over this course you will turn a deliberately simple HTTP endpoint into a genuine Node.js stream pipeline. The project is an npm workspace with three packages: backend/ (a native node:http server on port 3001), frontend/ (a React + Vite observation panel on port 3000 that proxies /api to the backend), and shared/ (TypeScript contracts and endpoint constants used by both sides). A PostgreSQL sidecar backs an unrelated /api/datasets endpoint, which exists only so the app has a realistic data boundary — it stays untouched here.
The interesting endpoint is GET /api/lab/sample. Today it responds with a small NDJSON fixture, but it does so in the least stream-like way possible: it reads the entire file into memory, computes a Content-Length, and ends the response with that single buffer. That is a correct HTTP response, and it is also a perfect starting point, because it isolates exactly one boundary — "how do the file's bytes reach the client?" — behind a tiny module.
In this unit that boundary is replaced. By the end, the endpoint creates a file-backed Readable stream, connects it to the HTTP response Writable with the Promise-based pipeline() helper, stops advertising a precomputed body length, and identifies itself as file-stream everywhere the application describes its own behavior: the shared TypeScript contract, the lab-config response the UI reads, and the backend regression tests. That linear composition — source → destination — is the pattern the rest of the course extends.
Tracing the request path that will stay stable
Before changing anything, it is worth knowing which layers are not changing. Almost all of them.
Routing happens in backend/src/http/router.ts. It resolves the pathname, rejects non-GET methods, and dispatches by comparing against the shared API_PATHS constants.
Two details matter for stream work. First, the sample branch is return sendSample(response, deps.samplePath) — route() hands back the route handler's promise, so anything the handler awaits stays inside the request's async flow. Second, the router never touches the filesystem; it forwards deps.samplePath, a plain string. The router therefore needs no changes when the body becomes a stream.
Where does samplePath come from? backend/src/config.ts resolves it once, relative to the project root:
Path configuration lives here and nowhere else. That is a boundary worth respecting deliberately: the stream module added later accepts a path rather than computing one. Otherwise the fixture location would be encoded in two places.
backend/src/app.ts wires the dependencies into the server and, importantly for streaming, owns the last-resort error handler:
That .catch is worth reading carefully, because it is the reason a streaming route must be written a particular way. If a route's promise rejects, this handler logs the failure. If headers have not been sent yet, it can still produce a clean JSON 500. If headers have already been sent — which is exactly the situation mid-body — a status line is no longer negotiable, so it destroys the socket instead. A streamed response commits to 200 before all bytes exist, so this headersSent branch is the realistic failure path. That only works if the route's promise actually settles when the stream does; a fire-and-forget body would leave this handler with nothing to catch.
The stream-specific work begins at the isolated file-access boundary.
