In the previous lesson, GET /api/lab/sample changed from a buffered response into an awaited pipeline() that sends a fresh file Readable directly into the HTTP ServerResponse. That browser-facing stream remains responsible for delivering fixture bytes to a client. This lesson adds a separate server-side capability for application code that needs to consume a fresh source and report what it observed.
The important distinction is the observation boundary:
The browser can count chunks delivered through an HTTP response. Those chunks describe how the response body was divided when it reached the browser's stream reader, not necessarily how the original file stream produced its values.
Server application code can count chunks yielded directly by a Node.js file Readable. This measurement happens before the data crosses the HTTP transport boundary.
Both consumers can describe the same complete fixture bytes because each operation reads the same file contents from its own fresh source.
Their chunk counts do not need to match because chunk boundaries can change as data moves through filesystem reads, buffering layers, HTTP transport, proxies, and browser APIs.
A chunk is not a permanent property of a file. Filesystem reads, stream buffering, HTTP delivery, proxies, and browser readers can all choose different boundaries. When both consumers finish reading the fixture, their byte totals can describe the same data. Chunk totals only describe the boundary where they were measured.
The Existing Browser-Facing Stream
backend/src/routes/sample.ts owns the existing path from the file source to the HTTP response. Its responsibility is to set response headers and forward fixture bytes; it does not inspect individual source chunks.
The first part of the route imports the HTTP response type, Node's promise-based pipeline() helper, and the factory that creates a fresh file stream.
TypeScript
import type { ServerResponse } from "node:http";import { pipeline } from "node:stream/promises";import { createSampleReadStream } from "../streams/file-source.js";
ServerResponse describes Node's writable HTTP response object. Importing it as a type makes the expected route dependency explicit without introducing a runtime import that the application does not need.
pipeline comes from node:stream/promises, so it returns a promise that settles when the streaming operation completes or fails. This allows the route to participate naturally in the application's asynchronous request lifecycle.
createSampleReadStream is kept outside the route itself. The route therefore knows that it can request a source for the fixture without having to know which filesystem API creates that source.
The route handler first writes the successful response headers before any fixture bytes are sent.
sendSample is asynchronous because completion of the route depends on completion of the streaming operation that follows. Returning Promise<void> communicates that callers can wait for that work to finish.
response.writeHead(200, ...) commits a successful HTTP status together with the response metadata before the body begins flowing.
The content-type remains application/x-ndjson; charset=utf-8, which accurately describes the sample fixture being streamed.
x-stream-lab-mode: file-stream makes the implementation mode visible to the browser and tests. It identifies that the response is coming from the file-stream implementation rather than from a previously buffered body.
There is intentionally no content-length header here. The route has not created one complete in-memory buffer whose size it can advertise before streaming begins.
After the headers are ready, the route connects a fresh file Readable directly to the HTTP response.
createSampleReadStream(samplePath) creates the readable source for this specific request. Because the factory is called inside the route, the request receives a new stream instance rather than attempting to reuse a previously consumed stream.
response is the writable destination. pipeline() repeatedly moves data from the file source into the HTTP response while handling stream completion and propagation of stream failures.
The route awaits the pipeline. That means the route's promise does not resolve merely because streaming started; it resolves only when the pipeline has completed successfully.
pipeline() ends the writable destination when the source finishes, so the route must not call response.end() again afterward. Doing so would duplicate lifecycle responsibilities already handled by the pipeline.
Nothing in this route counts source chunks. Its job is transportation: take the bytes produced by the fresh file source and forward them to the browser.
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Creating a Fresh File Source
backend/src/streams/file-source.ts isolates source creation from routing, HTTP behavior, and fixture-path configuration.
The module needs only Node's file-stream API and its corresponding stream type.
import { createReadStream, type ReadStream } from "node:fs";
createReadStream is the runtime function that creates a readable filesystem stream.
ReadStream is imported as a type so the factory can state precisely what kind of stream it returns.
Keeping these imports in a dedicated stream module prevents route code from becoming responsible for low-level source construction.
The function accepts the fixture path instead of resolving that path internally. This keeps configuration outside the source factory and lets application setup decide which file should be used.
Every call to createReadStream(samplePath) creates a new ReadStream object. That fresh-instance behavior becomes important once more than one operation needs to consume the same fixture.
The function does not read the whole file before returning. It creates the stream object, while actual data consumption happens when another part of the application begins reading from that stream.
A stream instance is consumable state. Once one operation has read through a stream, another independent operation should not expect to rewind and reuse the same object.
The sample route and the inspection route can therefore work with the same fixture independently because each calls createSampleReadStream(samplePath) and receives its own source.
Browser Measurements Are Transport Measurements
The frontend deliberately treats the streamed sample and the inspection summary as different API contracts. frontend/src/api/client.ts returns the raw browser Response for the sample but parses JSON for the inspection endpoint.
The client begins by importing the shared API paths and shared response types it already uses.
import { API_PATHS, type HealthResponse, type LabConfigResponse, type ServerInspection,} from "@stream-lab/shared";
API_PATHS contains the endpoint paths that are part of the existing shared API contract.
HealthResponse and LabConfigResponse provide TypeScript types for JSON endpoints already used by the frontend.
The inspection endpoint will deliberately remain local rather than being added to API_PATHS, so its path does not appear in this import list.
The frontend then defines the shape of the server-side inspection result.
source is the literal value "sample-file". Using a literal type documents exactly which source the current endpoint reports rather than allowing any arbitrary string.
chunks records the number of values yielded while the backend directly consumed the file Readable.
bytes records the total number of bytes represented by all of those yielded values.
This interface describes metadata about a completed server-side read. It does not contain the NDJSON fixture body itself.
For JSON endpoints, the client continues to use a small generic helper.
const getJson = async <T>(path: string): Promise<T> => { const response = await fetch(path); if (!response.ok) { throw new Error(`Request failed with HTTP ${response.status}`); } return response.json() as Promise<T>;};
getJson<T>() centralizes the repeated sequence of fetching a path, checking whether the HTTP response succeeded, and parsing its body as JSON.
The generic type parameter T allows each caller to specify the expected JSON result while keeping the runtime fetching logic in one place.
The response.ok check converts unsuccessful HTTP statuses into an application-level error instead of allowing the caller to accidentally treat an error response as successful data.
response.json() consumes the response body and produces the completed JSON value. That behavior is appropriate for small metadata responses such as the inspection summary.
This helper is not appropriate for the sample stream because parsing the sample immediately would hide the raw Response.body stream that the browser probe needs to observe.
health() and config() continue to use getJson() because both endpoints return ordinary JSON documents.
sample() deliberately calls fetch() directly and returns the raw browser Response. It does not call response.json() or response.text().
Keeping the raw sample response allows the browser probe to access Response.body.getReader() and observe values as the browser's response stream produces them.
inspect() behaves differently because the backend has already consumed its own file source by the time this response is sent. The frontend receives only the resulting JSON metadata.
/api/lab/inspect is written locally here instead of being added to API_PATHS, matching the decision that this inspection path is not part of the shared endpoint constants.
Keeping Browser and Server State Separate
frontend/src/App.tsx keeps browser and server observations in separate state. That separation prevents two measurements taken at different boundaries from being treated as though they were the same result.
The component starts with distinct result and loading state for the two operations.
result stores the browser-side ProbeResult. Its chunk and byte values come from reading the HTTP response body in the browser.
inspection stores the ServerInspection returned by the backend. Its chunk and byte values come from directly iterating over the server-side file source.
Separate probeLoading and inspectionLoading flags allow either operation to run without pretending that the other operation is also in progress.
The shared error state provides one place for the UI to surface failures from either action.
Most importantly, the state model keeps result and inspection as different concepts. Even if the numbers happen to look similar, their chunk measurements come from different stream boundaries.
Running the Browser Probe
The browser probe requests the raw sample response and passes it to the existing response reader.
setProbeLoading(true) marks only the browser probe as active, leaving the inspection operation independently controlled.
Clearing the previous error at the start prevents an old failure from remaining visible while a new request is attempted.
performance.now() captures the browser-side starting point used by the probe to calculate elapsed time.
api.sample() returns the raw Response, preserving access to the streaming body instead of parsing it eagerly.
readResponse() consumes that browser response stream. Its chunk count therefore describes values delivered through the browser's Response.body, not values originally emitted by the Node.js file source.
The 1_000 argument continues to limit the amount of preview data retained for display while allowing the reader to measure the complete response.
The remainder of the action handles failures and restores the loading state.
setInspectionLoading(true) marks the server inspection operation as active independently of the browser probe.
api.inspect() uses the JSON helper, so the frontend waits for the complete inspection summary rather than reading the fixture stream itself.
The backend performs the actual file consumption before responding. By the time this promise resolves, chunks and bytes already represent a completed server-side traversal.
setInspection(...) stores those server measurements separately from the browser's ProbeResult.
This action does not call api.sample() and does not pass anything to readResponse(). The browser therefore does not inspect the fixture body as part of this operation.
The action finishes with the same predictable error and loading-state handling.
} catch (cause) { setError( cause instanceof Error ? cause.message : "The server inspection failed.", ); } finally { setInspectionLoading(false); }};
Inspection failures are converted into a message specifically associated with the server inspection action.
The fallback text makes it clear that this operation is different from the sample probe even though both actions share the component's error state.
setInspectionLoading(false) runs from finally, so the inspection control returns to its normal state after either success or failure.
The two frontend actions are now independent requests with independent observations: one measures browser delivery, while the other asks the server to measure direct source consumption.
Rendering the Two Observation Boundaries
frontend/src/components/ProbeResults.tsx makes the browser and server measurements visible without implying that their chunk counts should match.
The component first imports the two result types it needs.
import type { ProbeResult } from "@stream-lab/shared";import type { ServerInspection } from "../api/client";
ProbeResult represents what the browser observed while reading the HTTP response.
ServerInspection represents what the backend observed while directly consuming its own fresh file source.
Importing two distinct types helps preserve the conceptual boundary in the UI instead of collapsing both results into one generic structure.
result can exist without inspection because the user can run only the browser probe.
inspection can similarly exist without result because server inspection is a separate action.
Making both values nullable allows the component to represent every stage: no measurements, only browser measurements, only server measurements, or both.
Before rendering metrics, the component handles the empty state.
export const ProbeResults = ({ result, inspection,}: ProbeResultsProps) => { if (!result && !inspection) { return ( <p className="empty-state"> Run a browser probe or server inspection to compare observations. </p> ); }
The component checks both values because either type of measurement is enough to display useful results.
When neither operation has produced data, the message explicitly mentions both available actions.
The wording also introduces the intended mental model: these values are observations to compare, not duplicate measurements that are required to be identical.
When a browser result exists, the component labels it as a browser response measurement.
BROWSER RESPONSE establishes where these measurements were taken: after the data traveled through the HTTP response and reached browser APIs.
HTTP status and Media type are transport-facing properties, which further reinforces that this result represents an HTTP response observation.
The label Observed chunks is intentionally general rather than claiming that these are file-stream chunks. They are the values observed by the browser's response reader.
Bytes can be meaningfully compared with the server's byte total because complete consumption on both sides should describe the same fixture contents.
Elapsed belongs specifically to the browser probe and describes the timing measured around that client-side operation.
The browser result also renders the preview produced while reading the response.
result.preview contains the limited body preview assembled by the browser reader while it consumed the streamed HTTP response.
Falling back to "(empty body)" gives the interface meaningful output even if no preview text is available.
The preview exists only for the browser-facing result. The inspection endpoint intentionally returns metadata instead of returning the fixture contents again.
The server result uses separate labels that describe direct Readable consumption.
SERVER READABLE explicitly identifies a different observation boundary from the browser response shown above it.
Source displays "sample-file", making it clear which server-side source was consumed to create the metadata.
Readable chunks refers specifically to the values yielded by direct async iteration over the server's file source.
Readable bytes is the sum of the byte lengths of those yielded values after the entire source has been consumed.
The component does not present the server chunk count as an expected match for the browser count. The labels intentionally describe where each number came from.
Matching chunk counts can happen, but they are not a contract and do not prove that source chunk boundaries survived the HTTP path. The complete byte sequence is the meaningful shared property.
New in this unit: direct source consumption
The new helper consumes an async iterable itself, records only metadata, and leaves the browser-facing pipeline unchanged.
A Generic Async-Iteration Helper
backend/src/streams/inspect-readable.ts contains the reusable inspection helper. Its responsibility is to consume any compatible asynchronous source once and return only numeric totals.
The module first defines the small result object returned after inspection finishes.
chunks represents the number of individual values yielded by the asynchronous source during one complete traversal.
bytes represents the total byte length of all supported chunks encountered during that traversal.
The result intentionally contains only metadata. It does not retain chunks, reconstruct a body, or expose source-specific information.
Keeping this interface small makes the inspection behavior useful independently of HTTP routing or file-specific code.
Measuring Individual Chunk Sizes
Because the generic helper accepts values typed as unknown, it needs a small boundary function that determines which chunk shapes are supported and how many bytes each contains.
const chunkByteLength = (chunk: unknown): number => { if (typeof chunk === "string") { return Buffer.byteLength(chunk); } if (chunk instanceof Uint8Array) { return chunk.byteLength; } throw new TypeError("Expected stream chunks to be strings or byte arrays.");};
The parameter begins as unknown because the inspection helper does not assume that every asynchronous source has already provided a narrow TypeScript chunk type.
Strings use Buffer.byteLength(chunk) rather than chunk.length. JavaScript string length counts UTF-16 code units, while Buffer.byteLength() reports the UTF-8 encoded byte size used intentionally by this helper.
Uint8Array values already represent sequences of bytes, so their .byteLength property directly provides the required size.
Node.js Buffer values are compatible with the Uint8Array check because Buffer extends Uint8Array. The helper therefore handles ordinary binary chunks produced by a file Readable.
Unsupported values produce a clear TypeError instead of being silently ignored or converted into an unreliable byte count.
This validation keeps the counting loop itself simple: once chunkByteLength() returns, the caller knows it received a supported value and a meaningful byte measurement.
Consuming the Source with for await...of
The inspection function accepts an asynchronous iterable rather than a file-specific stream type.
export const inspectReadable = async ( source: AsyncIterable<unknown>,): Promise<ReadableInspection> => { let chunks = 0; let bytes = 0;
AsyncIterable<unknown> describes the actual capability the function needs: a source that can asynchronously yield values over time.
The helper does not require ReadStream, samplePath, ServerResponse, or any other file- or HTTP-specific concept.
A Node.js file Readable supports asynchronous iteration, so it satisfies this contract without requiring the helper to depend directly on filesystem APIs.
chunks and bytes begin at zero because both totals are accumulated incrementally as the source produces values.
Only numeric counters are stored. There is no array for retaining chunks and no buffer for reconstructing the complete body.
The central consuming operation is a for await...of loop.
for await (const chunk of source) { chunks += 1; bytes += chunkByteLength(chunk); }
for await...of requests values from the asynchronous source one at a time and waits when the next value is not yet available.
Each iteration corresponds to one value yielded at this particular source boundary, so chunks += 1 records exactly one additional observed chunk.
chunkByteLength(chunk) validates the yielded value and calculates its byte size before that size is added to the running total.
Once an iteration finishes, the helper does not need to retain the chunk. The local loop variable can be released as iteration moves forward.
This makes the helper fundamentally different from buffering logic. It can inspect a stream completely while retaining only constant-size numeric state rather than a growing copy of the body.
The loop ends naturally when the asynchronous source signals completion.
After the source ends, the helper returns the completed measurements.
return { chunks, bytes };};
The return statement executes only after for await...of has finished consuming the source.
chunks therefore represents the complete number of values observed during this traversal rather than a partial snapshot.
bytes represents the complete sum of supported chunk byte lengths.
The consumed data is not returned. Only the two counters survive the operation.
There is no second traversal. Streams are consumed once, and the helper derives all required metadata during that one pass.
Reusing the Existing JSON Response Helper
backend/src/http/response.ts centralizes JSON response behavior. The inspection route should reuse this existing behavior instead of manually repeating response serialization.
The module starts with the HTTP response type and the project's shared error-envelope type.
import type { ServerResponse } from "node:http";import type { ErrorEnvelope } from "@stream-lab/shared";
ServerResponse provides the type of the writable HTTP response passed into these helpers.
ErrorEnvelope describes the project's established JSON shape for error responses.
Keeping this response behavior centralized allows individual routes to focus on producing values rather than repeatedly implementing HTTP serialization details.
sendJson() handles successful or otherwise arbitrary JSON values.
The helper constructs the standard { error: { code, message } } structure expected by the project.
Optional headers allow callers to attach information such as Allow: GET for method errors without duplicating response logic.
Delegating to sendJson() guarantees that error responses use the same JSON content type and serialization path as other JSON responses.
The new inspection route does not need to modify either helper. It can simply reuse sendJson() when its completed metadata is ready.
Returning Inspection Metadata Instead of Fixture Bytes
backend/src/routes/inspect.ts composes three existing responsibilities: creating a fresh source, inspecting that source, and sending the resulting metadata as JSON.
The route first imports those dependencies.
import type { ServerResponse } from "node:http";import { sendJson } from "../http/response.js";import { createSampleReadStream } from "../streams/file-source.js";import { inspectReadable } from "../streams/inspect-readable.js";
ServerResponse defines the route's HTTP response parameter.
sendJson owns JSON serialization and response completion.
createSampleReadStream owns construction of a fresh file source.
inspectReadable owns asynchronous consumption and measurement of that source.
The route itself can therefore remain a composition layer rather than duplicating the responsibilities of any of these modules.
The handler begins by accepting the response and the configured fixture path.
Like the streaming route, the inspection route returns Promise<void> because it performs asynchronous work before the response can be completed.
samplePath arrives as a dependency rather than being resolved inside the route. That keeps environment and application configuration outside request-handling logic.
The handler does not accept an existing stream instance because every inspection request should create and consume a fresh source.
The route creates that source and waits for complete inspection.
createSampleReadStream(samplePath) is evaluated for this request, producing a fresh ReadStream.
That new source is passed directly to inspectReadable(), whose for await...of loop consumes it exactly once.
await is essential here. inspection should not exist as a completed result until the file source has ended and all yielded values have been counted.
If the route attempted to send JSON before awaiting completion, the totals could describe only part of the source.
The helper retains only its counters while reading, so this operation measures the fixture without reconstructing the entire body in application memory.
Once inspection has completed, the route sends a small JSON description.
source: "sample-file" identifies the specific source represented by the returned metadata.
...inspection contributes the completed chunks and bytes values from the generic helper.
The route returns metadata rather than fixture bytes. The file contents have already been consumed on the server and are not forwarded to the browser by this endpoint.
sendJson() serializes the object and ends the response, so no additional response-ending logic is required.
This route is intentionally different from sendSample(): sendSample() transports fixture bytes, while inspectSample() consumes those bytes in application code and reports what that consumption observed.
Keeping the Inspection Path Local
shared/src/index.ts continues to define the existing endpoint constants that belong to the shared API contract.
These paths remain unchanged because they already represent the project's shared endpoint constants.
API_PATHS.inspect makes the inspection endpoint available through the same shared contract as the browser-facing endpoints.
The frontend and backend can still use the inspection endpoint by defining that path locally where it is needed.
Keeping the distinction explicit prevents every internal or lab-specific route from automatically becoming part of the shared API contract.
Registering the Inspection Route
backend/src/http/router.ts needs to recognize the local inspection path and dispatch successful GET requests to inspectSample().
The router imports its existing dependencies together with the new inspection handler.
import type { IncomingMessage, ServerResponse } from "node:http";import { API_PATHS } from "@stream-lab/shared";import type { DatabaseClient } from "../db/client.js";import { sendError, sendJson } from "./response.js";import { health } from "../routes/health.js";import { labConfig } from "../routes/lab-config.js";import { sendSample } from "../routes/sample.js";import { datasets } from "../routes/datasets.js";import { inspectSample } from "../routes/inspect.js";
IncomingMessage and ServerResponse describe the raw Node.js request and response objects handled by the router.
API_PATHS continues to provide the shared endpoint paths.
The existing route handlers remain unchanged, while inspectSample adds the new server-side inspection behavior.
Importing the inspection route does not make its path shared. Path ownership and handler implementation remain separate concerns.
The router dependencies continue to include the configured sample path.
database supports the existing database-backed routes.
startedAt supports the existing health behavior.
samplePath is the dependency both sample-related routes need. The router can pass the same configured path to each route while each route independently creates a fresh stream.
Passing dependencies through this object prevents request handlers from reaching outward to rediscover application configuration.
The inspection path is defined locally and added to the set of routes the router recognizes.
const INSPECT_PATH = "/api/lab/inspect";const knownPaths = new Set<string>([ ...Object.values(API_PATHS), INSPECT_PATH,]);
INSPECT_PATH gives the backend one local constant for the route instead of repeating the string throughout the router.
Object.values(API_PATHS) brings the existing shared paths into knownPaths.
Adding INSPECT_PATH means the router knows that /api/lab/inspect is a real endpoint even before it checks whether the incoming method is supported.
This distinction matters for error semantics. A request can target a known path with the wrong method, which should be different from requesting a path that does not exist at all.
Preserving GET-Only Method Handling
The request handler first resolves the pathname from the incoming URL.
The router is asynchronous because some route handlers await operations such as stream pipelines or async iteration.
request.url ?? "/" provides a fallback if the incoming URL is unexpectedly missing.
new URL(..., "http://localhost") supplies a base URL so relative request URLs can still be parsed using the standard URL API.
Reading .pathname isolates route matching from query-string details.
Before selecting a GET handler, the router rejects unsupported methods.
if (request.method !== "GET") { if (knownPaths.has(pathname)) { sendError( response, 405, "METHOD_NOT_ALLOWED", "Only GET is supported by this endpoint.", { allow: "GET" }, );
The method check happens before successful route dispatch, so every registered endpoint consistently follows the application's GET-only policy.
knownPaths.has(pathname) distinguishes a real endpoint from an unknown URL.
A non-GET request to a known endpoint receives HTTP 405 Method Not Allowed.
The existing "METHOD_NOT_ALLOWED" error code and message are reused instead of inventing inspection-specific error behavior.
{ allow: "GET" } adds the standard Allow response header so the client can see which method the endpoint supports.
Because INSPECT_PATH was added to knownPaths, POST /api/lab/inspect correctly follows this branch.
Unknown paths with unsupported methods retain their existing 404 behavior.
} else { sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", ); } return; }
If the pathname is not registered at all, changing the HTTP method does not turn it into a known resource. The router therefore returns 404 Not Found.
The return prevents execution from continuing into the successful GET dispatch logic after either error response has already been sent.
This preserves the existing distinction between an unsupported operation on a real route and a request for a route that does not exist.
Dispatching GET Requests
Once the request is known to use GET, the router can select the appropriate handler.
The existing health route continues to await its asynchronous result before passing that result to sendJson().
The lab configuration route remains synchronous and is serialized immediately.
Returning from each branch ensures only one route handler is responsible for a request.
The existing sample and dataset routes remain registered as before.
if (pathname === API_PATHS.sample) { return sendSample(response, deps.samplePath); } if (pathname === API_PATHS.datasets) { return datasets(response, deps.database); }
The sample route receives deps.samplePath, so it can create a fresh file source and stream it to the browser.
The router returns the promise produced by sendSample(), preserving the asynchronous pipeline as part of the request lifecycle.
The datasets route continues to receive the database dependency and remains independent of the new inspection functionality.
The new local path dispatches to the inspection handler using the same configured sample path.
if (pathname === INSPECT_PATH) { return inspectSample(response, deps.samplePath); }
Matching INSPECT_PATH invokes the new inspection route only after the router has established that the request method is GET.
deps.samplePath is passed into the handler rather than resolving the fixture location inside inspectSample().
Although both sample-related routes receive the same path, they do not share a stream instance. Each handler independently calls createSampleReadStream() when it executes.
Returning the handler's promise keeps completion or failure of the entire inspection operation connected to the router's promise.
Finally, unmatched GET requests keep the existing 404 response.
sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", );};
This final branch handles GET requests whose path matched none of the registered endpoints.
The same standard error envelope is used, so adding inspection does not create a separate routing error format.
Together with the earlier method handling, the router now guarantees that POST /api/lab/inspect receives 405 while a request such as GET /api/nope receives 404.
Keeping Async Work in the Request Lifecycle
backend/src/app.ts creates the HTTP server and connects asynchronous route failures to request-level error handling.
The module begins with its server and routing dependencies.
import { createServer, type Server } from "node:http";import type { DatabaseClient } from "./db/client.js";import { route } from "./http/router.js";
createServer constructs the Node.js HTTP server that receives requests.
Server provides the return type of the application factory.
DatabaseClient describes one application dependency, while route contains the request dispatch behavior.
The application layer does not need to understand how inspection iterates over a file. Its responsibility is to provide dependencies and manage request-level failures.
The application dependency type includes the sample path used by both sample-related routes.
database and samplePath are required application dependencies because routes need them to perform their work.
startedAt remains optional so tests or application setup can inject a value when needed.
Keeping samplePath at this level ensures both the streaming and inspection routes receive configuration through dependency injection instead of hard-coding fixture locations.
The application factory calculates the start time and creates the server.
route() can return a promise because handlers such as sendSample() and inspectSample() perform asynchronous stream work.
void makes it explicit that the server callback is not returning that promise to Node's createServer() API.
Instead, .catch(...) is attached directly to the routing promise so asynchronous failures remain connected to this request.
The error is logged using its message when possible, while still handling unknown thrown values safely.
An inspection failure from for await...of can therefore propagate through inspectReadable(), inspectSample(), and route() until it reaches this request-level catch handler.
If headers have already been sent, the application cannot replace the response with a new JSON error body.
if (response.headersSent) { response.destroy(); return; }
response.headersSent tells the application whether the HTTP response has already been committed.
This is particularly relevant for streaming. sendSample() writes headers before its pipeline begins, so a later stream failure can happen after the successful status has already been sent.
At that point, attempting to send a second status and JSON error document would produce an invalid response.
Destroying the response terminates the failed connection instead.
The immediate return prevents the handler from falling through into the JSON 500 path.
When no response headers have been sent yet, the application can still return its normal internal-error response.
A failure that occurs before headers are committed can still be represented as an ordinary HTTP 500 Internal Server Error.
The application uses the established JSON content type and error-envelope shape.
For the inspection route, this can cover failures that happen while opening or consuming the file before sendJson() has written the successful inspection response.
Both sample-related routes therefore fit the same request-lifecycle model even though their response behavior differs: sendSample() awaits pipeline(), while inspectSample() awaits inspectReadable().
The Fixture Is the Byte Authority
backend/data/sample.ndjson is the byte authority for both file-stream operations.
The fixture contains three NDJSON records, with one JSON object represented on each line.
Those three logical records should not be interpreted as three guaranteed stream chunks. NDJSON record boundaries and stream chunk boundaries describe different concepts.
A file Readable may yield the entire fixture in one chunk, split it across several chunks, or choose other boundaries depending on buffering and runtime behavior.
The browser may observe yet another segmentation after the data passes through the HTTP stack.
The stable property is the complete byte sequence represented by the fixture. Any complete consumer should account for those bytes regardless of how many chunks it observes.
Testing Through the Real Application
backend/test/app.test.ts starts the real application and verifies the behavior through HTTP. The tests focus on deterministic contracts rather than assuming a particular chunk segmentation.
The test module first imports its runtime and application dependencies.
import assert from "node:assert/strict";import { readFile } from "node:fs/promises";import { once } from "node:events";import test from "node:test";import type { AddressInfo } from "node:net";import { createApp } from "../src/app.js";import type { DatabaseClient } from "../src/db/client.js";
assert provides strict assertions for response statuses, headers, JSON values, and body bytes.
readFile is used in the tests to load the complete expected fixture bytes. This buffering happens in test code to create an authority for comparison; it does not change the production streaming implementation.
once allows the test to await server lifecycle events such as "listening" and "close".
test provides Node's built-in test runner API.
AddressInfo gives TypeScript the information needed to read the ephemeral port selected by the operating system.
createApp ensures tests exercise the actual routing and request-lifecycle behavior rather than testing only isolated helper functions.
Providing a Database Stub
These stream-focused tests still need to satisfy the application's database dependency, so they provide a deliberately unavailable test implementation.
state() and ping() consistently report "degraded", giving the health endpoint predictable behavior.
query() throws because these tests do not provide a functioning database.
close() resolves without performing work because there is no real test connection to close.
The object satisfies DatabaseClient, allowing the real application to start without introducing a database dependency into the stream inspection behavior being tested.
Starting and Stopping a Test Server
The withServer() helper starts the real application on an ephemeral loopback port and guarantees cleanup afterward.
Passing port 0 asks the operating system to allocate an available ephemeral port, which avoids hard-coded port conflicts during tests.
Binding to 127.0.0.1 keeps the server local to the test environment.
Waiting for "listening" ensures HTTP requests are not attempted before server startup has completed.
server.address() exposes the selected port, which is incorporated into the base URL passed to the test callback.
Because the callback communicates through HTTP, the tests exercise route selection, source creation, stream consumption, and response behavior together.
A successful 200 confirms that the browser-facing sample endpoint still operates normally.
x-stream-lab-mode must remain "file-stream", preserving the explicit implementation indicator.
The absence of content-length verifies the route's deliberate contract not to calculate or advertise a representation length. It is not evidence that the body must be streamed.
These assertions focus on the HTTP contract of the streaming route rather than on internal chunk boundaries.
The test then compares the complete response body with the fixture bytes.
body.source === "sample-file" confirms that the route identifies the expected source.
body.chunks > 0 confirms that the source actually yielded data. The test deliberately does not require a specific number such as one, three, or any other exact count.
body.bytes === expected.byteLength is the important completeness assertion. It proves that the helper accounted for the complete fixture even though the runtime was free to choose its own chunk boundaries.
No body-content copy needs to be returned by the endpoint. The expected fixture is loaded only by the test so its total size can be compared with the reported server-side measurement.
This is exactly the kind of assertion a stream test should prefer: verify stable data properties while remaining flexible about implementation-dependent segmentation.
Verifying 404 and 405 Behavior
The final integration test protects existing database and routing error semantics while also confirming that the inspection path is recognized as a real GET-only endpoint.
test("database, unknown route, and bad method responses are explicit", async () => { await withServer(async (baseUrl) => { const datasets = await fetch(`${baseUrl}/api/datasets`); assert.equal(datasets.status, 503); assert.equal( (await fetch(`${baseUrl}/api/nope`)).status, 404, );
The dataset request continues to return 503 with the deliberately unavailable database, confirming that unrelated application behavior remains intact.
/api/nope returns 404, demonstrating that genuinely unknown paths are still classified as missing routes.
These assertions establish the baseline needed to distinguish an unknown path from a known endpoint requested with the wrong method.
The test then sends an unsupported method to the inspection endpoint.
POST /api/lab/inspect targets a real route, so the correct response is 405 Method Not Allowed rather than 404 Not Found.
The Allow header must contain "GET", matching the router's existing GET-only contract.
This assertion indirectly verifies that INSPECT_PATH was included in knownPaths before the method check executes.
Together, the tests confirm three separate cases: an unavailable application dependency can produce 503, an unknown URL produces 404, and an unsupported method on a known inspection endpoint produces 405.
Comparing the Two Measurements
At this point, the application can read the same fixture through two intentionally separate paths.
When the browser probe runs, the data flow looks conceptually like this:
The browser's chunk count is measured only at the final Response.body boundary shown here.
The Node.js source, HTTP implementation, networking layers, and browser can all influence how data becomes grouped before the browser reader observes it.
The browser still receives the same complete fixture bytes even if those bytes arrive using different chunk boundaries from the original file Readable.
inspectReadable() observes values directly from a fresh server-side source instead of forwarding them through an HTTP response.
Its chunks value therefore describes the file Readable at the point of direct asynchronous iteration.
Once the source ends, the route returns only metadata. The fixture bytes themselves are not forwarded as the inspection response.
Both paths can report the same total bytes while reporting different chunk counts, and that difference is completely valid.
Recap
The application now has two deliberately separate ways to work with the sample fixture:
backend/src/routes/sample.ts creates a fresh file Readable and streams it to the browser through an awaited pipeline(). This route is concerned with transportation, so it forwards fixture bytes without inspecting source chunks individually.
backend/src/streams/file-source.ts remains the common source factory. Every call creates a new ReadStream, which allows independent operations to consume the same underlying fixture without trying to reuse an already-consumed stream object.
backend/src/streams/inspect-readable.ts accepts a generic AsyncIterable<unknown> and consumes it once with for await...of. It validates supported chunk types, counts every yielded value, adds their byte lengths, and retains only numeric totals instead of collecting the body.
backend/src/routes/inspect.ts creates a fresh file source, waits until inspectReadable() has consumed that source completely, and then returns { source, chunks, bytes } as JSON rather than returning the fixture contents.
frontend/src/api/client.ts preserves two different client contracts. api.sample() returns the raw Response so the browser can inspect its response stream, while api.inspect() parses a completed JSON metadata response.
frontend/src/App.tsx keeps the two operations in separate state and sends separate requests, reinforcing that browser transport measurements and server source measurements represent different observation boundaries.
frontend/src/components/ProbeResults.tsx labels the values accordingly. Observed chunks refers to browser-delivered response values, while Readable chunks refers to values directly yielded by the server's file source.
backend/src/http/router.ts recognizes /api/lab/inspect as a local GET-only path while preserving the difference between 404 Not Found for unknown paths and 405 Method Not Allowed for unsupported methods on known paths.
backend/src/app.ts keeps asynchronous stream work attached to the active request. Both pipeline() failures and async-iteration failures can propagate through the router into the application's existing request-level error handling.
backend/test/app.test.ts verifies deterministic properties rather than implementation-dependent segmentation. The inspection must yield at least one chunk and must report the fixture's complete byte length, but the test never requires a specific chunk count or equality between server and browser chunk totals.
The central mental model is simple: the same bytes can cross several boundaries, and each boundary can observe different chunking. The browser probe measures chunks delivered through HTTP. The inspection endpoint measures chunks consumed directly from the server-side file Readable. Those are complementary observations, not interchangeable ones.
import type { ServerResponse } from "node:http";import { sendJson } from "../http/response.js";import { createSampleReadStream } from "../streams/file-source.js";import { inspectReadable } from "../streams/inspect-readable.js";
import type { IncomingMessage, ServerResponse } from "node:http";import { API_PATHS } from "@stream-lab/shared";import type { DatabaseClient } from "../db/client.js";import { sendError, sendJson } from "./response.js";import { health } from "../routes/health.js";import { labConfig } from "../routes/lab-config.js";import { sendSample } from "../routes/sample.js";import { datasets } from "../routes/datasets.js";import { inspectSample } from "../routes/inspect.js";
if (request.method !== "GET") { if (knownPaths.has(pathname)) { sendError( response, 405, "METHOD_NOT_ALLOWED", "Only GET is supported by this endpoint.", { allow: "GET" }, );
TypeScript
} else { sendError( response, 404, "NOT_FOUND", "The requested API route does not exist.", ); } return; }
import assert from "node:assert/strict";import { readFile } from "node:fs/promises";import { once } from "node:events";import test from "node:test";import type { AddressInfo } from "node:net";import { createApp } from "../src/app.js";import type { DatabaseClient } from "../src/db/client.js";