Tracing NDJSON Backpressure
Tracing Pressure Through the NDJSON Pipeline
Hello, pressure tracer! 🔬 In the previous lesson, you built a manual byte-flow diagnostic where a slow writable can make write() return false, the caller waits for "drain", and final metrics are reported only after the destination finishes. This lesson keeps that reliable manual model, moves destination peak accounting into the destination itself, compares it with pipeline() lifecycle ownership, and then traces pressure through a structured NDJSON flow where queue units change from bytes to objects and back again.
The completed backend endpoint is:
It accepts these controls:
mode:manual,pipeline, orndjsonrecords: generated NDJSON record countdelayMs: delay for each destination writesourceHwm: readable-side high-water marksinkHwm: writable-side high-water mark
The fixed Backpressure Lab control sends this backend request and displays its completed JSON response.
Lab-only diagnostic: This intentionally slow endpoint has no deadline, disconnect cancellation, or concurrency control. Do not expose it publicly; a real service needs bounded aggregate work plus timeouts, cancellation, and rate or concurrency limits.
A constrained NDJSON request makes pressure observable:
Readable and Writable Pressure Have Different Owners
Streams have separate readable and writable queues.
| Operation | Queue being measured | Owner that responds to false |
|---|---|---|
readable.push(chunk) | Readable output queue | The source producer |
writable.write(chunk) | Writable input queue | The code writing to the destination |
backend/src/streams/observation-source.ts produces serialized NDJSON as byte-oriented chunks. Its _read() method owns readable-side pressure.
push()adds serialized NDJSON text to the readable queue. Because this source does not enable object mode, the queue is measured in bytes.- When
push()returnsfalse, the source stops producing for now. Node can invoke_read()again after downstream consumption creates room. nextIndexadvances before the result is checked because the current record has already entered the readable queue.- This is distinct from waiting for a writable
"drain"event. A source responds topush(), while a manual destination caller responds towrite().
A high-water mark is a configured buffering threshold. A queue length is the current amount buffered. Neither is automatically a count of NDJSON records.
