In the previous lesson, you built an Express server that exposes AI agents through REST endpoints and stores agent states in memory using a plain JavaScriptMap. While this worked well for learning the basics, it had a critical flaw: when the server restarts, all agent states disappear. For Factor 6 — Launch / Pause / Resume with simple APIs to work reliably, an agent paused yesterday must still be resumable today. SQLite database persistence solves this by storing states on disk. In this lesson, you will replace in-memory storage with SQLite using the StateStore class and implement progress callbacks that save the state after each agent step, enabling real-time progress tracking for clients. This persistent foundation is what makes the pause and resume endpoints that you will build in the next lesson truly practical. Before we write any code, let's see where the new pieces fit in the project.
Project Structure
The clean separation between the core agent logic and the server layer that you saw in the last lesson still holds. The key change in this lesson is a brand-new database.ts module inside the server directory, which encapsulates all persistence logic behind the StateStore class. A couple of existing files also gain small but important upgrades:
text
.├── src/│ ├── server/│ │ ├── app.ts # Express app — now reads/writes via StateStore│ │ ├── main.ts # Entry point: starts the HTTP server (unchanged)│ │ └── database.ts # NEW: StateStore class wrapping SQLite│ └── core/│ ├── agent.ts # Agent reducer — now accepts a progress callback│ ├── models/│ │ └── state.ts # State types + AGENT_STATUSES runtime array│ ├── tools/ # Tool functions and JSON schemas│ ├── prompts/ # System prompt│ └── utils/ # Context serialization helpers└── test.ts # Client script: launch + poll with live progress
Here's how each file changes relative to the previous lesson:
src/server/database.ts(new): Home of the StateStore class. It opens the SQLite connection, creates the states table, and provides get, save, and update methods. All SQL and all serialization logic lives here — nothing else in the app touches the database directly.
src/core/models/state.ts(updated): Adds the AGENT_STATUSES runtime array so that a status string read back from the database can be validated at runtime, not just at compile time.
src/core/agent.ts(updated): The run method now accepts an optional ProgressCallback that fires after every step, letting the server persist progress incrementally.
src/server/app.ts(updated): Swaps the in-memory Map for a StateStore instance, and passes a save callback into agent.run(...).
src/server/main.ts(unchanged): Still just imports app and calls app.listen(...).
test.ts(updated): Prints status, steps, and pending_tool_calls on every poll so you can watch progress accumulate in real time.
Keeping all persistence behind database.ts means the rest of your server never has to think about SQL, JSON serialization, or file paths — it just calls store.save(...) and store.get(...). This is the same decoupling principle from Factor 11 applied to storage: the core agent stays a pure reducer, and the database is just another thin adapter around it.
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Setting Up better-sqlite3
To begin working with a database, you must open a connection to a SQLite file and configure where that file resides on disk. The better-sqlite3 library handles this within the StateStore class constructor:
import Database from "better-sqlite3";import { mkdirSync } from "node:fs";import { dirname, join } from "node:path";import { fileURLToPath } from "node:url";function defaultDatabasePath(): string { const currentDirectory = dirname(fileURLToPath(import.meta.url)); return join(currentDirectory, "../../data/agent_states.db");}export class StateStore { private readonly db: Database.Database; constructor(dbPath = defaultDatabasePath()) { mkdirSync(dirname(dbPath), { recursive: true }); this.db = new Database(dbPath); this.db.pragma("journal_mode = WAL"); this.createTables(); }}
The defaultDatabasePath function uses fileURLToPath and dirname to compute the absolute path of the current file at runtime and then resolves the database file to data/agent_states.db two levels up. Because ES modules do not expose __dirname directly, this pattern is the standard replacement. The mkdirSync call with { recursive: true } ensures the data/ directory exists before better-sqlite3 tries to create the file — otherwise, the database constructor would throw an error if the directory were missing. new Database(dbPath) opens a synchronous connection to the SQLite file, creating it if it does not already exist. Finally, this.db.pragma("journal_mode = WAL") switches the database to Write-Ahead Logging (WAL) mode, which improves concurrent read performance and is the recommended setting for most SQLite applications.
Defining the Database Model for Agent State
Unlike an ORM that maps class properties to columns, better-sqlite3 works directly with plain SQL and plain JavaScript objects. Two TypeScript interfaces describe the shapes involved:
// The rich application type used everywhere in the agent coreexport interface State { id: string; steps: number; status: AgentStatus; context: ContextItem[]; pending_tool_calls: PendingToolCall[]; error: string | null; final_answer: string | null;}// The flat, serialized shape that maps one-to-one with a SQLite rowinterface StateRow { id: string; steps: number; status: string; context: string; // JSON-serialized ContextItem[] pending_tool_calls: string; // JSON-serialized PendingToolCall[] error: string | null; final_answer: string | null;}
State is the live, typed application model; the context property holds a proper array of ContextItem objects and pending_tool_calls holds an array of PendingToolCall objects. StateRow mirrors the actual columns of the states table. Here, context and pending_tool_calls are plain string fields because SQLite has no native array type; therefore, those arrays are stored as JSON text. The status column is also a plain string at the row level; validation happens when you read the value back out, rather than at the storage layer. Keeping these two interfaces separate makes the boundary between 'what the agent works with' and 'what the database stores' explicit and easy to reason about. The status field uses the AgentStatus type.
Creating Database Tables
With the connection open, the StateStore constructor calls createTables() to ensure the states table exists before any reads or writes occur:
private createTables(): void { this.db .prepare( `CREATE TABLE IF NOT EXISTS states ( id TEXT PRIMARY KEY, steps INTEGER NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT 'running', context TEXT NOT NULL DEFAULT '[]', pending_tool_calls TEXT NOT NULL DEFAULT '[]', error TEXT, final_answer TEXT )` ) .run();}
The CREATE TABLE IF NOT EXISTS clause makes this safe to run on every server startup — SQLite simply skips the statement if the table is already there. Each column maps directly to a field in StateRow: id is the TEXT PRIMARY KEY that uniquely identifies each agent run, steps is an INTEGER, and status is TEXT constrained in your application code rather than at the database level. The context and pending_tool_calls columns use TEXT NOT NULL DEFAULT '[]' because they will always hold a JSON array string, never a raw NULL. The error and final_answer columns omit NOT NULL because they are genuinely absent until the agent either fails or completes.
Converting State to a Database Row
Before writing a State to SQLite, you must flatten it into a StateRow. The stateToRow helper handles this:
The most notable lines are the two JSON.stringify calls. Because state.context is an array of ContextItem objects — union-typed values that SQLite cannot store natively — JSON.stringify serializes it into a text string. The same applies to state.pending_tool_calls. All other fields are already primitive types (string, number, or null) that SQLite can store directly. This function is called inside StateStore.save() immediately before the prepared statement runs, keeping the serialization logic in one place and out of the SQL itself.
Converting a Database Row Back to State
Reading a row back out of SQLite gives you a StateRow, which must be inflated into a typed State. Three helpers work together to do this:
function rowToState(row: StateRow): State { return { id: row.id, steps: row.steps, status: parseStatus(row.status), context: parseJsonArray<ContextItem>(row.context), pending_tool_calls: parseJsonArray<PendingToolCall>(row.pending_tool_calls), error: row.error, final_answer: row.final_answer };}function parseStatus(value: string): AgentStatus { if ((AGENT_STATUSES as readonly string[]).includes(value)) { return value as AgentStatus; } throw new Error(`Unknown state status: ${value}`);}function parseJsonArray<T>(raw: string): T[] { const parsed = JSON.parse(raw) as unknown; if (Array.isArray(parsed)) { return parsed as T[]; } return [];}
rowToState is the mirror image of stateToRow; it calls parseStatus for the status column and parseJsonArray for the two JSON text columns. parseStatus validates the raw string against AGENT_STATUSES — the shared runtime array declared in state.ts — and throws an error rather than silently accepting an unknown value. This is genuine runtime validation: if a row in the database holds an unexpected status string, a runtime error occurs during retrieval rather than allowing a type-unsound value to drift through the system.
parseJsonArray, however, is weaker than it might first appear, and it is worth being precise about what it does and does not guarantee. It calls JSON.parse, then performs a real runtime check — Array.isArray(parsed) — to confirm the value is an array (returning [] as a safe fallback if it is not). But the final parsed as T[] is only a compile-timeTypeScript cast: it tells the compiler to treat the elements as ContextItem or PendingToolCall, without verifying at runtime that each element actually has the correct shape. If a malformed object were stored in the JSON column, it would pass through this function unchecked.
In other words, parseJsonArray guarantees you get an array — not that every element is a valid ContextItem or PendingToolCall. This is acceptable here because the only writer to these columns is your own stateToRow function, so the data is trusted. If you ever needed stronger guarantees — for instance, when accepting data from an untrusted source — you would validate each element's shape explicitly (for example, with a schema validator) rather than relying on the cast.
Adding Progress Callbacks to the Agent
The agent's run method is designed as a reducer: it takes a State as input and returns a State as output. To enable incremental persistence, it also accepts an optional ProgressCallback that is called after every step:
type ProgressCallback = (state: State) => void | Promise<void>;async run(inputState: State, progressCallback?: ProgressCallback): Promise<State> { let state = structuredClone(inputState); state.status = "running"; state.error = null; const maxStepsAllowed = state.steps > 0 ? state.steps + this.maxSteps : this.maxSteps; try { while (state.status === "running" && state.steps < maxStepsAllowed) { state = await this.nextStep(state); if (progressCallback) { await progressCallback(state); // Save after every step } } if (state.status === "running" && state.steps >= maxStepsAllowed) { state.status = "max_steps_reached"; } return state; } catch (error) { state.status = "failed"; state.error = error instanceof Error ? error.message : String(error); state.pending_tool_calls = []; if (progressCallback) { await progressCallback(state); // Save the failed state too } return state; }}
structuredClone(inputState) creates a deep copy of the incoming state so the agent never mutates the caller's object — which is important when the same state object is reused across multiple runs. The ProgressCallback type allows the callback to be either synchronous or asynchronous, and each call is awaited so the agent never moves on to the next step before the previous state has been persisted. The callback is also called inside the catch block, ensuring that failed states are saved with their error messages. Notice the maxStepsAllowed calculation: when the state already has completed steps (a resume scenario), the agent receives a full additional budget of this.maxSteps steps on top of the existing count, rather than being cut off immediately.
Creating the Progress Callback
With the callback mechanism in place, the actual callback used in the server is straightforward:
There is no closure factory, session management, or explicit transaction handling. store.save(state) uses an INSERT ... ON CONFLICT(id) DO UPDATE SET ...SQL statement under the hood, which means it behaves as an upsert: the first call inserts the row, and every subsequent call for the same id updates all fields in place. After each agent step, the database row reflects the latest step count, status, context, and pending_tool_calls. A client polling the state endpoint will see progress accumulate in real time because every write goes straight to disk — better-sqlite3 operates synchronously and commits each statement immediately by default, so there is no buffering to worry about.
How Route Errors Become JSON Responses
The endpoints you are about to build lean on three small helpers — HttpError, a global error-handling middleware, and asyncHandler. You already met the first two in Unit 1, where they turned any error thrown inside a synchronous route into a consistent JSON response. This unit adds the third, asyncHandler, and it is worth understanding why it is now necessary.
In Unit 1, an error thrown inside a synchronous handler was automatically caught by Express and forwarded to the error-handling middleware. But every handler in this server is now async, because each one has to await a database or agent call. When an async handler throws, the error becomes a rejected promise that Express does not observe by default — so the rejection goes unhandled and the client's request simply hangs. asyncHandler fixes this by wrapping an async handler and attaching a .catch(next), forwarding any rejection to Express's next function so it reaches the error middleware:
HttpError is a small Error subclass that carries an HTTP status code alongside its message, so a handler can express both "what went wrong" and "which status the client should get":
Finally, a single global error-handling middleware — registered after all the routes — inspects whatever error reached it and formats a JSON response. If the error is an HttpError, it uses that status; otherwise it falls back to 500:
The payoff is that a route can simply throw new HttpError(404, "State not found") and trust that the client receives { "detail": "State not found" } with status 404 — exactly as in Unit 1. The difference now is that this guarantee extends to async handlers too: with asyncHandler in place, every failure — validation errors from requireStringField, missing records, or unexpected exceptions escaping an async route — flows through the same one place and comes back as predictable JSON.
Updating the Launch Endpoint for Database Persistence
The launch endpoint builds an initial State, persists it to the database, and then fires off background execution before returning:
requireStringField validates that request.body is an object and that input_prompt is a non-empty string, throwing an HttpError(400, ...) with a descriptive message if either check fails — which, thanks to the asyncHandler wrapper and global middleware described above, becomes a clean 400JSON response. Once you have the prompt, the initial State is constructed with a fresh UUID and a single user message in context. store.save(initialState) writes this state to SQLite immediately — before background execution begins — so the record exists in the database even if the process crashes during the first agent step. void runAgentInBackground(initialState.id) starts the asynchronous background function without awaiting it, allowing the endpoint to respond immediately with the initial state while the agent runs in the background.
Implementing the Background Task with Progress Callbacks
The background function loads the state from the database, runs the agent with a save callback, and persists both the intermediate and final states:
The function starts by calling store.get(stateId) to load the persisted state from SQLite. This load-from-database pattern is important: because the state was written before runAgentInBackground was called, this is a confirmed disk read rather than a reliance on an in-memory reference. If no record exists (which would be unusual), the function returns early without running the agent. The agent runs with the inline callback (state) => { store.save(state); }, so every completed step is immediately durable. After agent.run resolves, store.save(finalState) performs one more upsert to guarantee that the terminal state (including the final status and answer) is recorded. The catch block handles unexpected errors that escape the agent by loading the latest saved state, marking it as "failed", and persisting it with a cleared pending_tool_calls array so clients see a clean terminal state.
Updating the State Retrieval Endpoint
The state retrieval endpoint reads a single record from StateStore and returns it as JSON:
app.get( "/agent/state/:stateId", asyncHandler(async (request, response) => { const stateId = requireRouteParam(request.params.stateId, "stateId"); const state = store.get(stateId); if (!state) { throw new HttpError(404, "State not found"); } response.json(state); }));
requireRouteParam validates that :stateId from the URL is a non-empty string before using it. store.get(stateId) runs a SELECT * FROM states WHERE id = ? prepared statement and returns either a fully parsed State object or null if no matching row exists. When the result is null, throwing new HttpError(404, "State not found") propagates the error to the global Express error handler, which — as described earlier — responds with { "detail": "State not found" } and status code 404. If the record is found, response.json(state) serializes the State object — including the context array and pending_tool_calls array that rowToState already parsed from JSON — and sends it to the client. Each polling request made by the client hits this endpoint, reads fresh data from disk, and returns it; consequently, clients always see the latest durably saved progress.
Running the Test Script with Real-Time Progress
When you run the test script against the updated server, you will see incremental progress printed on each polling iteration as the agent works through the problem:
The test script posts to /agent/launch with { "input_prompt": "..." }, parses the returned State to get the id, and then loops: fetching /agent/state/${state.id} every second, printing the current status, steps, and pending_tool_calls, and breaking when the status is one of "complete", "max_steps_reached", or "failed". The multiple lines with the same step count reflect polling during the interval between steps, when the agent is waiting for an LLM response. Each line represents a genuine read from SQLite, confirming that state is being saved to disk after every step. If you restart your server while the agent is running, background execution will stop because the Node.js process ends — but the data on disk remains intact. Polling again after a restart with the same state id returns the last saved step exactly where it left off.
Viewing the Complete Final State
After the agent finishes, the test script prints the full state object:
Final State:{ "id": "88b0fafd-b061-46ab-9ec4-cff14250bc1a", "steps": 6, "status": "complete", "context": [ { "role": "user", "content": "Solve the root of this equation: x^2 - 5x + 6 = 0" }, { "type": "function_call", "name": "subtract_numbers", "arguments": "{\"a\":25,\"b\":24}", "call_id": "call_ZIVW8HErxVmuEi8ZqEqdFABk" }, { "type": "function_call_output", "call_id": "call_ZIVW8HErxVmuEi8ZqEqdFABk", "output": "{\"result\":1}" }, ... { "type": "function_call", "name": "final_answer", "arguments": "{\"answer\":\"The equation x^2 - 5x + 6 = 0 has roots x = 2 and x = 3.\"}", "call_id": "call_FWy2PLhqHFKpdqQjZpUTLzcZ" } ], "pending_tool_calls": [], "error": null, "final_answer": "The equation x^2 - 5x + 6 = 0 has roots x = 2 and x = 3."}
The output shows that the agent solved the quadratic equation in six steps, with the context array containing a complete trace of every function call and its result. This full state is stored in data/agent_states.db and can be retrieved at any time by calling the state endpoint with the same id, even after the server has been restarted.
Summary and Next Steps
You have transformed your Express API from using ephemeral in-memory storage to a durable persistence layer built on better-sqlite3 and the StateStore class, which now lives in its own src/server/database.ts module. States are written to disk at three moments: when an agent is first launched, after every step via the progress callback, and once more when the agent reaches a terminal status. The stateToRow and rowToState helpers form a clean serialization boundary, converting typed State objects to flat SQLite rows and back, with parseStatus performing genuine runtime validation of the status string and parseJsonArray guaranteeing an array shape (while trusting your own writer for the element types) during retrieval. Alongside persistence, you added asyncHandler so that errors thrown in async routes reach the same HttpError and global error middleware you carried over from Unit 1, keeping every failure a predictable JSON response. While the execution model remains a lightweight background task — if the server stops, the task also stops — the state data on disk is always consistent with the last completed step. This persistent foundation is exactly what makes Factor 6 — Launch / Pause / Resume with simple APIs viable: without durable state, pausing and resuming would be meaningless. In the next lesson, you will build on this foundation to implement the pause and resume endpoints that complete Factor 6.
import Database from "better-sqlite3";import { mkdirSync } from "node:fs";import { dirname, join } from "node:path";import { fileURLToPath } from "node:url";function defaultDatabasePath(): string { const currentDirectory = dirname(fileURLToPath(import.meta.url)); return join(currentDirectory, "../../data/agent_states.db");}export class StateStore { private readonly db: Database.Database; constructor(dbPath = defaultDatabasePath()) { mkdirSync(dirname(dbPath), { recursive: true }); this.db = new Database(dbPath); this.db.pragma("journal_mode = WAL"); this.createTables(); }}
TypeScript
// The rich application type used everywhere in the agent coreexport interface State { id: string; steps: number; status: AgentStatus; context: ContextItem[]; pending_tool_calls: PendingToolCall[]; error: string | null; final_answer: string | null;}// The flat, serialized shape that maps one-to-one with a SQLite rowinterface StateRow { id: string; steps: number; status: string; context: string; // JSON-serialized ContextItem[] pending_tool_calls: string; // JSON-serialized PendingToolCall[] error: string | null; final_answer: string | null;}
TypeScript
private createTables(): void { this.db .prepare( `CREATE TABLE IF NOT EXISTS states ( id TEXT PRIMARY KEY, steps INTEGER NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT 'running', context TEXT NOT NULL DEFAULT '[]', pending_tool_calls TEXT NOT NULL DEFAULT '[]', error TEXT, final_answer TEXT )` ) .run();}