Introduction

This is a course, not a manual. By the end you will have built panoptes-control — the coordinator that runs the EXECUTE stage of the Panoptes harness as a small distributed system — and, more importantly, you will understand why every piece is shaped the way it is. We are going to move the way a good pair-programming session moves: I frame a concept, you ask questions until it is solid, then we build the piece together and the compiler grades the work.

What we are building

panoptes-control is a control plane. A client submits an eval run — a manifest of vignettes, a list of models, a number of epochs — and the coordinator does four things:

1. Accepts the submission over an HTTP API and answers immediately.
2. Persists it as a queued run, durably, so a crash loses nothing.
3. Splits it into jobs — one per model × epoch — the unit that fans out.
4. Dispatches those jobs across a pool of workers, local first and networked later, collecting every model response back into the store.

Stages upstream (generating the vignettes) and downstream (coding responses, statistics) are other people's problems — some of them the Python side, some of them Course 1's harness. We are building the engine that takes a submitted run and executes it to completion across machines, correctly, even when machines fail.

The end state, held in view

Everything we build serves one payoff you should keep in mind from the very first chapter: under worker failure, no eval is lost and none is double-counted.

Those two properties pull in opposite directions, and that tension is the whole course. To lose nothing when a worker dies mid-job, the coordinator must be willing to hand that job to another worker — at-least-once delivery. But at-least-once means a job can genuinely run twice: the first worker was slow, not dead, and both results come back. The escape is idempotent recording — the store is written so that recording the same job's outcome twice is indistinguishable from recording it once. At-least-once delivery made safe by idempotent recording: that is the sentence the last arc pays off, and it is worth holding from here.

The architectural spine

There is one structural idea that makes the distributed half of this system additive rather than a rewrite, and you will meet it in the very next concept chapter: a single trait, WorkerHandle, behind which the scheduler dispatches every job.

The scheduler holds a pool of dyn WorkerHandle and calls dispatch(job). It never learns whether the handle on the other side runs the eval in an in-process task (a LocalWorker) or ships it over a TCP socket to another machine (a RemoteWorker). Because the scheduler depends only on the trait, the entire networked layer — Parts VI through VIII — arrives as a new implementation of an existing seam, not a change to the code that uses it. Get that seam right in Part II and the payoff arc has somewhere to plug in.

How the arcs are sequenced

The order is chosen for understanding, not for delivery speed. Each arc teaches one core idea and hands the next arc something to stand on:

• Part II — The Core Arc. The load-bearing seam (WorkerHandle), the Run/Job domain, the id newtypes, the error taxonomy that knows what is worth retrying, and the wire protocol. Everything downstream points here.
• Part III — The Eval Workload Arc. The ModelClient trait object and run_eval, tested against a mock HTTP server, plus the append-only file contract that logs every raw response.
• Part IV — The Persistence Arc. sqlx and SQLite: schema, migrations, and the atomic claim + transactional recording that make crashes and concurrent schedulers safe.
• Part V — The Service Arc. axum: handlers, shared state, extractors, and turning the error taxonomy into HTTP status codes in one place.
• Part VI — The Scheduler Arc. Bounded concurrency, retrying only the retryable, the LocalWorker, and graceful shutdown — the coordinator becomes a running program.
• Part VII — The Telemetry Arc. tracing spans and a /stats endpoint that accounts for tokens and cost per model.
• Part VIII — The Cluster Arc. The payoff: a framed TCP protocol, a connection actor multiplexing one socket, the RemoteWorker behind the same trait, and at-least-once redelivery with heartbeat reaping made safe by the store's idempotency.

A note on where you are starting from

You come to this course strong in two areas and new in a third, and it helps to be honest about all three. The domain — LLM evaluation, models, prompts, epochs, token accounting — is familiar ground; you have built systems like the thing this coordinator runs. And Rust itself is no longer new: from Courses 1 and 2 you already own ownership and borrowing, serde, traits and generics, async/await and tokio basics, wiremock, clap, and the thiserror/anyhow split. This course does not re-teach any of that.

What is new is the shape of a service: code that persists state to a database, answers HTTP requests, schedules concurrent work, and talks to other processes over a socket. That is the whole frontier here — axum, sqlx, and a hand-rolled network protocol — and it is exactly where the concept chapters spend their time. If a networking or persistence chapter feels hard, that is not a signal about your ability; it is a new class of failure (a dropped connection, a partial write, a race between two schedulers) that the type system is about to teach you to make impossible.

Turn the page for the architecture at a glance, then we start with Phase 0 and the seam that holds the whole thing together.