Panoptes Control — Architecture

The story, end to end

It starts with a client POSTing an eval run to the coordinator's axum API. create_run reads three fields — a manifest path, a list of models, and an epochs count — validates them (at least one model, epochs at least 1), and builds a Run: the top-level sweep, with a fresh RunId, a RunStatus of Queued, and a job_count computed up front as models times epochs. insert_run persists it in the Store — the SQLite-backed persistence layer — and the handler returns 201 Created with the id. Nothing executes yet: the API's entire job is request into store. A run is a promise of work, durably recorded, waiting to be claimed.

The Scheduler is the coordinator's engine, and the hub the rest of the system turns around. On each tick it calls claim_next_run — the load-bearing operation of the store: a single UPDATE … RETURNING that atomically flips the oldest queued run to Running and hands it back, so two schedulers can never grab the same run. With a run in hand it loads the manifest — a JSONL file of Vignettes, each an id plus the exact prompt to pose — and plan_jobs splits the run into Jobs: one Job per model, per epoch (an epoch is a repeat of the whole sweep, run to measure a model's consistency), each carrying an EvalJob — the entire vignette batch bound to one model at one epoch. That EvalJob is the unit that fans out. The scheduler snapshots its worker pool, insert_jobs records them, and it dispatches across the pool with buffer_unordered — a bounded-concurrency stream, four jobs in flight at once — retrying any retryable failure on the next worker via run_job_with_retry.

Every worker in that pool is an Arc<dyn WorkerHandle>, and WorkerHandle is the seam the whole design hangs on: the scheduler holds only the trait and knows nothing about how a handle runs the work, which is what makes the network layer additive rather than a rewrite. A LocalWorker runs the job in-process — its dispatch calls run_eval, which poses each Vignette to a ModelClient (HttpModelClient POSTs to /v1/generate, outside the workspace) and collects one ResponseRecord per vignette: which vignette, which model and epoch, the prompt, the response text, and token Usage. A RemoteWorker implements the same trait but fronts a TCP connection to a separate panoptes-worker process. Over it a MessageStream ships length-delimited JSON Message frames — Register, Assign, Result, Heartbeat — so a raw byte stream regains message boundaries. A per-connection actor multiplexes many jobs over the one socket, matching returning Result frames to their waiters by job id; the worker's declared capacity becomes real backpressure.

What makes that seam safe under failure is at-least-once delivery paired with idempotent recording. When a connection dies, the actor fails every in-flight job with a retryable Worker error, and the scheduler redelivers each to a surviving worker — so a job may run more than once. That is fine because record_job_outcome is idempotent by job id: it writes the JobOutcome and bumps the run's done_count in one transaction, but only advances the count the first time a job lands (the UPDATE is guarded by status != 'done') — without which a worker that dies after finishing but before its Result is acked would double-count the run. As each outcome comes back the scheduler appends its ResponseRecords to an append-only responses.jsonl log via append_records, then records it; when the last job lands, the run flips to Done. Finally /stats reads back across every stored outcome, aggregating token usage per model through usage_by_model into ModelUsage rows and an estimated dollar cost — the sweep's final accounting.