Phase 0: Workspace, Toolchain, the Loop

Before any coordinator code, we verify three things: that your toolchain is current, that you understand the workspace-per-arc structure the whole course grows inside, and that you can write a failing test, see it fail legibly, fix it, and see it pass. The last one is deliberately trivial — that is the point. We are testing the machinery, not your ability.

The toolchain

This course targets Rust edition 2024, which needs a stable toolchain ≥ 1.96. If you finished Courses 1 and 2 you already have rustup; bring it current and confirm the versions:

rustup update
rustc --version   # want 1.96.0 or newer
cargo --version   # want 1.96.0 or newer

Edition 2024 is set per crate in each Cargo.toml (edition = "2024"), not globally. It is what lets us use the current async ergonomics and the newest lint defaults without ceremony. If rustc --version reports something older than 1.96, rustup update fixes it before you go further.

The workspace, one crate per concern

panoptes-control is not one crate — it is a Cargo workspace of five, and the split is the architecture. A workspace is a single repository, a single target/ build directory, a single cargo test, but several independently-compiled member crates with explicit dependencies between them. Drawing those dependencies as crate boundaries is what stops, at compile time, a lower layer from reaching up into a higher one.

The root Cargo.toml is a [workspace], not a package — it lists members and pins shared dependency versions once:

[workspace]
resolver = "2"
members = [
    "crates/control-core",
    "crates/control-eval",
    "crates/control-store",
    "crates/panoptes-control",
    "crates/panoptes-worker",
]

[workspace.dependencies]
serde = { version = "1", features = ["derive"] }
tokio = { version = "1", features = ["full"] }
# ... every shared version lives here; crates opt in with `{ workspace = true }`

The five crates, and what each one owns:

The dependency direction is the point. control-core knows about no one; everyone knows about control-core. That is what makes the WorkerHandle seam load-bearing: both the in-process LocalWorker (in panoptes-control) and the networked RemoteWorker implement a trait that lives in the crate at the bottom, so the scheduler can hold either without a dependency pointing the wrong way.

The no-infrastructure testing stance

A distributed system sounds like it needs infrastructure to test — a database server, a live model API, a second machine. This one needs none of that, and keeping it that way is a design constraint we hold from the first test to the last. Every arc runs from a plain cargo test:

• The store opens an in-memory SQLite database (sqlite::memory:). Each test gets a fresh, isolated schema in microseconds; nothing touches disk, and there is no server to start.
• The eval workload points its HTTP client at a wiremock mock server (you met this in Course 2) instead of a real model API — so responses are deterministic and no network leaves the machine.
• The cluster binds a real TCP listener to 127.0.0.1:0 — port zero, meaning "the OS picks a free port" — so a test spins up a genuine coordinator and worker over a loopback socket with no fixed port to collide.
• Time-dependent logic (heartbeats, timeouts) uses tokio::time::pause, so a test can fast-forward thirty seconds of "silence" instantly instead of actually waiting.

The payoff is that the feedback loop stays fast and every test is hermetic: it depends on nothing outside the process, so it cannot be flaky because of a busy port, a slow network, or leftover rows from a previous run.

The loop, on a workspace member

Let us confirm the machinery end to end. Suppose the very first control-core function is job_count — the split arithmetic, one job per model × epoch — with a test that asserts the wrong number on purpose:

#![allow(unused)]
fn main() {
pub fn job_count(models: u32, epochs: u32) -> u32 {
    models * epochs
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn one_job_per_model_times_epoch() {
        assert_eq!(job_count(2, 3), 5); // deliberately wrong
    }
}
}

Test one crate in isolation with -p:

cargo test -p control-core

running 1 test
test tests::one_job_per_model_times_epoch ... FAILED

failures:

---- tests::one_job_per_model_times_epoch stdout ----

thread 'tests::one_job_per_model_times_epoch' panicked at crates/control-core/src/lib.rs:11:9:
assertion `left == right` failed
  left: 6
 right: 5

Read that carefully, because you will read a hundred of these. It names the exact test, the exact file and line, and — critically — left: 6, right: 5: the computed value against the expected value. This is why we write assertions with the computed value on the left and the expected on the right; the output then reads naturally. Here the code is right and the test is wrong. Fix the expectation to 6:

#![allow(unused)]
fn main() {
        assert_eq!(job_count(2, 3), 6);
}

Run again:

running 1 test
test tests::one_job_per_model_times_epoch ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

That is the loop. -p control-core scopes the run to the crate you are editing; a bare cargo test runs the whole workspace and is what you do before every commit, to confirm no downstream crate broke:

cargo test          # the whole workspace, every crate

Done when

You have a current toolchain (cargo --version ≥ 1.96), you can name the five crates and which layer each occupies, and you can run cargo test -p <crate>, read a failure, and fix it without thinking about the mechanics. When that loop is automatic, turn the page: the next chapter is the one concept the entire distributed design rests on — the seam.