Concept: The Seam — dyn Trait Objects as Dependency Inversion

Kind: Concept (read, do not code). This is the chapter the whole distributed design rests on.

Here is the promise the introduction made: the networked half of this system will arrive as an addition, not a rewrite. Parts II through VII build a coordinator that dispatches every eval job to an in-process worker. Then Part VIII adds workers on other machines — and the scheduler, the code that actually hands out jobs, does not change by a single line. That is not luck or discipline. It is a structural property you install deliberately, in Part II, with one trait. This chapter is about why that works, before we build it.

You already know trait objects from Courses 1 and 2 — Box<dyn Trait>, dynamic dispatch, the vtable. So this chapter is not "what is a trait object." It is a sharper claim: a trait object placed at the right boundary is dependency inversion, and dependency inversion is precisely the tool that makes "local now, networked later" an additive change. We are going to make that claim concrete, argue why it holds, and only then map it onto the real seam.

The problem, stated as a dependency arrow

Think about who-depends-on-whom, because that is the whole game. The scheduler's job is to take a queued run, split it into jobs, and get each job executed somewhere. The naive way to write that is for the scheduler to know how execution happens:

• Version 1 (local): the scheduler calls run_eval(job) directly, in-process. Simple. But now the scheduler depends on the concrete way jobs run.
• Version 2 (networked): we want some jobs to run on other machines. If the scheduler called run_eval directly, we now have to go back into the scheduler and teach it about sockets, framing, retries, and reconnection. The scheduler — the piece we had working — gets rewritten to accommodate a lower-level detail. Every time execution changes, the scheduler changes.

That is a dependency pointing the wrong way: a high-level policy (how to schedule and retry a run) depending on a low-level mechanism (how one job physically executes). Dependency inversion is the fix, and its name is literal — you invert that arrow. Instead of the scheduler depending on a concrete worker, both the scheduler and every concrete worker depend on an abstraction in between: a trait.

The seam, in a toy that mirrors the real one

Let us build the smallest possible version of exactly this shape, in a domain with no networking to distract us. We want to notify someone that something happened. Some notifiers are in-process (append to a local log); some are remote (ship the message over the network). The code that decides what to notify about must not care how any notifier delivers.

The abstraction in the middle is a trait with one behavior:

#![allow(unused)]
fn main() {
/// The seam. `broadcast` will depend only on this.
trait Notifier {
    fn id(&self) -> &str;
    fn notify(&self, msg: &str) -> Result<(), String>;
}
}

Now two concrete implementations. The first is fully in-process — it does its whole job by writing locally, no network involved:

#![allow(unused)]
fn main() {
/// In-process: writes to the local log. No network at all.
struct LogNotifier {
    id: String,
}

impl Notifier for LogNotifier {
    fn id(&self) -> &str {
        &self.id
    }
    fn notify(&self, msg: &str) -> Result<(), String> {
        println!("[{}] {msg}", self.id);
        Ok(())
    }
}
}

The second stands in for the far side — the version that would open a socket and ship bytes to another machine. We are not writing that socket code here; the point of the toy is that the code holding the notifiers cannot tell the difference:

#![allow(unused)]
fn main() {
/// A sketch of the far side: this one would ship the message over a socket.
/// Here we only stand in for that — the point is that `broadcast` cannot tell.
struct RemoteNotifier {
    id: String,
    endpoint: String,
}

impl Notifier for RemoteNotifier {
    fn id(&self) -> &str {
        &self.id
    }
    fn notify(&self, msg: &str) -> Result<(), String> {
        // The real impl writes bytes to `self.endpoint`; we fake the send.
        println!("POST {} <- {msg}", self.endpoint);
        Ok(())
    }
}
}

And here is the payoff — the piece that is the high-level policy. It holds a pile of notifiers behind Box<dyn Notifier> and drives them. Read its type signature closely: &[Box<dyn Notifier>]. There is no LogNotifier, no RemoteNotifier, no enum of "kinds" anywhere in it. It literally cannot name the concrete types, so it cannot branch on them:

/// The payoff: this function holds a pile of notifiers and never learns which
/// kind any of them is. Add a third impl tomorrow — this code does not change.
fn broadcast(notifiers: &[Box<dyn Notifier>], msg: &str) {
    for n in notifiers {
        if let Err(e) = n.notify(msg) {
            println!("{} failed: {e}", n.id());
        }
    }
}

fn main() {
    let notifiers: Vec<Box<dyn Notifier>> = vec![
        Box::new(LogNotifier { id: "log-1".into() }),
        Box::new(RemoteNotifier {
            id: "remote-1".into(),
            endpoint: "10.0.0.5:9000".into(),
        }),
    ];
    broadcast(&notifiers, "run 7 started");
}

Predict the output before you run it — two lines, and which impl produces each.

[log-1] run 7 started
POST 10.0.0.5:9000 <- run 7 started

Two different mechanisms — a local log write and a (pretended) network send — driven by one loop that knows about neither. broadcast depends only on Notifier. LogNotifier and RemoteNotifier depend on Notifier. The arrow is inverted, and the consequence is the whole reason we are here: to add a third kind of notifier, you write a new impl Notifier and drop it in the vec. You do not touch broadcast. That is "additive, not a rewrite," in fourteen lines.

Why the dependency has to be a trait object here

You might ask: we know generics, why not broadcast<N: Notifier>(notifiers: &[N])? Because that would demand every element be the same concrete type N. Our vec is deliberately heterogeneous — a LogNotifier and a RemoteNotifier side by side — and the whole point is to hold a mix and decide the mix at runtime, when we discover which workers have actually connected. A generic is resolved at compile time to one type; dyn is resolved at runtime, per element, through the vtable. Distribution is inherently a runtime fact — you do not know at compile time how many workers will connect or of which kind — so the seam must be a trait object, not a generic. This is the case where dyn is not a stylistic choice but the only tool that fits.

The real seam is the same shape, asynchronously

The toy is synchronous so it runs anywhere. The real seam is identical in structure but its one method is async, because dispatching a job means awaiting a model call or a network round-trip. An async method in a trait you intend to use as dyn still, in this course's toolchain, wants the #[async_trait] macro — you used it in Course 2 — because it rewrites async fn into a method returning a boxed future, which is something a vtable can hold.

Here is the toy rewritten in the exact shape of the real WorkerHandle. It will not run on the playground (it needs the async-trait and tokio crates), so it is marked ignore:

// scratch Cargo.toml deps:
//   async-trait = "0.1"
//   tokio = { version = "1", features = ["full"] }
use async_trait::async_trait;
use std::sync::Arc;

#[async_trait]
trait Notifier: Send + Sync {
    fn id(&self) -> &str;
    async fn notify(&self, msg: &str) -> Result<(), String>;
}

struct LogNotifier {
    id: String,
}

#[async_trait]
impl Notifier for LogNotifier {
    fn id(&self) -> &str {
        &self.id
    }
    async fn notify(&self, msg: &str) -> Result<(), String> {
        println!("[{}] {msg}", self.id);
        Ok(())
    }
}

struct RemoteNotifier {
    id: String,
    endpoint: String,
}

#[async_trait]
impl Notifier for RemoteNotifier {
    fn id(&self) -> &str {
        &self.id
    }
    async fn notify(&self, msg: &str) -> Result<(), String> {
        // Real impl: open a socket to `self.endpoint`, write the frame, await ack.
        println!("POST {} <- {msg}", self.endpoint);
        Ok(())
    }
}

// The policy is unchanged in shape — it just `.await`s each call. Note `Arc`
// instead of `Box`: workers are shared across concurrent dispatch tasks.
async fn broadcast(notifiers: &[Arc<dyn Notifier>], msg: &str) {
    for n in notifiers {
        if let Err(e) = n.notify(msg).await {
            println!("{} failed: {e}", n.id());
        }
    }
}

Two details that carry straight into the real seam. First, the trait now requires Send + Sync: a dyn WorkerHandle will be moved between and shared across tokio tasks running on different threads, so the compiler must know it is safe to send and share. Second, the container is Arc<dyn Notifier>, not Box: a Box is a single owner, but a worker handle is dispatched to from several concurrent tasks at once, so it must be shared ownership. Those are the only differences between the toy and the production seam. The inversion — policy depends on trait, mechanisms depend on trait — is bit-for-bit the same.

One-for-one: the toy ↔ the real thing

Everything above maps onto the actual control-core seam you will build in Part II, one piece to one piece:

• Notifier (the trait, the seam) ↔ WorkerHandle — the trait the scheduler dispatches through. Its real method is async fn dispatch(&self, job: Job) -> Result<JobOutcome, ControlError>.
• LogNotifier (in-process, no network) ↔ LocalWorker — a WorkerHandle that runs the eval in an in-process task. This is all the coordinator uses through Part VII.
• RemoteNotifier (the far-side sketch) ↔ RemoteWorker — a WorkerHandle that ships the job over a TCP socket to a worker process. This is the entire new surface Part VIII adds.
• broadcast(&[Box<dyn Notifier>]) (holds a mix, knows no concrete kind) ↔ the scheduler — it holds Vec<Arc<dyn WorkerHandle>> and calls dispatch, never learning local from remote.
• notify taking &str, not a generic ↔ dispatch taking a concrete Job — the seam's methods stay object-safe on purpose, so the trait can be dyn.

Hold that last row against the trap above: the day the compiler tells you a trait "is not dyn compatible," it is enforcing exactly the discipline that keeps this seam usable. And hold the mapping as a whole against the promise this chapter opened with. When Part VIII adds RemoteWorker, it adds a new impl WorkerHandle and a pool that hands the scheduler Arc<dyn WorkerHandle> values — and the scheduler, like broadcast, does not change. The distributed arc is additive because the arrow was inverted here, at the start.

Questions to lock

Genuinely stop on each. This is the foundation the last five arcs stand on.

1. What does "invert the dependency" mean concretely, in terms of which direction the arrow points before and after? Why does the inverted arrow make a new worker kind additive?
2. Why must the worker pool be dyn WorkerHandle rather than a generic Vec<W>? What runtime fact about distribution forces that choice?
3. Why does the real seam add Send + Sync and use Arc instead of Box? What would go wrong without each?
4. What makes a trait not object-safe (dyn-compatible), and what error do you get if you try to form &dyn of it?

Next: Part II, where we build this seam for real — the WorkerHandle trait, the domain it dispatches, and the error taxonomy that tells the scheduler which failures are worth retrying.