Concept: Trait-Object Clients and Mocking the Network

Kind: Concept. New crate: async-trait — lets an async fn live inside a trait that you can still use behind dyn.

The whole workload is a network round-trip

Step back and look at what a job in this system actually does. It takes a batch of vignettes, and for each one it POSTs a prompt to a model endpoint and waits for the reply. That is the entire workload. Strip away the scheduling, the storage, the telemetry, and what remains is a loop around one HTTP call.

That single fact drives most of the architecture that follows. Because the work is a network round-trip — almost all of it spent waiting on a remote server — a worker is barely using its CPU while a job runs. One machine could keep hundreds of these in flight at once, and the work parallelizes trivially across machines because each call is independent. The workload is worth distributing precisely because it is I/O, not computation. Hold onto that; it is why Parts VI and VIII exist.

For this arc, the consequence is narrower: the one thing a job depends on is a way to turn a prompt into a response over the network. Name that dependency, put it behind a seam, and everything downstream — run_eval, the local worker, the remote worker — can be written and tested without ever touching a real model.

The seam: ModelClient

You already know the trait-object seam from Part I: depend on a trait, not a concrete type, so the concrete type can be swapped. Here the trait is the model call itself:

#[async_trait]
pub trait ModelClient: Send + Sync {
    fn model_name(&self) -> &str;
    async fn generate(&self, prompt: &str) -> Result<ModelResponse, ControlError>;
}

Two methods, and only two. model_name reports the pinned model string; generate takes the entire prompt (single-turn — the prompt is the whole input) and returns a ModelResponse or a ControlError. run_eval will accept &dyn ModelClient and never know whether it is talking to a real HTTP client or a stub. That is the whole payoff: the workload logic is provider-agnostic and testable, and the transport is a detail chosen at the edge.

The Send + Sync bound is not decoration. A future produced by generate will be moved onto a runtime and, later in the course, handed between worker tasks on different threads. Send + Sync is the compiler's promise that doing so is safe. You met these bounds in the async arc; here is where they earn their keep.

Why async fn in a trait needs #[async_trait]

Try the obvious thing — a bare async fn in the trait, used behind dyn — and the compiler stops you cold:

#![allow(unused)]
fn main() {
trait QuoteClient {
    async fn quote(&self, topic: &str) -> String;
}

fn take_seam(_c: &dyn QuoteClient) {}
}

error[E0038]: the trait `QuoteClient` is not dyn compatible
 --> src/lib.rs:6:19
  |
6 | fn take_seam(_c: &dyn QuoteClient) {}
  |                   ^^^^^^^^^^^^^^^ `QuoteClient` is not dyn compatible
  |
note: for a trait to be dyn compatible it needs to allow building a vtable
 --> src/lib.rs:3:14
  |
2 | trait QuoteClient {
  |       ----------- this trait is not dyn compatible...
3 |     async fn quote(&self, topic: &str) -> String;
  |              ^^^^^ ...because method `quote` is `async`

Read the reason on the last line: the method is async. An async fn desugars to a function returning impl Future, and every call site can produce a different concrete future type of a different size. A dyn object needs one fixed vtable with one fixed return layout — and "some future, size unknown" is not that. So the trait is not dyn compatible, and &dyn QuoteClient will not compile.

#[async_trait] is the fix. It rewrites each async fn in the trait to return Pin<Box<dyn Future + Send>> — a heap-allocated, fixed-size future behind a pointer. Now every method has one uniform return type, the vtable can be built, and &dyn ModelClient works. The cost is one allocation per call, which against a network round-trip is free. (Rust is steadily lifting the language-level restriction, but for a dyn-dispatched trait like this one, #[async_trait] is still the standard tool.)

A server error is a retryable Worker error

Recall the taxonomy from Part II: ControlError splits into retryable (Worker, Protocol) and terminal (Invalid, NotFound). The model call is where that distinction gets its first real workout, and the rule is deliberate:

• The HTTP call fails to complete, or the server answers with a non-2xx status → ControlError::Worker, which is_retryable() reports as true. A 503 means the model backend hiccuped; the same request to another worker may well succeed. That is exactly what retryable means.
• The server answers 2xx but the body is not the shape we expect → ControlError::Invalid, terminal. A malformed reply will be malformed no matter who asks; retrying only wastes time.

That mapping is a policy decision, and it is worth making it concrete before you wire it to HTTP. Here it is on a toy error type — a non-2xx becomes a retryable server fault, an unparseable body becomes a terminal one:

/// A toy of the harness's error taxonomy: retryable vs terminal.
enum QuoteError {
    /// The server answered, but with a 5xx — a transient fault. Retryable.
    Server(String),
    /// The reply body was not the shape we expected. Terminal.
    Bad(String),
}

impl QuoteError {
    fn is_retryable(&self) -> bool {
        matches!(self, QuoteError::Server(_))
    }
    fn message(&self) -> &str {
        match self {
            QuoteError::Server(m) | QuoteError::Bad(m) => m,
        }
    }
}

/// Turn a raw HTTP outcome into our error taxonomy, exactly as `quote` would:
/// a non-2xx status is a transient Server fault; a body we cannot parse is Bad.
fn classify(status: u16, parses: bool) -> Result<&'static str, QuoteError> {
    if !(200..300).contains(&status) {
        return Err(QuoteError::Server(format!("quote status {status}")));
    }
    if !parses {
        return Err(QuoteError::Bad("missing `text` field".into()));
    }
    Ok("a witty maxim")
}

fn main() {
    for (status, parses) in [(200, true), (503, true), (200, false)] {
        match classify(status, parses) {
            Ok(text) => println!("{status} parses={parses:<5} -> Ok({text:?})"),
            Err(e) => println!(
                "{status} parses={parses:<5} -> Err({:?})  retryable={}",
                e.message(),
                e.is_retryable()
            ),
        }
    }
}

200 parses=true  -> Ok("a witty maxim")
503 parses=true  -> Err("quote status 503")  retryable=true
200 parses=false -> Err("missing `text` field")  retryable=false

The toy, end to end: a QuoteClient

Here is the exact shape the build asks of you, on a service that cannot be mistaken for the answer key: a quote API. POST {base}/quote with a JSON body; the reply is { "text": ..., "credits": ... }; the client parses it into a clean public type and maps failures into the taxonomy above. It is behind an #[async_trait] trait, so a stub and the real HTTP client are interchangeable.

use async_trait::async_trait;
use serde::{Deserialize, Serialize};

/// A witticism plus the credits the call cost. Flat, typed, ours.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Quote {
    pub text: String,
    pub credits: u32,
}

/// The seam: anything that turns a topic into a quote. Mockable.
#[async_trait]
pub trait QuoteClient: Send + Sync {
    fn source_name(&self) -> &str;
    async fn quote(&self, topic: &str) -> Result<Quote, QuoteError>;
}

/// A concrete client over HTTP with an injectable base URL, so a test can point
/// it at a mock instead of a real provider.
pub struct HttpQuoteClient {
    base_url: String,
    source: String,
    http: reqwest::Client,
}

impl HttpQuoteClient {
    pub fn new(base_url: impl Into<String>, source: impl Into<String>) -> Self {
        Self { base_url: base_url.into(), source: source.into(), http: reqwest::Client::new() }
    }
}

#[derive(Serialize)]
struct QuoteRequest<'a> { source: &'a str, topic: &'a str }

#[derive(Deserialize)]
struct QuoteReply { text: String, credits: u32 }

#[async_trait]
impl QuoteClient for HttpQuoteClient {
    fn source_name(&self) -> &str {
        &self.source
    }

    async fn quote(&self, topic: &str) -> Result<Quote, QuoteError> {
        let resp = self.http
            .post(format!("{}/quote", self.base_url.trim_end_matches('/')))
            .json(&QuoteRequest { source: &self.source, topic })
            .send().await
            // A dropped or refused connection is worth another attempt.
            .map_err(|e| QuoteError::Server(e.to_string()))?;
        if !resp.status().is_success() {
            // A non-2xx is a transient server fault → retryable.
            return Err(QuoteError::Server(format!("quote status {}", resp.status())));
        }
        // A 2xx we cannot parse is terminal → Bad.
        let reply: QuoteReply = resp.json().await
            .map_err(|e| QuoteError::Bad(e.to_string()))?;
        Ok(Quote { text: reply.text, credits: reply.credits })
    }
}

The Raw* split is the same trick from the mocking chapter: QuoteReply models only the fields we consume, and serde ignores the rest. Quote is the flat, owned type callers actually want.

Mock it — briefly, since you know the drill

You met wiremock in Course 1: it starts a real HTTP server on a random local port and hands you server.uri(), which you pass in as the client's base_url. The client makes a genuine request; the server returns exactly the canned response you configured. No key, no spend, no flake, and the full request-building and response-parsing path runs for real. The only new wrinkle here is that the thing under test is an #[async_trait] implementation.

#[cfg(test)]
mod tests {
    use super::*;
    use wiremock::matchers::{method, path};
    use wiremock::{Mock, MockServer, ResponseTemplate};

    #[tokio::test]
    async fn quote_client_posts_topic_and_parses_reply() {
        let server = MockServer::start().await;
        Mock::given(method("POST"))
            .and(path("/quote"))
            .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({
                "text": "Fortune favors the bold.", "credits": 1
            })))
            .expect(1) // guard against the silent-404 trap
            .mount(&server)
            .await;

        let client = HttpQuoteClient::new(server.uri(), "seneca");
        let q = client.quote("courage").await.unwrap();
        assert_eq!(q, Quote { text: "Fortune favors the bold.".into(), credits: 1 });
    }

    #[tokio::test]
    async fn server_error_is_retryable() {
        let server = MockServer::start().await;
        Mock::given(method("POST"))
            .respond_with(ResponseTemplate::new(503))
            .mount(&server)
            .await;
        let client = HttpQuoteClient::new(server.uri(), "seneca");
        let err = client.quote("x").await.unwrap_err();
        assert!(err.is_retryable()); // the 503 became a retryable Server fault
    }
}

Both tests pass — one real HTTP round trip each, zero network beyond loopback. The second is the important one: it pins the policy, not just the plumbing. It asserts that a 503 from the endpoint surfaces as a retryable error, which is the promise the scheduler will rely on.

To run these two examples yourself, make a scratch crate with serde/serde_json, tokio (features ["full"]), async-trait, and reqwest (features ["json"]) as dependencies, and wiremock = "0.6" under [dev-dependencies] — the same set control-eval declares. async-trait, reqwest, and wiremock are not on the Rust playground, so there is no play button.

One-for-one with the build

The toy maps onto control-eval exactly:

• QuoteClient ↔ ModelClient (the #[async_trait] seam, Send + Sync)
• HttpQuoteClient ↔ HttpModelClient (injectable base_url, a reqwest::Client inside)
• POST /quote ↔ POST /v1/generate
• quote(topic) ↔ generate(prompt)
• QuoteRequest/QuoteReply ↔ the request/response structs serde maps to the wire
• Quote ↔ ModelResponse (flat, owned, { text, usage })
• QuoteError::Server on a non-2xx ↔ ControlError::Worker, retryable
• QuoteError::Bad on an unparseable body ↔ ControlError::Invalid, terminal

Same shape, different domain. Build the model client and you have built the quote client with the labels changed.

Questions to lock

1. Why is the eval workload described as "a network round-trip," and why does that make it worth distributing across workers?
2. What exact compiler error do you get from a bare async fn in a trait used behind dyn, and what does #[async_trait] change to fix it?
3. A model endpoint returns 503. Which ControlError variant should generate produce, is it retryable, and why is that the right call?
4. The mock test aims the client at server.uri() instead of a real provider. What design property of the client makes that possible, and where else does that same seam pay off?