Build: ModelClient + run_eval

Maps to: Phase 1 (control-eval). Kind: Build.

Objective

Create the control-eval crate. Define the ModelClient seam (with #[async_trait]), one concrete HttpModelClient over reqwest, and run_eval — the pure workload a job performs. By the end you have the thing every worker in this system ultimately runs, tested against a mock with zero network spend.

Scaffold

Create (new crate — add crates/control-eval to the workspace members):

• crates/control-eval/Cargo.toml
  • [dependencies]: control-core = { path = "../control-core" } (the domain and error taxonomy live there), plus serde, serde_json, async-trait, reqwest, tokio — all { workspace = true }.
    • reqwest (features ["json"]) is the HTTP client; .json() on requests and responses is why the feature matters.
    • async-trait is what lets generate be an async fn in a dyn-compatible trait (see the concept chapter for the E0038 you get without it).
  • [dev-dependencies]: wiremock, pretty_assertions, tokio — all { workspace = true }.
• crates/control-eval/src/lib.rs — pub mod client; and pub mod eval;, re-exporting the public types.
• crates/control-eval/src/client.rs — the trait, ModelResponse, and HttpModelClient.
• crates/control-eval/src/eval.rs — run_eval.

Dependencies this chapter exercises: async-trait (async method behind dyn), reqwest (the round trip), serde/serde_json (wire ↔ types), wiremock (dev — the mock server).

Expected result: cargo test -p control-eval → 3 tests pass (model_client_posts_prompt_and_parses_reply, server_error_is_a_retryable_worker_error, run_eval_produces_one_record_per_vignette).

The spec (givens)

ModelResponse and the ModelClient trait

/// A model's reply plus its token accounting.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ModelResponse {
    pub text: String,
    pub usage: Usage, // from control-core
}

#[async_trait]
pub trait ModelClient: Send + Sync {
    fn model_name(&self) -> &str;
    async fn generate(&self, prompt: &str) -> Result<ModelResponse, ControlError>;
}

Exactly two methods. model_name returns the pinned model string; generate is single-turn — the prompt is the entire input. The Send + Sync bound is required so futures can move across worker threads later.

→ Answer key

HttpModelClient and the wire shapes

HttpModelClient holds a base_url, a model string, and a reqwest::Client. Construct it with new(base_url, model) where both accept impl Into<String>. The base_url is injectable — a test passes server.uri(), production passes the real endpoint.

generate POSTs to {base_url}/v1/generate (trim a trailing / from the base first). The request and response bodies:

// Request body → POST {base_url}/v1/generate
{ "model": "claude", "prompt": "the vignette prompt" }

// Success response body (HTTP 2xx)
{ "text": "the model's reply", "usage": { "input_tokens": 120, "output_tokens": 8 } }

Define two private structs — a Serialize request borrowing &str fields, and a Deserialize response modelling only text and usage. serde ignores any other fields the endpoint sends.

→ Answer key

Error mapping (the policy the test pins)

send() fails to complete            → ControlError::Worker   (retryable)
response status is not 2xx          → ControlError::Worker   (retryable)   e.g. "model status 503"
2xx body fails to deserialize       → ControlError::Invalid  (terminal)

A non-2xx or a dropped connection is a transient fault worth re-dispatching; a malformed 2xx body will fail identically on retry and must surface. This is the Part II taxonomy applied to the network.

→ Answer key

run_eval

pub async fn run_eval(
    client: &dyn ModelClient,
    spec: &EvalJob,
) -> Result<Vec<ResponseRecord>, ControlError>;

Pose every vignette in spec.vignettes to the client, in order, and collect one ResponseRecord per vignette. Each record carries vignette_id and prompt from the vignette, model and epoch from the job spec (not from the client), and response/usage from the ModelResponse. A single failing call ?-propagates and fails the whole job — the scheduler decides later whether to retry, based on is_retryable.

→ Answer key

Concepts exercised

• An #[async_trait] trait as a mockable seam (&dyn ModelClient).
• Mapping HTTP outcomes onto a retryable-vs-terminal error taxonomy.
• Request/response modelling with serde — declaring only the fields you consume.
• Testing an async client against a wiremock server with an injected base URL.
• A pure workload function that depends on the seam, not the concrete client.

The build loop (you drive)

Test 1 — model_client_posts_prompt_and_parses_reply (in client.rs, #[tokio::test])

1. Write the failing test. Start a MockServer, mount a Mock matching method("POST") and path("/v1/generate") that responds 200 with the success body above. Build HttpModelClient::new(server.uri(), "claude"), call generate("decide"), and assert resp.text == "maneuver" and resp.usage == Usage { input_tokens: 120, output_tokens: 8 }.
2. Predict: before you implement generate, what status will the client see if you POST to /v1/generte (typo) instead? Recall the silent-404 trap.
3. Run — it fails to compile (no generate yet).
4. Implement ModelResponse, the trait, the request/response structs, and generate — POST, check status, parse. What it does is specified above; how you arrange it is yours.
5. Run green, commit.

Test 2 — server_error_is_a_retryable_worker_error (in client.rs, #[tokio::test])

1. Write the failing test. Mount a mock that responds 503 to any POST. Build the client, call generate("x"), take unwrap_err(), and assert err.is_retryable().
2. Predict: if generate treated every error the same and returned ControlError::Invalid, would this test pass? What would is_retryable() report, and what would the scheduler then do with a transient 503?
3. Run, confirm the assertion, adjust the status-check branch so a non-2xx maps to Worker.
4. Run green, commit.

Test 3 — run_eval_produces_one_record_per_vignette (in eval.rs, #[tokio::test])

1. Write the failing test. Define a StubClient that implements ModelClient with no network — it counts calls in an AtomicUsize and returns ModelResponse { text: format!("re: {prompt}"), .. }. Build an EvalJob with two vignettes ("a"/"one", "b"/"two"), model: "claude", epoch: 3. Call run_eval(&client, &spec) and assert: records.len() == 2, the client saw exactly 2 calls, records[0].model == "claude", records[0].epoch == 3, records[1].vignette_id == "b", records[1].response == "re: two".
2. Predict: the stub reports model_name() == "stub", but the job's model is "claude". Which one lands in each record's model field — and why is taking it from the spec rather than the client the correct choice for a blinded eval?
3. Run — fails to compile (no run_eval).
4. Implement run_eval: loop the vignettes, .await each generate, build one record each from the vignette + spec + response.
5. Run green, commit.

Done when

cargo test -p control-eval shows 3 passing tests, the 503 case is proven retryable, and run_eval produces exactly one record per vignette with model/epoch drawn from the job spec.