Concept: An Error Taxonomy That Knows What to Retry

Kind: Concept.

You already know thiserror — how to derive Error, how #[error("...")] writes the Display message, how #[from] wires up a conversion. This chapter is not about that machinery. It is about one design decision the machinery lets you make: putting the retry decision in the error type itself, so the whole system asks the type "should I try again?" and gets one authoritative answer.

The problem: a distributed system fails constantly, and only some failures are worth retrying

The control plane dispatches jobs to workers over a network. In that world failure is not exceptional — it is Tuesday. A worker process dies mid-job. A TCP connection drops between the coordinator and a worker. A client sends a malformed request. A run id that never existed gets looked up. A disk write fails.

Those failures are not equal, and the difference is the single most important thing the scheduler needs to know:

• A worker that died or a connection that dropped is transient. Re-dispatch the same job to a different worker and it may well succeed. Retrying is correct.
• A malformed request or a missing run is terminal. Retrying sends the exact same bad input and fails identically — retrying just wastes time and buries the real problem. Surfacing it is correct.

So every place that handles an error has to answer: is this one worth retrying? The question is: where does that answer live?

The wrong shape: the decision scattered at call sites

The tempting-but-wrong version spreads the retry logic across every caller:

// The anti-pattern — do NOT build this.
match dispatch(job).await {
    Err(e) if e.to_string().contains("worker") => retry(job),   // string-matching!
    Err(e) if e.to_string().contains("timed out") => retry(job),
    Err(e) => return Err(e),
    Ok(o) => o,
}

Every call site re-derives the retry policy, usually by sniffing the Display string — which is brittle (reword a message and the retry breaks silently) and duplicated (add a new retryable case and you must find every match and update it). The policy has no single home, so it drifts.

The right shape: the decision is a method on the type

Encode the taxonomy in the enum, and give it one method that answers the question. Every caller asks err.is_retryable(); the type answers. Here is the whole idea as a toy — a FetchError for a hypothetical HTTP fetcher — runnable in a scratch project:

// Scratch dep: thiserror = "1"
use thiserror::Error;

#[derive(Debug, Error)]
enum FetchError {
    /// The URL points at nothing. Terminal — retrying fetches the same 404.
    #[error("not found: {0}")]
    NotFound(String),

    /// The request is malformed. Terminal — it fails identically next time.
    #[error("bad request: {0}")]
    Invalid(String),

    /// The upstream took too long. Retryable — it may answer on the next try.
    #[error("upstream timed out: {0}")]
    Timeout(String),

    /// The connection dropped mid-flight. Retryable — reconnect and retry.
    #[error("connection dropped: {0}")]
    Connection(String),

    /// A local I/O failure, converted from std::io::Error by `#[from]`.
    #[error("io error: {0}")]
    Io(#[from] std::io::Error),
}

impl FetchError {
    /// The retry decision lives here, decided once, for the whole system.
    fn is_retryable(&self) -> bool {
        matches!(self, FetchError::Timeout(_) | FetchError::Connection(_))
    }
}

/// `#[from]` generated `From<std::io::Error>`, so `?` converts for free.
fn load_cache(path: &str) -> Result<String, FetchError> {
    let body = std::fs::read_to_string(path)?; // io::Error -> FetchError::Io
    Ok(body)
}

fn main() {
    let errs = [
        FetchError::NotFound("GET /x".into()),
        FetchError::Invalid("empty body".into()),
        FetchError::Timeout("30s elapsed".into()),
        FetchError::Connection("peer reset".into()),
    ];
    for e in &errs {
        // The caller asks the type; it does not re-decide per call site.
        let verb = if e.is_retryable() { "retry" } else { "surface" };
        println!("{verb:8} <- {e}");
    }

    // `?` funnels a missing file through #[from] into FetchError::Io.
    match load_cache("does-not-exist.json") {
        Err(e) => println!("{:8} <- {e}  (retryable={})", "io", e.is_retryable()),
        Ok(_) => unreachable!(),
    }
}

surface  <- not found: GET /x
surface  <- bad request: empty body
retry    <- upstream timed out: 30s elapsed
retry    <- connection dropped: peer reset
io       <- io error: No such file or directory (os error 2)  (retryable=false)

Read the output as a policy table. NotFound and Invalid surface; Timeout and Connection retry; and Io — a real failure, produced by ? converting a genuine std::io::Error from the missing file — surfaces too. The retry decision appears in exactly one place, is_retryable, and every caller defers to it.

Why #[from] for I/O — and only I/O

Notice that Io is the one variant with #[from]. That is deliberate. #[from] generates impl From<std::io::Error> for FetchError, which is what lets the ? operator automatically lift a bare I/O error into your error type. You want that for I/O because I/O errors bubble up from std functions you call constantly (read_to_string, socket reads, file writes) and threading them through by hand would be noise.

You do not want #[from] on the domain variants (NotFound, Invalid, …), because those carry a String message you construct deliberately at the point you detect the problem — there is no single source type to convert from, and an automatic conversion there would blur where the error was actually raised. The rule of thumb: #[from] for errors that arrive from libraries you call; explicit construction for errors your own logic decides to raise.

The trap: forgetting #[from] and reaching for ?

Here is the exact failure the build hits if the I/O variant is declared without #[from]. Predict the outcome before reading it.

// Scratch dep: thiserror = "1"
use thiserror::Error;

#[derive(Debug, Error)]
enum FetchError {
    #[error("io error: {0}")]
    Io(std::io::Error), // note: no #[from]
}

fn load_cache(path: &str) -> Result<String, FetchError> {
    let body = std::fs::read_to_string(path)?; // ? needs From<io::Error>
    Ok(body)
}

fn main() {
    let _ = load_cache("x");
}

error[E0277]: `?` couldn't convert the error to `FetchError`
  --> src/main.rs:11:45
   |
10 | fn load_cache(path: &str) -> Result<String, FetchError> {
   |                              -------------------------- expected `FetchError` because of this
11 |     let body = std::fs::read_to_string(path)?; // ? needs From<io::Error>
   |                -----------------------------^ the trait `From<std::io::Error>` is not implemented for `FetchError`
   |
note: `FetchError` needs to implement `From<std::io::Error>`

E0277, a missing-trait error — not a type mismatch. The ? operator desugars to "on Err, convert via From and return," and without #[from] there is no From<std::io::Error> to convert through. The fix is one attribute. This is the theme again: the compiler refuses to build the program until the conversion is spelled out, so a "we forgot to handle I/O errors here" bug cannot ship.

One-for-one: the toy ↔ the build

FetchError is ControlError with the labels changed:

The build adds a Store variant (persistence failures, terminal) that the toy omits, but the shape is identical: a closed taxonomy, #[from] on exactly the I/O variant, and a single is_retryable that names the two retryable cases and lets everything else fall through to terminal.

Questions to lock

1. Why is is_retryable as a method on the error type strictly better than each caller deciding retry-worthiness itself? Name two concrete failure modes of the scattered version.
2. Why does #[from] belong on the Io variant but not on Worker/Invalid/NotFound? What does #[from] actually generate, and what uses it?
3. If you add a new retryable variant to the taxonomy, how many lines of retry logic must change — and why is that number what makes this design worth it?

Next: the build that turns this taxonomy into error.rs, plus the WorkerHandle seam and the wire Message enum in worker.rs and proto.rs.