Concept: Append-Only Logs and the File Contract

Kind: Concept. No new crate — this is std::fs and serde_json doing one disciplined thing.

The log is the instrument's memory

The client chapter gave you the act of an eval: pose a prompt, get a response. This chapter is about what happens to that response after the call returns. It has to land somewhere durable, because the whole point of running an eval is to have a record you can go back to — re-score, audit, reproduce, compare against last month's run. A model response that lives only in memory is not evidence; it is a rumour.

So the coordinator writes every response to a file, and the shape of that file is a contract — the same JSON Lines contract the rest of Panoptes already speaks. The harness upstream generates a manifest of vignettes (one JSON object per line) and drops it where the coordinator can read it. The coordinator runs the eval and writes a response log (again, one JSON object per line) that the coding stage downstream reads. Neither side owns a shared database or an API; they hand each other files. That is deliberate — a file on disk is the most boring, most portable, most debuggable interface two stages can share, and jq can read it at 3 a.m. when nothing else works.

Two functions carry this contract, and they are almost aggressively small:

• read a manifest: parse JSONL text into Vec<Vignette>.
• append records: serialize each ResponseRecord to one line and add it to the log.

The reading half is unremarkable — you have parsed JSONL before. The writing half hides the one decision this chapter exists to make: the log is append-only.

Why append-only is a contract, not a convenience

Append-only means exactly one thing: a write may add lines to the end of the file, and it may never touch a byte that is already there. No rewrite, no truncate, no in-place edit. History only grows.

That sounds like a small implementation detail — .append(true) versus .write(true) on an OpenOptions. It is not. It is the property the entire instrument's trustworthiness rests on, for three reasons that compound:

• Durability. responses.jsonl is the dataset of record. If a run could truncate it, a single buggy re-run — pointed at the wrong path, restarted after a crash — could erase months of collected responses in one open(). Append-only makes that class of accident structurally impossible: the file-open mode simply cannot overwrite. The safety is in the mode flag, not in remembering to be careful.
• Reproducibility. Each response is one immutable line, written once and never edited. That means the log is the run — a faithful, ordered transcript of what the model actually said, byte for byte. Re-score it a year later and you are scoring the same responses, not a mutated copy where some later pass "fixed" a few. An eval you cannot reproduce is not a measurement; it is an anecdote.
• Concurrency and crash-safety. Because a write only ever extends the file, two things fall out for free. A crash mid-run leaves a log that is shorter than intended but never corrupt — every complete line before the crash is still a valid record. And append is the one file operation that composes safely across separate runs: a second run appends its lines after the first run's, and the first run's records are exactly where they were.

Hold onto the middle one especially — each eval response is one immutable line — because it is the sentence that turns "a log file" into "an eval instrument." The rest of this chapter is that sentence made mechanical.

The toy, end to end: an append-only event log

Here is the exact shape the build asks of you, on a domain that cannot be mistaken for the answer key: a tiny event log. Each event is one immutable line — a logical tick and a reading. One function parses the log back into events; one function appends events, append-only. It runs on std and serde_json alone, so there is a play button.

use serde::{Deserialize, Serialize};
use std::fs::OpenOptions;
use std::io::Write;

/// One immutable event: when it happened, and what was observed.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Event {
    at: u64, // a logical tick here; a wall-clock timestamp in the real instrument
    reading: String,
}

/// Parse a JSONL log into events — one line, one immutable record.
fn parse_log(text: &str) -> Vec<Event> {
    text.lines()
        .filter(|l| !l.trim().is_empty())
        .map(|l| serde_json::from_str(l).expect("each line is one JSON event"))
        .collect()
}

/// Append events to the log as JSONL. Append-only: `.append(true)` never
/// truncates what is already there — the write starts at the end of the file.
fn append_events(path: &std::path::Path, events: &[Event]) -> std::io::Result<()> {
    let mut file = OpenOptions::new().create(true).append(true).open(path)?;
    let mut buf = String::new();
    for e in events {
        buf.push_str(&serde_json::to_string(e).unwrap());
        buf.push('\n');
    }
    file.write_all(buf.as_bytes())
}

fn main() -> std::io::Result<()> {
    let path = std::env::temp_dir().join("p3fin-eventlog.jsonl");
    let _ = std::fs::remove_file(&path); // fresh start so the demo is repeatable

    // Two separate runs write to the same log — the second must not erase the first.
    append_events(
        &path,
        &[
            Event { at: 1, reading: "boot".into() },
            Event { at: 2, reading: "warm".into() },
        ],
    )?;
    append_events(&path, &[Event { at: 3, reading: "hot".into() }])?;

    // Read the whole history back.
    let text = std::fs::read_to_string(&path)?;
    let events = parse_log(&text);

    println!("lines on disk: {}", text.lines().count());
    println!("events parsed: {}", events.len());
    for e in &events {
        println!("  at={} {}", e.at, e.reading);
    }
    // The first run's record is still the first record — history is intact.
    println!(
        "first record still first: {}",
        events[0] == Event { at: 1, reading: "boot".into() }
    );
    Ok(())
}

lines on disk: 3
events parsed: 3
  at=1 boot
  at=2 warm
  at=3 hot
first record still first: true

Two things earn their keep here. The at: 1 event survives the second write untouched — first record still first: true is the append-only property, proven, not asserted. And the two writes came from two separate append_events calls, which is the crash-safety story in miniature: imagine the process had died between them, and you would be left with exactly the first two lines — short, but every one of them a whole, valid record.

The trap: File::create truncates, silently

The prediction matters because the wrong choice does not error, warn, or misbehave visibly — it quietly destroys data and returns Ok. std::fs::File::create opens for writing and truncates the file to zero length first. Point a second run at a log opened that way and every earlier record is gone before your first new byte lands. Here is that mistake, runnable, so you see the damage with your own eyes:

use std::fs::{File, OpenOptions};
use std::io::Write;

fn main() -> std::io::Result<()> {
    let path = std::env::temp_dir().join("p3fin-trap.jsonl");
    let _ = std::fs::remove_file(&path);

    // First "run" appends two records — the honest way.
    let mut f = OpenOptions::new().create(true).append(true).open(&path)?;
    writeln!(f, "{{\"at\":1,\"reading\":\"boot\"}}")?;
    writeln!(f, "{{\"at\":2,\"reading\":\"warm\"}}")?;
    drop(f);

    // Second "run" opens with File::create — the trap. This TRUNCATES to empty
    // before the first byte is written, so the earlier history is gone.
    let mut f = File::create(&path)?;
    writeln!(f, "{{\"at\":3,\"reading\":\"hot\"}}")?;
    drop(f);

    let text = std::fs::read_to_string(&path)?;
    println!("lines after the truncating write: {}", text.lines().count());
    print!("{text}");
    Ok(())
}

lines after the truncating write: 1
{"at":3,"reading":"hot"}

One line. The boot and warm records are gone, and nothing anywhere said so — no panic, no Err, no log line. This is why the append-only property lives in a test in the build and not in a comment: "remember to open in append mode" is exactly the kind of instruction a tired future maintainer forgets, and the failure is invisible until someone goes looking for data that no longer exists. A test that appends twice and asserts the line count grew is cheap, permanent insurance against silently shredding the dataset of record.

Reading is the easy half — but note who owns the failure

The reading side has no such trap, but it makes one small policy choice worth naming. A manifest line that is not valid JSON — a truncated file, a hand-edit gone wrong — is a terminal failure, not a retryable one. There is no server to blame and no point trying again: the same bad bytes will fail to parse identically on the next attempt. So the parse maps a serde error to ControlError::Invalid, the terminal variant from the Part II taxonomy — the same call you make when a model returns a 2xx body you cannot deserialize. Bad input is bad input, whether it arrives over a socket or off a disk.

enum ControlError {
    Worker(String),  // transient — worth retrying
    Invalid(String), // terminal  — a retry changes nothing
}

/// Parse a JSONL manifest into (id, prompt) pairs. A malformed line is terminal.
fn parse_manifest(text: &str) -> Result<Vec<(String, String)>, ControlError> {
    let mut out = Vec::new();
    for line in text.lines().filter(|l| !l.trim().is_empty()) {
        // Stand-in for `serde_json::from_str`: split "id|prompt".
        match line.split_once('|') {
            Some((id, prompt)) => out.push((id.to_string(), prompt.to_string())),
            None => return Err(ControlError::Invalid(format!("bad manifest line: {line:?}"))),
        }
    }
    Ok(out)
}

fn main() {
    let good = "ca_geo-030|decide\nca_geo-060|choose\n";
    match parse_manifest(good) {
        Ok(v) => println!("good -> Ok, {} vignettes", v.len()),
        Err(_) => println!("good -> unexpected error"),
    }

    let bad = "ca_geo-030|decide\nthis line has no delimiter\n";
    match parse_manifest(bad) {
        Ok(_) => println!("bad -> unexpectedly ok"),
        Err(ControlError::Invalid(m)) => println!("bad -> Invalid (terminal): {m}"),
        Err(ControlError::Worker(m)) => println!("bad -> Worker (retryable): {m}"),
    }
}

good -> Ok, 2 vignettes
bad -> Invalid (terminal): bad manifest line: "this line has no delimiter"

The real parse_manifest uses serde_json::from_str in place of that split_once, and maps its error to ControlError::Invalid the same way — a malformed manifest is a terminal fault the scheduler will not waste a retry on.

One-for-one with the build

The toy maps onto control-eval's contract.rs exactly:

• parse_log (JSONL text → Vec<Event>) ↔ parse_manifest (JSONL text → Vec<Vignette>)
• reading the file then parsing ↔ load_manifest (tokio::fs::read_to_string then parse_manifest)
• append_events with .create(true).append(true) ↔ append_records, append-only
• Event { at, reading }, one per line ↔ ResponseRecord, one immutable line per response
• a malformed line → terminal ↔ serde error → ControlError::Invalid
• "first record still first" after a second write ↔ the append_records_is_append_only test's line-count assertion

Same shape, different domain. The real thing swaps std::fs for tokio::fs (the coordinator is async) and the logical at for a real ResponseRecord, but the contract — parse in, append-only out, each response one immutable line — is identical.

Questions to lock

1. What exactly does "append-only" forbid, and which single OpenOptions flag enforces it? What does File::create do instead, and why is that dangerous for a response log?
2. Give the three properties append-only buys the eval instrument (durability, reproducibility, crash-safety) and one sentence on why each follows from "history only grows."
3. A manifest line fails to parse. Is that a retryable Worker error or a terminal Invalid one, and why does retrying change nothing?
4. Why is the append-only property pinned by a test that appends twice, rather than trusted to a comment telling the maintainer to open in append mode?