What's Next: Real Brokers, mTLS, Sharding

Kind: Wrap-up.

Let us be honest about what you built. The protocol at the heart of Part VIII — serde_json frames over tokio-util's LengthDelimitedCodec, a hand-rolled handshake, heartbeats, and at-least-once redelivery — is a pedagogical protocol. It was the right thing to hand-roll, because hand-rolling it is how framing, backpressure, and death detection stop being words and become code you wrote. But a hand-rolled protocol is also a liability you now maintain. This page is the map of where a production system goes from here, and the good news running through all of it: you already have the seam for most of it. The work ahead is filling in mechanisms behind boundaries you have already drawn, not tearing anything out.

(a) A real broker or gRPC, instead of the hand-rolled wire

The framed-TCP protocol is readable and debuggable, and it taught the lesson. In production you would likely reach for tonic (gRPC) or a message broker (NATS, RabbitMQ, Redis Streams) instead.

• What you gain. Backpressure and flow control you did not have to write. Reconnection and retry semantics as library concerns, not your bug surface. Schema evolution — gRPC's protobuf and a broker's typed subjects both give you a disciplined way to add a field without breaking every worker at once, where today a change to the Message enum is a change every peer must ship simultaneously.
• What you lose. A dependency, and — for a broker — an operational thing to run, monitor, and keep alive. gRPC keeps you broker-free but hands you protobuf codegen and a heavier toolchain. The hand-rolled protocol's one virtue is that it has no moving parts you did not build.

You already have the seam for this: the transport lives entirely behind RemoteWorker, which is one impl WorkerHandle. Swapping framed TCP for a tonic client is a rewrite of that one type. The scheduler — which knows only the trait — does not notice.

(b) Security: mTLS and auth on the worker connection

Right now, any process that can reach the coordinator's port can send Register and start receiving jobs. That is fine on loopback and inside a trusted network; it is unacceptable across one.

The production path is mTLS — mutual TLS, where the coordinator and each worker both present certificates, so the coordinator knows a connecting worker is one it issued a cert to, and the worker knows it is talking to the real coordinator. Layer an auth token in the Register handshake on top, and registration becomes a gate instead of an open door.

The seam is here too: TLS wraps the TcpStream before your MessageStream codec ever sees it. The framing and protocol code is unchanged; you are inserting a layer beneath it.

(c) Horizontal scale: sharding across coordinators

One coordinator with many workers takes you a long way. The next ceiling is the coordinator itself — its scheduler, its store. Sharding runs across multiple coordinators, each owning a slice of the run space.

Here is the part worth savoring: the hard primitive for this is already built. The atomic claim_next_run — the UPDATE … RETURNING that hands a queued run to exactly one claimant and no other — is exactly the coordination point two coordinators need to not step on each other. It was built so a crashed scheduler and its replacement could not both run the same run. That same atomicity means two live coordinators pointed at one store can safely race for work: whoever wins the claim owns the run, and the loser moves on. Sharding on a shared store is nearly free because you already paid for the atomic claim. Sharding on separate stores is a bigger step — now you need a router deciding which coordinator owns which run — but the per-coordinator engine does not change.

(d) Durability beyond at-least-once

You built at-least-once delivery made safe by idempotent recording: a dead worker's in-flight job fails retryably, gets redelivered, and the store refuses to double-count it. That is the correct foundation. Production hardens it further:

• A dead-letter path. A job that fails its retry budget should not vanish or wedge its run — it should land somewhere inspectable, so an operator can see why and decide what to do. Today a permanently failing job is a gap you would have to go looking for.
• The "exactly-once" illusion. There is no true exactly-once delivery over an unreliable network — the honest framing is at-least-once delivery plus idempotent processing, which is the combination you already have. Naming it correctly matters, because it tells you where to spend: not on a mythical exactly-once transport, but on making every side effect idempotent, the way the store's recording already is.

The store is the coordination point for all of it. It is already the single source of truth about what happened; dead-lettering is one more terminal state in a state machine you already own.

Where to look when you build these

Two artifacts close the loop. The architecture diagram shows every seam named on this page — RemoteWorker behind WorkerHandle, the store as the coordination point, the transport as a replaceable layer — so you can see exactly where each production path plugs in. And the answer key is the full worked task plan: when you extend the reference build toward any of these, it is the ground truth for the shape you are extending.

You set out to close a distributed-systems gap. You now have a running control plane, a payoff test that proves its hardest property, and a clear-eyed map of the four roads out. That is not a course that ended. That is a foundation with seams cut for what comes next.