Backup swarm

If you already run a handful of machines, you already have somewhere to put your backups: each other. In a swarm, every bakelite node stores and serves copies of the other nodes' backups over a peer-to-peer iroh (QUIC) mesh. The fleet becomes its own off-site target — no S3 bucket, no monthly bill, no third party holding your data. Lose a machine and it restores straight from its peers.

Experimental. It's built into the standard binary and its durability loop is enforced and tested, but read What's not there yet before you lean on it — it's a redundancy layer on top of a normal backend, not a replacement for one.

The shape of it

Each node in the fleet does three things at once:

Serves — it holds other nodes' backup objects in a local store and hands them back on request.
Pushes — it mirrors its own databases out to the fleet, the same way it would mirror to S3, off the replication hot path.
Proves and repairs — after each mirror pass its durability controller challenges the peers to prove they still hold each of its objects (a content-hash challenge a lost or corrupted copy fails), re-replicates anything that fell below the fleet's replication factor onto other peers, and publishes the fleet's restore tip only up to what is proven durable. A sleeping peer still counts — sleep is not loss — but a wiped store or a peer gone past the loss horizon triggers repair.

So the redundancy is mutual and maintained: node A's backups live on R of its peers, and when one of those copies degrades, another peer gets a copy — automatically. It's configured once, at the top of the config, and then any database opts in with a one-line mirror entry.

Backups are pushed as an async mirror, so a slow or unreachable peer never paces or fails your live replication — a swarm node always keeps a normal sync backend (a local disk, or S3) as its primary. The swarm is the extra copy. (Retention returns the courtesy: the primary never prunes a prefix the fleet's durable chain still rests on.)

Builds and the QUIC stack

The swarm ships in the standard build — nothing extra to enable. It pulls in a QUIC networking stack (iroh), which is the one heavy dependency the otherwise-lean binary carries. If you want a leaner, iroh-free binary — say a musl static build that must avoid that stack — build without default features:

cargo build --release -p bakelite --no-default-features

Such a binary refuses to start if the config has a [swarm] section, with a message telling you to build with the default features — so you can't half-configure it by accident.

Set up a fleet

The swarm is configured once at the top level — every node shares one identity, one roster, and one store — and each database opts in with a { type = "swarm" } async mirror. Here's a node in a three-machine fleet:

[swarm]
# This node's stable fleet name: the namespace its backups occupy on every peer, so two
# machines may back up a database of the same name. Keep it across reinstalls.
node_name = "alpha"
# This node's identity. Created on first start if absent; print its *public* id (what
# the other nodes list below) with `bakelite swarm id`.
identity_file = "/var/lib/bakelite/swarm.key"
# Where this node keeps the objects it holds on its peers' behalf.
store_dir = "/var/lib/bakelite/swarm-store"
# Bind a FIXED port — the peers dial you at the address they have in their roster.
bind = "0.0.0.0:5000"
# How many distinct peers must durably hold a backup (proven, repaired; default 2).
replication_factor = 2
# The other nodes in the fleet: each one's name, public id, and a reachable address.
peers = [
  { name = "bravo",   id = "<node-b-id>", addr = "10.0.0.2:5000" },
  { name = "charlie", id = "<node-c-id>", addr = "10.0.0.3:5000" },
]

[[database]]
name = "app"
path = "/data/app.db"
  [[database.backends]]
  type = "file"
  path = "/srv/replicas/app"       # a normal sync primary — the swarm is the extra copy
  [[database.backends]]
  type = "swarm"
  mode = "async"                   # mirror this database into the fleet

The full field reference — including relay (reach NAT'd peers via iroh's relay + discovery infrastructure), permanent_loss_horizon, and per-peer class = "ephemeral"/"anchor" — is under Configuration → Peer-to-peer swarm.

Bootstrapping the roster. On each machine, write the [swarm] section and run bakelite swarm id — it creates the identity if needed and prints the public id. Collect the ids, then fill in every node's peers list with the other nodes' names, ids, and addresses. Because the rosters are symmetric — A lists B and C, B lists A and C — every node both dials and is dialled by the rest. To skip the manual id-collection round-trip, use invite and join below.

Watching it work. bakelite swarm status dials the fleet and challenges each peer to prove possession, then reports per database: the proven-durable tip, the restorable-right-now tip, and anything under-replicated. bakelite swarm status --check exits non-zero on under-replication, for cron. bakelite verify runs the same redundancy check as an extra layer and fails on a breach.

Growing the fleet: invite and join

Hand-collecting ids is the tedious part of standing up a fleet: a peer entry needs that peer's public id, but you can't know an id until the node exists, and every node references every other — so you end up starting all nodes, running swarm id on each, and copying ids around. invite/join replace that round-trip with a live exchange.

On an existing member, print a one-time ticket and wait:

bakelite swarm invite            # prints a bakelite-join:… ticket, then blocks

On the new machine (which has its own [swarm] section — at least node_name, identity_file, and store_dir), redeem it:

bakelite swarm join bakelite-join:…

The two dial each other over a dedicated, mutually-authenticated channel, swap { name, id }, and each prints the exact [swarm].peers line for the other — no id is ever typed by hand. The inviter also hands back every member it already knew, so joining through any one node teaches the newcomer the whole fleet in one shot.

The ticket is the vouch: it's short-lived (--ttl, default 10m), single-use, and carries a random secret, so only someone you handed it to can join. Add --require-approve to gate admission behind an interactive y/n that shows the newcomer's name and id first. A joined-but-hostile member is still boxed in by the trust model — it can only write its own namespace and can't read plaintext if encryption is on — so the worst a leaked ticket does is add a useless holder.

Membership spreads on its own. When daemons are running, one invite/join converges the whole fleet with no config edits: the exchanged ids are recorded as signed vouches, and each node publishes its own signed roster (databases/<node>/.roster) that peers read, verify, and set-union into their own — so a newcomer becomes known, dialable, and write-authorized fleet-wide within a convergence cycle (~30s). The paste-ready lines the commands still print are the fallback for when no daemon is running on a node (or you prefer to hand-manage the roster); you don't need them in the normal running-daemon case.

Membership vouching is transitive: seed a node's genesis peers with your real trusted members, since a node that learns the entire fleet solely through one (possibly compromised) member inherits that member's claims.

Shrinking the fleet: leave and re-key

Removing a member self-spreads too, for the two cases that stay cryptographically safe. On the node itself:

bakelite swarm leave                       # retire this node from the fleet (graceful decommission)
bakelite swarm rekey --new-id <new-id>     # hand this node's name to a rebuilt identity

Each hands the running daemon a self-signed control record that spreads like a roster does: every peer drops the node from write-auth and placement (leave, its name then burned), or reserves its name for the named successor (rekey, which then joins normally under the same node_name). Run them while the local daemon is up so it can publish the record, give it a convergence cycle, then decommission.

The discipline is that removal is destructive (it strips a peer's write access and its place in the durability math), so — unlike adding a member, which any member may vouch — a removal must be authored by the subject's own key: only a node can retire itself or hand on its own name. That keeps one compromised member from evicting the fleet, but it also means the two cases where the key isn't available — a machine whose identity is lost (a from-scratch DR rebuild), or forcibly evicting a live, uncooperative node — still need a manual all-nodes edit (they're the quorum-signed step that's next). swarm status lists what's been retired or re-keyed.

Addresses and NAT. The ticket embeds the inviter's pairing address. On a node that binds a concrete address that's automatic; a node that binds a wildcard (0.0.0.0) without relay should pass --addr <local-interface:port> to bind the pairing listener to a reachable interface so the ticket carries a dialable address. --addr is a local bind address (it must be bindable on the host); for a genuinely NAT'd node whose public address isn't a local interface, run the fleet with relay = true instead (peers then find each other by id through discovery).

Losing a machine restores from the fleet

This is the point of the whole thing. When a database has a swarm mirror, bakelite restore and verify consult the fleet as a last read source, after the node's own sync destinations. So a host whose primary disk is gone — or whose local copy has bit-rotted — restores straight from its peers, with no change to the command:

bakelite restore --db app --output /tmp/dr.db --timestamp '2026-05-30T11:55:03Z'

Restore verifies every object's hash before writing it, the same as from any other backend, so a peer serving a corrupt or wrong object is refused rather than applied. The full disaster-recovery checklist — pre-flight, restore to scratch, prove it, swap — is in Operations → Disaster recovery, and works unchanged with the fleet as the source.

A fresh box with no local replica works too: it needs the [swarm] roster (so it knows who to ask), the same node_name the lost machine had (that names its slice of the fleet), and the encryption key (if the fleet holds ciphertext) — then it reads the chain back from the peers. The rebuilt machine gets a fresh identity; the other nodes just update its id in their rosters, and the namespace carries on.

What the fleet can and can't see

Every connection between nodes is mutually authenticated by iroh, so a node always knows exactly which peer it's talking to. On top of that:

Writes are roster-gated and namespace-scoped. A peer may push or delete objects on a node only if its id is in that node's roster, and only under its own slice of the namespace (databases/<its name>/…) — so someone who merely learns your node's id can't destroy your backups, and one compromised fleet member can't clobber another member's backups either.
Reads are open to any node that can reach the port and knows the id. That's deliberate — it's what lets a fresh, not-yet-rostered box recover from the fleet. Which means the content of your backups is only protected if you turn on at-rest encryption: the fleet then holds ciphertext, and a peer without the key sees nothing useful.
How much a peer can write is boundable. Two opt-in store quotas — per_peer_quota (a cap per peer namespace) and min_free_space (a free-disk floor that protects this node's own backups) — refuse a peer's writes past the limit, so one member can't fill another's disk. Reads stay unaffected.

Run a swarm on a trusted or isolated network — a tailnet, a private VLAN — and enable encryption. By default nodes reach each other directly by the addresses in the roster; relay = true opts into iroh's relay + discovery infrastructure for fleets that can't (note the third-party surface documented there).

What's not there yet

The durability loop — prove, repair, publish only what's proven — is enforced and tested, but the swarm is still young. Known limits, so you can decide whether it fits:

Redundancy, not your only copy. A database always keeps a normal sync primary holding the complete chain; the swarm is the extra, self-repairing copy on top. That's a design stance, not just a maturity caveat — the fleet is the off-site layer, not the system of record.
Self-healing needs the owner alive. Each node repairs its own backups (it's the one that knows their true content). While a machine is dead, the surviving peers keep serving its objects but don't re-spread them; its redundancy can decay until it — or its replacement — comes back and repairs. Size replication_factor with that window in mind.
Removal is self-signed only — two cases stay manual. Adding a member self-spreads, and so now do a graceful leave and a planned re-key (swarm leave / rekey → a self-signed, swarm-replicated retirement the fleet applies). Because removal is destructive, it must be authored by the subject's own key — so the two cases where that key isn't usable still need a manual all-nodes edit: a lost-key DR rebuild (nothing left to sign the handoff), and forcibly evicting a live, uncooperative or compromised node. Both need quorum co-signing, which is the next step. (Vouching is also transitive, so seed a node's genesis peers with your real trusted members.)
Possession proofs read whole objects. Each controller pass makes every peer re-hash the objects it holds for the proving node (streamed, so memory is flat — but it's a full disk read of the replica per pass, per holder). Fine for the gigabytes-per-node fleets this targets; a sampled proof for very large replicas is future work.