Upgrading Nova

Normative. Every step an upgrade needs is on this page; if an upgrade requires something that is not written here, that is a bug — report it rather than improvising. Manual out-of-band repair is not an acceptable upgrade path.

Three questions have to be answerable before the first command runs: what order do I upgrade, how do I prove it worked, and exactly how do I go back. The sections below are in that order.

v0.3.0


Product version	`v0.3.0`
Database schema	19
Platforms	`linux/amd64`
Support epoch	2026-08-11
Support window	2 immediate predecessors, or 180 days since a predecessor's declared support epoch — whichever still admits it

Supported predecessors. An artifact outside this list is untested, which is not the same as rejected: the interop contract is negotiated protocol plus capabilities, and a version string was never it.

Predecessor	Kind	Schema	Support epoch	Age-branch deadline
`commit:143c459`	commit	18	2026-08-08	2027-02-04

Capability profile. Core capabilities are required at registration. Everything else gates a ROLE, not admission: a donor missing one is excluded from that role and from nothing else.

core — pin-change-log/v1, snapshot/v1
assignment-destination — blob-transfer/v1
possession-audit — audit-block-hash/v1
read-source — blob-transfer/v1, read-source/v1
repair-source — repair-stream/v1

Claims this release makes, and the gate that proves each. A claim bound to a gate whose coverage is still a placeholder cannot reach a signed lock.

Claim	Gate	Runner	Coverage
`baseline-coordinator-on-schema-19`	`upgrade-schema-e2e`	`rc-docker`	derived from an executed run
`baseline-donor-interop`	`upgrade-wire-e2e`	`rc-docker`	derived from an executed run
`candidate-baseline-transition`	`upgrade-candidate-e2e`	`rc-docker`	derived from an executed run
`coordinator-upgrade-needs-no-donor-upgrade`	`mixed-fleet-e2e`	`rc-docker`	derived from an executed run

What the gates do NOT prove. Recorded because the absence of an entry is not a statement, and a reader who sees only what a gate proves will assume the rest.

backup-restore-e2e
any DECLARED claim. Restorability is not a statement about artifact compatibility, which is what a claim is; it is a statement about the procedure UPGRADING.md tells an operator to run before anything else
that a restore succeeds on the operator's own data. It restores what the drill backed up, and a backup nobody has ever restored is the thing this exists to stop being normal
anything about a partial or corrupted archive volume: the drill restores a whole snapshot
crossversion-e2e
any DECLARED claim: baseline-donor-interop is assigned to upgrade-wire-e2e, which runs the protocol against a fresh database. This gate overlaps it heavily but is not the same scope, and re-pointing a reviewed claim at a gate that happens to cover it is how coverage stops meaning anything
anything about the schema: every pairing gets a FRESH database migrated by the coordinator side's own binary, so no binary is ever run against a schema it did not produce
anything about released artifacts: both sides are built from source
donor-backed reads or drain WITH THE BASELINE COORDINATOR: the gate exercises those only when the coordinator is HEAD. Untested here, not known broken — the earlier claim that they were provably broken described the P2-M6 binary, and 143c459 contains the later TLS fix
mixed-fleet-e2e
anything about donors outside the declared support window: an unsupported donor is untested by definition, which is what unsupported means
real Nebula routing, MTU behaviour, NAT traversal or lighthouse failure. The fleet speaks federation mTLS over loopback with placeholder overlay material; federation-deploy-e2e owns the overlay and needs TUN
that a donor FETCHES bytes after an assignment. The gate asserts the coordinator's decisions — who is assignable, who is evicted, who holds a role — not the transfer, which upgrade-wire-e2e covers
upgrade-candidate-e2e
anything about a PUBLISHED release: no registry ref is resolved, no release asset is downloaded, and no signature is verified against a real bundle. That is upgrade-release-e2e, and it cannot run until the release exists
multi-coordinator ordering or fencing, which is a Phase 6 concern
upgrade-release-e2e — POST-PUBLICATION; runs after the release exists, gates completion state 4 rather than the lock
anything at lock time. It runs AFTER publication, so nothing it concludes can appear in the lock it verifies — that document is already signed
multi-coordinator ordering or fencing, which is a Phase 6 concern
upgrade-schema-e2e
donor behaviour: no donor participates
that a DOWN migration works — the contract is restore-from-backup, and nothing here exercises a downgrade of the schema itself
anything about released artifacts: both coordinators are built from source
any range other than the one it ran. The result is about (18, 19], not about old-binary compatibility in general
upgrade-wire-e2e
anything about the schema: this gate runs against a FRESH database, so it cannot say whether an old binary survives a forward schema
anything about released artifacts: it builds from source
the reverse direction. A baseline COORDINATOR serving a candidate donor is a different question and belongs to mixed-fleet-e2e

Rollback. migrate down does not exist. Going back means one of two things, and novactl upgrade status prints which one before you start:

redeploy the previous binary — the schema stays forward and the named predecessor runs against it;
restore from backup — a migration in the range destroyed information, or the range is not compatible with the predecessor.

Three things that changed in P2-M7.3

1. You run a PUBLISHED image, not a local build. docker/docker-compose.yml carries no build: section. Image digests come from a release env you install, separate from docker/.env, and passed LAST so a stale ref in your own file cannot outrank the release you verified. Developers add -f docker/docker-compose.dev.yml and build from source; operators do not.

2. Migrations announce themselves and can refuse. The entrypoint runs migrate auto, which applies a range unattended only when the WHOLE range is old-binary compatible, online, maintenance-free and free of operator procedures. Otherwise it prints the exact command and exits zero, and the coordinator's startup floor refuses to serve against a stale schema. migrate down no longer exists — see Going back.

3. NOVA_VERSION no longer overrides anything. Build identity comes from the linker stamp. An environment variable that can lie about immutable build information has no legitimate use, and the fleet census must not be able to launder a claim through one.

Step 0 — establish trust, before anything from the release runs

The bundle you downloaded is material, not authority. Verify the lock with a command whose issuer and identity are written HERE, not read from the bundle:

cosign verify-blob \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity https://github.com/nova-archive/nova/.github/workflows/release.yml@refs/heads/main \
  --bundle lock.sigstore.json \
  lock.json

Record the digest it authenticated:

LOCK_DIGEST="sha256:$(sha256sum lock.json | cut -d' ' -f1)"

Then check the bootstrap script's hash against the lock BEFORE running it:

python3 -c 'import json,hashlib,sys; \
  p=json.load(open("lock.json"))["payload"]["scripts/nova-release"]; \
  h="sha256:"+hashlib.sha256(open("scripts/nova-release","rb").read()).hexdigest(); \
  sys.exit(0 if p==h else "MISMATCH: "+h+" != "+p)'

Only now may it run:

./scripts/nova-release verify --bundle . --lock-digest "$LOCK_DIGEST"

That verifies every bundle member against the lock, audits the bundle's copy of the signer policy against the script's hard-coded one, and verifies the nova-admin image the lock names for signature and provenance. Nothing from the release executes until all of it passes.

Step 1 — record what you are running

docker compose --env-file docker/.env -f docker/docker-compose.yml ps --format '{{.Service}} {{.Image}}'
./scripts/nova-release exec --bundle . --lock-digest "$LOCK_DIGEST" -- upgrade status

upgrade status works with the database down. It prints binary identity, the compiled-in release catalog, the applied schema, the pending range, the rollback boundary, the fleet census and the last recorded upgrade run.

Step 2 — take a backup, and restore it

A backup that has never passed restore verification does not count as a backup. For some ranges — anything with RestoreToRevert — the backup IS the rollback path, and discovering it was incomplete at that moment is discovering it too late.

Back up all three, together:

# 1. Postgres — the archive's metadata.
docker compose --env-file docker/.env -f docker/docker-compose.yml \
  exec -T postgres pg_dump -U nova -Fc nova > nova.dump

# 2. nova-secrets — the master key, the OIDC seed, the swarm key.
#    Without the master key, a restored database is a catalogue of ciphertext
#    nobody can open.
docker run --rm -v nova_nova-secrets:/s:ro -v "$PWD":/out debian:bookworm-slim \
  tar czf /out/nova-secrets.tar.gz -C /s .

# 3. nova-fedpki — the federation and Nebula CA keys. Without them you cannot
#    issue, rotate or revoke a donor identity, and nothing fails until the
#    first certificate expires.
docker run --rm -v nova_nova-fedpki:/pki:ro -v "$PWD":/out debian:bookworm-slim \
  tar czf /out/nova-fedpki.tar.gz -C /pki .

Then verify the restore, on a scratch host or a scratch Postgres — never over the live one:

./scripts/backup_restore_e2e.sh   # the same drill, against a synthetic deployment

Against YOUR backup, check the same properties the drill checks: the schema version matches, a donor's registration returns under the SAME id, the master key is byte-identical, the secrets keep mode 0600, the federation CA's fingerprint is unchanged, the restored authority can still ISSUE, and a certificate issued before the backup still verifies against it. Missing any one of those means the backup is not a rollback path.

Step 3 — preflight

./scripts/nova-release exec --bundle . --lock-digest "$LOCK_DIGEST" -- \
  upgrade check --lock /release/lock.json --intent /release/intent.json \
                --expect-lock-digest "$LOCK_DIGEST"

Run from the TARGET admin image: an N−1 deployment cannot evaluate an N target, because it does not know what N requires. It fetches nothing.

It prints the rollback boundary FIRST, then seven checks. Each carries a requirement (required/optional) and an outcome (passed/failed/skipped), because "disk headroom could not be checked" must not read the same as "disk headroom is fine". A required check that could not run BLOCKS until you acknowledge it by id:

... upgrade check ... --acknowledge disk-headroom

Acknowledgement waives a SKIP and nothing else. A check that FAILED is a measurement, and there is no flag for disbelieving one.

An unsupported donor never blocks. Untested is not dead, and blocking on it would make the support window a weapon rather than a statement. A donor with no compatible protocol does block, and must be drained and replaced while the old protocol still exists.

Step 4 — install the release env

sudo ./scripts/nova-release install-env \
  --bundle . --lock-digest "$LOCK_DIGEST" --dest /etc/nova/release.env

Atomic: Compose may read that file at any moment, and a half-written one is a deployment pointed at half a release.

Step 5 — apply the migrations, explicitly

docker compose -f docker/docker-compose.yml \
  --env-file docker/.env --env-file /etc/nova/release.env \
  --profile upgrade run --rm migrate apply --to <target schema> \
  --expect-release-lock "$LOCK_DIGEST"

--to is mandatory. An unbounded apply can apply a set nobody reviewed; the target IS the reviewed decision, and the matrix above names it.

The apply takes an advisory lock, RE-READS the applied version under it, computes (applied, target] there, and refuses a migration set it cannot account for. Pass --expect-from <n> to make it refuse if the schema moved between preflight and now.

If the range carries operator procedures, it refuses and prints each with an id. Do them, then re-run with --acknowledge <id> for each. An instruction nobody read is an instruction nobody followed.

Every apply is journalled to /var/lib/nova/upgrade before it starts, because migration 0019 creates the very table that records upgrade runs — the first upgrade from schema 18 has nowhere in the database to say it began. If the apply fails, that journal is the record. Back it up with the rest.

Step 6 — bring up the released topology

docker compose -f docker/docker-compose.yml \
               -f deploy/operator/compose.federation.yaml \
               --env-file docker/.env --env-file /etc/nova/release.env \
               --profile prod --profile federation up -d

Order matters: the release env is LAST.

Step 7 — prove it worked

./scripts/nova-release verify-planes --bundle . --lock-digest "$LOCK_DIGEST" \
  --report-dir /var/lib/nova/upgrade/reports

One run id across the host plane and the four in-image planes, with per-plane reports written to a mounted directory. The planes do not collapse into one process: host image inspection needs a Docker socket, the doctor plane needs another container's network namespace, and nova-admin runs --rm.

Then watch one control interval before touching any donor.

Step 8 — roll donors forward, in bounded batches

Two parties. You drain and authorize; the volunteer applies; you confirm and undrain. The volunteer's script holds no coordinator authority and cannot reconfigure the federation.

For each donor, in batches small enough to keep the replication floor:

# 1. Convert their bundle (once per release). Preserves node id, both
#    certificates, the Kubo repo and every volume.
novactl node convert-bundle --bundle ./invites/<name> \
  --lock lock.json --intent intent.json --expect-lock-digest "$LOCK_DIGEST"

# 2. Authorize what they may run.
novactl node rollout authorize --id <node-uuid> \
  --lock lock.json --intent intent.json --expect-lock-digest "$LOCK_DIGEST"

# 3. Drain.
novactl node drain --id <node-uuid>

Send the volunteer the converted bundle and the donor-lock digest that convert-bundle printed. They run:

./donor-update.sh check
./donor-update.sh apply --expect-lock-digest sha256:<the digest you sent>

It writes .env before recreating — Compose reads it at up, so persisting afterwards would restart the old digest and report success — never removes a volume, and waits for the node to report healthy.

Then confirm and release:

novactl node list                      # the VERSION column and the reported digests
novactl node undrain --id <node-uuid>

Let repair debt settle before the next batch.

Going back

migrate down does not exist. The schema is forward-only, and a down migration would destroy information the forward one committed to. Going back is one of two operations, and upgrade status names which one before you start:

redeploy the previous binary — the schema stays forward and the named predecessor runs against it. Evidenced for (18, 19] by scripts/upgrade_schema_e2e.sh.
restore from backup — a migration in the range destroyed information, or the range is not compatible with the predecessor. This is Step 2's backup, and it is why Step 2 says to restore it first.

Donor rollback is per component and conditional. The donor topology holds node state, a persistent Kubo repo and Nebula config; reverting image refs is safe only where the donor lock declares evidence for that component. Where it does not, donor-update.sh rollback stops, writes a recovery report, and changes nothing — starting older software against state newer software wrote can corrupt the replicas the rollback was meant to protect.

The fleet policy

Unknown is not unsupported. A donor that reports no version blocks nothing. The interop contract is negotiated protocol plus capabilities.
A version string never gates anything. It is an unauthenticated self-report; it informs the census and does nothing else.
A digest mismatch is a supply-chain warning. The donor keeps its replicas, keeps receiving work its capabilities support, and keeps counting toward durability. Investigate it; do not let the system act on it.
Nonconformance advises. A donor outside the core profile or the support window receives a deprecation message on its heartbeat. Nothing is withheld.
Retirement is operator-initiated. Unsupported status alone never triggers replacement. Drain the donor, let the replacement finish, then decommission — in that order, so the replica is re-homed before the holder goes.

P2-M7.3 — Upgrade & release lifecycle

What changed. The three behaviour changes are at the top of this page, and the whole procedure above replaces the old ad-hoc sequence. In short: you run a published image, migrations announce themselves and can refuse, and NOVA_VERSION no longer overrides the build stamp.

Database migration: 0019. Additive — two new tables (upgrade_runs, upgrade_events) and ten nullable columns on nodes, plus a backfill of effective_capabilities from advertised_capabilities that touches only donor-scale rows. (18, 19] is old-binary compatible, evidenced rather than inspected: scripts/upgrade_schema_e2e.sh starts the coordinator built at commit 143c459 against schema 19 and confirms it serves a database-backed query path. Going back across this boundary is a redeploy, not a restore.

The range is auto-appliable, so migrate auto will apply it on start unless you have set NOVA_MIGRATE_ON_START=false.

If you do not use federation. Install the release env, apply 0019, recreate. Nothing else applies to you.

If you have donors. Coordinator first, then donors in bounded batches, using Step 8's two-party sequence. A donor on the previous build keeps working: the interop contract is negotiated protocol plus capabilities, and this release changes neither. Convert each bundle once with novactl node convert-bundle — it preserves the node id, both certificates, the Kubo repo and every volume.

One recovery you may need. If a donor has been offline long enough to be evicted, it now returns by itself: this release accepts an evicted node's heartbeat when it presents its registered certificate, restores it to active with a forced snapshot, and credits none of its old replicas until they are re-assigned and acknowledged. Previously such a donor was stranded permanently and the only remedy was re-enrollment. Nothing to do; it recovers on its next heartbeat, and the reactivation is written to audit_log.

Not yet available. There is no published release to install: the release workflow has never run. Two compatibility gates — upgrade-release-e2e and mixed-fleet-e2e — carry placeholder coverage, and a release candidate is impossible while either does. Until then this page describes the procedure and the tooling, both of which work today against a locally built deployment.

P2-M7.2 — Federation productization

What changed. Federation went from a set of manual steps to one supported path: federation init, federation doctor, node invite. Existing federations are adopted, not replaced.

No database migration. Schema is unchanged.

If you do not use federation

Nothing to do. Pull the release and recreate as usual — the base Compose file gained one volume declaration and no service changes.

If you have a hand-built federation

Your CA, your donors and your swarm key are all preserved. Nothing re-enrolls.

1. Collect your existing PKI into deploy/operator/import/:

File	What it is
`federation-ca.crt` / `federation-ca.key`	your federation CA
`coordinator-federation.crt` / `.key`	the coordinator's identity
`nebula-ca.crt` / `nebula-ca.key`	your Nebula CA
`swarm.key`	your Kubo swarm key

Copy them — do not move them. Nova reads from this directory and never writes to it; it is mounted read-only.

Your existing swarm.key is never regenerated. A new one would silently cut every current donor off from your storage network while leaving them looking perfectly healthy — registered, up, and unable to exchange anything.

If your hand-built setup genuinely never had some of these (a Nebula CA, for instance), leave them out rather than inventing them. Adoption creates what is missing and preserves what is not.

2. Adopt:

docker compose --env-file docker/.env \
               -f docker/docker-compose.yml \
               -f deploy/operator/compose.federation.yaml \
               --profile federation run --rm nova-admin \
  federation init \
    --adopt-from /import \
    --overlay-cidr 10.42.0.0/24 \
    --operator-overlay-ip 10.42.0.1 \
    --lighthouse-public 203.0.113.7:4242 \
    --hostname nova.example.org

Use the values your current federation already uses. If they disagree with your existing certificates, the command stops and shows the difference rather than overwriting anything.

Expected: a non-zero adopted: count, and created: covering only what your hand-built setup lacked — typically the coordinator client identity and the repair signing key, which earlier documentation never mentioned.

3. Recreate with the overlay:

docker compose --env-file docker/.env \
               -f docker/docker-compose.yml \
               -f deploy/operator/compose.federation.yaml \
               --profile prod --profile federation up -d

4. Verify:

docker compose --env-file docker/.env \
               -f docker/docker-compose.yml \
               -f deploy/operator/compose.federation.yaml \
               --profile federation run --rm nova-doctor \
  federation doctor --live

All checks should pass. Then confirm your donors are still there and unchanged:

docker compose exec coordinator novactl node list

Their NODE_ID values must be the same as before. If any donor has a new id, stop — something re-enrolled rather than adopting, and that should not happen.

Port change

The federation listener is 9443. Some older notes said 8443, which is the public HTTPS port. If you set federation.listen_addr by hand to an :8443 address, federation init rewrites it; verify with federation doctor.

Behaviour change: the coordinator waits instead of failing

Previously, a missing overlay interface stopped the coordinator from booting. It now starts, serves normally, and waits up to interface_wait_seconds (default 120) for the interface before binding the federation listener. On timeout it stays up with federation not ready.

If you prefer the old behaviour, set interface_wait_seconds: -1.

You will now see this line at startup, which is normal:

federation.listener.waiting  iface=nebula1 listen=10.42.0.1:9443

Behaviour change: donors no longer need a restart

A first-boot donor used to register but not serve until restarted. That is fixed. Donors on the new image start serving immediately after registering. No action needed — existing donors keep working on the old image and pick this up whenever they next update.

Rollback

Safe. No migration ran, so the previous release can be redeployed directly. The federation material init created is additive: an older coordinator ignores what it does not know about. Your CA and donor identities are untouched either way.

Upgrading donors

Donors update after the coordinator, in small batches, never all at once.

For each donor:

Confirm the federation stays above its replication floor if this donor disappears (novactl node list).
If the update will take more than a moment, drain first: sh novactl node drain --id <node-id>
Give the donor the exact digest to run. Never a moving tag.
The donor verifies and restarts — see the donor quickstart.
Confirm they return: registration, immediate heartbeat, and node.source.started.
novactl node undrain --id <node-id>.
Wait for repair debt to settle before the next batch.

An update must never look like a new enrollment. The donor keeps their federation certificate, Nebula certificate, node id, Kubo repository, swarm key and stored data. If a donor comes back with a new NODE_ID, something is wrong.

Do not auto-update

Do not point Watchtower or similar at Nova images. A broken or compromised release reaching every donor simultaneously defeats the diversity that makes the federation survivable in the first place. Update a few, watch, continue.