Skip to manual
gluon

Operations

Back up, restore and replay deletions

Recover Gluon from a backup without bringing back records deleted since.

Platform operator

A restore brings back everything the backup held, including records a Source deleted after the backup was taken, the chat answers that quoted them and the encrypted audit payloads of those conversations. So a Gluon restore has one step a plain database restore lacks: you replay the deletion log before the restored environment serves a single question.

What to back up

WhatHow
Postgres master and audit databasesCloudNativePG archives WAL continuously and takes a daily base backup, kept for 35 days, so you can recover to a point in time
Keysthe key service holds them, never the database; back it up on its own
Elastic indexa snapshot to a snapshot repository
Deletion manifesta copy outside the database, newer than the backup

The manifest copy already exists if you forward the audit stream to a SIEM: every instance.delete record carries the ids it hid. You can also export it:

Shell
gluon erasure manifest --since 2026-09-01

Restore

  1. Decide the recovery point, and whether the audit database, the index and object storage are affected as well as the master database.

  2. Create fresh databases and migrate them forward:

    Shell
    GLUON_DATABASE_URL=.../gluon_restore GLUON_AUDIT_DATABASE_URL=.../gluon_restore_audit gluon migrate
    
  3. Restore the data into them with pg_restore or your platform's point-in-time recovery, restore the key service export and restore the index snapshot.

  4. Replay deletions before any traffic reaches the restored environment. Give it the manifest exported after the backup, or the SIEM's OCSF export as JSON lines:

    Shell
    gluon erasure replay --manifest deletions-after-backup.json
    gluon erasure replay --siem siem-export.jsonl
    

    The result counts what it did: added is manifest rows the restored log lacked, hidden is every record in the log hidden again, removed is what it purged. Where the manifest holds a newer report for a record than the restored log, the newer one wins, since its policy may be stricter. Replay is idempotent, so run it again if you're unsure.

  5. Verify the audit hash chains against the external checkpoints:

    Shell
    gluon audit-verify
    
  6. Ask a test question and read back a conversation from before the backup. Check that deletions made after the backup are gone.

  7. Only then point clients at the restored environment.

Rehearse it

Run the restore drill every quarter. Locally it runs on the compose stack with the search profile, with a fresh API image for its replica:

Shell
docker compose -f ops/compose/docker-compose.yml --profile search up -d
docker build -f ops/docker/rust.Dockerfile --target api -t gluon-api:local .
cargo run -p gluon-cli --example drill-restore

The drill seeds a record, backs up the index, both databases and the keys, then deletes the record through a connector, so the backup still holds it. It deletes the index, restores everything into scratch databases, replays the manifest and checks the hash chains, that the deleted record is no longer served, that the restored answer's quote is redacted, and that a fresh API replica answers with citations. It prints a JSON report with ok, the restore time and the total time, and exits non-zero on any failed check. The recovery time objective is 4 hours. On a laptop (2026-10-11) the restore took 3.4 seconds and verification finished at 4.5 seconds; your own infrastructure needs its own measured drill.

The step-by-step runbook for a real incident is ops/runbooks/restore.md. While you restore, the service is down or read only, and a real data loss is a P1.

Terminal running gluon erasure replay with a manifest and printing added, hidden and removed counts.
Deletion replay after a restore: nothing added, one record hidden, none removed.