taiki.be

Vault Enterprise: HSM-rooted seal and DR replication

Context

Secrets management for eleven Kubernetes clusters across multiple sites. The existing setup was Vault OSS on Raft integrated storage. Two things forced a change: a regulatory requirement for hardware-rooted key custody, and a business continuity requirement that named an RPO and RTO the existing single-site deployment could not meet.

Hardware root of trust

Vault’s barrier key has to live somewhere. In OSS the unseal keys are Shamir shares held by humans, which means every restart is a ceremony and the key material’s custody chain is only as good as your process discipline. Auditors do not love this.

Vault Enterprise’s HSM integration moves the root key into a Thales Luna partition. Two distinct benefits, worth separating because they get conflated:

Auto-unseal. The barrier key is wrapped by a key that never leaves the HSM. Vault restarts unattended. No ceremony, no distributed shares, no 3am conference call to reassemble a quorum of humans.

Seal wrap. Critical values get an additional layer of HSM-backed encryption at rest, which is the part that actually satisfies FIPS-oriented requirements. Auto-unseal alone does not.

The integration itself is less interesting than the operational discipline around it: HSM partition credentials are themselves a secret, the HSM client library has to be present on every Vault node, and version drift between Luna client and Vault is a real upgrade constraint. Plan HSM client upgrades and Vault upgrades as one change, not two.

The replication decision

Vault Enterprise offers two replication modes and they solve genuinely different problems. Conflating them is the most common and most expensive mistake in this space.

Disaster recovery replication maintains a warm standby cluster with a complete copy of the primary’s data — including tokens and leases. The secondary serves no client traffic at all. It exists to be promoted. Its entire value is that when the primary site is gone, applications reconnect and their existing tokens still work, because lease state came across.

Performance replication creates secondaries that actively serve reads and manage their own tokens and leases locally, forwarding writes to the primary. Its value is latency: an application in a distant region reads secrets from a local cluster instead of crossing a WAN on every request.

We needed the first and not the second, and the tiering means that distinction has direct budget consequences.

The reasoning is about access pattern rather than any latency measurement. Our Vault consumers are overwhelmingly Kubernetes workloads on well-connected sites, and they fetch secrets at startup and on rotation — not per request. Secret reads are not on a latency-critical path, so a local read replica would have bought us nothing any consumer could observe, at meaningful cost.

What we did need was survivability of a full site loss with tokens intact. That is DR replication, and it sits in the lower tier.

Operational notes from the build

The upgrade path ran 1.15 to 1.16.31 to 1.19.19 to 2.0.3+ent on the primary, with a greenfield secondary built and paired as DR. Things worth writing down:

  • disable_upgrade_migration=true before rolling RPM upgrades. Without it, the upgrade path does things you did not ask for.
  • TLS certificate rotation is a SIGHUP. No unseal, no restart.
  • On 2.0.3+ent, verify Raft membership with vault operator raft autopilot state, not list-peers.
  • Load balancer health semantics, confirmed empirically rather than from documentation: active returns 200, standby returns 429, a DR secondary returns 472. Get this wrong and your LB will happily route traffic at a cluster that cannot serve it, or drain one that can.
  • Reference drives by serial, never by /dev/sdX.

Outcome

The generalisable point: replication topology is a requirements question, not a capability question. “We have multiple sites” does not imply you need performance replication. Ask what the secondary is for — serving traffic, or waiting to be promoted — and buy accordingly. Vendors are happy to sell you geo-distribution you will never read from.

What I’d do differently

Treat the HSM client as part of the Vault version matrix from day one. We discovered the coupling during upgrades rather than planning around it, and an upgrade campaign across four Vault versions is the wrong place to find out that your unseal mechanism has its own compatibility constraints.