Two-site Ceph object storage for a carrier
The problem
Two production Red Hat Ceph Storage 8 clusters, one per datacentre, serving object storage over RGW with multisite replication from the primary site to the secondary. Both cephadm-managed. Both, when I picked them up, effectively unmonitored outside the Ceph dashboard — which is fine right up until the dashboard is the thing that is down.
Shape of the clusters
Each site runs dedicated monitor and manager nodes, a pair of RGW gateways for client traffic, a separate pair of RGW instances dedicated to multisite sync traffic, and an HAProxy pair in front.
Separating sync gateways from client-facing gateways is the design decision worth copying. Replication backlog then cannot starve customer requests, and the two workloads can be reasoned about — and tuned, and alerted on — independently.
The bucket data pool is 3× replicated rather than erasure coded at the primary site. Worth stating plainly, because “Ceph object storage” is widely assumed to mean EC; here the access pattern and the recovery behaviour argued for replication.
RGW is tuned well above defaults: 1024 threads, 2048 concurrent requests. The shipped defaults are sized for a modest cluster, not carrier request rates.
Monitoring
The gap was alerting that survives a control-plane outage, so I built it outside Ceph: NRPE checks on the hosts, scraped by the existing Nagios estate, configuration held in Git alongside the rest of the monitoring.
- A dedicated Ceph client keyring for the check user, with only the capabilities the plugins need. Monitoring should not hold admin keys.
- An Ansible role deploying the plugin set and NRPE configuration, driven by inventory groups per role — monitors, OSDs, client RGWs, sync RGWs, HAProxy — so a host gets exactly the checks its role implies.
- Per-site group variables, so one play covers both clusters and site identity is data rather than a forked copy of the play.
The role-group split is what I would keep in any rebuild. “Is multisite sync current” is only a meaningful check on the sync gateways, and expressing that in inventory rather than in template conditionals keeps the role readable at the point someone else inherits it.
OSD topology, and a rebalance in progress
The secondary site grew unevenly: three hosts carry 22 SSD OSDs each and three carry six, while CRUSH uses a host-level failure domain with flat per-OSD weights. That combination quietly concentrates both capacity and recovery load on half the hosts — the failure domain is doing what you asked, and what you asked is not what you meant.
The fix underway is physical: enclosure extensions for the light hosts, then eight disks moved from each heavy host to its paired light one, so all six end with a comparable OSD count and a comparable mix of the two disk sizes. The OSDs move intact, no reinstall — donors weighted out first, receivers weighted back up as they land.
Two things make this survivable in production. Donors are drained by reweight before anything is unracked. And the DC hands get a bay-and-serial sheet, not an OSD ID, because an OSD ID means nothing at the front of a chassis.