Two-site Ceph object storage for a carrier
The problem
Two production Red Hat Ceph Storage clusters, one per datacentre, serving object storage over RGW with multisite replication from the primary site to the secondary. Both cephadm-managed. Both, when I picked them up, effectively unmonitored outside the Ceph dashboard - which is fine right up until the dashboard is the thing that is down.
Shape of the clusters
Each site runs dedicated monitor and manager nodes, a pair of RGW gateways for client traffic, a separate pair of RGW instances dedicated to multisite sync traffic, and an HAProxy pair in front.
Separating sync gateways from client-facing gateways is the design decision worth copying. Replication backlog then cannot starve customer requests, and the two workloads can be reasoned about - and tuned, and alerted on - independently.
The bucket data pool is 3× replicated rather than erasure coded at the primary site. Worth stating plainly, because “Ceph object storage” is widely assumed to mean EC; here the access pattern and the recovery behaviour argued for replication.
RGW is tuned well above defaults: 1024 threads, 2048 concurrent requests. The shipped defaults are sized for a modest cluster, not carrier request rates.
Moving 150 TB to a new cluster
Security’s network-zone and DMZ architecture meant object storage could not stay on the old cluster in the management zone. The primary site is the result: a new RHCS cluster built in the production zone, and 150 TB moved onto it.
- RGW users were imported with their existing access keys, so clients changed only the endpoint.
- Bucket names were made S3-compliant on the way: underscores to hyphens, eleven buckets renamed.
- Data was copied per tenant with rclone, bandwidth-capped and phased: smallest buckets first, the largest tenant last, with buckets of 1.8 million objects and more.
Monitoring
The gap was alerting that survives a control-plane outage, so I built it outside Ceph: NRPE checks on the hosts, scraped by the existing Nagios estate, configuration held in Git alongside the rest of the monitoring.
- A dedicated Ceph client keyring for the check user, with only the capabilities the plugins need. Monitoring should not hold admin keys.
- An Ansible role deploying the plugin set and NRPE configuration, driven by inventory groups per role - monitors, OSDs, client RGWs, sync RGWs, HAProxy - so a host gets exactly the checks its role implies.
- Per-site group variables, so one play covers both clusters and site identity is data rather than a forked copy of the play.
The role-group split is what I would keep in any rebuild. “Is multisite sync current” is only a meaningful check on the sync gateways, and expressing that in inventory rather than in template conditionals keeps the role readable at the point someone else inherits it.
Rebalancing the second site by moving disks
The secondary site grew unevenly: three hosts carried 22 SSD OSDs each and three carried six, while CRUSH uses a host-level failure domain with flat per-OSD weights. That combination quietly concentrates both capacity and recovery load on half the hosts: the failure domain is doing what you asked, and what you asked is not what you meant. In practice the light hosts ran around 52% full against 14% on the heavy ones, nearfull warnings on a mostly empty cluster, and multisite sync stalled.
Reweights and the balancer only hide that. The fix was physical: disk-bay
extensions bought for the three light hosts, then eight disks moved from
each heavy host to its paired light one, so all six ended at 14 OSDs. The
OSDs moved intact, no reinstall: removed from the donor, carried across,
re-adopted with ceph cephadm osd activate, with noout, nobackfill and
norebalance set for each pair.
It was harder than planned. After the first pair landed, every OSD on the receiving host went down, its original six included: the LVM devices file only knew the disks it had started with. Rebuilding it brought them back. For the other two pairs the donor OSDs were redeployed by hand with cephadm, from backed-up cephx keys.
What made it survivable in production: donors drained by reweight before anything was unracked, and DC hands working from a bay-and-serial sheet with the drive LEDs lit, because an OSD ID means nothing at the front of a chassis.
End state: all six hosts at CRUSH weight 14, the upmap balancer on, and RGW multisite sync, configured a year earlier but never enabled, finally switched on and scoped by a sync policy.
Red Hat Ceph Storage 8 to 9
Both sites and the lab went from RHCS 8 to 9. The one real snag was cephadm itself: the mgmt-gateway and oauth2-proxy services, deployed without a certificate source, sent the upgrade into a reconfigure loop. The workaround was to remove both services, let the upgrade finish, and re-apply the saved spec, with a Red Hat case open on which 9.x release carries the fix.