taiki.be

Ansible Tower to AWX, twice

Why we left Tower

Ansible Tower 3.8.1, licensed per managed node across the estate.

The licence covered every server. Actual automation was concentrated in a handful of places: the Splunk, Java, Python and Go development teams, plus our own infrastructure team. We were paying estate-wide for something a fraction of the estate used. I am not publishing the figures, but the shape is the point — the entitlement was sized to the server count, and the server count was not what drove value.

There was no end-of-life gun to our heads. Tower 3.8 was old and AAP 2.x was the vendor’s forward path, but the decision was economic rather than forced. AWX is the same controller without the licence, in exchange for self-support. For that usage profile the trade was easy.

Worth naming the intent behind it, because it matters later: at the time I wanted more Ansible CLI boxes, not fewer. Automation closer to the teams that used it, less centralised gatekeeping.

Move one: AWX on RKE2

RKE2 for three reasons: a mature Kubernetes distribution, fleet management through Rancher for the clusters we already had, and — in theory — the option to run other distributions including OpenShift underneath it.

Deployed with awx-operator using the operator-managed PostgreSQL. That is still what runs it. Moving to CloudNativePG is the outstanding item, and would have been the better choice from the start: the operator’s bundled database is fine until you want backup, failover and version upgrades to be someone else’s solved problem.

The real work was not the controller, it was execution environments. Tower’s virtualenv model let job templates depend on whatever happened to be installed on the controller host. Container images make every one of those dependencies explicit, which is the point and also the effort. We ended up splitting EEs by target OS — older RHEL 8 targets need a different execution environment from current ones, and pretending otherwise produces failures that look like playbook bugs.

The piece worth doing early, which is easy to defer and hard to retrofit: point the controller at a real secrets backend instead of carrying the file-on-the-controller model forward. Tower encouraged that model. Moving platforms does not remove it; it has to be replaced deliberately.

Then the platform decision changed

The telco project standardised on OpenShift. That decision was made for reasons that had nothing to do with automation tooling, and the automation had to follow. RKE2 stopped being where platform investment went, and keeping the controller there meant maintaining a second Kubernetes distribution for one workload.

Move two: to OpenShift

There is no in-place path for an AWX instance between clusters. Everything that matters — job templates, inventories, credentials, job history, schedules — lives in PostgreSQL, so the migration is a database migration with a controller rebuilt around it.

What made it tractable: the controller is not a customer-facing service. There was no uptime requirement, so we moved project by project rather than big-bang. Move a project’s templates and inventories, run its pipelines, confirm they work, move on. Change freezes went through the normal CAB process. Validation was simply that the pipelines ran.

AWX 24.6.1 on OpenShift — 4.18 at the time, 4.20 now.

The database is where this gets awkward, and it is why the cutover stayed manual rather than automated. AWX encrypts credential fields with an instance secret key. Restore a dump without moving that key and every stored credential comes back unreadable. It is a one-shot migration whose riskiest step needs human judgement; automating a cutover you run once is effort in the wrong place.

What actually happened

Two years, not two weekends.

The interesting failures were not technical:

  • Projects were missed. The old Tower kept being used by teams nobody had chased, and this only surfaced when I rebuilt the management host on RHEL 10 and went looking at what still pointed where. Migration ownership was split across people, and split ownership is how you end up with a half-migrated platform that everyone believes is finished.
  • An environment variable fired jobs at the wrong controller. TOWER_HOST leaked from one context into another and jobs ran against an instance nobody intended. Cheap to fix, expensive to notice.
  • Credential encrypted fields are stripped on export/import. The same secret-key problem as above, in the shape most people meet it first.
  • ansible-core 2.18 refuses undecrypted vault files, which broke SCM inventory sync until the inventories were reworked.
  • The Splunk CD pipeline needed rebuilding around the RHEL 8 execution environment, with a hand-built inventory.

The generalisable lesson: decommissioning is part of the migration, not cleanup afterwards. An idle controller left running is not neutral. It still holds credential material, and as long as it answers requests, someone will keep using it.

Where it landed

AWX 24.6.1, released 2 July 2024 — the last AWX release. Development is paused pending an architectural rewrite, with the maintainers stating that the existing application architecture limits their ability to change it. The awx.awx collection has been flagged as lacking active maintenance.

So after two migrations, the platform is stable, working, and static. A frozen release with no outstanding CVEs is operationally fine. But it turns the platform question from “does this work” into “when do we move, and to what” — which, given the history above, is worth answering before someone else’s decision answers it for us.

Successor options

AAP 2.x — Red Hat’s supported line, and now a natural fit given the estate runs OpenShift. Event-driven Ansible and certified collections are real advantages. Per-node pricing is the constraint, which is where this whole story started.

Ascender (CIQ) — downstream rebuild of AWX with commercial support, from the Rocky Linux vendor. Enterprise Licence Agreement pricing rather than per-node, which is a different cost curve for a growing estate. Ascender Pro adds an event layer, local content hub, and platform audit log. AWX-derived, so job templates and workflows carry over.

Semaphore UI — different philosophy. Go, MIT-licensed, single binary, no external runtime dependencies, backed by SQLite/MySQL/PostgreSQL, not Kubernetes-coupled. Treats Terraform/OpenTofu as first-class. Recent releases added extended RBAC and HA. Not an AWX drop-in — templates and workflows get rebuilt, not migrated.

Where I’d go, and the part I got wrong

Ascender, or back to AAP if the budget is there.

Which is worth sitting with, because it inverts the premise of this whole write-up. We left a licensed product because a small, senior team was not using enough of it to justify estate-wide pricing. The team has since taken on more juniors, and the things a supported platform sells you — RBAC, guardrails, audit, upgrades that are someone else’s problem — are worth more to that team than they were to the one that made the original decision.

The argument that got us out is not the argument that applies now. Almost none of this was driven by the automation platform itself: licence terms first, then a platform standard set elsewhere, then the shape of the team. Tooling decisions get made for you more often than you make them; the useful skill is noticing when the reasoning behind an old decision has quietly expired.