From standalone MySQL to an always-on, multi-site platform, for a Pacific telecommunications operator

For a telecommunications operator, database availability is not an IT statistic. When the database behind a critical application stops answering, the effect reaches operations, customer-facing services and revenue systems within minutes — and it does so in a market where there is often no second provider for customers to fall back on.

This operator was running a business-critical application against a standalone MySQL server. Every failure scenario ended the same way: somebody recovering a database by hand while the service was down.

The brief was not "add a replica". A second copy of the database answers exactly one of the ways this environment could fail, and the others would have been left untouched.

Group replication in single-primary mode on the InnoDB engine: three members in the primary environment and a fourth at the recovery site. One member accepts writes; the rest stay read-only. If the primary leaves the group unexpectedly, the group elects an eligible member to replace it and that member becomes read/write while the others carry on as secondaries. Database availability stops depending on the health of one server.

That is half a solution. An application pointed directly at a specific host does not benefit from a successful database failover — it reconnects to a server that is no longer the primary and gets errors instead of service. So applications connect through MySQL Router, which knows the role of each member and sends read/write traffic to whichever one currently holds it.

Which moves the problem rather than solving it, unless you deal with the router too. There are two router instances, with Keepalived holding a floating connection point in front of them: if the active router goes, the address moves to the survivor and the application path stays up.

The result is protection at two distinct levels. The database layer elects a new primary. The routing layer moves the application-facing connection between redundant routers. Both have to hold for the service to stay up, and each is now someone else’s single point of failure, not this operator’s.

Local high availability answers server failure. It does nothing about losing the site. A fourth member at the disaster-recovery location takes replicated changes from the group and normally stays read-only, available for promotion if the primary site becomes unavailable.

Those are genuinely two different capabilities, and conflating them is how estates end up discovering that their "high availability" was only ever server-level. This operator has both, and knows which one answers which question.

The valuable thing here was never four MySQL servers running group replication. It was removing, one at a time, every place where a single failure could interrupt application access — the database server, the routing layer, and the site itself — and then building the migration and the operating procedures around that architecture rather than bolting them on afterwards.

For a Pacific telecommunications operator, that is the difference between a server failure and an outage. And when something larger goes wrong, there is a defined path to recovery that somebody has already walked.