Healthcare · Australia · delivered 2016

Building an Always-On SQL Server Platform for an Australian Healthcare Provider

A business-critical healthcare database was running without database-level high availability. Now: a two-node Always On architecture with synchronous replication and automatic failover, a stable listener for the applications, and encryption at rest across the databases — so losing the primary server is not the same as losing the service.

SQL ServerAlways On availability groupsWindows Server Failover ClusteringTransparent Data Encryption

The problem

For a healthcare provider, database availability is not an infrastructure statistic. Clinical and administrative systems are read by people making decisions, and when the database behind one of them stops answering, the effect is immediate and visible to patients.

This provider’s SQL Server environment was running on standalone virtual machines. Backups existed and were sound, but a backup answers a different question from the one that matters at nine on a Monday morning: it tells you the data is recoverable, not that the service is available. Losing the production server meant somebody restoring a database while the applications waited.

And availability was only half the requirement. The databases hold health information, which Australian privacy law treats as a special category. A design that delivered resilience by making another unencrypted copy of the data would have solved one problem by creating another.

What it had to solve

Not just a second SQL Server

Standing up a secondary instance is the easy part and the part that gets mistaken for the job. Each of the following had to be answered before the environment could be called highly available.

  • High availability at the database layer, not only at the virtual machine layer
  • Windows failover clustering underneath it, with quorum that survives losing a node
  • Application connectivity after a failover — where the applications point once the primary moves
  • Controlled initial synchronisation, rather than assuming the databases were ready to be replicated
  • Encryption of the data at rest, extended to the secondary so encrypted databases can participate properly
  • Backup, recovery and scheduled maintenance on the new platform
  • Proactive alerting, so a replication problem is something the team is told about
  • A patching procedure that works with the architecture instead of against it
  • A documented quality review before handover, covering performance, security and recoverability

What we built

Two SQL Server Enterprise instances on virtualised Windows Server, joined by Windows Server Failover Clustering — quorum configured with a file-share witness so the cluster can lose a node and still know it has one, distributed transaction support configured, and the cluster properties tuned rather than left at their defaults.

On top of that, an availability group with synchronous commit and automatic failover. That is the change that matters. A standalone model asks "how quickly can we repair the database server?" This one asks "can another server take over?" — and answers it without waiting for an administrator to decide.

Applications connect through an availability group listener rather than a server hostname, so when the primary role moves between replicas the application tier keeps using the same endpoint. Without that, a successful failover still leaves every application pointed at a server that is no longer primary, which is how an HA environment manages to fail over and go down at the same time.

The databases were brought in under a controlled process rather than a switch: a fresh full backup and the required transaction log backup for each, added to the availability group, restored onto the secondary in the correct recovery state, then joined. Repeated per database. It is slower than the automatic seeding option and it means the initial synchronisation is something we watched rather than assumed.

Encrypted, without giving up the failover

Transparent Data Encryption was applied across the application databases, which is straightforward on a standalone instance and less so inside an availability group: the master key and certificate are created and protected on the primary, and the certificate and its private key have to be present on the secondary as well, or the encrypted database cannot join. Encryption was then enabled per database and the state verified through SQL Server’s own encryption metadata rather than taken on trust.

The point is that the client was never asked to choose between availability and encryption at rest. Both were engineered in, which is the only acceptable answer when the data is health information and the secondary replica is a second complete copy of it.

The part worth copying

Patching that the architecture survives

High availability has to survive routine maintenance as well as unexpected failure — and more HA environments are broken by a cumulative update than by a server dying. The procedure was written and handed over rather than left for the support team to work out after go-live.

Take the secondary out of the failover path

Its failover mode and synchronisation settings are changed first, so applying an update to it cannot trigger a failover of production in the middle of the work.

Patch the secondary and let it catch up

The update is applied while production carries on untouched on the primary, then the replica is brought back into synchronisation and confirmed healthy before anything else happens.

Move the primary role deliberately

A planned failover onto the patched replica — which also exercises the failover path on a day someone chose, rather than on a day that chose itself.

Patch the remaining node, then restore the settings

The former primary is updated in turn and the synchronisation and automatic-failover settings are returned to their normal state, leaving the environment exactly as it started, one version further on.

Reviewed before handover

Delivered as an operated platform, not a finished install

The common failure in HA projects is treating the work as complete the moment replication turns green. The environment went through a documented quality review across performance, security, recoverability and robustness before it was handed over.

  • Memory configuration, MAXDOP, trace flags and backup compression set deliberately rather than left at install defaults
  • TempDB configuration, file placement, database growth settings, page verification and checksums
  • Scheduled maintenance and integrity checks running, with backups taken and verified on the new platform
  • Database Mail, operators and alerts configured, plus blocked-process monitoring and automatic error-log cycling
  • Volume maintenance and lock-pages-in-memory privileges granted to the service accounts that need them
  • Security review covering service accounts, auditing, a non-default instance port and the SQL Server Browser service disabled
  • Operating system and SQL Server patch levels confirmed on both nodes before sign-off

The outcome

A standalone production dependency became an availability group. A server hostname became a listener. A passive recovery option became a synchronised secondary that can take the primary role on its own. The databases are encrypted at rest, on both replicas. And monitoring, maintenance, backups, alerts, patching and a documented quality review went in as part of the platform rather than onto a list of things to do later.

For an organisation whose critical systems cannot wait for a failed database server to be repaired, that is the whole difference: the question stopped being how fast the server can be fixed, and became whether anyone outside the IT team needs to know it broke.

How this engagement was run

Delivered as a fixed-price project against written acceptance criteria, to the agreed timeline and without a cost variation. Client not named: the work was done under confidentiality.

Discuss a projectOther engagements