Why availability groups fail in predictable ways
Always On Availability Groups (AGs) are robust once configured, but the failure modes that take them down are remarkably consistent across estates. Most production incidents fall into a handful of categories: a secondary stuck in the RESOLVING state, synchronous-commit replicas falling behind, automatic failover that does not trigger, and split-brain after a site partition. The fix is rarely a single setting — it is a diagnostic sequence that isolates whether the problem is the WSFC quorum, the endpoint, the database, or the network path between replicas.
This article walks through the problems I see most often on SQL Server 2016–2022 Enterprise estates, the diagnostic commands that pinpoint each, and the solution approach that avoids data loss. For the underlying architecture and design choices, see the companion guide on SQL Server Always On and high availability.
Problem 1: Secondary replica stuck in RESOLVING / NOT SYNCHRONIZED
A secondary database stuck in RESOLVING or NOT SYNCHRONIZED means the replica cannot join the AG or has lost its connection to the primary. The symptom is usually a monitoring alert showing the database as unhealthy, while the primary keeps accepting writes.
The diagnostic sequence: first check the WSFC cluster state with `Get-ClusterNode` — if a node is down or quarantined, the AG cannot establish quorum. Then verify the database mirroring endpoint exists and has CONNECT permission for the SQL service accounts on every replica. A missing or revoked endpoint permission is the single most common cause of a stuck secondary.
- Check WSFC node and quorum state before touching the AG
- Verify the database mirroring endpoint is started on every replica
- Confirm the service accounts have CONNECT permission on the endpoint
- Resume the data movement with ALTER DATABASE … SET HADR RESUME once the cause is fixed
Diagnostic query
Run these against each replica to see the real state:
SELECT r.replica_server_name, r.availability_mode, r.connected_state_desc,
d.database_name, d.synchronization_state_desc, d.synchronization_health_desc
FROM sys.dm_hadr_availability_replica_states r
JOIN sys.dm_hadr_database_replica_states d
ON r.replica_id = d.replica_id;
-- Endpoint and permissions
SELECT name, state_desc, role_desc FROM sys.database_mirroring_endpoints;
SELECT permission_name, state_desc, grantee.name AS grantee
FROM sys.database_mirroring_endpoints e
JOIN sys.server_permissions p ON p.major_id = e.endpoint_id
JOIN sys.server_principals grantee ON grantee.principal_id = p.grantee_principal_id;Problem 2: Synchronous-commit replica lag and transaction log growth
A synchronous-commit secondary that cannot keep up causes two symptoms at once: the primary reports high HADR_SYNC_COMMIT waits, and the primary database transaction log grows without bound because log blocks cannot be hardened on the secondary and cleared.
The root cause is almost always one of three: a slow disk subsystem on the secondary, a long-running undo/redo operation after a restart, or network latency exceeding the commit window. Diagnose with `sys.dm_hadr_database_replica_states` — look at `log_send_queue_size` (KB waiting to be sent) and `redo_queue_size` (KB waiting to be redone). A large redo queue means the secondary cannot apply the log fast enough; a large send queue points to network or endpoint throughput.
| Wait / counter | Meaning | Likely cause |
|---|---|---|
| HADR_SYNC_COMMIT | Primary waiting for secondary harden | Slow secondary disk or network |
| log_send_queue_size high | Log not leaving the primary | Network bandwidth or endpoint |
| redo_queue_size high | Log received but not applied | Secondary disk I/O or CPU |
| Transaction log not truncating | Cannot clear past last hardened LSN | Secondary behind or unavailable |
Problem 3: Automatic failover does not trigger
Automatic failover requires three conditions: the failing replica must be in synchronous-commit mode, the failover mode must be AUTOMATIC, and the secondary must be in a healthy synchronized state at the moment of failure. If any condition is unmet, the AG stays on a down primary and you must force failover manually — which risks data loss.
The most common trap is a secondary that was healthy hours ago but drifted to NOT SYNCHRONIZED before the failure. SQL Server will not fail over to a non-synchronized secondary. Monitor synchronization health continuously, not just at failover time, and alert the moment a synchronous secondary leaves the SYNCHRONIZED state.
- Verify availability_mode = SYNCHRONOUS_COMMIT and failover_mode = AUTOMATIC on the secondary
- Alert when synchronization_health drops below 2 (HEALTHY)
- Never force failover to an asynchronous secondary without accepting potential data loss
- Test automatic failover in a maintenance window — an untested failover plan is a gamble
Problem 4: Split-brain and Datacenter Activation Coordination (DAC)
Split-brain occurs when both the primary and a secondary data centre believe they own the AG after a network partition, and both accept writes. The result is divergent data that cannot be cleanly merged. The protection is DAC mode — Datacenter Activation Coordination — which requires a quorum of DAG members plus the witness before databases can mount.
If DAC mode was not enabled before the partition, recovery is painful: you must choose one side as authoritative, discard the other, and rebuild the discarded replica from a backup or by re-seeding. Enable DAC mode on every multi-site AG before you need it.
ALTER AVAILABILITY GROUP [AG-Prod] SET (REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT = 1);
-- DAC mode is enabled by default on AGs created in SQL Server 2012+,
-- but verify it is present and the cluster quorum model supports it:
SELECT name, failover_mode_desc, availability_mode_desc
FROM sys.availability_replicas;Trade-offs and next steps
Every AG decision is a trade-off between data safety and performance. Synchronous commit protects data (RPO = 0) but adds commit latency; asynchronous commit removes latency but accepts a small RPO. Readable secondaries offload reporting but can introduce blocking on the redo thread. The right configuration depends on the workload’s recovery objectives — define RPO and RTO per database before tuning the AG.
When a problem recurs, the next step is usually architectural, not a setting change: add a third voting node (a file-share or cloud witness) for quorum stability, move the synchronous secondary onto faster storage, or introduce a dedicated low-latency network for the mirroring endpoint. Pair the AG with a tested backup and DR strategy — an AG is high availability, not a substitute for backups.
- Define RPO/RTO per database before changing AG settings
- Add a file-share or cloud witness for stable quorum on two-node clusters
- Use a dedicated network for the mirroring endpoint to isolate replication traffic
- Pair the AG with a tested restore drill — see the backup and DR solutions guide
Lessons learned
- Check WSFC quorum and endpoint permissions before chasing AG state
- Monitor synchronization_health continuously, not only at failover time
- Enable DAC mode on every multi-site AG before a partition happens
- Treat transaction log growth as a symptom of secondary lag, not a storage problem
Written by
Jamshed Alam
Global MSP & IT Infrastructure Consultant with 20 years across Exchange, Microsoft 365, Azure, security and automation. Available for project and contract work worldwide.