synthetic monitoring
1 TopicAzure SQL Hyperscale Looks Healthy, but Logins Are Failing: Troubleshooting Partial Availability
Can an Azure SQL database look healthy while your users can't log in? Yes. It happens when the health signals and your application are measuring different things. We see this question in support cases fairly often: "Availability looked fine, so why couldn't my application connect?" This post explains why the two can disagree, especially on Hyperscale, and how to collect the evidence that shows what really happened. A quick look at Hyperscale Hyperscale separates compute from storage. At a high level: Compute replicas run your queries. These are the primary, plus any high-availability (HA) and named replicas. Each one keeps a local cache of hot data. Page servers store the data and serve pages the compute replica doesn't have cached. The log service receives the transaction log and passes changes on to the other components. This design is what makes Hyperscale scale so well. It also means "the database" is several components working together, and they don't all fail together. The Hyperscale architecture overview covers this in more detail. "Available" isn't one thing These are four questions that sound similar but aren't: Signal What it really tells you Service or resource availability The database is online at the platform level Primary replica health The compute node serving your writes is working normally Login success rate New connections are actually completing Query latency Work is finishing in a reasonable time once connected Storage can be healthy while the compute replica is struggling. The database can be online while some logins fail. And a health check can succeed between two failed attempts. Healthy storage + degraded compute ≠ healthy application experience. A real-world pattern In a recent Hyperscale case, the customer saw intermittent login failures and slow reads from remote storage on the primary replica. The storage components themselves looked healthy. The database was never fully down, and some health checks kept succeeding while the application had trouble connecting. Customer impact ended after a failover moved the workload to a different, healthy replica. However, the exact underlying cause on the original replica wasn't confirmed. The failover fixed the symptoms, and that tells you the problem was tied to that specific replica. It doesn't tell you why. Recovering service and finding the root cause are separate results, and an RCA should make clear which of the two it reached. Why averages hide the problem Most dashboards show data averaged over 1 minute, 5 minutes or 1 hour. A short burst of failures disappears in those averages. Suppose a probe connects every 5 seconds, and 6 attempts fail in a 30-second burst: View Probes Failures Success rate The 5-minute window containing the burst 60 6 90.0% The full hour 720 6 99.2% Your users felt a real outage. The hourly chart looks almost perfect. Measure what your users experience Platform metrics are useful but run your own lightweight probe from where your application runs. Here's a simple Python example: import time, csv, datetime, pyodbc CONN = ("Driver={ODBC Driver 18 for SQL Server};" "Server=tcp:<server>.database.windows.net,1433;" "Database=<db>;Encrypt=yes;Authentication=ActiveDirectoryMsi;") REGION = "<app-region>" with open("sql_probe.csv", "a", newline="") as f: w = csv.writer(f) while True: ts = datetime.datetime.now(datetime.timezone.utc).isoformat() login_ms = query_ms = None ok, err = False, "" try: t0 = time.perf_counter() conn = pyodbc.connect(CONN, timeout=15) login_ms = round((time.perf_counter() - t0) * 1000) t1 = time.perf_counter() conn.cursor().execute("SELECT 1").fetchone() query_ms = round((time.perf_counter() - t1) * 1000) conn.close() ok = True except pyodbc.Error as e: err = str(e)[:200] w.writerow([ts, ok, login_ms, query_ms, err, REGION]) f.flush() time.sleep(5) Each row records: The UTC timestamp Whether the connection succeeded Login time Time for a simple query The SQL error, if any The region the probe ran from Keep the raw rows rather than only averages. If login time rises before failures begin, that's often the earliest warning you'll get. Lining up the evidence When something goes wrong, put these side by side in UTC: Application errors, with error numbers and connection IDs Your probe results Resource Health and Service Health events Database performance data, such as sys.dm_db_resource_stats and Query Store wait statistics. See Hyperscale performance diagnostics for Hyperscale-specific waits. Any failover, scale or deployment events If you open a support case, include: The exact UTC time windows Sample error messages, with error numbers Connection IDs Your probe data Which application hosts or regions were affected With that information, we can start investigating straight away instead of spending the first day collecting it. Build for partial failure Retry transient errors with a retry limit, exponential backoff and jitter. (See our previous post on maintenance and failover.) Set sensible connection and command timeouts, so a slow login doesn't hold up a thread indefinitely. Alert on probe failures and login time, not just on platform availability. Know how to recover. For example, knowing when a failover is appropriate, and who can request it, saves time during an incident. Summary "Is the database online?" is often too simple a question. Ask instead: Can my application connect, how long does that take, and how fast are queries once connected? Measure those directly, at a fine enough granularity, and short partial outages stop being invisible.