Thursday Architecture Lesson #1
Azure Compute: VM, App Service, Container Apps or AKS? Don't memorize the services. Remember this: V → VM Need maximum control or have a legacy workload. A → App Service Want managed hosting for web apps or APIs. C → Container Apps Want containers without taking on Kubernetes complexity. K → AKS You genuinely need Kubernetes capabilities. 🧠 The Architect's Rule: “Don't choose AKS because you can. Choose AKS because you need it.” The right architecture isn't the one with the most powerful technology. It's the one that gives the business the right balance of: Capability • Cost • Security • Scalability • Operational complexity For example, if a simple web application can run effectively on App Service, introducing Kubernetes may add complexity without adding meaningful business value. The best architects don't ask: “What is the most advanced technology we can use?” They ask: “What is the simplest architecture that meets the requirements?” 💬 Your turn: If you were modernizing a traditional 3-tier application today, which would you choose first — VM, App Service, Container Apps or AKS? And most importantly, why?Understanding Microsoft Entra ID Group Membership Caching and Azure SQL Authentication Timing
Contributor: hudajazmawi Executive Summary Organizations frequently use Microsoft Entra ID groups to manage access to Azure SQL databases. This approach simplifies administration, improves security, and supports just-in-time access models. In some scenarios, users may experience temporary authentication failures shortly after being granted access through a Microsoft Entra ID group. These failures can appear inconsistent, especially when access succeeds to one database while failing against another. Understanding how group membership caching works during authentication can help explain this behavior and reduce unnecessary troubleshooting efforts. This article explains a real-world scenario involving temporary authentication failures after group assignment, describes the underlying authentication behavior, and provides practical recommendations for validation and mitigation. Issue Description A user was granted access to Azure SQL through membership in a Microsoft Entra ID group. Shortly afterward, the user attempted to connect using Microsoft Entra authentication. The observed behavior was: Authentication to certain databases succeeded immediately. Authentication to other databases failed temporarily. The issue appeared shortly after the group membership was granted. Access eventually began working without any configuration changes. The behavior resolved after a period of time without additional intervention. At first glance, the results appeared inconsistent because some connection attempts were successful while others failed, even though the same user credentials and group assignments were being used. Technical Background Azure SQL supports Microsoft Entra authentication, allowing access to be granted through users, groups, and service principals managed within Microsoft Entra ID. When a user authenticates, Azure SQL must determine the user's effective permissions. For users who belong to many Microsoft Entra groups, membership information may be cached to improve authentication efficiency and reduce repeated directory lookups. Caching is a common design pattern used throughout distributed systems to improve performance, scalability, and reliability. However, because caches contain information retrieved at a specific point in time, there can be a temporary delay before recently changed security information becomes visible to all authentication requests. This behavior is particularly important to understand when organizations use: Just-in-time access workflows Privileged access management processes Temporary group assignments Automated access provisioning Frequent permission validation testing Root Cause The investigation determined that the authentication failures were caused by Microsoft Entra ID group membership caching. A login attempt occurred before the user was added to the required Microsoft Entra ID group. During that earlier authentication attempt, the user's group memberships were retrieved and cached. After the user was added to the required group, subsequent authentication attempts continued using the previously cached membership information until the cache expired. As a result, authentication requests temporarily evaluated permissions using outdated group membership data. Because the newly assigned group membership had not yet been reflected in the cached information, authentication failed even though access had already been granted. Once the cached membership information expired and fresh group membership data was retrieved, authentication succeeded without any additional configuration changes. Detailed Explanation To understand the behavior, consider the following simplified sequence: Step 1: Initial Authentication A user attempts to connect to Azure SQL before being added to the required Microsoft Entra ID group. During this process: The user's current group memberships are evaluated. Membership information is cached. The required access group is not yet present. Authentication behavior reflects the permissions available at that moment. Step 2: Group Membership Change The user is added to the appropriate Microsoft Entra ID group. From an administrative perspective, the access assignment has been completed successfully. However, any previously cached authentication information may still reflect the user's earlier membership state. Step 3: Immediate Retesting The user immediately attempts another connection. Although the directory now contains the new group membership, the authentication process may still reference cached membership information created before the change occurred. The result can be a temporary authentication failure. Step 4: Cache Expiration After the cached data expires or is refreshed, authentication retrieves updated membership information. The newly assigned group is now visible during authorization evaluation. At this point, authentication succeeds as expected. Why Some Databases May Behave Differently One of the most confusing aspects of these scenarios is that different databases may appear to behave differently even when they use identical group assignment models. This typically occurs because authentication state and cache usage can differ depending on the sequence and timing of connection attempts. For example: Database A may be accessed for the first time after the group assignment occurs. Database B may have received a connection attempt before the group assignment occurred. As a result: Database A may evaluate fresh membership information and allow access. Database B may continue referencing previously cached membership information until the cache expires. This can create the appearance of inconsistent behavior even though the system is operating as designed. Mitigation and Recommendations The following practices can help reduce the likelihood of encountering similar authentication timing scenarios. 1. Assign Access Before Testing Whenever possible, add users to the required Microsoft Entra ID groups before any authentication attempts are made against Azure SQL resources. This helps ensure that fresh membership information is used during the first authentication request. 2. Avoid Immediate Validation After Permission Changes If a user has recently been granted group-based access, consider allowing time for authentication cache refresh behavior before conducting validation testing. Immediate testing can sometimes produce results based on older membership information. 3. Plan for Temporary Authentication Delays Organizations implementing just-in-time access should account for the possibility of short propagation and cache refresh intervals when designing operational procedures. 4. Use DBCC FLUSHAUTHCACHE When Appropriate For controlled testing and validation scenarios, administrators may use: DBCC FLUSHAUTHCACHE; DBCC FLUSHAUTHCACHE; This command can help refresh authentication cache behavior during troubleshooting and validation activities. As with any administrative operation, testing should be performed according to organizational change-management procedures. 5. Capture Precise Timing Information When investigating authentication behavior, collecting exact timestamps is extremely valuable. Recommended data points include: Time the user was added to the Microsoft Entra ID group Time of each authentication attempt Database target of each connection attempt Time any cache refresh operation was performed Time authentication eventually succeeded Accurate timestamps help establish a clear correlation between group membership changes and authentication behavior. Validation Guidance If you need to verify whether group membership caching is influencing authentication results, consider the following approach: Record the exact time a user is added to the required Microsoft Entra ID group. Record the time of every authentication attempt. Identify whether any login attempts occurred before the group membership change. Observe whether successful authentication occurs after a period of time without configuration changes. Where appropriate, perform controlled tests using authentication cache refresh procedures. Compare authentication outcomes against the timeline of group membership updates. This structured approach often helps determine whether the observed behavior is related to authentication caching rather than a permission configuration issue. Key Takeaways Temporary authentication failures immediately after group-based access assignment do not necessarily indicate a configuration problem. Authentication behavior may be influenced by previously cached Microsoft Entra ID group membership information. Login attempts that occur before a group membership change can affect subsequent authentication behavior until cached data expires. Different databases may appear to behave differently if they are accessed at different points in the authentication timeline. Capturing precise timestamps significantly improves troubleshooting accuracy. Proper testing practices and awareness of cache behavior can reduce confusion and accelerate issue resolution. Closing Summary Microsoft Entra ID group-based authorization provides a powerful and scalable way to manage Azure SQL access. However, like many modern cloud authentication systems, caching is used to optimize performance and improve efficiency. When group memberships change immediately before authentication testing, temporary differences between cached and current membership information may lead to short-lived authentication failures. Understanding this behavior can help administrators accurately interpret results, design effective validation procedures, and avoid unnecessary troubleshooting. By assigning permissions before authentication attempts, allowing appropriate time for cache refresh behavior, and capturing precise timing information during investigations, organizations can more effectively manage Microsoft Entra-based access and streamline their operational workflows. As always, when troubleshooting authentication scenarios, focusing on the exact sequence and timing of events often provides the clearest path to identifying the underlying cause and validating a successful resolution. Further Reading To learn more about Microsoft Entra authentication and Azure SQL security, review the following Microsoft documentation: Microsoft Entra authentication for Azure SQL https://learn.microsoft.com/azure/azure-sql/database/authentication-aad-overview Explains how Microsoft Entra authentication works with Azure SQL and the benefits of group-based access management. DBCC FLUSHAUTHCACHE (Transact-SQL) https://learn.microsoft.com/sql/t-sql/database-console-commands/dbcc-flushauthcache-transact-sql Describes how to clear the database authentication cache and notes that it clears cached Microsoft Entra group membership data stored in the database.Unexpected PITR Charges from restorableDroppedDatabases After BC → Hyperscale Migration
Why This Behavior Is by Design When migrating an Azure SQL Database from Business Critical (BC) to Hyperscale using a manual cutover, some customers notice unexpected Point-in-Time Restore (PITR) backup storage charges appearing under the following resource: /Microsoft.Sql/servers/<server>/restorableDroppedDatabases/<database> At first glance, this can be confusing—especially when: No customer-initiated drop or delete was performed The database is online and healthy post-migration Test migrations may not have shown similar charges This post explains why this happens, why it is expected by design, and how these charges naturally expire. The Observed Scenario After a BC → Hyperscale manual cutover, customers may see PITR charges tied to: restorableDroppedDatabases/<database-name> Despite the database being active and available in Hyperscale, these charges start appearing immediately after the migration cutover and gradually decrease over time. Why Does the Database Appear as “Dropped”? During a manual cutover migration, Azure SQL performs an internal platform-driven workflow to complete the transition between architectures. From a control-plane perspective: The source Business Critical logical database is internally dropped This drop is not initiated by the customer It is a required system step to complete the Hyperscale migration Telemetry confirms that the migration workflow transitions through states such as: Internal drop of the source physical and logical database Cleanup of metadata and completion of the migration This entire sequence completes within seconds and is fully platform managed. Why Are Backup Charges Generated? Although the source BC database is internally dropped, its pre-migration PITR backups are still retained according to the configured backup retention period. Here’s the key point: Backups taken before upgrading to Hyperscale are retained and billed using the dropped-database backup billing model. Because the source database is now considered dropped (from the BC perspective): The 1× database-size discount no longer applies The full data file size is added to the billable backup size Charges appear under restorableDroppedDatabases This behavior is explicitly documented as expected in internal Azure SQL billing guidance. Why Do Charges Decrease Over Time? These charges are not permanent. They: Decrease daily Continue only while the pre-migration PITR backups are retained Automatically stop once the retention window expires In practical terms: Charges stop when: days_since_migration > configured_backup_retention_days No cleanup action is required from the customer—the platform handles this automatically. Why Didn’t Test Migrations Show Similar Charges? In many reported cases, test or smaller databases migrated using the same method did not generate noticeable charges. This can be explained by two documented optimizations: Backup size threshold – very small backup footprints are not charged Low activity optimization – inactive or low-change databases generate fewer snapshots As a result, smaller or lightly used test databases may fall below the billing threshold, while larger production databases do not. Is This a Billing Error or Credit Scenario? No. Although the operation is platform-driven: The behavior is by design The charges are for temporary retention of valid PITR backups They naturally expire based on retention Therefore, this scenario is not considered a billing defect and does not typically warrant credits. How Can Customers Reduce Charges Faster? If needed, customers can: Reduce the PITR backup retention period (minimum is 1 day) Wait up to 24 hours for billing to reflect the change This shortens how long the pre-migration backups are retained and billed. FAQ – restorableDroppedDatabases Charges After BC → Hyperscale Migration Q1: Why am I seeing PITR charges for restorableDroppedDatabases when my database is still online? A: During a Business Critical → Hyperscale manual cutover, Azure SQL internally drops the source BC database as part of the migration workflow. While the Hyperscale database is active and healthy, the pre‑migration BC backups are retained and billed under restorableDroppedDatabases. Q2: Did the customer initiate a drop or delete operation? A: No. This drop is platform‑driven and required to complete the migration. It is not initiated by the customer. Q3: What exactly is being billed? A: The charges are for Point‑in‑Time Restore (PITR) backups taken before the migration. These backups are retained according to the configured backup retention period and are billed using the dropped database billing model. Q4: Why does the cost appear higher than expected? A: Once a database is considered “dropped” (from the BC perspective), the 1× database-size discount no longer applies, and the full data file size is included in the billable backup size. Q5: Will these charges continue indefinitely? A: No. The charges decrease daily and automatically stop once the pre‑migration backups expire based on the configured PITR retention period. Q6: Why didn’t this happen with smaller or test databases? A: Smaller or low‑activity databases may fall below the backup billing threshold, or benefit from low‑activity snapshot optimizations, resulting in no visible charges. Q7: Is this a billing bug or credit-worthy scenario? A: No. This behavior is by design and expected. The charges reflect valid backup retention and do not typically qualify for credits. Q8: Can the customer reduce these charges sooner? A: Yes. The customer can reduce the PITR backup retention period (minimum 1 day). Billing changes usually reflect within up to 24 hours. Key Takeaways The behavior is expected and by design Charges come from pre-migration BC backups, not the active Hyperscale database The database was internally dropped as part of migration, not by the customer Charges decrease daily and stop automatically No action is required unless the customer wants to reduce retention early Final Note As of the time of writing, this behavior is not clearly described in public customer-facing documentation, which explains why it often appears unexpected. Awareness of this mechanism can help set correct expectations when planning BC → Hyperscale manual cutover migrations.
