Forum Discussion

MikeRM2's avatar
MikeRM2
Tin Contributor
Aug 10, 2026

Query Store transaction preventing restoring database

I have a database backup that was taken on SQL Server 2019 instance. While trying to restore it to a SQL Server 2025 instance, the files restored without issue, but it remained "In Recovery" for three days before I tried to take it to Emergency, Single User, and back online, though it still remained in recovery.

I then took the backup to a SQL Server 2019 instance and was able to restore it and bring it online there. Though the logs on both servers had shown similar errors to these two. The Page and Log record Id were consistant.

```text

Database ID 5, Page (1:47416) is marked RestorePending, which may indicate disk corruption.

During undoing of logged operation in database (page(1:301040) if any), an error occurred at log record ID (15885:36832:180).

```

 

Doing some investigation has provided that there is a stuck transaction, and I was able to pin this down to Query Store.

 

OPENTRAN (This spid was from the original server.):

```text

ransaction information for database 'BrokenDatabase'.

 

Oldest active transaction:

SPID (server process ID): 26s

UID (user ID) : -1

Name : DELETE

LSN : (15885:19072:1)

Start time : Mar 1 2025 7:27:58:633PM

SID : 0x01

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

```

 

I had the Top and bottom 50 of 938 transactions from fn_dblog, but since it contains data this community will not allow it.  Makes it harder to get an answer without the data showing that query store is the issue, but to play by the rules you have give lackluster information, because the guidelines are even searching through code blocks, markdown, and tables.

 

With Query Store in the Read_Write mode, I was unable to return anything from sys.query_store_plan and / or sys.query_store_runtime_stats, because they were locked by the ASYNC_LOAD of QDS.

 

All that I have collected show that is internal to the Query Store.

sys.internal_tables:

 

| name | object_id | principal_id | schema_id | parent_object_id | type | type_desc | create_date | modify_date | is_ms_shipped | is_published | is_schema_published | internal_type | internal_type_desc | parent_id | parent_minor_id | lob_data_space_id | filestream_data_space_id |

| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |

| plan_persist_runtime_stats | 213575799 | NULL | 4 | 0 | IT | INTERNAL_TABLE | 2019-09-24 | 2019-12-28 | 0 | 0 | 0 | 243 | QUERY_DISK_STORE_RUNTIME_STATS | 0 | 0 | 0 | NULL |

 

sys.dm_tran_active_transactions:

 

| transaction_id | database_id | database_transaction_begin_time | database_transaction_type | database_transaction_state | database_transaction_status | database_transaction_status2 | database_transaction_log_record_count | database_transaction_replicate_record_count | database_transaction_log_bytes_used | database_transaction_log_bytes_reserved | database_transaction_log_bytes_used_system | database_transaction_log_bytes_reserved_system | database_transaction_begin_lsn | database_transaction_last_lsn | database_transaction_most_recent_savepoint_lsn | database_transaction_commit_lsn | database_transaction_last_rollback_lsn | database_transaction_next_undo_lsn |

| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |

| 255756374 | 5 | NULL | 1 | 4 | 524289 | 1 | 0 | 0 | 0 | 10677320 | 0 | 0 | 15885-00000-19072-00001 | 15885-00000-36952-00023 | NULL | NULL | NULL | 15885-00000-36832-00179 |

 

Last Thursday, I was able to force the database to turn query store off, I tried after that setting it for Read_only and then clearing Query Store, but the clearing went into a lock for ASYNC_LOAD of QDS, after 6 hours of nothing happening and checking requests and seeing the lock, I ended that task. It was not actively doing anything just sleeping.

 

My ultimate goal is to be able to get his database to the SQL Server 2025 instance and have it online. What can I do with the original backup or currently the online version of the database on the SQL Server 2019 instance to be able to make that happen?

7 Replies

    • MW_DEV's avatar
      MW_DEV
      Brass Contributor

      Thank you for the kind words. It was a really interesting problem to solve :)

  • MW_DEV's avatar
    MW_DEV
    Brass Contributor

    Hi. Very interesting case. I’m not entirely sure whether this will solve your problem, as I wasn’t able to reproduce this scenario, but these are the steps I would take. 

    Based on the information you collected, I would approach this by fixing the Query Store state on SQL Server 2019 first, creating a completely new clean backup, and only then restoring that backup to SQL Server 2025. The fact that the same backup can be restored and brought online on SQL Server 2019, but SQL Server 2025 remains in recovery, strongly suggests that the problem occurs while SQL Server 2025 is performing recovery and database upgrade.

     

    SQL Server recovery consists of three phases:

    1. Analysis
    2.  Redo
    3. Undo

     

    During the Undo phase, SQL Server rolls back transactions that were still active at the recovery point. 

    Microsoft documents this process here:

    https://learn.microsoft.com/en-us/sql/relational-databases/backup-restore/restore-and-recovery-overview-sql-server?view=sql-server-ver17  

    In your case, sys.dm_tran_active_transactions shows those LSNs are essentially adjacent, so it looks like recovery is failing while trying to undo this particular transaction. Since your investigation points to Query Store, I would not continue trying to repair the copy that is already stuck on SQL Server 2025. Instead, I would use the working SQL Server 2019 copy to remove or repair the Query Store state before taking a new backup.

     

    The sequence I would use is the following:

    1.Restore the database on SQL Server 2019

    Use the instance where you already know the database can successfully complete recovery and come online. Do not use the SQL Server 2025 copy for the repair process.

     

    2. Run DBCC CHECKDB first

    Before changing anything, verify that the source database is physically and logically consistent:

    DBCC CHECKDB (N'BrokenDatabase') 
    WITH NO_INFOMSGS, ALL_ERRORMSGS; 
    GO

    Microsoft recommends validating source databases with DBCC CHECKDB before migration:

    https://learn.microsoft.com/en-us/ssms/migrate/upgrade-sql-server 

    This step is especially important because the error log contains:

    Page (...) is marked RestorePending, which may indicate disk corruption.

    If CHECKDB reports allocation or consistency errors, I would stop here and investigate those first

    Ideally the result should be 0 allocation errors and 0 consistency errors

     

    3. Check the current Query Store state

    SELECT 
    actual_state_desc, 
    desired_state_desc, 
    current_storage_size_mb, 
    max_storage_size_mb, 
    readonly_reason, 
    interval_length_minutes, 
    stale_query_threshold_days, 
    size_based_cleanup_mode_desc,
    query_capture_mode_desc 
    FROM sys.database_query_store_options;

    Pay particular attention to: actual_state_desc, desired_state_desc and readonly_reason.

     

    4. Turn Query Store completely OFF

    I would not use READ_ONLY for this operation. Use:

    ALTER DATABASE BrokenDatabase 
    SET QUERY_STORE = OFF; 
    GO

    Then verify the state again. This distinction is important. READ_ONLY still leaves Query Store active. OFF disables Query Store, which is required before running the Query Store consistency repair procedure.

     

    5. Run Query Store consistency repair

    Starting with SQL Server 2017, Microsoft provides a specific procedure for recovering an inconsistent Query Store:

    EXEC BrokenDatabase.dbo.sp_query_store_consistency_check; 
    GO

    Microsoft explicitly states that Query Store must be disabled before running this procedure. The documented recovery procedure is:

     

    ALTER DATABASE [myDatabase] 
    SET QUERY_STORE = OFF;
    
     EXECUTE [myDatabase].dbo.sp_query_store_consistency_check; 
    
    ALTER DATABASE [myDatabase] 
    SET QUERY_STORE = ON; 
    
    ALTER DATABASE [myDatabase] 
    SET QUERY_STORE (OPERATION_MODE = READ_WRITE);

     

    For this migration, however, I would not enable Query Store again yet. I would leave it OFF until the database has successfully migrated to SQL Server 2025.

    Documentation: 

    https://learn.microsoft.com/en-us/sql/relational-databases/performance/best-practice-with-the-query-store?view=sql-server-ver17#verify-that-query-store-collects-query-data-continuously 

    So in your case:

    ALTER DATABASE BrokenDatabase SET QUERY_STORE = OFF; 
    GO
    
     EXEC BrokenDatabase.dbo.sp_query_store_consistency_check; 
    GO

    6. Optionally clear Query Store

    ALTER DATABASE BrokenDatabase 
    SET QUERY_STORE CLEAR; 
    GO

    Microsoft documents QUERY_STORE CLEAR as another recovery option when Query Store data is damaged. However, I would consider this secondary to the consistency check.

    If I understood correctly, You previously attempted to clear Query Store while it was still active and encountered the ASYNC_LOAD wait. The important difference here is that Query Store is first disabled and its internal consistency is checked. If CLEAR again becomes stuck, I would not necessarily block the migration on it. I would leave Query Store OFF and continue with the remaining validation

     

    7. Force a checkpoint - This gives SQL Server an opportunity to persist the clean database state

    USE BrokenDatabase; 
    GO 
    CHECKPOINT; 
    GO

     

    8. Check the oldest active transaction again

    DBCC OPENTRAN (BrokenDatabase); 
    GO

    This is a very useful validation step. I would specifically check whether that transaction is still present. If the transaction disappears after

    QUERY_STORE = OFF 
    sp_query_store_consistency_check
    CHECKPOINT

    that would be a strong indication that Query Store was involved in the recovery problem.

     

    9. Run DBCC CHECKDB again before taking the new backup:

    DBCC CHECKDB (N'BrokenDatabase') 
    WITH 
      NO_INFOMSGS, 
      ALL_ERRORMSGS; 
    GO

    I would not migrate the database until CHECKDB completes cleanly.

     

    10. Create a completely new full backup

    Do not reuse the original backup. Create a new backup from the cleaned SQL Server 2019 database

    BACKUP DATABASE BrokenDatabase 
    TO DISK = N'X:\Backup\BrokenDatabase_Clean.bak' 
    WITH 
     COPY_ONLY, 
     CHECKSUM,
     INIT, 
     STATS = 5; 
    GO

    WITH CHECKSUM is important because SQL Server validates page checksums while creating the backup and also writes backup checksums.

     

    11. Verify the new backup

    RESTORE VERIFYONLY 
    FROM DISK = N'X:\Backup\BrokenDatabase_Clean.bak' 
    WITH CHECKSUM; 
    GO

    Keep in mind that RESTORE VERIFYONLY does not replace DBCC CHECKDB

     

    12. Make sure SQL Server 2025 is fully patched - always worth to check it

    SELECT 
    @@VERSION, 
    SERVERPROPERTY('ProductVersion') AS ProductVersion, 
    SERVERPROPERTY('ProductLevel') AS ProductLevel, 
    SERVERPROPERTY('ProductUpdateLevel') AS ProductUpdateLevel; 
    GO

     

    I would test this on the current SQL Server 2025 CU rather than RTM or an older build. I did not find a documented SQL Server 2025 fix that explicitly says something like "restore of SQL Server 2019 database hangs during Query Store undo", so I would not claim that a particular CU fixes this exact problem. However, reproducing a Database Engine recovery problem on the latest CU is important before escalating it to Microsoft

     

    13. Finally - restore the new backup to SQL Server 2025

    Restore the newly created backup, not the original one. The original backup already contains the transaction/log state and every time you restore that same old backup, SQL Server has to reconstruct that same historical recovery state and eventually process that same transaction. You cannot remove that transaction from an existing backup

    RESTORE DATABASE BrokenDatabase 
    FROM DISK = N'X:\Backup\BrokenDatabase_Clean.bak' 
    WITH 
      MOVE N'<logical_data_file>' TO N'<new_data_path>', 
      MOVE N'<logical_log_file>' TO N'<new_log_path>', 
    RECOVERY, 
    STATS = 5; 
    GO

    SQL Server will perform both database recovery and the internal database-version upgrade.

     

    I know,  It may look a bit overwhelming or overly cautious, but by restoring it successfully on SQL Server 2019, you have an opportunity to allow recovery to complete, repair/disable Query Store, checkpoint the database, verify consistency, and then create a new backup representing a new clean recovery point. There is no reason to perform a destructive repair while you still have a database that can successfully come online on SQL Server 2019. Likewise, switching the SQL Server 2025 copy between EMERGENCY, SINGLE_USER, and ONLINE does not fix the underlying Undo operation - at least I am not aware of it. 

     

    If after disabling Query Store, running sp_query_store_consistency_check, executing CHECKPOINT, and creating a new backup, SQL Server 2025 still fails at exactly the same LSN or page, I would treat that as a strong candidate for a SQL Server engine issue rather than continuing to manipulate Query Store manually.

    At that point I would reproduce the problem on the latest SQL Server 2025 CU and open a Microsoft Support case with:

     

    • the SQL Server 2019 and 2025 exact builds,
    • the ERRORLOG from both servers,
    • DBCC CHECKDB results,
    • DBCC OPENTRAN output,
    • sys.dm_tran_active_transactions,
    • Query Store state,
    • the exact failing LSN,
    • and, if possible, a reproducible backup.

     

    I would definitely not modify sys.internal_tables or Query Store internal tables directly

     

    Sorry again for the wall of text. I hope this line of thinking is clear and will at least help point you in the right direction toward a solution.

    • MikeRM2's avatar
      MikeRM2
      Tin Contributor

      From the SQL Server 2019 Machine. I have restored the database.

      5 percent processed.

      10 percent processed.

      15 percent processed.

      20 percent processed.

      25 percent processed.

      30 percent processed.

      35 percent processed.

      40 percent processed.

      45 percent processed.

      50 percent processed.

      55 percent processed.

      60 percent processed.

      65 percent processed.

      70 percent processed.

      75 percent processed.

      80 percent processed.

      85 percent processed.

      90 percent processed.

      95 percent processed.

      100 percent processed.

      Processed 357688 pages for database 'BrokenDatabase', file 'BrokenDatabase' on file 1.

      Processed 7444260 pages for database 'BrokenDatabase', file 'BrokenDatabase_log' on file 1.

      Msg 829, Level 16, State 1, Line 2

      Database ID 5, Page (1:47416) is marked RestorePending, which may indicate disk corruption. To recover from this state, perform a restore.

      Errors occurred during recovery while rolling back a transaction. The transaction was deferred. Restore the bad page or file, and re-run recovery.

      Restore was successful but deferred transactions remain. These transactions cannot be resolved because there are data that is unavailable. Either use RESTORE to make that data available or drop the filegroups if you never need this data again. Dropping the filegroup results in a defunct filegroup.

      RESTORE DATABASE successfully processed 7801948 pages in 376.268 seconds (161.992 MB/sec).

       

      When I run CheckDB the first time, I get the following error:

       

      Msg 7929, Level 16, State 1, Line 8

      Check statement aborted. Database contains deferred transactions.

       

      I am able to get the options:

      | actual_state_desc | desired_state_desc | current_storage_size_mb | max_storage_size_mb | readonly_reason | interval_length_minutes | stale_query_threshold_days | size_based_cleanup_mode_desc | query_capture_mode_desc |
      | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
      | READ_WRITE | READ_WRITE | 60 | 1000 | 0 | 60 | 30 | AUTO | AUTO |

       

      To get Query Store to turn off I have to use the Forced, but I am able to get it to turn off.

       

      Second run of option:

      | actual_state_desc | desired_state_desc | current_storage_size_mb | max_storage_size_mb | readonly_reason | interval_length_minutes | stale_query_threshold_days | size_based_cleanup_mode_desc | query_capture_mode_desc |
      | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
      | OFF | OFF | 60 | 1000 | 0 | 60 | 30 | AUTO | AUTO |

       

      Now I am running the consistency check, but that is almost immediately suspending. I can see this with the dm_exec_requests and dm_os_waiting_tasks

      | session_id | status | command | wait_type | wait_time | blocking_session_id | database_id | percent_complete |
      | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
      | 65 | suspended | EXECUTE | LCK_M_IX | 44539 | -3 | 5 | 6.96263 |

       

      | session_id | wait_type | blocking_session_id | resource_description |
      | :--- | :--- | :--- | :--- |
      | 65 | LCK_M_IX | -3 | objectlock lockPartition=0 objid=213575799 subresource=FULL dbid=5 id=lock1774ee13380 mode=X associatedObjectId=213575799 |


      after I ran the consistency, I also checked the Server Logs and see this:

      Internal table access error: failed to access Query Store internal table with HRESULT: 0x80004005

      This makes me wonder if I should continue with what you have provided with the checkpoint or not?

      • MW_DEV's avatar
        MW_DEV
        Brass Contributor

        Hey, sorry for the late response. I had to think about it. Based on the latest results, I would stop at this point and not continue with CHECKPOINT yet.

        The new clue is:

        blocking_session_id = -3

        wait_type = LCK_M_IX

        object_id = 213575799

        In SQL Server, blocking_session_id = -3 means that the resource is owned by a deferred recovery transaction. So sp_query_store_consistency_check is not being blocked by a normal user session that can simply be killed.

         

        Also, we know that object_id = 213575799 is the Query Store internal table you identified earlier.

        So the likely sequence is:

        1. Page 1:47416 cannot be accessed.

        2. Recovery cannot finish UNDO for a transaction.

        3. The transaction becomes DEFERRED.

        4. That recovery transaction keeps locks on Query Store internal data.

        5. sp_query_store_consistency_check requests an IX lock.

        6. It waits on LCK_M_IX with blocker -3.

        So at this point I would treat the Query Store problem as a consequence of the deferred recovery transaction, not necessarily as the original cause.

         

        So here is a plan:

        1. First check the affected page

        Use this query:

        USE master;
        GO
        SELECT
        database_id,
        file_id,
        page_id,
        event_type,
        error_count,
        last_update_date
        FROM msdb.dbo.suspect_pages
        WHERE database_id = DB_ID(N'YourDatabaseName');
        GO

        Then try to identify page 1:47416

        DBCC TRACEON(3604);
        GO
         
        DBCC PAGE (N'YourDatabaseName', 1, 47416, 3);
        GO

        DBCC PAGE might fail because the page is already marked RestorePending, but the result is still useful if SQL Server can return any metadata. If you are on a version where sys.dm_db_page_info is available, I would also try:

        SELECT *FROM sys.dm_db_page_info
        (
        DB_ID(N'YourDatabaseName'),
        1,
        47416,
        'DETAILED'
        );

         

        2. Check whether this is the only bad page

        Also inspect suspect_pages for other entries. If page 1:47416 is only one of several damaged pages, restoring only this page might not be sufficient. It is also worth reviewing the SQL Server error log around the restore/recovery operation for errors such as: 823, 824, 825, 829 or 3414

        This is important because the deferred transaction is normally a symptom of SQL Server being unable to access data required during rollback.

         

        3. If you have a known-good backup and transaction log chain

        This would be my preferred recovery path. If the database is using FULL or BULK_LOGGED recovery and you have the required backup/log chain, consider a page restore of page 1:47416.

        The idea is:

        RESTORE DATABASE [YourDatabaseName]
        PAGE = '1:47416'
        FROM DISK = N'X:\Backups\YourDatabase_full.bak'
        WITH NORECOVERY;
        GO

         

        Then apply the necessary transaction log backups:

        RESTORE LOG [YourDatabaseName]
        FROM DISK = N'X:\Backups\YourDatabase_log_1.trn'
        WITH NORECOVERY;
        GO
         
         
        RESTORE LOG [YourDatabaseName]
        FROM DISK = N'X:\Backups\YourDatabase_log_2.trn'
        WITH NORECOVERY;
        GO

        And finish the recovery with the last required log:

        RESTORE LOG [YourDatabaseName]
        FROM DISK = N'X:\Backups\YourDatabase_log_last.trn'
        WITH RECOVERY;
        GO

        The exact restore sequence depends on your backup history, so I would verify the backup chain before executing this. I cannot tell you the exact order because I don't know your backup strategy. After the page is restored and recovery can successfully complete the deferred rollback, check whether the blocker has disappeared- I mean that we have no more blocking_session_id = -3:

        SELECT
        session_id,
        status,
        command,
        wait_type,
        blocking_session_id,
        wait_resource
        FROM sys.dm_exec_requests
        WHERE database_id = DB_ID(N'YourDatabaseName');

        4. Only after the deferred transaction is gone

         

        Then return to Query Store:

         
        USE [YourDatabaseName];
        GO
         
        SELECT
        actual_state_desc,
        desired_state_desc,
        readonly_reason,
        current_storage_size_mb,
        max_storage_size_mb
        FROM sys.database_query_store_options;
        GO

        With Query Store disabled, run:

        EXEC sys.sp_query_store_consistency_check;
        GO

        At this point it should no longer be waiting on the deferred recovery transaction.

        And now you can run:

        DBCC CHECKDB (N'YourDatabaseName')
        WITH NO_INFOMSGS, ALL_ERRORMSGS;
         
        GO

        If CHECKDB completes cleanly, I would then create a new full backup and use that new backup for the SQL Server 2025 restore.

        Hopefully you have a known-good backup available and the page restore path from step 4 resolves the deferred transaction. That would be by far the cleanest outcome here

         

        5. If you do NOT have a usable good backup

         

        That is a different situation. Before doing any destructive repair, I would first make a copy of the MDF/LDF files and preserve the current state, and then run:

        DBCC CHECKDB (N'YourDatabaseName')
        WITH NO_INFOMSGS, ALL_ERRORMSGS;
        GO

        If it still terminates because of the deferred transaction, the fun begins. The remaining recovery option might eventually involve emergency-mode repair:

        ALTER DATABASE [YourDatabaseName] SET EMERGENCY;
        GO
         
        ALTER DATABASE [YourDatabaseName]
        SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
        GO
         
        DBCC CHECKDB
        (
        N'YourDatabaseName',
        REPAIR_ALLOW_DATA_LOSS
        );
        GO

        But I would consider this a last-resort option only. REPAIR_ALLOW_DATA_LOSS can deallocate damaged pages or rebuild structures in a way that makes the database physically consistent while losing data. It is not equivalent to restoring the original data. So if a clean backup exists, restoring the affected page/database is strongly preferable.

         

        Good luck and fingers crossed this gets you unstuck. Let me know if it resolves the issue, or at least whether we’re making progress and the symptoms change.