SQL Server Database Corruption: Causes, Detection, and some details behind DBCC CHECKDB
SQL Server Database Corruption: Causes, Detection, and some details behind DBCC CHECKDB Database corruption in SQL Server is rare but usually high-impact. When it occurs, it threatens ACID compliance - the foundation of transactional integrity - and can lead to downtime, data loss, and operational risk. This article explores: Common causes of corruption How DBCC CHECKDB works under the hood Performance tuning tips for running CHECKDB Sample error messages and what they mean Best practices for prevention and recovery Why ACID Matters in Corruption Scenarios Before diving into causes and detection, remember that SQL Server guarantees Atomicity, Consistency, Isolation, and Durability: Atomicity: Transactions are all-or-nothing. If any part fails, the whole transaction has to fail. Corruption can break this, leaving partial writes. Consistency: Every transaction moves the database from one valid state to another. Corruption violates this by possibly introducing invalid states. Isolation: Concurrent transactions shouldn’t interfere with each other. Corruption in shared pages can cause phantom reads or deadlocks. Durability: Once committed, data must persist, even in the event of system failure or crash. Disk-level corruption undermines durability guarantees. DBCC CHECKDB exists primarily to validate consistency and durability, ensuring that the logical and physical structures adhere to ACID principles. In any internal decision-making that a database engine needs to do – ACID will be the main motivating factor, overriding concerns about performance, high availability or any other considerations. This is by design – ACID is the single most important principle that the engine cares about. Common Causes of SQL Server Database Corruption Corruption usually originates outside of SQL Server, often in the I/O path. Below I list the main causes, ordered in terms of probability - the higher on the list, the more probable it is: Hardware Failures Disk errors, RAID controller cache issues, or faulty RAM can corrupt pages during writes. Even with write-ahead logging, if the physical medium fails, durability is compromised. I/O Subsystem Issues SAN/NAS instability, outdated drivers, or virtualization misconfigurations can cause torn writes. SQL Server relies on the OS and storage stack for atomic page writes; instability breaks this assumption. Improper Shutdowns Power loss during write operations can leave pages partially written, violating atomicity. Torn-page detection mitigates this, but only if checksums are enabled. OS or SQL Server Bugs Rare, but missing cumulative updates can expose edge cases in buffer pool or checkpoint logic. File System Misconfiguration Compressed/encrypted volumes or sector size mismatches can corrupt allocation maps in edge cases. Human Error Manual deletion of MDF/LDF files or incorrect restore sequences can orphan pages in some unsupported scenarios. Malware Ransomware or malicious scripts altering system tables can break referential integrity. How DBCC CHECKDB Works Under the Hood DBCC CHECKDB is SQL Server’s integrity verification tool, validating both physical and logical consistency: 1. Snapshot Creation Creates a transactionally consistent snapshot using sparse files (page changes that happen at runtime are tracked in the files to ensure consistency). Ensures checks run without blocking user activity, preserving isolation, unless WITH TABLOCK option is used to run on live database taking locks during execution. 2. Phases of Execution Allocation Checks Validates GAM, SGAM, and IAM pages for correct page allocation. Detects orphaned extents or double allocations. System Table Checks Verifies metadata in system catalogs like sysobjects and sysindexes. Ensures schema-level consistency. Table and Index Structure Checks Traverses B-trees, validating key order and linkage. Detects broken pointers or incorrect page splits. Page-Level Validation Reads every page, checks checksums or torn-page bits. Critical for durability verification. LOB Checks Ensures integrity of large object chains in IAM and target pages (text, image, XML). Cross-Object Consistency Confirms referential integrity across tables and indexes. 3. Error Reporting Errors include severity and repair recommendation: REPAIR_REBUILD: Non-destructive, fixes structural issues. REPAIR_ALLOW_DATA_LOSS: Last resort, may delete corrupt pages – the integrity of the “repaired” database is not guaranteed when used. Performance Tuning Tips for DBCC CHECKDB CHECKDB is I/O and CPU intensive. Here are some options you can use to optimize its performance while running: Use PHYSICAL_ONLY for Faster Checks DBCC CHECKDB ('YourDatabase') WITH PHYSICAL_ONLY; Skips logical checks, reducing load. This could be used for daily scans, assuming a full scan still happens, just more rarely. Run on a Restored Copy Offload workload to a non-production server using recent backups. Leverage Availability Groups Execute on a readable secondary replica to protect the primary node. Control Parallelism DBCC CHECKDB ('YourDatabase') WITH MAXDOP = 4; Explicit MAXDOP ensures predictable performance. You can adjust the parallelism in accordance with available CPUs. Schedule During Low Activity Avoid peak hours; combine with Resource Governor for throttling. Break Down Large Databases Use DBCC CHECKTABLE for individual large tables if full CHECKDB is too costly. Sample Error Messages and Interpretation Msg 824 “SQL Server detected a logical consistency-based I/O error: incorrect checksum.” → Page-level corruption, often disk-related. Msg 8905 “Extent (1:12345) in database ID 5 is marked allocated in GAM, but not in SGAM.” → Allocation map inconsistency. Msg 2533 “Table error: Object ID 123456789, index ID 1. Page (1:98765) failed checksum.” → Corruption in index or data page. Msg 8928 “Object ID 123456789, index ID 2: Page (1:54321) could not be processed.” → Structural issues in index B-tree. Apart from the errors above, we also sometimes see issues directly related to our use of sparse files when creating the snapshot: The operating system returned error 665(The requested operation could not be completed due to a file system limitation) to SQL Server during a write at offset 0x00002a3ef96000 in file 'DBFile.mdf:MSSQL_DBCC18' The operating system returned error 1450 (Insufficient system resources exist to complete the requested service.) to SQL Server during a write at offset 0x00002a3ef96000 in file with handle 0x0000000000000D5C. This is usually a temporary condition and the SQL Server will keep retrying the operation. If the condition persists, then immediate action must be taken to correct it. Those are directly caused by NTFS File Record Segment objects running out of space due to ATTRIBUTE_LIST_ENTRY list growing out of bounds, and not by CHECKDB command itself - for more details check the article https://learn.microsoft.com/en-us/troubleshoot/sql/database-engine/database-file-operations/1450-and-665-errors-running-dbcc-checkdb. Prevention and Recovery Best Practices Run DBCC CHECKDB weekly either directly on the database or on restored backups. This makes sure that if corruption happens you have a recent last known good backup to restore from. Maintain verified backups with WITH CHECKSUM and regular restore tests. It’s very important to make sure a created backup can be fully restored to make sure there’s no unpleasant surprises if an issue happens. Use enterprise-grade hardware (RAID 10, ECC memory, UPS). Apply cumulative updates for SQL Server and Windows. Avoid unsupported storage configurations (compressed/deduplicated volumes). Key Takeaways Corruption is usually hardware or I/O related, not SQL Server bugs. DBCC CHECKDB is your first line of defence - schedule it regularly. Always restore from a clean backup if in any way possible; use repair options only as a last resort. Critical Note on Repair Options What triggered this blog was a number of issues we saw where the plan in case a database goes suspect was essentially to: Switch database to emergency mode Switch to single user mode Run DBCC CHECKDB with REPAIR_ALLOW_DATA_LOSS options without any previous tests to check if a corruption is present Switching the database back to multi user and using it normally Note that a database can go suspect for multiple reasons, not all related to corruption – basically if we hit something that stops the recovery process and doesn’t allow us to finish. For example – a deadlock with the recovery thread causes the thread to be killed? Database goes to suspect mode. This plan is not the recommended approach - neither in corruption cases nor other reasons for suspect database. Using REPAIR_ALLOW_DATA_LOSS can leave the database logically inconsistent. You always need to validate data post-repair and address root causes (hardware, OS issues) before attempting recovery. For detailed guidance, see https://learn.microsoft.com/en-us/troubleshoot/sql/database-engine/database-file-operations/troubleshoot-dbcc-checkdb-errors. Note that the repair recommendation provided after running is the lowest level of repair that can address all errors reported by CHECKDB. However, “minimum” doesn’t mean it will fix everything it finds - some errors simply can’t be repaired. You might also need to run the repair process more than once, since removing some data might expose additional non-linked pages that themselves needs to be dropped as part of repair. Keep in mind that not every error requires this level of repair and using REPAIR_ALLOW_DATA_LOSS doesn’t always lead to data loss. The only way to know if fixing an error will cause data loss is to actually run the repair and verify the data afterwards. A helpful tip: You can use DBCC CHECKTABLE on any table that shows errors. This will tell you the minimum repair level needed for that specific table. It’s extremely important to remember that after running CHECKDB repair with data loss, you must manually validate your data. The repair process doesn’t guarantee logical consistency. For example, REPAIR_ALLOW_DATA_LOSS might remove entire data pages with inconsistent data. If that happens, tables with foreign key relationships could end up with rows that no longer have matching parent keys. I hope this post was able to bring a bit of clarity to the complicated topic of corruption causes and recovery. It’s important to remember that once corruption happens your options are limited – you have to be prepared before the issue occurs. Further Reading https://learn.microsoft.com/en-us/sql/t-sql/database-console-commands/dbcc-checkdb-transact-sql https://learn.microsoft.com/en-us/troubleshoot/sql/database-engine/database-file-operations/troubleshoot-dbcc-checkdb-errors https://learn.microsoft.com/en-us/sql/sql-server/failover-clusters/automatic-page-repair-availability-groups-database-mirroring?view=sql-server-ver17
Recommendations for Index Maintenance with AlwaysOn Availability Groups
First published on MSDN on Mar 03, 2015 SYMPTOMSConsider the following scenarioThe database is part of AlwaysOn Availability GroupsYou run long and log-intensive transactions like Index maintenance/rebuildsYou observe one or more of the following symptoms:Poor performing DML operations in availability databases on the primary replica if synchronous secondary replicas are present.How It Works: Online Index Rebuild - Can Cause Increased Fragmentation
First published on MSDN on Sep 05, 2012 SQL Server Books Online alludes to the fragmentation possibility but does not fully explain that the Online Index rebuilding may increase fragmentation when it is allowed to run with MAX DOP > 1 and ALLOW_PAGE_LOCKS = OFF directives.Effectively troubleshoot latency in SQL Server Transactional replication: Part 1
Are you struggling with latency issues in SQL Server Transactional replication? This article provides clear, step-by-step instructions to effectively troubleshoot and resolve these challenges. Dive into proven techniques and best practices that will help you enhance your SQL Server's performance and ensure seamless data replication. Don't let latency slow you down—master the art of SQL Server troubleshooting today! I hope you find this teaser engaging! If you need any adjustments or additional content, feel free to let me know. Thanks to Collin Benkler, Senior Escalation Engineer in Microsoft for SQL Server for his valuable feedbacks.
How to troubleshoot “SQL Server is terminating because of fatal exception c06d007e”
Case 1, with Dump generated Issue definition: ======== Customer have SQL 2008/2012/2014 running on the same server, and then applied the SP4 for SQL Server 2012 in Windows update. But after this customer saw some issue from application side, then you rollback the update. And then Customer applied the SP3 for SQL Server 2012, but after that the SQL Server service can’t startup with following message and dump generated. Customer also tried to repair SQL Server many times, it doesn’t work. 2018-02-22 18:31:43.52 Server Error: 17311, Severity: 16, State: 1. 2018-02-22 18:31:43.52 Server SQL Server is terminating because of fatal exception c06d007e. This error may be caused by an unhandled Win32 or C++ exception, or by an access violation encountered during exception handling. Check the SQL error log for any related stack dumps or messages. This exception forces SQL Server to shutdown. To recover from this error, restart the server (unless SQLAgent is configured to auto restart). 2018-02-22 18:31:43.53 Server Using 'dbghelp.dll' version '4.0.5' 2018-02-22 18:31:43.53 Server **Dump thread - spid = 0, EC = 0x0000000000000000 2018-02-22 18:31:43.53 Server ***Stack Dump being sent to E:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\LOG\SQLDump0094.txt 2018-02-22 18:31:43.53 Server * ******************************************************************************* 2018-02-22 18:31:43.53 Server * 2018-02-22 18:31:43.53 Server * BEGIN STACK DUMP: 2018-02-22 18:31:43.53 Server * 02/22/18 18:31:43 spid 11352 2018-02-22 18:31:43.53 Server * 2018-02-22 18:31:43.53 Server * ex_handle_except encountered exception C06D007E - Server terminating Issue analysis and troubleshooting thinking ======== In the live meeting with customer, we first checking the setup log. And we saw that the SP4 has been applied, but after a while there is another setup folder generated reporting the failed error message: Exception type: Microsoft.SqlServer.Chainer.Infrastructure.ChainerInvalidOperationException Message: The NT service 'SQLSERVERAGENT' could not be started. HResult : 0x84b20001 FacilityCode : 1202 (4b2) ErrorCode : 1 (0001) Data: HelpLink.EvtType = 0xAE7B8BF3@0xF762992B@1202@1 DisableWatson = true Stack: at Microsoft.SqlServer.Configuration.SetupExtension.RunPatchAllInstanceAction.RestoreServiceStates() at Microsoft.SqlServer.Configuration.SetupExtension.RunPatchAllInstanceAction.ExecuteAction(String actionId) at Microsoft.SqlServer.Chainer.Infrastructure.Action.Execute(String actionId, TextWriter errorStream) at Microsoft.SqlServer.Setup.Chainer.Workflow.ActionInvocation.ExecuteActionHelper(TextWriter statusStream, ISequencedAction actionToRun, ServiceContainer context) Then we get the dump file from customer and performed a detailed analysis . This callstack seems that we have issue while Initialize Server Components(InitServerComponents), and then we get the detailed component name “SqlServerSpatial110.dll” # Child-SP RetAddr Call Site 00 00000000`01bf9cf8 000007fe`fd3810ac ntdll!ZwWaitForSingleObject+0xa 01 00000000`01bf9d00 00000000`ff6af3ee KERNELBASE!WaitForSingleObjectEx+0x79 02 00000000`01bf9da0 00000000`ff6af874 sqlservr!CDmpDump::DumpInternal+0x20e 03 00000000`01bf9e40 00000000`ff6af919 sqlservr!CDmpDump::DumpFilter+0x64 04 00000000`01bf9f10 00000000`74d712e3 sqlservr!`CDmpDump::Dump'::`1'::filt$0+0x19 05 00000000`01bf9f40 00000000`7769bf9d msvcr100!__C_specific_handler+0x97 06 00000000`01bf9fb0 00000000`776704ca ntdll!RtlpExecuteHandlerForException+0xd 07 00000000`01bf9fe0 00000000`77673c1f ntdll!RtlDispatchException+0x45a 08 00000000`01bfa6c0 000007fe`fd39a06d ntdll!RtlRaiseException+0x22f 09 00000000`01bfb070 00000000`ff6af8ec KERNELBASE!RaiseException+0x39 0a 00000000`01bfb140 000007fe`cd6bd4c4 sqlservr!CDmpDump::Dump+0x4c 0b 00000000`01bfb180 000007fe`cd6bdf06 sqllang!SQLDumperLibraryInvoke+0x2e4 0c 00000000`01bfb420 000007fe`cd684b6b sqllang!CImageHelper::DoMiniDump+0x426 0d 00000000`01bfb610 000007fe`cd683f6a sqllang!stackTrace+0xbdb 0e 00000000`01bfd050 000007fe`e0312a75 sqllang!stackTraceCallBack+0xca 0f 00000000`01bfd090 00000000`775abc20 sqldk!ex_handle_except+0x125 10 00000000`01bfd2e0 00000000`776a9035 kernel32!UnhandledExceptionFilter+0x160 11 00000000`01bfd3c0 00000000`77687398 ntdll!RtlUserThreadStart$filt$0+0x45 12 00000000`01bfd3f0 00000000`7769bf9d ntdll!__C_specific_handler+0x8c 13 00000000`01bfd460 00000000`776704ca ntdll!RtlpExecuteHandlerForException+0xd 14 00000000`01bfd490 00000000`77673c1f ntdll!RtlDispatchException+0x45a 15 00000000`01bfdb70 000007fe`fd39a06d ntdll!RtlRaiseException+0x22f 16 00000000`01bfe520 000007fe`ccb7bc0b KERNELBASE!RaiseException+0x39 17 00000000`01bfe5f0 000007fe`cdb8e40f sqllang!__delayLoadHelper2+0x199 18 00000000`01bfe6a0 000007fe`cdb49a75 sqllang!_tailMerge_SqlServerSpatial110_dll+0x3f 19 (Inline Function) --------`-------- sqllang!InitializeSpatialNativeImpl+0x26 1a 00000000`01bfe710 00000000`ff69af2a sqllang!SQLLangInitComponents+0x275 1b 00000000`01bfe770 00000000`ff69c094 sqlservr!InitServerComponents+0x8a 1c 00000000`01bfe7d0 00000000`ff693c48 sqlservr!sqlservr_main+0x3b4 1d 00000000`01bfea20 000007fe`fe32a82d sqlservr!sqlservr_main_thread+0x178 1e 00000000`01bffd30 00000000`775259cd sechost!ScSvcctrlThreadA+0x25 1f 00000000`01bffd60 00000000`7768383d kernel32!BaseThreadInitThunk+0xd 20 00000000`01bffd90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d Looked at the DLL being loaded szDll : 0x7fecdeef9c0 : "SqlServerSpatial110.dll" [Type: char *] Then we checked the SqlServerSpatial110.dll both in C:\Windows\System32 and C:\Windows\SystemWOW64. System32 WOW64: And then we checked this SqlServerSpatial110.dll properties in another working server as below. Then we got the root cause that the current server SqlServerSpatial110.dll original filename is SqlServerSpatial120.dll(it is SQL 2014). But in this case, somehow is has been changed, but we don’t know why, we only get to the result. Then we tried to copy the correct one from another server to current server(system32 and systemWOW64 respectively). But this doesn’t work. After we restart the SQL Server it failed again, with same error. Then we come up with another solution, we can rename the current SqlServerSpatial110.dll to SqlServerSpatial110.dll.old and then launch the repair again. This worked. After repairing, we can see the new SqlServerSpatial110.dll has been created automatically with correct SP3 version(same with the current SQL Server version) Root cause: ======== Somehow, SqlServerSpatial110.dll has been changed. Resolution: ======== Rename the SqlServerSpatial110.dll under C:\Windows\System32 and C:\Windows\SystemWOW64 Launch the repair for SQL Server 2012. This resolved this issue successfully. Case 2, there is no dump generated Issue definition: ======== After you patch the SQL Server 2016, 9 instances failed to start due to the following error. 2020-02-22 18:43:17.14 Server Error: 17311, Severity: 16, State: 1. 2020-02-22 18:43:17.14 Server SQL Server is terminating because of fatal exception c06d007e. This error may be caused by an unhandled Win32 or C++ exception, or by an access violation encountered during exception handling. Check the SQL error log for any related stack dumps or messages. This exception forces SQL Server to shutdown. To recover from this error, restart the server (unless SQLAgent is configured to auto restart). 2020-02-22 18:43:17.14 Server Using 'dbghelp.dll' version '4.0.5' 2020-02-22 18:43:17.14 Server **Dump thread - spid = 0, EC = 0x0000000000000000 2020-02-22 18:43:17.14 Server * ******************************************************************************* 2020-02-22 18:43:17.14 Server * 2020-02-22 18:43:17.14 Server * BEGIN STACK DUMP: 2020-02-22 18:43:17.14 Server * 02/22/20 18:43:17 spid 8832 2020-02-22 18:43:17.14 Server * 2020-02-22 18:43:17.14 Server * ex_handle_except encountered exception C06D007E - Server terminating 2020-02-22 18:43:17.22 Server Stack Signature for the dump is 0x00000001A124A749 2020-02-22 18:43:17.22 Server Unable to create dump because SQLDUMPER library is not available. 2020-02-22 18:43:17.22 Server SQL Server shutdown has been initiated Issue analysis: ======== Based on the log, this issue should happened when loading some DLL. But as there no dump generated due to Unable to create dump because SQLDUMPER library is not available(from the following analysis, we can also see that dbgheko.dll not loaded yet). So we captured a process monitor, and then analyze the process monitor log. From the process monitor log, we can’t get the callstack which can help us to identify the DLL. So we are not able to find the DLL from the callstack, but we can compare the DLL which has already been loaded with a working server, and find out all the non-loaded DLL, and maybe we can find more clue. Compared the working server loaded DLL and this problematic server. All the yellow-mark DLL are non-loaded DLL in the problematic SQL. Working Server: <<<<<<<<<<<<<<< SQLNCLIRDAR11.RLL 0x18261ee0000 0x38000 c:\Program Files\Microsoft SQL Server\140\Shared\1033\SQLNCLIRDAR11.RLL Microsoft Corporation 2011.0110.5069.066 ((SQL11_OLEDB_AAD).160321-0812 ) 3/21/2016 11:56:15 PM instapi140.dll 0x7ffc21c80000 0x13000 c:\Program Files\Microsoft SQL Server\140\Shared\instapi140.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:36:59 AM sqlnclirda11.dll 0x59e10000 0x359000 c:\Program Files\Microsoft SQL Server\140\Shared\sqlnclirda11.dll Microsoft Corporation 2011.0110.5069.066 ((SQL11_OLEDB_AAD).160321-0812 ) 3/21/2016 11:57:40 PM BatchParser.dll 0x7ffbf7990000 0x2c000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\BatchParser.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:36:47 AM dbghelp.dll 0x59c70000 0x196000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\dbghelp.dll Microsoft Corporation 6.12.0002.633 (debuggers(dbg).100201-1211) 2/2/2010 4:15:44 AM ftimport.dll 0x60000000 0x25000 c:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\ftimport.dll Microsoft Corporation 12.0.7031.9999 11/1/2006 12:27:20 PM hkcompile.dll 0x7ffbf86f0000 0x162000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\hkcompile.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:47:36 AM hkengine.dll 0x7ffbf7fe0000 0x70b000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\hkengine.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:51:33 AM hkruntime.dll 0x7ffbf8860000 0x2d1000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\hkruntime.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:52:55 AM MSFTE.DLL 0x49980000 0x3ae000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\MSFTE.DLL Microsoft Corporation 12.0.6828.0 8/28/2006 7:15:55 PM opends60.dll 0x7ffbf8cb0000 0x9000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\opends60.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:36:08 AM qds.dll 0x7ffbf8b80000 0x12e000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\qds.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:38:39 AM sqlevn70.rll 0x17cc64a0000 0x310000 c:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\Resources\1033\sqlevn70.rll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:36:13 AM xplog70.RLL 0x183ea3d0000 0x4000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\Resources\1033\xplog70.RLL Microsoft Corporation 2017.0140.1000.169 ((SQLServer).170822-2340) 8/23/2017 7:50:34 AM xpstar.rll 0x183e93a0000 0xd000 c:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\Resources\1033\xpstar.rll Microsoft Corporation 2017.0140.1000.169 ((SQLServer).170822-2340) 8/23/2017 7:50:29 AM secforwarder.dll 0x7ffbf8b40000 0xa000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\secforwarder.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:45:38 AM SqlAccess.dll 0x7ffbf7f60000 0x75000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\SqlAccess.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:37:39 AM sqldk.dll 0x7ffbf8cc0000 0x4fc000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\sqldk.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:35:28 AM sqllang.dll 0x7ffbf9af0000 0x2782000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\sqllang.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:44:02 AM sqlmin.dll 0x7ffbfc280000 0x31fe000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\sqlmin.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:38:05 AM SQLOS.dll 0x7ffbff480000 0x7000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\SQLOS.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:45:34 AM SQLSCM.dll 0x7ffbf7e30000 0x13000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\SQLSCM.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:40:48 AM sqlservr.exe 0x7ff7869e0000 0x7f000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\sqlservr.exe Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:35:24 AM sqlTsEs.dll 0x7ffbf91c0000 0x92f000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\sqlTsEs.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:35:33 AM svl.dll 0x7ffbf8b50000 0x2e000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\svl.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:57:13 AM xplog70.dll 0x7ffc2ea50000 0x14000 c:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\xplog70.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:55:14 AM xpsqlbot.dll 0x7ffbf7cd0000 0x8000 C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\xpsqlbot.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:38:05 AM xpstar.dll 0x7ffbf7c60000 0x6e000 c:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\xpstar.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:47:53 AM Non_working Server <<<<<<<<<<<<<<< HKRUNTIME.DLL 0x7fffea270000 0x2d0000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\HKRUNTIME.DLL 1/1/1970 8:00:00 AM OPENDS60.DLL 0x7ffffe550000 0x9000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\OPENDS60.DLL 1/1/1970 8:00:00 AM QDS.DLL 0x7fffeac80000 0xdc000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\QDS.DLL 1/1/1970 8:00:00 AM SQLEVN70.RLL 0x7fffe5180000 0x2f0000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\RESOURCES\1033\SQLEVN70.RLL 1/1/1970 8:00:00 AM SQLBOOT.DLL 0x7fffeb230000 0x2e000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\SQLBOOT.DLL 1/1/1970 8:00:00 AM SQLDK.DLL 0x7fffe97b0000 0x4c8000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\SQLDK.DLL 1/1/1970 8:00:00 AM SQLLANG.DLL 0x7fffddf90000 0x25c7000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\SQLLANG.DLL 1/1/1970 8:00:00 AM SQLMIN.DLL 0x7fffe0560000 0x24fa000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\SQLMIN.DLL 1/1/1970 8:00:00 AM SQLOS.DLL 0x7fffff390000 0x7000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\SQLOS.DLL 1/1/1970 8:00:00 AM SQLSERVR.EXE 0x7ff7c29d0000 0x68000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\SQLSERVR.EXE 1/1/1970 8:00:00 AM SQLTSES.DLL 0x7fffe5470000 0x88b000 \Device\CSVVolume137\MSSQL13.KCOM1S130EDAM01\MSSQL\BINN\SQLTSES.DLL 1/1/1970 8:00:00 AM We can see HKRUNTIME already loaded in our problematic SQL, so the next most possible DLL should be the HKEngine.dll HKCompile.dll. Then check them in your environment, they do have some problem in those DLLs, for example the size/the modified date. Then we rename those DLL, for example HKEngine.dll.Old, and then run repair SQL Server, and after that we confirmed 1 instance start successfully. Solution: ======== HKEngine.dll HKCompile.dll has some problem when SQL Server trying to load them. So rename them and then run SQL Repair, and new DLL will be generated, then SQL Server start successfully.
