%3CLINGO-SUB%20id%3D%22lingo-sub-1427906%22%20slang%3D%22en-US%22%3ESQL%20CLR%20.NET%20Framework%20runtime%20fatal%20error%20due%20to%20Uneven%20Number%20of%20CPUs%20Across%20Processor%20Groups%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1427906%22%20slang%3D%22en-US%22%3E%3CH1%20id%3D%22toc-hId-468436663%22%20id%3D%22toc-hId-468436663%22%3EProblem%3C%2FH1%3E%0A%3CP%3EIf%20you're%20using%20SQL%20CLR%2C%20you%20may%20have%20encountered%20an%20unexpected%20SQL%20Server%20service%20termination%20issue%20on%20SQL%20Servers%20with%20a%20large%20number%20of%20CPUs.%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId--1339017800%22%20id%3D%22toc-hId--1339017800%22%3ESymptoms%3C%2FH1%3E%0A%3CP%3ESQL%20Server%20service%20terminates%20unexpectedly.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20the%20ERRORLOG%2C%20you'll%20see%20the%20following%20message%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CEM%3E%3CDATETIME%3E%3C%2FDATETIME%3E%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3EServer%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20*%20*******************************************************************************%3C%2FP%3E%0A%3CP%3E%3CEM%3E%3CDATETIME%3E%20%3C%2FDATETIME%3E%3C%2FEM%3EServer%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20*%3C%2FP%3E%0A%3CP%3E%3CEM%3E%3CDATETIME%3E%3C%2FDATETIME%3E%3C%2FEM%3E%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20*%20BEGIN%20STACK%20DUMP%3A%3C%2FP%3E%0A%3CP%3E%3CEM%3E%3CDATETIME%3E%3C%2FDATETIME%3E%3C%2FEM%3E%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20*%26nbsp%3B%26nbsp%3B%2004%2F27%2F20%2010%3A30%3A16%20spid%2038076%3C%2FP%3E%0A%3CP%3E%3CEM%3E%3CDATETIME%3E%3C%2FDATETIME%3E%3C%2FEM%3E%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20*%3C%2FP%3E%0A%3CP%3E%3CEM%3E%3CDATETIME%3E%3C%2FDATETIME%3E%3C%2FEM%3E%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20*%20A%20fatal%20error%20occurred%20in%20.NET%20Framework%20runtime.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EIn%20addition%2C%20a%20memory%20dump%20will%20be%20created%3C%2FLI%3E%0A%3CLI%3EThe%20last%20message%20in%20ERRORLOG%20will%20show%20something%20like%3A%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CEM%3E%3CDATETIME%3E%3C%2FDATETIME%3E%3C%2FEM%3E%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Error%3A%206536%2C%20Severity%3A%2016%2C%20State%3A%201.%3C%2FP%3E%0A%3CP%3E%3CEM%3E%3CDATETIME%3E%3C%2FDATETIME%3E%3C%2FEM%3E%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20A%20fatal%20error%20occurred%20in%20the%20.NET%20Framework%20common%20language%20runtime.%20SQL%20Server%20is%20shutting%20down.%20If%20the%20error%20recurs%20after%20the%20server%20is%20restarted%2C%20contact%20Customer%20Support%20Services.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId-1148495033%22%20id%3D%22toc-hId-1148495033%22%3ECause%3C%2FH1%3E%0A%3CP%3ENot%20all%20fatal%20errors%20in%20.NET%20Framework%20runtime%20will%20be%20caused%20by%20same%20issue%2C%20but%20it%20is%20possibly%20due%20an%20issue%20with%20Threadpool%20Manager%20in%20handling%20large%20number%20of%20threads.%20If%20you're%20able%20to%20open%20the%20dump%20file%2C%20you'll%20see%20a%20stack%20like%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E1a%20ntdll!RtlDispatchException%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E1b%20ntdll!KiUserExceptionDispatch%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CFONT%20color%3D%22%23FF0000%22%3E1c%20clr!ThreadpoolMgr%3A%3AGetCPUBusyTime_NT%3C%2FFONT%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E1d%20clr!ThreadpoolMgr%3A%3AGateThreadStart%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E1e%20kernel32!BaseThreadInitThunk%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E1f%20ntdll!RtlUserThreadStart%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EMore%20information%20about%20%3CSTRONG%3EThreadpoolMgr%3A%3AGetCPUBusyTime_NT%3C%2FSTRONG%3E%20can%20be%20found%20at%20GitHub%20-ERR%3AREF-NOT-FOUND-Win32ThreadPool.cpp.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CFONT%20size%3D%225%22%3EIMPORTANT%20-%20The%20issue%20can%20occur%20when%20the%20following%20conditions%20are%20present%3A%3C%2FFONT%3E%3C%2FP%3E%0A%3CP%3E%3CFONT%20color%3D%22%23FF0000%22%3ENumber%20of%20processors%20in%20the%20NUMA%20node%20is%20unevenly%20distributed%20across%20nodes%3C%2FFONT%3E%3C%2FP%3E%0A%3CP%3E%3CFONT%20color%3D%22%23FF0000%22%3ENumber%20of%20processors%20in%20each%20NUMA%20node%20is%2040%20or%20more%3C%2FFONT%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe've%20observed%20this%20on%20multiple%20instances%20with%20systems%20with%20160%20processors.%20At%20the%20top%20of%20SQL%20Server%20ERRORLOG%20is%20a%20logging%20of%20the%20NUMA%20node%20configuration%20for%20the%20CPUs%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%7Bdatetime%7D%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Node%20configuration%3A%20node%200%3A%20CPU%20mask%3A%200x00000000000fffff%3A%3CFONT%20color%3D%22%23000000%22%3E0%20A%3C%2FFONT%3Ective%20CPU%20mask%3A%200x00000000000fffff%3A0.%20This%20message%20provides%20a%20description%20of%20the%20NUMA%20configuration%20for%20this%20computer.%20This%20is%20an%20informational%20message%20only.%20No%20user%20action%20is%20required.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%7Bdatetime%7D%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Node%20configuration%3A%20node%201%3A%20CPU%20mask%3A%200x00000000000fffff%3A1%20Active%20CPU%20mask%3A%200x00000000000fffff%3A1.%20This%20message%20provides%20a%20description%20of%20the%20NUMA%20configuration%20for%20this%20computer.%20This%20is%20an%20informational%20message%20only.%20No%20user%20action%20is%20required.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%7Bdatetime%7D%20Server%20%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3BNode%20configuration%3A%20node%202%3A%20CPU%20mask%3A%200x00000000000fffff%3A2%20Active%20CPU%20mask%3A%200x00000000000fffff%3A2.%20This%20message%20provides%20a%20description%20of%20the%20NUMA%20configuration%20for%20this%20computer.%20This%20is%20an%20informational%20message%20only.%20No%20user%20action%20is%20required.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%7Bdatetime%7D%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Node%20configuration%3A%20node%203%3A%20CPU%20mask%3A%200x000000fffff00000%3A1%20Active%20CPU%20mask%3A%200x000000fffff00000%3A1.%20This%20message%20provides%20a%20description%20of%20the%20NUMA%20configuration%20for%20this%20computer.%20This%20is%20an%20informational%20message%20only.%20No%20user%20action%20is%20required.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%7Bdatetime%7D%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Node%20configuration%3A%20node%204%3A%20CPU%20mask%3A%200x000000fffff000%3CFONT%20color%3D%22%23000000%22%3E00%3A0%20A%3C%2FFONT%3Ective%20CPU%20mask%3A%200x000000fffff00000%3A0.%20This%20message%20provides%20a%20description%20of%20the%20NUMA%20configuration%20for%20this%20computer.%20This%20is%20an%20informational%20message%20only.%20No%20user%20action%20is%20required.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%7Bdatetime%7D%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Node%20configuration%3A%20node%205%3A%20CPU%20mask%3A%200x000000fffff00000%3A2%20Active%20CPU%20mask%3A%200x000000fffff00000%3A2.%20This%20message%20provides%20a%20description%20of%20the%20NUMA%20configuration%20for%20this%20computer.%20This%20is%20an%20informational%20message%20only.%20No%20user%20action%20is%20required.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%7Bdatetime%7D%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Node%20configuration%3A%20node%206%3A%20CPU%20mask%3A%200x0fffff0000000000%3A1%20Active%20CPU%20mask%3A%200x0fffff0000000000%3A1.%20This%20message%20provides%20a%20description%20of%20the%20NUMA%20configuration%20for%20this%20computer.%20This%20is%20an%20informational%20message%20only.%20No%20user%20action%20is%20required.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%7Bdatetime%7D%20Server%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Node%20configuration%3A%20node%207%3A%20CPU%20mask%3A%200x0fffff0000000%3CFONT%20color%3D%22%23000000%22%3E000%3A0%20Ac%3C%2FFONT%3Etive%20CPU%20mask%3A%200x0fffff0000000000%3A0.%20This%20message%20provides%20a%20description%20of%20the%20NUMA%20configuration%20for%20this%20computer.%20This%20is%20an%20informational%20message%20only.%20No%20user%20action%20is%20required.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20we%20look%20at%20the%20NUMA%20node%20masking%20by%20CPU%20group%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3ENode%20configuration%3A%20node%200%3A%20CPU%20mask%3A%200x00000000000fffff%3A0%20Active%20CPU%20mask%3A%200x00000000000fffff%3A%3CFONT%20color%3D%22%23FF6600%22%3E0%3C%2FFONT%3E.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3ENode%20configuration%3A%20node%201%3A%20CPU%20mask%3A%200x00000000000fffff%3A1%20Active%20CPU%20mask%3A%200x00000000000fffff%3A%3CFONT%20color%3D%22%23008000%22%3E1%3C%2FFONT%3E.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3ENode%20configuration%3A%20node%202%3A%20CPU%20mask%3A%200x00000000000fffff%3A2%20Active%20CPU%20mask%3A%200x00000000000fffff%3A%3CFONT%20color%3D%22%230000FF%22%3E2%3C%2FFONT%3E.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3ENode%20configuration%3A%20node%203%3A%20CPU%20mask%3A%200x000000fffff00000%3A1%20Active%20CPU%20mask%3A%200x000000fffff00000%3A%3CFONT%20color%3D%22%23008000%22%3E1%3C%2FFONT%3E.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3ENode%20configuration%3A%20node%204%3A%20CPU%20mask%3A%200x000000fffff00000%3A0%20Active%20CPU%20mask%3A%200x000000fffff00000%3A%3CFONT%20color%3D%22%23FF6600%22%3E0%3C%2FFONT%3E.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3ENode%20configuration%3A%20node%205%3A%20CPU%20mask%3A%200x000000fffff00000%3A2%20Active%20CPU%20mask%3A%200x000000fffff00000%3A%3CFONT%20color%3D%22%230000FF%22%3E2%3C%2FFONT%3E.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3ENode%20configuration%3A%20node%206%3A%20CPU%20mask%3A%200x0fffff0000000000%3A1%20Active%20CPU%20mask%3A%200x0fffff0000000000%3A%3CFONT%20color%3D%22%23008000%22%3E1%3C%2FFONT%3E.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3ENode%20configuration%3A%20node%207%3A%20CPU%20mask%3A%200x0fffff0000000000%3A0%20Active%20CPU%20mask%3A%200x0fffff0000000000%3A%3CFONT%20color%3D%22%23FF6600%22%3E0%3C%2FFONT%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EEach%20'f'%20in%20the%20mask%20indicates%204%20CPUs.%20Therefore%2C%20the%20above%20translates%20to%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CFONT%20color%3D%22%23FF6600%22%3EProcessor%20Group%200%3C%2FFONT%3E%20has%205*4*3%3D60%20CPUs%3C%2FLI%3E%0A%3CLI%3E%3CFONT%20color%3D%22%23008000%22%3EProcessor%20Group%201%3C%2FFONT%3E%20has%205*4*3%3D60%20CPUs%3C%2FLI%3E%0A%3CLI%3E%3CFONT%20color%3D%22%230000FF%22%3EProcessor%20Group%202%3C%2FFONT%3E%20has%205*4*2%3D40%20CPUs%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20problem%20arises%20when%20there%20is%20a%20mismatch%20between%20the%20number%20of%20CPUs%20in%20the%20SQL%20CLR%20Host%20configuration%20and%20the%20number%20of%20CPUs%20in%20the%20SQL%20Server%20Engine%20for%20a%20given%20processor%20group.%20The%20exception%20and%20SQL%20Server%20service%20crash%20happens%20when%20CLR%20starts%20a%20new%20thread.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20you%20want%20to%20check%20the%20processor%20mapping%20on%20your%20system%2C%20you%20can%20use%20Windows%20Sysinternals%20tool%20-ERR%3AREF-NOT-FOUND-Coreinfo%20v3.5.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId--658959430%22%20id%3D%22toc-hId--658959430%22%3EResolution%3C%2FH1%3E%0A%3CP%3EThis%20issue%20was%20fixed%20in%20.NET%20Framework%20versions%204.6%20and%20beyond.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CSTRONG%3EPreview%20of%20Quality%20Rollup%20for%20.NET%20Framework%204.8%20for%20Windows%208.1%2C%20RT%208.1%2C%20and%20Windows%20Server%202012%20R2%20(KB4537482)%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fsupport.microsoft.com%252Fen-us%252Fhelp%252F4537482%252Fkb4537482%26amp%3Bdata%3D02%257C01%257CNathan.Schoenack%2540microsoft.com%257C1eabd60fb0ba4cf709c808d806496996%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637266259657568346%26amp%3Bsdata%3DihTifAyol3SyVOZjICWzF%252FN5tXxU4uSdIQ7rFeX%252FWrs%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Fhelp%2F4537482%2Fkb4537482%3C%2FA%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CSTRONG%3EPreview%20of%20Quality%20Rollup%20for%20.NET%20Framework%204.6%2C%204.6.1%2C%204.6.2%2C%204.7%2C%204.7.1%2C%204.7.2%20for%20Windows%208.1%2C%20RT%208.1%2C%20and%20Windows%20Server%202012%20R2%20(KB4537488)%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fsupport.microsoft.com%252Fen-us%252Fhelp%252F4537488%252Fkb4537488%26amp%3Bdata%3D02%257C01%257CNathan.Schoenack%2540microsoft.com%257C1eabd60fb0ba4cf709c808d806496996%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637266259657578344%26amp%3Bsdata%3DzTTt2JDecNcOM3qMifUSBYjZ1WXgrr6eL1KEctoVhBA%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Fhelp%2F4537488%2Fkb4537488%3C%2FA%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CSTRONG%3EOther%20versions%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fdevblogs.microsoft.com%252Fdotnet%252Fnet-framework-february-2020-preview-of-quality-rollup%252F%26amp%3Bdata%3D02%257C01%257CNathan.Schoenack%2540microsoft.com%257C1eabd60fb0ba4cf709c808d806496996%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637266259657588335%26amp%3Bsdata%3DwqkH28phxfowsSz912ZSwLfK9v3if7YlISq%252B0PbvgqI%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdevblogs.microsoft.com%2Fdotnet%2Fnet-framework-february-2020-preview-of-quality-rollup%2F%3C%2FA%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20the%20version%20of%20.NET%20Framework%20on%20your%20system%20is%20less%20than%204.6%2C%20you%20will%20need%20to%20upgrade%20to%20at%20least%20.NET%20Framework%204.6%20to%20get%20the%20fix.%20We%20recommend%20that%20update%20to%20.NET%20Framework%204.8.%20If%20you're%20unable%20to%20upgrade%2C%20please%20see%20the%20workarounds%20below.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId-1828553403%22%20id%3D%22toc-hId-1828553403%22%3EWorkaround%3C%2FH1%3E%0A%3CP%3ENote%3A%20the%20workaround%20will%20depend%20on%20the%20number%20of%20logical%20processors%20for%20the%20individual%20machine.%20The%20following%20is%20an%20example%20based%20on%20the%20160%20logical%20CPU%20machine%20used%20as%20example%20above.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThere's%20two%20workarounds%20we%20have%20identified%20for%20this%20issue.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3E1st%20Workaround%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EIssue%20doesn't%20happen%20after%20disabling%20the%20hyper-threading.%20Once%20after%20disabling%20hyper-threading%2C%20there%20will%20be%202%20different%20processor%20groups%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3EProcessor%20Group%200%20has%2040%20Logical%20Processors%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3EProcessor%20Group%201%20has%2040%20Logical%20Processors%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EUnder%20each%20Group%2C%20we%20have%2040%20processors%2C%20so%20there%20is%20no%20uneven%20distribution%20across%20processor%20groups%20and%20the%20issue%20doesn't%20reoccur.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3E2nd%20Workaround%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E-ERR%3AREF-NOT-FOUND-BCDEdit%20is%20a%20command-line%20tool%20for%20managing%20Boot%20Configuration%20Data%20(BCD).%20This%20tool%20allows%20you%20to%20edit%20the%20size%20of%20the%20processor%20group.%20In%20the%20scenario%20described%20above%20with%20160%20processors%2C%20you%20can%20execute%20the%20following%20command%20to%20create%20an%20evenly%20distributed%20number%20of%20CPUs%20across%20the%20processor%20groups%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3Ebcdedit.exe%20%2Fset%20groupsize%2040%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20need%20to%20reboot%20the%20server%20after%20running%20above%20command.%20After%20this%2C%20you'll%20see%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3EProcessor%20Group%200%20has%2040%20Logical%20Processors%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3EProcessor%20Group%201%20has%2040%20Logical%20Processors%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3EProcessor%20Group%202%20has%2040%20Logical%20Processors%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3EProcessor%20Group%203%20has%2040%20Logical%20Processors%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENathan%20Schoenack%3C%2FP%3E%0A%3CP%3ESupport%20Escalation%20Engineer%20-%26nbsp%3BSQL%20Server%20Engine%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1427906%22%20slang%3D%22en-US%22%3E%3CP%3EIf%20you're%20using%20SQL%20CLR%2C%20you%20may%20have%20encountered%20an%20unexpected%20SQL%20Server%20service%20termination%20issue%20on%20SQL%20Servers%20with%20a%20large%20number%20of%20CPUs.%3C%2FP%3E%3C%2FLINGO-TEASER%3E
Microsoft

Problem

If you're using SQL CLR, you may have encountered an unexpected SQL Server service termination issue on SQL Servers with a large number of CPUs.

Symptoms

SQL Server service terminates unexpectedly.

 

In the ERRORLOG, you'll see the following message:

 

<datetime>

Server      * *******************************************************************************

<datetime> Server      *

<datetime> Server      * BEGIN STACK DUMP:

<datetime> Server      *   04/27/20 10:30:16 spid 38076

<datetime> Server      *

<datetime> Server      * A fatal error occurred in .NET Framework runtime.

 

  • In addition, a memory dump will be created
  • The last message in ERRORLOG will show something like:

 

<datetime> Server      Error: 6536, Severity: 16, State: 1.

<datetime> Server      A fatal error occurred in the .NET Framework common language runtime. SQL Server is shutting down. If the error recurs after the server is restarted, contact Customer Support Services.

 

Cause

Not all fatal errors in .NET Framework runtime will be caused by same issue, but it is possibly due an issue with Threadpool Manager in handling large number of threads. If you're able to open the dump file, you'll see a stack like:

 

1a ntdll!RtlDispatchException

1b ntdll!KiUserExceptionDispatch

1c clr!ThreadpoolMgr::GetCPUBusyTime_NT

1d clr!ThreadpoolMgr::GateThreadStart

1e kernel32!BaseThreadInitThunk

1f ntdll!RtlUserThreadStart

 

More information about ThreadpoolMgr::GetCPUBusyTime_NT can be found at GitHub Win32ThreadPool.cpp.

 

IMPORTANT - The issue can occur when the following conditions are present:

Number of processors in the NUMA node is unevenly distributed across nodes

Number of processors in each NUMA node is 40 or more

 

We've observed this on multiple instances with systems with 160 processors. At the top of SQL Server ERRORLOG is a logging of the NUMA node configuration for the CPUs:

 

{datetime} Server      Node configuration: node 0: CPU mask: 0x00000000000fffff:0 Active CPU mask: 0x00000000000fffff:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

{datetime} Server      Node configuration: node 1: CPU mask: 0x00000000000fffff:1 Active CPU mask: 0x00000000000fffff:1. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

{datetime} Server      Node configuration: node 2: CPU mask: 0x00000000000fffff:2 Active CPU mask: 0x00000000000fffff:2. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

{datetime} Server      Node configuration: node 3: CPU mask: 0x000000fffff00000:1 Active CPU mask: 0x000000fffff00000:1. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

{datetime} Server      Node configuration: node 4: CPU mask: 0x000000fffff00000:0 Active CPU mask: 0x000000fffff00000:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

{datetime} Server      Node configuration: node 5: CPU mask: 0x000000fffff00000:2 Active CPU mask: 0x000000fffff00000:2. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

{datetime} Server      Node configuration: node 6: CPU mask: 0x0fffff0000000000:1 Active CPU mask: 0x0fffff0000000000:1. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

{datetime} Server      Node configuration: node 7: CPU mask: 0x0fffff0000000000:0 Active CPU mask: 0x0fffff0000000000:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

 

If we look at the NUMA node masking by CPU group:

 

Node configuration: node 0: CPU mask: 0x00000000000fffff:0 Active CPU mask: 0x00000000000fffff:0.

Node configuration: node 1: CPU mask: 0x00000000000fffff:1 Active CPU mask: 0x00000000000fffff:1.

Node configuration: node 2: CPU mask: 0x00000000000fffff:2 Active CPU mask: 0x00000000000fffff:2.

Node configuration: node 3: CPU mask: 0x000000fffff00000:1 Active CPU mask: 0x000000fffff00000:1.

Node configuration: node 4: CPU mask: 0x000000fffff00000:0 Active CPU mask: 0x000000fffff00000:0.

Node configuration: node 5: CPU mask: 0x000000fffff00000:2 Active CPU mask: 0x000000fffff00000:2.

Node configuration: node 6: CPU mask: 0x0fffff0000000000:1 Active CPU mask: 0x0fffff0000000000:1.

Node configuration: node 7: CPU mask: 0x0fffff0000000000:0 Active CPU mask: 0x0fffff0000000000:0.

 

Each 'f' in the mask indicates 4 CPUs. Therefore, the above translates to:

  • Processor Group 0 has 5*4*3=60 CPUs
  • Processor Group 1 has 5*4*3=60 CPUs
  • Processor Group 2 has 5*4*2=40 CPUs

 

The problem arises when there is a mismatch between the number of CPUs in the SQL CLR Host configuration and the number of CPUs in the SQL Server Engine for a given processor group. The exception and SQL Server service crash happens when CLR starts a new thread.

 

If you want to check the processor mapping on your system, you can use Windows Sysinternals tool Coreinfo v3.5.

 

Resolution

This issue was fixed in .NET Framework versions 4.6 and beyond.

 

Preview of Quality Rollup for .NET Framework 4.8 for Windows 8.1, RT 8.1, and Windows Server 2012 R2 (KB4537482)

https://support.microsoft.com/en-us/help/4537482/kb4537482

 

Preview of Quality Rollup for .NET Framework 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows 8.1, RT 8.1, and Windows Server 2012 R2 (KB4537488)

https://support.microsoft.com/en-us/help/4537488/kb4537488

 

Other versions

https://devblogs.microsoft.com/dotnet/net-framework-february-2020-preview-of-quality-rollup/

 

If the version of .NET Framework on your system is less than 4.6, you will need to upgrade to at least .NET Framework 4.6 to get the fix. We recommend that update to .NET Framework 4.8. If you're unable to upgrade, please see the workarounds below.

 

Workaround

Note: the workaround will depend on the number of logical processors for the individual machine. The following is an example based on the 160 logical CPU machine used as example above.

 

There's two workarounds we have identified for this issue.

 

1st Workaround

Issue doesn't happen after disabling the hyper-threading. Once after disabling hyper-threading, there will be 2 different processor groups:

 

Processor Group 0 has 40 Logical Processors

Processor Group 1 has 40 Logical Processors

 

Under each Group, we have 40 processors, so there is no uneven distribution across processor groups and the issue doesn't reoccur.

 

2nd Workaround

BCDEdit is a command-line tool for managing Boot Configuration Data (BCD). This tool allows you to edit the size of the processor group. In the scenario described above with 160 processors, you can execute the following command to create an evenly distributed number of CPUs across the processor groups:

 

bcdedit.exe /set groupsize 40

 

You need to reboot the server after running above command. After this, you'll see:

 

Processor Group 0 has 40 Logical Processors

Processor Group 1 has 40 Logical Processors

Processor Group 2 has 40 Logical Processors

Processor Group 3 has 40 Logical Processors

 

Nathan Schoenack

Support Escalation Engineer - SQL Server Engine