Aug 03 2021 06:16 PM
Following the docs here https://docs.microsoft.com/en-us/mem/configmgr/core/servers/manage/upgrade-on-premises-infrastructure about in place OS upgrade for a CM 2010 install. The servers went from 2012 R2 to 2019. The upgrade worked fine for the primary site server which holds most of the roles (SQL server, Reporting services, WSUS/SUP with shared content folder). I then upgraded our management point which also serves as a secondary SUP. I removed WSUS on both before the upgrades and added them back as described in the docs. The upgrade went fine. I did a site reset to ensure everything is good and there were no issues in the console. Clients are working fine for all features.
After a few days I noticed that the management point is no longer compliant with the configuration baselines. Turns out it hasn't run any of them since the upgrade. I reinstalled the client and it finished with no errors. It seems to register fine. The certificate is fine but then when it tries to download policy or upload messages to the MP (itself), the jobs all error out.
The DataTransferService.log show the following sorts of errors repeated:
=============
CDTSJob::HandleErrors: DTS Job '{E27D24C3-091D-4793-92D6-CB8040D35D4C}' BITS Job '{4EA2941A-D5F2-4760-9947-DC6EC8ACD937}' under user 'S-1-5-18' OldErrorCount 415 NewErrorCount 416 ErrorCode 0x80072EFE
CDTSJob::HandleErrors: DTS Job '{3C7E4C35-2A83-4614-9565-608585A49D1C}' BITS Job '{57E58E15-8BEF-41AF-BD25-2F47AA42BC17}' under user 'S-1-5-18' OldErrorCount 131 NewErrorCount 132 ErrorCode 0x80072EFE
CDTSJob::HandleErrors: DTS Job ID='{3C7E4C35-2A83-4614-9565-608585A49D1C}' URL='https : //<MP FQDN>:443/SMS_MP' ProtType=3
CDTSJob::HandleErrors: DTS Job '{ECDFDEFC-2DCF-4570-A0D9-03701B0FF9D2}' BITS Job '{C0EB62DB-F1DC-42E5-94E8-DAE216713B0F}' under user 'S-1-5-18' OldErrorCount 113 NewErrorCount 114 ErrorCode 0x80072EFE
CDTSJob::HandleErrors: DTS Job ID='{ECDFDEFC-2DCF-4570-A0D9-03701B0FF9D2}' URL='https : //<MP FQDN>:443/SMS_MP' ProtType=3
CDTSJob::HandleErrors: DTS Job '{8F8B924E-04B6-4CD1-8928-963E00DE343C}' BITS Job '{5EF47274-86E8-44AA-B9F0-EEDD903D0F37}' under user 'S-1-5-18' OldErrorCount 208 NewErrorCount 209 ErrorCode 0x80072EFE
CDTSJob::HandleErrors: DTS Job ID='{8F8B924E-04B6-4CD1-8928-963E00DE343C}' URL='https://<MP FQDN>:443/SMS_MP' ProtType=3
CDTSJob::HandleErrors: DTS Job '{AC6517A8-EEC9-42F3-9205-215B281F2240}' BITS Job '{64F4719A-9053-400C-BA90-C0AAE00210B9}' under user 'S-1-5-18' OldErrorCount 131 NewErrorCount 132 ErrorCode 0x80072EFE
========================
That error source is WinHTTP and means "The connection with the server was terminated abnormally". I see an entry in the IIS logs with a 403 error, but I don't know how to get more details about what is causing it.
Looking with bitsadmin it shows the following:
===================
{5211F5B3-89FF-47AC-9BDC-5C67B990CFA7} 'CCM Message Upload {9EF459CF-4EB2-4A75-8FFF-FF400050A3B8}' TRANSIENT_ERROR 0 / 1 0 / 14138
{73F46E79-CD10-4846-840E-7ECD0CCBB976} 'CCM Message Upload {6CE3E579-1C78-4E2D-A1C6-C7118BBAA75E}' TRANSIENT_ERROR 0 / 1 0 / 24650
{9B0D204F-0D88-4683-8800-ADE66C176A0C} 'CCMDTS Job' TRANSIENT_ERROR 0 / 4 0 / UNKNOWN
{63EA641C-266A-4EC5-A428-D3DC30266040} 'CCMDTS Job' TRANSIENT_ERROR 0 / 4 0 / UNKNOWN
{B6C82074-7794-4A34-999A-3938A2216933} 'CCMDTS Job' TRANSIENT_ERROR 0 / 28 0 / UNKNOWN
{D68CF136-7EFB-4F5F-8E56-7B03C6C83830} 'CCMDTS Job' TRANSIENT_ERROR 0 / 4 0 / UNKNOWN
{47152487-B6AA-4134-BBD8-83A85467B8E5} 'CCMDTS Job' TRANSIENT_ERROR 0 / 4 0 / UNKNOWN
{90F73916-3F8C-48CD-B201-7F3AD9A5003F} 'CCMDTS Job' TRANSIENT_ERROR 0 / 3 0 / UNKNOWN
{DF682297-9FFE-4348-B104-FB344B178C13} 'CCMDTS Job' TRANSIENT_ERROR 0 / 4 0 / UNKNOWN
{6A839CF0-45BB-453A-98D9-F75343BE97D5} 'CCMDTS Job' TRANSIENT_ERROR 0 / 4 0 / UNKNOWN
{EA51C2C6-EEB4-4018-AC8A-E0475BD53513} 'CCMDTS Job' TRANSIENT_ERROR 0 / 4 0 / UNKNOWN
{2471FF3C-FD78-4BB4-A037-7E146619FDB8} 'CCMDTS Job' TRANSIENT_ERROR 0 / 4 0 / UNKNOWN
{6E50A082-1771-4147-B06A-76DD40DD6DAD} 'CCMDTS Job' TRANSIENT_ERROR 0 / 28 0 / UNKNOWN
{454513C9-AC01-43E1-B0E8-102FBE25779E} 'CCMDTS Job' TRANSIENT_ERROR 0 / 57 0 / UNKNOWN
{34F8B723-1371-461C-A4C5-A4866323EC44} 'CCMDTS Job' TRANSIENT_ERROR 0 / 57 0 / UNKNOWN
===============
If I delete the bits jobs, they just come back again of course. I don't know if it is useful to anyone, but when I run a bitsadmin on the job I get the following info.
=================
bitsadmin /info {5211F5B3-89FF-47AC-9BDC-5C67B990CFA7} /verbose
BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.
GUID: {5211F5B3-89FF-47AC-9BDC-5C67B990CFA7} DISPLAY: 'CCM Message Upload {9EF459CF-4EB2-4A75-8FFF-FF400050A3B8}'
TYPE: UPLOAD STATE: TRANSIENT_ERROR OWNER: NT AUTHORITY\SYSTEM
PRIORITY: NORMAL FILES: 0 / 1 BYTES: 0 / 14138
CREATION TIME: 7/30/2021 5:26:37 PM MODIFICATION TIME: 8/3/2021 7:27:22 PM
COMPLETION TIME: UNKNOWN ACL FLAGS:
NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3
RETRY DELAY: 600 NO PROGRESS TIMEOUT: 1209600 ERROR COUNT: 1193
PROXY USAGE: NO_PROXY PROXY LIST: NULL PROXY BYPASS LIST: NULL
ERROR FILE: https : //<MP FQDN>:443/CCM_Incoming/{9EF459CF-4EB2-4A75-8FFF-FF400050A3B8} -> D:\SMS_CCM\ServiceData\LocalPayload\{9EF459CF-4EB2-4A75-8FFF-FF400050A3B8}
ERROR CODE: 0x80072efe - The connection with the server was terminated abnormally
ERROR CONTEXT: 0x00000005 - The error occurred while the remote file was being processed.
DESCRIPTION:
JOB FILES:
0 / 14138 WORKING https : //<MP FQDN>:443/CCM_Incoming/{9EF459CF-4EB2-4A75-8FFF-FF400050A3B8} -> D:\SMS_CCM\ServiceData\LocalPayload\{9EF459CF-4EB2-4A75-8FFF-FF400050A3B8}
NOTIFICATION COMMAND LINE: none
owner MIC integrity level: SYSTEM
owner elevated ? true
Peercaching flags
Enable download from peers :false
Enable serving to peers :false
CUSTOM HEADERS: NULL
CLIENT CERTIFICATE INFORMATION:
Certificate Store Location : CERT_STORE_LOCATION_LOCAL_MACHINE
Certificate Store Name : MY
Certificate Hash : 41C2067A522B94550F626B1A136015C4C6FE46D9
Certificate Subject Name : NULL
HTTP security flags
Enable CRL Check :true
Ignore invalid common name in server certificate :false
Ignore invalid date in server certificate :false
Ignore invalid certificate authority in server certificate :false
Ignore invalid usage of certificate :false
URL redirection policy :Redirects will be automatically allowed.
Redirection from HTTPS to HTTP allowed :false
=================
The certificate hash in the job is the same one that shows in the ClientIDManagerStartup.log in the client log folder.
======
>>> Client selected the PKI Certificate [Thumbprint 41C2067A522B94550F626B1A136015C4C6FE46D9] issued to '<MP FQDN>' ClientIDManagerStartup 8/3/2021 5:59:50 PM 5304 (0x14B8)
======
I've been scouring the web for a few days now and can't seem to find anything to help. I see no other obvious errors in the log files. Again, no other machines are having this issue.
I know I could probably remove the MP role and reinstall or bring up a new MP, but I would rather not as this is my dev server that I was testing the in place upgrade on before I did the same thing in prod and I'm not looking forward to having to make those sorts of changes this close to the start of the semester on the prod side if the same thing happens when I upgrade prod. Anyone have any ideas?
Aug 23 2021 09:15 AM
Aug 25 2021 07:21 PM
Aug 31 2021 09:31 AM
Aug 31 2021 11:59 AM
Aug 31 2021 12:57 PM
Aug 31 2021 01:38 PM
Aug 31 2021 05:37 PM
Aug 31 2021 08:36 PM
Sep 01 2021 04:58 AM
@Nathan Blasac yeah I did reboot in between.
Sep 02 2021 08:28 PM
Sep 03 2021 05:13 AM
Sep 06 2021 08:17 PM
Sep 07 2021 04:55 AM
Sep 20 2021 06:21 PM
Sep 21 2021 02:00 PM
Sep 21 2021 04:09 PM
Sep 23 2021 04:41 AM