Issue:
We are seeing increasing state messages in D:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\high of P03 and it is very slowly processing and not caught-up even after 3 days. We see some of the computers which are installed with SCCM client are showing as not installed in console & not software center not working, deployment issues etc. The files in D:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\high is below
Date | Time | Total files in D:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\high | Difference |
27-Apr-21 | 6:55AM | 373144 | |
27-Apr-21 | 2.57PM | 681528 | -308384 |
27-Apr-21 | 8:19PM | 781386 | -99858 |
28-Apr-21 | 7:14AM | 825415 | -44029 |
28-Apr-21 | 9:04AM | 864423 | -39008 |
28-Apr-21 | 11:26AM | 928736 | -64313 |
28-Apr-21 | 01:25PM | 1000481 | -71745 |
28-Apr-21 | 3:56PM | 1047501 | -47020 |
28-Apr-21 | 6:50PM | 1010011 | 37490 |
28-Apr-21 | 8:00PM | 972071 | 37940 |
28-Apr-21 | 8:09PM | 969529 | 2542 |
28-Apr-21 | 8:47PM | 954277 | 15252 |
28-Apr-21 | 10:28PM | 885755 | 68522 |
29-Apr-21 | 6:59AM | 625707 | 260048 |
29-Apr-21 | 7:51AM | 628693 | -2986 |
29-Apr-21 | 8:35AM | 635475 | -9768 |
29-Apr-21 | 12:35PM | 758597 | -123122 |
29-Apr-21 | 01:29PM | 781896 | -23299 |
29-Apr-21 | 2:00PM | 792456 | -10560 |
29-Apr-21 | 4:24PM | 846674 | -54218 |
29-Apr-21 | 4:55PM | 855049 | -8375 |
29-Apr-21 | 5:38PM | 865924 | -10875 |
29-Apr-21 | 6:12PM | 874074 | -8150 |
29-Apr-21 | 7:30PM | 890557 | -16483 |
29-Apr-21 | 8:47PM | 898822 | -8265 |
29-Apr-21 | 9:07PM | 899765 | -943 |
29-Apr-21 | 10.25PM | 907989 | -8224 |
30-Apr-21 | 6:46AM | 910451 | -2462 |
Troubleshooting Done:
None. We were hoping to catch up but not happening
Any other thing we are suspecting related to this error:
IsChangeTrackingDataCleanedUp | SiteCode | P0A | sqlServerName | ABCD1234.com |
| True | False | result | Found change tracking records that passed retention period. |
|
Ask:
- Can we delete the files in D:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\high?
We are asking this because some of the files in D:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\high are older than 2 days. Whereas Min Missing Message Age is set to 2880
- Doing ChangeTracking cleanup will help?
This requires few hours of DB downtime and we will if this only helps
- Is there any way to reduce the files clients sending to server? We have adjusted affected primary (P0A) client setting to 7 days to 10 days but not helping much ☹. Also, state messages set to 2000. (We have to set this back to original value once issue resolved)
- Can we change the below values until these messages are processed?
1148 352 3 Heartbeat Msg Interval 60
1143 352 3 Inbox Polling Interval 900
1144 352 3 Loader Chunk Size 256
1142 352 3 Loader Threads 4
1145 352 3 Max Chunks Fetched 100
1147 352 3 Min Missing Message Age 2880
Additional Data:
To obtain the specific settings for STATE_MESSAGE_SYSTEM, run the following query on a primary
select * from SC_Component_Property where ComponentID in (select ID from SC_Component where ComponentName like 'SMS_STATE_SYSTEM') and sitenumber in (select SiteNumber from SC_SiteDefinition where Sitecode = (Select ThisSiteCode from SMSData))
ID ComponentID SiteNumber Name Value1 Value2 Value3
1148 352 3 Heartbeat Msg Interval 60
1143 352 3 Inbox Polling Interval 900
1144 352 3 Loader Chunk Size 256
1142 352 3 Loader Threads 4
1145 352 3 Max Chunks Fetched 100
1147 352 3 Min Missing Message Age 2880
1146 352 3 Resync Check Interval 15
1473 352 3 Retry Config REG_SZ <Config><Retry PatternID="0" RetryQueue="0">7200,28800,86400</Retry></Config> 0
Environment is MECM 1910 with CAS and three primaries. Issue started when we moved SQL server to VM in one of the primary (because of VM performance) now we have moved back to physical but dealing with clearing these backlogs.
Renaming / deleting inbox folder and slowly moving the files to inbox folder using powershell script did not help - https://lab-geek.com/2016/04/09/using-powershell-to-help-clear-a-configmgr-inbox-backlog/
