Backlog increasing in Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incomi

Issue:

We are seeing increasing state messages in D:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\high of P03 and it is very slowly processing and not caught-up even after 3 days. We see some of the computers which are installed with SCCM client are showing as not installed in console & not software center not working, deployment issues etc. The files in D:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\high is below

Troubleshooting Done:

None. We were hoping to catch up but not happening

Any other thing we are suspecting related to this error:

Ask:

1. Can we delete the files in D:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\high?

We are asking this because some of the files in D:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\high are older than 2 days. Whereas Min Missing Message Age is set to 2880

1. Doing ChangeTracking cleanup will help?

This requires few hours of DB downtime and we will if this only helps

1. Is there any way to reduce the files clients sending to server? We have adjusted affected primary (P0A) client setting to 7 days to 10 days but not helping much ☹. Also, state messages set to 2000. (We have to set this back to original value once issue resolved)

1. Can we change the below values until these messages are processed?

1148    352      3          Heartbeat Msg Interval                                  60

1143    352      3          Inbox Polling Interval                          900

1144    352      3          Loader Chunk Size                               256

1142    352      3          Loader Threads                                   4

1145    352      3          Max Chunks Fetched                           100

1147    352      3          Min Missing Message Age                              2880

Additional Data:

To obtain the specific settings for STATE_MESSAGE_SYSTEM, run the following query on a primary

select * from SC_Component_Property where ComponentID in (select ID from SC_Component where ComponentName like 'SMS_STATE_SYSTEM') and sitenumber in (select SiteNumber from SC_SiteDefinition where Sitecode = (Select ThisSiteCode from SMSData))

ID         ComponentID  SiteNumber     Name  Value1 Value2 Value3

1148    352      3          Heartbeat Msg Interval                                  60

1143    352      3          Inbox Polling Interval                          900

1144    352      3          Loader Chunk Size                               256

1142    352      3          Loader Threads                                   4

1145    352      3          Max Chunks Fetched                           100

1147    352      3          Min Missing Message Age                              2880

1146    352      3          Resync Check Interval                         15

1473    352      3          Retry Config    REG_SZ            <Config><Retry PatternID="0"  RetryQueue="0">7200,28800,86400</Retry></Config>       0

Environment is MECM 1910 with CAS and three primaries. Issue started when we moved SQL server to VM in one of the primary (because of VM performance) now we have moved back to physical but dealing with clearing these backlogs.

Renaming / deleting inbox folder and slowly moving the files to inbox folder using powershell script did not help - https://lab-geek.com/2016/04/09/using-powershell-to-help-clear-a-configmgr-inbox-backlog/