IIS Admin Service unable to start - How and Why it can happen?

We often come across issue where IIS admin service fails to start with multiple error codes and often error codes points to machine key corruption..

 sharing one scenario below:

The IIS Admin Service terminated with “Key not valid for use in specified state “ which clearly means NTE_BAD_KEY_STATE. The machine key starting with c2319 for IIS admin service is corrupted

How it works?

--The IIS Admin Service attempts to load the IIS metabase, metabase.xml into memory upon startup.

--To do this, it has to be able to decrypt certain parts of the metabase, using a specific RSA machine key. if for some reason  its unable to decrypt   service cannot start

--The root cause is that the IISAdmin service cannot read the encrypted sections of the IIS Configuration file.

Corruption can happen due to multiple reasons

• By default IIS Machine Key is In  C:\documents and settings\all users\application data\microsoft\crypto\RSA\MachineKeys folder. If  there is any change like compression/permission change for this path we can see  issue
• AV /encryption software scanning file
• application like doubletake.exe touching machine key files on reboot
• abrupt restart/shutdown when the MachineKeys files were accessed
•  Metabase.xml is corrupt (due to reasons listed above)
• permission alternation of the machine key folder

We have seen usually patch /windows update  has always been the victim of such issues.. We always suggest below practice   so that “patch does not become victim of any issue”

1.             Before patching , reboot the server
2.             Once the server is up, check if all required application is working as expected
3.             Install patches
4.             Reboot the server
5.             check applications again

It is difficult for us to tell why the Machine Key file gets  corrupted without any data. Audtiing can help us  to provide more clues in case similar problem occurred again in future.

1 Enable auditing policy on the problematic IIS server:

1. Click Start, click Run, type Gpedit.msc, and then click OK.
2. Under Local Computer Policy, expand Computer Configuration, and then expand Windows Settings.
3. Expand Security Settings, expand Local Policies, and then click Audit Policy.
4. In the details pane, double-click Audit object access
5. Click to select the Success check box, and then click to select the Failure check box.
6. Click OK.

2 Enable auditing on the C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys folder.

1. Right click the folder C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys and choose "Properties".
2. Switch to "Security" tab, and click "Advanced" button.
3. Switch to "Auditing" tab, and click "Add".
4. Enter "Everyone" and then choose "OK".
5. Check following operations, then choose "OK".
6. Traverse Folder/Execute File
7. Create Files/Write Data
8. Create Folders/Append Data
9. Write Attributes
10. Write Extended Attributes
11. Delete Subfolders or files
12. Delete
13. Change Permissions
14. Take Ownership
15. Choose "OK" to confirm the change.