The PowerShell script that's worked for 2 years to find a signing certificate, stopped working

[Rod Falanga]

This is an on-prem TFS question. Yesterday I put a certificate in place to handle signing ClickOnce deployments. However, it fails to apply the certificate. The guy who wrote these release scripts before me, the former TFS administrator, was a PowerShell guru. I am not a PowerShell guru. However, I can muddle my way into trying to figure out what is going on, up to a point. The errors occur in the Release, not in the build. The first error that appears is, "You cannot call a method on a null-valued expression". I believe I've found the line where that error occurs, from the PowerShell++ script he uses. It is here:

$cert = ls cert:\ -Recurse -CodeSigningCert | ? {$_.Verify()} | Select -First 1

I've broken this line down into its parts. Ignoring the assignment to the variable $cert, this part of the PowerShell script works:

ls cert:\ -Recurse -CodeSigningCert

When I run it on the TFS build server, running PowerShell as the account used for performing builds and releases, it produces 4 lines of results. However, the rest of the line:

| ? {$_.Verify()} | Select -First 1

I believe rults in assigning a NULL to $cert, which agrees with the first error message in the TFS Release log. This same script has worked fine, for two years. Why has importing a new certificate into the TFS build server certificate store made this assignment fail?

[#2]

@Rod Falanga 

Hey, Rod.

Good diagnostic thinking - that's a great skill to have whether it's PowerShell or something else you're nutting through.

Having four certificates is bothersome since they can be returned in any order. I'd probably suggest tossing a "Sort-Object -Property NotAfter -Desc" in there before the "Select -First 1" to ensure you're at least always grabbing the most recently-obtained certificate.

But setting that aside, the things I'd suggest checking the returned certificate that:

1. It hasn't expired (NotAfter date);
2. The CRL is reachable;
3. The Active Directory policy endpoint is reachable (if relevant).

Cheers,

Lain

[#3]

Thank you, Lain, for the kind complements.

I'm just a better than average beginner PowerShell user, so I don't know how to use NotAfter. For example, do I do something like NotAfter 7/1/2022? Or does NotAfter -Desc sort a description field in descending order? And if so, is that description field always the expiration date?

And what does CRL mean?

I've no idea at all if Active Directory policy endpoint is involved, and certainly no idea if it is reachable, even if it is involved. But since it worked as recently as last week, I presume it either isn't involved or if involved it was reachable.

[#4]

@Rod Falanga 

Hey, Rod. Here's some short-form responses.

Sort-Object

Expanding on my "Sort-Object" comment, this is what I meant:

Going from this original line:

$cert = ls cert:\ -Recurse -CodeSigningCert | ? {$_.Verify()} | Select -First 1

To this:

$cert = ls cert:\ -Recurse -CodeSigningCert | Sort-Object -Property NotAfter -Desc | ? {$_.Verify()} | Select -First 1

This ensures that the command returns the certificate with the furtherest-away expiration date, and not just any randomly-ordered matching certificate.

CRL distribution point

An example of a CRL distribution point is shown below. If your TFS host can't reach the CRL location (multiple CRL locations can be listed, meaning you'd want to check them all) then the call within your command line to the .Verify() method will fail, which is why I listed this as an option.

If the CRL begins with "http" then you can simply try accessing that location (including the file name) from a browser. If it begins with ldap: then perhaps for now, just skip this test (you'd need to leverage something like certutil.exe to run this test) since there's a good chance this won't be the issue anyway.

Example of plugging a http-based CRL into the browser bar to test existence:

Active Directory policy endpoint

If you don't see a "Certificate Template Information" line as shown below in the Details tab, then just ignore this part as it's most likely you're not using an Active Directory policy anyway.

If you do see such a line, then you will have a policy and therefore an endpoint but as with the "ldap" statement above, I'd just assume this is working for now as it's not as easy to test as a "http"-based CRL.

Summary of my initial thoughts

My thoughts at this early stage are:

• I doubt the script is actually the issue;
• I feel like it's either selecting an expired certificate (i.e. current date > NotAfter value on the selected certificate); or
• All of the CRLs from the chosen certificate are unreachable.

While I don't think the script will be at fault, it does bother me that it's scoped to search both the local machine store (which is what I'd expect to see) but also the current user store (which I didn't expect to see.) That doesn't sit well with me since if the script selected a certificate from within the user store, I'd expect that TFS could not read it - at least not from where it sits in the user store.

While that bothers me, it's also not a reason for the call to .Verify() to fail, which your testing shows is happening (i.e. ".Verify()" will be returning $false.)

Purely as a testing exercise, you can run the following. If all the results back as "False" then that serves as confirmation about what you've hypothesized regarding the "null" return value:

cert:\ -Recurse -CodeSigningCert | ForEach-Object {$_.Verify()}

Naturally, there are other options but these are my initial guesses, as per my first response.

Make sure you take a peek at Event Viewer for clues, too.

Cheers,

Lain

Edited: Corrected multiple typos.

[#5]

Thank you very much for giving me that additional explanation. Especially the Sort-Object code. I thought you meant to replace the .Verify() code with Sort-Object. When I get back to work on Tuesday, I'll put that in place.

The rest will have to wait until I get back to work.

[#6]

@LainRobertson I'm at my work laptop so I can try what you suggested. I took this line:

ls cert:\ -Recurse -CodeSigningCert | Sort-Object -Property NotAfter -Descending | Select -First 1

Then changed it, per your suggestion, to this:

ls cert:\ -Recurse -CodeSigningCert | Sort-Object -Property NotAfter -Descending | ? {$_.Verify()} | Select -First 1

The first line does return 1 of the records from the Certificate store. The second line doesn't return anything from the Certificate store. And I know there's a valid certificate in the certificate store that won't expire until June of 2023.

[#7]

@Rod Falanga 

Hey, Rod.

The key difference between those lines is that your first line doesn't contain the call to ".Verify()", which almost confirms what I expected to be true, which is that the returned result is failing the call to .Verify().

Try running the following (which is mostly the same as what I posted earlier) and see if it returns True or False.

$cert = ls cert:\ -Recurse -CodeSigningCert | Sort-Object -Property NotAfter -Descending |  Select -First 1;
$cert | fl Thumbprint, NotAfter, Subject, @{n="Verified"; e={ $_.Verify() }};

You should see output like the following, with the value for Verified being what you're most interested in.

Failing verification doesn't mean the certificate is invalid (though it could be.) It just means the verification process failed, which takes me back to things like ensuring the CRL can be reached, etc.

For now, you're only interested in seeing if Verified is coming back as false. If it is, then that's why the script is failing to find a certificate to assign.

Cheers,

Lain

Edited to correct the two-line PowerShell example.

[#8]

Hi Lain,

I ran the two lines of PS script you gave me. It didn't produce anything, after running the second line. After running it, PS just returned to the PS prompt.

[#9]

@Rod Falanga 

My apologies, Rod.

I'd made a copy-and-paste fail since I forgot to remove the .Verify() section from line 1.

I've updated line 1 now, so perhaps try it again.

From what you described about not getting anything at all though, I expect the result will indeed be False.

Cheers,

Lain

[#10]

@LainRobertson 

I've been making several changes to the PS script to make it work. It still isn't. Here's what I've currently got for trying to sign the .exe and .dll files produced during the build:

Get-AuthenticodeSignature *.exe,*.dll | ? Status -eq NotSigned | % Path | %{&$signtool sign /debug /tr $timestamp /td sha384 /fd /sha1 $hash $_ }

And here's the error that I'm now getting:

##[error]SignTool Error: The specified algorithm cannot be used or is invalid 

 I do not know what algorithm should be used with the /td and /fd switches. And I'm still unsure if I should include /sha1 or not. Working with a colleague we looked at the properties of the new certificate and saw this:

• Signature algorithm: SHA384RSA
• Signature hash algorithm: SHA384
• Thumbprint algorithm: SHA1

Using those what does it tell you I should be using for /td and /fd. And do I still need to use /SHA1?

[#11]

@Rod Falanga 

I'm having to make an educated guess here - given all I can see is a variable named $signtool, but I'm assuming the tool being used is signtool.exe. That puts this outside the realm of being a PowerShell discussion but I'll try to help as best I can.

I'm going off the reference for signtool.exe:

• SignTool.exe (Sign Tool) - .NET Framework | Microsoft Docs

I'd try the following for the final part of your statement:

%{&$signtool sign /debug /tr $timestamp /td sha256 /fd sha256 /sha1 $hash $_ }

The basis for going with SHA256 as the lowest common denominator comes from point 8 in the following article - which I'm only using as a clue:

• Manage code signing certificates - Windows drivers | Microsoft Docs

If I was going to bump anything up to SHA384, it'd be the /fd switch. That's solely because the timestamping server is more of an unknown to me than the Windows operating system, which has supported SHA384 for some time now. But this is just a big assumption since I know nothing about your environment, or that of the timestamping server. Again, this is well outside of the PowerShell remit.

The /sha1 switch is fine. You can think of this as "/thumbprint" instead, if you prefer, as a certificate thumbprint is simply a dynamically-generated (i.e. it's not part of the certificate at all) SHA1-based hash of the certificate.

In the context of signtool.exe, /sha1 thumbprint is simply used to instruct signtool.exe which certificate is should select when more than one certificate is eligible for selection.

Cheers,

Lain

[#12 · Accepted solution]

@Rod Falanga 

I also meant to add: is there some reason Set-AuthenticodeSignature won't work for you?

The only gap I see between it and signtool.exe is the ability to control the timestamping algorithm, but I wouldn't have thought this would have mattered.

It would be a little easier/more readable to use Set-AuthenticodeSignature but if you need that finer-grain control from signtool.exe then that's fair enough.

Cheers,

Lain

[#13]

The only reason why Set-AuthenticodeSignature wasn't working for me is because I've never heard of it. I only learned of it yesterday. I switched out using SignTool with Set-AuthenticodeSignature. Set-AuthenticodeSignature works!!!!

Thank you very much for all of your help with this!!