Forum Discussion
did the Jun 9 Security Update remove trusted (amazon) certs?
We have a product that runs on Windows and uses AWS IoT to connect and transmit info.
We noticed yesterday that many - roughly 2/3 of our fleet went silent - no connections.
We have some of our own PCs that experienced this as well. Investigation yielded issues with TLS. The client (which is just using the default windows cert checking) actively terminated the connected. It didn't like the cert from the AWS IoT endpoint.
All of these PCs were happily connected on Jun 8.
Investigation seemed to indicate that some combination of Amazon Root CA (1-4) and some Starfield certs were not in the device cert mgr.
Also - when we manually added AmazonRootCA1.pem to the cert mgr, our service connected again.
So the evidence seems to strongly support that the security update removed trusted amazon root certs from the cert store.
I'm guessing some/many won't notice since they are making regular TCP connections and maybe the certs get auto-added if they are not there? But we are doing MQTT over the AWS port 8883. So perhaps Windows did not detect this and seek to refresh its cert store?
Can anyone confirm if they have seen the same?
2 Replies
This is a really useful writeup, and it lines up with what a few others have reported around the same timeframe after installing recent Windows cumulative updates. Windows normally relies on the Automatic Root Certificate Update mechanism to pull trusted roots like Amazon Root CA1 through 4 on demand the first time they're needed, but that mechanism depends on outbound connectivity to Windows Update or the Microsoft Trusted Root Program CTL, so if a device is on a locked down network, behind a proxy that blocks that specific traffic, or if the update itself altered the local disallowed or untrusted store, the auto fetch simply won't kick in and the connection fails silently exactly 1like you're describing. The fact that manually importing AmazonRootCA1.pem immediately fixed the connection strongly points to either the update pruning stale or duplicate root entries as part of its cleanup routine, or a change in how Windows handles intermediate chaining for Starfield-issued certs since Amazon's chain still partially relies on that legacy trust path. For anyone hitting this at scale, the safer short-term fix is scripting a bulk import of the current AWS IoT root CA bundle via certutil or Group Policy rather than relying on Windows to auto-heal the store, especially for devices doing MQTT over 8883 since that path doesn't trigger the same automatic root discovery that regular HTTPS traffic does. Would also be worth checking Windows Update history on the affected machines to pin down the exact KB so others troubleshooting the same symptom can cross-reference it.
We haven't seen it across our environment, but your findings do sound consistent with a trust store issue rather than an AWS IoT problem.
If manually importing Amazon Root CA 1 immediately restores the connection, that strongly suggests the TLS handshake is failing because the client can no longer build a trusted certificate chain. AWS IoT endpoints themselves haven't had a broad certificate change that would explain this behavior on its own.
One thing I would check is whether the affected machines have Automatic Root Certificates Update disabled (via Group Policy or registry) or if they were unable to reach Microsoft's root certificate update service. Windows doesn't necessarily have every trusted root preinstalled it can download roots on demand under the right conditions.
It's also interesting that only about two-thirds of your fleet was affected. That makes me wonder if there's a common factor, such as:
Different Windows builds or patch levels
Different GPOs affecting certificate management
Machines that haven't accessed HTTPS endpoints using the Amazon trust chain recently
Restricted internet access preventing automatic root updates
I'd be interested to know which Windows versions and KB updates the affected systems have installed. If they're all on the same cumulative update, that could help determine whether this was caused by a Windows update or by differences in how the root certificate store was maintained.
