I found this post while searching for a similar issue.
The difference in my case was that adding a Custom 401 makes the Windows Authentication prompt never show, and instead the IIS immediately rendered the error page in all situations.
Since our backend runs in PHP I have then tried creating a custom Error page also in PHP, for debugging.
This gave me more tools than a pure HTML page would.
Adding these lines to the start of my custom error page fixed the issue for me:
http_response_code(401);
header('www-authenticate: Basic realm=""');
Essentially, this makes sure my custom error page also returns HTTP status 401 (like the default pages do),
and it clears the IIS Authentication Header, which will then allow the prompt to return on page reload.
Hope this can help someone 🙂