@DanielMonger
Key requirement for tasks to update based on resource calendars in the Project Schedule mode. This is https://support.microsoft.com/en-us/office/scheduling-modes-in-microsoft-project-for-the-web-b2d1626b-8dca-40a1-922e-fbaffd56e589 within your Project for the web environment.
If you create a Project with "Fixed Units" as the scheduling mode then add a resource with non working time or time off the task duration will update to include the working days
If you create a Project with "Fixed Duration" as the scheduling mode then add a resource with non working time or time off the task effort will only include the "working days)
As you point out when you update non-working time or time off it does not automatically update existing tasks (yet). Existing tasks do update once the resource is added/removed to any task within the project (but you can't really rely on that). I've found that by reporting against capacity (user work calendars) and demand (all the assignments) in Power BI the best way to highlight any resourcing issues. This is pretty a pretty solid appoach as PM's will not be suprised by tasks updating in the background, but they can have access to reports that outline resourcing issues.
Hope this helps.
Duncan