Silly question about OS in place upgrade of site server

This feels like a silly question with an obvious no answer but figured I'd ask it and see if other people have tried.

I've done OS in place upgrades of my DP/MP servers and the primary SUP server.  These have been done via task sequence for the in place upgrade without issues.

however I suspect I cannot use the task sequence to do the in place of the primary site server?  maybe if I had a HA setup with and active passive but I do not.
Primary Site only runs:

• Component Server
• Fallback status point
• Site Server
• Site System
• SMS Provider

The DP/MP's host all the data but I suspect because they have to look back to the site server they would fail in half an hour during the task sequence build.