@kato_r I don't think this is possible using SharePoint UI.
Also, you can only notify people with existing access/permissions on a file. If you try to notify someone without access, you'll receive an error. So, I don't think there will be any problem keeping it as is.
Please click Mark as Best Response & Like if my post helped you to solve your issue. This will help others to find the correct solution easily. It also closes the item. If the post was useful in other ways, please consider giving it Like.