The second build of Windows Server vNext (based on Windows 11) came with an iso with installable optional features, via Dism. I plan to use in-place upgrade from build to build, not perform fresh installs on my test machine every time. I wonder if in-place upgrade shall be, practicly, blocked, if any of the optional features is installed, due to incompatibility between the installed optional features, on file/feature version number. Perhaps upgrade is possible, but optional features shall be uninstalled then, or kept, but needed to be reinstalled via Dism from new sources?
If create a custom image of the new build, with the new Feature on Demand, and use that as the OS to upgrade to, it should work. Take a look at the steps here to do this with the App Compatibility Feature on Demand for Server Core. See section, "Adding to an offline WIM Server Core image".