Edge Dev only runs in the background if Startup Boost is enabled. I don't know when this started to happen, as I only realized it now.
Another thing that happens if Startup Boost is not enabled is that the Edge Dev icon in the Task Manager becomes the Edge Stable icon. So I'm wondering if this could be a conflict between Edge Dev and Edge Stable as I have both installed. However, in Edge Stable the Startup Boost and Running in Backround options are disabled.