I did not get the setup running like I wanted it to. I still do not know if the flow with ADFS involved can be done the way I tried it.
So here is how I managed to solve my problem anyway:
I used an AzureAPP du get an Id token of the current user, which I parsed to get the AzureAD loginname of that user.
I used the OfficeDevPnP.Core.AuthenticationManager to call the GetADFSUserNameMixedAuthenticatedContext function and did everything within the context of the provided (elevated) user. I then did the Security Trimming myself, by checking if the current user (see client) has the needed permissions on a specific list Item for example.
This is not very fast I guess, as I have to loop through all List Items I get (because I am running the code as Admin) and then check each one for the permissions and sort it out, if the user should not be allowed to see it.
I am not really happy with the solution I finally came up with, but that was the solution I was able to implement after many days of research, anger, tears, questioning life...
So in case somebody else has the same problem: If you find a proper solution, please contact me. If not, feel free to use the same workaround as I did.