Active OneDrive makes whole environment 'sluggish'

Our client is using AVD for about 5 weeks in 'production' now. It is a smaller company with about 40 users. We have 6 VM's in AVD (D4s_v5) in use at the moment. We have tried D8s_v5 without any improvement.  
But we see memory and cpu usage at high levels most of the time with OneDrive at the top of the processes. Every server feels sluggish at the moment. Starting Excel takes about 30-40 seconds, Word about the same time. Opening a Word document takes about 20-25 seconds to open and opening folders from the right pane (yes only the right pane) in explorer takes abount 7-15 seconds per folder.

All those folders are folders from SharePoint document libraries. 
It doens't matter if a folder is fully synced with the server or it isn't. 

We are using FSlogic with profiles stored on azure. 

I was reading a blog of Christian Brinkhoff about almost the same kind of issues but can't really pinpoint the exact cause or find a solution for this issue.

Can you help me out?