Modern document library web part - slow?

This depends on the number of items in your document library.

For some reason the webpart does a lot of request to retrieve all data. Some of the requests wil return NO rows (depending on your filters/view).

So the problem here is the webpart's implementation. It retrieves items in "pages" (I guess 5.000 at a time), but for some reason it also returns empty pages.

In this case, if you have a library with 500.000 items, it will take 100 requests to load every item. Even if there are just 10 items to display. There's even a chance that just one of the 100 responses contains items.