How many documents is to much for a Document Set and Record center?

In my experience, a document library with document sets is easily doable for this volume.

We do this in SharePoint Online, and have a single library with hundreds of thousands of documents, with root-level document sets, standard folders within each document set, and then the subsequent documents inside of those folders. And a few other similar implementations, with tens of thousands of docs.

The keys are this:
1) Make sure you define any indexes you need on the front end. Use all 20 of them, and if you can't fill them up, consider creating some placeholder names just in case a new requirement in the future drives the need for a new one

2) Know how to script (in powershell) and how to iterate documents, in case you need to make mass changes

3) Leverage Search and Refiners as the primary interface

4) Make sure your root level document set has views to avoid threshold limits, and whatever folder structures you have underneath do the same. I know on-prem you can change the threshold values, in the cloud you can't, but if you do it right it may not matter. I'm not sure any recent updates on prem that might make it easier. If you are gonna automatically archive stuff, consider using subfolder by year or some other type of value.

You could move things to a record center if you want, but technically you dont HAVE to if you dont have a good reason to. A well architected solution will allow you continue to grow.

Technically, it will work, just takes some thought on the front end.