Breaking permission inheritance on large library with no unique document or folder permissions

We have a large library that was migrated using a tool. Library has around 500000 documents. When migrated to SPO, some documents had document level permissions. These permissions have been removed.

Currently, library has no unique permissions and all documents inside it inherit. However, when we try to break inheritance at library level, we still get the list view threshold error.

Is this expected? We thought list view threshold error only presents itself when trying to load more than 5000 items or if there are too many unique permissions in library. Breaking permissions are library level should work.

Any ideas?