Managing Large Lists

Deleted Forgive me as I dodge around your original question and ask about your end-goal instead. Assuming you are using ADFS, is the data you use for this roster also reside in ADFS? The reason I ask--my organization recently is exploring dealing with employee directory maintenance by leaning on Microsoft Teams' Who bot instead of a list. Who isn't ideal for browsing employees, but it is very good at returning AD information about specific people and even provides some extra information you can't get from a directory, such as what people that person most communicates with.

As for your original question--the best way I can think of to manage a list that large other than delete/add would be to set up a delta job. In theory, you could do this with a tool like SSRS via Kingswaysoft or maybe even Flow. To pseudo-code the process, you basically have this:

• Match unique identifiers (i.e., employee number) from new list (the most recent employee directory data) to master list (your SharePoint list)
  • If new list has a new unique identifier not found in the master list, add the new record to your master list
  • If master list has a unique identifier that isn't found in the new list, delete that record from the master list
  • If unique identifiers match, but any of the other fields in don't match between the master list and the new list, overwrite master list record with new list record information

Delta jobs should, in theory, run faster than a drop-and-load process.