I have replicated this issue across multiple browsers (chrome, IE and Edge) on several machines.
The issue only seems to occur after the list reaches a certain length. I am using custom JSON view formatting, the code for which can be seenhere. Though another userappears to have run into the same issue, and did not mention using view formatting. If I alter my formatting to be more dense (to show more items per line, thus reducing the height of the page), the issue does not occur. The issue also does not occur in folders with fewer items.
As you can see from the video, the problem happens as the page reaches the bottom of the currently loaded items. Hence why reducing the number of items or fitting more onto the screen prevents the issue (as all the items are loaded with the page). However this is not a solution as we require more than the ~50 items we can currently show, before the bug appears.
I have posted this to MS Answers, and an MS agent directed me here. He seemed to think the issue was related to my JSON code.