Forum Discussion

CRbot's avatar
CRbot
Copper Contributor
Aug 15, 2023

SPQueryThrottledException from file upload

We're working on an automation that's uploading a lot of files to SharePoint through the office365 library in python

 

We've started getting this exception - form what I've been able to find it looks like it's related to a query returning over 5000 items. I'm confused why I'm getting this response when I'm trying to upload a file, I assume some query happens on the backend in order to perform the upload

 

 

[2023-08-15T14:29:48.840]: office365.runtime.client_request_exception.ClientRequestException: ('-2147024860, Microsoft.SharePoint.SPQueryThrottledException', 'The attempted operation is prohibited because it exceeds the list view threshold.', "500 Server Error: Internal Server Error for url: https://mysharepointsite.sharepoint.com/sites/Podio/_api/Web/RootFolder/Folders('Shared%20Documents')/Folders('Folder%20Name')/Folders('Another%20Folder')/Folders('SubFolder')/Folders('Longer%20Folder%20%20%201234568%20%20%20Name%20%20987654321')/Folders('Files')/Files/add(overwrite=true,url='image.jpg')")

 
Here is a minimal example of how we upload the files

from office365.sharepoint.folders.folder import Folder
from office365.runtime.http.request_options import RequestOptions
from office365.sharepoint.client_context import ClientContext

def upload_file(file_name: str, folder: Folder, client_context: ClientContext) -> str:
    def _set_header(request: RequestOptions) -> None:
        request.set_header("Prefer", "bypass-shared-lock")

    with open(file_name, "rb") as content_file:
        file_content = content_file.read()

    upload_file = folder.upload_file(file_name, file_content)
    client_context.load(upload_file, before_loaded=_set_header)
    client_context.execute_query()
    return upload_file.serverRelativeUrl

 
Anyone know how we can fix this? I see now way to bump the list view threshold in the settings and I tried creating an indexed column for the Title column (thinking that might be the value used in the query) but we're still hitting this exception

2 Replies

  • CRbot's avatar
    CRbot
    Copper Contributor
    For anyone who faces this issue we mitigated it by uploading to a different directory

    0. Catch `ClientRequestException` and confirm it's _this_ exact issue
    1. Parsed the attempted upload path to identify the parent directory of the directory we're trying to upload a file to
    2. Append an integer to that file name
    3. Check for that directory and upload if it doesn't exist
    4. Upload to that directory instead
  • CRbot's avatar
    CRbot
    Copper Contributor

    Anyone know if there's any support team for the sharepoint API and how to contact them?

Resources