Inconsistent responses from SharePoint365 API

%3CLINGO-SUB%20id%3D%22lingo-sub-874752%22%20slang%3D%22en-US%22%3EInconsistent%20responses%20from%20SharePoint365%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-874752%22%20slang%3D%22en-US%22%3E%3CP%3EWe%20use%20the%20legacy%20Sharepoint%20API%20to%20ingest%20all%20content%20from%20not%20only%20SharepointOnline%2F365%20but%20also%20Sharepoint%202013%20and%202016.%3CBR%20%2F%3EUnfortunately%2C%20while%20ingesting%20data%20from%20a%20Sharepoint%20365%20site%2C%20the%20API%20is%20not%20behaving%20in%20a%20consistent%20way.%3CBR%20%2F%3EConsistently%2C%20it%20reports%20that%20certain%20items%20do%20not%20exist%20when%20the%20site%20has%20not%20been%20modified%20and%20we%20can%20see%20said%20items%20in%20the%20expected%20location.%3CBR%20%2F%3EFor%20example%3A%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EDEBUG%202019-07-29%2023%3A04%3A17%2C439%20(Worker%20thread%20'0')%20-%20http-outgoing-129363%20%26gt%3B%26gt%3B%20GET%20%2Fsites%2Fsonyhack%2F_api%2Fweb%2FGetFileByServerRelativeUrl('%252f%252fsites%252fsonyhackShared%2520Documents%252fFinance-1%252fSPI%252fFinance%252f21-SOX%2520-%2520Internal%2520Audit%252fFY14%252fRef%2520%252317.6%2520LON_Prod%2520Acctg%2520Fee_WE%2520140302.xlsx')%20HTTP%2F1.1%0ADEBUG%202019-07-29%2023%3A04%3A17%2C439%20(Worker%20thread%20'0')%20-%20http-outgoing-129363%20%26gt%3B%26gt%3B%20User-Agent%3A%20NONISV%7CExonar%7CSlurp%2F1.0%0ADEBUG%202019-07-29%2023%3A04%3A17%2C439%20(Worker%20thread%20'0')%20-%20http-outgoing-129363%20%26gt%3B%26gt%3B%20X-RequestDigest%3A%200x31C66DB00416F265FCD915032E30EF736B3AC796F59873EBF593105231E91D0683FD26F857E0854958C30C7D45AFD6CB5DA4B2DABC7D06133631A74FCA9A9F86%2C29%20Jul%202019%2022%3A40%3A45%20-0000%0ADEBUG%202019-07-29%2023%3A04%3A17%2C439%20(Worker%20thread%20'0')%20-%20http-outgoing-129363%20%26gt%3B%26gt%3B%20Host%3A%20exonarltd.sharepoint.com%0ADEBUG%202019-07-29%2023%3A04%3A17%2C439%20(Worker%20thread%20'0')%20-%20http-outgoing-129363%20%26gt%3B%26gt%3B%20Connection%3A%20Keep-Alive%0ADEBUG%202019-07-29%2023%3A04%3A17%2C439%20(Worker%20thread%20'0')%20-%20http-outgoing-129363%20%26gt%3B%26gt%3B%20Cookie%3A%20FedAuth%3D77u%2FPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48U1A%2BVjYsMGguZnxtZW1iZXJzaGlwfDEwMDM3ZmZlYTUwNzU0ZTJAbGl2ZS5jb20sMCMuZnxtZW1iZXJzaGlwfHN1cHBvcnRAZXhvbmFybHRkLm9ubWljcm9zb2Z0LmNvbSwxMzIwODg4NDgxNjAwMDAwMDAsMTMxNTIyNzA4MDQwMDAwMDAwLDEzMjA4OTg4OTEyNzA3MTQwOSwwLjAuMC4wLDIsMTFjZjVjZjUtYmQxZS00MmUyLTg3YjYtMTU2OWJiNTc5NGViLCwsOTkwNWY1OWUtZjBjOS05MDAwLTM0MWQtNzdhZWI2MTQ1OGIzLDdhMTZmNTllLTMwMDItOTAwMC1lMGE1LTgzZGU4OWRjY2NjMSwsMCwwLDAsLCwsbVAyTy9jZlFEMmF4UzVoVHBLUlJpTlZIT2o3VVNvYnJMN2hzRlNUT0RReFNIWTFqSUNhazMycW15b3R1anhTdE16ZzRWc1ZJSm4vSjE4RklIa3J0MGVZSlg3NWJ3S1dYMVVlWFBpbXViWGxNNzhHVW4zQWNoWVY4VjhBdmxLSzBxamZhWG1rUkxLWVB3aG8xdWxnL0Jqend5SVdQRFB1WDhjenZPNzcybU5GZm9MSmZLbVJvWjRHbXVVMENPQ08wTFowQW0zLzJSVjJpeTQ4bEZjc0JlK2dsMU5ac0tmZDZib1hyU2tqNEVBUDE2d1p3Q3BKQ0lHZW5TMUFNS2NwZVpEdVZiS0g1U3pTMSt4VFhPOVozUHk3RXJkOUZ4VHBDOGI2VURPTVpReSthUUhiWXo0cXhQSHRkbkNPamhFVWJka1k1YmhJcWxTNlNVQ1drV21MbzJRPT08L1NQPg%3D%3D%3B%20RpsContextCookie%3D%3B%20rtFa%3DyXf27yO84mqNo8Df1YyZAZNdFXHp86RmifcrIL5sf3UmMTFDRjVDRjUtQkQxRS00MkUyLTg3QjYtMTU2OUJCNTc5NEVCuxeux9MONhihqOajrJJASK7xHgk0lIVFEo6GDHydXFue5RYzCycoZUFUt27FQIFbYxlOF0J3kuxBbWR6NIdGTpscklpdAxhr66ZtGvqiLfXBNXT%2FezoERMFETTjK8yl6gRT0W4g8%2FFeArW2ME6QdmXm1Cy%2BNDjI%2F%2BgxOQ2hqpEAQ3vPsa0h%2BxAh235SiCMX0gBV%2BvFa8A7AFH%2BjzQzyuYlVH2Cwy6KVqdvNpqeyBQoVejio5ySQgljpY77o89sH5d35hJHuUOOQr7RsrZdZ69Ewm5B3p9AfqO0Es%2BT8uqHPOxowWe2tSIku0oo7JkILJAgda8LXXkNHwTXQEUT2CgUUAAAA%3D%0ADEBUG%202019-07-29%2023%3A04%3A17%2C439%20(Worker%20thread%20'0')%20-%20http-outgoing-129363%20%26gt%3B%26gt%3B%20Cookie2%3A%20%24Version%3D1%0ADEBUG%202019-07-29%2023%3A04%3A17%2C439%20(Worker%20thread%20'0')%20-%20http-outgoing-129363%20%26gt%3B%26gt%3B%20Accept-Encoding%3A%20gzip%2Cdeflate%0ADEBUG%202019-07-29%2023%3A04%3A17%2C513%20(Worker%20thread%20'0')%20-%20http-outgoing-129363%20%26lt%3B%26lt%3B%20HTTP%2F1.1%20404%20Not%20Found%3C%2FCODE%3E%3C%2FPRE%3E%3CP%3EAnd%20yet%2C%20the%20item%20is%20there%2C%20not%20having%20been%20modified%20for%20months.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIn%20other%20situations%2C%20these%20failures%20are%20intermittent%2C%20with%20an%20item%20being%20reported%20as%20missing%20on%20one%20run%20to%20then%20be%20seen%20the%20next%20run%20and%20disappear%20again%20on%20the%20following%20run.%3C%2FP%3E%3CP%3EThis%20makes%20us%20unable%20to%20get%20a%20consistent%20view%20on%20Sharepoint%20sites%20as%2C%20if%20one%20of%20those%20items%20is%20a%20folder%2C%20we%20end%20up%20not%20only%20not%20seeing%20it%20but%20any%20of%20the%20contents%20within.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIs%20this%20something%20others%20have%20experienced%3F%20Is%20there%20a%20workaround%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThank%20you%20for%20your%20help%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-874752%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAPI%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESharePoint%20Online%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-875302%22%20slang%3D%22en-US%22%3ERe%3A%20Inconsistent%20responses%20from%20SharePoint365%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-875302%22%20slang%3D%22en-US%22%3EWhat%20do%20you%20mean%20by%20legacy%20API%3F%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-876668%22%20slang%3D%22en-US%22%3ERe%3A%20Inconsistent%20responses%20from%20SharePoint365%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-876668%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F60%22%20target%3D%22_blank%22%3E%40Juan%20Carlos%20Gonz%C3%A1lez%20Mart%C3%ADn%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20authenticate%20with%20Sharepoint%20API%20using%26nbsp%3B%3CSTRONG%3ESharePoint%20App-Only%20principal%26nbsp%3B%3C%2FSTRONG%3Eas%20documented%20in%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fsharepoint%2Fdev%2Fsolution-guidance%2Fsecurity-apponly%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fsharepoint%2Fdev%2Fsolution-guidance%2Fsecurity-apponly%3C%2FA%3E%20as%20our%20app%20way%20to%20run%20software%20that%20integrates%20with%20both%20online%20and%20on%20premises%20Sharepoint%20installations.%20That%20was%20what%20was%20meant%20with%20legacy.%3C%2FP%3E%3CP%3EApologies.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20inconsistencies%20we%20are%20seeing%20are%20solely%20related%20to%20the%20Sharepoint%20REST%20API%20documented%20in%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fsharepoint%2Fdev%2Fsp-add-ins%2Fget-to-know-the-sharepoint-rest-service%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fsharepoint%2Fdev%2Fsp-add-ins%2Fget-to-know-the-sharepoint-rest-service%3C%2FA%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EFor%20full%20clarity%20purposes%2C%20the%20endpoints%20we%20are%20having%20issues%20with%20are%26nbsp%3B%3C%2FP%3E%3CP%20class%3D%22p1%22%3E%3CSPAN%20class%3D%22s1%22%3EGetFileByServerRelativeUrl%20and%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20class%3D%22s1%22%3EGetFolderByServerRelativeUrl%20(%3C%2FSPAN%3Edescribed%20in%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fsharepoint%2Fdev%2Fsp-add-ins%2Fworking-with-folders-and-files-with-rest%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fsharepoint%2Fdev%2Fsp-add-ins%2Fworking-with-folders-and-files-with-rest%3C%2FA%3E%26nbsp%3B)%20which%20we%20use%20to%20navigate%20Sharepoint%20sites.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Occasional Visitor

We use the legacy Sharepoint API to ingest all content from not only SharepointOnline/365 but also Sharepoint 2013 and 2016.
Unfortunately, while ingesting data from a Sharepoint 365 site, the API is not behaving in a consistent way.
Consistently, it reports that certain items do not exist when the site has not been modified and we can see said items in the expected location.
For example:

DEBUG 2019-07-29 23:04:17,439 (Worker thread '0') - http-outgoing-129363 >> GET /sites/sonyhack/_api/web/GetFileByServerRelativeUrl('%2f%2fsites%2fsonyhackShared%20Documents%2fFinance-1%2fSPI%2fFinance%2f21-SOX%20-%20Internal%20Audit%2fFY14%2fRef%20%2317.6%20LON_Prod%20Acctg%20Fee_WE%20140302.xlsx') HTTP/1.1
DEBUG 2019-07-29 23:04:17,439 (Worker thread '0') - http-outgoing-129363 >> User-Agent: NONISV|Exonar|Slurp/1.0
DEBUG 2019-07-29 23:04:17,439 (Worker thread '0') - http-outgoing-129363 >> X-RequestDigest: 0x31C66DB00416F265FCD915032E30EF736B3AC796F59873EBF593105231E91D0683FD26F857E0854958C30C7D45AFD6CB5DA4B2DABC7D06133631A74FCA9A9F86,29 Jul 2019 22:40:45 -0000
DEBUG 2019-07-29 23:04:17,439 (Worker thread '0') - http-outgoing-129363 >> Host: exonarltd.sharepoint.com
DEBUG 2019-07-29 23:04:17,439 (Worker thread '0') - http-outgoing-129363 >> Connection: Keep-Alive
DEBUG 2019-07-29 23:04:17,439 (Worker thread '0') - http-outgoing-129363 >> Cookie: FedAuth=77u/PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48U1A+VjYsMGguZnxtZW1iZXJzaGlwfDEwMDM3ZmZlYTUwNzU0ZTJAbGl2ZS5jb20sMCMuZnxtZW1iZXJzaGlwfHN1cHBvcnRAZXhvbmFybHRkLm9ubWljcm9zb2Z0LmNvbSwxMzIwODg4NDgxNjAwMDAwMDAsMTMxNTIyNzA4MDQwMDAwMDAwLDEzMjA4OTg4OTEyNzA3MTQwOSwwLjAuMC4wLDIsMTFjZjVjZjUtYmQxZS00MmUyLTg3YjYtMTU2OWJiNTc5NGViLCwsOTkwNWY1OWUtZjBjOS05MDAwLTM0MWQtNzdhZWI2MTQ1OGIzLDdhMTZmNTllLTMwMDItOTAwMC1lMGE1LTgzZGU4OWRjY2NjMSwsMCwwLDAsLCwsbVAyTy9jZlFEMmF4UzVoVHBLUlJpTlZIT2o3VVNvYnJMN2hzRlNUT0RReFNIWTFqSUNhazMycW15b3R1anhTdE16ZzRWc1ZJSm4vSjE4RklIa3J0MGVZSlg3NWJ3S1dYMVVlWFBpbXViWGxNNzhHVW4zQWNoWVY4VjhBdmxLSzBxamZhWG1rUkxLWVB3aG8xdWxnL0Jqend5SVdQRFB1WDhjenZPNzcybU5GZm9MSmZLbVJvWjRHbXVVMENPQ08wTFowQW0zLzJSVjJpeTQ4bEZjc0JlK2dsMU5ac0tmZDZib1hyU2tqNEVBUDE2d1p3Q3BKQ0lHZW5TMUFNS2NwZVpEdVZiS0g1U3pTMSt4VFhPOVozUHk3RXJkOUZ4VHBDOGI2VURPTVpReSthUUhiWXo0cXhQSHRkbkNPamhFVWJka1k1YmhJcWxTNlNVQ1drV21MbzJRPT08L1NQPg==; RpsContextCookie=; rtFa=yXf27yO84mqNo8Df1YyZAZNdFXHp86RmifcrIL5sf3UmMTFDRjVDRjUtQkQxRS00MkUyLTg3QjYtMTU2OUJCNTc5NEVCuxeux9MONhihqOajrJJASK7xHgk0lIVFEo6GDHydXFue5RYzCycoZUFUt27FQIFbYxlOF0J3kuxBbWR6NIdGTpscklpdAxhr66ZtGvqiLfXBNXT/ezoERMFETTjK8yl6gRT0W4g8/FeArW2ME6QdmXm1Cy+NDjI/+gxOQ2hqpEAQ3vPsa0h+xAh235SiCMX0gBV+vFa8A7AFH+jzQzyuYlVH2Cwy6KVqdvNpqeyBQoVejio5ySQgljpY77o89sH5d35hJHuUOOQr7RsrZdZ69Ewm5B3p9AfqO0Es+T8uqHPOxowWe2tSIku0oo7JkILJAgda8LXXkNHwTXQEUT2CgUUAAAA=
DEBUG 2019-07-29 23:04:17,439 (Worker thread '0') - http-outgoing-129363 >> Cookie2: $Version=1
DEBUG 2019-07-29 23:04:17,439 (Worker thread '0') - http-outgoing-129363 >> Accept-Encoding: gzip,deflate
DEBUG 2019-07-29 23:04:17,513 (Worker thread '0') - http-outgoing-129363 << HTTP/1.1 404 Not Found

And yet, the item is there, not having been modified for months.

 

In other situations, these failures are intermittent, with an item being reported as missing on one run to then be seen the next run and disappear again on the following run.

This makes us unable to get a consistent view on Sharepoint sites as, if one of those items is a folder, we end up not only not seeing it but any of the contents within.

 

Is this something others have experienced? Is there a workaround?

 

Thank you for your help

2 Replies
What do you mean by legacy API?

@Juan Carlos González Martín 

We authenticate with Sharepoint API using SharePoint App-Only principal as documented in https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly as our app way to run software that integrates with both online and on premises Sharepoint installations. That was what was meant with legacy.

Apologies.

 

The inconsistencies we are seeing are solely related to the Sharepoint REST API documented in https://docs.microsoft.com/en-us/sharepoint/dev/sp-add-ins/get-to-know-the-sharepoint-rest-service

 

For full clarity purposes, the endpoints we are having issues with are 

GetFileByServerRelativeUrl and GetFolderByServerRelativeUrl (described in https://docs.microsoft.com/en-us/sharepoint/dev/sp-add-ins/working-with-folders-and-files-with-rest ) which we use to navigate Sharepoint sites.