Inconsistent behavior in using Taxonomy API

%3CLINGO-SUB%20id%3D%22lingo-sub-184696%22%20slang%3D%22en-US%22%3EInconsistent%20behavior%20in%20using%20Taxonomy%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-184696%22%20slang%3D%22en-US%22%3E%3CP%3EWe're%20currently%20developing%20a%20solution%20that%20involves%20several%20types%20of%20Taxonomy%20operations%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CUL%3E%3CLI%3ELoading%20Terms%26nbsp%3B%3C%2FLI%3E%3CLI%3ECreating%20Terms%20(if%20not%20exist)%26nbsp%3B%3C%2FLI%3E%3CLI%3ECreating%20Term%20Labels%20(if%20not%20exist)%3C%2FLI%3E%3CLI%3EReusing%20Terms%20(if%20not%20already%20done)%3C%2FLI%3E%3C%2FUL%3E%3CP%3EIn%20our%20logic%20many%20of%20the%20operations%20need%20to%20be%20combined%20to%20achieve%20our%20desired%20results.%26nbsp%3BWe're%20using%20.NET%20Managed%20CSOM.%20We're%20experiencing%20very%20inconsistent%20behavior%20in%20using%20the%20Taxonomy%20API's.%20Running%20the%20same%20code%20with%20the%20same%20input%20several%20times%20can%20have%20different%20results%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CUL%3E%3CLI%3E80-90%25%20of%20the%20time%20no%20errors%20at%20all%20and%20results%20as%20expected%3C%2FLI%3E%3CLI%3EServer%20Exception%20when%20reusing%20or%20loading%20a%20Term%3A%26nbsp%3B%20%22The%20object%20is%20invalid%20or%20outdated.%20Please%20re-fetch%20the%20object%22%3C%2FLI%3E%3CLI%3EServer%20Exception%20when%20reusing%20or%20loading%20a%20Term%3A%20%22Taxonomy%20item%20instantiation%20failed.%20Unable%20to%20create%20a%20client%20object%20with%20ID%20%7Bsome%20id%7D%22%3C%2FLI%3E%3CLI%3EThe%20HTTP%20service%20located%20at%20http%3A%2F%2Fusr19769-643%3A32843%2F3115c78f4d804003b5964d1cb5b56042%2FMetadataWebService.svc%20is%20unavailable%3C%2FLI%3E%3CLI%3EThere%20was%20no%20endpoint%20listening%20at%20http%3A%2F%2Fusr19769-643%3A32843%2F3115c78f4d804003b5964d1cb5b56042%2FMetadataWebService.svc%20that%20could%20accept%20the%20message%3C%2FLI%3E%3C%2FUL%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20Taxonomy%20API%20doesn't%20seem%20to%20be%20very%20robust.%20We've%20currently%20implemented%20retry%20mechanisms%20with%20delays%20in%20our%20code%20so%20that%20ultimately%20everything%20ends%20up%20in%20the%20desired%20state%2C%20but%20this%20feels%20very%20dirty.%20Since%20most%20of%20the%20time%20the%20results%20are%20what%20we%20expect%2C%20I%20can't%20imagine%20the%20problems%20are%20caused%20by%20our%20usage.%20I%20would%20expect%20consistently%20bad%20results%20if%20the%20problem%20was%20in%20our%20API%20usage%2C%20not%20this%20random%20behavior.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EDoes%20anyone%20have%20similar%20experiences%3F%20What%20are%20your%20approaches%20in%20usage%20and%20error%20handling%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-277886%22%20slang%3D%22en-US%22%3ERe%3A%20Inconsistent%20behavior%20in%20using%20Taxonomy%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-277886%22%20slang%3D%22en-US%22%3E%3CP%3EI%20implemented%20the%20follow%20extension%20to%20get%20around%20the%20problem.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CPRE%3Epublic%20static%20async%20Task%20ExecuteQueryRetry(this%20ClientContext%20clientContext%2C%20int%20delaySeconds%20%3D%202)%3CBR%20%2F%3E%7B%3CBR%20%2F%3E%20while%20(delaySeconds%20%26lt%3B%2030)%3CBR%20%2F%3E%20%7B%3CBR%20%2F%3E%20%20%20try%3CBR%20%2F%3E%20%20%20%7B%3CBR%20%2F%3E%20%20%20%20%20clientContext.ExecuteQuery()%3B%3CBR%20%2F%3E%20%20%20%20%20return%3B%3CBR%20%2F%3E%20%20%20%7D%3CBR%20%2F%3E%20%20%20catch%20(ServerException%20ex)%3CBR%20%2F%3E%20%20%20%7B%3CBR%20%2F%3E%20%20%20%20%20var%20retry%20%3D%20ex.Message%20!%3D%20null%20%3F%3CBR%20%2F%3E%20%20%20%20%20ex.Message.Contains(%22MetadataWebService.svc%22)%3CBR%20%2F%3E%20%20%20%20%20%7C%7C%20ex.Message.Contains(%22GetDefaultSiteCollectionTermStore%22)%20%3A%20false%3B%3CBR%20%2F%3E%3CBR%20%2F%3E%20%20%20%20%20if%20(!retry)%3CBR%20%2F%3E%20%20%20%20%20%7B%3CBR%20%2F%3E%20%20%20%20%20%20%20throw%3B%3CBR%20%2F%3E%20%20%20%20%20%7D%3CBR%20%2F%3E%3CBR%20%2F%3E%20%20%20%20%20var%20delay%20%3D%20TimeSpan.FromSeconds(delaySeconds)%3B%3CBR%20%2F%3E%20%20%20%20%20await%20Task.Delay(delay)%3B%3CBR%20%2F%3E%3CBR%20%2F%3E%20%20%20%20%20delaySeconds%20%3D%20delaySeconds%20*%202%3B%3CBR%20%2F%3E%20%20%20%7D%3CBR%20%2F%3E%20%7D%3CBR%20%2F%3E%3CBR%20%2F%3E%20throw%20new%20TimeoutException(%22Stopped%20retrying%20ExecuteQuery%20after%2030%20seconds.%22)%3B%3CBR%20%2F%3E%7D%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-274744%22%20slang%3D%22en-US%22%3ERe%3A%20Inconsistent%20behavior%20in%20using%20Taxonomy%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-274744%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Tom%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHaving%20the%20sleep%20for%20180%20seconds%20fixed%20it%20for%20me%20too.%20Have%20you%20found%20any%20other%20way%20to%20get%20this%20working%20other%20than%20putting%20in%20some%20delay%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThank%20you.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-200798%22%20slang%3D%22en-US%22%3ERe%3A%20Inconsistent%20behavior%20in%20using%20Taxonomy%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-200798%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Tom%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ENo%20real%20progress%20regarding%20this%20on%20my%20side.%20My%20code%20is%20full%20of%20retry-mechanisms%20(basically%20try-catch-retry)%20to%20work%20around%20the%20errors%20I've%20listed%20and%20the%20end-results%20are%20fine%20now%E2%80%A6%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHope%20your%20workaround%20suffices%20too!%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ERegards%2C%3C%2FP%3E%3CP%3EPaul%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-200776%22%20slang%3D%22en-US%22%3ERe%3A%20Inconsistent%20behavior%20in%20using%20Taxonomy%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-200776%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Paul%20-%3C%2FP%3E%3CP%3EI'm%20curious%20if%20you've%20made%20progress%20on%20this%20this.%26nbsp%3B%20I%20am%20also%20seeing%20the%20error%3A%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3EThe%20HTTP%20service%20located%20at%20http%3A%2F%2Fusr19769-643%3A32843%2F3115c78f4d804003b5964d1cb5b56042%2FMetadataWebService.svc%20is%20unavailable%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIn%20my%20case%2C%20I%20was%20using%20PnP%20PowerShell%20(Apply-PnPProvisioningTemplate)%20to%20deploy%20new%20Managed%20Metadata%20fields%20and%20anchor%20them%20to%20an%20existing%20term%20set.%26nbsp%3B%20I%20was%20running%20this%20after%20creating%20a%20new%20site.%26nbsp%3B%20I%20added%20a%20Sleep%20for%20120%20seconds%20after%20site%20creation%20and%20before%20applying%20the%20template.%26nbsp%3B%20I%20still%20saw%20this%20error.%26nbsp%3B%20Then%20I%26nbsp%3Bset%20it%20to%20sleep%20for%20180%20seconds%20and%20I%20did%20not%20see%20the%20error.%26nbsp%3B%20I%20need%20to%20do%20more%20testing%20to%20see%20if%20180%20seconds%20is%20sufficient%20in%20all%20cases%20or%20if%20I'll%20need%20to%20implement%20retries%20as%20you%20have%20done.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ERegards%2C%3C%2FP%3E%3CP%3ETom%20Castiglia%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-185197%22%20slang%3D%22en-US%22%3ERe%3A%20Inconsistent%20behavior%20in%20using%20Taxonomy%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-185197%22%20slang%3D%22en-US%22%3E%3CP%3EAhh%20yeah%20there%20is%20a%20max%20memory%20usage%20on%20functions%2C%20but%20I%20doubt%20you%20are%20hitting%20it%20with%20taxonomy%20changes.%26nbsp%3B%20Similarly%20though%20if%20you%20are%20using%20large%20amounts%2C%20try%20scaling%20down%20the%20load%20and%20see%20what%20happens.%26nbsp%3B%20A%20bit%20painful%20for%20performance%20testing%20unless%20you%20have%20some%20%22real%22%20performance%20testing%20tools%20at%20your%20disposal%20that%20play%20nicely%20with%20Azure.%26nbsp%3B%20Good%20luck%20or%20hopefully%20someone%20can%20give%20a%20better%20answer%20than%20me!%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EMax%3A%201%2C536MB%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESource%3A%20%3CA%20title%3D%22Memory%20Consumption%20Limit%20on%20Azure%20Function%20Apps%22%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fpricing%2Fdetails%2Ffunctions%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fpricing%2Fdetails%2Ffunctions%2F%3C%2FA%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-185088%22%20slang%3D%22en-US%22%3ERe%3A%20Inconsistent%20behavior%20in%20using%20Taxonomy%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-185088%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Joel%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20for%20your%20response!%20Sorry%20I%20forgot%20to%20mention%20the%20environment%20I'm%20working%20in%20is%20SharePoint%20Online%20so%20no%20option%20to%20increase%20hardware.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20logic%20is%20running%20in%20an%20Azure%20Function%20App%20by%20queueu-triggered%20Functions.%20I%20have%20also%20written%20some%20Unit%20Tests%20where%20I%20see%20the%20same%20happening%20(e.g.%20Unit%20Tests%20succeed%20most%20of%20the%20times%2C%20but%20frequently%20fail%20due%20to%20one%20of%20the%20errors%20liststed%20above).%20Very%20unpredictable...%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-184851%22%20slang%3D%22en-US%22%3ERe%3A%20Inconsistent%20behavior%20in%20using%20Taxonomy%20API%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-184851%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Paul%20this%20sounds%20like%20a%20SharePoint%20timer%20or%20indexing%20issue.%26nbsp%3B%20Some%20things%20to%20try%20out%20and%20test%3A%3C%2FP%3E%3COL%3E%3CLI%3EHow%20many%20terms%20are%20you%20loading%20or%20creating%3F%20Try%20reducing%20the%20number%20as%20see%20if%20that%20resolves%20the%20issue.%3C%2FLI%3E%3CLI%3EHow%20often%20are%20you%20running%20this%20solution%3F%20Try%20reducing%20the%20number%20of%20runs%20and%20see%20if%20that%20resolves%20the%20issue.%3C%2FLI%3E%3C%2FOL%3E%3CP%3EIf%20either%20of%20the%20above%20resolves%20the%20issue%20there%20are%202%20ways%20off%20the%20top%20of%20my%20head%20for%20handling%20this%3A%3C%2FP%3E%3COL%3E%3CLI%3EEasiest%20solution%20is%20to%20%E2%80%9Cthrow%20more%20power%20at%20it%E2%80%9D.%20Increase%20the%20RAM%20and%20%2F%20or%20processors%20and%20%2F%20or%20disk%20speeds%20and%20see%20if%20it%20can%20handle%20large%20loads.%3C%2FLI%3E%3CLI%3EAlternatively%2C%20evaluate%20the%20SharePoint%20timer%20and%20indexing%20jobs%20if%20you%20can.%20You%20may%20be%20able%20to%20increase%20the%20frequency.%26nbsp%3B%20Alternatively%2C%20this%20should%20give%20you%20clues%20on%20the%20delays%20you%20may%20need%20to%20add%20to%20your%20code%20to%20properly%20pause%20the%20record%20passing.%26nbsp%3B%20If%20you%20are%20bound%20to%20the%20speed%20of%20the%20SharePoint%2C%20then%20coding%20to%20pause%20to%20those%20speeds%20is%20the%20best%20you%20can%20do.%3C%2FLI%3E%3C%2FOL%3E%3CP%3EHope%20these%20ideas%20help%20and%20there%20may%20be%20more.%26nbsp%3B%20If%20you%20need%20assistance%2C%20my%20company%20specializes%20in%20these%20things%20and%20may%20be%20able%20to%20help.%26nbsp%3B%20%3CA%20title%3D%22eSoftware%20Associates%22%20href%3D%22https%3A%2F%2Fwww.eswcompany.com%2Fsharepoint-consulting-2%2F%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3EClick%20here%20for%20assistance%20in%20a%20new%20browser%20tab%3C%2FA%3E.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
Super Contributor

We're currently developing a solution that involves several types of Taxonomy operations:

 

  • Loading Terms 
  • Creating Terms (if not exist) 
  • Creating Term Labels (if not exist)
  • Reusing Terms (if not already done)

In our logic many of the operations need to be combined to achieve our desired results. We're using .NET Managed CSOM. We're experiencing very inconsistent behavior in using the Taxonomy API's. Running the same code with the same input several times can have different results:

 

  • 80-90% of the time no errors at all and results as expected
  • Server Exception when reusing or loading a Term:  "The object is invalid or outdated. Please re-fetch the object"
  • Server Exception when reusing or loading a Term: "Taxonomy item instantiation failed. Unable to create a client object with ID {some id}"
  • The HTTP service located at http://usr19769-643:32843/3115c78f4d804003b5964d1cb5b56042/MetadataWebService.svc is unavailable
  • There was no endpoint listening at http://usr19769-643:32843/3115c78f4d804003b5964d1cb5b56042/MetadataWebService.svc that could accept the message

 

The Taxonomy API doesn't seem to be very robust. We've currently implemented retry mechanisms with delays in our code so that ultimately everything ends up in the desired state, but this feels very dirty. Since most of the time the results are what we expect, I can't imagine the problems are caused by our usage. I would expect consistently bad results if the problem was in our API usage, not this random behavior.

 

Does anyone have similar experiences? What are your approaches in usage and error handling?

7 Replies
Highlighted

Hi Paul this sounds like a SharePoint timer or indexing issue.  Some things to try out and test:

  1. How many terms are you loading or creating? Try reducing the number as see if that resolves the issue.
  2. How often are you running this solution? Try reducing the number of runs and see if that resolves the issue.

If either of the above resolves the issue there are 2 ways off the top of my head for handling this:

  1. Easiest solution is to “throw more power at it”. Increase the RAM and / or processors and / or disk speeds and see if it can handle large loads.
  2. Alternatively, evaluate the SharePoint timer and indexing jobs if you can. You may be able to increase the frequency.  Alternatively, this should give you clues on the delays you may need to add to your code to properly pause the record passing.  If you are bound to the speed of the SharePoint, then coding to pause to those speeds is the best you can do.

Hope these ideas help and there may be more.  If you need assistance, my company specializes in these things and may be able to help.  Click here for assistance in a new browser tab.

Highlighted

Hi Joel,

 

Thanks for your response! Sorry I forgot to mention the environment I'm working in is SharePoint Online so no option to increase hardware.

 

The logic is running in an Azure Function App by queueu-triggered Functions. I have also written some Unit Tests where I see the same happening (e.g. Unit Tests succeed most of the times, but frequently fail due to one of the errors liststed above). Very unpredictable...

Highlighted

Ahh yeah there is a max memory usage on functions, but I doubt you are hitting it with taxonomy changes.  Similarly though if you are using large amounts, try scaling down the load and see what happens.  A bit painful for performance testing unless you have some "real" performance testing tools at your disposal that play nicely with Azure.  Good luck or hopefully someone can give a better answer than me!

 

Max: 1,536MB

 

Source: https://azure.microsoft.com/en-us/pricing/details/functions/

Highlighted

Hi Paul -

I'm curious if you've made progress on this this.  I am also seeing the error: 

The HTTP service located at http://usr19769-643:32843/3115c78f4d804003b5964d1cb5b56042/MetadataWebService.svc is unavailable

 

In my case, I was using PnP PowerShell (Apply-PnPProvisioningTemplate) to deploy new Managed Metadata fields and anchor them to an existing term set.  I was running this after creating a new site.  I added a Sleep for 120 seconds after site creation and before applying the template.  I still saw this error.  Then I set it to sleep for 180 seconds and I did not see the error.  I need to do more testing to see if 180 seconds is sufficient in all cases or if I'll need to implement retries as you have done.

 

Regards,

Tom Castiglia

Highlighted

Hi Tom,

 

 

No real progress regarding this on my side. My code is full of retry-mechanisms (basically try-catch-retry) to work around the errors I've listed and the end-results are fine now…

 

Hope your workaround suffices too!

 

 

Regards,

Paul

Highlighted

Hi Tom,

 

Having the sleep for 180 seconds fixed it for me too. Have you found any other way to get this working other than putting in some delay?

 

Thank you.

Highlighted

I implemented the follow extension to get around the problem.

 

public static async Task ExecuteQueryRetry(this ClientContext clientContext, int delaySeconds = 2)
{
while (delaySeconds < 30)
{
try
{
clientContext.ExecuteQuery();
return;
}
catch (ServerException ex)
{
var retry = ex.Message != null ?
ex.Message.Contains("MetadataWebService.svc")
|| ex.Message.Contains("GetDefaultSiteCollectionTermStore") : false;

if (!retry)
{
throw;
}

var delay = TimeSpan.FromSeconds(delaySeconds);
await Task.Delay(delay);

delaySeconds = delaySeconds * 2;
}
}

throw new TimeoutException("Stopped retrying ExecuteQuery after 30 seconds.");
}