Multi tenant rate throttling for external resources

%3CLINGO-SUB%20id%3D%22lingo-sub-1323087%22%20slang%3D%22en-US%22%3EMulti%20tenant%20rate%20throttling%20for%20external%20resources%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1323087%22%20slang%3D%22en-US%22%3E%3CP%3EI'm%20building%20a%20reporting%20app%20that%20consumes%20data%20from%20a%20customer's%20Azure%20Devops.%20There%20will%20be%20a%20daily%20import%20that%20runs%20maybe%2050k%20requests%20to%20the%20customer's%20installation%20of%20Azure%20Devops%20(this%20takes%20maybe%20an%20hour)%2C%20but%20there%20is%20also%20the%20initial%20setup%20that%20runs%20~10x%20that%20amount.%20There%20are%20multiple%20customers%2C%20with%20new%20setups%20happening%20while%20others%20are%20doing%20their%20daily%20regular%20loads.%20I'd%20like%20to%20do%20the%20loads%20with%20a%20series%20of%20functions%2C%20as%20the%20utilization%20is%20sporadic%20and%20spiky.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAt%20some%20point%20in%20the%20chain%20of%20functions%2C%20there%20will%20be%20one%20that%20creates%20a%20record%20in%20a%20queue%20for%20each%20and%20every%20test%20result%2C%20where%20we%20load%20the%20result%20and%20attachments.%20This%20is%20where%20the%20numbers%20get%20big%20and%20durations%20get%20long.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EGoals%3A%3C%2FP%3E%3CP%3EI'd%20like%20to%20minimize%20any%20Azure%20resources%20that%20are%20configured%20per%20customer.%3C%2FP%3E%3CP%3EI%20know%20I%20have%20to%20throttle%20the%20number%20of%20concurrent%20calls%20per%20customer%20(not%20everyone%20is%20in%20the%20cloud).%3C%2FP%3E%3CP%3EI%20do%20not%20want%20to%20throttle%20the%20number%20of%20concurrent%20calls%20across%20customers.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWould%20I%20create%20multiple%20queues%2C%20rate%20limit%20each%20queue%20and%20write%20new%20batches%20to%20whatever%20queue%20is%20the%20shortest%3F%20Maybe%20put%20in%20some%20sort%20of%20affinity%20logic%2C%20and%20special%20initial%20load%20queues.%20This%20all%20feels%20somewhat%20hacky%20so%20I%20hope%20there%20is%20a%20cleaner%20solution.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ECreating%20a%20Queue%20%26amp%3B%20Worker%20per%20customer%20automatically%20might%20be%20the%20best%20solution%20from%20a%20performance%20perspective.%20My%20gut%20says%20that%20is%20not%20a%20great%20idea%2C%20but%20maybe%20I'm%20wrong.%3CBR%20%2F%3E%3CBR%20%2F%3EI'm%20open%20to%20other%20technologies%2C%20it%20does%20not%20have%20to%20be%20a%20queue%20%26amp%3B%20function%2C%20although%20the%20on-demand%20behavior%20of%20the%20function%20seems%20hard%20to%20beat.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1330117%22%20slang%3D%22en-US%22%3ERe%3A%20Multi%20tenant%20rate%20throttling%20for%20external%20resources%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1330117%22%20slang%3D%22en-US%22%3E%3CP%3EThe%20solution%20I'll%20try%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EFor%20the%20part%20where%20I%20expect%20the%20load%20to%20become%20problematic%20I'll%20do%20the%20following%3B%3C%2FP%3E%3CP%3EEach%20tenant%20will%20have%20their%20own%20resource%20group%20which%20will%20contain%20a%26nbsp%3B%3C%2FP%3E%3CP%3EQueue%3C%2FP%3E%3CP%3EFunction%3C%2FP%3E%3CP%3EAPI%20gateway%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20API%20gateway%20will%20be%20in%20front%20of%20the%20external%20system%20and%20will%20take%20care%20of%20the%20rate-limiting%20for%20the%20function.%20This%20could%20create%20a%20possibly%20high%20reprocess%20rate%20from%20the%20queue%20(and%20I'd%20pay%20to%20fail%20a%20request%20I'm%20paying%20for).%20I'd%20limit%20the%20function%20from%20scaling%20as%20well%2C%20but%20the%20only%20way%20to%20get%20fine-grained%20control%20is%20to%20put%20in%20an%20API%20gateway.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThis%20will%20only%20work%20for%20so%20many%20tenants%20as%20I'd%20hit%20limits%20on%20resource%20groups%20and%20functions.%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3EI'm%20not%20thrilled%20with%20this%20solution%2C%20but%20it%20looks%20plausible.%3C%2FP%3E%3C%2FLINGO-BODY%3E
New Contributor

I'm building a reporting app that consumes data from a customer's Azure Devops. There will be a daily import that runs maybe 50k requests to the customer's installation of Azure Devops (this takes maybe an hour), but there is also the initial setup that runs ~10x that amount. There are multiple customers, with new setups happening while others are doing their daily regular loads. I'd like to do the loads with a series of functions, as the utilization is sporadic and spiky.

 

At some point in the chain of functions, there will be one that creates a record in a queue for each and every test result, where we load the result and attachments. This is where the numbers get big and durations get long.

 

Goals:

I'd like to minimize any Azure resources that are configured per customer.

I know I have to throttle the number of concurrent calls per customer (not everyone is in the cloud).

I do not want to throttle the number of concurrent calls across customers. 

 

Would I create multiple queues, rate limit each queue and write new batches to whatever queue is the shortest? Maybe put in some sort of affinity logic, and special initial load queues. This all feels somewhat hacky so I hope there is a cleaner solution. 

 

Creating a Queue & Worker per customer automatically might be the best solution from a performance perspective. My gut says that is not a great idea, but maybe I'm wrong.

I'm open to other technologies, it does not have to be a queue & function, although the on-demand behavior of the function seems hard to beat.

1 Reply

The solution I'll try

 

For the part where I expect the load to become problematic I'll do the following;

Each tenant will have their own resource group which will contain a 

Queue

Function

API gateway

 

The API gateway will be in front of the external system and will take care of the rate-limiting for the function. This could create a possibly high reprocess rate from the queue (and I'd pay to fail a request I'm paying for). I'd limit the function from scaling as well, but the only way to get fine-grained control is to put in an API gateway.  

 

This will only work for so many tenants as I'd hit limits on resource groups and functions. 

I'm not thrilled with this solution, but it looks plausible.