I'm building a reporting app that consumes data from a customer's Azure Devops. There will be a daily import that runs maybe 50k requests to the customer's installation of Azure Devops (this takes maybe an hour), but there is also the initial setup that runs ~10x that amount. There are multiple customers, with new setups happening while others are doing their daily regular loads. I'd like to do the loads with a series of functions, as the utilization is sporadic and spiky.
At some point in the chain of functions, there will be one that creates a record in a queue for each and every test result, where we load the result and attachments. This is where the numbers get big and durations get long.
Goals:
I'd like to minimize any Azure resources that are configured per customer.
I know I have to throttle the number of concurrent calls per customer (not everyone is in the cloud).
I do not want to throttle the number of concurrent calls across customers.
Would I create multiple queues, rate limit each queue and write new batches to whatever queue is the shortest? Maybe put in some sort of affinity logic, and special initial load queues. This all feels somewhat hacky so I hope there is a cleaner solution.
Creating a Queue & Worker per customer automatically might be the best solution from a performance perspective. My gut says that is not a great idea, but maybe I'm wrong.
I'm open to other technologies, it does not have to be a queue & function, although the on-demand behavior of the function seems hard to beat.
