Load Testing Azure OpenAI With JMeter

sergiis this blog is on testing the AOAI endpoints or an APIM which handles the multiple endpoints. This is needed to handle the concurrency expected in production scenarios and the impact of throttling and latency. Based on the performance results, there are options to explore increasing quota or look at PTUs or split the quota into multiple endpoints. Load testing of the applications which invoke AOAI as part of their logic is not the intended scope here. You have to hit the AOAI endpoints with expected load to understand the response patterns.