Code First Distillation with Stored Completions in Azure OpenAI Service

Former Employee

Feb 25, 2025

We are thrilled to announce the Public Preview release of the Stored Completions API and SDK in Azure OpenAI Service! Following our recent announcement on the enhanced Azure OpenAI Service distillation and Fine-Tuning Capabilities, we are excited to introduce a set of new API capabilities and an SDK experience that will empower our customers to interact with Stored Completions through code.

What were we supporting before?

Model distillation empowers developers to use the outputs of large, complex models to fine-tune smaller, more efficient ones. This technique allows the smaller models to perform just as well on specific tasks, all while significantly cutting down on both cost and latency. Azure OpenAI Service distillation involves three main components:

Stored Completions: Capture and store input-output pairs from models like GPT-4o to generate datasets for evaluation and fine-tuning. The feature offers an interface for reviewing, filtering, and exporting data based on predefined criteria.
Evaluation: Create and run custom evaluations to measure model performance using data from Stored Completions or existing datasets. The Evaluation experience provides a UI-based approach to score data.
Fine-tuning: Integrate Stored Completions and Evaluation with Azure OpenAI fine-tuning. Use datasets from Stored Completions in fine-tuning jobs and run evaluations on fine-tuned models.

Together, these steps create a comprehensive distillation process: collecting live traffic, filtering and subsetting it in the Stored Completions UI, exporting it to the Evaluation UI for quality scoring, and fine-tuning from the collected data.

What is new?

In addition to our UI offering for Stored Completions, we are now providing a code-first approach to an end-to-end distillation flow in Azure OpenAI Service.

We are introducing Stored Completions API that help our customers to:

Retrieve stored completions by ID: GET /chat/completions/:id
Get stored completions messages: GET /chat/completions/:id/messages
Get a full list of stored completions: GET /chat/completions
Update metadata of stored completions: POST /chat/completions/:id
Delete stored completions by ID: DELETE /chat/completions/:id

These new capabilities will enable developers to seamlessly integrate stored completions into their workflows, providing greater flexibility and control over their data and models in the Distillation Process.

Maximizing Efficiency and Control with Stored Completions API and SDK

The value of using the API and SDK for retrieving, updating, or deleting stored completions lies in the enhanced flexibility, efficiency, and control they offer developers. The Stored Completions UI provides an interface for reviewing, filtering, and exporting data based on predefined criteria. The new APIs not only enable the retrieval of stored completions but also support update and delete functionalities.

Retrieving Stored Completions: The API and SDK allow developers to programmatically access stored completions, enabling seamless integration into their workflows. This is particularly useful for automated dataset generation, ensuring that the data is always up-to-date and relevant for training machine learning models.
Updating Stored Completions: By using the API and SDK to update metadata or other attributes of stored completions, developers can maintain accurate and organized datasets. This helps in refining the data for better evaluation and fine-tuning of models.
Deleting Stored Completions: The API and SDK provide the ability to delete stored completions that are no longer needed, ensuring that the dataset remains clean and manageable. This is crucial for maintaining data quality and relevance.

Overall, the API and SDK enhance the efficiency of managing stored completions, making it easier for developers to generate, update, and maintain high-quality datasets for their machine learning projects.

Getting Started with the Stored Completions API and SDK

Here is an example of an API call to get list of stored completions:

Example Request and Response:

To get started with the SDK, you can follow these steps: Install the SDK, Initialize the client instance to interact with the APIs and start using the APIs to interact with stored completions.

And here is an example of how to use the SDK:

The introduction of the Stored Completions API and SDK in Azure OpenAI Service marks a significant milestone in our journey to enhance the capabilities of our platform. These new functionalities provide developers with greater flexibility, efficiency, and control over their data and models, enabling them to create high-quality datasets seamlessly. We are excited to see how our customers will leverage these new features to drive innovation and achieve their goals. Stay tuned for more updates and happy coding!