Forum Discussion
Introduction to Content filtering and Embeddings in Azure Open AI Service
Content Filter : Generated using Microsoft Desginer
Content filtering and Embeddings in Azure AI Open Service:
Abuse Monitoring
- Content Classification: Classifier models detect harmful language and/or images in user prompts (inputs) and completions (outputs). The system looks for categories of harms as defined in the Content Requirements, and assigns severity levels as described in more detail on the Content Filtering page.
- Abuse Pattern Capture: Azure OpenAI Service’s abuse monitoring looks at customer usage patterns and employs algorithms and heuristics to detect indicators of potential abuse. Detected patterns consider, for example, the frequency and severity at which harmful content is detected in a customer’s prompts and completions.
- Human Review and Decision: When prompts and/or completions are flagged through content classification and abuse pattern capture as described above, authorized Microsoft employees may assess the flagged content, and either confirm or correct the classification or determination based on predefined guidelines and policies. Data can be accessed for human review only by authorized Microsoft employees via Secure Access Workstations (SAWs) with Just-In-Time (JIT) request approval granted by team managers. For Azure OpenAI Service resources deployed in the European Economic Area, the authorized Microsoft employees are located in the European Economic Area.
- Notification and Action: When a threshold of abusive behavior has been confirmed based on the preceding three steps, the customer is informed of the determination by email. Except in cases of severe or recurring abuse, customers typically are given an opportunity to explain or remediate—and implement mechanisms to prevent recurrence of—the abusive behavior. Failure to address the behavior—or recurring or severe abuse—may result in suspension or termination of the customer’s access to Azure OpenAI resources and/or capabilities.
Content filtering
Azure OpenAI Service includes a content management system that works alongside core models to filter content.
If the system identifies harmful content, you'll receive either
- An error on the API call
- Content_filter as the finish_reason on the response
Mitigate
Mitigating harms presented by large language models such as the Azure OpenAI models requires an iterative, layered approach that includes experimentation and continual measurement.
Best practices – Content filtering
- Consider the following best practices
- Check the finish_reason to see if the generation is filtered
- Check that there's no error object in the content_filter_result
Applications serving multiple end-users should pass the user parameter with each API call.
Embedding: Cosine similarity
- Cosine similarity measures the cosine of the angle between two vectors projected in a multi-dimensional space.
- If two documents are far apart by Euclidean distance because of size, they could still have a smaller angle between them and therefore higher cosine similarity.
Azure OpenAI embeddings rely on cosine similarity to compute similarity between documents and a query.
More details: Azure OpenAI Service embeddings - Azure OpenAI - embeddings and cosine similarity | Microsoft Learn
