Introducing Azure OpenAI Service On Your Data in Public Preview

[Andy-Beatman]

We are excited to announce the launch of Azure OpenAI Service on your data in public preview, a groundbreaking new feature that allows you to harness the power of OpenAI models, such as GPT-4, with your own data. This new and highly requested customer capability revolutionizes the way you interact with and analyze your data, providing greater accuracy, speed, and valuable insights. Let’s explore the features, use cases, data sources, and next steps for leveraging Azure OpenAI Service on your data.

Unlocking the Potential of Your Data

Azure OpenAI Service on your data empowers you to unlock the full potential of your data by running OpenAI models directly on it, eliminating the need for training or fine-tuning. With the advanced conversational AI capabilities of ChatGPT and GPT-4, you can streamline communication, enhance customer service, and boost productivity throughout your organization. These models not only leverage their pre-trained knowledge but also access specific data sources, ensuring that responses are based on the latest available information.

Key Use Cases

Azure OpenAI Service on your data offers a wide range of applications that can revolutionize businesses across various industries. Whether it's enabling self-service data requests, enhancing customer service, driving revenue generation, or improving productivity in B2C and B2B interactions, the possibilities are endless. This powerful service simplifies processes such as document intake and indexing and provides quick access to legal insights and financial data for better decision-making. Additionally, it empowers your organization to tap into resources for accurate marketing responses and streamline software development and HR procedures.

By harnessing Azure OpenAI Service on your data, you can harvest valuable customer insights, monetize access to data, and gain deep industry and competitor insights. The transformative impact on productivity, revenue, and strategic decision-making can help your business thrive in today's fast-paced market. We are excited to learn how our customers leverage this service to transform their operations, improve customer experiences, and gain a competitive edge.

How it works

Azure OpenAI Service on your data ingests and connects your data from any source, regardless of its location. Whether your data is stored locally or in the cloud, we provide seamless connectivity to unlock its full potential. With our advanced tools, you can efficiently process, organize, and optimize your data, gain valuable insights and enhancing its quality. Additionally, our user-friendly API and SDK enable easy integration with your existing systems, and we offer a customizable sample app for quick implementation. Sharing and utilizing your data is made effortlessly, allowing you to distribute information within your organization or with your customers promptly.

Azure OpenAI Service on your data supports connecting to multiple sources, including:

• Azure Cognitive Search index: You can connect your data to an Azure Cognitive Search index, enabling seamless integration with OpenAI models.
• Azure Blob storage container: Connect your data to an Azure Blob storage container and easily access it for analysis and conversation using Azure OpenAI Service
• Local files: Connect to files in your Azure AI portal, providing flexibility and convenience in connecting your data. We ingest and chunk the data into your Azure Cognitive Search index. File formats such as txt, md, html, Word files, PowerPoint, and PDF can be utilized for analysis and conversation.

Azure AI Studio: Chat Playground and Deployment Options

Explore the capabilities of Azure OpenAI Service models with the chat playground, a no-code environment where you can experiment, iterate, and generate completions based on your prompts. The chat playground provides an easy-to-use interface and also offers Python and curl code samples for integration into your own applications.

You have the option to deploy a web app directly from Azure AI studio, creating a conversational AI platform accessible to your organization. The deployment process is seamless, allowing you to choose a name, select the appropriate subscription, resource group, location, and pricing plan for your web app. Web apps are also customizable by the user. You can use the API and endpoint to integrate this feature in the service, not just in an app.

Azure OpenAI Service on your data is a game-changer in the field of conversational AI and data analysis. By leveraging OpenAI models with your own data, you can unlock powerful insights, improve decision-making, and enhance productivity across your organization. Whether you choose to utilize the chat playground in Azure AI Studio for experimentation or deploy a web app for broader accessibility, Azure OpenAI Service on your data offers a seamless and efficient solution.

How to use Azure OpenAI on your data

To begin utilizing Azure OpenAI Service on your data, you need to have an approved Azure OpenAI Service application and an Azure OpenAI Service resource with either the gpt-35-turbo or the gpt-4 models deployed. Once you meet the prerequisites, follow these steps:

• Connect your data source: Use Azure AI Studio to connect your desired data source, whether it's an Azure Cognitive Search index, Blob storage container, or by uploading files locally.
• Ask questions and chat on your data: Once your data source is connected, you can start asking questions and conversing with the OpenAI models through Azure AI Studio. This enables you to gain valuable insights and make informed business decisions.

To get started with Azure OpenAI Service on your data ensure you have the necessary prerequisites, connect your data source using Azure AI studio, and begin asking questions and chatting with the models. Experience the transformative power of conversational AI and take your data analysis to new heights with Azure OpenAI Service on your data.

A Responsible Approach

Microsoft has a layered approach for generative models, guided by Microsoft’s responsible AI principles. In Azure OpenAI Service, an integrated safety system provides protection from undesirable inputs and outputs and monitors for misuse. In addition, Microsoft provides guidance and best practices for customers to responsibly build applications using these models and expects customers to comply with the Azure OpenAI Code of Conduct. With Open AI’s GPT-4, new research advances from OpenAI have enabled an additional layer of protection. Guided by human feedback, safety is built directly into the GPT-4 model, which enables the model to be more effective at handling harmful inputs, thereby reducing the likelihood that the model will generate a harmful response.

Data, Privacy and Security
Learn about the processing, usage, and storage of data in the Azure OpenAI Service.

Resources

• Apply now for access to Azure OpenAI Service 
• Review the new documentation for Azure OpenAI Service on your data
• Explore the playground and customization in Azure AI Studio
• Check out the new QuickStart
• Learn more about Data, Privacy and Security for Azure OpenAI Service
• Create a resource
• Bookmark the What's New page

[NetaH]

It's Live now ! You should see add your data (preview) in the Azure OpenAI portal ChatGPT playground and an option Bring your own data (preview) 

[Martin Liljenberg]

Only for new distribution? I have GPT3.5 turbo, not showing up there unfortunately-

[NetaH]

@Martin Liljenberg You should see it. Please refresh and if you can't still can't see it and have access to Azure OpenAI Service please open a support ticket so we can assist. 

[Martin Liljenberg]

Thank you now it is there! I will start playing =))

[tsupreet]

Had been waiting for this for some time. Its also showing up in my playground & I've already started playing with it now. Thank you. 

The documentation links are still returning 404. 

A follow up question, is adding SQL DB as custom data possible/ in works? That would be a game changer. 

[NetaH]

@tsupreet documentation will be live also today after 3PM PST, stay tuned. 
If your SQL DB also has descriptions and unstructured information within for example a supports tickets DB etc you can ingest it into a Cognitive Search index and use the description column for example as the content column in  Azure OpenAI on your data and try it out. If your SQL DB is totally structured than using Azure OpenAI on your data with structured data is not yet supported. 

[Sachidanand_Sharma_1]

It is amazing .. Could you share the architecture of this model where it is connecting with Enterprise database?

[CooperWang]

I can't wait....  Could you provide more instructions about how to create a web portal which allows our company to chat with AI? Is the web portal hosted on Azure Web App Service?

[Bytewolf]

Hey MSFT team

Can Azure OpenAI update available models please? Function calling feature would be great requires models gpt-3.5-turbo-0613 and gpt-4-0613

gpt4 is not available and on gpt3.5 Im only getting access to gpt-3.5-turbo-0301

https://platform.openai.com/docs/guides/gpt/function-calling

As you know getting useful JSON responses can at times be challenging and this can help.

Thank you, Sergio

[Joel_Borellis]

Excited to try this out.  Look forward to trying this with Azure Cognitive Search Index and the new vector search.

[arnabbiswas]

Is there any plan to release Python APIs for this feature?

[NetaH]

@arnabbiswas Yes, SDK is coming soon with Python API. 
@CooperWang see here more details on how to create a Web App and share it https://learn.microsoft.com/en-us/azure/cognitive-services/openai/use-your-data-quickstart?tabs=comm... and the WebApp github - https://github.com/microsoft/sample-app-aoai-chatGPT/tree/main

[jigmoth]

I get an error when asking, do anyone get this or know solutions.?

Error

Missing header 'chatgpt_url' in request.

[rameshbkanagaraj]

Yes, I am getting the same error: Missing header 'chatgpt_url' in request.

[Mariama13]

hello I try to download a pdf file but every time that I only have the final click to save, I get the error that this service is unavailable

[wangjueliang]

Hey guys,

Quick question, in the data source, how do I point it to SharePoint Online?

Cheers,

[NetaH]

@wangjueliang Sharepoint Online is not yet supported as a data source. 

[mpjjonker]

Is this the productization of this Azure Samples? https://learn.microsoft.com/en-us/answers/questions/1179152/azure-cognitive-search-azure-openai-exam...

[vkrishnaganga]

Hi All, We have tried using this option and added storage account as a source. But the model is still answering generic questions. We tried providing various prompts to restrict the non organizational data but no luck. Any immediate guidelines on how to restrict model to answer only from attached source and not any other questions.

[cocxy]

Hello,

2 questions:

How to save / get back an "Add your data" session in the Studio? It's not persistent.

I get a deployment failed error today for the web app but managed yesterday, how to get detailed errors?

Best regards.

[Jani_Iivari]

Rest API doesn't seem to work:

https://[Endpoint]/openai/deployments/[modelDeployment]/extensions/chat/completions?api-version=2023-06-01-preview

Method: POST

Headers:

Content-Type:application/json
api-key:[MYKEY]
chatgpt_url:[Endpoint]
chatgpt_key:[MYKEY]

Body:

{

  "dataSources": [

    {

      "type": "AzureCognitiveSearch",

      "parameters": {

        "endpoint": "[COGNITIVESEARCHENDPOINT]",

        "key": "[COGNITIVESEARCHKEY]",

        "indexName": "[COGNITIVESEARCHINDEXNAME]"

      }

    }

  ],

  "messages": [

      {

            "role": "user",

            "content": "what are the 10 traps to avoid in data catalog project"

        }

   ],

  "temperature": 0.5,

  "top_p": 0.95,

  "frequency_penalty": 0,

  "presence_penalty": 0,

  "max_tokens": 800,

  "stop": null

}

Response:

{

    "error": {

        "code": "InternalServerError",

        "message": "Backend returned unexpected response. Please contact Microsoft for help."

    }

}

Note: works fine from the playground, and the Cognitive Search responds to GET: https://[COGNITIVESEARCHENDPOINT]/indexes/[COGNITIVESEARCHINDEXNAME]?api-version=2021-04-30-Preview

[Vishal251943]

Informative 

[Mischa_van_oijen]

This a great stuff, but it seems limited at the moment (which is understandable as its preview)

Some questions:

1. When will Sharepoint as a data repo be supported, so Upload document to SharePoint, SharePoint Online indexer?

2. When will the hybrid search (embeddings) be supported for this? 

3. Will there also be templates to translate it to a virtual agent, or a normal teams bot?

4. Will you be able to query your own data from the powerplatform (PowerApp, PowerAutomate etc)

5. Will this integrate with the upcoming plugins announced? In other words could the source be a mix of structured and unstructured?

6. Do the Azure openAI T&A allow us to share it with a customer, for example to support a demo based on their own data?

Definition of awesome

1. User uploads files to SharePoint Online

2. SharePoint Online indexer, add to cog search index + embeddings (to support vector search and hybrid search) => https://github.com/Azure/cognitive-search-vector-pr

3. You can use plugins to also connect to SQL databases => https://techcommunity.microsoft.com/t5/ai-cognitive-services-blog/generative-ai-for-developers-explo...

4. You can use functions to also get data from other sources => https://openai.com/blog/function-calling-and-other-api-updates

5. The bring your own data feature, will search vectors and returns better results as just the cognitive semantic search 

[Mark_Rubio]

A few bugs that we consider very critical. I understand this is just a preview. But this sometimes produce unwanted results outside of "your own data".

Also, the REST API documentation is not somehow fully documented. I have copied the sample cURL command so I can test it out using a REST client but I am getting an error "Backend returned unexpected response. Please contact Microsoft for help."..

To resolve the error, I captured the Network traffic from the Playground and imported it to my Postman client.

[NetaH]

@vkrishnaganga and @Mark_Rubio and @Jani_Iivari  can you please open a support ticket and we can assist in resolving the issue. 

[wangjueliang]

Hi @NetaH,

This app supports ACS Index now so would it worth exploring a workaround to convert the existing SPO data into the index?

If the SPO direct support is coming soon, then there is no point wasting money indexing our SPO with ACS.

Cheers,

[vkrishnaganga]

Hi @NetaH 
Thank you for the acknowledgement. I have raised the Microsoft support ticket "2306230030002857". Request you to expedite the process.

[tdang740]

Hi,

I've connected to the cognitive search data source and got this error when chatting in the playground:

Error

Azure Search Error: 400, message='Server responded with status 400. Error message: {"error":{"code":"FeatureNotSupportedInService","message":"Parameter 'speller' is not supported on 'free' tier search services, nor search services created before 2019-01-01. See https://aka.ms/azs-speller for more information.\r\nParameter name: speller","details":[{"code":"SpellerNotAvailable","message":"Parameter 'speller' is not supported on 'free' tier search services, nor search services created before 2019-01-01. See https://aka.ms/azs-speller for more information."}]}}', url=URL('https://xxx.search.windows.net/indexes/xxx/docs/search?api-version=2021-04-30-Preview')
Call to Azure Search instance failed.
API Users: Please ensure you are using the right instance, index_name and provide admin_key as the api_key.

We use Basic tier and our service has been created recently.

[Ron_Maraykhin]

Hello,

Is it possible to save the chat history and access it from the WebApp like on the Chat Gpt site? 

And is it possible to upload files directly from the WebApp and update the existing WebApp ?

Thanks!

[PascalBurume]

Chat with your own data has worked for me with Power Platform, specially Power Pages.

[Jani_Iivari]

Did the same as Mark. Captured the REST API from network traffic when using the chat playground.
Here's the updated REST API Call

URL: https://[YOURENDPOINT]/openai/deployments/[YOURMODEL]/extensions/chat/completions?api-version=2023-06-01-preview

Method: POST
Headers:
Content-Type:application/json
api-key: [YOURAPIKEY]
chatgpt_url: https://[YOURENDPOINT]/openai/deployments/[YOURMODEL]/chat/completions?api-version=2023-03-15-preview
chatgpt_key:  [YOURAPIKEY]

{

    "messages": [

        {

            "role": "user",

            "content": "what are the 10 traps to avoid in data catalog project"

        }

    ],

    "temperature": 0.5,

    "top_p": 0.95,

    "frequency_penalty": 0,

    "presence_penalty": 0,

    "max_tokens": 800,

    "stop": null,

    "deployment": "gtp-35-turbo",

    "stream": false,

    "dataSources": [

        {

            "type": "AzureCognitiveSearch",

            "parameters": {

                "endpoint": "[OPENAIENDPOINT]",

                "key": "[OPENAIENDKEY]",

                "indexName": "[CSSEARCHINDEX]",

                "semanticConfiguration": "",

                "queryType": "simple",

                "fieldsMapping": {

                    "contentFieldsSeparator": "\n",

                    "contentFields": [

                        "content"

                    ],

                    "filepathField": "metadata_storage_name",

                    "titleField": null,

                    "urlField": null

                },

                "inScope": false,

                "roleInformation": "You are an AI assistant that helps people find information."

            }

        }

    ]

}

[jigmoth]

@Jani_Iivari  I get this error in postman same as i get from portal. Any suggestions?

{

    "error": {

        "requestid": "3483b2a0-36d2-4efc-9eed-14f58af3e204",

        "code": 400,

        "message": "Missing header 'chatgpt_url' in request."

    }

}

[aymiee]

This works great for me if I am just browsing pdf, .docx, and txt files, i.e. connecting via Sharepoint Indexer and targeting the index and seeing it in the Chat Playground.  But my challenge has always been excel files.  I have a Sharepoint Indexer pointing to a Document Library using the "useQuery".  Even though there are 2 folders and one file, only contents from the first folder was indexed.  The excel files has two worksheets but only the first worksheet was indexed.  What was indexed in the content field was a bunch of gibberish in the Field Explorer of that index.  In the Playground, data is not queryable.  

Has anyone successfully index excel files?  I can control what is indexed via Microsoft Graph API but I still have the same problem with gibberish in the index and data not being queryable for Excel worksbooks.  What approach are others using?  Do I need to look at embeddings?  Any insight would be great! 

Thank you in advance.

[rawan0819]

@jigmoth , I assume your AOAI resource is using a private endpoint or vnet ? We detected an issue around it and are rolling out mitigation. The fix will hit all the regions by the end of tomorrow. Please try again later and let us know if the issue still existed

[rawan0819]

@Jani_Iivari , the chatgpt_url needs to be the chat/completion endpoint like this : https://[ENDPOINT]/openai/deployments/[DEPLOYMENT]/chat/completions?api-version=2023-03-31-preview

[im_bmw]

Probably I'm misunderstanding a bunch of this--it's all very new to me (sorry!).  I was able to create a cognitive search datasource and index based on a SQL table of info.  When connected here, I expected to be able to chat about the data by referring to column names, but there doesn't seem to be an automatic association between column names and chat terms--is this correct?

Also, when I view a citation, or try to ask questions about my non-textual fields (dates, integers, etc) they seem to be ignored.  Is this also expected?  I was hoping to be able to chat things like:  Show me book checked out after mm/dd/yyyy, or How many books were checked out in April, or What is the total overdue amount from East Library...

Maybe I'm expecting things this method isn't attempting to provide...

Thanks for any clarification!

[jigmoth]

@rawan0819 The behavior is inconsistent with different errors, now i started getting different error, i am making sure i don't have any private endpoint for my OpenAI service.

Error: sequence item 15: expected str instance, list found

Also for your information, if i have private endpoint, its still not working, and gives the same error for private endpoint, i believe you said its now already fix but its not actually

[Andre Heim]

hi all, has anyone already tried to use it with Power Flow? I'm in search of how to configure the HTTP step correctly.

[cocxy]

What should I do when the Azure Cognitive Search index created by the Az OAI studio stays at 0 document, 0 byte?

Thanks.

[janahza]

Hi Everyone,

We would like to limit the access to the files based on the profils they have in RBAC. 
For ex: Profil A can only get answers about topic 1-2 but Profil B can get answers about topic 1-2-3-4-5. 

Do you think it's possible to implement this here? If yes, how, please? 

Thanks in advance! 

NB: Topic 1-2-3-4-5 are pdf files :)

[FabioBielli]

Hi

I did same test in the playground publishing a web App on a dev tenant I use for testing. Before moving on the production tenant I am wondering if:

• I understand the cost is around 0,001835 for 1000 token. To simplify can I assume this would be the price of a couple of autogenerated answers? 
• I did a test with a pdf (a toolkit we have). What is the cost for the document libraries I use as knowledge base for the model? 
• In case could it be possible to customize some graphic part of the chatbot with company brand identity or layout or other web parts?

Thank you!

[Mariama13]

Hello, I don't know if this is the right place to ask my question but I'm trying my luck.

I work on a project which consists in inserting my own documents (pdf) in azure openAI and from GPT 3.5 Turbo, I deploy a web application, in which I ask my questions in the assistant and the answers will be based on of the document.

Only I can not savor the distinction between the standard model and the personalized model, because it seems to me they do the same thing.

What I understood is that the custom model if it does not find information in our imported data, it will not be able to look for additional information in the pretrainer model of GTP3 unlike the standard model. Am I wrong or not?

PS: I have already read the documents in the Microsoft site.

[gozcom8888]

Playground Chat does not support tabular file formats - eg. CSV.  Even if you save the CSV as a TXT (supported by Chat) it's been my experience the model doesn't recognize the file.  This presents a limitation if you want to use a web app as a UI.  

I'm wondering if others have had the same experience.

However, the playground Assistant does support CSV files.  How can we make this model available from a web app?  Is that a supported config?

I have a case open with MS, waiting to hear back.  Existing documentation does not address this.

Thanks in advance for future replies.