Azure AI Assistants with Logic Apps
Introduction to AI Automation with Azure OpenAI Assistants Intro Welcome to the future of automation! In the world of Azure, AI assistants are becoming your trusty sidekicks, ready to tackle the repetitive tasks that once consumed your valuable time. But what if we could make these assistants even smarter? In this post, we’ll dive into the exciting realm of integrating Azure AI assistants with Logic Apps – Microsoft’s powerful workflow automation tool. Get ready to discover how this dynamic duo can transform your workflows, freeing you up to focus on the big picture and truly innovative work. Azure OpenAI Assistants (preview) Azure OpenAI Assistants (Preview) allows you to create AI assistants tailored to your needs through custom instructions and augmented by advanced tools like code interpreter, and custom functions. To accelerate and simplify the creation of intelligent applications, we can now enable the ability to call Logic Apps workflows through function calling in Azure OpenAI Assistants. The Assistants playground enumerates and lists all the workflows in your subscription that are eligible for function calling. Here are the requirements for these workflows: Schema: The workflows you want to use for function calling should have a JSON schema describing the inputs and expected outputs. Using Logic Apps you can streamline and provide schema in the trigger, which would be automatically imported as a function definition. Consumption Logic Apps: Currently supported consumption workflows. Request trigger: Function calling requires a REST-based API. Logic Apps with a request trigger provides a REST endpoint. Therefore only workflows with a request trigger are supported for function calling. AI Automation So apart from the Assistants API, which we will explore in another post, we know that we can Integrate Azure Logic Apps workflows! Isn’t that amazing ? The road now is open for AI Automation and we are on the genesis of it, so let’s explore it. We need an Azure Subscription and: Azure OpenAI in the supported regions. This demo is on Sweden Central. Logic Apps consumption Plan. We will work in Azure OpenAI Studio and utilize the Playground. Our model deployment is GPT-4o. The Assistants Playground offers the ability to create and save our Assistants, so we can start working and return later, open the Assistant and continue. We can find the System Message option and the three tools that enhance the Assistants with Code Interpreter, Function Calling ( Including Logic Apps) and Files upload. The following table describes the configuration elements of our Assistants: Name Description Assistant name Your deployment name that is associated with a specific model. Instructions Instructions are similar to system messages this is where you give the model guidance about how it should behave and any context it should reference when generating a response. You can describe the assistant’s personality, tell it what it should and shouldn’t answer, and tell it how to format responses. You can also provide examples of the steps it should take when answering responses. Deployment This is where you set which model deployment to use with your assistant. Functions Create custom function definitions for the models to formulate API calls and structure data outputs based on your specifications Code interpreter Code interpreter provides access to a sandboxed Python environment that can be used to allow the model to test and execute code. Files You can upload up to 20 files, with a max file size of 512 MB to use with tools. You can upload up to 10,000 files using AI Studio. The Studio provides 2 sample Functions (Get Weather and Get Stock Price) to get an idea of the schema requirement in JSON for Function Calling. It is important to provide a clear message that makes the Assistant efficient and productive, with careful consideration since the longer the message the more Tokens are consumed. Challenge #1 – Summarize WordPress Blog Posts How about providing a prompt to the Assistant with a URL instructing it to summarize a WordPress blog post? It is WordPress cause we have a unified API and we only need to change the URL. We can be more strict and narrow down the scope to a specific URL but let’s see the flexibility of Logic Apps in a workflow. We should start with the Logic App. We will generate the JSON schema directly from the Trigger which must be an HTTP request. { "name": "__ALA__lgkapp002", // Remove this for the Logic App Trigger "description": "Fetch the latest post from a WordPress website,summarize it, and return the summary.", "parameters": { "type": "object", "properties": { "url": { "type": "string", "description": "The base URL of the WordPress site" }, "post": { "type": "string", "description": "The page number" } }, "required": [ "url", "post" ] } } In the Designer this looks like this : As you can see the Schema is the same, excluding the name which is need only in the OpenAI Assistants. We will see this detail later on. Let’s continue with the call to WordPress. An HTTP Rest API call: And finally mandatory as it is, a Response action where we tell the Assistant that the Call was completed and bring some payload, in our case the body of the previous step: Now it is time to open our Azure OpenAI Studio and create a new Assistant. Remember the prerequisites we discussed earlier! From the Assistants menu create a [+New] Assistant, give it a meaningful name, select the deployment and add a System Message . For our case it could be something like : ” You are a helpful Assistant that summarizes the WordPress Blog Posts the users request, using Functions. You can utilize code interpreter in a sandbox Environment for advanced analysis and tasks if needed “. The Code interpreter here could be an overkill but we mention it to see the use of it ! Remember to save the Assistant. Now, in the Functions, do not select Logic Apps, rather stay on the custom box and add the code we presented earlier. The Assistant will understand that the Logic App named xxxx must be called, aka [“name”: “__ALA__lgkapp002“,] in the schema! In fact the Logic App is declared by 2 underscores as prefix and 2 underscores as suffix, with ALA inside and the name of the Logic App. Let’s give our Assistant a Prompt and see what happens: The Assistant responded pretty solidly with a meaningful summary of the post we asked for! Not bad at all for a Preview service. Challenge #2 – Create Azure Virtual Machine based on preferences For the purpose of this task we have activated System Assigned managed identity to the Logic App we use, and a pre-provisioned Virtual Network with a subnet as well. The Logic App must reside in the same subscription as our Azure OpenAI resource. This is a more advanced request, but after all it translates to Logic Apps capabilities. Can we do it fast enough so the Assistant won’t time out? Yes we do, by using the Azure Resource Manager latest API which indeed is lightning fast! The process must follow the same pattern, Request – Actions – Response. The request in our case must include such input so the Logic App can carry out the tasks. The Schema should include a “name” input which tells the Assistant which Logic App to look up: { "name": "__ALA__assistkp02" //remove this for the Logic App Trigger "description": "Create an Azure VM based on the user input", "parameters": { "type": "object", "properties": { "name": { "type": "string", "description": "The name of the VM" }, "location": { "type": "string", "description": "The region of the VM" }, "size": { "type": "string", "description": "The size of the VM" }, "os": { "type": "string", "description": "The OS of the VM" } }, "required": [ "name", "location", "size", "os" ] } } And the actual screenshot from the Trigger, observe the absence of the “name” here: Now as we have number of options, this method allows us to keep track of everything including the user’s inputs like VM Name , VM Size, VM OS etc.. Of Course someone can expand this, since we use a default resource group and a default VNET and Subnet, but it’s also configurable! So let’s store the input into variables, we Initialize 5 variables. The name, the size, the location (which is preset for reduced complexity since we don’t create a new VNET), and we break down the OS. Let’s say the user selects Windows 10. The API expects an offer and a sku. So we take Windows 10 and create an offer variable, the same with OS we create an OS variable which is the expected sku: if(equals(triggerBody()?['os'], 'Windows 10'), 'Windows-10', if(equals(triggerBody()?['os'], 'Windows 11'), 'Windows-11', 'default-offer')) if(equals(triggerBody()?['os'], 'Windows 10'), 'win10-22h2-pro-g2', if(equals(triggerBody()?['os'], 'Windows 11'), 'win11-22h2-pro', 'default-sku')) As you understand this is narrowed to Windows Desktop only available choices, but we can expand the Logic App to catch most well know Operating Systems. After the Variables all we have to do is create a Public IP (optional) , a Network Interface, and finally the VM. This is the most efficient way i could make, so we won’t get complains from the API and it will complete it very fast ! Like 3 seconds fast ! The API calls are quite straightforward and everything is available in Microsoft Documentation. Let’s see an example for the Public IP: And the Create VM action with highlight to the storage profile – OS Image setup: Finally we need the response which can be as we like it to be. I am facilitating the Assistant’s response with an additional Action “Get Virtual Machine” that allows us to include the properties which we add in the response body: Let’s make our request now, through the Assistants playground in Azure OpenAI Studio. Our prompt is quite clear: “Create a new VM with size=Standard_D4s_v3, location=swedencentral, os=Windows 11, name=mynewvm02”. Even if we don’t add the parameters the Assistant will ask for them as we have set in the System Message. Pay attention to the limitation also . When we ask about the Public IP, the Assistant does not know it. Yet it informs us with a specific message, that makes sense and it is relevant to the whole operation. If we want to have a look of the time it took we will be amazed : The sum of the time starting from the user request till the response from the Assistant is around 10 seconds. We have a limit of 10 minutes for Function Calling execution so we can built a whole Infrastructure using just our prompts. Conclusion In conclusion, this experiment highlights the powerful synergy between Azure AI Assistant’s Function Calling capability and the automation potential of Logic Apps. By successfully tackling two distinct challenges, we’ve demonstrated how this combination can streamline workflows, boost efficiency, and unlock new possibilities for integrating intelligent decision-making into your business processes. Whether you’re automating customer support interactions, managing data pipelines, or optimizing resource allocation, the integration of AI assistants and Logic Apps opens doors to a more intelligent and responsive future. We encourage you to explore these tools further and discover how they can revolutionize your own automation journey. References: Getting started with Azure OpenAI Assistants (Preview) Call Azure Logic apps as functions using Azure OpenAI Assistants Azure OpenAI Assistants function calling Azure OpenAI Service models What is Azure Logic Apps? Azure Resource Manager – Rest OperationsAzure OpenAI Service - Features Overview and Key Concepts
Azure artificial intelligence services including a variety services related to language and language processing (speech recognition, speech formation, translations), text recognition, and image and character recognition. What is Azure OpenAI Service? Azure OpenAI Service provides REST API access to OpenAI's powerful language models including the GPT-3, Codex and Embeddings model series. Azure OpenAI Model Azure OpenAI provides access to many different models, grouped by family and capability. A model family typically associates models by their intended task. Azure OpenAI Service Model capabilities Each model family has a series of models that are further distinguished by capability. These capabilities are typically identified by names, and the alphabetical order of these names generally signifies the relative capability and cost of that model within a given model family. Azure OpenAI models fall into a few main families: GPT-4:A set of models that improve on GPT-3.5 and can understand as well as generate natural language and code. GPT-3.5:A set of models that improve on GPT-3 and can understand as well as generate natural language and code. Embeddings:A set of models that can convert text into numerical vector form to facilitate text similarity. DALL-E:A series of models that can generate original images from natural language. Key concepts: Prompts & Completion The completions endpoint is the core component of the API service. This API provides access to the model's text-in, text-out interface. Users simply need to provide an input prompt containing the English text command, and the model will generate a text completion. Token Azure OpenAI processes text by breaking it down into tokens. Tokens can be words or just chunks of characters.For example, the word “hamburger” gets broken up into the tokens“ham”, “bur” and “ger” The total number of tokens processed in a given request depends on the length of your input, output and request parameters. The quantity of tokens being processed will also affect your response latency and throughput for the models. Resource Azure OpenAI is a new product offering on Azure. You can get started with Azure OpenAI the same way as any other Azure product where you create a resource, or instance of the service, in your Azure Subscription. You can read more about Azure's resource management design. Deployment Once you create an Azure OpenAI Resource, you must deploy a model before you can start making API calls and generating text. This action can be done using the Deployment APIs. These APIs allow you to specify the model you wish to use. In-context learning The models used by Azure OpenAI use natural language instructions and examples provided during the generation call to identify the task being asked and skill required. When you use this approach, the first part of the prompt includes natural language instructions and/or examples of the specific task desired. The model then completes the task by predicting the most probable next piece of text. This technique is known as "in-context" learning. There are three main approaches for in-context learning: Few-shot: In this case, a user includes several examples in the call prompt that demonstrate the expected answer format and content. One-shot: This case is the same as the few-shot approach except only one example is provided. Zero-shot: In this case, no examples are provided to the model and only the task request is provided. Model The service provides users access to several different models. Each model provides a different capability and price point. GPT-4 models are the latest available models. Due to high demand access to this model series is currently only available by request. The GPT-3 base models are known as Davinci, Curie, Babbage, and Ada in decreasing order of capability and increasing order of speed. The Codex series of models is a descendant of GPT-3 and has been trained on both natural language and code to power natural language to code use cases. Use cases: GPT 3.5 Generating natural language for chatbots and virtual assistants with awareness of the previous history of chat Power chatbots that can handle customer inquiries, provide assistance, and converse but doesn’t have memory of conversations Automatically summarize lengthy texts Assist writers by suggesting synonyms, correcting grammar and spelling errors, and even generating entire sentences or paragraphs Help researchers by quickly processing large amounts of data and generating insights, summaries, and visualizations to aid in analysis Generate good quality code based on natural language Use cases: GPT 4.0 Generating and understanding natural language for customer service interactions, chatbots, and virtual assistants – doesn’t have memory of conversations Generating high-quality code for programming languages based on natural language input. Providing accurate translations between languages Improving text summarization and content generation Provides for multi-modal interaction (text and images) Substantial reduction in Hallucinations Consistency between different runs is high Multi-Modal Transformer Architecture Multi-modal models combine text and other types of input (such as graphics, images etc.) and are more task-specific. One multi-modal model in the collection has not been pre-trained in the same self-supervised manner. These models have performed state-of-the-art tasks, including visual question answering, image captioning, and speech recognition. Pricing Pricing will be based on the pay-as-you-go consumption model with a price per unit for each model, which is similar to other Azure Cognitive Services pricing models. Language models Image models Fine-tuned models Embedding models DALL-E Image Generation Editing an image Creating variations of image Embedding models The embedding is an information dense representation of the semantic meaning of a piece of text. Microsoft currently offers three families of Embeddings models for different functionalities: Similarity embedding:are good at capturing semantic similarity between two or more pieces of text. Text search embedding:help measure whether long documents are relevant to a short query. Code search embedding:are useful for embedding code snippets and embedding natural language search queries.
Andruino R2 recognize objects in an unstructured enviroment (ROS to Cognitive Service interface)
Andruino R2 robot uses cognitive services to recognize objects in an unstructured environment. Video: https://www.youtube.com/watch?v=Ra1HNqW0bA0 Andruino R2 is an open educational low-cost (components about 35€/USD) modular and extendable mobile robot based on Android and Arduino, integrated in the cloud, to be used as an educational tool in labs and classrooms of STEM, ICT vocational training or engineering courses, as well as in e-learning or MOOC courses as an alternative or, complementary, to virtual labs and soft robotics simulation. It is a first step introducing what we call “BYOR: Bring Your Own Robot” education policy equivalent to “BYOD: Bring your own devices”. Andruino it is powerful but easy to construct and extremely low cost, as it is based in the student's smartphone. Andruino R2 is compatible with ROS (Robot Operating System) and with iot clouds and try to bring deep learning and other advanced techniques to the STEM's and VET's classrooms. Author: @andruinos License: CC-BY-SA https://creativecommons.org/licenses/by-sa/2.0/Announcing new milestones for Microsoft Cognitive Services vision and search services in Azure
Artificial Intelligence (AI) has emerged as one of the most disruptive forces behind the digital transformation of business. At Microsoft, we believe everyone—developers, data scientists and enterprises—should have access to the benefits of AI toaugment human ingenuity in unique and differentiated ways. We’ve beenconducting research in AIfor more than two decades and infusing it into ourproducts and services. Now we’re bringing it to everyone through simple, yet powerful tools. One of those tools isMicrosoft Cognitive Services, a collection of cloud-hosted APIs that let developers easily add AI capabilities for vision, speech, language, knowledge and search into applications, across devices and platforms such as iOS, Android and Windows. To date more than a million developers have already discovered and tried our Cognitive Services, and many new customers are harnessing the power of AI, such as major auto insurance providerProgressivebest known for Flo, its iconic spokesperson. The company wanted to take advantage of customers’ increasing use of mobile channels to interact with its brand. Progressive used Microsoft Azure Bot Service and Cognitive Services to quickly and easily build theFlo Chatbot—currently available on Facebook Messenger—which answers customer questions, provides quotes and even offers a bit of witty banter in Flo’s well-known style. Today, we’re announcing new milestones forCognitive Services vision and search services in Azure. Read about it in the Azure blog.CS in eDiscovery
Very curious to hear about plans to integrate CS into content search and Adv. eDiscovery--especially for setting up query-based eDiscovery holds early in a case (but also within review sets later). When it comes, it's going to be a quantum leap in document recall compared to standard keyword queries.
Personal AI
How could Microsoft get it so wrong. I tried to use Azure and AI and its just matrix calculations. Open-AI is like a brute force training method, but at least its visualized. I just want a dog, a pet cat. An annoying robot that helps me with math and does my homework and that i can play games with. That will learn and grow with me and all you have is just a bunch of worthless databases like your CD collection, or vintage games. Bunch of comic book nerds with their libraries of books. Its like walking into a library and you don't know how to read. Here i was thinking that paperclip from word would have progressed into an annoying helper, but at least i have dancing girls hows **bleep** i can vibrate. Seriously can you not just make something simple, someone somewhere in that huge office must be someone who knows where the AI program is and can make a simple program and is like a "ferbe". Thats as helpful as the paperclip with a comprehensive help toolkit. Something that works like an operating system of something robotic i want to build. Could someone please help me make something simple, that can do basic reading comprehension so i can at least talk to it, and get it to read a book and answer questions. Maybe one that can learn maths and can 3d model things so i can visualize it. Or maybe something that can just learn by watching me and can recall something i need for reference. "i saw something on facebook" (pulls out phone to show person) "asks ai something vague" instantly pulls it up.
Searching pdf files using Natural Language
I'm new to all the Azure services and have been tasked with finding a solution to search a collection of pdf files using natural language queries. One use case would be to search resumés by saying (or entering) something like. "Find me all the candidates who have worked for XYZ company within the last 5 years". It looks like Azure Cognitive Search could be very useful for this. I've created a search service, and resource and data source etc and have successfully done a simple text search of my file. The resulting JSON looks like it did a good identifying all the various text concepts (names and places etc). I'm overwhelmed with all the various services that are available. If anyone can give a little newbie guidances as to which direction to go it would be greatly appreciated and would save me insane amounts of time running down rabbit holes looking for the correct services to use for my purposes. Thanks.
