Video Transcript:
- Today we’ll look at the latest updates for building intelligent document processing apps using Azure AI services to extract text automatically from forms, structured or even unstructured documents, and text-based images at scale. From using Azure’s Form Recognizer Studio to leverage both pre-trained models and even bring and build your own to leveraging its raw data and code outputs to help speed up your development of custom apps. And joining us today from the Azure AI engineering team is Bema Bonsu. Welcome to Mechanics.
- Happy to be here.
- And thanks so much for joining us today, Bema. You know, document processing is one of those areas that’s always been rife for AI. It removes the manual effort needed to understand and classify documents as part of everyday information management or more custom solutions in areas like contract management or tax processing. So how are we moving this area forward?
- You are right. This is definitely a prime area for AI. If you think about the different structured and unstructured formats, content types, different spoken languages, handwritten inputs, and custom data fields that exist across different domains, it would be impossible to be able to handle this without leveraging machine learning and AI. A big focus for us has been on the precise application of AI to continually improve document understanding capabilities that you can call programmatically from Azure AI services and integrate with your app experiences regardless of where you are in the world.
- To that point, by the way, Azure AI services are in fact foundational to the intelligent document processing capabilities that are in a number of Microsoft services today, whether that’s information management in Microsoft Syntex where business users can point to whole document libraries and Syntex will just scan, analyze, and process items at scale, or you can use low-code document processing with AI Builder in the Power Platform as part of apps and workflows. And in Azure, all of this comes together with the Form Recognizer service for custom apps. So can you walk us through the core experiences and some of the recent updates?
- Sure, so for custom apps, there are a number of key updates that make document understanding better. I’m in Azure Form Recognizer Studio. I’m going to point out a few here. First, you’ll see we’ve added more document analysis capabilities, allowing you to extract text, tables, and key-value pairs. We’ve also added updates to aid with navigation, search, and table labeling. Beyond that, we have also extended our list of pre-built models for tax documents and business cards. There are now more models as well as regional-specific content like invoices, receipts, IDs, and health insurance cards. We are also offering a new pre-built contract model that you can use to extract signatory information. And something that continually gets better, our custom extraction models, lets you extract key-value pairs, selection marks, tables, signature fields, selected regions, and just about any other text or text-based image patterns you define.
- And these models are really great, by the way, for understanding both structured and unstructured documents like financial reports, insurance claims, proposals, mortgage documents, and more.
- They are. And it’s worth pointing out that as a global service, we support more than 300 spoken languages. I’m also really excited about the additional work we are doing for document extraction and search with the integration of large language models using Azure OpenAI. For example, using the General Document model, you can use natural language to describe the values within documents that you might be looking for, and it will identify the content even if your documents contain slight variations. Also, in the area of document classification, the service will now identify document types within the same file and split those out as separate components for analysis.
- And the improved reasoning, by the way, over unstructured content is something that I know a lot of people watching are going to be excited about. So I’d love to see how this works.
- Of course, and I’ve got a very timely example for you. For anyone living in the US dealing with tax season, you’ve probably heard of the tax preparation company H&R Block. They prepare more than 20 million tax returns each year. It’s not uncommon for many of their customers to literally show up with piles of printed paper tax forms and receipts. No one wants to manually enter those items. For the last couple of years, they’ve turned to Azure AI services to automate many of their processes. Form Recognizer can work across tax forms to extract data and help automate that process. In the US, we have common tax forms like W2s, 1099s, 1040s, and W-9s that we use to file taxes. Form Recognizer has a pre-built model for W2s and you can easily train it to handle the other forms, so we’ll start there. In Form Recognizer Studio, we have sample W2 forms preloaded, as you can see here on the left. The first one is an image scan from a paper form, which you can see from the scanned text. And the second one is a lot clearer, like a screenshot of a digital form. I’ll use the first one because the text and the markings are a little more difficult to read. Once I hit analyze, you’ll see all of the scanned fields on the right, along with percentage confidence levels. If I go into the result tab, it shows the output of the analyze step as a raw JSON file. I can use this JSON file to display the information in a custom app or enter the captured fields into a table or database. And in the code tab, I can see Python code here by default, which I can use in a custom app as a starting point. And I can also view it as Javascript as well as C# depending on my needs.
- And this also works for handwritten text and, like you said, across many different languages.
- Absolutely. Receipts for expenses often have handwritten elements and that’s where you can leverage the models pre-built in Form Recognizer Studio. Here, you can see a few common receipt types for retail goods, hotel accommodation, food service with handwritten tips, receipts in other languages like French, Spanish, and Japanese, and these span different currencies and use different date formats. I’ll start with the receipt that has a handwritten tip, and you’ll see that even though the handwriting wasn’t super easy to read, the service is successfully able to pull the data correctly throughout with high confidence. Now, let’s look at the Japanese receipt next. I’ll analyze it and you’ll see it’s extracted with address elements, the date using year/month/day format, and the transaction amounts. If I expand the items, you’ll see that each line item was correctly scanned and outputted as correct Japanese characters, even though a few of the lines in the characters are thin and hard to read. With these pre-built models and samples, it’s super easy to get started and you can even upload your own examples to verify that they’re extracted well.
- Right, and this worked really well even for double-byte characters that were scanned on printed paper. Now, you mentioned there were forms like Form 1099 and also 1040 and W-9 and, you know, those aren’t built into the service. So how difficult would it be to add one of those forms into the service?
- It’s actually really easy. The best thing to do is to start with a couple of forms you’re trying to scan. I’ve pre-scanned some W-9s in advance. So let me show you that experience. Back in the Form Recognizer Studio, I’ll choose to create a custom model this time. Here’s where you’ll return to your custom projects list if you have a few already. I’ll create a new project, give it a name W-9, choose my subscription, resource group, and resource along with a few more fields for storage. The storage account is used to store the training data set that you create. From there, I can start building and training new models. In the custom model, you’ll see that there are tabs for Label Data, Models, Test, and Settings. The documents you see here are my scanned W-9 forms. You’ll need at least five examples to get started. I’ll select one of my documents and start by selecting the plus sign on the right and adding a field. These can be text, key-value pairs, or selection marks. That can be used, for example, with the tax classification section and the checkmarks inside. You can also define signatures or tables. I’ll start by choosing a text field and just create one for name. And from here, you’ll repeat this process for other text, signature, or table fields, and once you’ve added all of the fields you need, you can start to identify the parts in the document that correspond to your fields.
- So to save a little time, you’ll see that I have a dozen or so fields and a signature defined. Now I can match each of the fields in the document with my labels. I just need to click on the word in the document and assign it to the corresponding field from the list. You’ll need to repeat this step for every field in the document for each of the five documents in this case. Then once you’re finished, you can train your model. I’ll hit train, then I’ll give my model a name W-9model. I’ll skip the description, choose the model type, in this case, Neural, and this process will take several minutes. And when it’s done, you can test it out. So from the test tab, I’ll upload another two sample forms, and just like we did before with the pre-built models, I’ll just need to hit analyze. When it’s finished, you can see that it has successfully detected all of our fields with our defined labels. And just like I showed before, I can get to the JSON results and the code to add this to a custom app or workflow. As you can see, it took just a few minutes to build a custom neural model that’ll extract the data automatically from any other scanned W-9 forms I may have.
- Right, and because, by nature, taxes are pretty seasonal, this is also a great example of how you take advantage of the elasticity of the cloud to really scale out when you need to. And you also mentioned that it’s possible then to extract data from unstructured documents, right?
- That is right. Contract processing has been available for a few months as a pre-built neural document model. This extracts the important elements of a contract as key-value pairs to speed up many of the workflows around contract management. Let me show you how Contracts 365, a leading provider of contract management software, are leveraging our AI services to do this. So this is the Contracts 365 experience for requesting new counterparties, contracts, and documents. I’ll start with requesting a new contract. I can select the request type to request a contract from a template, file an executed contract, or review a third-party contract. Request Priority lets me set system conditions for how quickly a contract needs to be processed. These conditions can force the user to supply additional information per company policy. I’ll review a new third-party contract with an urgent one-to-two-day priority The AI processing will extract the contract record type, in this case, a clinical trial agreement, and I’ll upload the contract as a PDF, and that will initiate AI processing. Here, Contracts 365 uses Microsoft AI to extract contract data and then transforms, validates, and visualizes the data in the system with its AI engine. You can see that it has extracted the contract value and other important metadata about the clinical trial and its status and placed the data in the corresponding fields. Contracts 365 lets you then validate what has been inferred with highlighted citations in the PDF file. And here you can see it also finds counterparty data automatically and accurately extracts all the relevant title and address fields and populates the data in the backend system. So as you can see, the contract was completely unstructured and this is a huge benefit of leveraging a large language model. These models can easily process the various structures and terms people might use in a document like this.
- So how would I use then as a developer maybe some of these experiences with Form Recognizer and some of my custom apps?
- Let me show you a custom app we’ve built, what it does, and the code behind it. Here, I’ve built a simple app for processing invoices. I can see some basic stats about the number of documents processed, and below that are some details from a few processed invoices. Now, let me show you an invoice to get an idea of what we’re processing. Pay attention to the fields, like the invoice number and the amounts here from this invoice. Now I’ll head back over to my app and I’ll upload the PDF file we just saw. I’ll first select the file, then I’ll choose the model. In this case, it’s a pre-built invoice model, but we have quite a few additional model options to choose from. I’ll go ahead and upload it. And while this is uploading and getting processed, let me show you where this data is going. Here, we’re using Cosmos DB. If you remember the JSON output I showed earlier in Form Recognizer Studio, this is essentially what we’re adding to Cosmos DB. I have the Data Explorer already open, so I’ll refresh my items list, and this is the file I just uploaded and processed. And there’s our JSON output, again, just like we saw before, with all the key-value pairs and fields we’ve extracted. Now, if I go back to the app, I can see a clean, formatted view of the data. And here it is in our dashboard view from before, right in the top row. And even though this is a pretty sophisticated app in terms of its document processing capabilities, the nice thing from a developer standpoint is just how easy it is to code and call the web services needed.
- Let me show you what’s behind it using Visual Studio Code. Notice this is a Python app and we’re using the Python SDK. These are the environment variables we’ve defined for Azure App Service. Here you can see we’re creating the clients we need. This is so we can send our data to blob storage and the results to the Cosmos DB. This is the code that handles the upload and stores the file in Azure Storage. Here is where the Form Recognizer client sends the document to the service for analysis. And this is where we’ve instructed it to save the outputted JSON results in Cosmos DB. Once everything is finished, we return the message of completion to the user. So in just a few lines of code, we are able to build a fully functioning solution.
- And I can see this being a massive time saver, you know, for processing documents at scale. Now, for anyone that’s watching right now looking to get started, what do you recommend?
- The best way is to try Azure AI services out. And for that, you’ll need the Azure Form Recognizer Studio. If you have an Azure account set up, you can get to it by going to aka.ms/formrecognizerstudio. Then after that, to integrate Form Recognizer into a custom application or workflow, check out our guided quickstart with all the steps documented in your programming language of choice, whether that’s C#, Java, Python, or our REST API.
- Thanks so much for joining us today, Bema. And, of course, keep checking back to Microsoft Mechanics for all the latest updates. Thanks for watching and we’ll see you next time.
