Extract Data from PDFs using Form Recognizer with Code or Without!
Form Recognizer is a powerful tool to help build a variety of document machine learning solutions. It is one service however its made up of many prebuilt models that can perform a variety of essential document functions. You can even custom train a model using supervised or unsupervised learning for tasks outside of the scope of the prebuilt models! Read more about all the features of Form Recognizer here. In this example we will be looking at how to use one of the prebuilt models in the Form Recognizer service that can extract the data from a PDF document dataset. Our documents are invoices with common data fields so we are able to use the prebuilt model without having to build a customized model. Sample Invoice: After we take a look at how to do this with Python and Azure Form Recognizer, we will take a look at how to do the same process with no code using the Power Platform services: Power Automate and Form Recognizer built into AI Builder. In the Power Automate flow we are scheduling a process to happen every day. What the process does is look in the raw blob container to see if there is new files to be processed. If there is new files to be processed it gets all blobs from the container and loops through each blob to extract the PDF data using a prebuilt AI builder step. Then it deletes the processed document from the raw container. See what it looks like below. Power Automate Flow: Prerequisites for Python Azure Account Sign up here! Anaconda and/or VS Code Basic programming knowledge Prerequisites for Power Automate Power Automate Account Sign up here! No programming knowledge Process PDFs with Python and Azure Form Recognizer Service Create Services First lets create the Form Recognizer Cognitive Service. Go to portal.azure.com to create the resource or click this link. Now lets create a storage account to store the PDF dataset we will be using in containers. We want two containers, one for the processed PDFs and one for the raw unprocessed PDF. Create an Azure Storage Account Create two containers: processed , raw Upload data Upload your dataset to the Azure Storage raw folder since they need to be processed. Once processed then they would get moved to the processed container. The result should look something like this: Create Notebook and Install Packages Now that we have our data stored in Azure Blob Storage we can connect and process the PDF forms to extract the data using the Form Recognizer Python SDK. You can also use the Python SDK with local data if you are not using Azure Storage. This example will assume you are using Azure Storage. Create a new Jupyter notebook in VS Code. Install the Python SDK !pip install azure-ai-formrecognizer --pre Then we need to import the packages. import os from azure.core.exceptions import ResourceNotFoundError from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential import os, uuid from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__ Create FormRecognizerClient Update the endpoint and key with the values from the service you created. These values can be found in the Azure Portal under the Form Recongizer service you created under the Keys and Endpoint on the navigation menu. endpoint = "<your endpoint>" key = "<your key>" We then use the endpoint and key to connect to the service and create the FormRecongizerClient form_recognizer_client = FormRecognizerClient(endpoint, AzureKeyCredential(key)) Create the print_results helper function for use later to print out the results of each invoice. def print_result(invoices, blob_name): for idx, invoice in enumerate(invoices): print("--------Recognizing invoice {}--------".format(blob_name)) vendor_name = invoice.fields.get("VendorName") if vendor_name: print("Vendor Name: {} has confidence: {}".format(vendor_name.value, vendor_name.confidence)) vendor_address = invoice.fields.get("VendorAddress") if vendor_address: print("Vendor Address: {} has confidence: {}".format(vendor_address.value, vendor_address.confidence)) customer_name = invoice.fields.get("CustomerName") if customer_name: print("Customer Name: {} has confidence: {}".format(customer_name.value, customer_name.confidence)) customer_address = invoice.fields.get("CustomerAddress") if customer_address: print("Customer Address: {} has confidence: {}".format(customer_address.value, customer_address.confidence)) customer_address_recipient = invoice.fields.get("CustomerAddressRecipient") if customer_address_recipient: print("Customer Address Recipient: {} has confidence: {}".format(customer_address_recipient.value, customer_address_recipient.confidence)) invoice_id = invoice.fields.get("InvoiceId") if invoice_id: print("Invoice Id: {} has confidence: {}".format(invoice_id.value, invoice_id.confidence)) invoice_date = invoice.fields.get("InvoiceDate") if invoice_date: print("Invoice Date: {} has confidence: {}".format(invoice_date.value, invoice_date.confidence)) invoice_total = invoice.fields.get("InvoiceTotal") if invoice_total: print("Invoice Total: {} has confidence: {}".format(invoice_total.value, invoice_total.confidence)) due_date = invoice.fields.get("DueDate") if due_date: print("Due Date: {} has confidence: {}".format(due_date.value, due_date.confidence)) Connect to Blob Storage Now lets connect to our blob storage containers and create the BlobServiceClient. We will use the client to connect to the raw and processed containers that we created earlier. # Create the BlobServiceClient object which will be used to get the container_client connect_str = "<Get connection string from the Azure Portal>" blob_service_client = BlobServiceClient.from_connection_string(connect_str) # Container client for raw container. raw_container_client = blob_service_client.get_container_client("raw") # Container client for processed container processed_container_client = blob_service_client.get_container_client("processed") # Get base url for container. invoiceUrlBase = raw_container_client.primary_endpoint print(invoiceUrlBase) HINT: If you get a "HttpResponseError: (InvalidImageURL) Image URL is badly formatted." error make sure the proper permissions to access the container are set. Learn more about Azure Storage Permissions here Extract Data from PDFs We are ready to process the blobs now! Here we will call list_blobs to get a list of blobs in the raw container. Then we will loop through each blob, call the begin_recognize_invoices_from_url to extract the data from the PDF. Then we have our helper method to print the results. Once we have extracted the data from the PDF we will upload_blob to the processed folder and delete_blob from the raw folder. print("\nProcessing blobs...") blob_list = raw_container_client.list_blobs() for blob in blob_list: invoiceUrl = f'{invoiceUrlBase}/{blob.name}' print(invoiceUrl) poller = form_recognizer_client.begin_recognize_invoices_from_url(invoiceUrl) # Get results invoices = poller.result() # Print results print_result(invoices, blob.name) # Copy blob to processed processed_container_client.upload_blob(blob, blob.blob_type, overwrite=True) # Delete blob from raw now that its processed raw_container_client.delete_blob(blob) Each result should look similar to this for the above invoice example: The prebuilt invoices model worked great for our invoices so we don't need to train a customized Form Recognizer model to improve our results. But what if we did and what if we didn't know how to code?! You can still leverage all this awesomeness in AI Builder with Power Automate without writing any code. We will take a look at this same example in Power Automate next. Use Form Recognizer with AI Builder in Power Automate You can achieve these same results using no code with Form Recognizer in AI Builder with Power Automate. Lets take a look at how we can do that. Create a New Flow Log in to Power Automate Click Create then click Scheduled Cloud Flow . You can trigger Power Automate flows in a variety of ways so keep in mind that you may want to select a different trigger for your project. Give the Flow a name and select the schedule you would like the flow to run on. Connect to Blob Storage Click New Step List blobs Step Search for Azure Blob Storage and select List blobs Select the ellipsis click Create new connection if your storage account isn't already connected Fill in the Connection Name , Azure Storage Account name (the account you created), and the Azure Storage Account Access Key (which you can find in the resource keys in the Azure Portal) Then select Create Once the storage account is selected click the folder icon on the right of the list blobs options. You should see all the containers in the storage account, select raw . Your flow should look something like this: Loop Through Blobs to Extract the Data Click the plus sign to create a new step Click Control then Apply to each Select the textbox and a list of blob properties will appear. Select the value property Next select add action from within the Apply to each Flow step. Add the Get blob content step: Search for Azure Blob Storage and select Get blob content Click the textbox and select the Path property. This will get the File content that we will pass into the Form Recognizer. Add the Process and save information from invoices step: Click the plus sign and then add new action Search for Process and save information from invoices Select the textbox and then the property File Content from the Get blob content section Add the Copy Blob step: Repeat the add action steps Search for Azure Blob Storage and select Copy Blob Select the Source url text box and select the Path property Select the Destination blob path and put /processed for the processed container Select Overwrite? dropdown and select Yes if you want the copied blob to overwrite blobs with the existing name. Add the Delete Blob step: Repeat the add action steps Search for Azure Blob Storage and select Delete Blob Select the Blob text box and select the Path property The Apply to each block should look something like this: Save and Test the Flow Once you have completed creating the flow save and test it out using the built in test features that are part of Power Automate. This prebuilt model again worked great on our invoice data. However if you have a more complex dataset, use the AI Builder to label and create a customized machine learning model for your specific dataset. Read more about how to do that here. Conclusion We went over a fraction of the things that you can do with Form Recognizer so don't let the learning stop here! Check out the below highlights of new Form Recognizer features that were just announced and the additional doc links to dive deeper into what we did here. Additional Resources New Form Recognizer Features What is Form Recognizer? Quickstart: Use the Form Recognizer client library or REST API Tutorial: Create a form-processing app with AI Builder AI Developer Resources page AI Essentials video including Form Recognizer
Apps can now narrate what they see in the world as well as people do
How would you leverage technology capable of generating natural language image descriptions that are, in many cases, just as good or better than what a human could produce? What if that capability is just one cloud API call away? Would you create live scene captions for people who are blind or low vision to better understand the world around them, like Seeing AI?
Customer support: How Azure Metrics Advisor can help improve customer satisfaction
Anomaly detection is an instance where AI (ARTIFICIAL INTELLIGENCE) can help by detecting the anomaly, notifying both the business process and planning team, providing the root cause, and either automatically fixing the issue or enabling the support operations managers to make better-informed decisions.
