Form Recognizer is a powerful tool to help build a variety of document machine learning solutions. It is one service however its made up of many prebuilt models that can perform a variety of essential document functions. You can even custom train a model using supervised or unsupervised learning for tasks outside of the scope of the prebuilt models! Read more about all the features of Form Recognizer here. In this example we will be looking at how to use one of the prebuilt models in the Form Recognizer service that can extract the data from a PDF document dataset. Our documents are invoices with common data fields so we are able to use the prebuilt model without having to build a customized model.
Sample Invoice:
After we take a look at how to do this with Python and Azure Form Recognizer, we will take a look at how to do the same process with no code using the Power Platform services: Power Automate and Form Recognizer built into AI Builder. In the Power Automate flow we are scheduling a process to happen every day. What the process does is look in the raw
blob container to see if there is new files to be processed. If there is new files to be processed it gets all blobs from the container and loops through each blob to extract the PDF data using a prebuilt AI builder step. Then it deletes the processed document from the raw
container. See what it looks like below.
Power Automate Flow:
Prerequisites for Python
- Azure Account Sign up here!
- Anaconda and/or VS Code
- Basic programming knowledge
Prerequisites for Power Automate
- Power Automate Account Sign up here!
- No programming knowledge
Process PDFs with Python and Azure Form Recognizer Service
Create Services
First lets create the Form Recognizer Cognitive Service.
- Go to portal.azure.com to create the resource or click this link.
Now lets create a storage account to store the PDF dataset we will be using in containers. We want two containers, one for the processed
PDFs and one for the raw
unprocessed PDF.
- Create an Azure Storage Account
- Create two containers:
processed
,raw
Upload data
Upload your dataset to the Azure Storage raw
folder since they need to be processed. Once processed then they would get moved to the processed
container.
The result should look something like this:
Create Notebook and Install Packages
Now that we have our data stored in Azure Blob Storage we can connect and process the PDF forms to extract the data using the Form Recognizer Python SDK. You can also use the Python SDK with local data if you are not using Azure Storage. This example will assume you are using Azure Storage.
-
Create a new Jupyter notebook in VS Code.
-
Install the Python SDK
!pip install azure-ai-formrecognizer --pre
- Then we need to import the packages.
import os
from azure.core.exceptions import ResourceNotFoundError
from azure.ai.formrecognizer import FormRecognizerClient
from azure.core.credentials import AzureKeyCredential
import os, uuid
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__
Create FormRecognizerClient
- Update the
endpoint
andkey
with the values from the service you created. These values can be found in the Azure Portal under the Form Recongizer service you created under theKeys and Endpoint
on the navigation menu.
endpoint = "<your endpoint>"
key = "<your key>"
- We then use the
endpoint
andkey
to connect to the service and create the FormRecongizerClient
form_recognizer_client = FormRecognizerClient(endpoint, AzureKeyCredential(key))
- Create the
print_results
helper function for use later to print out the results of each invoice.
def print_result(invoices, blob_name):
for idx, invoice in enumerate(invoices):
print("--------Recognizing invoice {}--------".format(blob_name))
vendor_name = invoice.fields.get("VendorName")
if vendor_name:
print("Vendor Name: {} has confidence: {}".format(vendor_name.value, vendor_name.confidence))
vendor_address = invoice.fields.get("VendorAddress")
if vendor_address:
print("Vendor Address: {} has confidence: {}".format(vendor_address.value, vendor_address.confidence))
customer_name = invoice.fields.get("CustomerName")
if customer_name:
print("Customer Name: {} has confidence: {}".format(customer_name.value, customer_name.confidence))
customer_address = invoice.fields.get("CustomerAddress")
if customer_address:
print("Customer Address: {} has confidence: {}".format(customer_address.value, customer_address.confidence))
customer_address_recipient = invoice.fields.get("CustomerAddressRecipient")
if customer_address_recipient:
print("Customer Address Recipient: {} has confidence: {}".format(customer_address_recipient.value, customer_address_recipient.confidence))
invoice_id = invoice.fields.get("InvoiceId")
if invoice_id:
print("Invoice Id: {} has confidence: {}".format(invoice_id.value, invoice_id.confidence))
invoice_date = invoice.fields.get("InvoiceDate")
if invoice_date:
print("Invoice Date: {} has confidence: {}".format(invoice_date.value, invoice_date.confidence))
invoice_total = invoice.fields.get("InvoiceTotal")
if invoice_total:
print("Invoice Total: {} has confidence: {}".format(invoice_total.value, invoice_total.confidence))
due_date = invoice.fields.get("DueDate")
if due_date:
print("Due Date: {} has confidence: {}".format(due_date.value, due_date.confidence))
Connect to Blob Storage
- Now lets connect to our blob storage containers and create the BlobServiceClient. We will use the client to connect to the
raw
andprocessed
containers that we created earlier.
# Create the BlobServiceClient object which will be used to get the container_client
connect_str = "<Get connection string from the Azure Portal>"
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
# Container client for raw container.
raw_container_client = blob_service_client.get_container_client("raw")
# Container client for processed container
processed_container_client = blob_service_client.get_container_client("processed")
# Get base url for container.
invoiceUrlBase = raw_container_client.primary_endpoint
print(invoiceUrlBase)
HINT: If you get a "HttpResponseError: (InvalidImageURL) Image URL is badly formatted." error make sure the proper permissions to access the container are set. Learn more about Azure Storage Permissions here
Extract Data from PDFs
We are ready to process the blobs now! Here we will call list_blobs
to get a list of blobs in the raw
container. Then we will loop through each blob, call the begin_recognize_invoices_from_url
to extract the data from the PDF. Then we have our helper method to print the results. Once we have extracted the data from the PDF we will upload_blob
to the processed
folder and delete_blob
from the raw folder.
print("\nProcessing blobs...")
blob_list = raw_container_client.list_blobs()
for blob in blob_list:
invoiceUrl = f'{invoiceUrlBase}/{blob.name}'
print(invoiceUrl)
poller = form_recognizer_client.begin_recognize_invoices_from_url(invoiceUrl)
# Get results
invoices = poller.result()
# Print results
print_result(invoices, blob.name)
# Copy blob to processed
processed_container_client.upload_blob(blob, blob.blob_type, overwrite=True)
# Delete blob from raw now that its processed
raw_container_client.delete_blob(blob)
Each result should look similar to this for the above invoice example:
The prebuilt invoices model worked great for our invoices so we don't need to train a customized Form Recognizer model to improve our results. But what if we did and what if we didn't know how to code?! You can still leverage all this awesomeness in AI Builder with Power Automate without writing any code. We will take a look at this same example in Power Automate next.
Use Form Recognizer with AI Builder in Power Automate
You can achieve these same results using no code with Form Recognizer in AI Builder with Power Automate. Lets take a look at how we can do that.
Create a New Flow
- Log in to Power Automate
- Click
Create
then clickScheduled Cloud Flow
. You can trigger Power Automate flows in a variety of ways so keep in mind that you may want to select a different trigger for your project. - Give the Flow a name and select the schedule you would like the flow to run on.
Connect to Blob Storage
- Click
New Step
List blobs
Step- Search for
Azure Blob Storage
and selectList blobs
- Select the ellipsis click
Create new connection
if your storage account isn't already connected- Fill in the
Connection Name
,Azure Storage Account name
(the account you created), and theAzure Storage Account Access Key
(which you can find in the resource keys in the Azure Portal) - Then select
Create
- Fill in the
- Once the storage account is selected click the folder icon on the right of the list blobs options. You should see all the containers in the storage account, select
raw
.
- Search for
Your flow should look something like this:
Loop Through Blobs to Extract the Data
- Click the plus sign to create a new step
- Click
Control
thenApply to each
- Select the textbox and a list of blob properties will appear. Select the
value
property - Next select
add action
from within theApply to each
Flow step. - Add the
Get blob content
step:- Search for
Azure Blob Storage
and selectGet blob content
- Click the textbox and select the
Path
property. This will get theFile content
that we will pass into the Form Recognizer.
- Search for
- Add the
Process and save information from invoices
step:- Click the plus sign and then
add new action
- Search for
Process and save information from invoices
- Select the textbox and then the property
File Content
from theGet blob content
section
- Click the plus sign and then
- Add the
Copy Blob
step:- Repeat the add action steps
- Search for
Azure Blob Storage
and selectCopy Blob
- Select the
Source url
text box and select thePath
property - Select the
Destination blob path
and put/processed
for the processed container - Select
Overwrite?
dropdown and selectYes
if you want the copied blob to overwrite blobs with the existing name.
- Add the
Delete Blob
step:- Repeat the add action steps
- Search for
Azure Blob Storage
and selectDelete Blob
- Select the
Blob
text box and select thePath
property
The Apply to each
block should look something like this:
- Save and Test the Flow
- Once you have completed creating the flow save and test it out using the built in test features that are part of Power Automate.
This prebuilt model again worked great on our invoice data. However if you have a more complex dataset, use the AI Builder to label and create a customized machine learning model for your specific dataset. Read more about how to do that here.
Conclusion
We went over a fraction of the things that you can do with Form Recognizer so don't let the learning stop here! Check out the below highlights of new Form Recognizer features that were just announced and the additional doc links to dive deeper into what we did here.
Additional Resources
Quickstart: Use the Form Recognizer client library or REST API