Automating tax documents processing with Azure Form Recognizer

Tax documents such as W2, 1099, 1040, W9 are part of several scenarios and processes such as underwriting, business loans, mortgage loans, tax calculations and more. Extracting information from these documents is usually tedious, time consuming, and prone to errors. The cost and burden of extracting data manually and building out a team to do it is a huge effort with high costs.

That is where Azure Form Recognizer can assist. Using Form Recognizer pre-built W2 model and its ability to easily train additional custom models for your specific documents makes it possible and easy to automate data extraction from tax documents enabling you to develop an application that extracts data from these documents in scale and within seconds. No longer will you have to manually input data from tax documents; Azure From Recognizer can assist and do it for you.

Challenges in extracting information from tax documents

Extracting information from Tax documents has several challenges:

1. Document variations – although tax documents are usually structured documents, each type has several variations and changes over the years.   
   
    
2. Complex key value pairs – Tax documents include complex key value pair extraction for example W-2 includes section #12 with a variety of key value pairs and matching codes, and 1040 has complicated layouts.  
   
3. Handwritten – Tax forms can be handwritten and scanned causing the handwritten text to be unclear or not easily recognizable. A high-quality OCR is needed to extract handwritten numbers and characters.
4. Variety of file formats – tax documents that are received for extraction have a variety of formats – images (jpg, png, tiff), digital PDFs and scanned PDFs. Support for all formats is needed to extract the information.
5. Compliance and PII (personally identifiable information) – Tax documents include lots of Personal Identifiable Information (PII) such as social security numbers, names, addresses and more, which needs to be handled with care and by a compliant and secure service only.

Azure Form Recognizer overcomes these challenges and enables you to extract information from tax documents easily, securely and with high accuracy.

H&R Block uses Azure AI to transform tax returns - Customer Story 

A new job. A change in relationship status. A move across the country. H&R Block understands that tax returns are personal—a diary of your year’s most impactful events. When the organization’s 21 million clients rely on the tax professionals at H&R Block, providing individualized support for each client’s unique needs is paramount. And whether clients come in with a shoebox full of tax forms or file online, H&R Block is using Microsoft technology to transform the tax filing experience. In 2021, the 60,000 tax professionals at H&R Block prepared more than 21.6 million tax returns. That’s 21.6 million individuals with varying needs, concerns, complex circumstances—and each with a shoebox full of tax forms.  With AI-powered services like Azure Form Recognizer and Azure Cognitive Search, H&R Block tax professionals can spend more time building meaningful, personalized client experiences—and helping each client get the most out of their tax return.

See how H&R Block is using Azure Form Recognizer at https://www.aka.ms/hrblockignite2022.

Extracting information from tax documents with Azure Form Recognizer

Azure Form Recognizer enables you to extract information from tax documents with a W-2 prebuilt model and for other documents like W-9, 1040, 1099 etc. with a custom model. You can extract information from a W-2 model with 3 simple steps:

For other types of documents, you can train a Form Recognizer custom model to extract the fields and information you need with 4 simple steps:

Why use Form Recognizer for tax document automation processing?

Automating extraction of information from documents with Azure Form Recognizer has the following benefits:

1. Simple and Easy - Easily pull data and organize information from your Tax documents with the W2 prebuilt model and the custom models features.
2. Privacy and Security - Rely on enterprise-grade security and privacy applied to both your data and any trained models.
3. Fast – Process tax documents within seconds

Form Recognizer pre-built W2 model

The Form W-2, Wage and Tax Statement, is a US Internal Revenue Service (IRS) tax form. It is used to report employees' salary, wages, compensation, and taxes withheld. Employers send a W-2 form to each employee on or before January 31 each year and employees use the form to prepare their tax returns. W-2 is a key document used in employee's federal and state taxes filing, as well as other processes like mortgage loan and Social Security Administration (SSA).

A W-2 is a multipart form divided into state and federal sections and consisting of more than 14 boxes that detail an employee's income from the previous year. The Form Recognizer W-2 model combines Optical Character Recognition (OCR) with deep learning models to analyze and extract information reported in each box on a W-2 form. The model supports standard and customized forms from 2018 to the present. Both single and multiple forms are also supported. See detailed field extraction here.

Learn more about Form Recognizer W2 prebuilt model here https://docs.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-w2. 
Get started here – https://formrecognizer.appliedai.azure.com/studio/prebuilt?formType=tax.us.w2

Form Recognizer custom model

Form Recognizer uses state-of-the-art machine learning technology to detect and extract information from forms and documents and returns the extracted data in a structured JSON output. Custom models extract and analyze distinct data and use cases from forms and documents specific to your business. Standalone custom models can be combined to create composed models. To create a custom model for your tax documents all you need to do is label a dataset of documents with the values you want extracted and train the model on the labeled dataset. You only need five examples of the same form or document type to get started.

Learn more about Form Recognizer custom models here - https://docs.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-custom

Get started here - https://formrecognizer.appliedai.azure.com/studio/custommodel/projects

As tax season is approaching now is the time to start automating your tax documents workflows.

Get started 

• Try out the Form Recognizer Studio
• Get started with the new SDK or  REST API
• Learn more on prebuilt W2 model and Custom models

Additional Resources

1. Watch session topics featured in this blog
2. Get started with Microsoft Learn to build skills
3. Explore Azure announcements at Microsoft Ignite

Form Recognizer continues to improve AI quality and service performance. If you have any questions or feedback on either the preview APIs or the service, please contact us via email.