What is intelligent document processing?

Intelligent document processing (IDP) refers to a set of business solutions that use deep learning tools to automate document processing. With the help of RPA bots, AI and computer vision, IDP extracts unstructured data from documents (e.g., email text, PDF, and scanned documents) and converts it into structured data.

As defined by Deloitte, IDP “automates the processing of data contained in documents ― understanding what the document is about, what information it contains, extracting that information, and sending it to the right place.”

IDP differs from optical character recognition (OCR) and Intelligent Character Recognition (ICR), legacy softwares that can turn a scanned image into text (i.e. check scanning at the bank), as it not only captures data from documents, but also extracts, categorizes and exports relevant data for further processing using AI technologies.

IDP solutions tend to be “non-invasive” and easily integratable into existing systems, business applications and platforms. They also run the gamut from pre-built, out-of-the-box solutions to more complex, bespoke implementations. 

Potential use cases include:

• Invoice Processing
• Digital Document Archiving
• Insurance Claims Processing
• Fraud Detection
• Case Reviews
• Contract Administration
• Mortgage Loan Application Processing
• Customer Onboarding

Intelligent Data Processing Stages & Components 

*Image sourced from https://www.automationanywhere.com/company/blog/rpa-thought-leadership/take-a-platform-approach-to-idp 

Pre-Processing

As documents arrive to be processed in various conditions, IDP applies techniques such as noise reduction, binarization and de-skewing to maximize the quality of documents. 

Image Processing 

IDP first uses computer vision to understand document structure and identify “features” such as text, graphs and pictures. Older technologies such as OCR and ICR can then be leveraged to extract text from the document. During this process, some IDP solutions essentially create a digitized version of the document, or “digital twin,” that is primed for machine reading.

Classification & Data Extraction

Natural Language Processing (NLP).  NLP is a subset of Artificial Intelligence (AI) that enables machines to understand, contextualize and analyze human language. When it comes to IDP, after OCR extracts text from the document, NLP is used to make sense of it. 

Using machine learning (ML) and AI-based techniques such as NLP, IDP automatically identifies, separates and classifies document components. For example, a loan application might include a person’s pay stubs, a scan of their driver’s license, tax forms, bank statements, etc. An IDP’s classification engine is responsible for parsing out all of these different components, accurately categorizing them and routing them to their next destination.

One of the key deliverables of IDP systems is their ability to pinpoint important information and extract it for further analysis or processing. To accomplish this, IDPs often include a library of pre-trained extraction models or a pattern matching tools such as Regular Expressions (RegEx).

Data Validation

To ensure data accuracy and integrity, IDP platforms leverage external databases and pre-configured lexicons to validate the data extracted from documents. Not only does this process ensure the data quality, but that data is collected in the right format and prepped for immediate usage.  

The data validation process typically leverages a HITL (Human-in-the-Loop) machine learning framework, whereby problematic data is routed to humans to review and correct. This approach enables the validation model to continuously learn and improve its accuracy over time.

Integration

The final step of the IDP process is to integrate the validated data into large enterprise systems and workflows. 

Benefits of Intelligent Document Processing?

“Intelligent Document Processing (IDP), in particular, will emerge as a major tool for businesses to successfully navigate a completely remote workforce. Every organization will need to be able to process structured and unstructured data autonomously to work efficiently. IDP allows bots to process emails, signatures, and PDFs—enabling document-intensive processes such as insurance claims, loan applications, and invoices to be automated.”

-Jon Knisley, the Principal of Automation and Process Excellence at FortressIQ 

Increased effectiveness and efficiency

For far too long, organizations have been beholden to highly expensive, slow and error-prone manual document processing processes. To put things in perspective:

• Standard invoice processing has an average error rate of 10%
• On average, simple manual document processing costs around $6-8 per document. For more complex documents, average cost per document can be upwards of $40-50
• The average office worker uses about 10,000 sheets, or two full cases, of paper,per year. 
• More than 70% of businesses would fail within 3 weeks if they suffered a catastrophic loss of paper-based records due to fire or flood

At large document-heavy companies, such as banks and insurance agencies, the costs and risks of manual document processing can add up fast. However, by embracing IDP, organizations can dramatically increase the speed at which documents are processes as well as the effectiveness. 

In fact, according to industry leaders such as UiPath, IDP can:

• Reduce the risk of errors by 52% or more
• Reduce expense of manual document processing by 35% 
• Reduce time spent on document-related tasks by 17%
• Reduce document processing times by 50-70%

Case in point, Community Brands, a leading-edge technology provider for private schools and NGOs, partnered with Scalehub to automate the financial aid application process using IDP. As a result, they were able to:

• Reduce operating costs by 30% YOY
• Reduce document verification time down by 85%
• Reduce the entire financial aid application process down from 6 weeks to just a couple of days
• Achieve a 99% accuracy rate

IDP is also considered a potent solution for exception handling. Take invoicing. As roughly 20-30% of invoices include errors or some sort of other problem, it’s not surprising that 48% of businesses consider handling exceptions as the top accounts payable problem. IDP systems can be “trained” to flag errors or fraudulent information. In a matter of seconds, the issue can be identified and re-routed for human intervention.

For example, at Community Brands, using manual document processing it took an average of 10-12 days to simply verify whether or not the correct documentation was uploaded. With IDP, it now takes a matter of seconds. 

Improved Compliance & Security

IDP’s impressive accuracy rate makes it the ideal solution for handling any compliance-related document or those that include sensitive information such as personally identifiable information (PII) or health records. As IDP eliminates the need for humans to open up, review or handle any of the data included documents, it minimizes the risk of exposing sensitive information to outside parties. In addition, IDP can help streamline and maximize the accuracy of regulatory reporting. 

Enhanced Data Quality & Usability

On average, 80% of an organization’s data is “dark data” — meaning it’s locked in emails, text, PDFs, and scanned documents. However, using RPA and AI-based tools, IDP unlocks the value of dark data by transforming into high quality, structured data that is primed for analysis.

As the experts at Mckinsey explain, “by combining the data derived from paper documents with the wealth of digital data already available, a comprehensive data landscape can be established, significantly enhancing data evaluation and analytics possibilities.”

IDP Promotes & Helps Scale Automation

Along with workflow management tools, IDP is a powerful enabler of end-to-end process automation. As IDP can be tied to any platform, it helps link together all of the various systems that go into automating complex business processes and achieving hyperautomation. 

Furthermore, cognitive technologies such as RPA and AI need structured high quality data to “learn” from and operate. By transforming unstructured data found in documents into streams of cleaned, structured data, IDP optimizes data for RPA/AI consumption as well.

IDP Solution Provider Snapshot

Hyperscience -  an input-to-outcome platform for the automation of document-based workflows. Hyperscience embraces a human-centric approach to product design, creating solutions that not only provide stellar UX, but aim to enhance human behavior and decision making rather than replace it. Customers include TD Ameritrade, ONE Insurance and Fidelity Investments. 

Automation Anywhere - uses AI technologies such as natural language processing (NLP), Computer Vision, deep learning and machine learning (ML) to classify, categorize, and extract relevant information, and validate the extracted data.

Amazon Textract - fully managed machine learning service that automatically extracts handwriting, printed text, and data from scanned documents. With Textract you can quickly automate manual document activities, enabling you to process millions of document pages in hours.

Datamatics TruCap+ - performs intelligent data capture from various unstructured documents with over 99% accuracy. Also includes an AI layer that helps in document classification, data extraction, formatting of information, and input of the information into the downstream system.

Appian - a pre-built solution that converts unstructured data locked in documents to structured data. To do this, they use a combination of cutting-edge machine learning and AI capabilities augmented by humans when needed. Can be set up in minutes and seamlessly integrated into existing business processes. Clients include Cigna, Ryder and the GSA.

Infrrd - uses advanced AI technologies to extract data trapped in complex, unstructured documents as well as images, tables, graphs, and more. Clients include Adobe, Allstate and Kohl’s. 

Have 2 minutes? Take our survey and let us know, how is your organization benefitting (or not) from IDP.

Create your own user feedback survey