Unstructured data processing trends and Its future
Exlporing unstructured data processing trends and its future
Add bookmark
What is Unstructured Data?
Experts estimate that the average person generates more than 1.7 MB of digital data per second, amounting to over 2.5 quintillion bytes per day. However, as the world becomes increasingly digitized and networked, experts predict that, on average, people will produce 463 exabytes of data per day by 2025.
While a small percentage of the data produced every day is structured data – digital information that adheres to a predefined data model or schema – the vast majority (80- 90 percent) is what’s known as unstructured data, data that lacks metadata or any sort of identifiable structure. As such, this “dark data” is unreadable by machines.
| Points of Comparison | Structured Data | Unstructured Data |
|
|
|
|
|
|
|
|
|
Hidden within this sea of unmanaged and untapped data lies incredible transformational value. While structured data can show you what is happening, it’s unstructured data that will tell you why. Furthermore, unstructured data is the lifeblood of predictive, end-to-end automation and commercialized artificial intelligence (AI).
The key for global corporate enterprise is to benefit from the collective intelligence presented by RPA and cognitive technologies along with human workers. Only by having technology combine with human talent can global corporate enterprise achieve scalable intelligent automation. And only with scalable intelligent automation enterprise resiliency be realized. Join that community for lessons learned at SRIA Live.Scalable RPA & Intelligent Automation Live
As recently as 2019, a mere 18% of enterprises were able to make use of unstructured data. However, with the emergence of AI-enabled Unstructured Data Processing (UDP) technology organizations of all types and sizes can now transform this “dark” data into powerful strategic assets.
Using a wide variety of machine learning techniques such as Natural Language Processing (NLP), video analytics, computer vision, speech and voice recognition technology, UDP solutions acquire, analyse, and act on unstructured data much like a human would. This not only allows organizations to automate content ingestion, but also access a treasure trove of decision intelligence.
Examples of UDP use cases include:
- A consumer goods company might use UDP to track conversations and mentions related to their brand on social media platforms. Once gathered, this information can be mined for insights related to customer engagement, marketing and new product development.
- Cities are increasingly using UDP tools such as computer vision and video analytics to not only detect problems like bottlenecks, traffic violations and dangerous driving conditions, but develop new strategies for mitigating these issues.
- A call center might use UDP to automate aspects of the quality control process. Using speech recognition technology and analytics, organizations can now monitor 100% of customer calls for compliance issues and even alert agents in real time if they make a mistake.
Given the expansiveness of UDP’s potential, over the years we at IAN have become curious about how organizations are deploying the technology in the real world and where they plan to apply the technology next.
To gain a better understanding of current UDP adoption trends, we surveyed more than 300 digital transformation, intelligent automation and RPA leaders on how, when and why they’re embracing UDP. The following is an overview of the key trends and areas of growth we uncovered.
ALSO READ: The Evolution of Unstructured Enterprise Data
Unstructured Data Processing Trends
UDP Technology Uncovered
What types of unstructured data processing (UDP) solutions and technologies are you currently using?
Unstructured data processing (UDP) is an umbrella term that encompasses dozens of different technologies, strategies and AI techniques.
The most established, well-known and widely adopted UDP solution is Intelligent Document Processing (IDP) and Automation (IDA).
With the help of RPA bots, AI and computer vision, IDP extracts text data from documents (e.g., email text, PDF, and scanned documents) and converts it into structured data. Especially popular in paper-heavy industries such as banking, insurance and legal services, IDP/IDA solutions enable organizations to not only automate manual data entry workflows, but also derive meaningful insights into business operations.
Reliable and (relatively) easy to integrate into existing IT systems, while also proven to deliver quick efficiency wins, it was no surprise that 50% of our respondents have already adopted IDP/IDA solutions and 42% say they’re looking to invest in this area down the line.
Following IDP/IDA, in terms of current deployments, optical character recognition (OCR) (49%) and computer vision (42%), followed closely behind. As both of these tools are fundamental components of IDP and UDP architecture, this too is no surprise. Furthermore, the computer vision market is projected to explode over the next few years, ballooning from $9.45 billion in 2020 to $41.11 billion by 2030, according to research by Allied Market Research.
When asked what solutions they were looking to adopt in the near-future, 54% of respondents selected Polygon Annotation, making it the top area of growth amongst all of the technologies listed.
Polygon annotation is a precise way to annotate objects by selecting a series of x, y coordinates along its edges. In other words, it makes irregular, polygonal-shaped objects recognizable.
For example, polygon annotation techniques help self-driving cars recognize objects on the road such as other cars, pedestrians, street signs and sidewalks. It’s also commonly used in factory settings to examine products for potential quality control issues and by retailers looking to label asymmetrical shaped objects (i.e., shoes, clothes, furniture, etc.).
Following polygon annotation, 49% of respondents said they were looking to adopt cognitive machine reading (CMR).
CMR platforms use AI and advanced pattern recognition techniques to process multiple types of unstructured data (i.e., checkboxes, tables, handwriting, cursive, and images) from multiple sources all at once. Unlike traditional OCR tools, CMR are self-learning and improve on their own without human intervention.
Last but not least, 46% of our respondents selected sentiment analysis as a key area of investment. Leveraging AI and NLP, sentiment analysis tools mine text in order to gauge people’s opinions or attitudes. Frequently used to analyse social media posts and customer service interactions (i.e., chat, email, video and call transcripts), sentiment analysis can help organizations understand how people view and interact with their brand.
It should also be noticed that in the “other” option, some interesting industry specific and innovative applications came up including sustainability/energy efficiency, engineering, clinical research and medical AI development. To us this indicates just how versatile UDP technology is.
UDP Adoption by Function
1. Corporate Finance
Though UDP can add tremendous value to many parts of the organization, it is often first deployed in finance and accounting. In fact, 40% of our respondents have already deployed UDP in finance, making it the top area of current adoption.
Corporate finance is fertile ground for UDP for a number of reasons. To start, it tends to be document-heavy and transactional. Take for example the accounts payable (AP) process, a notoriously inefficient and cumbersome activity. Not only does UDP eliminate the need for manual data entry and invoice matching, but it also uses machine learning algorithms to route documents to the appropriate individuals for validation and exceptions handling, enabling “touchless,” straight-through processing (STP).
In addition, ensuring accuracy and precision is paramount due to the various corporate, local, state and federal regulations finance departments must adhere to. AI-enabled UDP systems help support auditing and compliance by automatically monitoring documents against rules and laws, flagging those with issues. In addition, machine learning algorithms can monitor financial transactions for anomalies and instances of fraud. Considering the Association of Certified Fraud Examiners (ACFE) estimate that organizations lose 5% of revenue to fraud each year, the ROI for such tools can add up fast.
2. Customer Service
Representing a top area of growth, 37% of our respondents stated that they were already leveraging UDP to optimize the customer experience with an additional 37% indicating they planned to expand into this area within the next two years. In addition, 56% of our respondents said “improved customer satisfaction” was a top benefit of UDP (more to come on that in Part Three).
In which functional areas is your organization leveraging unstructured data processing (UDP)?
Do you truly understand what your customers really think about your company, brand and products? UDP can help you find out by extracting data from and analysing millions of customer transactions (i.e., emails, social media posts, video reviews, call recording, chat transcripts) at once, identifying those key areas of success as well as improvement.
In addition, UDP tools can be used to monitor and enhance contact center performance in real-time. These days, call centers are equipping call center agents with AI virtual assistants that can help with a variety of unstructured data-rich tasks such as:
- “Listening in” in on calls to provide the agent with real-time suggestions and next best actions in order to maximize call quality.
- Automatically complete tasks without human intervention (i.e., cancel accounts, locate customer files, update customer profiles, etc.)
UDP-enabled tools such as chatbots, voice agents and computer vision can then enable self-service, allowing human customer service workers to focus on more complex cases and strategic priorities. In addition, these tools have enabled organizations to create new, cutting-edge customer experiences.
3. Information Technology
The role information technology (IT) plays in organizational success is rapidly expanding. As such, IT teams must increase both their operational performance and strategic effectiveness. UDP can help them do both.
Using NLP, IT departments can process, classify, analyse and remediate helpdesk tickets in seconds. Due to its advanced pattern recognition capabilities, UDP solutions can also be used to identify hidden IT issues that may be undetectable by traditional structured data methods. For example, re-clustering incidents around keywords such as “password,” “log in,” or “slow” can help IT departments more effectively identify correlated incidents and their root-causes.
Use cases for UDP don’t end at the help desk. UDP tools can also be used to enable fingerprinting, anomaly detection and other cyber security technologies. In addition, developers, programmers and engineers frequently use UDP tools to enhance customer facing applications and data management ecosystems. Furthermore, it’s the IT team that typically spearheads the implementation of UDP technology across the enterprise.
As of now, 39% of our respondents have deployed UDP in IT with 27% confirming they plan to apply the technology to IT processes in the near future.
4. Supply Chain & Operations
One of the most exciting areas of UDP adoption is supply chain and operations. In fact, a 2021 McKinsey report found that AI-enabled supply chain management improved logistics costs by 15%, inventory levels by 35%, and service levels by 65%. Overall, 38% of our respondents have already deployed UDP in this area while 36% plan to in the near future.
To start, UDP can transform the unstructured data generated by IoT sensors in warehouses, transit vehicles and other points along the supply chain to better understand how environmental conditions impact both the products themselves and the equipment used to transport them. If, for example, the temperature spiked in a refrigerated truck transporting temperature sensitive vaccine, an AI-enabled UDP solution could identify the change in temperature and alert the truck driver and other relevant parties in real time.
In addition, by centralizing data pulled from hundreds of systems across the end-to-end supply chain, UDP solutions can help organizations obtain unprecedented levels of visibility into supply chain performance.
Using computer vision-enabled cameras and IoT sensors, organizations can track their products in real time from production to delivery, automate quality control inspections and, with all of the high-quality data they’ve collected, optimize forecasting.
Last but not least, large organizations responsible for juggling thousands of different vendors at once use UDP-enabled tools to not only automate contract management and compliance, but monitor the internet (i.e., trade publications, social media, vendor websites, video reviews, news articles, industry discussion boards, digital advertisements, etc.) for indicators of supplier risk and competitive advantage.
5. Marketing & Sales
Few functions have been more impacted by the digital revolution than sales and marketing. In fact, these disciplines are almost unrecognizable from what they were 20, even 10, years ago. Social media, digital media, smartphones and email have irrecoverably changed the way we sell, market and purchase all types of products.
As a result, AI adoption in marketing has exploded. In fact, Salesforce’s 2020 Marketing Report found that the percentage of marketers leveraging AI jumped from 29% in 2018 to 84% in 2019.
Furthermore, according to MarketsandMarkets, the marketing AI market is expected to grow from $6.46 Billion in 2018 to $40.09 Billion by 2025 (29.79% CAGR).
When it comes to AI-powered UDP specifically, 30% of our respondents have already applied the technology to marketing & sales activities and an additional 30% plan to do so in the next two years.
In addition to helping organizations analyse social media posts and product reviews for customer sentiments, UDP can also help marketing and sales teams automate and optimize numerous aspects of their day-to-day responsibilities. For example, UDP can be used to help marketing teams more effectively tag, classify and manage their digital assets library.
Furthermore, using machine learning and NLP, UDP tools can analyse customer data from numerous sources (past purchasing behaviour, customer service transcripts, social media posts, etc) and use those insights to generate personalized marketing messaging and campaigns. In addition, marketers can now use AI to automate the creation of synthetic content. With specially tailored UDP solutions, organizations can now create new promotional videos, images, blog posts and even songs in minutes instead of weeks.
Why Unstructured Data Processing (UDP)?
What value has unstructured data processing (UDP) brought to your organization?
The number one goal of any business is to deliver value to its shareholders and customers. As you may have surmised from the early sections of this report, UDP is an elegant tool for achieving this objective.
When it comes to delivering bottom line results, UDP dramatically increases efficiency and reduces overhead costs by replacing cumbersome manual tasks with seamless, automated processes. In fact, when asked to select the top benefits of UDP, 67% of our respondents chose improved efficiency and 53% selected increased cost savings.
When it comes to improving the customer experience (56%), not only does UDP have the potential to optimize CX directly through the creation of new, cutting-edge customer experiences, by analysing high volumes of video, speech and text data in real time, it can also provide organizations with potent insight into customer behaviour, attitudes and needs.
Unstructured Data Processing (UDP) Vendor Matrix
What UDP vendors are you currently or considering working with?
In terms of current adoption, the 3 most popular UDP vendors are Tableau (43%), Automation Anywhere (30%), and UIPath (26%).
For those looking to transform unstructured data into sophisticated visualizations, Tableau will likely remain a top choice given its advanced data representation capabilities and impressive UX.
Likewise, when it comes to more traditional intelligent document processing needs, our respondents indicate they will continue to look towards established intelligent document processing and RPA players UiPath (38%), Automation Anywhere (37%) and Datamatics (36%).
However, the appetite for UDP around video, audio, and images is clearly growing.
In fact, our respondents expressed growing interest in AI platforms such as Primer.AI (36%), Veritone (35%) and Super.ai (33%).
Though the line between AI platforms and legacy IDP players is certainly blurry, AI platforms go beyond document support to deliver insights into any data source. Providing end-to-end AI model creation, testing, deployment, and monitoring capabilities, these AI platforms can be used to create AI-enabled applications ranging from fraud detection to synthetic content creation to video object identification and tracking.
ALSO READ: The Trends that will Define 2022
