This article provides a framework for measuring productivity in an IDP platform.
What is an IDP system
An Intelligent Document Processing (IDP) system is a platform that provides capabilities to ingest, classify and separate documents and extract data to drive business processes. A successful IDPsolution ensures the seamless execution of transactions in operational productivity applications and reduces manual work resulting in accurate and efficient operations.
Most documents received by businesses that require attention contain a WHO and a WHAT. We need to find who is making a request from us and what they are asking us. We usually have to know what type of document we are dealing with first. In machine learning, we call this the classification problem. The document type informs us about the specific metadata to expect from a given document. In addition, we define fields to represent the WHO and the WHAT. The process of automatically finding the WHO and WHAT fields is called extraction.
Let’s use insurance as a vertical and focus on a claim document that an insurance company receives through email or mail. In the insurance world, Member ID is defined as a field to find the WHO. The extraction process uses algorithms, machine learning, and other techniques to find the Member ID and other key pieces of data on the document. Once the metadata is found on the document, either automatically by AI or by a human, the data is then passed down to a case management or business process management system to process the claim further and reimburse the insurer.
The factory metaphor
An IDP system is very much like a document factory. The raw material is documents in the forms of paper, email, fax, pdf, image, word etc., and the output is a document with verified metadata (fields). In other words, you have unstructured data as input, and the output is accurate structured data. Usually, a downstream system uses the metadata to process the WHAT for the WHO. In the case of our claim example, the data is used for adjudicating a claim.
The factory uses labour (human in the loop) for verifying classification and extraction results, and it requires capital (servers) to process the documents. So any improvement in technology (the big T in economics) improves productivity. Improvement in technology may be in the shape of an advanced AI or a seamless user experience enabled by software.
What should we measure
An accurate KPI for measuring productivity is calculating how many documents per hour of human labour our factory produces before and after implementing a and IDP solution. You could also change the denominator and the numerator and calculate how long it takes to process one document per second of human labour. Both numbers tell the same story.
Let’s look at the example of a system that currently produces 100 documents per hour of human labour and has 2 full-time staff. This factory can output 200 (2 people * 100 documents each) documents in an hour. If we hire two more employees, the factory’s output will increase to 400 documents per hour; however, the metric remains at 100 documents per 1 hour of human labour. Thus, the increase in output isn’t the result of productivity improvement but additional labour.
Now let’s assume that we still have four employees, but we implement an AI that automatically classifies the documents and extracts some metadata with minimum human intervention. We measure the factory output, and we notice it has gone up to 520 documents per hour. We then measure the metric, and we notice that employees can now process 130 documents per hour of labour; since machines are doing some of the work, humans are afforded time to process additional documents.
In this case, the productivity gain is (130 - 100) / 100 or 30%.
To get a dollar value representing the cost-saving, we can multiply the productivity gain by the number of full-time staff and then multiply the result by the average salary.
e.g. if our factory has 4 employees in total and the average salary of our employee is $60,000, the cost-saving would be 30% * 4 * $60,000 or $7,200
Case study from a past client
in 2019-2020 I spent over a year rebuilding a massive IDP solution for a client that processes over 18 million documents a year. An army of developers from the vendor and IBM had built the original system in 2010. With the newly redesigned system, built by 1 and a half developers plus myself, we leveraged every capability available in the platform. On top of that, we innovated so much above and beyond the platform that our project qualified for the Scientific Research and Experimental Development Tax Incentive Program. As a result, the time it took to process one document went down from 51.14 seconds of labour per document to 10.85 seconds of labour per document. The net productivity gain from AI improvements, smart algorithms, and a better user experience was an incredible ~ 80%.
Another way to look at it is to calculate the number of documents we can output per hour of labour. With this KPI, our employees were able to process 70.3 documents per hour, and now each can process 331.8 documents per hour. It’s still an ~80% improvement but from a slightly different view.
The document department in the aforementioned company employs 40 people to process these 18 million documents. The average salary of an employee is $50,000. Can you calculate the gains in dollar value?
80% * 40 * 50,000, a whopping $1.6M of savings in a year.
In hindsight, I would have been better off if I had asked my compensation to be minimum wage plus 1 year of cost-saving. Live and learn!