Productivity Framework Part 2 – A step by step guide

This article aims to dive a little deeper into how to measure productivity in a cognitive capture system. The principles introduced here are generic and can be applied to any system. However, we use a Kofax/KTM platform to provide specific examples.

If you haven’t read the introductory post about the framework, I strongly recommend reviewing it before reading this post.

A typical capture workflow

We already discussed what a (cognitive) capture system is trying to solve. A capture system is basically a workflow of attended modules and unattended services that enables processing documents. The flow below depicts a typical workflow:

Document Ingestion

Documents, in the form of the email body, email attachments, scanned paper, faxes, file imports, etc., are ingested and imported into the system and kickstart the workflow.

Classifiaction Service

A classifier runs a pre-trained model on documents in this service to predict whether a document belongs to pre-defined buckets. A real example would be classifying documents received in a mailbox to correspondents, application forms, supporting documents, etc.

If documents were scanned as paper, an additional classifier is run to determine the start and end of a document. The process of determining the start and end of a document is called separation.

Document Review

When the classifier is not confident about a document, it will flag it and send it to document review for a human to review the results. Humans can change classes, merge, split or even delete documents and pages in this module.

Extraction Service

The next step in a typical flow is the extraction service. Any AI, algorithm, formatters, validation rule or custom logic is executed as part of this service to extract values for your predefined fields.

Validation

If a document contains a field that the system was not confident about its value, it will route the document to the Validation module for human review. Humans will review all the unconfident fields and either confirm a value extracted by the system or retype the correct value.

Export

The last step is to hand over the document and its associated metadata to a downstream system/s performed by the Export service.

Quality Control

Documents can end up in this module either by the automatic rejection of unattended services or by a human in one of the attended modules. Either way, the sheer routing of a document to this module means something has gone wrong and needs further review and action by a human.

A senior resource usually works in this module to resolve the exceptions.

What values to collect

If you remember from the previous article, the KPI we are interested in is how long it takes to process one document per second of human labour before and after implementing a capture system.

Number of Documents

This is the number of documents that went through the export service successfully in a specific time frame.

Cumulative seconds of human labour

This number is the total seconds all the operators spent processing documents in attended modules in a specific time frame. Please note that unattended services must be excluded from this calculation.

The Query

Here is a query you can run against Kofax Analytics for Capture Database. Of course, the idea behind the query applies to other capture systems as well.

Select distinct BatchExports.ExportDate
,BatchExports.BatchClassName
,BatchExports.CtExpDoc ExportDocCt
,TotProcSec TotProcSec

from

(SELECT distinct dateadd(day,datediff(day,0,SessionEndTime),0) ExportDate
,BatchClassName
,sum(DocumentCount) CtExpDoc

from

[KAFC_Data].[dbo].[LastVersionOfBatch]
where ModuleName='Export' and [State]='Released'
and SessionEndTime>'2021-01-01' and SessionEndTime<'2021-08-01'
and BatchClassName in ('My BatchClass')
group by  dateadd(day,datediff(day,0,SessionEndTime),0) ,BatchClassName) BatchExports

inner join

(select distinct dateadd(day,datediff(day,0,SessionEndTime),0) ProcessDate
,BatchClassName
,sum(DocumentCount) DocCt
,sum(ProcessingSeconds) TotProcSec

from

[KAFC_Data].[dbo].[HistoryOfBatch]
where Operator not in ('Service Accounts/Admin Accounts')
and ModuleName in (/*'Batch Manager',*/'KTM Document Review','KTM Validation','KTM Validation 2'/*,'KTM Validation 3'*/,'Quality Control')
and SessionEndTime>'2021-01-01' and SessionEndTime<'2021-08-01'
group by dateadd(day,datediff(day,0,SessionEndTime),0),BatchClassName/*,Operator,BatchGuid,DocumentCount,[PageCount],ModuleName*/
) People
on BatchExports.ExportDate=People.ProcessDate and BatchExports.BatchClassName=People.BatchClassName 


order by BatchExports.ExportDate,BatchExports.BatchClassName

Result Set

Here is the result you would expect from the above query. The data here is manufactured for demonstration purposes.

Export DateBatch Class NameExport Doc CountTotal Processing Time in Seconds
2020-11-02Batch Class A6850169510
2020-11-02Batch Class B323123108
2020-11-03Batch Class A6986208207
2020-11-03Batch Class B20820755254

Pivot table

The last step is pretty trivial, but I will share the steps just in case.

First, create a custom field, call it total seconds per document and use “=TotProcSec/ExportDocCt” in the formula.

Next, use the following pivot table setting :

You would see a result similar to this:

Row LabelsBatch Class ABatch Class BGrand Total
2020-11-0224.757.1519.11
2020-11-0229.805.2415..03
Grand Total27.305.6916.52
The magic number we are looking for here is 16.52 seconds. This means that, on average, it takes 16.52 seconds to process one document in this system. Therefore, to compare an old solution to a newer one, you would need to repeat this process for both systems and make a comparison to calculate the improvement.

Productivity by Module

Another interesting and useful view for productivity is breaking it down by module.

The query

Here is the query for getting the data for each attended module:

Select *
from
(select distinct dateadd(day,datediff(day,0,SessionEndTime),0) ProcessDate
,BatchClassName
,ModuleName
,sum(DocumentCount) DocCt
,sum(ProcessingSeconds) TotProcSec
FROM [KAFC_Data].[dbo].[HistoryOfBatch]
where Operator not in ('Service Accounts/Admin Accounts')
and ModuleName in (/*'Batch Manager',*/'KTM Document Review','KTM Validation','KTM Validation 2'/*,'KTM Validation 3'*/,'Quality Control')
and SessionEndTime>'2020-11-01' and SessionEndTime<'2021-03-06'
group by dateadd(day,datediff(day,0,SessionEndTime),0),BatchClassName,ModuleName/*,DocumentCount,*/
) People
order by 1,2,3

When you create the pivot table, divide the average seconds a document spent in each module by the number of exported documents in the same period of time. The result would be similar to the overall productivity report with the addition of the breakdown per module.

The module productivity report is super insightful and can be extremely useful in showing which module is the bottleneck in the workflow, and which leg of your project had the highest ROI. In addition, it tells the complete story of your labour and where they are spending their time.

Leave a comment