How Different an AI Product Manager than the Traditional Product Manager?
What makes an AI Product Manager different than traditional Product Manager?
In this post, I am going to share my experience and I request everyone to put their comments by which the gap between my involvement and other’s experience can be abridged.
When it comes to AI, then first question comes to our mind how can we define AI?
When we say, Artificial Intelligence, then we get the basic definition is as the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.
Here, each decision area is again a vast domain in itself.
For example: When we say about Speech Recognition or Language Translation, then we refer it to Natural Language Processing, when we say about Visual Perception, then we refer to Computer Vision and when we say about intelligent decision making system, we refer to Reinforcement Learning.
So, any Product Manager solving a use case in these areas can be defined as a AI Product Manager. Here itself we clearly say what should be the knowledge area and the skill set of an AI Product Manager, that differentiates them from the traditional Product Managers.
To know more about the market expectations from an AI Product Manager, please refer the below blog:
https://www.zdnet.com/article/ai-is-transforming-product-management/
So, lets start a simple case study.
Automation is a very commonly used term now a days. We had to make certain processes in our product as automated. So for this, I need to understand the complete process, sub processes, input and output for all these sub processes. This is very crucial, without which it is very difficult to find out the process which can be automated.
Lets assume a product providing incident processing in Retail industry.
When it is Incident Processing, then there must be input of these incidents into the system, which can be manually done by data entry operators and also incident information provided through various reports which can be pdf and sometimes scanned copy of pdfs which are in image files, should be input as a case to the system. It is very error prone to manually enter these data to create an incident.
So, I got one automation use case to auto-extract the data from these files and create a case.
Now, the question come how will be this built?
Lets understand the process designed for this:
1. These files should come through email attachments.
2. A scheduler should run to push all the files from email server and put those into a common repository.
3. A REST service API will pull all these files to get processed through a Natural Language Processing (NLP) model, which is to extract the information present in those files.
4. Once data get extracted, the child system notifies and sends the data in the form of xml file to the source system.
5. Source system creates a case with the data available in xml files with a receipt id.
Now, the question comes do I have enough knowledge to visualize the feasibility of NLP model, which will be used for data extraction?
Steps for Model Building:
A. Data Preparation: As part pf data preparation, I prepared a small checklist to
There are two types of files: PDF and Image.
· Do all the files have same templates or different templates?
· What are all different kinds of templates, which is required to create a layout template that contains the coordinates of all areas that should be recognized?
· What are the attributes need to be extracted from the forms?
B. Model Building:
I designed an overall architecture where OCR Tesseract will be used for PDF data extraction and OpenCV’s EAST text detector for image data extraction using python language. The complete architecture will be discussed in a separate blog. I just want to say here that, a AI PM should be able to visualise the feasibility of the use case and provide a solution.
Here the challenge is to overcome the change in the template of these files, which I solved with the following:
§ Since the extracted information of the fields, will be exported into a database or a spreadsheet application it is important to define the correct data type. So if the matching form template can be detected, then (in theory) all coordinates of the relevant areas as well as the data types for the export are known.
§ In real life forms-processing different forms have to be processed with one system. Pre-sorting would be too expensive, so it is important to match the filled and scanned paper-forms with the previously defined form templates.
C. Model Validation: After the data extracted, I had to manually validate the data against each form to know to what extent the model was accurate. Totally, I had validated 100 PDF files.
D. Model Performance Metrics: I had defined the performance metric of each attribute through F1 score.
Before understanding F1 score, lets understand Confusion Matrix.
A confusion matrix is a table that is often used to describe the performance of a classification model (or “classifier”) on a set of test data for which the true values are known. It allows the visualization of the performance of an algorithm.
TP=True Positive: Observation is positive, and is predicted to be positive.
FP=False Positive: Observation is negative, but is predicted positive.
TN=True Negative : Observation is negative, and is predicted to be negative.
FN=False Negative : Observation is positive, but is predicted negative..
Precision= TP/(TP+FP), which signifies how many selected items are relevant.
Recall=TP/(TP+FN), which signifies how many relevant items are selected.
High recall, low precision: This means that most of the positive examples are correctly recognized (low FN) but there are a lot of false positives.
Low recall, high precision: This shows that we miss a lot of positive examples (high FN) but those we predict as positive are indeed positive (low FP)
We calculate an F-measure which uses Harmonic Mean in place of Arithmetic Mean as it punishes the extreme values more.
The F-Measure will always be nearer to the smaller value of Precision or Recall.
F1 score =2(Precision*Recall)/(Precision+Recall)
E. Model Performance Improvement: Now, the model has to run through multiple iterations to improve its accuracy.
So, without having these knowledges, it is difficult for an AI Product Manager to design an automation use case. In a traditional PM role, he has to provide the functional use case and process flow, but in AI PM role, he has to understand the way model behaves with data, which in turn impacts the business turnover.
So, on periodic basis, the model needs to be retrained in order to address any data or target or concept drift.
Productising of the model along with more use cases are coming in next blog…
