Date: 10/27/2020
Challenges of Executing AI Machine
Learning Projects
Umesh Rao Hodeghatta, Ph.D
According
to a recent research report by Wall Street Journal, AI project failure rates
near 50%, more than 53% terminates at proof of concept level and do not make it
to production. Gartner report says that nearly 80% of the analytics projects
are not delivering any business value. That means for every 10 projects, only 2
projects are useful to the organization. Let us pause here a moment, rather
than looking at what makes AI projects fail, let’s look at the challenges
involved in AI projects and find a solution to overcome these challenges.
AI
projects are different from traditional software projects. Typical software
projects, consist of well-defined software requirements, high level design,
coding, unit testing, system testing, and deployment along with beta testing or
field testing. Now, organizations are adopting Agile process instead of
traditional V or waterfall model, but all the steps mentioned are still valid.
However,
AI and Machine Learning projects’ methodology is different from the above. Our experience
working on many AI/ML projects has given us insights on some of the challenges
of executing AI projects. Also, we are in regular touch with senior executives
and thought leaders from different industries who understand the success
formula. The following discussion is based on our practical experience and
knowledge gained in the field.
Successful execution of
AI projects depends on the following factors:
1. Clearly aligned
Business Expectations
2. Clarity on
Terminologies
3. Meeting Data
Requirements
4. Tools and Technology
5. Right Resources
6. Understanding Output
Results
7. Project Planning and
the Process
Before, I go further
onto details, I want to reiterate the fact that there are two sectors of
organizations who are keen on AI projects.
The first category is,
Tech(nology) companies like Microsoft, IBM, Google, Amazon, Intel, Cisco, etc.
They have been inventing technologies and solutions and have developed 1000s of
software products and applications. For
these tech companies, succeeding with AI projects is no big deal!! They know
the technology very well, data is not a problem, requirements are noticeably
clear, and they have all the resources and expertise who have been working on
the technology and even on machine learning and AI projects for decades. AI or
Machine learning is not a new term to them - “Old wine in a new bottle”.
The second category is
the rest of the non-tech companies, who are exploring the benefits of
initiating AI in their organizations, as every other big company CEO is talking
about AI.
Whereas, the second
category, non-tech companies, only adopt the technology, applications and tools
developed by the first category tech companies and are curious to apply the new
technology – AI, Machine Learning, Deep Learning, Neural Network, Reinforcement
Learning, etc., to solve the business problems. This becomes a new initiative
for the CEOs in the organization in the yearly plan and will trickle down to
the bottom of the pyramid. The subsequent levels of executives have to adopt
the new AI term in their projects’ but they may lack expertise and skills
required to implement and succeed.
Let’s
dive into each one of the challenges I mentioned in detail.
1. Clearly Aligned Expectations
Every
CEO in an organization is obsessed with AI and asking his people “What
percentage of business decisions are we making with help from AI?”,
Aligning
Business is necessary for an AI Project. Whatever may be the technology,
whether it is AI or non-AI, every CEO or Senior Executives are interested in
these three factors:
1.Can
AI boost customer satisfaction level
2.
Is AI helping in making quick Business Decisions
3.
Is AI helping in overall company performance and ROI
Most projects fail
because of the mismatch of expectations. Software projects have well defined
requirements, and they may not change much till the product is released. Further, the Agile methodology provides the
flexibility to handle the changes in a better way.
Whereas in AI and ML
projects, understanding requirements itself can be challenging. It is important
to know the problem an organization wants to solve and whether it can be
solvable using AI/ML techniques.
In case of AI and ML
projects, sometimes an organization defines a problem which can be solved
without AI or ML. In other cases, though the organization has a well defined
business problem, the project manager has aligned the resources, and kicked off
the project, the data scientists report that they need “data” to solve the problem,
and the team may not get access to the right data that is required to solve the
problem for a long time or the organization may not have the required data.
After months pass by, project is still in initial stage with no progress
towards solving the problem. This can make the management unhappy and
cancelling the project.
Sometimes, the technical
people may define the problem which the business is not really interested
in. If the organization is not
interested, even if the problem is solved, the solution may not be deployed
leading to wastage of time and resources.
Also, it is possible that the technical people may define it wrongly
leading to not usable solutions.
Hence, based on our
experience, we feel, before you agree to start the project, it is necessary to
set the expectations. Know what is the problem an organization is trying to
solve, whether the problem practical, is it really a Machine Learning project
or just an automation software project, do you have proper data, are you
authorized to access data, etc. You must set the organization expectations
properly, while explaining the data requirements or whether the problem itself
is AI/ML related or not.
When
an AI Project is initiated, it is important to set the expectation with the
Business and the Data Science/AI team:
1.
Deployment – How long it takes for deployment and how you integrate with the
production systems, and some of the concerns with data/cyber security.
2.
What is the value add of the new AI System
Always remember, every
CEO is interested in leveraging cutting edge technology to:
a. Achieve customer
satisfaction
b. Quick Business
Decisions
c. Company performance
and ROI
In
Summary, Set proper expectations:
-
Is
it really AI/ML problem?
-
Set
the DATA expectations
-
Align
business with the problem and requirements?
2. Clarity on Terminologies
Because of the current
trend in technologies and the BUZZ it creates, too many terms like Machine
Learning, Data Science, Business Analytics, Data Analytics, Data Visualization,
Data modeling, Predictive analytics, Diagnostic analytics (I heard this recently…something
I had never heard before), Prescriptive analytics, AI, Deep Learning,
Reinforcement learning, etc, are heard by people. Are these terms same? If not,
what is the difference?, which term means what? is very confusing for people,
especially those who are new. This is so
because various people define each term differently and when any such term is
used, people are not clear what it really means in the context.
Each industry may differ
in the definition of these terms - what
is Data Science for Home Depot1 may be Machine Learning for Google2
and vice versa. Hence, based on our experience, if possible, it is not worth
defining these terms and convincing the company executive about the “confusing
terminology”. Instead, it is better to focus on the problem and how it can be
solved, whether to apply supervised machine learning or simple descriptive
analytics.
3. Data Requirements
For Business, Data is everything.
Agree?
Data is Information
Data is MONEY and
Data MEANS BUSINESS
For Business, DATA is sensitive
Business DATA is under
surveillance, being monitored by government regulations and other regulatory
bodies.
But, FOR AI /Analytics projects, without “DATA”,
there is no Analytics or AI/Machine Learning projects. Data plays an important
role in AI/ML projects.
However, Data is information, Data is the secret of business.
Exposing data is exposing your business. Organizations are overly sensitive
about their business data and moreover, CEOs, CFOs are held responsible for any
breach of data, data falling into wrong hands, used for wrong purpose. If any
small data breach happens, they must answer to investors, government regulators
and to consumers. How much risk can they afford to take? Hence, organizations
have strict process on who should access and what data should be accessed.
There can be several approval processes involved before getting access to data
and the whole process may simply take lot of time. Or else, we may get only a portion of data
which may not be useful unless other related portion of the data is also
obtained.
Without “DATA”, there is no Analytics or AI/Machine Learning
projects. Data plays an important role in AI/ML projects. Data is information,
Data is the secret of business. Exposing data is exposing your business.
Organizations are overly sensitive about their business data and moreover,
CEOs, CFOs are held responsible for any breach of data, data falling into wrong
hands, used for wrong purpose. If any small data breach happens, they must
answer to investors, government regulators and to consumers. How much risk can
they afford to take? Hence, organizations have strict processes on who should
access data and what data should be accessed. There can be several approval
processes involved before getting access to data and this whole process may
simply take lot of time. Or else, we
may get only a portion of data which may not be useful unless other related
portion of the data is also obtained.
AI / ML Depends on DATA. Hence, AI
Project team must understand the Organizations’ challenges with respect to data
and work with them closely. Understand the company process, Understand DATA
Compliances, etc.
Though the team has planned to work on a well-defined AI
project, without data they simply have to wait for approval process. You can
imagine the weekly updates that would be on the project or daily scrum standup
meeting updates.
If the AI project involves applying supervised machine
learning, then can you get labeled data? Why would a company have labeled data
in the first place? Who will label the data for you? Will anyone understand
what is labeled data? What is the cost of labeling exercise? Is the company willing to bear this cost?
Imagine you are working on a healthcare project, classifying
X-Ray images. You have to ask hospitals/healthcare providers to give you not
only X-Ray images but also labeled X-Ray images with the corresponding issues
found in them. Do you think, they have labeled X-ray images as per your defined
classes?
Inaccurate DATA can lead to wrong predictions and wrong
decisions
Minimal Data may not be useful.
Hence, you must set the “data” expectations at the beginning
of your project. Otherwise, you are bound to FAIL!!
4. Tools and Technology
Tools support
implementing technology required to solve a specific problem. All the tools may
not support all the technology. Different technology is applied to solve
different problems. For example, using “deep learning” to solve simple “credit
approval” problem, may be an over kill. Or using decision tree for image
classification may not give desired results. Or there is no need of applying
machine learning to clean a database.
There are open source
tools and commercial tools. The decision to use the appropriate tool depends on
the organization’s long term goal and objective. Many factors can play a role
in selecting the tools including IT infrastructure and production environment
and the compatibility of the tool with these.
Also, the cost of acquiring these tools may be a factor in case of
commercial tools. What is important is choosing the tools which support the
right technology and integrates well with your existing environment.
Commercial
tools such as PowerBI, Tableu, are extremely good at Data Visualization and
descriptive analytics.
You
don’t need expensive tools for data plumbing and cleaning jobs. Any software
can be programmed to do data preprocessing required for building ML model.
5. Resources
Resource allocation
varies from organization to organization. Definition of a data scientist, what
are the tasks of a data scientist, should you hire a Ph.D. or not, what type of
“scientist” is required to handle your data - is something people in the industry
differ and argue on. However, we will not be getting into the debate of
defining who is a “data scientist” or the role of a data scientist.
We believe that the data
scientist should spend more time on developing AI/ML /analytical solutions and
the type of model she/he should be creating. Data scientists should seek help
from the software coding experts in preparing the data. Data preparation may
involve tasks like handling missing values with appropriate techniques,
discarding or rectifying wrong data, deleting certain columns, merging multiple
columns to come up with a new feature, transformation of data, scaling of the
data (in short data wrangling and data plumbing), reduction of features to
meaningful short features, etc. This requires coding skills such as python, C++
or Java or any other programming language. When it comes to text, preprocessing
is more involved such as tokenizing, removing punctuations, removing
unnecessary junk characters, removing emojis, etc. This requires a solid
knowledge of “regular expressions.” And we believe this can be achieved by
anyone who has good programming skills.
As per our resource
definitions, a “data scientist”, should spend more time creating different Deep
Learning/Machine Learning models, in training the model, understanding the
output of the model, improving the performance of the model by tuning the model
parameters, in deciding which model is best suitable for the business problem
and finally helping in deployment of the model. We believe this requires an advance
knowledge of probability, deep understanding of statistics, different machine
learning algorithms (not just the usage of scikit-learn or TensorFlow
libraries), and creating various deep networks.
Similarly, Data or
Busines Analysts can perform descriptive analytics using MSBI or Tableau,
According to our
experience, hiring a data scientist based on “coding test” may not be wise and
you may end up hiring “coders” not a “data scientist”. Further, each project should be staffed with
the right type of people with right competence to deliver best results. Wrong type of resources even in good numbers
may lead to delay of the project.
6. Output Results, Trust and Ethics
Interpreting
Output results of AI/ML models proves the Trust and ethics of AI Project. Wrong
interpretation of the results with non-validated/not completely and effectively
validated model can lead to misguided decisions which can jeopardize the very
purpose for which we are deploying the model.
What is the acceptable
accuracy? What is the kind of samples selected to arrive at the model? Are
these samples sufficient? Have we considered the entire set of relevant data?
Or have we omitted an important aspect or feature from the data which may lead
to seemingly good but misleading results.
For a model which gives general information like if this is right time
to book the flight ticket, it may not matter much. However, if it has to do with the life and
safety of human beings, then even a few false positives or a few false
negatives can seriously undermine the use of the model in spite of high
accuracy.
Further, different
algorithms may have to be validated through different but relevant
metrics. If we use wrong evaluation
metrics it can lead us in wrong directions.
Similarly, overfit models may not be relied upon as they throw up
different results on unseen / new data.
We should be absolutely
clear that the models have generalized the solution and are not an overfit to
the training data. Further, the
evaluation metrics on the test/validation data including new data should give
us the confidence that the model is deployable for production use. Such decisions should be purely on the
fulfillment of the purpose/objective of the model.
Output of AI Machine
Learning models depend on several factors but mainly on the “data”. Let’s say,
your data scientists have developed a face recognition AI system, with an
accuracy of 99%. Or let us say your data
scientists have developed a model which can predict a disease based on the data
input to it and the accuracy is 99.5%.
The organization has to make a decision whether to deploy this model or
not.
7. Project Planning and Process
Traditional software
project planning has a well defined process, timeline, milestones,
deliverables, tasks and subtasks allocated to resources, risks and mitigation
plans, and weekly tracking. All these can be defined using standard methods and
a lot of prior knowledge which is already existing among the resources.
The same project
planning cannot be applied for AI and ML projects because of many unknowns and
uncertainties. You may wonder, how can you execute a project with so many
unknowns and uncertainties?
The success of AI and
Machine Learning project depends primarily on the “data”. The first step in any project planning is
defining business requirements. Since AI and ML projects rely on data, you have
a couple of choices. The first choice is you explore the data provided by
business and then define what problems that can be solved OR the second choice
is, the organization will define the business requirements/problem and then you
explore the available data and check if you can get the required data.
With the first choice,
you already have data and it may take sometime to explore the data, analyse the
data, understand the data and defining the problem that can be solved, however,
the problem that you are defining to solve may not be relevant or appealing to
the business and they may reject your proposal. The entire time you have spent
is then considered as a waste.
On the other hand, if
business defines the problem, you must be clear about what data you need to
solve the problem. Once the problem definition and requirements are clear, then
the exploration of data starts. If such data does not exist, then you can
either stop the project or try to implement methods to start getting such
data. If you have the data, the step is
to get the data from various sources applicable, consolidate them and clean the
data as relevant. Here, the aspects like access permissions, etc. matter. Next step is, to determine the AI/ML method
and algorithm to be used, run the same against the cleaned data, analyze and
validate the results. Where the results
are not up-to-the expectations, we identify the alternative methods/algorithms
and/or tune the hyper parameters and carry out further analysis till we are
satisfied with the results per the objective of the project.
We use various metrics
to measure the performance of the model and once our validation gives the
requisite comfort and our metrics suggests good generalized learning and
meeting business requirements of prediction of the model, we go ahead to deploy
the model on to the production.
Figure 2: AI and ML Project Lifecycle
In both the scenarios,
it is hard to define the project plan, milestones & deliverables, risks
& mitigation plans, tracking the project on Sprints. There are challenges
at each stage, whether getting access to data, exploring data, labeling data
for supervised machine learning, or which algorithm will give best results, or
deploying the models in fields. Hence, executing and managing AI and ML project
is much more challenging than other software projects.
It is a complete
reversal of Software Project process.
To
summarize,
Success
of AI project depends on the following factors:
1. Aligning Expectations
2. Setting Data
Expectations
3. Using of proper Tools
and Technology
4. Aligning Right
Resources
5. Asserting Output
Results
6. Aligning Project
Planning and the Process
References:
2. https://www.forbes.com/sites/gilpress/2019/07/19/this-week-in-ai-stats-up-to-50-failure-rate-in-25-of-enterprises-deploying-ai/#34ea003272ce;
3. https://www.forbes.com/sites/gilpress/2020/01/13/ai-stats-news-only-146-of-firms-have-deployed-ai-capabilities-in-production/#4da0ee526500
