- [Instructor] I think it's really important
for organizations new to AI to understand
that deploying AI isn't always easy and straight forward.
The organization might only see a return on investment
after several AI projects have been deployed
and the AI squad is more mature and proficient
in creating a successful product.
One of the biggest problems I have seen
is that many organizations think that they have
the necessary data, but when they dig a little deeper
the data is not readily available.
So it's in several different systems
or extracting the data is really hard,
and perhaps they haven't been capturing all
of the necessary data points.
I worked on one AI project where we were trying
to do predictive maintenance on a certain device.
The manufacturer's suggested lifespan for the device
was between 13 and 15 years.
Unfortunately, the team that I was working with
only had maintenance and repair data
and electronic records for eight years.
Earlier records were stored on paper in four-ring binders.
There was a limited amount of predictive maintenance
we could do with the data available.
AI models fail in production because of model drift.
The results might be very promising
when training the models,
however they perform poorly in production.
Perhaps there hasn't been a team
actively monitoring the performance of the model.
So how do you get around this?
Understand that you still need data scientists
and machine-learning engineers supporting the model
after it is in production
and factor this into the project time.
The end goal is not creating an accurate AI model.
The end goal is continually monitoring the model
that is in production, and making sure
that it performs within the accuracy boundaries
as expected by the business.
In software engineering
when an app is deployed into production,
engineers will measure latency, disc usage,
CPU usage and so on.
A similar level of monitoring is required
for AI models in production.
Perhaps the primary business problem
wasn't an AI one.
Let's go back to our e-commerce example.
Let's say the AI-based recommendation engine
has been deployed and it hasn't made
a significant difference to revenue from the site.
Why didn't it work?
It could well be that the business does not have a good idea
of a shopper's journey.
Had they tracked and studied the journey
of the majority of shoppers,
they might have found that most of the shoppers
were buying additional items
because they were using the store's excellent search engine
rather than using the product's page
or even a recommendation engine.
It might make sense here, to then have a project
that focuses on further improving the search engine,
which will provide a better return on investment.
This isn't a failure of AI per se,
but in this instance,
the primary route to success was not an AI-based solution.
What can sometimes happen is that the product owners
and the data scientists are convinced
that the AI projects hold the key to digital transformation.
Now, if the AI team continues to deploy AI products
and they don't work well,
then this can cause significant problems
within an organization.
So, for example, the models in production are degrading
and they're behaving unpredictably.
Perhaps the AI team isn't mature enough
and the models are not robust enough
to handle real-world data.
Deep-learning AI models, for example,
are not always easily interpretable.
Now what happens is that after repeated failures,
organizations might switch back to legacy systems
where at least if things went wrong,
you could troubleshoot the system.
The management team might lose confidence
and their trust in AI in general.
The problem is that this can cause an organization
to miss out on the powerful impact
that AI can have on their business.
So how do we get around this?
Set correct expectations.
Understand that some failure in AI projects
is to be expected.
Pick projects that are small, well-defined
and in line with the team's ability to deliver.
And as the AI squad grows in experience and confidence,
pick more complex and challenging projects.
And finally, ensure that there are enough
data science resources to monitor
and manage models after going into production.