- [Instructor] As an organization,
you need to recognize that there's never
a final version of an AI model.
Your AI model work is never done.
It would need to be updated and improved over time.
This means that the data scientist still need to review
and monitor work on previous projects even
after the models are on production.
This is to ensure that the models stay accurate
as defined by the business.
One of the things you and your team need
to be thinking about is how to monitor the models.
How can you make sure that your model is healthy?
Are the models degrading?
Let's go back to the e-commerce example.
There are going to be new products
that will need to be added to the store.
Some of the items in the store might be seasonal
and the sales reflect that.
New techniques or algorithms might be available
that data scientists might want to try out
and see if they can get better performances with.
Many applications have a dashboard to be
able to monitor the health, CPU usage, and response times.
Why not have a dashboard for your AI models?
If you have the e-commerce store,
you can track spikes or drops in
some of the metrics you are using.
Was there any underlying issue with some of the data?
Can you take the recent data set
and pass it through a development model
and replicate the model behavior?
Perhaps the model that we've created
requires certain input features,
but for some reason, those sets
of inputs are missing in production.
How does the model behave in this instance?
So how often should a model be retrained?
It depends on what they're predicting.
There are always unforeseen external changes
that can affect the accuracy of machine learning models.
Seasonal changes and weather would
have an impact on purchases of an online retailer.
Marketing campaigns can affect sales.
Here's an example.
We are showing the model performance over time.
The darker the shade of red in the heat map,
the lower the model performance.
The darker the shade of blue,
the better the model performance.
Another consideration is that the model that works
for one geographical location might not work for another.
Demographics, weather, and tastes vary across geography,
and need to be taken into consideration
for models to work well.
Now, let's say your core business
was around streaming sports programs globally.
And you want it to provide recommendations
to customers for what they should watch.
Then geography is hugely important.
Soccer, as it's known in the US,
or football, as it's called everywhere else,
is hugely popular in Europe,
South America, and parts of Asia.
The Super Bowl is typically popular mainly in the US,
and cricket is very popular in South Asia.
Things are complicated by say the football World Cup
that comes around every four years.
Those who don't normally watch football are drawn
to support their national teams.
The AI model needs to be
able to take into account both geography
and timing of events, and
as you can imagine, this is not trivial.
I was working on a health project
where we were trying to predict
if patients would require unplanned emergency care.
So that's things like patients turning up at an accident
and emergency unit or requiring an ambulance.
And as you can imagine,
demographic, geography,
and past health can play a significant role.
This is further complicated in cities
where different parts of the population might live
in certain geographies and a model developed
for one hospital in one location might not apply
or work in another part of the country.
So how often do we need to update the models?
If you have a deep-learning application
using say face recognition,
you would not expect there
to be significant changes over the months.
Many banks in the UK are starting
to use voice recognition as a means of authentication.
Now, I'm quite interested
to find out how soon they prompt me
to re-record my voice as part of its model training.
Similarly, you would not expect significant changes
for natural language models.
There might be new vocabulary added,
but this is over several months or years.
Economic models need to change a little more frequently.
For a country, this might be based on a monthly
or quarterly basis when growth results
are released by economists or the government.
Fraud and spend analytic models need
to be updated fairly regularly
to capture the latest means of fraud.
And finally, any algorithmic trading system needs to be
able to deal with real-time changes
and information that it's getting from buyers,
changes in the economic outlook, news updates and so on.
So frequent changes would need to be made for such a model.
This means that real-time AI models
might require dedicated teams working on them
to update daily changes.
Fraud and spend analytic teams need to be tracking
and updating models frequently
to capture any new techniques or methods.
Fraudulent activity makes up a very small percentage
which adds to the challenge
of reducing the number of false positives.
Economic models typically require monthly changes
and will probably not require a dedicated team.
Similarly, monthly changes
to image-based deep learning models is adequate.
The key thing here is to track the models
and to be able to determine when
there is model degradation and to deal with it.
What are some of the signs
that indicate you need to retrain your AI model?
For mobile and telecom providers,
one of the biggest challenges they face is customer churn.
Often, telecom providers will profile their customers,
so those who are loyal,
those who are likely to leave,
which are the most valuable customers based on their spend
or which package they're on,
and now, normally, the highest churn
are from the new customers who've been around
for less than six months.
So if you plot these different profiles out
and find that the distribution seems
to change from the training data to real-world data,
then it's probably an indication
that you need to review the model.
Model retraining normally begins with reviewing
and understanding the data
that caused the model to behave unexpectedly.