- [Instructor] Now, remember that the purpose
of this course is to identify what's required
to introduce AI into an organization and how to go about it.
I won't get into the technical details
of creating AI models here.
I've covered some of these topics in my other
machine learning, deep learning framework courses
with Charis and PyTorch and pandas for data analysis.
So the start of the AI project might include things
like setting up cloud accounts and credentials,
extracting data from the platform.
It might even be providing only samples of data
to the data science team to work on
while the rest is ingested.
Now, this is where it really helps
to have a generalist team, one where you can
share out tasks within the team.
At the start of the project, the data scientists
might not have access to all of the data.
There will still be many tasks that they can help with
in ingesting some of the data from the platform
to the AI system.
One thing to be very wary of in projects at this stage
is for the data scientists to understand
what is the minimum amount of data required
to create a model.
It can be common for data scientists to want to collect
all of the data upfront to only realize later
that some of it is irrelevant.
And it's important for the data scientist,
the product owner, and the scrum master to talk
this through and negotiate a way forward.
It's much better for the data scientist to spend time
trying to get a working model with a minimum amount
of data and iterate from there.
Now, once more data becomes available, the focus
of the data science team shifts to cleaning the data,
getting it into the required format, and doing some
exploratory data analysis.
The platform support team and the data engineer
might continue to work on making more data available
and writing scripts to help automate some of the tasks.
In some of the projects I've been involved in,
the initial part of the project, the data engineer
was working on scoping and getting the right number
of nodes in the cluster and ensuring that some
of the services don't monopolize the cluster resources.
Initially, there can be a tricky balance
of getting resources right, where both the data scientists
and the data engineers are accessing services
at the same time.
As the data science team makes more progress
with exploratory data analysis and start getting
the data in a format, they can try out some simple models.
Once even the most simple AI model is available
it's important that this work is shared
with the platform team.
There are several reasons for this.
Perhaps the results that are provided from the model
are not in the required format.
If these results are shared earlier, then the platform team
can provide the necessary feedback.
Continuously sharing intermediate work
is hugely helpful for feedback.
Perhaps the data science team have misunderstood
one of the requirements and this can be quickly rectified
with the product owner.
AI model development and data science will probably
not work at the first attempt.
There might be a lot of pivoting involved.
There might be changes in the requirements.
The model might work but not be as accurate
as the business requires and more work needs to be done.
But there's no reason for the data scientists
and the machine learning engineers to hide away
in a room until you have the perfect model.
Continuous conversation with the rest
of the scrum team is essential.
Now, using the example of a recommendation engine
for an e-commerce site, perhaps if customers
are shopping from a laptop or PC you might display
four or five complementary purchases.
For those on a mobile phone, there might not
be enough space on the screen to display
complementary purchases and so perhaps fewer are displayed.
So it's possible that you might want to engage someone
who has some experience with user experience design.
Towards the middle of the project, the focus
is primarily on the AI model that has been created.
The data science team might have created
more than one model.
At this stage, the data science team will be looking
at parameter tuning, feature engineering,
and possibly model ensembling.
Back to our example of an e-commerce site.
For a certain customer, there might be different
recommendations from different models and the team
are trying to determine which model's predictions
are better and what metric will be used.
One of the biggest challenges when it comes
to working on the AI models algorithms and analysis
is that there's a very low level of collaboration
between the data scientist and the team.
This is probably because this requires more
specialist skills and is still a relatively new skill.
But it's essential that some sort of collaboration
is introduced and this is not the work
of a single person end-to-end.
The scrum master can help facilitate knowledge-sharing
sessions throughout the scrum.
If you're trying to introduce AI projects
into an organization, then the developers in the team
will benefit from learning how this is done.
Additionally, should there be only one data scientist
in the team and they leave, this project and others
need to still be able to progress without them.