- [Instructor] Back several generations in my family tree,
I have some ancestors who were miners.
They would go down to work in the mines early Monday morning
and wouldn't come back up until Saturday night,
it was backbreaking and lung blackening work
that I'm grateful I would never have to do,
but to be perfectly clear,
that's not what we're talking about
when we're talking about data mining,
rather data mining is the practice
of working with large datasets
to find potentially valuable insight.
Think of it this way, computer algorithms,
especially deep learning neural networks
are able to go through millions of images
and automatically identify all of the pictures
with cats in them and show you where the cats are,
they draw the frame around them.
Data mining's a little like that,
but instead of finding cats,
maybe you're trying to find new customers,
or new treatments for diseases,
or students with exceptional potential,
really anything that's important to you
using the same general concept.
The idea is pretty simple, you get a lot of data
in your computer, millions of cases
and maybe thousands of variables,
and you have the algorithm start looking for patterns
of some kind in the data.
Now, there is one major risk in doing all of this,
and that is what I personally call Baseball Stats,
that's not to be confused with the real data of baseball,
but you get things like this,
the the batter is a career of three for three
against left-handed relief pitchers
on interleague play before the eighth inning
on away games when the team is leading
by at least four runs after the all-star break,
and you got to wonder how the batter's
even been up to bat three times.
This is capitalizing on chance, these are such small numbers
that these are meaningless patterns,
even if they give people something to talk about.
The major challenge in data mining is to find patterns,
but not get these kinds of false positives,
and in fact, it is false positives
that are one of the major risks,
that's if somebody rolls a 12 with two dice,
that might mean something if it's a good outcome,
and if they do it again you might suddenly think,
they know how to roll a 12.
Well, that's not really the case
because what you're failing to do
is to adequately account for the roll of chance,
lots of things can happen at random.
A person can flip 10 heads in a row,
it doesn't happen very often,
but there's a known probability for that,
and it's important to always know
what is the normal variation and maybe slightly beyond that
to say, this is normal, and when you start looking at many,
many possible effects, that becomes critical,
and that's one reason any time you do a data mining project,
you especially need to be careful about doing verification,
that's the double checking that scene
if your pattern holds up in a different dataset,
it's why it's common in data science
to break up your original dataset into what's called
a training dataset and a testing or validation dataset
to control for these tendencies towards false positives,
but if you can do that, there's some great insights
you can get out of data mining,
for instance, you can find new market segments,
groups of people who's interests have not
been adequately addressed before,
you got another niche there, something you can do.
You can also look for new kinds of product and market match,
especially when it turns out that a product
can be used in unexpected, novel ways.
There are things like market basket analysis
where you can say that anytime a person buys this and this,
there's an increased probability they will also buy this,
so if you're doing e-commerce,
you show them that extra item online
or you give them an incentive to buy it.
And then there's a whole separate category
of time series mining where you're looking for events
that happened over time,
or the cluster related sequence mining
where you're not saying when they happened,
just that it happened before this and after that,
both of those can be enormously useful
in predicting trends with the interactions you have
with businesses or customers.
Now, if you'd like more information on this,
we have another course called
Data Science Foundations: Data Mining,
which gives you some concrete examples
of how each of these kinds of analyses can worked,
and how the results can be interpreted and put into effect,
but either way, the basic idea here is the same,
when you've got data and when you've got a lot of data,
you have something that potentially has great value
if you know how to go through it, and mine through it,
and find those patterns that you
and your organization can act on.