- [Instructor] Sometimes when you're working with big data,
you get to play the part of the detective,
putting together the clues and drawing
the connections between them.
Anomaly detection is one place
where this practice is absolutely critical.
There are a few places where anomaly detection
can be really important.
One is process failure.
You're trying to see if a machine is likely
to break down soon or if a software system is going
to fail under a particular load.
You might be looking for new areas of potential value,
say, for instance, markets that you might have missed
or new product areas or particular clients
that you can use them to identify outliers
that have not been adequately addressed previously
like gold in the rough here.
And one really important area
of anomaly detection is fraud detection,
or the ability to stop crimes
before they get carried out.
In all of these cases, you have a few general approaches.
Number one, you can look for cases
that don't fit the usual patterns.
You can look for outliers that are curious
and say, "What's going on here?"
And then get some further investigation to try
to understand what gave rise to that anomalous observation.
Or in certain cases, you can simply look
for ones that fit known patterns of fraud.
When I try to sell things online
and a person wants to send me a cashier's check,
and they want me to mail it out of town,
I know it's a fraud.
I don't have to do anything else.
And both approaches, looking for curious anomalies
and cases that fit known fraud can be very useful
for anomaly detection and fraud detection.
In terms of the actual algorithms,
you have the standard list of choices.
You can do things like regression,
classification trees and decision trees.
You can use neural networks.
Any of these would work for identifying both outliers
and cases that fit known patterns of fraud.
The kind of data that you're going
to be including might include things
like the purchase characteristics,
what was purchased for how much, when, where,
through what means and so on.
Credit card purchases of gift cards are always suspect
and they're going to set of alarms no matter where.
Then there are personal measures, so things like biometrics,
scanning your fingerprint or your face,
the latency in typing your password or your user name,
height as estimated by the angle
that you hold your cell phone,
the signature squiggle that you use
when writing your name online
or even just finding the mouse on your computer.
All of these can be recorded,
and all of these can be used as data
to identify the authenticity of the person
who's intending to make the purchase.
And then of course there are general trends.
There are new scams going around all the time,
and they might be more common in one area or another.
Have scammers found a way around all technology?
All of these can be used as the data point
in identifying whether a particular proposed transaction
is fraudulent or legitimate.
And what's important to say about all of this is
this is all at the very heart of big data.
Big data plays a critical role
because we don't have just numerical information.
We have biometric information.
We have GPS locations.
We have all sorts of data that don't fit
into the standard rows and columns of a spreadsheet.
That gets at the variety of big data,
and because these decisions often have to be made
in real-time while a person is attempting
to process the credit card,
that gets at the velocity of big data,
and it's compared to the huge volume
of transactions that accrue every day that allows you
to make these decisions about what is normal,
what's anomalous, what's legitimate, and what's fraudulent.