- [Instructor] Big data isn't just a supersized version
of small data.
It's qualitatively different in a number of ways
that actually matter to what you're doing.
In the book, Principles of Big Data:
Preparing, Sharing, and Analyzing Complex Information,
Jules Berman outlines 10 ways,
maybe you could say they're the top 10,
that big data is different from small data,
and I want to walk you through those.
First, they differ in their goals.
For small data,
the goal is usually a specific, singular goal.
We're trying to accomplish this one task
by analyzing the data.
On the other hand, with big data,
the goals evolve and they can redirect over time.
You may have one when you start,
but things can take unexpected directions.
Second is location.
Small data is usually in one place and often,
in one computer file or one floppy disk.
But big data is much larger because it is big,
and the data may be spread across multiple servers
in multiple locations anywhere on the internet.
Third, the data structure and content.
Small data is typically structured in a single table,
rows and columns in a table,
like a spreadsheet.
Big data, on the other hand,
can be semi-structured or unstructured
across many different sources,
and it's putting those things together
that constitutes one of the major challenges.
The fourth difference is data preparation.
Small data is usually prepared by the end user
for their own goals so the person who's putting in,
they know why it's there
and they know what they're trying to accomplish.
On the other hand, big data is generally a team sport
and the data can be prepared by many people
who may not be the end users
or the analysts at all,
and so the degree of coordination
that is required for big data
is extraordinarily more advanced.
The fifth difference has to do with longevity.
Small data may be kept for only a limited amount of time
after the project is finished.
Maybe a few months, maybe a few years, but after that,
it can go away.
It doesn't really matter.
Big data, on the other hand,
may be stored in perpetuity
and it actually may become part of later projects,
so future data might be added to existing data,
historical data, and data from other sources can come in.
It evolves over time.
The six difference has to do with measurements.
Small data is typically measured
in standardized units using one protocol,
usually because one person's doing it
and it all happens in one point in time.
On the other hand,
big data can come in many different formats
that are measured in many different units
and gathered with many different protocols
by different people and different times and places.
And so there is no assumption of standardization
or uniformity, at least in the raw version of the data.
The seventh difference has to do with reproducibility.
With small data, projects can usually be reproduced,
if needed, if your data goes bad
or if your data goes missing,
you can do it over again.
In scientific circles,
this kind of replicability is actually considered
a very good thing.
With big data, however,
replication, meaning gathering all the data over again
simply may not be possible or feasible.
Bad data may have to be identified
through some forensic process,
and there may be attempts to directly repair things
or you may have to do without it.
The eighth difference has to do with the stakes involved.
With small data,
because the projects are not huge,
the risks are generally limited.
If the project doesn't work out
or if a mistake is made, it's usually not catastrophic.
But with big data projects,
because there's so much time and effort
and people's work invested in it,
the risks are enormous.
They can cost hundreds of millions of dollars
and lost data or bad data can doom a project,
maybe even the organization that's running the project.
The ninth difference goes by a peculiar title,
for those of you who are not in computer science,
and that's introspection.
This is a term that comes from object-oriented
programming languages like C++, Java, Perl, and so on.
It has to do with where the data is
and identifying the data.
In small data,
you generally have well-organized
and individual data points
that are easy to locate
and they often have clear metadata
so you know where they come from.
You know what the values mean.
With big data, however,
you have many different files
in potentially many different formats,
and it can be difficult to actually locate the data points
that you're looking for
and if they are not well documented
and it's easy for things to slip through the cracks,
it may be difficult to interpret the data
and know exactly what each value means.
And the 10th difference has to do with analysis.
In small data, you can generally analyze all the data
in one procedure on one machine,
but big data, again,
because it's big,
the data may need to be broken apart.
It may need to be analyzed in several different steps
using different methods
and then trying to combine the results at the end.
All of these differences
combined to make it so big data isn't just bigger.
It's not just more data or lots of data
sometimes, people call it.
It's a qualitatively different approach
to solving problems
that requires qualitatively different methods
and that is both the challenge and the promise of big data.