- [Narrator] Salmon can be farmed
or they can be caught wild.
But either way it takes a fair amount of work
before they are turned into this.
Everybody knows that food prep
is an important although time consuming
and frequently tedious part of cooking.
There is a similar principle in any big data project.
The rule of thumb is about 80 percent of the time
on a big data project is spent preparing the data.
And that's been my own experience.
Now there are several reasons
why this may be the case.
It includes things like how is the data entered?
If you're using wild caught data,
meaning data that you found out there in the world
and that maybe was entered with free text.
You have to look at things like place names.
Here are four different ways of indicating California.
You can write it out, you can use various abbreviations
and the inclusion of a period.
At least by default marks it as a separate answer
than the one without a period.
Or when people are putting in dates.
Here are four different ways of writing the same dates.
Now humans looking at these know
that they refer to the same thing.
But a computer will read them by default as different things
and you have to do a little bit of clean up work
to make sure that the dates are all read the same way
and formatted the same way.
Or phone numbers.
Again, the same phone number in four different ways.
You know for instance that if you have these
in a spreadsheet
they're going to sort separately.
But you can reformat them
so it knows that they are all phone numbers,
it does them appropriately.
And this kind of clean up work,
this becomes an important part of getting things ready
for your big data project.
Then they're are other issues like units of measurement.
If you ask someone's weight and they put 150
or 68 or 10.7
well, 150 is pounds, 68 kilograms is the same weight
and 10.7 stone
if you're dealing with someone from the United Kingdom.
That also is 150 pounds.
So several different ways of measuring these.
And we all know about examples where, for instance,
NASA has not adequately converted from one unit to another
and space probes have crashed as a result of that.
Then there are errors.
If someone puts in 2.7 children, that's not possible.
You can't have fractional children.
You can have an average that's fractional
or 270 children, too many.
Or negative 27, you can't have a negative number
on a counted value like this.
And then of course missing data.
In some data sets, an eight might indicate
a missing value.
`A negative 999 might indicate a missing or a period
or a simply empty space.
And there are different kinds of missingness.
You can have not a number, not available,
missing completely or random.
All of them mean different things.
And when you're preparing your data,
you have to go through these and be thorough
and getting them all in the same format
with the same meaning and interpretation
all the way through.
And of course that's to say nothing of the amount
of processing required for dealing with unstructured data
like cell phone videos.
And it gets to an interesting question.
Where is the return on investment
and where's the balance point?
Should you clean wild data?
Or should you gather new data that is in the structure
and with the encodings that you want?
So for instance, here's a microscope image
of cancer cells.
And if you're dealing things like health records
the project can be a nightmare.
Which is why companies like
for instance recursion pharmaceuticals
who do drug development
where they find applications of existing drugs
to orphan diseases.
They figured out that it was cheaper, faster, and easier
to develop their own technology
to gather and analyze data
from half a million experiments per week
with the goal is to hitting a million per week.
They did the math and realized
that gathering their own data
with their own propriety methods
so they could then feed it into their algorithms
for analyzing the data
would be the best way to reach their goals
as opposed to trying to clean the wild caught data.
And so, it becomes a trade off
and depending on your exact circumstances,
you might be able to use the wild data.
You may need to gather new custom bespoke data.
But no matter how you decide to go about it,
you do need to give yourself adequate time
for the demanding work
of preparing data
in your big data project.
Remember that 80/20 rule.
Again about 80 percent of the time on a project
is spent getting ready for it.
20 percent for analysis.
You can find ways to facilitate that,
to streamline that
and get to the meaning and the insight
that you're looking for faster and more efficiently.