- [Narrator] My friend is a choregrapher
and they've got a box full of early work
recorded on Super Eight, on Betamax,
and a whole bunch of other orphan video formats.
One of these days they'll get them transferred to digital,
but until then, these videos constitute
a sort of data dead zone.
There's data there, it's in there hands,
they just can't do anything with it.
It's actually an example of data governance gone wrong.
Data governance is a lot like taking care of
your dance videos or maybe your garden.
In a garden there's a lot of work to do
to make sure your plants are healthy and happy,
and the same holds true for taking care of your data.
There are four major elements of data governance.
They concern the availability of data,
the usability, the integrity of that data,
and the security, and I'll talk a little bit
about each of these separately.
But I just want to mention, these things are hard enough to do
when you have only one machine to take care of.
I think all of us have had an example
of losing the data file, having it be gone for good,
or trying to open a file and have it be corrupted.
Well it's exponentially more complicated
when you have big data, for several reasons.
Number one, there's more people,
there's more hands in the data.
There's more people working
who may not be communicating well
and that dramatically increases the risk
of working at cross-purposes.
There are many more machines, instead of a single laptop,
you've got a server or collections of servers,
or who knows what out there in the cloud.
And so you have a lot of different hardware
and potentially different software involved.
And then you have more formats for your data
'cause big data necessarily involves
many different kinds of data, think about the variability.
You have a lot of different formats,
and you may be working
in many different programming languages
'cause they work well for different things.
All of these multiply the complexity of data governance
and make that project much more important and challenging
when working with big data.
Let's talk about the availability first.
The question is, where is the data, can you find it?
Is it connected to your current systems,
or is it sitting on a hard drive somewhere
that you don't where it is?
Is the data restricted, can only certain people
get access to it?
As important as computer security is,
the moment you don't have the password
to the data that you need,
you know you're facing an important availability issue.
Second is usability.
What format is used for storing the data?
Is it on a floppy disk back in your backpack?
What language is used for processing the data?
There's actually a fair amount of work
still done in Fortran,
but you're going to have to work a little harder
to find somebody who's very comfortable working in that.
Or is metadata available, is there something
that describes where the data comes from,
how it was collected, who the sample was,
what the variables mean and what the values mean?
These are all critical questions
to make the data usable and interpretable
within the context of your project.
Then there's integrity.
Are the data files free from corruption?
Anybody who's had to deal with this for a single file
knows how difficult it can be when you get muck.
If it happens to an entire server
you got a extraordinarily huge problem.
Is the code that's used to process and analyze the data,
is the code complete and is it functional?
And you have to worry about whether, for instance,
the code has updated between now and then,
which is why, for instance, it's common practice
to include the usable copies of the packages
that you used when you're doing the programming.
So those update a lot,
this way you always have the version that you know worked.
And then are the relations intact?
And by that I mean the connections
between the different tables and your data.
You have to have a way of referring between them
to make a complete data set.
Are those still valid?
And then finally there is the issue of security.
Are the data and the code safe from hackers?
Remember, data's enormously valuable,
and it's a target for anybody
who's looking to get a little bit of extra stuff for free.
Or in a big data project,
because there's so many people involved
you have to be sure that the data and the code
are safe from accidental changes by unauthorized users.
I think about, for instance, just my genealogical data.
I log in there, somebody's made a change,
they went back to the same wrong thing
that got corrected and you got this big problem.
Think about what happens on Wikipedia,
the little editing wars.
But people sometimes log in and if they have access,
they might make a change without realizing
how much impact they're having on other major projects.
And then the last one, to get back to our Fortran issue,
are they future-proof?
Can you store the data, can you store the code,
in such a way that you know it will be usable
at least a few years from now, maybe not forever,
but at least a few years?
So for instance, if possible,
don't store your data in a proprietary format
which requires a license and requires the company
that made it to update things.
If possible, store it in an open format.
CSV files are nice, they don't keep the metadata
so you have other options for that.
But something that you know will be around
for at least a few years,
that way you can guarantee at least the short-term longevity
of your data and the whole thing makes it easier to manage
and to govern the data that is using
your data science projects over time,
so you can get both more insight and more meaning,
and truthfully, more value, too, out of your project.