- [Instructor] I love old books.
I love the feel, I love the look.
I love the formal typesetting,
the page-long paragraphs, the formal language,
the complete absence of footnotes and references.
But they're musty, they're fragile,
and really, they're risky to lend out.
You don't know if they're going to come back intact.
They're hardly how I would want to store my operational data.
And so after the era of books as data
we got to go to digital data sets.
But so often the data did, and frequently still does,
exist as a local file on one person's own computer
where it may not be able to do much good,
except for that one person's desktop.
Organizations have always been trying to find ways
to get the message out.
And several developments in data storage have helped
with that goal and helped deal with the massive quantities
of data that are part and parcel of big data.
The first solution, the first step, in this direction
was the creation of data warehouses.
A data warehouse is a unified place
to keep an organization's data sets.
So think of it as a server that has all the data sets on it.
Now there're several advantages to data warehouses.
Number one, the data
is typically well-structured and curated.
And it's well-structured in part because data warehouses
often could only take one kind of data,
and it would usually be a relational database
with the rows and columns of tables.
And in addition to that,
a data warehouse may contain discrete organizational units.
So, for instance, accounting might have their own part
of the warehouse, and sales might have their own part
of the warehouse.
And this was one way to organize things.
Now these were very popular, say, for instance,
back in the '70s and '80s.
But there was a problem in that you ran the risk
of ending up with a data dead zone,
an abandoned data warehouse.
The reliance on structured data worked well
for a lot of purposes, but the explosion
of big data brought in enormous amounts
of unstructured and semi-structured data
that didn't fit well in the systems.
In addition, warehouses sometimes perpetuated the existence
of organizational silos, or these walls between units,
that didn't communicate well.
And truthfully, that undermined some of the promise
of the data revolution.
And so for many organizations the data warehouse kind of came,
it solved some problems, but then it seemed
to bring along some potential problems of its own.
That led to the development of our next solution
and next metaphor, the data lake.
This is a data storage
that holds data that is structured,
and semi-structured, and unstructured.
Any kind of data can go in there, so it's enormously flexible.
And it's designed to do away with data silos
that all of the organization's data
from every unit can go in there.
And so it's potentially available to whoever needs it.
And really, it's the existence of data lakes
through things like Hadoop
that made the big data revolution possible.
But just as with data warehouses,
there are some potential risks with data lakes,
and that is you can end up with a data swamp.
If the data lake isn't well maintained,
if mistakes in the data aren't fixed,
if adequate metadata isn't provided,
if multiple, but different, versions
of the same data are coexisting,
if formats aren't adapted to the software used,
you run the risk of getting this swamp-like situation,
which can be a dangerous place to get lost,
and really becomes a place where data go to die.
And so people came to recognize there were some limitations
to this approach as well, which leads us
to the current preferred solution, which is the cloud.
The idea here is that a data cloud is something
that contains data, like a data lake,
that is structured, semi-structured, and unstructured,
so it's very flexible in terms of the contents.
But it doesn't rely on a local server.
It doesn't rely on local maintenance
of the equipment, because it's off somewhere else.
I mean, this is when you use things
like Amazon Web Services, better known as AWS,
or Microsoft Azure, or Google Cloud, among others.
And also, because it's fully online, it's more accessible.
And truthfully, it's more adaptable.
You can get more space immediately.
You can get more computing resources immediately as you go.
And so there's some huge advantages
to storing and working with big data in the cloud.
But, as with every other solution, there are some risks,
and I'm going to call it data dissipation.
The major risks of storing data in the cloud
have to do, first, with security,
where the organization may not be keeping track
of who has access to the resources,
or it might be susceptible to hacking.
And second, there's cost containment.
When things are infinitely expandable at any time,
you may not be keeping track of the costs
that you're racking up.
And third, there's the remote possibility
that the services you use
might become temporarily inaccessible,
or they may disappear completely.
And so you do run the risk of your data kind of dissipating,
or not knowing what's happening there.
Because we're in the data cloud era,
it's worth pointing out a couple of variations
that have become particularly popular.
One is the multicloud, also called polynimbus.
This is using several different cloud storage
and computing providers simultaneously.
So you might use both Microsoft Azure, and Google Cloud,
and use some IBM cloud computing resources.
And the point here is you actually can integrate them
so they're in a single heterogeneous environment,
so it's as though they were in a single thing.
Now, that's one popular approach
because it lets you use the strengths
of each of these providers.
There's also hybrid cloud.
This is where you're using both a public cloud,
like AWS, and you're using a secure private cloud.
Think of a local server.
And what that lets you do
is it lets you keep a lot tighter control
on what happens here.
And it's possible also to move back and forth.
You can take things from the private cloud,
push them into the public cloud we need to expand
or if you need transparency.
But times you need extra security,
you bring it from the public to the private.
And so there're a lot of variations that can be used
to meet the demands of your organization's data computing
and the regulatory environment.
And so there're a few things to think about
when looking at these different approaches:
the data warehouse, the data lake, the data cloud,
along with the multicloud and the hybrid cloud.
Number one is accessibility.
Some of these allow more access to more people
and from different situations.
The cloud's going to be the best at that.
The data warehouse is going to be the most limited
in that particular respect.
There's also the flexibility.
What kinds of data can you put into it?
And also, can you scale up your resources as needed?
And then the third one is security.
How much control do you have over your data?
How much control do you have over your costs?
And how are you able to meet the demands
of the regulations that you're working with?
Using these three criteria, as well as things
like the cost of purchase to maintain
and the learning curve to get started
on any one of these, your organization can choose.
Do you want a data warehouse?
Do you want a data lake?
Or do you want to work in the cloud?
Or maybe there's even a situation
where you do want to keep those files on a single computer
for the tightest possible control.
But any one of these is going to allow you
to make an informed choice about how you deal
with you data, especially all the variations
that constitute big data and the promise that it has
for your organization.