- [Instructor] I have a friend who worked as an archivist
for a major fashion designer in New York City.
Their job was to go through the disorganized racks
and piles of clothes from several decades of runway shows,
match them with the videos of the shows
and then label and document and organize everything.
It was a monumental task that took over a year
but now the design house has a documented history
of their creative work
and in addition they can actually find
what they're looking for.
In fashion, as in life,
a little bit of structure can go a long way.
When it comes to data and structure,
there are three major rubrics or categories,
there is structured data,
semi-structured data,
and unstructured
and I'll say a couple words about each of these.
Structured data is the kind that you generally think of
when you think of data.
It's rows and columns organized and labeled.
The variables are predefined,
the last name goes here, the first name goes here,
the zip code goes here.
Spreadsheets are a perfect example of structured data,
provided that each row is a case
and each column is a variable or field.
Relational databases are also a perfect example
of structured data,
things are well-defined, you know where things are,
you know how they connect
and it makes analysis very easy.
On the other hand,
here in the data science world
we get to deal a lot with semi-structured data.
Now, this involves variables that are defined
but they're defined on a one on one basis
and marked with tags
and those tags can vary freely.
Good examples of this are HTML,
that's a Hypertext Markup Language
that creates webpages
or XML, Extensible Markup Language
and JSON, the JavaScript Object Notation,
and all of these are used extensively on the web.
And what you see here on the right
is some data that I've used in other examples
but saved not as a CSV file
'cause it's normally in a spreadsheet
but I've saved it as a JSON file,
and so you can see within the curly brackets
are each case and then in each line
it defines first what the name of that variable or field is,
like state and then it gives the value for that case,
California and so on.
Now, these have consistent names
but they can vary from one document
or one case to another.
And then finally, there's unstructured data,
this is when the variables or the fields
are not labeled or identified,
instead you have a lot information there
that you have to figure out what it is.
Free text is an example,
if you import the text of a book
or you take photos or videos or audio waves,
like we have right here,
those are extremely common forms of data
but they're not used traditionally a lot in analysis
because they're not structured, they're not labeled,
they're not processed in a way to make analysis easy.
And that gets it a little paradox here,
if we take these three versions of data
and look at their ease of analysis,
structured data is by far the best,
it's ready to go, it's all set up.
Semi-structured data takes a little bit of work
but you can import a package into your programming language
that knows how to pass it
and it adds a few extra steps
but it's not a big deal.
Unstructured data on the other hand,
requires sometimes an amazing amount
of mental and statistical gymnastics
to get the data set for analysis.
So from ease, it's structured to semi-structured,
to unstructured being the hardest.
On the other hand,
if you want to talk about the quantity of data
that is theoretically available out there in the world,
well there's more unstructured data than anything else,
especially when you consider all the social media posts,
all the books ever written, all the songs ever recorded,
all the movies ever made,
all of those are unstructured.
And there's an enormous wealth there
but it takes some work to get at it.
Semi-structured, there's a lot of data that,
for instance, is on the internet,
social media data is in certain respects semi-structured
and webpages are largely semi-structured
at least in the gross elements.
And then structured data,
the stuff that's all set up and ready to go,
well it's the easiest to work with
and it's what I turn to first,
it is probably the least common
in terms of overall quantity of data
that's theoretically available.
This means that there's a problem,
this mismatch between the ease of analysis and the quantity
and that means that basically you have to find a way
to take the unstructured data
and to put some kind of structure on it.
You can do that by either manually labeling the data,
like grading papers and giving a score to them
or taking a document and saying what it's about.
You can do it yourself,
you can hire other people to do it,
you can wait for users to label their own photographs,
or what's shown itself to be really useful these days
is to use machine learning to create and apply the labels,
this is then one of the great benefits
of deep learning URL networks.
They take messy unstructured data, like audio, video,
MRI scans, and so on,
and through a sophisticated procedure
they are able to label the data and do so at scale,
which is one of the great benefits.
So it's one way of dealing with this mismatch
between the ease of analysis
and the quantity of the data available.
But what to let you know overall
is that there are these different kinds of data
that the structured is easy and ready to go,
semi-structured is pretty common,
unstructured is an enormous untapped resource
and that's one of the major challenges of big data
and by extension, data signs and machine learning
and artificial intelligence
to take that unstructured data
and get the value and the meaning out of it.