- [Instructor] I've been teaching in universities
for over 20 years,
but it took me more than a decade to come to grips
with one common element of that job, grading papers.
The fact is, reading research papers,
giving useful feedback,
and assigning grades takes a lot of time,
and frequently I could get overwhelmed by the effort,
the cognitive energy it took.
But if you're working in the online world, then you know
that that sort of unstructured text information
can come in much, much faster than even student papers.
Dealing with that fire hose
of feedback is a huge challenge for anyone,
and that's the entire point of text analytics in big data.
Now, there are a few major challenges
that prompt all of this.
Number one, text is unstructured data.
It's there.
It's in sentences and paragraphs and words.
It's not broken down
into this is a variable, this is a topic.
Also, you can have text from multiple sources.
You can have comments and reviews online.
You can have people forwarding tweets and adding posts.
And then you might also want to look at the connections
between the people and the text that they put together.
That can give you some very rich context
on what's going on there.
So this is hard to do,
but there are some major benefits to this.
Number one is that you're getting unprompted data.
In the research world, if you can only get data
when you ask people the question
and they know you're asking,
you don't necessarily get to know their unfiltered thoughts.
And you do get to know that through social media.
You also frequently get the metadata.
So when a person posts a tweet,
they don't just have the text.
You get to know where they were, the time of day.
You can look up the weather.
You can see how many connections there were,
how many times it got liked.
You've got a lot information there
that's potentially available
to help you interpret that text.
And really more than anything
when you start dealing with unstructured text,
you have a massive quantity of data available to you.
Now, there are a few general steps in text analytics.
Number one is to parse the text.
It's like taking the newspaper
and cutting it up into little bits and pieces.
You split it up into words.
Maybe you remove the punctuation.
You put it all together into this list.
And then at the most basic level,
you count the words and the phrases,
their absolute frequency or their relative frequency.
And then using that data, you are then able
to do things like model the topics.
How often does this set of words having to do
with this particular topic come up?
Are people generally positive or negative about that?
Although that ties into sentiment analysis,
which we'll talk about elsewhere.
But you can look at some great examples of this.
For instance, back in the early '60s,
Frederick Mosteller and David Wallace pioneered
the statistical analysis of text
in a project to determine the authorship
of several disputed Federalist Papers from the 1700s
which led to the founding of the United States.
They did this by looking at how common particular words
and punctuation were in the papers
that they knew who wrote them
and used it to compare it with the other ones.
Now, more recently, Tahir Hemphill and the Rap Research Lab
created the Hip Hop Word Count,
one of my very favorite free text projects.
And that allowed them to explore the ways that concepts
as revealed in hip hop music evolve,
both across time and across geography.
It's a fascinating project that builds on the idea
of text analytics using these very general principles
within a big data environment.
And using any of these general principles,
you can start getting some of the great insights analysis
from your unprompted, unstructured,
and truly massive troves of data that are available to you
through the analysis of unstructured text.