- [Narrator] Jazz music is inseparably connected
with improvisation.
With a small combo
it's possible to have just the outlines of the tune
and then create the whole thing on the go.
When it works well, it can be an extraordinary experience.
But once you have more than a few musicians,
say the 15 to 20 that are in a big band,
you can still have improvised solos
but everything else usually has to be composed in advance
to work well, to coordinate,
and make a wonderful experience.
There's an analogy between this,
between improvisation and composition,
in music and in data.
In jazz, the improvisation,
the coming up in the stream of the moment
versus the composition where the work has to be done
ahead of time,
and you got to put a bow on it before you move on,
that's a lot like in data, what is called stream processing.
Where you're processing the data as it comes in,
live, in real time,
versus batch processing where you take data that's been
wrapped up and put a bow on it,
it's in a data set,
it's set and you run with it.
Now let's take a quick look at this.
Batch processing is data at rest,
meaning it's static,
it's stationary.
Now I'm not trying to talk it down.
This is the kind of data that I work with.
It's data that's in a data set
and that's what it is.
And you can analyze that data
in as a much detail as you desire.
You have time to go back and forth with it.
You can look for patterns.
You can find groups.
You can make predictions of, based on rather sophisticated
and time consuming models.
All of those are possibilities with standard, batch data.
Create the data,
put it together,
say it's done,
then do the analysis and do your work.
Compare that to stream.
Instead of having your baskets of apples,
they are now on a conveyor belt.
The data is constantly arriving.
You may not even keep the old data.
Maybe you just have a window of data that you're saving.
Because maybe you're just looking for changes over
a short period of time.
Or you're looking for significant anomalies that might,
for instance, trigger a problem with the system.
A potential break down with the machine,
an attempt at fraud,
as something that triggers the next actions
that result from your analysis.
Now I do want to make one important distinction.
Everybody likes to have an organization that operates
efficiently and responds well to changing situations.
That's one reason why Agile development
and management are so popular.
And so there may be an automatic preference
for a data approach that seems
speedy and responsive and maybe agile too.
But the thing to remember is that
stream processing isn't better,
it's not a new and improved version of batch processing.
It's a different process.
Streaming data and streaming analysis
often relies on a different infrastructure.
You use different kinds of storage systems.
You have different kinds of databases.
You have different ways of incorporating that data.
It often uses different methods.
You can't do an elaborate neural network
on streaming data.
You often to simplify the model
depending on the kind of data
and the volume that you have coming in.
And most significantly,
stream processing has different
goals than batch processing.
Again stream processing is designed to look for
very quick trends
or immediate anomalies.
Something that requires immediate action.
Something where the business simply can't survive
without that moment to moment responsiveness.
Batch processing is designed to allow you to
look at things in more depth.
And that is the traditional method of working with data.
And so my point here is simply to tell you that these
two approaches exist.
They're two different paths.
And depending on the purposes of your organization
you will want one over the other.
But generally, it's important to remember
that they serve different goals,
use different methods.
And so it's a decision that you need to consult with
the people on your data science team
and when you're doing your big data projects
you need to think about it carefully
and see how well that matches up
with the goals of your organization.