- [Instructor] And now a little aside to take
a finer look at monitoring.
When building deployment systems,
I try to distinguish between monitoring and forensics,
also called logging.
Monitoring tends to be about right now.
Is the system up?
Did I just break that?
And forensics tends to be about what just happened.
What was that request?
Where's this bug?
Monitoring tends to be aggregated.
I like systems that will tell me,
"What is the error rate right now?"
What is the distribution of latencies?
What is the slowest request in the last 10 minutes?
Things like that.
Whereas forensics tends to be detailed.
Do you want to store each individual thing
so that when you discover that there was a bug,
you can go back and say, "Hmm, what was the request
"that caused this bug?
"Ah, it was this request."
And then you look up the request
and the response and maybe if you're lucky,
you've got a system where you can feed the request
back in as an input and watch it flow
through the system and debug.
Forensics and logging are extremely useful for debugging.
And monitoring systems are really useful for operations
and figuring out what's going on with the system right now.
It's amazing how long 10 minutes can feel
when I'm waiting for the AWS Cloud Watch metrics update
for load balance or just to determine,
"Have I fixed it?"
So I like monitoring for alerts.
I want systems to come and tell me
when something's wrong.
And most hosting providers include
such systems in their offerings.
You just have to get the time to use them.
Logging is also very important.
In fact, the cloud formation stack we used
in this previous example includes logging.
It's there and ready to be looked at
as your system grows.
Logging is tremendously important too
because debugging without it can be a bit of a chore.
When you're trying to look at automatic monitoring system
and decide where do you put an alert.
At what point should your system send an email
and tell the administrator hey you better wake up
and fix this?
It's useful to start asking questions
and narrow them down like, "Did a release take
"the site down?"
Probably, but how do you explain that to a computer?
Did the recent release make the error rate worse?
Now that's getting closer to a question we can program.
How 'about did the error rate increase by more than 50%
within 10 minutes of any release?
Now that's something we can make an alert for.
And there's always the ever question caveat of did
the latency increase because of a failure
or did someone just release a new feature
that is slow but exactly the speed it needs to be.
It's always useful to include enough margin
in these systems to keep the number
of false positives reasonable.
When your system does detect an alert,
what are the options?
Well, the most common option is call a human.
Humans are good at stepping in and deciding what's
a false alarm and what's not.
If you have a really solid system
that doesn't wake you up in the middle
of the night unnecessarily too often,
you might consider tying it in Jenkins
through to a revert job.
That revert job can be a roll forward revert job
that simply runs the deployer again
on the thing tagged one back from the release.
So in other words, roll back by just re-deploying
the previous release.
Or roll backwards based systems where you keep
the old one running, then you just reactivate
the old servers and then deactivate the new servers.
This has the advantage of letting you flip back and forth,
but it can be a bit harder to build too.
And like all such things, this really is
a personal decision.
It depends on your environment, what you're working on,
how much time you have to devote to the effort,
and whether or not you need such systems.