- [Instructor] There are many specific practices
that individuals and organizations have found
that help them in their DevOps journey.
Here's the 10 cited by James Wickett and Ernest Mueller
in their lynda.com course DevOps Foundation.
ICS, or Incident Command System,
is a method of dealing with major incidents
that feature self-similar organizational units,
self-organizing to quickly
and effectively coordinate efforts across teams
and scale up and down and evolve
as the incident changes in scope, scale, and focus.
Adapting ICS for the DNA of your major incident management
and IT service continuity processes is kind of a no-brainer
for enterprise IT shops because ICS was born and bred
to scale massively up and down.
I've included a link to Brent Chapman's presentation,
Incident Command System for IT:
What We Can Learn from the Fire Department.
It explains how Incident Command works
in the real world of emergency services
and explains how the same process can work for IT.
The DevOps practices of putting Devs on call
for the services they create is based on the golden idea
of you dealt it, you deal with it.
The idea that Devs will take more care
of the quality of what they produce
if they'll have to suffer the pain of dealing with it later
and not just chuck it over the fence
for somebody else to test and support.
The Devs on call DevOps practice has the greatest impact
on ITIL-driven application management, service desk,
and the IT operations function.
You put Devs on call for the service they create
to create a fast feedback loop
that helps rapidly improve logging and deployment
and enables faster resolution of application issues.
Lenny Rachitsky, in his blog transparentuptime.com,
advocates for creating public status pages,
clearly communicating the status and status history
of a service and communicating promptly and clearly
when service issues arise.
This is because transparent, easily-accessible communication
during outages can actually increase customer satisfaction.
So implement public status pages for each of your services.
That's what you can do here.
Establish a status page that gets updated
when there's an issue so customers and users can know
what's happening, what's being done about it,
and, after the issue is resolved, lessons learned.
Lenny Rachitsky cites four prerequisites
for public status pages.
First, admit and own your failures.
Secondly, sound like a human.
Third, have a communication channel.
And fourth, be authentic.
With public status pages and these four prerequisites,
you can build trust with current
and prospective customers and users.
Check out statuspage.io for an example
of status pages as a service,
which, besides the push mechanism of the page,
allows users to opt in to pull communications
for status updates.
A culture of blamelessness is a hallmark of DevOps,
and blameless postmortems are its most striking feature,
balancing safety for admitting failure with accountability
and focusing postmortems on learning and taking action,
rather than placing blame.
John Allspaw, CTO at Etsy, wrote an article
called Blameless Postmortems and a Just Culture
that describes the practice,
and I've included a link here for you to learn more.
What you can do here in your ITIL-driven shop is
to engender a culture of blamelessness.
This is key.
If your culture is one of kill the messenger
or the nail that sticks up gets hammered down,
you've got to face that down
and address what behaviors get rewarded
in your organization.
If you want to adopt blameless postmortem practices
for postmortems after major incidents, problems,
and IT service continuity or disaster recovery events,
you can start with a better blameless agenda
to enforce behavior.
For example, see the agenda I lay out
in my other course on Lynda, Problem Management Techniques,
for post-implementation reviews.
Embedded teams, most typically embedding Ops engineers
on Dev team, allows for close coordination
between individuals and teams around the shared goal
of overall success of the product or service
with quick time to value, quality,
and continuous improvement.
Embedding helps bring down the barriers of separate goals.
In traditional IT shops,
Devs want to innovate and ship product,
and Ops want to keep production stable and minimize risk.
Embedding helps bring down the barrier
of these separate goals by establishing a singular goal
and providing a conduit for greater shared understanding,
communication, collaboration, context,
trust, respect, and even terminology and language.
So at this intersection,
consider embedding an Ops engineer on each Dev team,
and make the team responsible for all its own work
instead of the old way
of Dev and Ops throwing requests over the wall
into some queue for someone else to dispatch.
In larger organizations,
since you can't put everyone on the same team,
I often see this put into practice as Ops engineers
as floating consultants who can attach to teams as needed.
The cloud and DevOps are joined at the hip.
Cloud solutions give you an entirely API-driven way
to create and control infrastructure
so you can treat infrastructure as code
and move at a much higher velocity
than with legacy platforms.
How can you integrate the cloud
into your services and service management approach?
Well, here consider three different approaches.
The first is technology-driven.
Simply use the cloud for greenfield projects
and take on CD, CM,
and DevOps monitoring and logging practices
using a lean style and continuous everything
and everything is code.
The second approach applies to legacy.
It's the automate where you can,
adopt practices where they're useful approach.
Note that some legacy platforms are more modern
and amenable to this than others.
You may be surprised to find out
what's available these days.
I know I was when I heard what IBM was providing
for mainframes these days.
The third approach is to work mostly
on cultural aspects of DevOps,
bringing teams together and breaking down silos.
The ultimate form of this is the honest to goodness
DevOps team with an SRE model and embedding.
Andon cords were originally physical cords
like the cords on buses you pull
when you ask the driver to stop.
They were used by Toyota on its production line.
It empowers anyone to pull the cord
to stop the production line if they see a problem or defect
that's going to be passed downstream.
You can do the same thing
with your build pipeline for software
to stop bugs from propagating downstream.
Ideally, you'll have automated testing
that catches most issues,
but it's important to have a mechanism
like a big panic button you can push
when there's a problem with the pipeline itself,
like some weird thing happened to Jenkins
that's affecting the build pipeline itself,
or when your automated testing didn't catch a bug
that you see will break things downstream.
So consider adding Andon cords to your terminology
and key principles and models
that you're using in your ITIL-driven shop,
starting with your build pipeline.
Unnecessary dependencies are a key source
of process overhead, conflict, and outages.
In dependency injection,
sometimes it's called inversion of control,
dependencies are not known to the application,
but instead are passed to the application at runtime.
A dependency is an object that can be used,
for example a service,
and an injection is the passing of a dependency
to a dependent object, for example a client
that would use it, with the intent of decoupling objects
so that no client code must be changed simply
because an object it depends on is changed
to a different one, and also that no service breaks
due to a tightly coupled dependency.
Dependency injection is vital for a well-behaved application
in an infrastructure as code environment.
Other patterns like service discovery can be used
to obtain the same goal.
So, as applied to your ITIL-driven technology
and architecture, the thing to do here is
to relentlessly root out unnecessary dependencies
that create breaking changes
and require lots of discussion time, overhead,
and effort for teams.
I've seen unbelievable transformations
in the speed and quality of teams who've done this,
using techniques like versioned APIs,
data contracts, and dependency injection.
Blue/green deployment is supplanting rolling upgrades
as a method for releasing software code,
where two environments, blue and green,
are initially configured identically,
with one active and the other environment inactive.
New code is released to the inactive environment
where it's thoroughly tested,
then the active and inactive environments are switched.
If problems are discovered after the switch,
traffic can be directed back to the inactive configuration
that still runs the original version.
Once the new code has proven itself in production,
the team updates the code in the inactive environment
to match the active environment.
The process reverses itself
when the next software iteration is ready for release.
This technique eliminates downtime
due the application deployment and also reduces risks
and dramatically lowers the transaction cost
and time required to revert to a prior known good state.
If something unexpected happens
with your new version on the green,
you can immediately roll back the last version
by switching back to blue.
For ITIL-driven shops, implementing this technique
affects service management technology and architecture
and the release and deployment management process primarily.
To achieve high availability,
one approach is to focus on working from the bottom up,
making each component of a system
as highly available as possible
with the idea is that the sum result
will be high availability in the overall system.
Chaos Monkey, a software tool developed
by Netflix engineers and part of their Simian Army set
of tools they share, takes a different approach.
It works on the principle that the best way
to avoid major failures is to constantly fail.
This forces developers and operators creating the systems
to engineer resiliency into their services
instead of being lulled into a sense of complacency
and making the mistake of thinking
that their infrastructure is always on.
For ITIL-driven shops,
Chaos Monkey is new terminology, a new principle and model.
What I see most shops do is cherry-pick pieces
out of the Simian Army set of tools,
for example security bits for integration testing.
Also, I see teams approaching this two ways,
sitting around a table and talking through who does what
if a bomb is dropped, kind of a simulation,
or doing it live, actually dropping the bomb
and seeing how we react.
They do it, check metrics, the impact on performance,
then review the root causes of failures
in a blameless review,
then come up with agile ways to improve it,
then they do it all over again.