- [Instructor] Let's prepare to collect scores
from your experts.
We'll start by setting up a single place
to record the scores you'll gather.
This is your working document,
and not meant to be a questionnaire.
In the future, I'll explain the workflow
for collecting the data.
Put your experts' names with the controls
they will score into your spreadsheet.
Here you can see, I have the recovery controls
from the NIST Cybersecurity Framework
listed by their unique codes in the first column called ID.
Then in the second and third columns to the right,
I have listed two experts by name
along with their titles.
Notice Bob will not score three of the controls,
so I've colored the intersecting cells black,
and will not include them in the calculation
of the summary scores.
Because I'm using the NIST Cybersecurity Framework,
the three columns to the right
are labeled Outcome, Activity, and Function,
which are the three levels of controls
defined in that framework.
Since you already know who will be
providing scores for each control,
making your score sheet should take you
about two hours unless you have
dozens of experts to interview.
This is what your data might look like
after you collect all the scores.
Notice the colors of the cells
follow along with the score key.
In Excel, you can use conditional formatting
to make that happen automatically.
Also notice the scores in the three right-hand columns
are calculated as simple averages,
also called the arithmetic mean.
Let's step through the roll up of the averages.
The Outcome Average is the mean
of Alice and Bob's scores for each outcome.
The Activity Average is the mean of
the outcomes for each activity,
and the Function Average is the mean
of all the activities.
Your next major decision is to choose
the level of data quality you need.
This will be determined by several factors.
Your organizational risk appetite,
management style, available budget and timeline,
and your internal culture.
Make sure to discuss this choice
thoroughly with your boss,
as it's crucial to have their support
when the data is analyzed,
and you bring back the top risks.
In my experience, there are three levels
of data quality to choose from.
With the good quality option,
you will briefly train your experts,
and they will submit their scores
using an online survey tool,
such as SurveyMonkey, or maybe you'll use
an internal survey tool.
When you have dozens or hundreds of experts
to work with, this is the most scalable,
and least expensive data gathering option.
It will generate actionable data
assuming your experts participate fully and sincerely,
without fear of retaliation for pointing out problems.
Some people have doubts about the usefulness
of self-reported scores,
and there is some cause for concern.
Self-reported answers may be exaggerated,
and some experts may be too embarrassed to tell the truth,
thinking it will reflect poorly on them and their work.
Also experts are inherently biased
by their feelings at the time they
fill out the questionnaire.
You can address these concerns with additional training,
arranging for their supervisors to encourage honesty,
or by choosing one of the higher data quality levels.
With the better quality option,
again you briefly train your experts,
and then interview them, either in person,
or by voice or video call.
This approach requires two data collection hours per expert.
This choice is good when you suspect
the experts are unable to participate fully and sincerely
using the online response method.
The interview allows you to watch for signs
during your interaction that your expert
is giving you biased responses.
If you suspect that's happening,
take a curious tone and ask the expert
to explain more fully why they're giving you
that particular score.
The more they talk, the closer you'll get to reality.
With the best quality approach,
you briefly train your experts,
and then interview them, either in person,
or by voice or video call.
Then, they provide evidence to justify their scores,
which you should gather as soon as possible.
This approach requires 50% to 100%
more data collection hours than
the better quality choice,
mostly for follow-up to collect the evidence.
You might also choose this option
as a readiness assessment for an upcoming audit
or standards-based certification.