- In order to collect reliable data from experts,
you'll need a consistent approach
that guides them to translate their observations
into a numeric score.
We'll base our approach
on the zero through 10 scoring system
we previously learned about,
and we'll create a score key for the experts to use.
We'll also create a questionnaire for each expert
in advance of the data collection.
Here's how you create a score key.
For each numeric score, zero through 10,
prepare a statement or two that describes
how well any given control is actually performed.
The number we get from each expert
is called the actual score,
and we record each one in our spreadsheet.
Actual scores are determined by an expert
based on how well the organization has performed the control
over the past and anticipated future six to 12 months.
Here's what my score key looks like
at the four major scoring points.
Remember that I'm using NIST Cybersecurity Framework,
so my lowest level controls are called outcomes.
If the expert agrees with this statement,
the outcome rarely or never happens when needed,
then the score is zero.
What if the expert agrees with this statement?
The outcome happens consistently
with some minor flaws from time to time.
Re-work sometimes happens but is uncommon.
Then the score is five.
An eight means the expert believes
the outcome happens consistently
with great effectiveness and high quality.
And a 10 means the outcome happens
at excessive financial cost to no benefit.
Of course, experts are welcome
to choose a score between these numbers.
Note there are two other possible responses,
unknown and not applicable.
Although based on how carefully
we prepared for the interview,
these responses should be rare.
Once your score key is ready,
prepare one questionnaire for each expert.
Start by turning each control into a question.
Start the question with How Well.
Let me show you how this works.
Here's an original control
from the NIST Cybersecurity Framework.
The development and testing environments
are separate from the production environment.
Now, here's the control written as a question.
How well are the development and testing environments
separated from the production environment?
Here's what a questionnaire for an expert would look like.
I've put in one control from each of the five main functions
of the NIST Cybersecurity Framework.
Notice I've color-coded the functions
without using red, yellow, or green.
Since those colors have implied meaning,
it would likely cause confusion
unless those colors were associated with a score.
Would you show this version to an expert
in a face-to-face interview,
or would you dress it up a bit?
That's entirely up to you.