- [Instructor] Let's look at the workflow to collect
scores from your experts.
You can begin this work after you identify
your experts and your questionnaires have been created.
When gathering data using an online tool,
under the good quality option,
your next step is to prepare for
questionnaire distribution and tracking.
You'll know you're done when your experts
can remotely access and complete
the questionnaire, and you can analyze the data.
Expect a lot of upfront work,
setting up the online system,
but the payback will happen when you
don't have to travel overnight or spend
lots of hours conducting interviews.
The more experts you have to collect data from,
the greater the return on investment.
It can take between 30 and 40 hours of your time,
over a period of three or four weeks
to convert the questionnaires into
your online system, do basic testing,
and send the notifications.
You may need help from an expert
on whatever survey system you're using.
Before you distribute the survey,
recruit someone to receive a notification
and try the questionnaire.
However, when you're gathering data
by conducting interviews using
the better or best quality methods,
instead of preparing an online survey,
your next step in the workflow is different.
You start by designing the interviews.
This will take between two and three hours of your time.
You'll need to decide if you want to interview
one expert at a time, or you could also
meet with several experts from
the same work group, and generate consensus scores.
Depending on time, distance, and budget,
the interviews could be in person,
or done by conference call or video call.
Be sure to prepare enough score keys
and questionnaires so every expert
will have their own copies during the interview.
I've gone as far as to print the materials out
and send them in advance.
Instead of printing handouts,
you might want to show the information
on a monitor or projector.
That works fine in many situations,
but could be awkward during the interview
unless the experts can see the questions
and the score key at the same time,
but you may be able to do it
if you carefully plan to have the right
equipment available in the room
for every interview.
With either the online or interview-based approaches,
your next step is to prepare the expert's supervisor
for the meetings.
Ideally, this is a meeting or phone call
that will take between 15 and 30 minutes per supervisor.
You can save time if you can speak
with multiple supervisors all at once.
Ask the supervisors to tell their experts
ahead of time that the purpose of the scoring
is to find opportunities for improvement,
not to catch people doing things wrong
and get them into trouble.
Mention that the raw data gathered
will not be shared with executives,
only the summary data.
Ask the supervisors to encourage their expert
to tell the respectful truth when assigning scores.
To increase the likelihood the expert
will score accurately and truthfully,
ask their supervisors to not attend
your meeting with their expert.
If they insist on attending the interview,
ask what it would take for them to
feel comfortable not attending,
and then try to meet their request.
If they continue to insist on attending,
you should let them, or select a different expert.
After giving the supervisors a few days
to talk with their experts,
send a meeting request to each expert to gather scores.
Budget 15 minutes per expert to set up appointments.
Expect to spend up to two hours per meeting with experts
to gather scores.
The main drivers of interview time
are the number of people, and the number of questions.
Every interview is a balancing act.
Keeping an eye on the clock,
while trying to not rush through it.
If you have concerns about how it will go,
do a practice interview with a friend.
Here's the standard agenda that assumes a 90-minute meeting.
At the start of the meeting,
using words similar to those you said
to their supervisor, take 10 minutes
to briefly explain the purpose of the meeting,
then briefly explain how the data
will be used after it's collected.
Next, pass out the supporting materials
and give the experts a 20-minute training.
Show them how the score key works.
When determining a score, ask them to consider
what they have experienced with that control
in the previous six to 12 months,
and what they expect will happen
six to 12 months into the future.
Note, this approach emphasizes learning
to score as you go.
Experts may feel awkward at first using this approach.
Your job during this early phase
is to minimize your own talking,
and give them time to figure it out.
Finally, step through the questions one by one,
and collect scores over the final 60 minutes.
Read each question out loud
while the expert follows along.
After each question is read,
allow silence to enter the room,
to give the expert a chance to consider the question
and use the score key.
If the expert asks for clarifications,
do your best to help, but let the expert assign the score.
Document all scores in your spreadsheet as you go,
including any insightful comments
made by the expert.
Let's look at a few time estimates
for gathering scores from experts.
These estimates do not include the setup effort.
Using the good data quality method,
I spent 46 hours over four weeks
collecting data online from over 150
experts across the world at a $1 billion
financial services company.
On another occasion, using the better data quality method,
the chief information security officer
at a $4 billion county government spent
29 hours over five weeks interviewing 22 experts.
And finally, using the better data quality method,
I spent 12 hours over three weeks
interviewing experts at a $2 million non-profit agency.