Mystery Shop Sample Size and Customer Experience Variation

Mystery shop programs measure human interactions; interactions with other humans and increasingly human interactions with automated machines.  Given that humans are on one or both sides of the equation, it is not surprising that variation in the customer experience exists.

When designing a mystery shop program, a central decision is the number of shops to deploy.  This decision is dependent on a number of issues including: desired reliability, number of customer interactions, and the budgetary resources available for the program.  However, one additional and very important consideration, which frankly doesn’t get much attention, is the amount of variation expected in the customer experience to be measured.

The level of variation in the customer experience is an important consideration.  Consistent customer experience processes require less mystery shops than those with a high degree of variation.  To illustrate this, consider the following:

Assume a customer experience process is 100% consistent with zero variation from experience to experience.  Such a process would require only one shop to accurately describe the experience as a whole.  Now, consider a customer experience process with an infinite level of variation in the experience.  Such a process would require far more than one shop.   In fact, assuming an infinite level of variation, 400 shops would be required to achieve a margin of error of plus or minus five percent.

Obviously, the variation of most customer experience processes reside somewhere between perfect consistency and infinite variation. So how do managers determine the level of variation in their process?  The answer to this question will probably be more qualitative than quantitative.   Ask yourself:

  • Do you have a set of standardized customer experience expectations?
  • Are these expectations clearly communicated to employees?
  • Other than mystery shopping, do you have any processes in place to monitor the customer experience? If so, are the results of these monitoring tools consistent from month-to-month or quarter-to-quarter?

To make it easy, I always ask new clients to give a qualitative estimate of the level of variation in their customer experience from: high, medium to low.  The answer to this question will also be considered along with the level of statistical reliability desired and budgetary resources available for the program in determining the appropriate number of shops.

So – ask yourself; how much variation can we expect in our customer experience?

 

Mystery_Shopping_Page

 

Tags: , ,

About Eric Larse

Eric Larse is co-founder of Seattle-based Kinesis CEM, LLC, which helps clients plan and execute their customer experience strategies through the intelligent use of customer satisfaction surveys and mystery shopping, linked with training and incentive programs. Visit Kinesis at: www.kinesis-cem.com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: