Mystery Shop Sample Size and Customer Experience Variation

Mystery shop programs measure human interactions; interactions with other humans and increasingly human interactions with automated machines.  Given that humans are on one or both sides of the equation, it is not surprising that variation in the customer experience exists.

When designing a mystery shop program, a central decision is the number of shops to deploy.  This decision is dependent on a number of issues including: desired reliability, number of customer interactions, and the budgetary resources available for the program.  However, one additional and very important consideration, which frankly doesn’t get much attention, is the amount of variation expected in the customer experience to be measured.

The level of variation in the customer experience is an important consideration.  Consistent customer experience processes require less mystery shops than those with a high degree of variation.  To illustrate this, consider the following:

Assume a customer experience process is 100% consistent with zero variation from experience to experience.  Such a process would require only one shop to accurately describe the experience as a whole.  Now, consider a customer experience process with an infinite level of variation in the experience.  Such a process would require far more than one shop.   In fact, assuming an infinite level of variation, 400 shops would be required to achieve a margin of error of plus or minus five percent.

Obviously, the variation of most customer experience processes reside somewhere between perfect consistency and infinite variation. So how do managers determine the level of variation in their process?  The answer to this question will probably be more qualitative than quantitative.   Ask yourself:

• Do you have a set of standardized customer experience expectations?
• Are these expectations clearly communicated to employees?
• Other than mystery shopping, do you have any processes in place to monitor the customer experience? If so, are the results of these monitoring tools consistent from month-to-month or quarter-to-quarter?

To make it easy, I always ask new clients to give a qualitative estimate of the level of variation in their customer experience from: high, medium to low.  The answer to this question will also be considered along with the level of statistical reliability desired and budgetary resources available for the program in determining the appropriate number of shops.

So – ask yourself; how much variation can we expect in our customer experience?