Mystery Shop Sample Plans – How many shops?

Previously we examined best practices in mystery shop scoring.

Decisions regarding the number of shops are primarily driven by budgetary resources available and the level of statistical reliability required.

Reliability at Individual or Store Level

The most appropriate measure of reliability at the individual or store level is maximum possible shop distortion (MPSD).  Given that shops are snapshots of specific moments in time, it is possible for unique events to influence the outcome of any one shop.  It is possible, therefore, that the experience observed by the mystery shopper is not representative of what normally happens.  Consider the following examples: a retail location is shopped hours after it was held up, or a bank teller is shopped on the day after her child was up sick all night, or a server at a restaurant just had an extremely bad day.  In each of these cases, it is possible these external events impacted employee performance and the customer experience.

How do we know if the experience is typical or not?

Maximum possible shop distortion is the maximum influence any unique event can have on a set of shops to an individual or location.

With one shop to a given location, we do not know if it is typical or not; we only have one data point, so the MPSD is 100%.  It is possible the experience is not representative of what is typical.  With two shops, the MPSD is 50%.  If there are discrepancies within the shops, we do not know which is normal and which is the outlier.  With three shops, we now have potentially two shops to point to the outlier (MPSD 33%).  The MPSD continues to decline with each additional shop.

As this graph illustrates, maximum possible shop distortion begins to flatten out relative to the incremental program cost as we approach 3 to 4 shops per store.   This is where ROI in terms of improved reliability is maximized.

Click here for the next installment in this series: Taking Action on Mystery Shop Results.