Not All Customer Experience Variation is Equal: Use Control Charts to Identify Actual Changes in the Customer Experience

Variability in customer experience scores is common and normal. Be it a survey of customers, mystery shops, social listening or other customer experience measurement, a certain amount of random variation in the data is normal. As a result, managers need a means of interpreting any variation in their customer experience measurement to evaluate if the customer experience is truly changing, or if the variation they are seeing is simply random.

One solution to this need is control charts. Control charts are a statistical tool commonly used in Six Sigma programs to measure variation. They track customer experience measurements within upper and lower quality control limits. When measurements fall outside either limit, the trend indicates an actual change in the customer experience rather than just random variation.

To illustrate this concept, consider the following example of mystery shop results:

In this example the general trend of the mystery shop scores is up, however, from month to month there is a bit of variation.  Managers of this customer experience need to know if July was a particularly bad month, conversely, is the improved performance of in October and November something to be excited about.  Does it represent a true change in the customer experience?

To answer these questions, there are two more pieces of information we need to know beyond the average mystery shop scores: the sample size or count of shops for each month and the standard deviation in shop scores for each month.

The following table adds these two additional pieces of information into our example:

Now, in order to determine if the variation in shops scores is significant or not, we need to calculate upper and lower quality control limits, where any variation above or below these limits is significant, reflecting an actual change in the customer experience.

The upper and lower quality control limits (UCL and LCL, respectively), at a 95% confidence level, are calculated according to the following formulas:

Where:

x = Grand Mean of the score

n = Mean sample size (number of shops)

SD = Mean standard deviation

Applying these equations to the data in the above table, produces the following control chart, where the upper and lower quality control limits are depicted in red.

This control chart tells us that, not only is the general trend of the mystery shop scores positive, and that November’s performance has improved above the upper control limit, but it also reveals that something unusual happened in July, where performance slipped below the lower control limit. Maybe employee turnover caused the decrease, or something external such as a weather event was the cause, but we know with 95% confidence the attributes measured in July were less present relative to the other months.  All other variation outside of November or July is not large enough to be considered statistically significant.

So…what this control chart gives managers is a meaningful way to determine if any variation in their customer experience measurement reflects an actual change in the experience as opposed to random variation or chance.

In the next post, we will look to the causes of this variation.

Next post:

Not All Customer Experience Variation is Equal: Common Cause vs. Special Cause Variation