Use the Right Research Tool: Avoid NPS with Mystery Shopping

Net Promoter Score (NPS) burst on the customer experience scene 15 years ago in a Harvard Business Review article with the confident (some might say over confident) title “The One Number You Need to Grow.”  NPS was introduced as the one survey question you need to ask in a customer survey.

Unfortunately, I’ve seen many customer experience managers include NPS in their mystery shopping programs, which is frankly a poor research practice.

The NPS methodology is relatively simple.  Ask customers a “would recommend” question, “How likely are you to recommend us to a friend, relative or colleague?”  on an 11-point scale from 0-10.

Next, segment respondents according to their responses to this would recommend question.  Respondents who answered “9” or “10” are labeled “promoters”, those who answered “7” or “8” are identified as “passive referrers”, and finally, those who answered 0-6 are labeled “detractors”.  Once this segmentation is complete, the Net Promoter Score (NPS) is calculated by subtracting the proportion of “detractors” from the proportion of “promoters.”  This yields the net promoters, the proportion of promoters after the detractors have been subtracted out.

The theory behind NPS is simple.  It is used as a proxy for customer loyalty.  Loyalty is a behavior, surveys best measure attitudes, not behaviors.  Therefore customer experience researchers need a proxy measurement for loyalty.  NPS is considered an excellent proxy for loyalty under the theory that if one is likely to put their reputation at risk by referring a brand to others, they are more likely to be loyal to the brand.  In contrast, to those who are not willing to put their reputation at risk are less likely to be loyal.

Fads in customer experience measurement come and go.  The NPS fad has been particularly stubborn.  Mostly because the theory behind it is intuitive, it is a solution to the problem of measuring loyalty within a survey, and it is simple.  I personally think it was oversold as the “one number you need to grow.”  Overselling it as the one number you need to grow doesn’t do justice to the complexities of managing the customer experience, nor does one NPS number give any direction in terms of how to improve your NPS score.  An NPS score alone is just not very actionable.

While NPS is an excellent loyalty proxy and has a lot of utility is a customer experience survey, it is not an appropriate tool to use in a mystery shopping context.  Mystery shopping is a snapshot of one experience in time, where a mystery shopper interacts with the representative of the brand.  NPS is a measure of one’s likelihood to refer the brand to others.  The problem is the likelihood to refer the brand to others is almost never the result of a snapshot in time.  Rather, it is a holistic measure of the health of the entire relationship with the brand, and as such does not work well in a mystery shop context where the measurement is of a single interaction.  As such, NPS is a measure of things unrelated to the specific experience measured in the mystery shop; things like: past-experiences, overall branding, alignment of the brand to customer expectations, etc.

Now, I understand the intent of inserting NPS in the mystery shop.  It is to identify a dependent variable from which to evaluate the efficacy of the experience.  NPS is just the wrong solution for this objective.

There is a better way.

Instead of blindly using NPS in the wrong research context, focus on your business objectives.  Ask yourself:

• What are our business objectives with respect to the experience mystery shopped?
• What do we want to accomplish?
• How do we want the customer to feel as a result of the experience?
• What do we want the customer to do as a result of the experience shopped?

Once you have determined what business objectives you want to achieve as a result of the customer experience, design a specific question to measure the influence of the customer experience on this business objective.

For example, assume your objective of the customer experience is purchase intent.  You want the customer to be more motivated to purchase after the experience than before.  Ask a purchase intent question, designed to capture the shopper’s change in purchase intent as a result of the shop.

Now, you have a true dependent variable from which to evaluate the behaviors measured in the mystery shop.  This is what we call Key Driver Analysis – identifying the behaviors which are key drivers of the desired business objective.  In the example above we want to identify key drivers of purchase intent.

I like to think of different question types and analytical techniques as tools in a tool box.  Each is important for its specific purpose, but few are universal tools which work in every context.  NPS may be a useful tool for customer experience surveys.  It is not, however, an appropriate tool for mystery shopping.