Most mystery shopping programs score shops according to some scoring methodology to distill the mystery shop results down into a single number.
Scoring methodologies vary, but the most common methodology is to assign points earned for each behavior measured and divide the total points earned by the total points possible, yielding a percent of points earned relative to points possible. It is a best practice in mystery shopping to calculate the score for each business unit independently (employee, store, region, division, corporate).
Not all Behaviors are Equal
Some behaviors are more important than others. As a result, best in class mystery shop programs weight behaviors by assigning more points possible to those deemed more important. Best practices in mystery shop weighting begin by assigning weights according to management standards (behaviors deemed more important, such as certain sales or customer education behaviors), or according to their importance to a desired outcome such as purchase intent or loyalty. Service behaviors with stronger relationships to the desired outcome, identified through Key Driver Analysis, receive stronger weight. Again, see the subsequent discussion of Key Driver Analysis.
Don’t Average Averages!
It is a mistake to calculate business unit scores by averaging unit scores together (such as calculating a region’s score by averaging the individual stores or even shop scores for the region). This will only yield a mathematically correct score if all shops have exactly the same points possible, and if all business units have exactly the same number of shops. However, if the shop has any skip logic, where some questions are only answered if specific conditions exist, different shops will have different points possible, and it is a mistake to average them together. Averaging them together gives shops with skipped questions disproportionate weight. Rather, points earned should be divided by points possible for each business unit independently. Just remember – don’t average averages!
What Is A Good Score?
This is perhaps the most common question asked by mystery shop clients – one for which there is no simple answer. It amazes me how many mystery shop providers I’ve heard pull a number out of the air, say 90%, and quote that as the benchmark with no thought given to the context of the question. The fact of the matter is much more complex. Context is key. What constitutes a good score varies dramatically from client-to-client, program-to-program based on the specifics of the evaluation. One program may be an easy evaluation, measuring easy behaviors, where a score must be near perfect to be considered “good” – others may be difficult evaluations measuring more difficult behaviors, in this case a good score will be well below perfect. The best practice in determining what constitutes a good mystery shop score is to consider the distribution of your shop scores as a whole, determine the percentile rank of each shop (the proportion of shops that fall below a given score), and set an appropriate cut off point. For example, if management decides the 60th percentile is an appropriate standard (6 out of 10 shops are below it), and a shop score of 86% is in the 60th percentile, then a shop score of 86% is a “good” shop score.
Work Toward a Distribution
When all is said and done, the product of a best in class mystery shop scoring methodology will produce a distribution of shop scores, particularly on the low end of the distribution. Mystery shop programs with tight distributions around the average shop score offer little opportunity to identify areas for improvement. All the shops end up being very similar to each other, making it difficult to identify problem areas and improve employee behaviors. Distributions with scores skewed to the low end, make it much easier to identify poor shops and offer opportunities for improvement via employee coaching. If questionnaire design and scoring create scores with tight distributions, consider a redesign.
Keep it Simple
Often mystery shopping programs are designed by committee which can result in an overly complicated and cumbersome program. Unrealistic scenarios combined with long, overly complex questionnaires result in frustration for mystery shopper, mystery shop provider and the end client. In such cases the likelihood of shopper exposure is increased and the accuracy of the observations suffers. Keep it simple – simpler designs work better and provide more value.
Anticipate the Analysis
Finally, identify what specific desired outcome you want from the customer as a result of the experience. Do you want the customer to purchase something? Do you want them return for another purchase? The answer to this question will anticipate the analysis and build in mechanisms for Key Driver Analysis to identify which behaviors are more important in driving this desired outcome – behaviors that matter.
What, How & Why
A best practice in mystery shop questionnaire design is to include observations of objective behaviors, subjective impressions and comments. Each of these serves a specific purpose in identifying the service behaviors that matter – behaviors which drive profitability. Together, these three elements of questionnaire design reveal the “what”, “how” and “why” of the customer experience.
Objective Behaviors: Observations of objective behaviors form the backbone of best in class mystery shops. These observations identify what specific sales and service behaviors were observed. Mystery shopping is primarily an observational form of research, and as such, a best practice in mystery shopping is to focus on observations of specific objective and observable behaviors. These objective observations serve two purposes: First, they measure and motivate expected sales and service behaviors. Second, they serve as a foundation for Key Driver Analysis, where the other two subjective elements of the questionnaire are used to determine the relationship between employee behaviors and a desired outcome, such as purchase intent or customer loyalty.
Subjective Impressions: Subjective impressions are primarily captured through scientifically designed and strategically selected rating scales. These questions reveal how the shopper felt about the experience. They add both a quantitative and qualitative perspective to the objective behaviors observed and provide a basis for interpretation of not only individual shops, but also an analytical means to determine the relationship between each service behavior and the desired outcome. We will explore this in more detail in a discussion of Key Driver Analysis.
Subjective Comments: Beyond measuring what behaviors were observed and how the shopper felt about the experience, open-ended comments capture why shoppers felt the way they did about the experience. While objective behaviors are the backbone of the shop, many of Kinēsis’ clients consider these comments the heart of the shop, providing a qualitative texture to understand specifically what the shopper felt about the experience. They not only serve as a framework for understanding each shop individually, but provide raw material for content analysis to determine key qualitative key drivers of the desired outcome (purchase intent and customer loyalty).
Anticipate the Analysis
A best practice in mystery shopping program design is to anticipate the analysis. Together, these three design elements provide input into Key Driver Analysis techniques to identify key sales and service drivers of purchase intent and loyalty – behaviors that matter.
The first step in building a best in class mystery shop program is defining your objectives. Defining research objectives prior to making any other decisions about the program will ensure your program starts right, stays on track, on budget, and produces positive results. The mystery shop best practice in defining objectives for a program is a fairly simple process. First, generate a list of specific behavioral expectations you have of your employees.
What Do You Expect?
Ask yourself what sales and service behaviors you expect from employees. This list of behaviors is going to vary broadly from industry-to-industry, channel-to-channel, and brand-to-brand. Some of the questions you might ask yourself look like this:
- What specific service behaviors do we expect?
- When greeting a customer, what specific behaviors do we expect from staff?
- When meeting with customers after the greeting, what specific behaviors do we expect?
- If a phone interaction, what specific hold/transfer procedures do we expect (for example asking to be placed on hold, informing customer of the destination of the transfer)?
- Are there specific profiling questions we expect to be asked? – If so, what are they?
- What closing behaviors do you expect? How do you want employees to ask for the business?
- At the conclusion of the interaction, how do you want the employee to conclude the conversation or say goodbye?
- Are there specific follow-up behaviors that you expect, such as getting contact information, suggesting another appointment, or offering to call the customer?
- What other specific behaviors do we expect?
Map Expectations to the Shop Questionnaire
Once you have developed a list of the specific behaviors you expect, the next step is to map each of your behavioral expectations to a question or set of questions on the mystery shop questionnaire. Remember. these behaviors must be specific, objective and observable.
“You can expect what you inspect.”
This management philosophy is as true today as it was 50 years ago when W. Edwards Deming coined it. Managers of the customer experience have several tools available to them to inspect or monitor the customer experience. However, when it comes to monitoring employee behaviors – service and sales behaviors that drive customer experience success – no tool is better suited for that objective than mystery shopping.
Mystery shopping programs, when administered in accordance with certain mystery shopping best practices, not only test for the presence of service behaviors, but identify which sales and service behaviors matter most. These behaviors – the ones that matter most – are those which drive either purchase intent or customer loyalty. Mystery shopping provides a vehicle to not only measure but motivate these key behaviors. Central to the success of any customer experience initiative is understanding and adhering to certain best practices. This white paper advances several mystery shopping best practices.
Central to monitoring the customer experience is an understanding of the brand-customer interface. At the center of the customer experience are the various channels which form the interface between the customer and the brand. Together, these channels define the brand more than any external messaging. Different research tools have different research purposes. Some are designed to monitor the customer experience from the customer side of this interface; others, like mystery shopping monitor it from the brand side of the interface. Best in class mystery shopping programs focus on the behavioral side of the equation, answering the question: are our employees exhibiting appropriate sales and service behaviors and are these behaviors the ones that matter?
Types of Mystery Shopping
Before discussing best practices in mystery shopping, it is instructive to consider how brands use mystery shopping to measure and motivate the desired customer experience. Just about any channel in the brand-customer interface can be shopped at any point in the customer journey.
Some of the types of shops include:
In-Person: While distribution channels shift to more self-administered on-line channels, in many industries the in-person channel continues to be the embodiment of the brand – central to a multichannel strategy. This role will put new pressures on store personnel as brand advocates. In-person mystery shopping evaluates and motivates sales and service behaviors as part of this role.
Contact Center: Contact center mystery shopping provides managers a unique opportunity to evaluate the customer experience using predetermined scenarios. Most contact centers employ call monitoring to evaluate agent performance. Best in class mystery shopping programs augment call monitoring by giving managers a tool to present specific scenarios to agents to test their performance.
Web/Mobile Shops: Across many industries, self-administered channels are increasingly becoming key to opening and deepening the customer relationship. Mystery shopping website and mobile channels provides managers tools to test ease of use, navigation and the overall customer experience of online and mobile channels.
Life Cycle Shops: Life cycle mystery shops are designed to evaluate the customer experience through the entire customer journey across a variety of delivery channels, and a spectrum of transactions, over an extended period of time.