A Guide to QA Calibration: Ca-li-brate Good Times, Come On

Regular conversation reviews are an excellent method for providing consistent, actionable feedback to your support reps and impurrroving your support quality.

However, this assumes that reviewers are aligned and consistent with your company’s quality standards.

But what happens when reviewers are misaligned, inconsistent, or biased in their grading criteria

• Your quality metrics become skewed,
• Your support reps become frustrated,
• There is no longer one true path to Qualityville.

That’s where support QA calibration can get you back on track. 

What is QA calibration? 

Customer service QA calibration is the process where internal quality reviewers align their rating techniques to ensure that support reps receive consistent feedback from all reviewers.

In essence, QA calibrations help reviewers synchronize their assessments, provide uniform feedback to support reps, and eliminate bias from quality ratings. The goal is to ensure everyone is on the same page. Support reps should receive the same quality of feedback regardless of who reviews their customer interactions.

Typically, the aspects that require quality assurance calibration include:

• Rating scale: Ensure all reviewers understand the different ratings consistently. The larger the rating scale, the more crucial the calibration.
• Failed vs. non-applicable cases: When a ticket is handled correctly but a specific aspect of the conversation is missing, reviewers might rate it differently.
• Free-form feedback (additional comments): While tricky to calibrate, reviewers should agree on the “length” of feedback, feedback techniques, and the overall style and tone of the comments.

Now that we’ve covered the definition of calibration in quality assurance, let’s start setting up your own QA calibration process.

Plan your calibration sessions

Customer service QA calibration involves comparing how different reviewers evaluate the same support ticket during a calibration session.

Imagine a group of referees preparing for the World Cup. The players (support reps) expect consistent, fair, and unbiased refereeing. The referees (reviewers) have guidelines and rulebooks (scorecards) to follow.

Before the World Cup, referees convene to discuss rule changes or delve into particular interpretations of rules. These discussions ensure the referees view the rules uniformly and deliver consistent team performance.

QA calibration sessions aim to achieve this consistency for your reviewers.

2. Review and discuss together 

Calibrating feedback collectively with all reviewers is ideal for discussing and (re)defining quality standards in your support team. In this setup, the team reads customer conversations together for the first time and collaboratively decides how to score them. If conflicts or disagreements arise, the leader of the QA program typically makes the final decision.

➕ Pro: This strategy removes the feeling of being graded—and the possible stress that this can cause—from the sessions. By allowing reviewers to discuss tickets together, you strengthen the common understanding of your quality criteria among your reviewers.

➖ Con: You will not be able to measure the current discrepancies in your reviewers’ work. Without an understanding of how consistent and unbiased your reviewers are, it might be difficult to address and resolve these differences.

3. Review together with support reps

Invite your support reps to meetings where you discuss the best ways to handle specific situations. This engages them in the quality process, helping them better understand quality expectations and view customer interactions from another perspective. It also allows support reps to provide additional context when reviewers disagree on the final rating.

➕ Pro: This is the most transparent way to handle quality calibrations. Discussing review scores together allows support reps and reviewers to learn from each other and aligns the entire team around the same goals and QA standards.

➖ Con: You won’t have your reviewers’ original scores to compare to the agreed benchmark. Additionally, some support reps may feel uncomfortable defending their actions in front of a room of reviewers.

Before inviting your QA team to a calibration session, consider the following:

• How many conversations to calibrate? The ideal number varies based on factors like ticket complexity, the number of reviewers, and the time allocated for the session. Start with five conversations in your first session and adjust as needed for future sessions.
• How often will you hold your sessions? Monthly sessions are typically ideal for most teams. However, if you have more reviewers, you might need to calibrate more frequently.
• What is your calibration goal? Setting a clear goal for your sessions helps keep discussions on track. Whether you aim to update rating scales, discuss internal processes around escalations, or align reviewers with QA standards, having a specific objective prevents the discussion from veering off course.

Choose a facilitator to lead QA calibration sessions

Let’s face it: holding regular customer service QA calibrations takes time and organizational effort. That’s why having dedicated facilitators responsible for making the process successful is essential.

Here are three tips for successfully engaging facilitators in your QA calibration process:

1. Let facilitators take the lead. Calibration sessions involve numerous mini-decisions, from selecting tickets for review to determining the final rating when there are differences of opinion. Allow your facilitators to take charge to prevent unnecessary disputes over each aspect of the process.
2. Rotate facilitators. Instead of having the same person run all sessions, rotate the responsibility among all reviewers. This ensures everyone on the QA team shares the responsibility and understands the complexity of aligning all reviewers to the same rating style.
3. Create facilitator guidelines. Provide specific guidelines to help facilitators lead sessions successfully. Outline potential scenarios and offer tips on handling different situations.

Allowing your reviewers to facilitate sessions keeps your team engaged and motivated. It fosters a sense of shared responsibility for customer service quality and helps everyone work together towards the same goals.

Define your quality calibration baseline

The quality assurance calibration baseline sets the allowable variance in your reviewers’ ratings, typically around 5%. This percentage accounts for minor fluctuations and helps you focus on significant discrepancies.

Here’s how to use your baseline effectively:

• Below the baseline: If the difference in your reviewers’ ratings is within the baseline, it indicates consistent evaluations. You can consider the calibration process successful, ensuring your team receives uniform feedback.
• Above the baseline: If the difference exceeds the baseline, it signals discrepancies in the feedback. You’ll need to continue the calibration process and discuss the details with your team to address these inconsistencies.

Having a measurable baseline simplifies calibrations. It helps you determine which differences in reviewers’ ratings need attention and which can be overlooked.

Use a quality management solution to facilitate quality assurance calibration

Calibration sessions, like the support QA process itself, can be labor-intensive if done manually. Switching to a dedicated tool with built-in calibration features can save significant time spent on manual tasks such as copy-pasting, file sharing, and sending notifications. This is especially important as your team scales, helping you avoid quality growing pains.

With Zendesk QA (formerly Klaus), you can easily find and share conversations for calibration, manage score visibility, and analyze your results all in one platform.

Here’s how simple it is:

1. Activate sessions: Go to your workspace’s ‘Calibration’ tab settings to activate calibration sessions.
2. Ensure unbiased reviews: Adjust visibility settings for your workspace reviewers and managers to ensure unbiased calibration reviews.
3. Access everything in one place: Use the ‘Conversations’ view to get an overview of past and active sessions, and schedule new ones.
4. Filter and focus: Select the most insightful conversations for calibration and add them to the relevant session.
5. Review and rate: Have reviewers rate the selected conversations using the same logic and scorecard setup as regular reviews.
6. Compare results: Use the dedicated calibration dashboard to compare results and ensure consistency.

By utilizing these features, you can streamline your QA calibration process, making it efficient and effective.

Make use of the findings from your quality assurance calibrations

One often overlooked aspect of calibration sessions is how to utilize the results effectively. The solution lies in two key actions:

1. Keep agents informed: Ensure that your support agents are kept in the loop regarding the outcomes of calibration sessions. This transparency fosters understanding and alignment with the quality standards upheld by your team.
2. Update quality standards: Use the insights gained from calibration sessions to refine and update your quality standards as needed. For instance, if a calibration session reveals ambiguity in a rating category, take the opportunity to clarify and define it further. It’s essential to communicate these updates to your support reps, as they directly impact their understanding and application of quality criteria.

By documenting and sharing the findings and outcomes of calibration sessions, you ensure that your support team remains cohesive and aligned in their approach to maintaining quality standards.

A clear path to Qualityville

Like conversation reviews, your calibration sessions should be a recurring activity. The benefits of regular sessions include:

• Keeping a relevant scorecard: Perhaps a rating category is poorly defined, or a 4-point rating scale does not make sense for one of your categories. Sessions bring all of your reviewers together, allowing you to discuss what is working and what is not working in your QA scorecard. 

• Internal process updates: You may calibrate cases where you notice the support rep did everything purr-fectly, however, it was still not a great customer experience. Your sessions can help bring to light processes in need of an update, while also helping to identify coaching opportunities for your team.
• A clear path to Qualityville: By having these discussions regularly, you are able to achieve consistency and keep an up-to-date definition of what quality means for your team. This is crucial for making sure that your reps are all heading in the same direction.

Want more? You can learn everything there is to know about calibration sessions and other conversation review topics in our first customer service quality course – Setting up the purr-fect customer service QA program. See you there!

Originally published on December 10th, 2020, last updated on March 23st, 2023.