This website requires JavaScript.

Why You Should be Skeptical of Reviewing Conversations at Random

Conversation reviews6 MIN READNov 23, 2021

Conversation reviews

When it comes to impurroving the quality of your support, we want you to be experts. 

A robust QA program consists of setting up a scalable way to conduct conversation reviews and using the information you gather to improve your customer service interactions. Quality is an ongoing process – one that Klaus makes easy, but a process all the same.

One way to save time in this purrocess is to make sure you are reviewing the right conversations.

How to review support conversations

The majority of our users use the default sorting option and review their most recent conversations first. Other users, likely aware that selecting consecutive interactions can lead to skewed sampling and thus, skewed results, use the random sorting option. 

Random is clever, you think. Random will provide a nice variety of conversations, you think. Random might be wasting your team hours of work per week, though.

Pour yourself a coffee – let’s get into why.

What is random?

Randomization is the act of scrambling your content arbitrarily, without method. The problem with this is that conversation reviews should have a method. 

Random = no method

Conversation reviews … need a method. 

See the problem?

The myth: a random sample gives you a diverse set of conversations.
The truth: a random sample is good at giving you more of the same. 

Diversity matters. When the percentage of conversations you review is small, you want every review to matter. Don’t worry, we’re not telling you you have to increase your review capacity ten-fold. We’re here to tell you how to improve the sample of conversations you review. 

As we know from our users, companies on average only have time to review 1% of their conversations. Using customer service QA software helps you make that 1% count. Read about how the makers of WordPress.com keep their CSAT above 95% with Klaus. 

“Why doesn’t a random sample give me a broad enough variety?!” you might be thinking. We hear you. Let’s look closer. 

Put on Your Thinking Cat

The complexity of human interaction contains multitudes. It is impossible to fully understand complexity using graphs and numbers, but (thankfully) it is possible to gauge complexity using certain parameters. Our data scientists examine interactions by word and character count, number of messages in a conversation, number of customer replies, and more. 

We snuck into the data lab to get our claws into some of the numbers. 

Let’s look at just one of these parameters – character count – to prove why random fails the diversity test

Check out this nifty histogram:

Histogram of conversation character count

As the median black line illustrates, half of all conversations consist of less than 2,000 characters, half of them consist of over 2,000 characters.

For context, at this point in the blog post, you’ve read just over 2,000 characters. (Way to go! Keep reading – there’s still good stuff to come.)

The distribution of interaction character count is skewed to the right. In other words, there is a long tail of conversations on the right side with far longer character counts than the median. It is not a balanced spectrum.

As the pink line illustrates, 25% of all conversations exceed 4,000 characters. These lengthier interactions are likely the most worthwhile for you to review: they help you detect the underlying problems in your support team(s). These are the conversations that consist of more than just a few back-and-forths, ones in which the customer concerns are not typical, ones in which your agents have no easy answers. Ones that demand improvement.

Data from our Pins for Coaching feature proves that complex interactions are the most useful for coaching.

These are the conversations you want to examine.

Time is precious

This is just an example of one parameter. It’s safe to say that this right-skewed trend is prevalent across all of the parameters we use to study interactions. 

Complexity is, um, complex. 

We have calculated that approximately 9% of all conversations classify as complex. Random sampling makes the well of possibility for finding them shallow. 

Again, complexity is determined by combined criteria. If we were to add all other parameters into the probability, the reality is that the amount of time and effort wasted by random sampling is much higher. 

Time to trim that a little. 

Finding the right conversations to review

You know that using the random filter isn’t the optimal method of conversation selection. So what to do meow?

You want to look at the more complex conversations. But setting up the filters to do so yourself is time-consuming and complex in itself, even if you do have the expertise.

Ahem, ushering in the customer service QA software of your dreams. Designed by cats.

Klaus’ Complexity Filter does a lot of the detective work for you. 

We created a statistical model to calculate the complexity of your company’s conversations. It filters your conversations through a specific lens that takes into account both language and channel. 

For example, we have realized that Latin languages tend to have higher message counts. And the average character count in email can differ vastly from the average character count in chat. The Complexity Filter uses this information to detect which of your conversations are relatively non-complex and removes them from the review selection. 

Sort by complexity: the best conversations to review will rise to the top (freshest first).

Your takeaway time-saving actions 

  1. Sort by complexity to review the most diverse, worthwhile conversations.
  2. Understand the why of complex conversations with Klaus’ Conversation Insights: filter out the non-complex tickets to examine, for example, how CSAT or Sentiment relates to these more valuable conversations.
  3. Highlight the most valuable conversations for training with Pins for Coaching to make your 1:1s more structured and purposeful.

Oh, and one last question you might be asking yourself: When should you sort by random? We didn’t just put it there for fun. It’s a perfect way to get a quick snapshot of the most common conversations on your helpdesk. We also want to leave the option open for those (super talented) customers who wish to construct their own filter in a way that catches what is interesting for them. Then they can apply random sorting on top. 

Never miss an update