AI Support Metrics: 5 KPIs To Track
Artificial intelligence might be taking over (metaphorically speaking), but it’s crucial not to adopt AI software blindly.
If you're considering (or have already implemented) AI tools to improve your support team’s efficiency, measuring success is essential to ensuring the investment is worth it.
In this article, we’ll review the key performance indicators (KPIs) you should monitor to ensure that the AI tools in your customer service tech stack help you deliver a great support experience and maximize your return on investment (ROI).
Why you need to track AI customer service metrics
We all know (or at least think we know) that AI is useful for improving support efficiency. But like American business theorist W. Edwards Deming famously said, “Without data, you’re just another person with an opinion.”
Building a business case for a new AI tool based only on gut feelings probably won’t convince your CFO. And even though researchers predict that chatbots will become the primary support channel for a quarter of all organizations by 2027, simply highlighting all the hype around AI probably won’t do the trick, either.
Without tracking AI customer support metrics, you won’t be able to justify the cost (which can be significant) or understand the ROI of your investment.
Moreover, you won’t know if your customers and team members benefit from the AI tools you’re using.
Recent research shows that six out of 10 customers would prefer if companies didn’t use AI for customer service. But, in my experience (and that of many others), that’s often because AI is implemented poorly, usually in a way that makes it hard for customers to get the help they need.
If you can get AI right, customers tend to appreciate self-service solutions when they work well.
Similarly, adding AI to your tech stack will likely save your support team time and save your business money.
Given that, it’s business critical to figure out how to measure AI performance to ensure it has a positive impact on your support metrics and on your customers — and to be able to make adjustments swiftly when required.
5 key AI support metrics to track
With dozens of AI tools available on the market, each offering its own reporting and analytics, setting the right targets can feel overwhelming.
To simplify this task, I’ve picked the five most important AI support metrics for you to monitor. These metrics will help you understand the performance of your AI tools and how they fit into your broader customer service goals.
Here’s a key thing to remember: The success of AI customer service tools largely hinges on how effectively they reduce the customer service burden on human agents.
They do this in two ways:
By resolving customer issues without requiring human interaction, in which case it’s critical the customer has a satisfying or positive interaction with the AI.
By making life easier for your human support agents through improving efficiency and streamlining their workflows. Features like Help Scout’s AI assist, AI summarize, and AI drafts all fall into this category.
The metrics you use to gauge your customer service AI’s success will reflect these two options. Metrics like deflection and resolution rates measure the first, whereas metrics like first response and resolution time help you gauge the second.
1. Deflection rate
Deflection rate indicates the percentage of customers who did not require human assistance after interacting with your AI tools, be it a chatbot or an AI-powered search on your knowledge base.
Deflection rate tells you how many tickets your human team was able to avoid thanks to your AI tools — and although that can sound a bit negative (why are you avoiding customers?), as long as those customers are having a good experience and are happy with the outcome, then it’s a win-win.
To calculate your deflection rate, divide the number of tickets deflected by AI without human intervention by the total number of support requests over a given period. Then, multiply the result by 100%:
For example, if you receive 100 tickets in a month and 30 of them are deflected by your AI chatbot without human assistance, your deflection rate would be 30%. Assuming customers stop the interaction because they get satisfying resolutions from the bot, the deflection rate allows you to measure its effectiveness.
The benchmark for deflection rate typically stands around 50% to 60%, but it can vary widely across industries and products.
A higher deflection rate can signal an efficient AI configuration that successfully handles customer requests. If your bot deflection rate is low, you can improve it by reviewing the tickets that required human intervention and training your AI model to better handle more scenarios over time.
However, a high deflection rate doesn't necessarily mean that all customer issues have been resolved, which brings us to the next crucial AI support metric.
2. Resolution rate
Unlike deflection rate, resolution rate takes the customer perspective into account, indicating the percentage of users who confirmed that their issue was resolved after interacting with your AI-powered channels.
The resolution rate shows how well your AI tools solve customer issues.
To calculate your resolution rate, divide the number of requests resolved by AI by the total number of support requests received. Then, multiply the result by 100%:
Here’s the tricky part: How do you know if an issue was resolved?
There are two common ways:
Ask your customers outright. If you’re using a chatbot, this can be as simple as adding a “Did this solve your issue?” prompt to the end of every conversation. It’s easy for the customer, but there’s also a chance that some percentage of customers won’t respond and will just exit the chat, skewing your numbers (or forcing you to assume issues were resolved).
Observe who follows up. With this approach, you consider issues resolved if a customer ends a conversation with your AI tool and doesn’t open a new conversation within a certain threshold (e.g., 48 hours). This can be harder to calculate, but it avoids making the assumptions of the first approach.
Let’s review our previous example. If your chatbot deflected 30 tickets but only 20 customers confirmed that their issues were resolved before closing the chat, then your deflection rate would still be 30%, but the resolution rate would be 20%.
The higher your resolution rate, the better. Resolving issues quickly and effectively is what customer support is all about, and if AI can help you do that better, then everyone wins.
If your resolution rate is low, you probably need to provide further training to your AI model. It also can be a signal that your AI is trying to handle conversations for which it isn’t equipped — such as complex technical issues — so you may need to update your conversation routing to send those customers straight to a human team member.
3. Cost per resolution
While this isn’t a performance metric in the same way as deflection or resolution rates, tracking the cost per resolution is crucial for evaluating the ROI of your AI implementation. That’s because this metric helps you understand the financial impact of your investment into AI.
To calculate the cost per resolution, divide the cost of your AI tech stack by the number of requests resolved by AI over the same period:
For example, if your AI bot costs you $300 per month and resolves an average of 1,000 tickets, the cost per resolution would be $0.30. The results are often mind-blowing when compared to cost per resolution by a human agent. In North America, for example, average cost per resolution is above $15 across email and chat channels.
Comparing the result with your cost per resolution before AI implementation will help assess the effectiveness of your investment. Ideally, you’d want to see a lower cost per resolution after AI tools are implemented.
However, avoid comparing the cost per resolution of AI and human agents once your AI channels are live.
Typically, human agents will start handling increasingly complex requests while AI takes care of simpler questions. More challenging tickets naturally take more time and resources, increasing their cost per resolution. Seeing a higher cost per resolution by agents doesn’t mean you should seek to eliminate those costs — instead, you should uncover the context and find ways to make your teams’ workflows more efficient.
The exact amount saved depends on the volume of deflected tickets and the average cost of a support interaction in your context. For larger organizations, the savings stack up quickly.
These savings can be reinvested into hiring more skilled employees, increasing salaries for existing agents, or implementing other initiatives to enhance customer experience, such as process automation, customer success programs, robust QA workflow, and knowledge base software.
4. Response and resolution times
First response time is how long it takes for a customer to receive the initial response to their query. You can also track your average or median response time, which would track how quickly your team is responding to every inbound customer message (not just the initial message).
Since AI tools offer instant responses, measuring response time for AI channels like chatbots isn't particularly useful. Where it can be impactful, though, is when you’re trying to understand how AI-features have impacted the efficiency of your whole support team.
For instance, let’s say your average first response time to emails is eight hours. You don’t have the budget to hire new team members, and your existing team simply can’t get to new conversations any quicker.
To make matters worse, you also don’t have the money to invest in a major new AI implementation — even if the vendor is promising really high deflection rates.
What can you do?
Well, if you’re using Help Scout, you could enable the AI drafts feature, which lets your team draft replies to customers with one click. Drafts are based on your past conversations and the Docs articles in your Help Scout account, so they’ll be tailored to your unique voice and product(s).
Instead of spending five minutes drafting a response, your team can generate a response, edit it as needed, and send it off within a minute or two.
This is a great example of using AI to speed up the things your support team does every day. While it’s not a customer-facing application of AI, it can dramatically improve metrics like response time.
Resolution time is another great AI support metric, and it’s closely related to response time. As you’d expect, resolution time measures the average time it takes your team to resolve customer issues.
It’s not uncommon for average response and resolution times to decrease due to AI efficiency but for those metrics to increase on tickets your human agents are handling (given their increased complexity).
To accurately gauge the performance of both your team and your AI bot, create separate reports to track how quickly resolutions are achieved by AI and human agents.
Since your human team will handle more complex and time-consuming issues, you'll need new benchmarks to accommodate this shift for your team as well. If you notice agents’ handling times climbing, look at other metrics — such as CSAT scores — to determine if this slowdown is impacting the overall customer experience.
5. Customer satisfaction (CSAT)
Your CSAT score reflects the level of customer satisfaction after support interactions, including those interactions that only involve AI.
Typically, customers are prompted to complete a satisfaction survey immediately following an interaction, rating their experience on a scale from 1 to 5. Your CSAT score is calculated as the percentage of satisfied customers who rated their experience as 4 or 5.
In competitive industries like SaaS and ecommerce, the CSAT benchmark is typically around 80%. High-performing teams can often achieve a CSAT score of 95+%.
Adding an open-ended feedback field to your CSAT survey can provide valuable insights into the reasons behind low and high scores, helping you adjust workflows to enhance the customer experience.
Tracking CSAT ratings separately for interactions with and without AI involvement will help you understand the impact of AI tools on customer satisfaction. Remember that AI should make your customer service team more efficient, but not at the expense of creating a great experience and fostering customer loyalty.
To gauge if your AI is achieving this goal, look at a metric like CSAT, then ask questions like:
Are customers consistently satisfied after interacting with AI channels?
How do your AI bots compare to human agents in terms of customer satisfaction?
Are there any trends in the open-ended feedback for AI-assisted vs. human-assisted interactions?
To improve your CSAT score, review AI interactions and customer feedback — you might identify request types where customers tend to have a better experience if they speak to a human agent right away. On the other hand, you might find topics where your AI tools excel.
Customers generally appreciate quick responses from bots for simple questions or requests. However, more complex issues requiring backend log reviews or involving account billing history may not be ideal for AI yet. In ecommerce support, for example, customers often appreciate automated return and exchange processes, so these are easy areas where you can double down on AI use.
It’s worth highlighting that AI support channels may sometimes have a slightly lower CSAT score, but the cost savings can be significant enough to justify their use. These tradeoffs aren’t always clear at first glance, so it’s important to approach them thoughtfully.
For example, customers might respond with a lower CSAT score if they occasionally need to repeat their questions, but if introducing AI also means they’re benefitting from 24/7 support that you couldn’t afford otherwise, is that a worthwhile trade to make?
While I’d never encourage companies to shift focus off creating satisfied customers, if using AI unlocks major benefits or enables you to invest the saved money in building a more reliable product or better self-service resources, that might offset any minor dips in satisfaction.
Bring your customer support to the next level
By leveraging AI-powered customer service and closely monitoring these five key AI service metrics, you can strike the right balance between efficiency and customer satisfaction — ensuring that your AI tools contribute positively to the overall customer experience.
You’ll also be better positioned to reduce costs and improve operational efficiency, helping you stay ahead of the competition.
Sign up for a 15-day free trial of Help Scout or check out our newest AI features to start optimizing your service operations today!
Maryna Paryvai
Maryna is a results-driven CX executive passionate about efficient processes and human-centric customer support. With a track record of scaling ecommerce support operations, she firmly believes that exceptional customer experiences lie at the heart of every successful business.
