Tracking and Measuring Success With AI in Customer Experience

Gladly Team

5 minute read

As AI continues to reshape the customer experience landscape, businesses are grappling with how to effectively implement and measure the success of their AI initiatives. In a recent webinar, Lauren Inman-Semerau, Head of CX at Rothy’s, shared valuable insights on tracking and measuring AI success in customer service.

WEBINARS

How Rothy's Tracks AI Customer Service Impact

Watch now

Setting the Stage: Defining AI Success Metrics

Before diving into measurement, it’s crucial to establish clear goals for your AI implementation.

As Lauren notes, “In the beginning, it was really important for me to track CSAT of Sandi [Rothy’s AI chatbot] as well as CSAT for my agents. SLA was very important in those first few months.”

Key metrics to consider include:

However, Lauren cautions against relying solely on traditional metrics: “I actually thought average handle time would be much more important than it has turned into over time.”

Evolving Metrics for AI-Driven Customer Service

As AI becomes more integrated into customer service operations, it’s essential to adapt your measurement strategies.

Lauren explains, “Overall metrics are going to start shifting, and what we think about as industry standards is going to shift because of AI. We don’t want to give up those basic KPIs like SLA, average handle time, FCR, and CSAT — those are base health metrics. But now we have the opportunity to layer onto them things like resolution rate for your bot and transfer rate to your agents.”

Some emerging metrics to consider include:

AI Resolution Rate: The percentage of inquiries successfully resolved by AI without human intervention
Transfer Rate: How often queries are escalated from AI to human agents
AI Adoption Rate: The proportion of customers engaging with AI-powered solutions
Sidekick Conversation Handle Time (CHT): A measurement of time saved through Gladly’s conversational and collaborative AI handling routine inquiries, helping optimize workforce planning and agent allocation.

Holistic Measurement Approach

Lauren emphasizes the importance of looking at metrics holistically: “I think you need to look at it holistically as a business. Is my overall average handle time coming down, is it staying the same, and also looking at the agent? So you still need reasonable average handle times, but if you’re not taking into account the topics related to what your agent brand ambassadors are resolving, I think that’s a miss.”

This approach allows you to understand the full impact of AI on your customer service operations, including how it affects human agent performance and overall efficiency.

Measuring AI’s Impact on Revenue Generation

An often overlooked aspect of AI in customer service is its potential to drive revenue.

Lauren shares an exciting insight: “We’re seeing, depending on the month and our volume, [significant conversion rate improvements] just by talking to my agents and talking to somebody who can inform you about the product. So, as you’re thinking about KPIs for your AI, think about the flip side. How are your KPIs also changing for your brand ambassadors and your agents? Are you repurposing their work? Are you elevating their work? Is it just about FCR, average handle time, cost per contact, SLA, or how can they actually start to [generate revenue]?”

This reveals an opportunity to measure AI’s impact on:

Practical Tips for Implementing AI Measurement

Establish a baseline: Measure key metrics before implementing AI to gauge its impact accurately.
Use consistent measurement methods: Lauren explains their approach to CSAT measurement: “We have our CSAT triggered at the close of our conversation. So when you [are] done talking to either Sandi or an agent, 24 hours later, an email is triggered to rate your interaction.”
Segment AI and human agent performance: “My email trigger doesn’t know the difference between a brand ambassador and Sandi. It sends the same satisfaction email, and then we sort it by agent.”
Monitor for unexpected outcomes: Keep an eye out for “hallucinations” or incorrect responses from your AI system.
Continuously refine and adapt: Regularly review your metrics and adjust your AI strategy accordingly.

Embracing the Future of AI Measurement

As AI continues to evolve, so too must our approaches to measuring its success.

Lauren summarizes this sentiment perfectly: “We’re in this cool place where we can help craft what that looks like for our companies…at the end of the day, customers want to be heard, serviced, and delivered to. As AI evolves, customer needs may not change, but how we deliver those experiences will.”

By adopting a holistic, forward-thinking approach to AI measurement, businesses can track the success of their current implementations and pave the way for future innovations in customer experience.

Remember, the goal isn’t just to implement AI for its own sake but to leverage it thoughtfully to create more personalized, efficient, and satisfying customer interactions. With the right metrics, you’ll be well-equipped to navigate the exciting future of AI-driven customer service.

Read more about how Rothy’s partnered with Gladly to achieve:

83% SLA across channels
93%+ AI CSAT
80% reduction in agent attrition

Learn More About the Leading Customer Service AI Conference of the Year

GLADLY CONNECT LIVE