The following is a guest blog written by Joey Greenwald, Senior Director of Marketing, Directly.
Creating a truly effective AI-powered virtual assistant (VA) isn’t simple. You don’t just drop in a few lines of code and a data set and then let an algorithm do its thing. As we wrote recently, there are four main reasons AI projects fail and one of them is a lack of maintenance. It takes a dedicated and ongoing commitment to do it right. Just ask Amazon — the company has spent billions of dollars on research and development and reportedly has 10,000 employees dedicated to making Alexa more understanding, empathetic and ultimately effective.
Almost like a parent, organizations that successfully deploy AI-powered products are continuously teaching and nurturing them over time — even after they are mature, effective and by all standards “intelligent.” They raise them.
In our space, AI is being deployed as a tool to help companies automate portions of their customer service operations. The problem we at Directly are trying to solve is to help businesses efficiently resolve customer support issues. We do this with AI-powered virtual agents that can quickly identify the intent of the question, serve automatic content to resolve the issue, and escalate when needed to human support agents.
The key metric for support automation is “resolution rate.” Simply put, how many support cases can be completely resolved by AI-powered virtual agent without escalating those issues to an agent?
When we discuss resolution rate with our clients, we look at the curve shown below and segment potential customer support questions into three sections: The head (red) represents the most common questions, while the middle tail (green), and the long tail (teal) are the least commonly asked and most complex questions.
The area of this curve where an AI-powered virtual agent is able to successfully answer questions depends largely upon how companies (and their vendors) support their AI initiative over time.
In this post, I’ll explain the steps involved in raising an AI-powered virtual agent. Most importantly, I’ll show how to increase its level of maturity and its success rate. I’ll also share some rough benchmarks for support resolution rates with each stage of AI maturity, as illustrated by our first contact resolution curve.
Childhood: Creating and launching your virtual support agent
Let’s say your company has committed to building and launching a virtual support agent. Here’s how you’d first bring it to life.
Step 1: Choose a VA vendor
The first step is to choose a vendor from which to license your virtual assistant. This decision typically depends upon the level of features you need, though it may also be influenced by any pre-existing vendor relationships (e.g. if you use Azure or other Microsoft technology, that might steer you to Microsoft Bot Framework). Many companies start with broadly applicable services such as these: Amazon Lex, Microsoft Bot Framework, Zendesk Smooch, Salesforce’s Einstein, Meya, and Google Dialogflow. For more robust (and expensive) solutions, companies look to Nuance and [24]7.ai.
Step 2: Cluster historical data
Next, you’ll need to gather all relevant customer support data representing your history of customer questions and interactions. Your AI will need this baseline structured data, likely in the form of a .CSV file, as you start building its intelligence. And then you’ll group related questions into clusters.
Step 3: Convert clusters into intents and create labels
In this step, you convert the clusters into “intents.” The intents represent the user goal for different questions they may ask. For example, one user may ask, “My order hasn’t arrived — where is it?” while another might say, “I put an order into the system on Wednesday and have not received it.” With each, the user intent is to find out the status of an order. For these clusters of intents, you would then create labels. In this case, the label may simply be “Order Status.” Part of this step includes determining which labels occur frequently enough to justify the effort of turning it into an intent. Once you have a series of intents, you then edit the questions associated with the intents into simpler and unambiguous “training phrases,” which will be applied to the machine learning model.
Step 4: Write content
Now it’s time to deploy subject matter experts to craft the specific language to be used by your virtual support agent. For any customer question that is part of the “Order Status” label, your assistant might respond with, “Let me check on the status of your order,” and then ultimately go find the information and present the specific answer. That copy will need to be customized for each channel in which you’re using a virtual support agent. For example, a web-based chatbot is much different than a voice-enabled agent like Alexa. If your virtual agent is international, you’ll need to localize content for each language you plan to support.
Step 5: Load, train, and launch
Now that your customer data is structured, your AI foundation is in place and your content is written, it’s time to put it together and launch it. If you work with Directly, we have an integration team that can help you with this. Using tools provided by your VA platform, you’ll launch your VA and upload your data sets — including content — onto a staging site. You then test and train the virtual agent until it’s performing up to expectations. And at that point, it’s time to move your virtual agent to a production site and bring it to life.
Expected success rate
Now that you launched your VA, what can you expect during its “childhood”? Immature consumer AI-powered virtual agents should resolve between 30-50% of questions, while rates for more complex B2B solutions would be lower. Those cases, represented in stripes, would likely all appear in the lower section of the head of our graph, representing only the most frequently asked questions and least complex questions.
However, if the company launches and doesn’t have the resources in place to actively maintain the AI, performance won’t continue at this level. Inevitably, your virtual support agent’s effectiveness will degrade over time because the external factors related to your business and your services evolve.
A few months after launch, the curve will show regression and look something like this:
Adolescence: Maintenance and optimization of your agent
So, let’s say you launch your virtual support agent and your goal is to simply maintain the curve and your contact resolution rate. You’ll need dedicated training to ensure that the performance you see at launch stays steady. Here are the three areas you should be monitoring with your virtual agent — and the related tactics to improve in those areas:
- Intent precision: How accurate is your AI in matching the questions to the correct intents? Internal or external subject matter experts should actively review and repair the link between intents and questions. Over time, you also should be steadily growing the number of questions that are linked to each intent.
- Recall: How often is your virtual agent attempting to answer question variations that relate to existing intents? You want the AI to attempt to answer as many of the relevant questions as possible, while only escalating issues that are more complex.
- Content success: Make sure that when your virtual agent does answer questions, the content is appropriate for the intent. As part of ongoing training and maintenance, this content should be optimized over time based on customer satisfaction ratings — this is where subject matter experts also play a role — as they can evaluate and revise content as needed.
Expected success rate
Again, if the goal here is to simply maintain your AI-powered virtual agent, then your curve (below) and resolution rates should look very similar to what they do at launch. But, simply maintaining these rates means that your virtual agent may never get out of that awkward adolescent phase.
Adulthood: Pushing down the curve by advancing your AI
So, you don’t want to settle for just maintaining your virtual support agents base level performance? That’s great — because neither do we.
The good news is that improving the performance of your VA so that it reaches “adulthood” requires a similar methodology as simply maintaining it. Just a lot more of it.
With the performance attributes above — intent precision, recall and content success — companies that commit incremental increases to resources should see a correlated improvement in their contact resolution rates.
The big difference in “adulthood:” you’ll be creating many more intents and content for events that are more unusual and complex (and more likely to fall into the middle or long tail). When there are new questions emerging, you need to make sure there’s a process for capturing them and then quickly converting them into intents and creating content. For example, if new features aren’t working, that requires new intents. And as the number of intents grows, that means you’ll need to have a correlated increase in training in place to improve the intent precision, recall, and content success.
And if you want your AI-powered agent to take on more short-term issues like service outages (further down the curve), you’ll want resources and processes in place to quickly arm your virtual agent to help what we call “surge capturing.”
Expected success rate
So, if companies apply more resources to increase their resolution rate, how much can they realistically improve? That depends on the business and the resources it can dedicate. Every case is different — but you can think of the resolution rate similarly to the 80/20 rule (a.k.a. the Pareto principle). Companies can expend minimal resources to get to a base level resolution rate, but it requires increasingly more effort/resources the further down the curve you move.
With the additional committed resources, we expect that B2C company VAs can realistically achieve a 50-70% contact resolution rate and your adult virtual agent’s graph starts to look more like this:
AI Superhero: Adding the human touch your virtual agent
No virtual assistant — nor human agent, for that matter — will be able to successfully resolve 100 percent of support-related questions. Our secret sauce at Directly is the combination of AI and human experts in what we often refer to as “expert-in-the-loop” AI. The Directly CX automation platform helps our clients incentivize their networks of expert users to train the AI at all phases of virtual agent maturity. And, importantly, that expert network serves as a backstop for questions the AI can’t answer.
Our integrated platform helps companies seamlessly transition issues from an AI virtual agent to an expert, so the customer experience is painless. (Unlike the pain we’ve all encountered when we can’t find answers online — then give up and call a company’s phone support, and often remain on hold for an aggravating period of time.) So when you combine a thoroughly trained, AI-powered virtual assistant with human expertise, you’ve now achieved peak CX automation. A true AI superhero.
Expected success rate
This CX automation superhero solution can achieve contact resolution in the 80-90% range. And the first contact resolution graph is a thing of beauty, as you see near perfection in the head and significantly improved results in the medium tail and further out in the long tail.
Want to see an AI ‘superhero’ virtual agent in action?
If you’d like to see an expert-powered AI in action, visit directly.com.
