Co-Authors: Rashi Kukke & Nilay Chheda
Blog Overview: In this technical blog for developers, we dive deep into the implementation of the Autosuggest feature, including best practices and lessons learned. Learn about data ingestion technology, tuning of Cognitive Search and Semantic Search match scores, telemetry, and performance analysis. We will also provide benchmarks and resources for performance analysis with Azure Cognitive Search.
Introduction
In today's fast-paced world, customers expect immediate solutions to their problems and quick access to support. To meet this need, companies must provide a seamless and efficient customer support experience. The Azure Customer Support Experiences Team recognized this and in 2020, initiated a project to enhance the customer experience through the Support and Troubleshooting platform with the introduction of the Autosuggest feature. This innovative solution leverages Azure Cognitive Search and AI/ML technologies to deliver highly relevant search results, streamlining the support process and providing quick access to key customer requests. Through its successful integration of 12 Azure services, and with a goal to onboard 35 products by the end of March 2023, the Autosuggest feature has become a critical component in delivering a superior customer support experience. What’s more, autosuggest has gone on to win a Microsoft hackathon competition.
Figure 1: Azure Support Portal Request Form with Autosuggest for Summaries
Indeed, the Autosuggest feature has proven to be a popular tool among customers, with many requesting its troubleshooting abilities. Here we uncover insights into the development and implementation of Autosuggest, including best practices and lessons learned.
What is Azure Cognitive Search
Microsoft was recently recognized as a Leader in the 2022 Gartner® Magic Quadrant™ for Insight Engines, Azure Cognitive Search delivers a cloud-based search service from Microsoft that enables developers to add advanced search capabilities to their applications. With features like natural language processing, machine learning, and AI technologies, Cognitive Search makes it easy to find relevant information from unstructured data. Developers can create and manage search indexes, customize search experiences, and analyze data using built-in analytics tools. Semantic search, a feature of Azure Cognitive Search, goes beyond traditional search methods by understanding the meaning and context of search queries and data. This enables more accurate and relevant results, even with queries phrased differently than the data. Semantic search also enables searching across multiple fields and extracting entities and relationships within the data, making it a valuable tool for creating sophisticated search experiences.
Autosuggest Application Architecture
The Azure Customer Support Experiences Team is dedicated to delivering top-notch customer support experiences for Azure services. The team has leveraged AI and ML technologies to improve their support experiences while ensuring scalability and resilience. The team's core architecture has been optimized to handle these improvements, benefiting both existing and new use cases.
The Autosuggest feature in the Azure portal was designed to offer lightning-fast latency and highly relevant suggestions, considering various user context information. To achieve this, the team utilized Azure Cognitive Search and its Semantic search feature to create a user experience like the Azure Cognitive Search suggester or Autocomplete APIs, using the search API itself.
The Autosuggest infrastructure was set up and scaled globally using "infrastructure as code" (IaC) and Azure Resource Manager (ARM) templates. The Autosuggest architecture includes 6 components and is designed to be idempotent. Data is replicated to regional storage, attached to the Cognitive Search instance, and AI skill sets are added for improved indexing.
Semantic Search is not just for search queries. How we implemented its results as a suggester
Semantic search is designed for search queries. However, our team saw the value of leveraging the out-of-the-box deep learning capabilities from Microsoft Bing for a different area of search. Azure Cognitive Search suggests using at least two replicas for search queries, as they are usually read-intensive operations. We chose the standard S1 tier, which offers enough replicas and partitions for our current and future needs, and is the minimum tier required to use the semantic search feature. It's important to review Azure Cognitive Search's documentation to understand the available performance tuning options and determine the best approach based on specific needs.
Figure 2: Azure Cognitive Search Application Architecture
The Autosuggest architecture has six components, starting with the core infrastructure, followed by configuring the Azure Cognitive Search instance, and ending with setting up the Autosuggest microservice and API endpoint. The components are executed in the order listed and are designed to be idempotent, meaning that resources in the core infrastructure are created only once and updated only if the ARM template changes. The data source configuration, AI skill set, and indexer are also idempotent, but the Cognitive Search index and data snapshot are treated as unique artifacts for each deployment, so a new index and snapshot are created with each deployment.
The Autosuggest data is stored in a single JSON array file on cost-efficient blob storage and the search index is updated each time new data arrives. The data is first replicated to regional storage and attached to the Cognitive Search instance. The built-in key phrase extraction skillset is added to some data fields to enhance the cognitive search index. After creating the index and indexer, the Autosuggest service is set up and the end-to-end Autosuggest application scenario is tested by simulating a call to the newly deployed Autosuggest API instance with the updated index.
Results
After testing the Semantic search feature with Autosuggest, the Azure Customer Support Experiences Team decreased the number of support requests by 4% and mitigated an additional 2% of initiated requests after customers received an autosuggest solution. According to Azure Supportability leaders,
“The task of understanding our customers' intent and providing the right solution can be challenging, especially when they describe their issues using just a single word or their messages are too brief or too long. Fortunately, we're able to overcome this obstacle through the Autosuggest feature in the Azure portal for Commerce Products, which presents us with suggestions to capture the customer's true intent accurately. Thanks to Autosuggest, our Commerce Engineering Operations team has experienced impressive results, not just in terms of classification accuracy but also in terms of customer experience and engagement. We're using autosuggest suggestion to train our problem classification model for even greater impact. Over the last quarter, we're thrilled to report that we've seen a significant 5.3% increase in SHS from the addressable market using Autosuggest. This has allowed us to serve our customers better, quickly resolve their issues and improve overall satisfaction. We're excited to see what the future holds as we continue to leverage Autosuggest to achieve even more impressive results.” - Vinay Kumar, Microsoft Principal Program Manager, Commerce Engineering Operations
“The semantic search capability arrived just in time for us to integrate it with our Autosuggest infrastructure while team was working on an internal hackathon project, which has improved our customers' support and troubleshooting experience in the Azure portal. The inherent capabilities of Semantic search, such as its ability to go beyond traditional keyword search by taking into account the context and meaning of the search and the data provided, have been a great addition to our system. We are pleased to see the positive impact on our business KPIs due to Autosuggest built on top of the Azure Cognitive Search and semantic search technologies” - Ashutosh Agrawal, Principal Group Engineering Manager, Azure CXP
Best Practices and Lessons Learned
Key Lessons Learned:
- Understanding the search score and determining the best threshold for a specific use case is important in order to improve the relevance of search results.
- The choice of data ingestion technology plays a crucial role in ensuring that the data is properly indexed and optimized for search.
- Threshold tuning of cognitive search and semantic search match scores can have a significant impact on the relevance of search results.
- Collecting and analyzing telemetry data helps to identify potential performance bottlenecks and areas for improvement.
- Regular performance analysis and benchmarking helps to ensure that the system is running optimally and to identify areas for improvement.
In this blog, we discussed the key lessons learned during the development of the Autosuggest feature for Azure Support. These lessons include the importance of understanding the search score, the impact of the choice of data ingestion technology, the impact of threshold tuning, the importance of telemetry, and the importance of regular performance analysis and benchmarking. By following these key lessons, organizations can improve the relevance of their search results and ensure that their systems are running optimally.
To learn more about Azure Cognitive Search, visit: https://learn.microsoft.com/azure/search/search-what-is-azure-search
Gartner, Magic Quadrant for Insight Engines, Stephen Emmott | Anthony Mullen | David Pidsley | Tim Nelms, 12 December 2022
Gartner is a registered trademark and service mark and Magic Quadrant is a registered trademark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
Microsoft
