Mithun Prasad, PhD, Data Scientist at Microsoft
Chatbots
The rise of artificial intelligence (AI) in the last few years has given a massive push to the idea of conversational interfaces, commonly known as chatbots. Chatbots can maintain a conversation with a user in natural language, understand the user’s intent and send responses based on the organization’s business rules and data. Many chatbots including Cortana, Siri and Alexa can also interact verbally with consumers. In short, chatbots provide a conversational interface using natural language. They are expected to engage users in extended conversations involving followup dialogue.
LUIS
Language Understanding Intelligent Service (LUIS) offers a fast and effective way of adding language understanding to applications. With LUIS, you can use pre-existing, world-class, pre-built models from Bing and Cortana as suited to your purposes — and when you need specialized models, LUIS guides you through the process of quickly building them. The intent-entity model is very popular among many chatbot frameworks including LUIS. LUIS interprets user goals (intents) and distills valuable information (entities) from within sentences, for a high quality, nuanced language model.
Searching with conversations
While LUIS is commonly used for chatbots, it can also be used for understanding search queries. Search queries are very common in retail applications, food delivery applications, shopping applications, etc. where users are searching for a specific product or entity. Common examples of searching with a free text query are:
- Show me 8” blue shower tiles
- Thai restaurants in Los Angeles
- Flights to Los Angeles from Seattle today
The query resolution from the search query would result in showing a result list often sorted by field(s). With LUIS, the intents and entities extracted from the search query would produce field and field values to perform a search against an inverted index (such as Azure Search, solr, elastic search) or a database to obtain a result list.
- In the case of Show me 8” blue shower tiles, LUIS can be trained to extract entities such as [size: 8 inch, color: blue] and intent: ShoppingFlooringTiles. Using the intent and entities, results can be obtained by searching certain fields capturing product attributes such as size and color. If the product attributes are not indexed, either the title or description field can be searched to identify the entities. The intent is an indication of query classification to filter on categories in this scenario. The result list in the area of retail for example can include product attributes such as color, shape, materials, price, product description, etc.
- With the second case, Thai restaurants near me, LUIS can be trained to extract the following entities [cuisine: thai, location: santa monica] and intent: RestaurantSearch. If restaurants along with its cuisine and location are indexed, these fields can be filtered or sorted using the entity values obtained from the query analyzed by LUIS.
- With the third query, Flights to Los Angeles from Seattle today, the entities extracted would be [fromLocation: Seattle, toLocation: Los Angeles, departureDate: today ]. Another way of extracting the entities can be in a hierarchical structure such as [location: {from: Seattle, to: Los Angeles}, departureDate: today}. Time expressions such as today can be further mapped to a date value. Using intent and entities extracted, entity resolution can be performed by searching structured fields related to locations, dates, etc.
LUIS also provides prebuilt entities that can help with detecting entities. Generic entities such as age, currency expressions, or date expressions are available, and when included in your application, LUIS includes the corresponding entity prediction in the endpoint response.
In summary, LUIS or a custom Natural Language Understanding (NLU) engine built with some of the techniques below can be used to extract intent and entities for processing search queries. A robust NLU engine needs to be deterministic, generalize, and have the ability to resolve entities.
Routing Strategies
With conversational search, queries can be open-ended and can result in queries that are not domain related. For example, What’s the weather like today? would result in a None intent or an intent that was not part of the domain used to build the model. In such scenarios, we would not treat the query as a search query. In many other scenarios, for the chatbot to respond to a request, it would need to be aware of different domains. This is usually captured with intent which classifies a given query into a finite set of topics.