Forum Discussion

Copper Contributor

Mar 10, 2021

Search integration with ETL/DW/Synapse(DL)

We're looking at implementing an enterprise search, which includes a revamp of our EDP (enterprise data platform). Creating an etl which will ingest data from various sources via ADF into a DL to be ...

aowens-jmt

Copper Contributor

Mar 10, 2021

We'd like for our users to be able to ask natural language questions of the data and get results (and better yet, actual answers to questions)

liamca-msft

Microsoft

Mar 10, 2021

Natural Language (NLP) is a really broad area, so what I would suggest is to break down what is actually need by the user. For example, sometimes the goal is to be able to use NLP to ask questions and get answers. Or othertimes, it is to be able to let users enter a search query that is not a set of words, but more like a phrase and have the search understand it. For example, when I search "hot dog" it should know that I am not referring to "puppies", but rather food. For each of these cases, I highly recommend that you take a look at the semantic search capability we just introduced (https://docs.microsoft.com/en-us/azure/search/semantic-search-overview). In some cases, there are really advanced NLP cases. For example "find me all the homes in seattle that have 3 bedrooms and are less than $1m". This is advanced because you need to understand that some of the query ("3 bedrooms") need to be mapped to a filter query (bedrooms == 3). for this, you can do training of the users intent using LUIS (https://www.luis.ai/), but this requires a lot of training, and I would not likely suggest that you start here.

aowens-jmt
Copper Contributor
Mar 10, 2021
But if we have data that contains information about people connected to a particular project (via timesheets, CRM data, etc) and we ask it to give me a list of people working on project XXXX would semantic search be a good option for that or LUIS need to be brought in?
- liamca-msft
  Microsoft
  Mar 10, 2021
  aowens-jmt that is a great example. You could definitely use LUIS to help here because you could break down the intent of this query into a search where project == XXXX. In this particular case, I don't think Semantic Search would help because if you sent a query of "give me a list of people working on project XXXX", the only term in this that would actually help find relevant content in the search index is XXXX. It would not know that the user wants to limit search to the field "project". It is not to say that you need to use LUIS, and you could certainly have you web app intercept this search query and rewrite it as a search query (search=*&$filter=project eq XXXX).