Search integration with ETL/DW/Synapse(DL)

[aowens-jmt]

We're looking at implementing an enterprise search, which includes a revamp of our EDP (enterprise data platform). Creating an etl which will ingest data from various sources via ADF into a DL to be used by DW/Synapse as a source of truth. What are the best ways to connect the cognitive search to that gold data (may contain un-, semi-, fully-structured data)?

[#2]

We'd like for our users to be able to ask natural language questions of the data and get results (and better yet, actual answers to questions)

[#3]

Hi! Since you are staging the content into a DL, I suspect that might be the best place to go to connect cognitive search. In the event your DL is ADLS Gen 2, you might find the Indexer we have for this (https://docs.microsoft.com/en-us/azure/search/search-howto-index-azure-data-lake-storage) to be an easy way to bring the data in to the search service, at which point it is then searchable. As part of this ingestion, you could also potentially use some of the Cognitive Skills (https://docs.microsoft.com/en-us/azure/search/cognitive-search-concept-intro) to further enrich this content, in the event you wanted to do so.

[#4]

Adding onto @liamca-msft response, @aowens-jmt - you could use semantic search (including answers and captions) to provide a natural language query experience. You can sign up for the preview here: https://aka.ms/semanticpreview

[#5]

I thought I (our company) already signed up for that preview; we're already having conversations with MS about prototyping solutions with CS. Thank you, I wouldn't mind seeing some examples of that.

[#6]

Natural Language (NLP) is a really broad area, so what I would suggest is to break down what is actually need by the user. For example, sometimes the goal is to be able to use NLP to ask questions and get answers. Or othertimes, it is to be able to let users enter a search query that is not a set of words, but more like a phrase and have the search understand it. For example, when I search "hot dog" it should know that I am not referring to "puppies", but rather food. For each of these cases, I highly recommend that you take a look at the semantic search capability we just introduced (https://docs.microsoft.com/en-us/azure/search/semantic-search-overview). In some cases, there are really advanced NLP cases. For example "find me all the homes in seattle that have 3 bedrooms and are less than $1m". This is advanced because you need to understand that some of the query ("3 bedrooms") need to be mapped to a filter query (bedrooms == 3). for this, you can do training of the users intent using LUIS (https://www.luis.ai/), but this requires a lot of training, and I would not likely suggest that you start here.

[#7]

But if we have data that contains information about people connected to a particular project (via timesheets, CRM data, etc) and we ask it to give me a list of people working on project XXXX would semantic search be a good option for that or LUIS need to be brought in?

[#8]

@aowens-jmt that is a great example.  You could definitely use LUIS to help here because you could break down the intent of this query into a search where project == XXXX.  In this particular case, I don't think Semantic Search would help because if you sent a query of "give me a list of people working on project XXXX", the only term in this that would actually help find relevant content in the search index is XXXX.  It would not know that the user wants to limit search to the field "project".  It is not to say that you need to use LUIS, and you could certainly have you web app intercept this search query and rewrite it as a search query (search=*&$filter=project eq XXXX).