All Azure AI posts

Localize your website with Microsoft Translator

Krishna_Doss — Thu, 22 Apr 2021 23:57:02 GMT

Web Localization and Ecommerce

Using Microsoft Azure Translator service, you can localize your website in a cost-effective way. With the advent of the internet, the world has become a much smaller place. Loads of information are stored and transmitted between cultures and countries, giving us all the ability to learn and grow from each other. Powered by advanced deep learning, Microsoft Azure Translator delivers fast and high-quality neural machine-based language translations, empowering you to break through language barriers and take advantage of all these powerful vehicles of knowledge and data transfer.

Research shows that 40% of internet users will never buy from websites in a foreign language[1]. Machine translation from Azure, supporting over 90 languages and dialects, helps you go to market faster and reach buyers in their native languages by localizing your web assets: from your marketing pages to user-generated content, and everything in-between.

Up to 95% of the online content that companies generate is available in only one language. This is because localizing websites, especially beyond the home page, is cost prohibitive outside of the top few markets. As a result, localized content seldom extends one or two clicks beyond a home page. However, with machine translation from Azure Translator Service, content that wouldn’t otherwise be localized can be, and now most of your content can reach customers and partners worldwide.

How to localize your website in a cost-effective way?

The first step is to understand the nature of your website content and classify them. It is critical as each of them needs different levels of localization. There are four types of content: a) static and dynamic, b) generated by you and posted by customer, c) sensitive like ‘Terms of Use’, d) part of UX elements.

Static content like about the organization, product or service description, user guides, terms of use, etc. can be translated once (or less frequently) offline into all required target languages. Translation results could be cached and served from your webserver. This could substantially reduce the cost of translation. Machine translation models which powers Azure Translator service are regularly updated to improve quality. Hence consider refreshing the translations once a quarter if not every month.

User generated content like customer reviews, information requests, etc. are dynamic in nature, not all of them requiring translations, and to be translated on need basis only. You could plan for an UX element in the webpage which could initiate translation on need basis. Target language for translation could be identified based on user browser language. Likewise, responses to customer could be translated back into the language of original request or comment.

Sensitive content like terms of use, company policies, are recommended to do a human review post-machine translation.

Text in UX elements of the webpage like menu, labels in forms, etc. are typically one or two words and have restricted space. Hence recommended to do a UX testing post translation for fit and finish. If necessitates look for alternate translation or human review.

Due to the speed and cost-effective nature that Azure Translator Service provides, you can easily test which localization option is optimal for your business and your users. For example, you may only have the budget to localize in dozens of languages and measure customer traffic in multiple markets in parallel. Using your existing web analytics, you will be able to decide where to invest in human translation in terms of markets, languages, or pages. For example, if the machine translated information passes a defined page view threshold, your system may trigger a human review of that content. In addition, you will still be able to maintain machine translation for other areas, to maintain reach.

By combining pure machine translation and paid translation resources, you can select different quality levels for the translations based on your business needs.

How to use Azure Translator service to translate static content

Pre-requisite:

Create an Azure subscription
Once you have an Azure subscription, create a Translator resource in the Azure portal.
Once Translator resource it created, go to the resource, and select ‘Keys and Endpoint’ which is used to connect your application to the Translator service.

Translating static webpage content:

Below code sample shows how to translate an element in the webpage. You could use it and iterate for each element in your webpage requiring translation.

import os, requests, uuid, json subscription_key = "YOUR_SUBSCRIPTION_KEY" endpoint = "https://api.cognitive.microsofttranslator.com" path = '/translate' constructed_url = endpoint + path params = { 'api-version': '3.0', 'to': ['de'], # target language 'textType': 'html' } headers = { 'Ocp-Apim-Subscription-Key': subscription_key, 'Content-type': 'application/json', 'X-ClientTraceId': str(uuid.uuid4()) } # You can pass more than one object in body. body = [{ "text": "The samples on this page use hard-coded keys and endpoints for simplicity. \ Remember to remove the key from your code when you're done, and \ never post it publicly. For production, consider using a secure way of \ storing and accessing your credentials. See the Cognitive Services security article \ for more information." }] request = requests.post(constructed_url, params=params, headers=headers, json=body) response = request.json() print (response[0]['translations'][0]['text']) # shows how to access the translated text from response

Localization is just a fraction of the things that you can do with Translator, so don't let the learning stop here. Check out recent new Translator features, additional doc links to dive deeper, and join the Translator Ask Microsoft Anything session on 4/27.

Get started:

Sign up for Azure trial
Join Translator engineering team on Ask Microsoft Anything on 4/27
Learn about Document Translation (Preview)
Create a language translator application with Unity and Azure Cognitive Services
Translator documentation

[1] CSA Research – Can’t Read, Won’t Buy – B2C Analyzing Consumer Language Preferences and Behaviors in 29 Countries https://insights.csa-research.com/reportaction/305013126/Marketing

Big data preparation in Azure Machine Learning – powered by Azure Synapse Analytics

Xun_Wang — Tue, 20 Apr 2021 16:00:00 GMT

Many customers who embark on a machine learning journey deal with big data, and need the power of distributed data processing engines to prepare their data for ML. By offering Apache Spark® (powered by Azure Synapse Analytics) in Azure Machine Learning (Azure ML), we are empowering customers to work on their end-to-end ML lifecycle including large-scale data preparation, featurization, model training, and deployment within Azure ML workspace without the need to switching between multiple tools for data preparation and model training. The ability to build the full ML lifecycle within Azure ML will reduce the time required for customers to iterate on a machine learning project which typically includes multiple rounds of data preparation and training.

With the preview of managed Apache Spark in Azure ML, customers can use Azure ML notebooks to connect to Spark pools in Azure Synapse Analytics, to do interactive data preparation using PySpark. Customers have the option to configure Spark sessions to quickly experiment and iterate on the data. Once ready, they can leverage Azure ML pipelines to automate their end-to-end ML workflow from data preparation to model deployment all in one environment, while maintaining their data and model lineage. Customers who prefer to train in the Spark environment can choose to install relevant libraries such as Spark MLlib, MMLSpark, etc. to complete their training on Spark pools.

Customers in preview will be able to benefit from the following key capabilities:

Reuse Spark pools from Azure Synapse workspace in Azure ML

Customers can leverage existing Spark pools from Azure Synapse Analytics (Azure Synapse) in Azure ML by just linking their Azure ML and Synapse workspaces via the Azure ML Studio, the Python SDK, or the ARM template. Customers just need to follow the widget in UI or leverage a few lines of code as described in the documentation here.

Once the workspaces are linked, customers can attach existing Spark pools into Azure ML workspace and can also register the supported linked services (data store sources).

Perform interactive data preparation via Spark magic from Azure ML notebooks

Customers can use Azure ML notebooks to start Spark sessions in PySpark via Spark Magic on attached Spark pools. Customers can register Azure ML datasets to load data from storage of choice. For data in Gen1 and Gen2, customers can use their own identities to authenticate access to data by leveraging AML datasets. The attached Spark pools can be used normally in Azure ML experiments, pipelines, and designer. More information on leveraging Spark Magic for data preparation on AML notebooks here

Productionize via Azure ML pipelines to orchestrate E2E ML steps including data preparation

After completing the interactive data preparation, customers can leverage Azure ML pipelines to automate data preparation on Apache Spark runtime as a step in the overall machine learning workflow. Customers can use the SynapseSparkStep for data preparation and choose either TabularDataset or FileDataset as input. Customers can also set up HDFSOutputDatasetConfig to generate the sparkstep output as a FileDataset, to be consumed by the following AzureML pipeline step. More details on How to use Apache Spark (powered by Azure Synapse) in your machine learning pipeline here.

Get started with big data preparation in Azure ML via Apache Spark powered by Azure Synapse

Get started by visiting our documentation and let us know your thoughts. We are committed to making the data preparation experience in Azure ML better for you!

Learn more about the Azure Machine Learning service and get started with a free trial.

Learn more about Azure Synapse big data preparation experience in Azure ML
Learn more about how to use Apache Spark in your machine learning pipelines
Learn more about Apache Spark
Learn more about Azure Synapse Analytics

Re: Introducing Multivariate Anomaly Detection

Acash — Wed, 14 Apr 2021 17:51:32 GMT

Great article. I just had a couple of questions?

1) I see a preprocessing block. What kind of data preprocessing does it support? Also, does it support preprocessing during training as well as inference?

2) After training a model what metric does the service provide for verifying the accuracy?

3) Do you have limitations on the number of sensors or timestamps supported? Do you have any metric on the latency of training or inference?

Analyzing COVID Medical Papers with Azure and Text Analytics for Health

shwars — Tue, 13 Apr 2021 19:42:11 GMT

Automatic Paper Analysis

Automatic scientific paper analysis is fast growing area of studies, and due to recent improvements in NLP techniques is has been greatly improved in the recent years. In this post, we will show you how to derive specific insights from COVID papers, such as changes in medical treatment over time, or joint treatment strategies using several medications:

The main idea the approach I will describe in this post is to extract as much semi-structured information from text as possible, and then store it into some NoSQL database for further processing. Storing information in the database would allow us to make some very specific queries to answer some of the questions, as well as to provide visual exploration tool for medical expert for structured search and insight generation. The overall architecture of the proposed system is shown below:

We will use different Azure technologies to gain insights into the paper corpus, such as Text Analytics for Health, CosmosDB and PowerBI. Now let’s focus on individual parts of this diagram and discuss them in detail.

If you want to experiment with text analytics yourself - you will need an Azure Account. You can always get free trial if you do not have one. And you may also want to check out other AI technologies for developers

COVID Scientific Papers and CORD Dataset

The idea to apply NLP methods to scientific literature seems quite natural. First of all, scientific texts are already well-structured, they contain things like keywords, abstract, as well as well-defined terms. Thus, at the very beginning of COVID pandemic, a research challenge has been launched on Kaggle to analyze scientific papers on the subject. The dataset behind this competition is called CORD (publication), and it contains constantly updated corpus of everything that is published on topics related to COVID. Currently, it contains more than 400000 scientific papers, about half of them - with full text.

This dataset consists of the following parts:

Metadata file Metadata.csv contains most important information for all publications in one place. Each paper in this table has unique identifier cord_uid (which in fact does not happen to be completely unique, once you actually start working with the dataset). The information includes:
- Title of publication
- Journal
- Authors
- Abstract
- Data of publication
- doi
Full-text papers in document_parses directory, than contain structured text in JSON format, which greatly simplifies the analysis.
Pre-built Document Embeddings that maps cord_uids to float vectors that reflect some overall semantics of the paper.

In this post, we will focus on paper abstracts, because they contain the most important information from the paper. However, for full analysis of the dataset, it definitely makes sense to use the same approach on full texts as well.

What AI Can Do with Text?

In the recent years, there has been a huge progress in the field of Natural Language Processing, and very powerful neural network language models have been trained. In the area of NLP, the following tasks are typically considered:

Text classification / intent recognition: In this task, we need to classify a piece of text into a number of categories. This is a typical classification task. Sentiment Analysis; We need to return a number that shows how positive or negative the text is. This is a typical regression task. Named Entity Recognition (NER); In NER, we need to extract named entities from text, and determine their type. For example, we may be looking for names of medicines, or diagnoses. Another task similar to NER is keyword extraction.; Text summarization; Here we want to be able to produce a short version of the original text, or to select the most important pieces of text.; Question Answering; In this task, we are given a piece of text and a question, and our goal is to find the exact answer to this question from text.; Open-Domain Question Answering (ODQA); The main difference from previous task is that we are given a large corpus of text, and we need to find the answer to our question somewhere in the whole corpus.

In one of my previous posts, I have described how we can use ODQA approach to automatically find answers to specific COVID questions. However, this approach is not suitable for serious research.

To make some insights from text, NER seems to be the most prominent technique to use. If we can understand specific entities that are present in text, we could then perform semantically rich search in text that answers specific questions, as well as obtain data on co-occurrence of different entities, figuring out specific scenarios that interest us.

To train NER model, as well as any other neural language model, we need a reasonably large dataset that is properly marked up. Finding those datasets is often not an easy task, and producing them for new problem domain often requires initial human effort to mark up the data.

Pre-Trained Language Models

Luckily, modern transformer language models can be trained in semi-supervised manner using transfer learning. First, the base language model (for example, BERT) is trained on a large corpus of text first, and then can be specialized to a specific task such as classification or NER on a smaller dataset.

This transfer learning process can also contain additional step - further training of generic pre-trained model on a domain-specific dataset. For example, in the area of medical science Microsoft Research has pre-trained a model called PubMedBERT (publication), using texts from PubMed repository. This model can then be further adopted to different specific tasks, provided we have some specialized datasets available.

Text Analytics Cognitive Services

However, training a model requires a lot of skills and computational power, in addition to a dataset. Microsoft (as well as some other large cloud vendors) also makes some pre-trained models available through the REST API. Those services are called Cognitive Services, and one of those services for working with text is called Text Analytics. It can do the following:

Keyword extraction and NER for some common entity types, such as people, organizations, dates/times, etc.
Sentiment analysis
Language Detection
Entity Linking, by automatically adding internet links to some most common entities. This also performs disambiguation, for example Mars can refer to both the planet or a chocolate bar, and correct link would be used depending on the context.

For example, let’s have a look at one medical paper abstract analyzed by Text Analytics:

As you can see, some specific entities (for example, HCQ, which is short for hydroxychloroquine) are not recognized at all, while others are poorly categorized. Luckily, Microsoft provides special version of Text Analytics for Health.

Text Analytics for Health

Text Analytics for Health is a cognitive service that exposes pre-trained PubMedBert model with some additional capabilities. Here is the result of extracting entities from the same piece of text using Text Analytics for Health:

Currently, Text Analytics for Health is available as gated preview, meaning that you need to request access to use it in your specific scenario. This is done according to Ethical AI principles, to avoid irresponsible usage of this service for cases where human health depends on the result of this service. You can request access here.

To perform analysis, we can use recent version Text Analytics Python SDK, which we need to pip-install first:

pip install azure.ai.textanalytics==5.1.0b5

Note: We need to specify a version of SDK, because otherwise we can have current non-beta version installed, which lacks Text Analytics for Health functionality.

The service can analyze a bunch of text documents, up to 10 at a time. You can pass either a list of documents, or dictionary. Provided we have a text of abstract in txt variable, we can use the following code to analyze it:

poller = text_analytics_client.begin_analyze_healthcare_entities([txt]) res = list(poller.result()) print(res)

This results in the following object:

[AnalyzeHealthcareEntitiesResultItem(
  id=0, entities=[
     HealthcareEntity(text=2019, category=Time, subcategory=None, length=4, offset=20, confidence_score=0.85, data_sources=None, 
        related_entities={HealthcareEntity(text=coronavirus disease pandemic, category=Diagnosis, subcategory=None, length=28, offset=25, confidence_score=0.98, data_sources=None, related_entities={}): 'TimeOfCondition'}), 
     HealthcareEntity(text=coronavirus disease pandemic, category=Diagnosis, subcategory=None, length=28, offset=25, confidence_score=0.98, data_sources=None, related_entities={}), 
     HealthcareEntity(text=COVID-19, category=Diagnosis, subcategory=None, length=8, offset=55, confidence_score=1.0, 
        data_sources=[HealthcareEntityDataSource(entity_id=C5203670, name=UMLS), HealthcareEntityDataSource(entity_id=U07.1, name=ICD10CM), HealthcareEntityDataSource(entity_id=10084268, name=MDR), ...

As you can see, in addition to just the list of entities, we also get the following:

Enity Mapping of entities to standard medical ontologies, such as UMLS.
Relations between entities inside the text, such as TimeOfCondition, etc.
Negation, which indicated that an entity was used in negative context, for example COVID-19 diagnosis did not occur.

In addition to using Python SDK, you can also call Text Analytics using REST API directly. This is useful if you are using a programming language that does not have a corresponding SDK, or if you prefer to receive Text Analytics result in the JSON format for further storage or processing. In Python, this can be easily done using requests library:

uri = f"{endpoint}/text/analytics/v3.1-preview.3/entities/ health/jobs?model-version=v3.1-preview.4" headers = { "Ocp-Apim-Subscription-Key" : key } resp = requests.post(uri,headers=headers,data=doc) res = resp.json() if res['status'] == 'succeeded': result = t['results'] else: result = None

(We need to make sure to use the preview endpoint to have access to text analytics for health)

Resulting JSON file will look like this:

{"id": "jk62qn0z", "entities": [ {"offset": 24, "length": 28, "text": "coronavirus disease pandemic", "category": "Diagnosis", "confidenceScore": 0.98, "isNegated": false}, {"offset": 54, "length": 8, "text": "COVID-19", "category": "Diagnosis", "confidenceScore": 1.0, "isNegated": false, "links": [ {"dataSource": "UMLS", "id": "C5203670"}, {"dataSource": "ICD10CM", "id": "U07.1"}, ... ]}, "relations": [ {"relationType": "Abbreviation", "bidirectional": true, "source": "#/results/documents/2/entities/6", "target": "#/results/documents/2/entities/7"}, ...], }

Note: In production, you may want to incorporate some code that will retry the operation when an error is returned by the service. For more guidance on proper implementation of cognitive services REST clients, you can check source code of Azure Python SDK, or use Swagger to generate client code.

Using Cosmos DB to Store Analysis Result

Using Python code similar to the one above we can extract JSON entity/relation metadata for each paper abstract. This process takes quite some time for 400K papers, and to speed it up it can be parallelized using technologies such as Azure Batch or Azure Machine Learning. However, in my first experiment I just run the script on one VM in the cloud, and the data was ready in around 11 hours.

Having done this, we have now obtained a collection of papers, each having a number of entities and corresponding relations. This structure is inherently hierarchical, and the best way to store and process it would be to use NoSQL approach for data storage. In Azure, Cosmos DB is a universal database that can store and query semi-structured data like our JSON collection, thus it would make sense to upload all JSON files to Cosmos DB collection. This can be done using the following code:

coscli = azure.cosmos.CosmosClient(cosmos_uri, credential=cosmos_key) cosdb = coscli.get_database_client("CORD") cospapers = cosdb.get_container_client("Papers") for x in all_papers_json: cospapers.upsert_item(x)

Here, all_papers_json is a variable (or generator function) containing individual JSON documents for each paper. We also assume that you have created a Cosmos DB database called ‘CORD’, and obtained required credentials into cosmos_uri and cosmos_key variables.

After running this code, we will end up with the container Papers will all metadata. We can now work with this container in Azure Portal by going to Data Explorer:

Now we can use Cosmos DB SQL in order to query our collection. For example, here is how we can obtain the list of all medications found in the corpus:

-- unique medication names SELECT DISTINCT e.text FROM papers p JOIN e IN p.entities WHERE e.category='MedicationName'

Using SQL, we can formulate some very specific queries. Suppose, a medical specialist wants to find out all proposed dosages of a specific medication (say, hydroxychloroquine), and see all papers that mention those dosages. This can be done using the following query:

-- dosage of specific drug with paper titles SELECT p.title, r.source.text FROM papers p JOIN r IN p.relations WHERE r.relationType='DosageOfMedication' AND CONTAINS(r.target.text,'hydro')

You can execute this query interactively in Azure Portal, inside Cosmos DB Data Explorer. The result of the query looks like this:

[ { "title": "In Vitro Antiviral Activity and Projection of Optimized Dosing Design of Hydroxychloroquine for the Treatment of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)", "text": "400 mg" },{ "title": "In Vitro Antiviral Activity and Projection of Optimized Dosing Design of Hydroxychloroquine for the Treatment of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)", "text": "maintenance dose" },...]

A more difficult task would be to select all entities together with their corresponding ontology ID. This would be extremely useful, because eventually we want to be able to refer to a specific entity (hydroxychloroquine) regardless or the way it was mentioned in the paper (for example, HCQ also refers to the same medication). We will use UMLS as our main ontology.

--- get entities with UMLS IDs SELECT e.category, e.text, ARRAY (SELECT VALUE l.id FROM l IN e.links WHERE l.dataSource='UMLS')[0] AS umls_id FROM papers p JOIN e IN p.entities

Creating Interactive Dashboards

While being able to use SQL query to obtain an answer to some specific question, like medication dosages, seems like a very useful tool - it is not convenient for non-IT professionals, who do not have high level of SQL mastery. To make the collection of metadata accessible to medical professionals, we can use PowerBI tool to create an interactive dashboard for entity/relation exploration.

In the example above, you can see a dashboard of different entities. One can select desired entity type on the left (eg. Medication Name in our case), and observe all entities of this type on the right, together with their count. You can also see associated UMLS IDs in the table, and from the example above once can notice that several entities can refer to the same ontology ID (hydroxychloroquine and HCQ).

To make this dashboard, we need to use PowerBI Desktop. First we need to import Cosmos DB data - the tools support direct import of data from Azure.

Then we provide SQL query to get all entities with the corresponding UMLS IDs - the one we have shown above - and one more query to display all unique categories. Then we drag those two tables to the PowerBI canvas to get the dashboard shown above. The tool automatically understands that two tables are linked by one field named category, and supports the functionality to filter second table based on the selection in the first one.

Similarly, we can create a tool to view relations:

From this tool, we can make queries similar to the one we have made above in SQL, to determine dosages of a specific medications. To do it, we need to select DosageOfMedication relation type in the left table, and then filter the right table by the medication we want. It is also possible to create further drill-down tables to display specific papers that mention selected dosages of medication, making this tool a useful research instrument for medical scientist.

Getting Automatic Insights

The most interesting part of the story, however, is to draw some automatic insights from the text, such as the change in medical treatment strategy over time. To do this, we need to write some more code in Python to do proper data analysis. The most convenient way to do that is to use Notebooks embedded into Cosmos DB:

Those notebooks support embedded SQL queries, thus we are able to execute SQL query, and then get the results into Pandas DataFrame, which is Python-native way to explore data:

%%sql --database CORD --container Papers --output meds SELECT e.text, e.isNegated, p.title, p.publish_time, ARRAY (SELECT VALUE l.id FROM l IN e.links WHERE l.dataSource='UMLS')[0] AS umls_id FROM papers p JOIN e IN p.entities WHERE e.category = 'MedicationName'

 Here we end up with meds DataFrame, containing names of medicines, together with corresponding paper titles and publishing date. We can further group by ontology ID to get frequencies of mentions for different medications:
unimeds = meds.groupby('umls_id') \
              .agg({'text' : lambda x : ','.join(x), 
                    'title' : 'count', 
                    'isNegated' : 'sum'})
unimeds['negativity'] = unimeds['isNegated'] / unimeds['title']
unimeds['name'] = unimeds['text'] \
                  .apply(lambda x: x if ',' not in x 
                                     else x[:x.find(',')])
unimeds.sort_values('title',ascending=False).drop('text',axis=1)

This gives us the following table:

umls_id	title	isNegated	negativity	name
C0020336	4846	191	0.039414	hydroxychloroquine
C0008269	1870	38	0.020321	chloroquine
C1609165	1793	94	0.052426	Tocilizumab
C4726677	1625	24	0.014769	remdesivir
C0052796	1201	84	0.069942	azithromycin
…	…	…	…	…
C0067874	1	0	0.000000	1-butanethiol

From this table, we can select the top-15 most frequently mentioned medications:

top = { x[0] : x[1]['name'] for i,x in zip(range(15), unimeds.sort_values('title',ascending=False).iterrows()) }

To see how frequency of mentions for medications changed over time, we can average out the number of mentions for each month:

# First, get table with only top medications imeds = meds[meds['umls_id'].apply(lambda x: x in top.keys())].copy() imeds['name'] = imeds['umls_id'].apply(lambda x: top[x]) # Create a computable field with month imeds['month'] = imeds['publish_time'].astype('datetime64[M]') # Group by month medhist = imeds.groupby(['month','name']) \ .agg({'text' : 'count', 'isNegated' : [positive_count,negative_count] })

This gives us the DataFrame that contains number of positive and negative mentions of medications for each month. From there, we can plot corresponding graphs using matplotlib:
medh = medhist.reset_index()
fig,ax = plt.subplots(5,3)
for i,n in enumerate(top.keys()):
    medh[medh['name']==top[n]] \
    .set_index('month')['isNegated'] \
    .plot(title=top[n],ax=ax[i//3,i%3])
fig.tight_layout()

Visualizing Terms Co-Occurrence

Another interesting insight is to observe which terms occur frequently together. To visualize such dependencies, there are two types of diagrams:

Sankey diagram allows us to investigate relations between two types of terms, eg. diagnosis and treatment
Chord diagram helps to visualize co-occurrence of terms of the same type (eg. which medications are mentioned together)

To plot both diagrams, we need to compute co-occurrence matrix, which in the row i and column j contains number of co-occurrences of terms i and j in the same abstract (one can notice that this matrix is symmetric). The way we compute it is to manually select relatively small number of terms for our ontology, grouping some terms together if needed:

treatment_ontology = { 'C0042196': ('vaccination',1), 'C0199176': ('prevention',2), 'C0042210': ('vaccines',1), ... } diagnosis_ontology = { 'C5203670': ('COVID-19',0), 'C3714514': ('infection',1), 'C0011065': ('death',2), 'C0042769': ('viral infections',1), 'C1175175': ('SARS',3), 'C0009450': ('infectious disease',1), ...}

Then we define a function to compute co-occurrence matrix for two categories specified by those ontology dictionaries:
def get_matrix(cat1, cat2):
    d1 = {i:j[1] for i,j in cat1.items()}
    d2 = {i:j[1] for i,j in cat2.items()}
    s1 = set(cat1.keys())
    s2 = set(cat2.keys())
    a = np.zeros((len(cat1),len(cat2)))
    for i in all_papers:
        ent = get_entities(i)
        for j in ent & s1:
            for k in ent & s2 :
                a[d1[j],d2[k]] += 1
    return a

 Here get_entities function returns the list of UMLS IDs for all entities mentioned in the paper, and all_papers is the generator that returns the complete list of paper abstracts metadata.

To actually plot the Sankey diagram, we can use Plotly graphics library. This process is well described here, so I will not go into further details. Here are the results:

Plotting a chord diagram cannot be easily done with Plotly, but can be done with a different library - Chord. The main idea remains the same - we build co-occurrence matrix using the same function described above, passing the same ontology twice, and then pass this matrix to Chord:

def chord(cat): matrix = get_matrix(cat,cat) np.fill_diagonal(matrix,0) names = cat.keys() Chord(matrix.tolist(), names, font_size = "11px").to_html()

The results of chord diagrams for treatment types and medications are below:


Treatment types	Medications

Diagram on the right shows which medications are mentioned together (in the same abstract). We can see that well-known combinations, such as hydroxychloroquine + azitromycin, are clearly visible.

Conclusion

In this post, we have described the architecture of a proof-of-concept system for knowledge extraction from large corpora of medical texts. We use Text Analytics for Health to perform the main task of extracting entities and relations from text, and then a number of Azure services together to build a query took for medical scientist and to extract some visual insights. This post is quite conceptual at the moment, and the system can be further improved by providing more detailed drill-down functionality in PowerBI module, as well as doing more data exploration on extracted entity/relation collection. It would also be interesting to switch to processing full-text articles as well, in which case we need to think about slightly different criteria for co-occurrence of terms (eg. in the same paragraph vs. the same paper).

The same approach can be applied in other scientific areas, but we would need to be prepared to train a custom neural network model to perform entity extraction. This task has been briefly outlined above (when we talked about the use of BERT), and I will try to focus on it in one of my next posts. Meanwhile, feel free to reach out to me if you are doing similar research, or have any specific questions on the code and/or methodology.

Learn about Bot Framework Composer’s new authoring experience and deploy your bot to a telephone

KelvinChen — Wed, 14 Apr 2021 16:41:33 GMT

Customer expectations continue to increase, looking for immediate response and rapid issue resolution, across multiple channels 24/7. Nowhere is this more apparent than the contact center, with this landscape is driving the need for efficiencies, such as reducing call handling times and increasing call deflection rates – all whilst aiming to deliver a personalized and tailored customer experience.

To help respond to this need, we announced the public preview of the telephony channel for Azure Bot Service in February 2021, expanding the already significant number of touch points offered by the service, to include this increasingly critical method of communication.

Built on state-of-the-art speech services

The new telephony channel, combined with our Bot Framework developer platform, makes it easy to rapidly build always-available virtual assistants, or IVR assistants, that provide natural language intent-based call handling and the ability to handle advanced conversation flows, such as context switching and responding to follow up questions and still meeting the goal of reducing operational costs for enterprises.

This new capability combines several of our Azure and AI services, including our state-of-the-art Cognitive Speech Service, enabling fluid, natural-sounding speech that matches the patterns and intonation of human voices through Azure Text-to-Speech neural voices, with Azure Communications Services powering various calling capabilities. The channel also provides support for full duplex conversations and streaming audio over PSTN, support for DTMF, barge-in (allowing a caller to interrupt the virtual assistant) and more. Follow our roadmap and try out one of our samples on the Telephony channel GitHub repository.

Improving our Conversational AI SDK and tools for speech experiences

To compliment the introduction of the telephony channel and ensure our customers can create industry leading experiences, we have added new features to Bot Framework Composer, an open-source conversational authoring tool, featuring a visual canvas, built on top of the Bot Framework SDK, allowing you to extend and customize the conversation with code and pre-built components. Updates to Composer to support speech experiences include,

The ability to add tailored speech responses in seconds, either for a voice only or multi-modal (text and speech) agent.
Addition of global application settings for your bot, allowing you to set a consistent voice font to be used on speech enabled channels, including taking care of setting the required base SSML tags.
Authoring UI helpers that allow you to add additional common SSML (Speech Synthesis Markup Language) tags to control the intonation, speed and even the style of the voice used, including new styles available for our neural voice fonts, such as a dedicated Customer Service style.

Comprehensive Contact Center solution through Dynamics 365

Microsoft announced the expansion of Microsoft Dynamics 365 Customer Service omnichannel capabilities to include a new voice channel, that is built on this telephony channel infrastructure. With native voice, businesses receive seamless, end-to-end experiences within a single solution, ensuring consistent, personalized, and connected support across all channels of engagement. This new voice channel for Customer Service enables an all-in-one customer service solution without fragmentation or manual data integration required, and enables a faster time to value. Learn more here.

Get started building for telephony!

Sign up for Azure trial
Join us on live stream of AI Show on 4/16 11AM PDT
Sign up for conversational AI Ask Microsoft Anything (4/28)
To get started developing a virtual agent, that you can surface via the new telephony channel today, download Bot Framework Composer
Read more about the telephony channel preview, including documentation and samples, visit the Bot Framework telephony channel GitHub repository

Re: Introducing Multivariate Anomaly Detection

Alex Thomas — Tue, 13 Apr 2021 00:59:44 GMT

Fantastic!

Introducing Multivariate Anomaly Detection

Tony_Xing — Mon, 12 Apr 2021 15:11:56 GMT

Microsoft partners and customers have been building metrics monitoring solutions for AIOps and predictive maintenance, by leveraging the easy-to-use time-series anomaly detection Cognitive Service: Anomaly Detector. Because of its ability to analyze time-series individually, Anomaly Detector is benefiting the industry with its simplicity and scalability.

What's new

We are pleased to announce the new multi-variate capability of Anomaly Detector. The new multivariate anomaly detection APIs in Anomaly Detector further enable developers to easily integrate advanced AI of detecting anomalies from groups of metrics into their applications without the need for machine learning knowledge or labeled data. Dependencies and inter-correlations between different signals are now counted as key factors. The new feature protects your mission-critical systems and physical assets, such as software applications, servers, factory machines, spacecraft, or even your business, from failures with a holistic view.

Imagine 20 sensors from an auto engine generating 20 different signals, e.g., vibration, temperature, etc. The readings of those signals individually may not tell you much on system-level issues, but together, could represent the health of the engine. When the synergy of those signals turns odd, the multivariate anomaly detection feature can sense the anomaly like a seasoned floor expert. Moreover, the AI models are trained and customized for your data such that it understands your business. With the new APIs in Anomaly Detector, developers can now easily integrate the multivariate time-series anomaly detection capabilities as well as the interpretability of the anomalies into predictive maintenance solutions, or AIOps monitoring solutions for complex enterprise software, or business intelligence tools.

Customer love

“Medical device production demands unprecedented precision. For this reason, the Siemens Healthineers team uses Multivariate Anomaly Detector (MVAD) in medical device stress tests during the final inspection in the production. We found MVAD easy to use and work almost out of the box with promising performance. With the ready-to-use model, we don't need to develop a custom AD model, which ensures a short time to market. We plan to expand this technology also to other use cases. It is made easy due to good integration into our ML platform and processes.” - Dr. Jens Fürst, Head Digitalization and Automation at Siemens Healthineers

To better understand the health and condition of the aircraft and foresee and fix potential problems before they occur, Airbus deployed Anomaly Detector, part of Cognitive Services, to gather and analyze the telemetry data. It began as a proof of concept of the aircraft-monitoring application by loading telemetry data from multiple flights for analysis and model training. “Early tests have shown that for many cases, the out-of-the-box solution works beautifully, which helps us deploy our solutions faster. I would say that we save up to three months on development for our smaller use cases with Anomaly Detector.”
Marcel Rummens: Product Owner of Internal AI Platform, Airbus

AI horsepower

Time-series anomaly detection is an important research topic in data mining and has a wide range of applications in the industry. Efficient and accurate anomaly detection helps companies to monitor their key metrics continuously and alert for potential incidents on time. In many real-world applications like predictive maintenance and SpaceOps, multiple time-series metrics are collected to reflect the health status of a system. Univariate time-series anomaly detection algorithms can find anomalies for a single metric. However, it could be problematic in deciding whether the whole system is running normally. For example, sudden changes of a certain metric do not necessarily mean failures of the system. As shown in Figure 1, there are obvious boosts in the volume of TIMESERIES RECEIVED and DATA RECEIVED ON FLINK in the green segment, but the system is still in a healthy state as these two features share a consistent tendency. However, in the red segment, GC shows an inconsistent pattern with other metrics, indicating a problem in garbage collection. Consequently, it is essential to take the correlations between different time series into consideration in a multivariate time-series anomaly detection system.Fig.1

In this newly introduced feature, we productized a novel framework — MTAD-GAT (Multivariate Time-series Anomaly Detection via Graph Attention Network), to tackle the limitations of previous solutions. Our method considers each univariate time-series as an individual feature and tries to model the correlations between different features explicitly, while the temporal dependencies within each time-series are modeled at the same time. The key ingredients in our model are two graph attention layers, namely the feature-oriented graph attention layer and the time-oriented graph attention layer. The feature-oriented graph attention layer captures the causal relationships between multiple features, and the time-oriented graph attention layer underlines the dependencies along the temporal dimension. In addition, we jointly train a forecasting-based model and a reconstruction-based model for better representations of time-series data. The two models can be optimized simultaneously by a joint objective function.

The magic behind the scenes can be summarized as follows:

A novel framework to solve the multivariate time-series anomaly detection problem in a self-supervised manner. Our model shows superior performances on two public datasets and establishes state-of-the-art scores in the literature.
For the first time, we leverage two parallel graph attention (GAT) layers to learn the relationships between different time-series and timestamps dynamically. Especially, our model captures the correlations between different time-series successfully without any prior knowledge.
We integrate the advantages of both forecasting-based and reconstruction-based models by introducing a joint optimization target. The forecasting-based model focuses on single-timestamp prediction, while the reconstruction-based model learns a latent representation of the entire time-series.
Our network has good interpretability. We analyze the attention scores of multiple time-series learned by the graph attention layers, and the results correspond reasonably well to human intuition. We also show its capability of anomaly diagnosis.

Multivariate anomaly detection API overview

This new feature has a different workflow compared with the existing univariate feature. There are two phases to obtain the detection results, the training phase, and the inference phase. In the training phase, you need to provide some historical data to let the model learn past patterns. Then in the inference phase, you can call the inference API to acquire detection results of multivariate time-series in a given range.

APIs	Functionality
/multivariate/models	Create and train model using training data
/multivariate/models/{modelid}	Get model info including training status and parameters used in the model
multivariate/models[?$skip][&$top]	List models of a subscription
/multivariate/models/{modelid}/detect	Submit inference task with user's data, this is async
/multivariate/results/{resultid}	Get anomalies + root causes (the contribution scores of each variate for each incident)
multivariate/models/{modelId}	Delete an existing multivariate model according to the modelId
multivariate/models/{modelId}/export	Export Multivariate Anomaly Detection Model as Zip file

Get started!

Learning more from our documentation
QuickStarts: C#, Python, JavaScript, Java
Artificial Intelligence for developers

Re: Introducing QnA Maker managed: now in public preview

HesselW — Fri, 09 Apr 2021 06:57:57 GMT

question wrt performance. My chatbots using qnamaker all suffer from the same issue. When the bots are not used for a while (couple of hours) the qnamaker server seems to idle. The first answer of the service take at least 10 seconds. All calls after take less than a second. Is there some kind of Allways on for the QnaService?

Supercharge Azure ML code development with new VS Code integration

Sid_Unnithan — Thu, 08 Apr 2021 15:25:26 GMT

This post is co-authored by Abe Omorogbe, Program Manager, Azure Machine Learning.

The Azure Machine Learning (Azure ML) team is excited to announce the release of an enhanced developer experience for ‘compute instance’ and ‘notebooks’ users, through a VS Code integration in the Azure ML Studio! It is now easier than ever to work directly on your Azure ML compute instances from within Visual Studio Code, , and with full access to a remote terminal, your favorite VS Code extensions, Git source control UI, and a debugger.

Bringing VS Code to Azure Machine Learning

The Azure Machine Learning and VS Code teams have been working in collaboration over the past couple of months to better understand user workflows for authoring, editing, and managing code files. The demand for VS Code became clear after speaking to a wide variety of users tasked with managing larger projects and operationalizing their models. Users were eager to continue working on their Azure ML compute resources and retain the development context initially defined through the Studio UI.

The first step to enabling a better editing experience for users was to evaluate what was currently used in VS Code. Users were familiar with extensions such as Remote-SSH and , the former used to connect to their remote compute and the latter to author notebook files. The advantage of using Jupyter, JupyterLab, or Azure ML notebooks was that they could be used for all compute instance types without requiring any additional configuration or networking changes.

To enable users to work against their compute instances without requiring SSH or additional networking changes, the Azure ML and VS Code teams built a Notebook-specific compute instance connect experience. The Azure ML extension was responsible for facilitating the connection between VS Code – Jupyter and the compute instance, taking care of authenticating on the user’s behalf. After a month or so of releasing this capability, it was clear that users were excited about connectivity without SSH and being able to work from directly within VS Code. However, working in the editor implied expectations around being able to use other VS code features such as the remote terminal, debugger, and language server. Users expressed their frustration with being limited to working in a single Notebook file, being unable to view files on the remote server, and not being able to use their preferred extensions.

VS Code Integration: Features

Learning from prior releases and talking to users led the Azure ML and VS code teams, to build a complete VS Code experience for compute instances without using SSH. Getting started with this experience is trivial – entry points have been integrated within the Azure ML Studio in both the Compute Instance and Notebooks tabs.

Studio UI Compute Entry Point

Studio UI Notebooks Entry Point

Through this VS Code integration customers will now have access to the following features and benefits:

Full integration with Azure ML file share and notebooks: All file operations in VS Code are fully synced with the Azure ML Studio. For example, if a user drags and drops files from their local machine into VS Code connected to Azure ML, all files will be synced and appear in the Azure ML Studio.
Git UI Experiences: Fully manage Git repos in Azure ML with the rich VS Code source control UI.
Notebook Editor: Seamlessly click out from the Azure ML notebooks and continue to work on notebooks in the new native VS code editor.
Debugging: Use the native debugging in VS Code to debug any training script before submitting it to an Azure ML cluster for batch training.
VS Code Terminal: Work in the VS Code terminal that is fully connected to the compute instance.
VS Code Extension Support: All VS Code extensions are fully supported in VS Code connected to the compute instance.
Enterprise Support: Work with VS Code securely in private endpoints without additional, complicated SSH and networking configuration. AAD credentials and RBAC are used to establish a secure connection to VNET/private link enabled Azure ML workspaces.

VS Code Integration: How it Works

Clicking out to VS Code will launch a desktop VS Code session which initiates a secondary remote connection to the target compute. Within the remote connection window, the Azure ML extension creates a WebSocket connection between your local VS Code client and the remote compute instance.

The connected window now provides you with:

Access to the mounted file share, with consistent syncing between what is seen in Jupyter* and the Azure ML Notebooks experience.
Access to the machine’s local SSD in case you would like to clone and manage repos outside of the shared file share.
The ability to manage repositories through the source control UI.
The ability to create, interact and debug running applications.
A remote terminal for executing commands directly against the remote compute.

Below is a high-level overview of the remote connection

Remote Connection Architecture Diagram (High-Level)

This new connect capability and direct integration in the Azure ML Studio creates a better-together experience between Azure ML and VS Code! When working on your machine learning projects you can get started with a notebook in the Azure ML Studio for early data prep and exploratory work, when you’re ready to start fleshing out the rest of your project, work on multiple file types, and use more advanced editing capabilities and VS Code extension, you can seamlessly transition over to working in VS Code. The retained context and file share usage enables you to move bi-directionally (from notebooks to VS Code and vice-versa) without requiring additional work.

Getting Started

You can initiate the connection to VS Code directly from the Studio UI through either the Compute Instance or Notebook pages. Alternatively, there are routes starting directly within VS Code if you would prefer. Given you have the Azure Machine Learning extension installed, you can find the compute instance in the tree view and right-click on it to connect. You can also invoke the command “Azure ML: Connect to compute instance” and follow the prompts to initiate the connection.

Azure ML extension command

Azure ML extension tree view context menu option

For more details on how you can get started with this experience, please take a look at our public documentation.

Both the Azure ML and VS Code extension teams are always looking for feedback on our current experiences and what we should work on next. If there is anything you would like us to prioritize, please feel free to suggest so via our GitHub repo; if you would like to provide more general feedback, please fill out our survey.

Eleven more languages are generally available for Azure Neural Text-to-Speech

GarfieldHe — Wed, 31 Mar 2021 15:21:04 GMT

This post is co-authored with Lihui Wang, Gang Wang, Xinfeng Chen, Qinying Liao, Garfield He and Sheng Zhao

Six new languages

12 voices from 6 brand-new languages, with one male and one female voice in each language are available now: Nia in ‘cy-GB’ Welsh (United Kingdom), Aled in ‘cy-GB’ Welsh (United Kingdom), Rosa in ‘en-PH’ English (Philippines), James in ‘en-PH’ English (Philippines), Charline in ‘fr-BE’ French (Belgium), Gerard in ‘fr-BE’ French (Belgium), Dena in ‘’nl-BE Dutch (Belgium), Arnaud in ‘nl-BE’ Dutch (Belgium), Polina in ‘uk-UA’ Ukranian (Ukraine), Ostap in ‘uk-UA’ Ukranian (Ukraine), Uzma in ‘ur-PK’ Urdu (Pakistan), and Asad in ‘ur-PK’ Urdu (Pakistan).

Hear the samples below or try them with your own text in our product demo on Azure.

Locale code	Language	Gender	Voice name	Audio sample
cy-GB	Welsh (UK)	Female	cy-GB-NiaNeural	Mae'r ysgol ar agor drwy'r wythnos.
cy-GB	Welsh (UK)	Male	cy-GB-AledNeural	Mae Bangor 8 milltir o Gaernarfon.
en-PH	English (Philippines)	Female	en-PH-RosaNeural	I need to buy a mineral water.
en-PH	English (Philippines)	Male	en-PH-JamesNeural	Let's meet tomorrow at 6 pm.
fr-BE	French (Belgium)	Female	fr-BE-CharlineNeural	On se voit pour dîner demain ?
fr-BE	French (Belgium)	Male	fr-BE-GerardNeural	Il existe 2 manières de participer.
nl-BE	Dutch (Belgium)	Female	nl-BE-DenaNeural	Hij is al urenlang online.
nl-BE	Dutch (Belgium)	Male	nl-BE-ArnaudNeural	Ik vond vele kabouters in hun tuin.
uk-UA	Ukrainian (Ukraine)	Female	uk-UA-PolinaNeural	У Києві завершили реставрацію Андріївської церкви.
uk-UA	Ukrainian (Ukraine)	Male	uk-UA-OstapNeural	Загалом було оновлено 4 395 км доріг.
ur-PK	Urdu (Pakistan)	Female	ur-PK-UzmaNeural	واہ! کیا ہی خوبصورت نظارہ ہے۔
ur-PK	Urdu (Pakistan)	Male	ur-PK-AsadNeural	سورج گرہن پاکستانی وقت کے مطابق شام 6 بج کر 34 منٹ پر شروع ہو گا۔

With this update, Azure TTS now supports 60 languages in total. Check out the figure below for more details or see the full language list.

Five preview languages now GA

Last November, we released 5 languages in preview with 10 voices for European locales. Now these languages become generally available in all Neural TTS regions/datacenters. Azure TTS now has full support for all 24 European languages.

Locale code	Language	Gender	Voice name	Audio samples
et-EE	Estonian (Estonia)	Female	et-EE-AnuNeural	Pese voodipesu kord nädalas või vähemalt kord kahe nädala järel ning ära unusta pesta ka kardinaid.
et-EE	Estonian (Estonia)	Male	et-EE- KertNeural	Ametlikku meetodit sellise pettuse avastamiseks ei olegi olemas.
ga-IE	Irish (Ireland)	Female	ga-IE- OrlaNeural	Tá an scoil sa mbaile ar oscailt arís inniu.
ga-IE	Irish (Ireland)	Male	ga-IE- ColmNeural	Ritheadh próiseas comhairliúcháin faoin scéal sa bhfómhar.
lt-LT	Lithuanian (Lithuania)	Female	lt-LT- OnaNeural	Derinti motinystę ir kūrybą išmokau jau po pirmojo vaiko gimimo.
lt-LT	Lithuanian (Lithuania)	Male	lt-LT- LeonasNeural	Aišku, anksčiau ar vėliau paaiškės tos priežastys.
lv-LV	Latvian (Latvia)	Female	lv-LV-EveritaNeural	Daži tumšās šokolādes gabaliņi dienā ir gandrīz būtiska uztura sastāvdaļa.
lv-LV	Latvian (Latvia)	Male	lv-LV- NilsNeural	Aizvadīto gadu uzņēmums noslēdzis ar 6,3 miljonu eiro zaudējumiem.
mt-MT	Maltese (Malta)	Female	mt-MT-GraceNeural	Fid-diskors tiegħu, is-Segretarju Parlamentari fakkar li dan il-Gvern daħħal numru ta’ liġijiet u inizjattivi li jħarsu lill-annimali.
mt-MT	Maltese (Malta)	Male	mt-MT- JosephNeural	Anki tfajjel tal-primarja jaf li l-popolazzjoni tikber fejn hemm il-prosperità.

How to integrate with the new voices/languages

Azure TTS now covers more languages of the world. Applications using Azure TTS can be easily updated to support coverage of additional countries. All the voices are available in the same API and SDK. Developers can just edit the voice and locale list in their applications to use these new voices without code logic modifications.

For instance, Microsoft Teams auto attendants lets people call your organization and navigate a menu system to speak to the right department, call queue, person, or an operator. It uses Azure TTS to render customized prompts as a call response. To better localize audio prompts for different countries, Teams has been integrated with the new TTS languages to serve more customers around the world.

Want more languages or voices?

If you find that the language which you are looking for is not supported by Azure TTS, reach out to your sales representative, or file a support ticket on Azure. We'd be happy to engage and discuss how to support the languages you need. You can also customize and create a brand voice with your speech data for your apps using the Custom Neural Voice feature.

Tell us your experiences!

By offering more voices across more languages and locales, we anticipate developers across the world will be able to build applications that change experiences for millions. Whether you are building a voice-enabled chatbot or IoT device, an IVR solution, adding read-aloud features to your app, converting e-books to audio books, or even adding Speech to a translation app, you can make all these experiences natural sounding and fun with Neural TTS.

Let us know how you are using or plan to use Neural TTS voices in this form. If you prefer, you can also contact us at mstts [at] microsoft.com. We look forward to hearing about your experience and look forward to developing more compelling services together with you for the developers around the world.

Get started

Add voice to your app in 15 minutes

Explore the available voices in this demo

Build a voice-enabled bot

Deploy Azure TTS voices on prem with Speech Containers

Build your custom voice

Azure Speech and Batch Ingestion

Panos Periorellis — Tue, 30 Mar 2021 20:21:49 GMT

Getting started with Azure Speech and Batch Ingestion Client

Batch Ingestion Client is as a zero-touch transcription solution for all your audio files in your Azure Storage. If you are looking for a quick and effortless way to transcribe your audio files or even explore transcription, without writing any code, then this solution is for you. Through an ARM template deployment, all the resources necessary to seamlessly process your audio files are set-up and set in motion.

Why do I need this?

Getting started with any API requires some amount of time investment in learning the API, understanding its scope, and getting value through trial and error. In order to speed up your transcription solution, for those of you that do not have the time to invest in getting to know our API or related best practices, we created an ingestion layer (a client for batch transcription) that will help you set-up a full blown, scalable and secure transcription pipeline without writing any code.

This is a smart client in the sense that it implements best practices and optimized against the capabilities of the Azure Speech infrastructure. It utilizes Azure resources such as Service Bus and Azure Functions to orchestrate transcription requests to Azure Speech Services from audio files landing in your dedicated storage containers.

Before we delve deeper into the set-up instructions, let us have a look at the architecture of the solution this ARM template builds.

The diagram is simple and hopefully self-explanatory. As soon as files land in a storage container, the Grid Event that indicates the complete upload of a file is filtered and pushed to a Service bus topic. Azure Functions (time triggered by default) pick up those events and act, namely creating Tx requests using the Azure Speech Services batch pipeline. When the Tx request is successfully carried out an event is placed in another queue in the same service bus resource. A different Azure Function triggered by the completion event starts monitoring transcription completion status and copies the actual transcripts in the containers from which the audio file was obtained. This is it. The rest of the features are applied on demand. Users can choose to apply analytics on the transcript, produce reports or redact, all of which are the result of additional resources being deployed through the ARM template. The solution will start transcribing audio files without the need to write any code. If -however- you want to customize further this is possible too. The code is available in this repo.

The list of best practices we implemented as part of the solution are:

Optimized the number of audio files included in each transcription with the view of achieving the shortest possible SAS TTL.
Round Robin around selected regions in order to distribute load across available regions (per customer request)
Retry logic optimization to handle smooth scaling up and transient HTTP 429 errors
Running Azure Functions economically, ensuring minimal execution cost

Setup Guide

The following guide will help you create a set of resources on Azure that will manage the transcription of audio files.

Prerequisites

An Azure Account as well as an Azure Speech key is needed to run the Batch Ingestion Client.

Here are the detailed steps to create a speech resource:

NOTE: You need to create a Speech Resource with a paid (S0) key. The free key account will not work. Optionally for analytics you can create a Text Analytics resource too.

Go to Azure portal
Click on +Create Resource
Type Speech and
Click Create on the Speech resource.
You will find the subscription key under Keys
You will also need the region, so make a note of that too.

To test your account, we suggest you use Microsoft Azure Storage Explorer.

The Project

Although you do not need to download or do any changes to the code you can still download it from GitHub:

git clone https://github.com/Azure-Samples/cognitive-services-speech-sdkcd cognitive-services-speech-sdk/samples/batch/transcription-enabled-storage

Make sure that you have downloaded the ARM Template from the repository.

Batch Ingestion Client Setup Instructions

Click on +Create Resource on Azure portal as shown in the following picture and type ‘ template deployment ’ on the search box.

2. Click on Create Button on the screen that appears as shown below.

3. You will be creating the relevant Azure resources from the ARM template provided. Click on click on the ‘Build your own template in the editor’ link and wait for the new screen to load.

You will be loading the template via the Load file option. Alternatively, you could simply copy/paste the template in the editor.

Saving the template will result in the screen below. You will need to fill in the form provided. It is important that all the information is correct. Let us look at the form and go through each field.

NOTE: Please use short descriptive names in the form for your resource group. Long resource group names may result in deployment error

First pick the Azure Subscription Id within which you will create the resources.
Either pick or create a resource group. [It would be better to have all the resources within the same resource group so we suggest you create a new resource group].
Pick a region [May be the same region as your Azure Speech key].

The following settings all relate to the resources and their attributes

Give your transcription enabled storage account a name [you will be using a new storage account rather than an existing one]. If you opt to use existing one then all existing audio files in that account will be transcribed too.

The following 2 steps are optional. Omitting them will result in using the base model to obtain transcripts. If you have created a Custom Speech model using Speech Studio, then:

Enter optionally your primary Acoustic model ID
Enter optionally your primary Language model ID

If you want us to perform Language identification on the audio prior to transcription you can also specify a secondary locale. Our service will check if the language on the audio content is the primary or secondary locale and select the right model for transcription.

Transcripts are obtained by polling the service. We acknowledge that there is a cost related to that. So, the following setting gives you the option to limit that cost by telling your Azure Function how often you want it to fire.

Enter the polling frequency [There are many scenarios where this would be required to be done couple of times a day]
Enter locale of the audio [you need to tell us what language model we need to use to transcribe your audio.]
Enter your Azure Speech subscription key and Locale information

NOTE: If you plan to transcribe large volume of audio (say millions of files) we propose that you rotate the traffic between regions. In the Azure Speech Subscription Key text box you can put as many keys separated by column ';'. In is important that the corresponding regions (Again separated by column ';') appear in the Locale information text box. For example if you have 3 keys (abc, xyz, 123) for east us, west us and central us respectively then lay them out as follows 'abc;xyz;123' followed by 'east us;west us;central us'

The rest of the settings related to the transcription request. You can read more about those in our docs.

Select a profanity option
Select a punctuation option
Select to Add Diarization [all locales]
Select to Add Word level Timestamps [all locales]

Do you need more than transcription? Do you need to apply Sentiment to your transcript? Downstream analytics are possible too, with Text Analytics Sentiment and Redaction being offered as part of this solution too.

If you want to perform Text Analytics please add those credentials.

Add Text analytics key
Add Text analytics region
Add Sentiment
Add data redaction

If you want to further analytics we could map the transcript json we produce to a DB schema.

Enter SQL DB credential login
Enter SQL DB credential password

You can feed that data to your custom PowerBI script or take the scripts included in this repository. Follow this guide for setting it up.

Press Create to trigger the resource creating process. It typically takes 1-2 mins. The set of resources are listed below.

If a Consumption Plan (Y1) was selected for the Azure Functions, make sure that the functions are synced with the other resources (see this for further details).

To do so, click on your StartTranscription function in the portal and wait until your function shows up:

Do the same for the FetchTranscription function

Important: Until you restart both Azure functions you may see errors.

Running the Batch Ingestion Client

Upload audio files to the newly created audio-input container (results are added to json-result-output and test-results-output containers). Once they are done you can test your account.

Use Microsoft Azure Storage Explorer to test uploading files to your new account. The process of transcription is asynchronous. Transcription usually takes half the time of the audio track to be obtained. The structure of your newly created storage account will look like the picture below.

There are several containers to distinguish between the various outputs. We suggest (for the sake of keeping things tidy) to follow the pattern and use the audio-input container as the only container for uploading your audio.

Customizing the Batch Ingestion Client

By default, the ARM template uses the newest version of the Batch Ingestion Client which can be found in this repository. If you want to customize this further clone the repo

To publish a new version, you can use Visual Studio, right click on the respective project, click publish and follow the instructions.

What to build next

Now that you’ve successfully implemented a speech to text scenario, you can build on this scenario. Take a look at the insights Text Analytics provides from the transcript like caller and agent sentiment, key phrase extraction and entity recognition. If you’re looking specifically to solve for Call centre transcription, review this docs page for further guidance

Re: Form Recognizer now reads more languages, processes IDs and invoices, trains on tables, and mor

Woocash — Sat, 20 Mar 2021 00:00:39 GMT

When will the client library support latest API versions? When I build a model with the labeling tool I get an error saying that the model is built with different version than the client is targeting.

Microsoft named a Leader in 2021 Gartner Magic Quadrant for Cloud AI Developer Services

maddybutzbach — Fri, 19 Mar 2021 16:17:37 GMT

Gartner recently released their Magic Quadrant for 2021 Cloud AI Developer Services. Microsoft is in the Leaders quadrant and was positioned highest on the ability to execute axis. You can download a complimentary copy of the Magic Quadrant for Cloud AI Developer Services for the full report. In this post, we’ll look at why, we think, Microsoft was placed in the Leaders quadrant.

According to the report, “Gartner defines cloud AI developer services (CAIDS) as cloud-hosted or containerized services/models that allow development teams and business users to leverage artificial intelligence models via APIs, SDKs, or applications without requiring deep data science expertise.”

They specifically evaluated services with capabilities in language, vision, and automated machine learning. For Azure, this includes Azure Cognitive Services, Azure Machine Learning, and Microsoft’s conversational AI portfolio. For Power Platform, this includes AI Builder and Power Virtual Agents.

“Gartner believes that enterprise development teams will increasingly incorporate models built using AI and ML into applications. These services currently fall into three main functional areas: language, vision and automated machine learning (autoML). The language services include natural language understanding (NLU), conversational agent frameworks, text analytics, sentiment analysis and other capabilities. The vision services include image recognition, video content analysis and optical character recognition (OCR). The autoML services include automated tools that will let developers do data preparation, feature engineering, create models, deploy, monitor and manage models without having to learn data science.”

Azure AI enables you to develop AI applications on your terms, apply AI responsibly, and deploy mission-critical AI solutions.

Develop on your terms

Azure AI allows you to build AI applications in your preferred software development language and deploy in the cloud, on-premises, or at the edge. Azure provides options for data scientists and developers of all skill levels – no machine learning expertise required. See the Microsoft section of the Magic Quadrant for Cloud AI Developer Services.

Apply AI responsibly

Azure offers tools and resources to help you understand, protect, and control your AI solutions, including responsible ML toolkits, responsible bot development guidelines, tools to help you explain model behavior and test for fairness, and more. We never use your data to train our models, and we keep principles like inclusiveness, fairness, transparency, and accountability in mind at every stage of our AI research, development, and deployment. See the Microsoft section of the Magic Quadrant for Cloud AI Developer Services.

Deploy mission-critical solutions

Azure lets you access the same AI services that power products like Microsoft Teams and Xbox, and that are proven at global scale. Azure leads the industry when it comes to security, and we have the most comprehensive compliance coverage of any cloud service provider. We continue to innovate and our Microsoft Research team has made significant breakthroughs, most recently reaching human parity with image captioning. See the Microsoft section of the Magic Quadrant for Cloud AI Developer Services.

Whether you’re a professional developer or data scientist, or just getting started, we hope that you can use Azure AI services to build impactful AI-powered applications that solve complex problems and enhance customer experience.

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Microsoft. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

The Tenets of Knowledge Management Adoption

Sonia Ang — Fri, 19 Mar 2021 16:00:00 GMT

Knowledge Management Systems and Adoption Key Tenets:

Sonia M. Ang – CSA

In today’s competitive business environment organizations need a clear roadmap that aligns with their training needs and focuses on both short-and long-term objectives. Knowledge Management is a tool that can be implemented to identify and appeal to the training needs of the modern employee. A successful Enterprise Learning system goes beyond the organizational level and allows employees access to information and knowledge, thus creating better alignment at the enterprise level. A well-designed Knowledge Management System can break down barriers by providing partners, clients, and customers with not only essential information and robust training, but also opportunities to promote and inform your organization’s products and services.

Employee retention and customer satisfaction are essential to any organizations long-term success. When considering your organization’s ROI, the benefits of Enterprise Learning are threefold: retention (customers and employees), satisfaction, and improved profitability. Enterprise Learning can be leveraged to provide better development and training opportunities thus promoting a feeling of empowerment amongst your team. Enterprise Learning promotes efficiencies in training, in turn your organization will recognize the cost-saving benefits due to lower employee turnover and customer churn.

Enterprise Knowledge Management Adoption

1. Management Sponsorship and a COE

An executive sponsor provides that critical link between executive leadership and project management and helps support projects successfully to their completion at their expected performance.  Sponsorship of the Enterprise Knowledge Management Project will enhance your product base and create opportunities for your company in the rapidly advancing Knowledge Management sector.  Executive sponsorship will align with our company’s strategy to be experts in Knowledge Management.  Microsoft will be at the forefront of Knowledge Management as organizations rush to adopt more efficient and effective training strategies that better align to the modern worker's needs.

2. Beyond Training but Execution

Training programs at the company level are often too extensive in complexity and the amount of learning material can be downright overwhelming. Training professionals need to venture beyond traditional learning and utilize learning strategies that support the needs of today’s modern professionals. Microlearning, for example, provides a host of benefits to your organization in terms of increased learner participation, memorability of courses, and quick deployment with easy updates to your digital learning assets.

Knowledge Management allows for learning concepts to be extracted from larger training programs and utilized as checklists and instructional videos that are easily accessible at a moment’s notice. When learning material is successfully mined it allows the learning process to be refined, thus challenging concepts can be identified and made easier to process and understand.

3. Collaborative and Social Learning

How does your organization support collaboration, problem-solving and the co-creation of knowledge? A successful Enterprise Learning Strategy will push an organization forward by improving collaboration via robust communities and meaningful discussion forums. Building professional learning communities on platforms like Slack and Microsoft TEAMS can help break the walls down in organizations where ideas and knowledge may be siloed. In addition to collaborative discussion platforms that give all community members a voice, the development of Expert Finders can be an important catalyst for creating a robust culture of collaboration. Hidden ideas and knowledge will organically emerge from people in your organization that may hold previously unearthed niche expertise.

4. Where is the Data?

Generating data for analysis is the foundation of a robust Enterprise Learning strategy. The key to success is to build data collection directly into technical systems supported by a centralized knowledge repository. Data collection in the education sphere has traditionally focused on summative assessments like exams that are intended to measure a learner’s mastery of objectives. Using the specifications in Enterprise Learning provides another set of metrics by allowing an organization to track formative assessments, such as data and social learning activity. For example, these metrics allows an organization to collect new data, adding another layer of data to your knowledge repository to support the creation of more meaningful formative and summative assessments.

5. Reusable Content and Reproducibility

How do organizations move past the traditional content development models where bulky training manuals were the norm? Learning data and insights help organizations to build learning solutions that are reusable and provide 24/7 access to learning. Additionally, a Headless CMS allows your organization’s content creators to move away from the rigid templates that most traditional learning management systems utilize. This means content creators have more control over the quality of their content, and this streamlines the process of creating unique digital learning experiences for both your employees and customers. Perhaps your organization releases a short instructional video on company’s rules and regulations. This learning asset has value as both a stand-alone content object in a knowledge base as well as a learning module in a more extensive communications course. Content creators in your organization, ranging from instructional designers to marketing professionals, often create multiple versions of the same material. Creating content that is reusable and not redundant is more efficient as you are not reinventing the wheel. Additionally, you are not burdened with trying to maintain and keep up-to-date multiple versions of the same learning assets.

6. “Findability” of Learning Assets: Digitization and Technology

The successful implementation of KM tools can enhance the user experience through successful mining and classification of metadata. The power of KM is powerful in that it provides a taxonomy, ontology, and a finely tuned search system. A well-designed metadata strategy will take advantage of your learning assets which may include courses, webinars, professional learning communities (PLCs) and subject matter experts in your organization. This myriad of data exits across multiple systems in your organization. Using metadata that is contextualized and consistent will ensure that your data is findable. Taking it one-step further ontologies can be tapped to support a complete network of shareable and reusable knowledge across a domain for each unique user. 

Summary

The adoption of an Enterprise Management system can reframe your organization’s knowledge and learning infrastructure. Modern learners need to consume information quickly thus allowing them to efficiently apply and master essential skills and strategies. A well-engineered Enterprise Learning Plan that focuses on empowering your community will result in higher learner engagement, enhanced workplace skills and a robust culture-of-knowledge across your organization. Curiosity will drive success rather than traditional command and control training strategies. Empower your community by giving them the autonomy to self-direct their own learning. As a result, your organization will enjoy increased engagement, enhanced workforce skills and, in turn, a robust learning culture will grow and flourish

Re: Form Recognizer now reads more languages, processes IDs and invoices, trains on tables, and mor

francescofucci — Thu, 18 Mar 2021 15:19:52 GMT

is there a manual about Draw Region feature?

Re: Introducing QnA Maker managed: now in public preview

julianportelli — Wed, 17 Mar 2021 08:42:28 GMT

Found a solution! See https://techcommunity.microsoft.com/t5/cognitive-services/qna-maker-bot-returns-both-short-and-long-answer-in-test-in-web/m-p/1882908

Extract Data from PDFs using Form Recognizer with Code or Without!

cassieview — Tue, 16 Mar 2021 16:43:56 GMT

Form Recognizer is a powerful tool to help build a variety of document machine learning solutions. It is one service however its made up of many prebuilt models that can perform a variety of essential document functions. You can even custom train a model using supervised or unsupervised learning for tasks outside of the scope of the prebuilt models! Read more about all the features of Form Recognizer here. In this example we will be looking at how to use one of the prebuilt models in the Form Recognizer service that can extract the data from a PDF document dataset. Our documents are invoices with common data fields so we are able to use the prebuilt model without having to build a customized model.

Sample Invoice:

After we take a look at how to do this with Python and Azure Form Recognizer, we will take a look at how to do the same process with no code using the Power Platform services: Power Automate and Form Recognizer built into AI Builder. In the Power Automate flow we are scheduling a process to happen every day. What the process does is look in the raw blob container to see if there is new files to be processed. If there is new files to be processed it gets all blobs from the container and loops through each blob to extract the PDF data using a prebuilt AI builder step. Then it deletes the processed document from the raw container. See what it looks like below.

Power Automate Flow:

Prerequisites for Python

Azure Account Sign up here!
Anaconda and/or VS Code
Basic programming knowledge

Prerequisites for Power Automate

Power Automate Account Sign up here!
No programming knowledge

Process PDFs with Python and Azure Form Recognizer Service

Create Services

First lets create the Form Recognizer Cognitive Service.

Go to portal.azure.com to create the resource or click this link.

Now lets create a storage account to store the PDF dataset we will be using in containers. We want two containers, one for the processed PDFs and one for the raw unprocessed PDF.

Create an Azure Storage Account
Create two containers: processed, raw

Upload data

Upload your dataset to the Azure Storage raw folder since they need to be processed. Once processed then they would get moved to the processed container.

The result should look something like this:

Create Notebook and Install Packages

Now that we have our data stored in Azure Blob Storage we can connect and process the PDF forms to extract the data using the Form Recognizer Python SDK. You can also use the Python SDK with local data if you are not using Azure Storage. This example will assume you are using Azure Storage.

Create a new Jupyter notebook in VS Code.
Install the Python SDK

!pip install azure-ai-formrecognizer --pre

Then we need to import the packages.

import os
from azure.core.exceptions import ResourceNotFoundError
from azure.ai.formrecognizer import FormRecognizerClient
from azure.core.credentials import AzureKeyCredential
import os, uuid
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__

Create FormRecognizerClient

Update the endpoint and key with the values from the service you created. These values can be found in the Azure Portal under the Form Recongizer service you created under the Keys and Endpoint on the navigation menu.

endpoint = "<your endpoint>"
key = "<your key>"

We then use the endpoint and key to connect to the service and create the FormRecongizerClient

form_recognizer_client = FormRecognizerClient(endpoint, AzureKeyCredential(key))

Create the print_results helper function for use later to print out the results of each invoice.

def print_result(invoices, blob_name):
    for idx, invoice in enumerate(invoices):
        print("--------Recognizing invoice {}--------".format(blob_name))
        vendor_name = invoice.fields.get("VendorName")
        if vendor_name:
            print("Vendor Name: {} has confidence: {}".format(vendor_name.value, vendor_name.confidence))
        vendor_address = invoice.fields.get("VendorAddress")
        if vendor_address:
            print("Vendor Address: {} has confidence: {}".format(vendor_address.value, vendor_address.confidence))
        customer_name = invoice.fields.get("CustomerName")
        if customer_name:
            print("Customer Name: {} has confidence: {}".format(customer_name.value, customer_name.confidence))
        customer_address = invoice.fields.get("CustomerAddress")
        if customer_address:
            print("Customer Address: {} has confidence: {}".format(customer_address.value, customer_address.confidence))
        customer_address_recipient = invoice.fields.get("CustomerAddressRecipient")
        if customer_address_recipient:
            print("Customer Address Recipient: {} has confidence: {}".format(customer_address_recipient.value, customer_address_recipient.confidence))
        invoice_id = invoice.fields.get("InvoiceId")
        if invoice_id:
            print("Invoice Id: {} has confidence: {}".format(invoice_id.value, invoice_id.confidence))
        invoice_date = invoice.fields.get("InvoiceDate")
        if invoice_date:
            print("Invoice Date: {} has confidence: {}".format(invoice_date.value, invoice_date.confidence))
        invoice_total = invoice.fields.get("InvoiceTotal")
        if invoice_total:
            print("Invoice Total: {} has confidence: {}".format(invoice_total.value, invoice_total.confidence))
        due_date = invoice.fields.get("DueDate")
        if due_date:
            print("Due Date: {} has confidence: {}".format(due_date.value, due_date.confidence))

Connect to Blob Storage

Now lets connect to our blob storage containers and create the BlobServiceClient. We will use the client to connect to the raw and processed containers that we created earlier.

# Create the BlobServiceClient object which will be used to get the container_client
connect_str = "<Get connection string from the Azure Portal>"
blob_service_client = BlobServiceClient.from_connection_string(connect_str)

# Container client for raw container.
raw_container_client = blob_service_client.get_container_client("raw")

# Container client for processed container
processed_container_client = blob_service_client.get_container_client("processed")

# Get base url for container.
invoiceUrlBase = raw_container_client.primary_endpoint
print(invoiceUrlBase)

HINT: If you get a "HttpResponseError: (InvalidImageURL) Image URL is badly formatted." error make sure the proper permissions to access the container are set. Learn more about Azure Storage Permissions here

Extract Data from PDFs

We are ready to process the blobs now! Here we will call list_blobs to get a list of blobs in the raw container. Then we will loop through each blob, call the begin_recognize_invoices_from_url to extract the data from the PDF. Then we have our helper method to print the results. Once we have extracted the data from the PDF we will upload_blob to the processed folder and delete_blob from the raw folder.

print("\nProcessing blobs...")

blob_list = raw_container_client.list_blobs()
for blob in blob_list:
    invoiceUrl = f'{invoiceUrlBase}/{blob.name}'
    print(invoiceUrl)
    poller = form_recognizer_client.begin_recognize_invoices_from_url(invoiceUrl)

    # Get results
    invoices = poller.result()

    # Print results
    print_result(invoices, blob.name)

    # Copy blob to processed
    processed_container_client.upload_blob(blob, blob.blob_type, overwrite=True)

    # Delete blob from raw now that its processed
    raw_container_client.delete_blob(blob)

Each result should look similar to this for the above invoice example:

The prebuilt invoices model worked great for our invoices so we don't need to train a customized Form Recognizer model to improve our results. But what if we did and what if we didn't know how to code?! You can still leverage all this awesomeness in AI Builder with Power Automate without writing any code. We will take a look at this same example in Power Automate next.

Use Form Recognizer with AI Builder in Power Automate

You can achieve these same results using no code with Form Recognizer in AI Builder with Power Automate. Lets take a look at how we can do that.

Create a New Flow

Log in to Power Automate
Click Create then click Scheduled Cloud Flow. You can trigger Power Automate flows in a variety of ways so keep in mind that you may want to select a different trigger for your project.
Give the Flow a name and select the schedule you would like the flow to run on.

Connect to Blob Storage

Click New Step
List blobs Step
- Search for Azure Blob Storage and select List blobs
- Select the ellipsis click Create new connection if your storage account isn't already connected
  - Fill in the Connection Name, Azure Storage Account name (the account you created), and the Azure Storage Account Access Key (which you can find in the resource keys in the Azure Portal)
  - Then select Create
- Once the storage account is selected click the folder icon on the right of the list blobs options. You should see all the containers in the storage account, select raw.

Your flow should look something like this:

Loop Through Blobs to Extract the Data

Click the plus sign to create a new step
Click Control then Apply to each
Select the textbox and a list of blob properties will appear. Select the value property
Next select add action from within the Apply to each Flow step.
Add the Get blob content step:
- Search for Azure Blob Storage and select Get blob content
- Click the textbox and select the Path property. This will get the File content that we will pass into the Form Recognizer.
Add the Process and save information from invoices step:
- Click the plus sign and then add new action
- Search for Process and save information from invoices
- Select the textbox and then the property File Content from the Get blob content section
Add the Copy Blob step:
- Repeat the add action steps
- Search for Azure Blob Storage and select Copy Blob
- Select the Source url text box and select the Path property
- Select the Destination blob path and put /processed for the processed container
- Select Overwrite? dropdown and select Yes if you want the copied blob to overwrite blobs with the existing name.
Add the Delete Blob step:
- Repeat the add action steps
- Search for Azure Blob Storage and select Delete Blob
- Select the Blob text box and select the Path property

The Apply to each block should look something like this:

Save and Test the Flow
- Once you have completed creating the flow save and test it out using the built in test features that are part of Power Automate.

This prebuilt model again worked great on our invoice data. However if you have a more complex dataset, use the AI Builder to label and create a customized machine learning model for your specific dataset. Read more about how to do that here.

Conclusion

We went over a fraction of the things that you can do with Form Recognizer so don't let the learning stop here! Check out the below highlights of new Form Recognizer features that were just announced and the additional doc links to dive deeper into what we did here.

Additional Resources

New Form Recognizer Features

What is Form Recognizer?

Quickstart: Use the Form Recognizer client library or REST API

Tutorial: Create a form-processing app with AI Builder

AI Developer Resources page

AI Essentials video including Form Recognizer

Re: Computer Vision for spatial analysis at the Edge

hussnain_ahmed — Thu, 11 Mar 2021 19:26:27 GMT

recent announcement of Azure Percept, capable of running Spatial Analysis suites edge environments much better. Is Azure Percept DK good for production deployments or as a dev kit, it is only meant for prototyping? Secondly, do you have any roadmap for adding intel Movidius Myriad x devices to the list of recommended devices for the spatial analysis? The price of NVIDIA T4 based hardware is heavy on the business case for spatial analysis applications. Also, such servers are not suited to many edge environments.

Model understanding with Azure Machine Learning

mithigpe — Tue, 16 Mar 2021 19:05:14 GMT

This post is co-authored by Mehrnoosh Sameki, Program Manager, Azure Machine Learning.

Overview

Model interpretability and fairness are part of the ‘Understand’ pillar of Azure Machine Learning’s Responsible ML offerings. As machine learning becomes ubiquitous in decision-making from the end-user utilizing AI-powered applications to the business stakeholders using models to make data-driven decisions, it is necessary to provide tools at scale for model transparency and fairness.

Explaining a machine learning model and performing fairness assessment is important for the following users:

Data scientists and model evaluators - At training time to help them to understand their model predictions and assess the fairness of their AI systems, enhancing their ability to debug and improve models.
Business stakeholders and auditors - To build trust with defined ML models and deploy them more confidently.

Customers like Scandinavian Airlines (SAS) and Ernst & Young (EY) put interpretability and fairness packages to the test to be able to deploy models more confidently.

SAS used interpretability to confidently identify fraud in its EuroBonus loyalty program. SAS data scientists could debug and verify model predictions using interpretability. They produced explanations about model behavior that gave stakeholders confidence in the machine learning models and assisted with meeting regulatory requirements.
EY utilized fairness assessment and unfairness mitigation techniques with real mortgage adjudication data to improve the fairness of loan decisions from having an accuracy disparity of 7 percent between men and women to less than 0.5 percent.

We are releasing enhanced experiences and feature additions for the interpretability and fairness toolkits in Azure Machine Learning, to empower more ML practitioners and teams to build trust with AI systems.

Model understanding using interpretability and fairness toolkits

These two toolkits can be used together to understand model predictions and mitigate unfairness. For this demonstration, we shall take a look at a loan allocation scenario. Let’s say that the label indicates whether each individual repaid a loan in the past. We will use the data to train a predictor to predict whether previously unseen individuals will repay a loan or not. The assumption is that the model predictions are used to decide whether an individual should be offered a loan.

Identify your model's fairness issues

Our revamped fairness dashboard can help uncover the harm of allocation which leads to the model unfairly allocating loans among different demographic groups. The dashboard can additionally uncover harm of quality of service which leads to a model failing to provide the same quality of service to some people as they do to others. Using the fairness dashboard, you can identify if our model treats different demographics of sex unfairly.

Dashboard configurations

When you first load the fairness dashboard, you need to configure it with desired settings, including:

selection of your sensitive demographic of choice (e.g., sex[1])
model performance metric (e.g., accuracy)
fairness metric (e.g., demographic parity difference).

Model assessment view

After setting the configurations, you will land on a model assessment view where you can see how the model is treating different demographic groups.

Our fairness assessment shows an 18.3% disparity in the selection rate (or demographic group difference). According to that insight, 18.3% more males are receiving qualifications for loan acceptance compared to females. Now that you’ve seen some unfairness indicators in your model, you can next use our interpretability toolkit to understand why your model is making such predictions.

Diagnose your model’s predictions

The new revamped interpretability dashboard greatly improves the user experience of the previous dashboard. In the loan allocation scenario, you can understand how a model treats female loan applicants differently than male loan applicants using the interpretability toolkit:

Dataset cohort creation: You can slice and dice your data into subgroups (e.g., female vs. male vs. unspecified) and investigate or compare your model’s performance and explanations across them.
Model performance tab: With the predefined female and male cohorts, we can observe the different prediction distributions between males and female cohorts, with females experiencing higher probability rates of being rejected for a loan.
Dataset explorer tab: Now that you have seen in the model performance tab how females are rejected at a higher rate than males, you can use the data explorer tab to observe the ground truth distribution between males and females. For males, the ground truth data is well balanced between those receiving a rejection or approval whereas, for females, the ground truth data is heavily skewed towards rejection thereby explaining how the model could come to associate the label ‘female’ with rejection.
Aggregate feature importance tab: Now we observe which top features contribute to the model’s overall prediction (also called global explanations) towards loan rejection. We sort our top feature importances by the Female cohort, which indicates that while the feature for “Sex” is the second most important feature to contribute towards the model’s predictions for individuals in the female cohort, they do not influence how the model makes predictions for individuals in the male cohort. The dependence plot for the feature “Sex” also shows that only the female group has positive feature importance towards the prediction of being rejected for a loan, whereas the model does not look at the feature “Sex” for males when making predictions.
Individual feature importance & What-If tab: Drilling deeper into the model’s prediction for a specific individual (also called local explanations), we look at the individual feature importances for only the Female cohort. We select an individual who is at the threshold of being accepted for a loan by the model and observe which features contributed towards her prediction of being rejected. “Sex” is the second most important feature contributing towards the model prediction for this individual. The Individual Conditional Expectation (ICE) plot calculates how a perturbation for a given feature value across a range can impact its prediction. We select the feature “Sex” and can see that if this feature had been flipped to male, the probability of being rejected is lowered drastically. We create a new hypothetical What-If point from this individual data point and switch only the “Sex” from female to male, and observe that without changing any other feature related to financial competency, the model now predicts that this individual will have their loan application accepted.

Once some potential fairness issues are observed and diagnosed, you can move to mitigate those unfairness issues.

Mitigate unfairness issues in your model

The unfairness mitigation part is powered by the Fairlearn open-source package which includes two types of mitigation algorithms: postprocessing algorithms (ThresholdOptimizer) and reduction algorithms (GridSearch, ExponentiatedGradient). Both operate as “wrappers” around any standard classification or regression algorithm. GridSearch, for instance, treats any standard classification or regression algorithm as a black box, and iteratively (a) re-weight the data points and (b) retrain the model after each re-weighting. After 10 to 20 iterations, this process results in a model that satisfies the constraints implied by the selected fairness metric while maximizing model performance. ThresholdOptimizer on the other hand takes as its input a scoring function that underlies an existing classifier and identifies a separate threshold for each group to optimize the performance metric, while simultaneously satisfying the constraints implied by the selected fairness metric.

The fairness dashboard also enables the comparison of multiple models, such as the models produced by different learning algorithms and different mitigation approaches. Bypassing the dominated models of GridSearch for instance, you can see the unmitigated model on the upper right side (with the highest accuracy and highest demographic parity difference) and can click on any of the mitigated models to observe them further. This allows you to examine trade-offs between performance and fairness.

Comparing results of unfairness mitigation

After applying the unfairness mitigation, we go back to the interpretability dashboard and compare the unmitigated model with the mitigated model. In the figure below, we see a more even probability distribution for the female cohort for the mitigated model on the right:

Revisiting the fairness assessment dashboard, we also see a drastic decrease in demographic parity difference from 18.8% (unmitigated model) to 0.412% (mitigated model):

Saving model explanations and fairness metrics to Azure Machine Learning Run History

Azure Machine Learning’s (AzureML) interpretability and fairness toolkits can be run both locally and remotely. If run locally, the libraries will not contact any Azure services. Alternatively, you can run the algorithms remotely on AzureML compute and log all the explainability and fairness information into AzurML’s run history via the AzureML SDK to save and share them with other team members or stakeholders in AzureML studio.

Azure ML’s Automated ML supports explainability for its best model as well as on-demand explainability for any other models generated by Automated ML.

Learn more

Explore this scenario and other sample notebooks in the Azure Machine Learning sample notebooks GitHub.

Learn more about the Azure Machine Learning service.

Learn more about Responsible ML offerings in Azure Machine Learning.

Learn more about interpretability and fairness concepts and see documentation on how-to guides for using interpretability and fairness in Azure Machine Learning.

Get started with a free trial of the Azure Machine Learning service.

[1] This dataset is from the 1994 US Census Bureau Database where “sex” in the data was limited to binary categorizations.

Re: Form Recognizer now reads more languages, processes IDs and invoices, trains on tables, and mor

smf723 — Wed, 10 Mar 2021 01:44:58 GMT

So we're 9 months since the last release of the Form Recognizer (2.0). Should we be expecting a 12 month cadence for releases? Like most organizations we can't use "preview" software in production.

Advance Resource Access Governance for AML

mufy — Thu, 11 Mar 2021 09:28:08 GMT

Access control is a fundamental building block for enterprise customers, where protecting assets at various levels is absolutely necessary to ensure that only the relevant people with certain positions of authority are given access with different privileges. This is more so prevalent in machine learning, where data is absolutely essential in building ML models, and companies are highly cautious about how the data is accessed and managed, especially with the introduction of GDPR. We are seeing an increasing number of customers seeking for explicit control of not only the data, but various stages of the machine learning lifecycle, starting from experimentation and all the way to operationalization. Assets such as generated models, cluster creation and model deployment require to be governed to ensure that controls are in line with the company’s policy.

Azure traditionally provides Role-based Access Control [1], which helps to manage access to resources; who can access these and what they can access. This is primarily achieved via the concept of roles. A role defines a collection of permissions.

Existing Roles in AML

Azure Machine Learning provides three roles [3] for enterprise customers to provision as a coarse-grained access control, which is designed for simplicity in mind. The first role (Owner) has the highest level of privileges, that grants full control of the workspace. This is followed by a Contributor, which is a bit more restricted role that prevents users from changing role assignment. Reader having the most restrictive permissions and is typically read or view only (see figure 1 below).

Figure 1 - Existing AML roles

What we have found with the customers is that while Coarse-grained Access Control immensely simplifies the management of the roles, and works quite well with a small team, primarily working in the experimentation environment. However, when a company decides to operationalize the ML work, especially in the enterprise space, these roles become far too broad, and too simplistic. In the enterprise space, the deployment tends to have several stages (such as dev, test, pre-prod, prod, etc.), and require various skillset (data scientist, data engineer, etc.) with a greater control in each stage. For example, a Data Scientist may not operate in the production environment. A Data Engineer can only provision resources and should not have the ability to commission and decommission training clusters. Such governance policies are crucial for companies to be enforced and monitored to maintain integrity of their business and IT processes.

Unfortunately, such requirements cannot be captured with the existing roles. Enterprise needs a better mechanism to define policies for various assets in AML to satisfy their business specific requirements.

This is where the new exciting feature of advanced Role-based Access Control really shines. It is based on Fine-grained Access Control at component level (see figure 2) with a number of pre-built out of the box roles, plus the ability to create custom roles that can capture more complex governance access processes and enforce them.

Advance Fine-grained Role-based Access Control

The new advance Role-based Access Control feature of AML is really going to solve many of the enterprise problems around the ability to restrict or grant user permissions for various components. Azure AML currently defines 16 components with varying permissions.

Figure 2 - Components Level RBAC

Each component defines a list of actions such as read, write, delete, etc. These actions can then be amalgamated together to create a custom specific role. To illustrate this with an example of a list of actions currently available for a Datastore component (see Figure 3 below).

Figure 3 - Datastore Actions

A datastore along with Dataset are important concepts in Azure Machine Learning, since they provide access to various data sources, with lineage and tracking ability. Many enterprises have built global Datalake that contain terabytes of data which can contain highly sensitive information. Companies are quite protective of who can access these data, along with various business justifications for how these data are being accessed/used. It is therefore imperative that a tighter access control is mandated for a specific role, such as a Data Engineer to accomplish such a task.

Fortunately, AML advance access control provide custom roles. to cater for their company specific access control, which may be a hybrid of these roles. For such requirements, Azure caters for custom roles.

Custom Role

Custom role [4] allows creation of Fine-grained Access Control on various components, such as the workspace, datastore, etc.

Can be any combination of data or control plane actions that AzureML+AISC support.
Useful for creating scoped roles to a specific action like an MLOps Engineer

These controls are defined in a JSON policy definition, for example.

{ "Name": "Data Scientist", "IsCustom": true, "Description": "Can run experiment but can't create or delete datastore.", "Actions": ["*"], "NotActions": [ "Microsoft.MachineLearningServices/workspaces/*/delete", "Microsoft.MachineLearningServices/workspaces/ datastores/write", "Microsoft.MachineLearningServices/workspaces/ datastores /delete", “Microsoft.MachineLearningServices/workspaces/datastores/write”, "Microsoft.Authorization/*/write" ], "AssignableScopes": [ "/subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>" ] }

The above code defines a Data Scientist who can run an experiment but cannot create or delete a Datastore. This role can be created using the Azure CLI (az role definition create -role-definition filename), however, the CLI ML extension needs to be installed first.

Role Operation Workflow

In an organization, the following activities are to be undertaken by various role owners.

Sub admin comes in for an enterprise and requests Amlcompute quota
They create an RG and a workspace for a specific team, and also set workspace level quota
The team lead (aka workspace admin), comes in and starts creating compute within the quota that the sub admin defined for that workspace
Data Scientist comes in and uses the compute that workspace admin created for them (clusters or instances).

Roles for Enterprise

AML provides a single environment for doing end-to-end experimentation to operationalization. For a start-up this is really useful as they tend to operate in a very agile manner, where many iterations can happen in a short period of time and having the ability to quickly move from ideation to production really reduces their cycle time. Unfortunately, this may not be the case for the enterprise customers, where they would typically be using either two or three environments to carry out their production workload such as: Dev, QA and Prod.

Dev is used to do the experimentation, while QA is catered for satisfying various functional and non-functional requirements, followed by Prod for deployment into the production for consumer usage.

The environments would also have various roles to carry out different activities, such as Data Scientist, Data Engineer and MLOps Engineer (see figure 8 below).

Figure 8 - Enterprise Roles

A Data Scientist normally operates in the Dev environment and has full access to all the permissions related to carrying out experiments, such as provisioning training clusters, building models, etc. While some permissions are granted in the QA environment, primarily related model testing and performance, and very minimal access to the Prod environment, mainly telemetry (see below Table 1).

A Data Engineer on the other hand primarily operates in the Build and QA environment. The main focus is related to the data handling, such as data loading, doing some data wrangling, etc. They have restricted access in the Prod environment.

Table 1 - Role/environment Matrix

An MLOps Engineer has some permission in the Dev environment, but full permissions in the QA and Prod. This is because an MlOPs Engineer is tasked with building the pipeline, gluing things together, and ultimately deploying models in production.

The interesting part is how do all these roles and environments and other components fit together in Azure to provide the much-needed access governance for the enterprise customers.

Enterprise AML Roles Deployment

It is impressive for enterprises to be able to model these complex roles/environments mapping as shown in Table one. Fortunately these can be achieved in Azure using a combination of AD groups, roles and resource groups.

Figure 9 - Enterprise AML Roles Deployment

Fundamentally, Azure Active Directory groups play a major part in gluing all these components together to make it functional.

First step is to group the users specific to role(s) in a “Role AD group” for a given persona (DS, DE, etc.,). Then assign roles with various RBAC actions (Data Writer, MLContributor, etc.) to this AD group. All these users will now inherit the permissions specific to this role(s). Multiple AD groups will be created for different persona roles.

Separate AD groups (‘AD group for Environment’) are created for each environment (i.e. Dev, QA and Prod), the Role AD Groups are added to these Environment AD groups. This creates a mapping of users belonging to a specific role persona with given permissions to an environment.

The ‘AD group for Environment’ is then assigned to a resource group, which contains a specific AML Workspace. This ensures that the role permissions assigned to users will be enforced at the workspace level.

Summary

In this blog, we have discussed the new advance Role-based Access Control, and how it is being applied in a complex enterprise with various environments with different user personas.

The important point to note is the flexibility that comes with this new feature which can operate at any of the 16 AML components and be able to define Fine-grained Access Control for each through custom roles, and out of box four roles which should be sufficient for the majority of the customers.

References

[1] https://docs.microsoft.com/en-us/azure/role-based-access-control/overview

[2] https://azure.microsoft.com/en-gb/services/machine-learning/

[3] https://docs.microsoft.com/en-us/azure/machine-learning/concept-enterprise-security

[4] https://docs.microsoft.com/en-us/azure/role-based-access-control/custom-roles

Additional Links:

https://docs.microsoft.com/en-us/azure/machine-learning/how-to-assign-roles

co-author: @Nishank Gupt and @John Wu

Improving collaboration and productivity in Azure Machine Learning

abeomor — Wed, 10 Mar 2021 18:28:13 GMT

This post is co-authored by Sharon Xu Program Manager, Azure Notebooks.

Today we are very proud to announce the next set of productivity features and improvements for the notebook experience. Since we announced the GA release of Notebooks in Azure Machine Learning (Azure ML), we have learned a lot from our customers. Over the past few months, we have incrementally improved the notebook experience while simultaneously contributing back to the open source nteract project. The Azure ML team recently released a robust set of new functionalities designed to improve data scientist productivity and collaboration in Azure ML Notebooks.

Data scientist & Developer Productivity

We have spoken to several data scientists and developers to fully understand the additional features needed to improve productivity while developing machine learning projects. From feedback, we have found that users constantly needed the following enhancements to speed up their workflow: a clear indication that a cell has finished running, a way to templatize common code excerpts, a way to check variable contents, and more. The following list is a culmination of the most highly requested productivity features:

Cell Status Bar. The status bar located in each cell indicates the cell state: whether a cell has been queued, successfully executed, or run into an error. The status bar also displays the execution time of the last run.
Variable Explorer. provides a quick glance into the data type, size, and contents of your variables and dataframes, allowing for quicker and simpler debugging.

Figure 1: (1) Cell status bar (2) Variable explorer

Notebook snippets (preview). Common Azure ML code excerpts are now available at your fingertips. Navigate to the code snippets panel, accessible via the toolbar, or activate the in-code snippets menu using Ctrl + Space.

Figure 2 (1) Notebook snippets panel, showing all useful snippets

IntelliCode. IntelliCode provides intelligent auto-completion suggestions using an ML algorithm that analyzes the context of your notebook code. IntelliCode suggestions are designated with a star.

Figure 3: IntelliCode in Azure ML Notebooks

Keyboard shortcuts with full Jupyter parity. Azure ML now supports all the keyboard shortcuts available in Jupyter and more.
Table of Contents. For large notebooks, the Table of Contents panel then allows you to navigate to the desired section. The sections of the notebook are designated by the Markdown headers.
Markdown Side-by-side Editor in Notebooks. Within each notebook, the new side-by-side editor allows you to view the rendered results of your Markdown cells directly in your notebook editing.

Figure 4: (1) Table of content pane (2) Markdown side by side

Collaboration and Sharing

An increasing number of data scientists and developers are creating notebooks collaboratively and sharing these notebooks across their team We heard feedback that most users feel like they are missing adequate tools to edit notebooks simultaneously or share their notebooks with a broader audience. Users often resort to screen shares and calls to complete or present work within a notebook. We recently just release a few new features to help combat some of these issues:

Co-editing (preview). Co-editing makes collaboration easier than ever. The notebook can now be shared by sending the notebook URL, allowing multiple users to edit the notebook in real-time.

Figure 5: Live Co-editing in Azure ML

Export Notebook as Python, LaTeX or HTML. When you feel satisfied with the results from your notebook and ready to present to your colleagues, you can export the notebook to various formats for easy sharing. LaTeX, HTML, and .py are currently supported.

Figure 6: Export Notebooks as Python and more in Azure ML

Get Started Today

To begin using these features in Azure ML Notebooks, you will first need to create an Azure Machine Learning. Your Azure ML workspace serves as your one-stop-shop for all your machine learning needs, where you can create and share all your machine learning assets.

Once you have your workspace set up, you can get started using all the features in the Azure ML Notebooks experience. The notebooks experience aims to provide you with an integrated suite of data science tools. Users can start working with a highly productive and collaborative Jupyter notebook editor directly in their workspace as well as quickly access other ML assets such as experiment details, datasets, models, and more.

With the addition of this host of features, notebooks in Azure ML aims to improve every aspect of your development needs – collaboration, code editing, debugging. Give these features a try and leave your feedback. The feedback provided by our community is what drives us to improve and build new features. As we continue to push out new releases, keep an eye out, because the team has a few more exciting features coming out soon.

Integrating AI: Prototyping a No-Code solution with Power Apps

Yonet — Mon, 15 Mar 2021 21:14:46 GMT

You might have the cutting edge AI features but it is hard to know how useful it will be before letting your users beta test your prototype. You can build fast, deploy and deliver your app and iterate without writing any code, using AI Builder and Power Platform.

This article explains what Power Platform is, as well as go through a step by step process to create an application that detects objects from photos using Power Apps and AI Builder. Check out the video below to see the app we will build to detect different Mixed Reality Headsets such as HoloLens version 1 and 2 Augmented Reality and Virtual Reality headsets and their hand controllers.

What is Power Platform?

Power Platform is a set of tools, API's and SDK's that helps you analyze your data and build automations, applications and virtual agents with or without having to write any code.

What are Power Apps?

Power Apps, allows you to create applications with a drag and drop UI and easy integration of your data and 3rd party APIs through connectors.

A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows. For example, you can use the Twitter connector to get tweet data and visualize it in a dashboard or use the Twilio connector to send your users text messages without having to be an expert in Twitter or Twilio APIs or having to write a line of code.

Check out the list of connectors for Power Apps to see all the APIs that are available. Notice Power Automate or Logic App connectors might not be the same.

What is AI Builder?

AI Builder is one of the additional features of Power Apps. With AI Builder, you can add intelligence to your apps even if you have no coding or data science skills.

What are some of the use cases for AI Builder?

You can use pre-trained models to:

Detect objects from images
Analyze your customers' sentiment from feedback
Detect keywords from text
Extract specific information about your business from text

Is AI Builder the right choice?

Great question! There are so many tools out there and many ways to do the same thing. How do you know which one is the right solution before investing time and effort?

‌

I have a rule of thumb when I want to build something, use whatever is available and easy to use first. When your needs exceed what the tool you are using covers, look into another solution or building it yourself.

Use the tool easiest to get started when you are building your idea. When your needs exceed the capabilities of the tool you are using, find a solution that enables you. Don't invest in building things from scratch before you know it is worth it to do so.

For example, if you have an app idea, it is better to have a prototype running as easily as possible. You can test your ideas before investing your time into building custom designed UI or features. In our specific case, you can first prototype your app with the drag and drop UI of Power Apps and using prebuilt AI models. When your specific needs surface, such as recognizing a particular object or keyword, you can invest your time into creating your custom models to train for the object or keyword detection.

Can I use Power Apps and AI Builder for production?

Yes you can. As any tool that does things magically, AI Builder in Power Apps comes with a cost. That does not mean you can't try your ideas out for free.

What will my production app cost?

If you want to go to production with Power Apps, it is a good idea to consider the costs. Thankfully there is an app for that. AI Builder Calculator lets you input what AI tools you will need and how many users will be accessing your app's AI features and gives you the price it will cost you.

What are preview features?

AI Builder was released for public preview on June 10, 2019 in Europe and the United States. Preview release features are subject to change and may have restricted functionality before the official release for general availability. Preview releases are not meant for production use. You can try them out and influence the final product by giving feedback.

The General Availability (GA) release will occur in a phased manner, with some features remaining in preview status while others are released for GA. You can check out the release status on the AI Builder documentation.

What is Object Detection?

AI Builder Object detection is an AI model that you can train to detect objects in pictures. AI models usually require that you provide samples of data to train before you are able to perform predictions. Prebuilt models are pre-trained by using a set of samples that are provided by Microsoft, so they are instantly ready to be used in predictions.

Object detection can detect up to 500 different objects in a single model and support JPG, PNG, BMP image format or photos through the Power Apps control.

How to try out Object Detection capabilities?

You can try out and see how object detection works before having to create and accounts or apps yourself on the Azure Computer Vision page.

What can you do with Object Detection?

Object counting and inventory management
Brand logo recognition
Wildlife animal recognition

How to detect objects from images?

To start creating your AI model for your app, sign in to Power Apps and click on AI Builder on the left hand menu. Select Object Detection from the "Refine Model for your business needs" option.

Name your new AI model with a unique name. Select Common Objects and proceed to next section.

Name the objects that you are going to detect.

Upload images that contain the object you will detect. To start with you can upload 15 images for each object.

Make sure each object has approximately the same amount of images tagged. If you have more examples of one object, the training data will be likely to detect that object when it is not.
Tag your objects by selecting a square that your object is in and choosing the name of the object.

Once you are done, choose Done Tagging and Train. Training process will take some time.
If you choose to not use an image or clear any tags, you can do that at any time by going back to your model under the AI Builder on the left hand side menu and choose your model and choose edit.

AI Builder will give you a Performance score over 100 and a way to quickly test your model before publishing. You can edit your models and retrain to improve your performance. Next section will give you some best practices to improve your performance.

How to Improve Your Custom Model Performance?

Getting the best model performance for your business can be an iterative process. Results can vary depending on the customizations you make to the model, and the training data you provide.

‌

To help facilitate this process, AI Builder allows you to have multiple versions of your model so you can use your model and continue to improve it at the same time.

What are some best practices for training for object detection?

Use diverse images to train with all possible use cases. For example if you are training your data to detect a VR headset, use images of the headset used in different environments as well as the out of the box images. If you only train with images with people wearing the headset, your model would not recognize images of the same device when it is in its box.

Use images with a variety of backgrounds. Photos in context are better than photos in front of neutral backgrounds.

Use training images that have different lighting. For example, include images taken with flash, high exposure, and so on.

Use images of objects in varied sizes. Different sizing helps the model generalize better.
Use images taken from different angles. If all your photos are from a set of fixed cameras such as surveillance cameras, assign a different label to each camera. This can help avoid modeling unrelated objects such as lampposts as the key feature. Assign camera labels even if the cameras capture the same objects.

How to share your models?

By default, only you can see the models you create and publish. This feature allows you to test them and use them within apps and flows without exposing them.

If you want others to use your model, you can share it with specific users, groups, or your whole organization.

How to use your Custom Vision model in a Power App?

Once you are happy with your model's performance, you can add it to a new app by choosing Use model and New app.

You will be redirected to Power App editor and an Object Detection component that uses your model will be added automatically. In the editor, you can add new pages to navigate, design and customize your pages.

Once you are happy with the design, you can publish and share your app. You can use your new app by downloading Power Apps from Apple, Android or Microsoft stores. Once you sign in, your app will be listed in the mobile Power Apps.

What's next?

Now you have your app's prototype, you can add more features, get feedback and test your app.

Should I keep using my power app or rebuild it?

When your needs change, you can consider refactoring your application to a serverless backend and a custom built UI. If the app is working fine for you and your users, you can continue using and improving overtime using Power Apps.

‌

What would be the changes that require the upgrade? There are two possibilities for the changed requirements for your app:

Feature
Budget

How to create a custom feature for Power Apps?

Ready made tools are always limited to the features the product team decides to include. If you are writing custom code, you can add any feature that you need. Thankfully, for the features that are not implemented yet, it is always possible to author a custom connector that you can use with or without Power Apps.

A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows.

Check out the list of Power Apps Connectors and how to build a custom connector yourself.

How to compare costs for Power Apps and Logic Apps?

Once you start using your app, you will have a better idea about the number of users accessing AI capabilities and the number of images that you need to train. You can use AI Builder Cost Calculator and Logic App Cost Calculator to compare options. You can check any other service price through Azure Product Cost Calculator.

Additional Resources

Leave a comment below for your AI application use cases and the tutorials you would like to see.

Re: Introducing QnA Maker managed: now in public preview

julianportelli — Tue, 09 Mar 2021 09:47:32 GMT

would it be possible to add the ability to toggle displaying of the short answer, possibly via an Application Setting in the bot's App Service? Showing both short and long answers would be extremely confusing for end users.

Mask detection now available in preview via Azure Cognitive Services

vaparth — Mon, 08 Mar 2021 21:11:35 GMT

COVID-19 spread has changed our day-to-day life in unprecedented ways. Organizations around the world are taking action to, contain and help prevent further spread of the disease by using AI technologies like computer vision, help ensure the safety of the employees and customers.

Azure Cognitive Services now, provides Mask detection functionality, to assist application developers in building solutions that can help monitor and contain the spread. Mask detection can be deployed anywhere, from the cloud leveraging Face service, to the edge using Spatial analysis service.

Mask detection on the edge

Spatial analysis is, a capability of Computer Vision, part of Azure Cognitive Services. This capability understands people’s movements in a physical space by analyzing real-time video, significantly increasing efficiency, and providing valuable insights for enabling various scenarios including,

Counting people in a space for maximum occupancy
Understanding the distance between people for social distancing measures
Determining customer footfall such as in retail spaces
Determining wait time in a checkout line
Determining trespassing in protected areas

Spatial analysis can now detect whether a person is wearing a protective face covering or not. With this new capability, businesses can leverage insights to build applications that can measure safety and enhance compliance. For example, a business can aggregate data of percentage of people wearing masks in a physical space to improve compliance measures. To help ensure the safety of people working in a given space, mask detection can also be used to notify when a person may accidentally enter the space without a face mask.

Mask detection can be enabled for the following spatial analysis operations - personcount, personcrossingline and personcrossingpolygon. The classifier model can be enabled by configuring the ‘ENABLE_FACE_MASK_CLASSIFIER’ parameter to True, this is disabled by default. The attributes, face_mask or face_noMask, will be returned as metadata with confidence score for each person detected in the video stream.

Face mask and Person detection with Spatial analysis

Spatial analysis operations provide a real-time video analysis pipeline on new and existing RTSP cameras. The deployment of the spatial analysis container on edge devices is facilitated by Azure IoT Hub. When video is streamed and processed by spatial analysis, the container emits AI insight events about people’s movement which in turn are sent to Azure IoT Hub as IoT telemetry. From IoT Hub you can create various routes to other Azure services and build your business solutions.

Spatial analysis container deployment with Azure IoT

The events from each operation are egressed to Azure IoT Hub on JSON format. Sample JSON for an event output by cognitiveservices.vision.spatialanalysis-personcount operation.

{ "events": [ { "id": "3733eb36935e4d73800a9cf36185d5a2", "type": "personLineEvent", "detectionIds": [ "90d55bfc64c54bfd98226697ad8445ca" ], "properties": { "trackingId": "90d55bfc64c54bfd98226697ad8445ca", "status": "CrossLeft" }, "zone": "doorcamera" } ], "sourceInfo": { "id": "camera_id", "timestamp": "2020-08-24T06:06:53.261Z", "width": 608, "height": 342, "frameId": "1340", "imagePath": "" }, "detections": [ { "type": "person", "id": "90d55bfc64c54bfd98226697ad8445ca", "region": { "type": "RECTANGLE", "points": [ { "x": 0.491627341822574, "y": 0.2385801348769874 }, { "x": 0.588894994635331, "y": 0.6395559924387793 } ] }, "confidence": 0.9005028605461121, "metadata": { "attributes": { "face_Mask": 0.99 } } } ], "schemaVersion": "1.0" }

Learn how to build business applications with spatial analysis, follow these instructions to deploy a sample Azure Web Application that presents a live view of people counting events in a physical space. You can modify this app with other spatial analysis operations and make modifications based on the event output of the container.

Mask detection in the cloud

Mask detection is also available through the Face Detection cloud endpoint in Azure Cognitive Face API Service. This capability analyses images, detects one or more human faces along with attributes for each face in the image. Face mask attribute is available with the latest detection_03 model, along with additional attribute “noseAndMouthCovered” that provides insight about whether the mask covers both the nose and mouth.

To leverage the latest mask detection capability, users need to specify the detection model in the API request - assign the model version with the detectionModel parameter to detection_03. Refer to How to specify a detection model to learn more about the capabilities of each detection model and sample code to call it.

Face mask detection with Face Service

Detection_03 API response with face mask attribute:

{ "faceId": "eee58bd3-0b54-4f48-9a96-c9c60724ee80", "faceRectangle": { "top": 171, "left": 1212, "width": 79, "height": 125 }, "faceAttributes": { "mask": { "type": "faceMask", "noseAndMouthCovered": “true” } }, { "faceId": "2d83c3c1-7266-4b84-b47b-a65645368021", "faceRectangle": { "top": 364, "left": 600, "width": 66, "height": 80 }, "faceAttributes": { "mask": { "type": "faceMask", "noseAndMouthCovered": “true” } },

Responsible AI and Deployment Guide

Microsoft’s principled approach enables developers to build rich solutions while ensuring responsible use.

Responsible deployment recommendations for spatial analysis is provided in accordance with Microsoft Responsible AI Principles: fairness, reliability & safety, privacy & security, inclusiveness, transparency, and human accountability. For general guidelines and specific recommendations for height, angle, and camera-to-focal-point-distance, see Camera placement guide. And refer to Face API Transparency Note to get clear guidance on use of facial recognition to help ensure it fits your goals and achieve accurate results.

Get Started

Learn more with our documentation Spatial analysis, QuickStart: Face Service

Follow the tutorial to Create a People Counting Web App, Detect faces using the .NET SDK

Learn about Azure Stack Edge and Azure IoT Hub

Put AI into practice with Microsoft's Azure AI Hackathon

wmendoza — Mon, 08 Mar 2021 17:52:41 GMT

If you’ve been looking for a reason to get started with AI to solve a particular problem or use case, look no further! We invite you to put your skills to the test and apply Azure AI to a new or existing project. As you may have seen in an earlier post by Anand Raman, we have been hosting an Azure AI hackathon in which you can submit your project and be eligible to win prizes. Developers of all backgrounds and skill levels are welcome to join and submit any form of AI project, whether using Azure AI to enhance existing apps with pre-trained machine learning (ML) models with Cognitive Services or building your own custom ML models with Azure Machine Learning.

AI Hackathon homepage

If you’re interested in participating, visit the Azure AI Hackathon page to get started. The deadline is April 5^th so you still have time to build and submit a project! Use one or more of the following Azure AI services to build a new project or update an existing project: Azure Machine Learning, Azure Cognitive Services, Bot Framework and Azure Cognitive Search. Projects may integrate with other Azure services, open source technologies (including but not limited to frameworks, libraries, and APIs) and physical hardware of your choice.

If you’re looking for a little inspiration, below are a few examples of past winners:

2019 First Place– Trashé

Trashe Smarter Recycling solution

Submitted by Nathan Glover and Stephen Mott, Trashé is a SmartBin that aims to help people make more informed recycling decisions. While the idea is super impactful, it’s even more powerful when you see it in action- not just the intelligence, but the end-to-end scenario of how it can be applied in a real-world environment.

This team used many Azure services to connect the hardware, intelligence, and presentation layers—you can see this is a well-researched architecture that is reusable in multiple scenarios. Azure Custom Vision was a great choice in this case, enabling the team create a well performing model with very little training data. The more we recycle, the better the model will get. It was great to see the setup instructions included to help build unique versions of Trashé so users can contribute to helping the environment by recycling correctly within their local communities—this community approach is incredibly scalable.

2019 Second Place- AfriFarm

Niza Siwale’s app recognizes crop diseases from images using Azure Machine Learning service and publishes the findings so anyone can track disease breakouts. This also provides a real-time update for government agencies to act quickly and provide support to affected communities. As quoted by Niza, this project has an incredible reach to a possible 200 million farmers whose livelihoods depend on farming in Africa.

Creating a simple Android application where users can take photos of crops to analyze if each farmer is getting information when they need it, users can also contribute their own findings back to the community around them keeping everyone more informed and connected. Using the popular Keras framework along with the Azure Machine Learning service, this project built and deployed a good plant disease recognition model which could be called from the application. Any future work or improved versions of models can be monitored and deployed in a development cycle. From this, the progression of the model can be tracked over time.

2019 Third Place- Water Level Anomaly detector

Roy Kincaid’s project identifies drastic changes in water levels using an ultrasonic sensor that could be useful for detecting potential floods and natural disasters. This information can then be used to provide adequate warning for people to best prepare to major changes in their environment. Water Level Anomaly Detector could also be beneficial for long-term analysis of the effects of climate change. This is another great example of an end-to-end intelligent solution.

Roy is well skilled in the hardware and connection parts of this project, so it was brilliant to see the easy integration of the Anomaly Detector API from Azure Cognitive Services and to hear how quickly Roy could start using the service. Many IoT scenarios have a similar need for detecting rates and levels; in fact, Roy had hinted at coffee level detector in the future. In a world where we all want to do our part to help the environment, it’s a great example of how monitoring enables us to measure changes over time and be alerted when issues arise.

These are just 3 of the past winners and submissions. For more inspiration, visit our gallery of past submissions.

Resources to get started

Sign up for the Azure AI hackathon
Visit our AI for Developers resources page for tutorials and a curated 30-day learning journey
Visit our ML for Data Scientists resources page for tutorials and a curated 30-day learning journey

Re: Introducing semantic search: Bringing more meaningful results to Azure Cognitive Search

Luis Cabrera-Cordon — Wed, 03 Mar 2021 17:02:52 GMT

, we are starting with English for now. The models have been trained to support other languages (like Spanish), but we want to do some more testing on those before we release them. Stay tuned.

Form Recognizer now reads more languages, processes IDs and invoices, trains on tables, and more

christina-lee — Thu, 04 Mar 2021 23:59:38 GMT

Documents contain invaluable information powering core business processes. Extracting information from these documents with minimum manual intervention helps bolster organizational efficiency and productivity. As more and more processes and workflows get automated, the need for new features to help extract text and structures increases.

Today, we are excited to announce newest updates to Form Recognizer that will be available on March 15, 2021.

What’s New?

Form Recognizer v2.1 public preview 3 will be available on March 15, 2021, and it will include:

Extract data from invoices

Invoices are complex documents that vary in structure and contain data that is vital to organizations business processes. One of the most challenging tasks in extracting data from invoices is extracting data from invoices line items. The Form Recognizer Invoice API now supports line-item extraction, it also extracts now the full line item and its parts – description, amount, quantity, product ID, date and more. With a simple API \ SDK call you can extract all the data from your invoices – text, tables, key value pairs and line items.

Figure 1 Line items are extracted from invoices

Extract data from IDs

The new pre-built ID model enables customers to take worldwide passports and U.S. drivers license and return structured data representing the information available on the IDs. The new ID API extract the text and values of interest from IDs such as document number, last name, first name, date of expiration, country and more.

Figure 2 Pre-built ID model can extract information from passports and US drivers licenses

Supervised table labeling and training, empty-value labeling

In addition the Form Recognizer state of the art deep learning automatic table extraction capabilities it now also enables customer to train and label tables. This new release includes the ability to label line items/tables (dynamic and fixed) and train a custom model to extract key-value pairs and line items. Once a model is trained and documents are analyzed using this model the new line items will be extracted as part of the JSON output in the documentResults section.

Figure 3 Label tables in your training dataset

In addition to labeling tables, you can now label empty values and regions; if some documents in your training set do not have values for some fields, you can use this so that your model will know to extract values properly from analyzed documents.

Natural reading order, handwriting classification, and page selection

With this update, you can choose to get the text line outputs in the natural reading order instead of the default left-to-right and top-to-bottom ordering. Use the new readingOrder query parameter to “natural” value for a more human-friendly reading order output as shown in the following example. Note the first column’s text lines output in order before listing the second, and the third, column.

In addition, for Latin languages, Form Recognizer will classify Latin-languages only text lines as handwritten style or not and give a confidence score, as seen below.

Furthermore, when analyzing a multi-page PDF or TIFF, you can now specify which pages you want to analyze.

Pre-built Receipt model quality improvements

This new update includes a number of quality improvements for the pre-built Receipt model, especially around line item extraction.

Our Customers & Partners

AvidXchange has developed an account payable automation solution leveraging Form Recognizer. “By partnering with Microsoft, we’re able to deliver an accounts payable automation solution for the middle market that’s truly powered by machine learning,” said Chris Tinsley, Chief Technology Officer at AvidXchange. “Our customers will benefit from faster invoice processing times and increased accuracy so we can help ensure their suppliers are paid the right amount, at the right time.”

WEX has developed a tool to process Explanation of Benefits documents using Form Recognizer. Matt Dallahan, Senior Vice President of Product Management and Strategy, said “The technology is truly amazing. I was initially worried that this type of solution would not be feasible, but I soon realized that the Form Recognizer can read virtually any document with accuracy.”

GEP has developed an invoice processing solution for a client using Form Recognizer. “At GEP, we are seeing AI and automation make a profound impact on procurement and the supply chain. By combining our AI solution with Microsoft Form Recognizer, we automated the processing of 4,000 invoices a day for a client, saving them tens of thousands of hours of manual effort, while improving accuracy, controls and compliance on a global scale,” said Sarateudu Sethi, GEP’s Vice President of Artificial Intelligence.

“At Cross Masters, using cutting-edge AI technologies is not only a passion, it is an essential part of our work culture that requires continuous innovation. One of our latest success stories is automation of manual paperwork, required to process thousands of invoices. Thanks to Microsoft Form Recognizer’s AI engine we were able to develop a unique customized solution that provides to our clients market insights from large set of collected invoices. What we find the most convenient is human beating extraction quality and continuous introduction of new features, such as model composing or table labelling. This assures our client’s market advantage and helps our product to be the best-in-class solution” Jan Hornych, Head of Marketing Automation, Cross Masters

Try out Form Recognizer

To get started with Form Recognizer, please login to the Azure Portal to create a Form Recognizer resource. Once your resource is created, you can start exploring Form Recognizer, with the improvements mentioned above coming on March 15. You can learn more about Form Recognizer here.

Re: Introducing semantic search: Bringing more meaningful results to Azure Cognitive Search

Alberto Diaz Martin — Wed, 03 Mar 2021 07:35:26 GMT

Hi Luis, in which language are semantic search supported? Is it Spanish supported in this preview?

Re: Introducing QnA Maker managed: now in public preview

julianportelli — Wed, 03 Mar 2021 07:23:44 GMT

Any response to 's question "how to remove short answer after publishing qna maker KB?" I am facing the same issue!

Introducing semantic search: Bringing more meaningful results to Azure Cognitive Search

Luis Cabrera-Cordon — Fri, 05 Mar 2021 00:00:36 GMT

A few years ago, it became clear to our team that AI could bring value to our customers, from improvements in ingestion to data exploration. We knew we had a lot of these valuable assets around Microsoft, so our team set out on a mission to bring as much “intelligence” as we could to the product then known as Azure Search.

In the first phase of this mission, we took on "unsearchable" content; about 80% of business relevant data is in unstructured formats such as PDFs, PowerPoints, Word documents, JPEGs, CSVs, etc. We added AI powered enrichments to our ingestion process, enabling the ability to extract structure, insights and transform information from your data. These capabilities were well received by our customers, culminating in a product rebrand as "Azure Cognitive Search".

I am happy to announce that in our continuation of this journey, we are bringing state of the art AI capabilities to the “head” of our product, the core search sub-system. In partnership with the Bing team, we have integrated their semantic search investments (100s of development years and millions of dollars in compute time) into our query infrastructure, effectively enabling any developer to leverage this investment over searchable content that you own and manage. We believe semantic search on Azure Cognitive Search offers the best combination of search relevance, developer experience, and cloud service capabilities available on the market.

This post explains what new capabilities are available to you and how you can get started today. I would also encourage you to look at the post called “Bing’s AI behind semantic search” that goes deeper into the Bing technology that made semantic search possible.

Today, we are launching several exciting semantic search features in a public preview:

Semantic Ranking

Customers have grown accustomed to using natural language queries in web search engines, but these queries usually do not perform as well when using a traditional keyword-based retrieval approach with ranking only based on term frequencies. To demonstrate this, consider what happens when a customer types a query like “how to add a user in Office” in the Microsoft documentation. For this purpose, we loaded all the Microsoft documentation dataset into Azure Cognitive Search so that we could compare the results between the default lexical based ranking algorithm and the semantic ranking algorithm.

Traditional retrieval and ranking approach

The default ranker (BM25) uses words as discrete units and predicts relevance by using the frequencies of terms in the corpus. BM25 works well when searching for keywords, but it struggles to find the most relevant documents when issuing a natural language query.

Note that the results do meet the lexical frequency requirements. For instance, inspecting the top document “Troubleshoot user errors with Office Add-ins” shows that there are lot of mentions of terms like “office”, “user”, “add” and “how to” in the document – but unfortunately the article is not providing the information we meant to query for.

Semantics-based ranking

With the release of semantic search, now we can enable a ranking algorithm that will use deep neural networks to rank the articles based on how “meaningful” they are relative to the query. Internally, this is a ranker that is applied on top of the results returned by the BM25-based ranker. Using semantic search capabilities, these are the top results for our query:

I read the content of the top-document called “Add users and assign licenses at the same time”, and it is clear that this is exactly the document I need! Semantic search made this connection even though the title and content are not syntactically close to my query.

Semantic Answers

In the previous example, the title of the document by itself did not make it very easy for me to catch if that was a relevant document or not. I still had to read it to find the snippet in the documentation that told me how to add a user to Office.

The good news is that now you can also get semantic answers! It is one of my favorite features; it uses an AI model that extracts relevant passages from the top documents, and then ranks them on their likelihood of being an answer to the query. If we find a passage that has a high likelihood of answering the question, we will promote it as a semantic answer.

This is what it looks like, in this case. Note that we even leveraged a model from Bing to provide highlights for the most relevant section in the semantic answer.

Semantic Captions

Similarly, we can extract the most relevant section of each document returned so you can quickly skim through the results and see if they have the content that you care about; making it easier for you to triage the results briefly and go deeper into the ones that you think are relevant given your context.

Get started today!

Using semantic search is easy. After you sign up for the preview at http://aka.ms/semanticpreview, all you need to do is change your query parameters as part of the request as shown below. Note that there is no need to re-index any of your content!

Query parameter
queryType	Set to “semantic” to indicate that you would like semantic ranking and answers. Other values supported: “simple” and “full”.
searchFields	Ordered list of fields that semantic ranking should be applied on. If you have a title or a short field that describes your document, we recommend that to be your first field. Follow that by the url (if any), then the body of the document, and then any other relevant fields.
queryLanguage	“en-us” is the only supported value today. We will be adding more languages soon. Stay tuned.
speller	Set to “lexicon” if you would like spell correction to occur on the query terms. Otherwise set to “none”.
answers	Set to “extractive” if you would like to get extractive answers. Otherwise set to “none”.

Sample Query

POST https://[service name].search.windows.net/indexes/[index name]/docs/search?api-version=2020-06-30-preview { "search": " Where was Alan Turing born?", "queryType": "semantic", "searchFields": "title,url,body", "queryLanguage": "en-us", "speller": "lexicon", "answers": "extractive" }

Sample Response

{ "@search.answers": [ { "key": "a1234", "text": "Turing was born in Maida Vale, London, while his father, Julius…", "highlights": " Turing was born in Maida Vale, London , while …", "score": 0.87802511 } ], "value": [ { "@search.score": 51.64714, "@search.rerankerScore": 1.9928148165345192, "@search.captions": [ { "text": " Alan Mathison Turing, (born June 23, 1912, London, England—died June 7, 1954…", "highlights": " Alan Mathison Turing, (born June 23, 1912, London, England—died June…", } ], "id": "b5678", "body": "…" }, … ] }

Learn more about Semantic Search in our documentation.

I am personally super excited about these new capabilities, the efficiencies that they will bring to you, and the progression of our vision to bring the best AI capabilities at Microsoft to Azure developers!

Luis Cabrera – on behalf of the Azure Cognitive Search team

Customers & Partners

Case Study: PPL Electric Utilities Corporation, a utilities company, is working with Neudesic to create a web application with Azure Cognitive search to empower its field workers to find the most relevant information wherever they are.

Case Study: Howden teamed up with OrangeNXT to further improve Smart Records using key elements of their digitalNXT Search, a fully managed cloud solution powered by Azure Cognitive Search.

Call to Action

Preview sign-up form

Cognitive Search Team Ask Me Anything (March 10 2021)

Resources

Documentation

Mechanics Video

Bing science behind semantic search

Re: Simplify and accelerate AI for the entire data science team with Azure Machine Learning designer

Gowindarajan — Mon, 01 Mar 2021 05:15:04 GMT

We have a subscription to Azure Classic, without additional charges are we able to use the new Azure Machine Learning?

Re: Get skilled on AI and ML – on your terms with Azure AI

h2Productsh2__di — Fri, 26 Feb 2021 23:16:25 GMT

{

"type": "object",

"properties": {

"id": {

"type": "integer",

"format": "int64"

"scaDetails": {

"description": "Strong Customer Authentication challenge event information",

"type": "object",

"required": [

"clientEventId",

"eventId"

"example": {

"clientEventId": "47875409",

"eventId": "abcd1234"

"properties": {

"clientEventId": {

"type": "string",

"description": "2FA client event id"

"eventId": {

"type": "string",

"description": "2FA event id"

"link": {

"type": "string",

"description": "2FA event resource reference"

}

"link": {

"type": "object",

"properties": {

"href": {

"type": "string"

"rel": {

"type": "string"

}

"status": {

"type": "string",

"enum": [

"PROCESSED",

"FAILED",

"CANCELED",

"CUSTOMER_REDIRECT",

"SMS_VERIFICATION_REQUIRED",

"MISSING_SENDER_DOB",

"MISSING_SENDER_ADDRESS",

"BALANCE_NOT_ENOUGH",

"IN_PROGRESS",

"PENDING",

"PENDING_REVIEW",

"PENDING_KYC",

"PENDING_SCREENING",

"SCHEDULED",

"B{

"type": "object",

"properties": {

"clientEventId": {

"type": "string",

"description": "Transaction related with this SCA Event"

"eventId": {

"type": "string",

"description": "Unique SCA Event Id"

74 1050 1575 1000 0097 3495 4788

123456789101112131415161718192021222324252627282930313233343536
# This file is part of PulseAudio.
#
# PulseAudio is free software; you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# PulseAudio is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU

}

}ANK_UNSUPPORTED",

"SCA_CHALLENGE"

]

}

Re: Computer Vision for spatial analysis at the Edge

jfilcik — Thu, 25 Feb 2021 22:11:42 GMT

Thanks for your interest in Spatial Analysis. We recommend NVIDIA T4 for production deployments. You may use a CUDA Compute Capable devices 6.0 or higher for testing ( e.g.: NVIDIA 1080Ti or 2080Ti ).

Ombromanie: Creating Hand Shadow stories with Azure Speech and TensorFlow.js Handposes

jelooper — Thu, 25 Feb 2021 19:09:52 GMT

Have you ever tried to cast hand shadows on a wall? It is the easiest thing in the world, and yet to do it well requires practice and just the right setup. To cultivate your #cottagecore aesthetic, try going into a completely dark room with just one lit candle, and casting hand shadows on a plain wall. The effect is startlingly dramatic. What fun!

Even a tea light suffices to create a great effect

In 2020, and now into 2021, many folks are reverting back to basics as they look around their houses, reopening dusty corners of attics and basements and remembering the simple crafts that they used to love. Papermaking, anyone? All you need is a few tools and torn up, recycled paper. Pressing flowers? All you need is newspaper, some heavy books, and patience. And hand shadows? Just a candle.

This TikTok creator has thousands of views for their handshadow tutorials

But what's a developer to do when trying to capture that #cottagecore vibe in a web app?

High Tech for the Cottage

While exploring the art of hand shadows, I wondered whether some of the recent work I had done for body poses might be applicable to hand poses. What if you could tell a story on the web using your hands, and somehow save a video of the show and the narrative behind it, and send it to someone special? In lockdown, what could be more amusing than sharing shadow stories between friends or relatives, all virtually?

Hand shadow casting is a folk art probably originating in China; if you go to tea houses with stage shows, you might be lucky enough to view one like this!

A Show Of Hands

When you start researching hand poses, it's striking how much content there is on the web on the topic. There has been work since at least 2014 on creating fully articulated hands within the research, simulation, and gaming sphere:

MSR throwing hands

There are dozens of handpose libraries already on GitHub:

There are many applications where tracking hands is a useful activity:

• Gaming
• Simulations / Training
• "Hands free" uses for remote interactions with things by moving the body
• Assistive technologies
• TikTok effects :trophy:
• Useful things like Accordion Hands apps

One of the more interesting new libraries, handsfree.js, offers an excellent array of demos in its effort to move to a hands free web experience:

Handsfree.js, a very promising project

As it turns out, hands are pretty complicated things. They each include 21 keypoints (vs PoseNet's 17 keypoints for an entire body). Building a model to support inference for such a complicated grouping of keypoints has provenn challenging.

There are two main libraries available to the web developer when incorporating hand poses into an app: TensorFlow.js's handposes, and MediaPipe's. HandsFree.js uses both, to the extent that they expose APIs. As it turns out, neither TensorFlow.js nor MediaPipe's handposes are perfect for our project. We will have to compromise.

TensorFlow.js's handposes allow access to each hand keypoint and the ability to draw the hand to canvas as desired. HOWEVER, it only currently supports single hand poses, which is not optimal for good hand shadow shows.
MediaPipe's handpose models (which are used by TensorFlow.js) do allow for dual hands BUT its API does not allow for much styling of the keypoints so that drawing shadows using it is not obvious.

One other library, fingerposes, is optimized for finger spelling in a sign language context and is worth a look.

Since it's more important to use the Canvas API to draw custom shadows, we are obliged to use TensorFlow.js, hoping that either it will soon support multiple hands OR handsfree.js helps push the envelope to expose a more styleable hand.

Let's get to work to build this app.

Scaffold a Static Web App

As a Vue.js developer, I always use the Vue CLI to scaffold an app using vue create my-app and creating a standard app. I set up a basic app with two routes: Home and Show. Since this is going to be deployed as an Azure Static Web App, I follow my standard practice of including my app files in a folder named app and creating an api folder to include an Azure function to store a key (more on this in a minute).

In my package.json file, I import the important packages for using TensorFlow.js and the Cognitive Services Speech SDK in this app. Note that TensorFlow.js has divided its imports into individual packages:

"@tensorflow-models/handpose": "^0.0.6",
"@tensorflow/tfjs": "^2.7.0",
"@tensorflow/tfjs-backend-cpu": "^2.7.0",
"@tensorflow/tfjs-backend-webgl": "^2.7.0",
"@tensorflow/tfjs-converter": "^2.7.0",
"@tensorflow/tfjs-core": "^2.7.0",
...
"microsoft-cognitiveservices-speech-sdk": "^1.15.0",

Set up the View

We will draw an image of a hand, as detected by TensorFlow.js, onto a canvas, superimposed onto a video suppled by a webcam. In addition, we will redraw the hand to a second canvas (shadowCanvas), styled like shadows:

<div id="canvas-wrapper column is-half">
<canvas id="output" ref="output"></canvas>
    <video
        id="video"
        ref="video"
        playsinline
        style="
          -webkit-transform: scaleX(-1);
           transform: scaleX(-1);
           visibility: hidden;
           width: auto;
           height: auto;
           position: absolute;
         "
    ></video>
 </div>
 <div class="column is-half">
    <canvas
       class="has-background-black-bis"
       id="shadowCanvas"
       ref="shadowCanvas"
     >
    </canvas>
</div>

Load the Model, Start Keyframe Input

Working asynchronously, load the Handpose model. Once the backend is setup and the model is loaded, load the video via the webcam, and start watching the video's keyframes for hand poses. It's important at these steps to ensure error handling in case the model fails to load or there's no webcam available.

async mounted() {
    await tf.setBackend(this.backend);
    //async load model, then load video, then pass it to start landmarking
    this.model = await handpose.load();
    this.message = "Model is loaded! Now loading video";
    let webcam;
    try {
      webcam = await this.loadVideo();
    } catch (e) {
      this.message = e.message;
      throw e;
    }

    this.landmarksRealTime(webcam);
  },

Setup the Webcam

Still working asynchronously, set up the camera to provide a stream of images

async setupCamera() {
      if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
        throw new Error(
          "Browser API navigator.mediaDevices.getUserMedia not available"
        );
      }
      this.video = this.$refs.video;
      const stream = await navigator.mediaDevices.getUserMedia({
        video: {
          facingMode: "user",
          width: VIDEO_WIDTH,
          height: VIDEO_HEIGHT,
        },
      });

      return new Promise((resolve) => {
        this.video.srcObject = stream;
        this.video.onloadedmetadata = () => {
          resolve(this.video);
        };
      });
    },

Design a Hand to Mirror the Webcam's

Now the fun begins, as you can get creative in drawing the hand on top of the video. This landmarking function runs on every keyframe, watching for a hand to be detected and drawing lines onto the canvas - red on top of the video, and black on top of the shadowCanvas. Since the shadowCanvas background is white, the hand is drawn as white as well and the viewer only sees the offset shadow, in fuzzy black with rounded corners. The effect is rather spooky!

async landmarksRealTime(video) {
      //start showing landmarks
      this.videoWidth = video.videoWidth;
      this.videoHeight = video.videoHeight;

      //set up skeleton canvas
      this.canvas = this.$refs.output;
      ...

      //set up shadowCanvas
      this.shadowCanvas = this.$refs.shadowCanvas;
      ...

      this.ctx = this.canvas.getContext("2d");
      this.sctx = this.shadowCanvas.getContext("2d");

      ...

      //paint to main

      this.ctx.clearRect(0, 0, this.videoWidth, 
  this.videoHeight);
      this.ctx.strokeStyle = "red";
      this.ctx.fillStyle = "red";
      this.ctx.translate(this.shadowCanvas.width, 0);
      this.ctx.scale(-1, 1);

      //paint to shadow box

      this.sctx.clearRect(0, 0, this.videoWidth, this.videoHeight);
      this.sctx.shadowColor = "black";
      this.sctx.shadowBlur = 20;
      this.sctx.shadowOffsetX = 150;
      this.sctx.shadowOffsetY = 150;
      this.sctx.lineWidth = 20;
      this.sctx.lineCap = "round";
      this.sctx.fillStyle = "white";
      this.sctx.strokeStyle = "white";

      this.sctx.translate(this.shadowCanvas.width, 0);
      this.sctx.scale(-1, 1);

      //now you've set up the canvases, now you can frame its landmarks
      this.frameLandmarks();
    },

For Each Frame, Draw Keypoints

As the keyframes progress, the model predict new keypoints for each of the hand's elements, and both canvases are cleared and redrawn.

      const predictions = await this.model.estimateHands(this.video);

      if (predictions.length > 0) {
        const result = predictions[0].landmarks;
        this.drawKeypoints(
          this.ctx,
          this.sctx,
          result,
          predictions[0].annotations
        );
      }
      requestAnimationFrame(this.frameLandmarks);

Draw a Lifelike Hand

Since TensorFlow.js allows you direct access to the keypoints of the hand and the hand's coordinates, you can manipulate them to draw a more lifelike hand. Thus we can redraw the palm to be a polygon, rather than resembling a garden rake with points culminating in the wrist.

Re-identify the fingers and palm:

     fingerLookupIndices: {
        thumb: [0, 1, 2, 3, 4],
        indexFinger: [0, 5, 6, 7, 8],
        middleFinger: [0, 9, 10, 11, 12],
        ringFinger: [0, 13, 14, 15, 16],
        pinky: [0, 17, 18, 19, 20],
      },
      palmLookupIndices: {
        palm: [0, 1, 5, 9, 13, 17, 0, 1],
      },

...and draw them to screen:

    const fingers = Object.keys(this.fingerLookupIndices);
      for (let i = 0; i < fingers.length; i++) {
        const finger = fingers[i];
        const points = this.fingerLookupIndices[finger].map(
          (idx) => keypoints[idx]
        );
        this.drawPath(ctx, sctx, points, false);
      }
      const palmArea = Object.keys(this.palmLookupIndices);
      for (let i = 0; i < palmArea.length; i++) {
        const palm = palmArea[i];
        const points = this.palmLookupIndices[palm].map(
          (idx) => keypoints[idx]
        );
        this.drawPath(ctx, sctx, points, true);
      }

With the models and video loaded, keyframes tracked, and hands and shadows drawn to canvas, we can implement a speech-to-text SDK so that you can narrate and save your shadow story.

To do this, get a key from the Azure portal for Speech Services by creating a Service:

You can connect to this service by importing the sdk:

import * as sdk from "microsoft-cognitiveservices-speech-sdk";

...and start audio transcription after obtaining an API key which is stored in an Azure function in the /api folder. This function gets the key stored in the Azure portal in the Azure Static Web App where the app is hosted.

async startAudioTranscription() {
      try {
        //get the key
        const response = await axios.get("/api/getKey");
        this.subKey = response.data;
        //sdk

        let speechConfig = sdk.SpeechConfig.fromSubscription(
          this.subKey,
          "eastus"
        );
        let audioConfig = sdk.AudioConfig.fromDefaultMicrophoneInput();
        this.recognizer = new sdk.SpeechRecognizer(speechConfig, audioConfig);

        this.recognizer.recognized = (s, e) => {
          this.text = e.result.text;
          this.story.push(this.text);
        };

        this.recognizer.startContinuousRecognitionAsync();
      } catch (error) {
        this.message = error;
      }
    },

In this function, the SpeechRecognizer gathers text in chunks that it recognizes and organizes into sentences. That text is printed into a message string and displayed on the front end.

Display the Story

In this last part, the output cast onto the shadowCanvas is saved as a stream and recorded using the MediaRecorder API:

const stream = this.shadowCanvas.captureStream(60); // 60 FPS recording
      this.recorder = new MediaRecorder(stream, {
        mimeType: "video/webm;codecs=vp9",
      });
      (this.recorder.ondataavailable = (e) => {
        this.chunks.push(e.data);
      }),
        this.recorder.start(500);

...and displayed below as a video with the storyline in a new div:

      const video = document.createElement("video");
      const fullBlob = new Blob(this.chunks);
      const downloadUrl = window.URL.createObjectURL(fullBlob);
      video.src = downloadUrl;
      document.getElementById("story").appendChild(video);
      video.autoplay = true;
      video.controls = true;

This app can be deployed as an Azure Static Web App using the excellent Azure plugin for Visual Studio Code. And once it's live, you can tell durable shadow stories!

Try Ombromanie here. The codebase is available here

Take a look at Ombromanie in action:

Learn more about AI on Azure
Azure AI Essentials Video covering speech and language
Azure free account sign-up

Responsible Machine Learning with Error Analysis

besmiranushi — Thu, 18 Feb 2021 16:00:00 GMT

Website: ErrorAnalysis.ai

Github repository: https://github.com/microsoft/responsible-ai-widgets/

Machine Learning (ML) teams who deploy models in the real world often face the challenges of conducting rigorous performance evaluation and testing for ML models. How often do we read claims such as “Model X is 90% on a given benchmark.” and wonder what does this claim mean for practical usage of the model? In practice, teams are well aware that model accuracy may not be uniform across subgroups of data and that there might exist input conditions for which the model fails more often. Often, such failures may cause direct consequences related to lack of reliability and safety, unfairness, or more broadly lack of trust in machine learning altogether. For instance, when a traffic sign detector does not operate well in certain daylight conditions or for unexpected inputs, even though the overall accuracy of the model may be high, it is still important for the development team to know ahead of time about the fact that the model may not be as reliable in such situations.

Figure 1 - Error Analysis moves away from aggregate accuracy metrics, exposes the distribution of errors to developers in a transparent way, and enables them to identify & diagnose errors efficiently.

While there exist several problems with current model assessment practices, one of the most obvious is the usage of aggregate metrics to score models on a whole benchmark. It is difficult to convey a detailed story on model behavior with a single number and yet most of the research and leaderboards operate on single scores. At the same time, there may exist several dimensions of the input feature space that a practitioner may be interested in taking a deep dive and ask questions such as “What happens to the accuracy of the recognition model in a self-driving car when it is dark and snowing outside?” or “Does the loan approval model perform similarly for population cohorts across ethnicity, gender, age, and education?”. Navigating the terrain of failures along multiple potential dimensions like the above can be challenging. In addition, in the longer term, when models are updated and re-deployed frequently upon new data evidence or scientific progress, teams also need to continuously track and monitor model behavior so that updates do not introduce new mistakes and therefore break user trust.

To address these problems, practitioners often have to create custom infrastructure, which is tedious and time-consuming. To accelerate rigorous ML development, in this blog you will learn how to use the Error Analysis tool for:

Getting a deep understanding of how failure is distributed for a model.
Debugging ML errors with active data exploration and interpretability techniques.

The Error Analysis toolkit is integrated within the Responsible AI Widgets OSS repository, our starting point to provide a set of integrated tools to the open source community and ML practitioners. Not only a contribution to the OSS RAI community, but practitioners can also leverage these assessment tools in Azure Machine Learning, including Fairlearn & InterpretML and now Error Analysis in mid 2021.

If you are interested in learning more about training model updates that remain backward compatible with their previous selves by minimizing regress and new errors, you can also check out our most recent open source library and tool BackwardCompatibilityML.

Prerequisites

To install the Responsible AI Widgets “raiwidgets” package, in your python environment simply run the following to install the raiwidgets package from pypi. If you do not have interpret-community already installed, you will also need to install this for supporting the generation of model explanations.

pip install interpret-community pip install raiwidgets

Alternatively, you can also clone the open source repository and build the code from scratch:

git clone https://github.com/microsoft/responsible-ai-widgets.git

You will need to install yarn and node to build the visualization code, and then you can run:

yarn install yarn buildall

And install from the raiwidgets folder locally:

cd raiwidgets pip install –e .

For more information see the contributing guide.

If you intend to run repository tests, in the raiwidgets folder of the repository run:

pip install -r requirements.txt

Getting started

This post illustrates the Error Analysis tool by using a binary classification task on income prediction (>50K, <50K). The model under inspection will be trained using the tabular UCI Census Income dataset, which contains both numerical and categorical features such as age, education, number of working hours, ethnicity, etc.

We can call the error analysis dashboard using the API below, which takes in an explanation object computed by one of the explainers from the interpret-community repository, the model or pipeline, a dataset and the corresponding labels (true_y parameter):

ErrorAnalysisDashboard(global_explanation, model, dataset=x_test, true_y=y_test)

For larger datasets, we can downsample the explanation to fewer rows but run error analysis on the full dataset. We can provide the downsampled explanation, the model or pipeline, the full dataset, and then both the labels for the sampled explanation and the full dataset, as well as (optionally) the names of the categorical features:

ErrorAnalysisDashboard(global_explanation, model, dataset=X_test_original_full,true_y=y_test, categorical_features=categorical_features, true_y_dataset=y_test_full)

All screenshots below are generated using a LGBMClassifier with three estimators. You can directly run this example using the jupyter notebooks in our repository.

How Error Analysis works

1. Identification

Error Analysis starts with identifying the cohorts of data with a higher error rate versus the overall benchmark error rate. The dashboard allows for error exploration by using either an error heatmap or a decision tree guided by errors.

Error Heatmap for Error Identification

The view slices the data based on a one- or two-dimensional grid of input features. Users can choose the input features of interest for analysis. The heatmap visualizes cells with higher error with a darker red color to bring the user’s attention to regions with high error discrepancy. This is beneficial especially when the error themes are different in different partitions, which happens frequently in practice. In this error identification view, the analysis is highly guided by the users and their knowledge or hypotheses of what features might be most important for understanding failure.

Figure 2 - While the overall error rate for the dataset is 23.65%, the heatmap reveals that the error rates are visibly higher, up to 83%, for individuals with higher education. Error rates are also higher for males vs. females.

Decision Tree for Error Identification

Very often, error patterns may be complex and involve more than one or two features. Therefore, it may be difficult for developers to explore all possible combinations of features to discover hidden data pockets with critical failure. To alleviate the burden, the binary tree visualization automatically partitions the benchmark data into interpretable subgroups, which have unexpectedly high or low error rates. In other words, the tree leverages the input features to maximally separate model error from success. For each node defining a data subgroup, users can investigate the following information:

Error rate - a portion of instances in the node for which the model is incorrect. This is shown through the intensity of the red color.
Error coverage – a portion of all errors that fall into the node. This is shown through the fill rate of the node.
Data representation - number of instances in the node. This is shown through the thickness of the incoming edge to the node along with the actual total number of instances in the node.

Figure 3 – Decision tree that aims at finding failure modes by separating error instances from success instances in the data. The hierarchical error pattern here shows that while the overall error rate is 23.65% for the dataset, it can be as high as 96.77% for individuals who are married, have a capital gain higher than 4401, and a number of education years higher than 12.

Cohort definition and manipulation

To specialize the analysis and allow for deep dives, both error identification views can be generated for any data cohort and not only for the whole benchmark. Cohorts are subgroups of data that the user may choose to save for later use if they wish to come back to those cohorts for future investigation. They can be defined and manipulated interactively either from the heatmap or the tree. They can also be carried over to the next diagnostical views on data exploration and model explanations.

Figure 4 - Creating a new cohort for further investigation that focuses on individuals who are married and have capital gain lower than 4401.

2. Diagnosis

After identifying cohorts with higher error rates, Error Analysis enables debugging and exploring these cohorts further. It is then possible to gain deeper insights about the model or the data through data exploration and model interpretability.

Debugging the data

Data Explorer: Users can explore dataset statistics and distributions by selecting different features and estimators along the two axes of the data explorer. They can further compare the subgroup data stats with other subgroups or the overall benchmark data. This view can for instance uncover if certain cohorts are underrepresented or if their feature distribution is significantly different from the overall data, hinting therefore to the potential existence of outliers or unusual covariate shift.

Figure 5 - In figure 1 and 2, we discovered that for individuals with a higher number of education years, the model has higher failure rates. When we look at how the data is distributed across the feature “education_num” we can see that a) there are fewer instances for individuals with more than 12 years of education, and b) for this cohort the distribution between lower income (blue) and higher income (orange) is very different than for other cohorts. In fact, for this cohort there exist more people who have an income higher than 50K, which is not true for the overall data.

Instance views: Beyond data statistics, sometimes it is useful to merely just observe the raw data along with labels in a tabular or tile form. Instance views provide this functionality and divide the instances into correct and incorrect tabs. By eyeballing the data, the developer can identify potential issues related to missing features or label noise.

Debugging the model

Model interpretability is a powerful means for extracting knowledge on how a model works. To extract this knowledge, Error Analysis relies on Microsoft’s InterpretML dashboard and library. The library is a prominent contribution in ML interpretability lead by Rich Caruana, Paul Koch, Harsha Nori, and Sam Jenkins.

Global explanations

Feature Importance: Users can explore the top K important features that impact the overall model predictions (a.k.a. global explanation) for a selected subgroup of data or cohort. They can also compare feature importance values for different cohorts side by side. The information on feature importance or the ordering is useful for understanding whether the model is leveraging features that are necessary for the prediction or whether it is relying on spurious correlations. By contrasting explanations that are specific to the cohort with those for the whole benchmark, it is possible to understand whether the model behaves differently or in an unusual way for the selected cohort.

Dependence Plot: Users can see the relationship between the values of the selected feature to its corresponding feature importance values. This shows them how values of the selected feature impact model prediction.

Figure 6 - Global feature explanations for the income prediction model show that marital status and number of education years are the most important features globally. By clicking on each feature, it is possible to observe more granular dependencies. For example, marital statuses like “divorced”, “never married”, “separated”, or “widowed” contribute to model predictions for lower income (<50K). Marital status of “civil spouse” instead contributes to model predictions for higher income (>50K).

Local explanations

Global explanations approximate the overall model behavior. For focusing the debugging process on a given data instance, users can select any individual data points (with correct or incorrect predictions) from the tabular instance view to explore their local feature importance values (local explanation) and individual conditional expectation (ICE) plots.

Local Feature Importance: Users can investigate the top K (configurable K) important features for an individual prediction. Helps illustrate the local behavior of the underlying model on a specific data point.

Individual Conditional Expectation (ICE): Users can investigate how changing a feature value from a minimum value to a maximum value impacts the prediction on the selected data instance.

Perturbation Exploration (what-if analysis): Users can apply changes to feature values of the selected data point and observe resulting changes to the prediction. They can save their hypothetical what-if data points for further comparisons with other what-if or original data points.

Figure 7 - For this individual, the model outputs a wrong prediction, predicting that the individual earns less than 50K, while the opposite is true. With what-if explanations, it is possible to understand how the model would behave if one of the feature values changes. For instance, here we can see that if the individual were 10 years older (age changed from 32 to 42) the model would have made a correct prediction. While in the real world many of these features are not mutable, this sensitivity analysis is intended to further support practitioners with model understanding capabilities.

Other relevant tools

Error Analysis enables practitioners to identify and diagnose error patterns. The integration with model interpretability techniques testifies to the joint power of providing such tools together as part of the same platform. We are actively working towards integrating further considerations into the model assessment experience such as fairness and inclusion (via FairLearn) as well as backward compatibility during updates (via BackwardCompatibilityML).

Our team

The initial work on error analysis started with research investigations on methodologies for in-depth understanding and explanation of Machine Learning failures. Besmira Nushi, Ece Kamar, and Eric Horvitz at Microsoft Research are leading these efforts and continue to innovate with new techniques for debugging ML models. In the past year, our team was extended via a collaboration with the RAI tooling team in the Azure Machine Learning group as well as the Analysis Platform team in Microsoft Mixed Reality. The Analysis Platform team has invested several years of engineering work in building internal infrastructure and now we are making these efforts available to the community as open source as part of the Azure Machine Learning ecosystem. The RAI tooling team consists of Ilya Matiach, Mehrnoosh Sameki, Roman Lutz, Richard Edgar, Hyemi Song, Minsoo Thigpen, and Anup Shirgaonkar. They are passionate about democratizing Responsible AI and have several years of experience in shipping such tools for the community with previous examples on FairLearn, InterpretML Dashboard etc. We also received generous help and expertise along the way from our partners at Microsoft Aether Committee and Microsoft Mixed Reality: Parham Mohadjer, Paul Koch, Xavier Fernandes, and Juan Lema. All marketing initiatives, including the presentation of this blog, were coordinated by Thuy Nguyen.

Big thanks to everyone who made this possible!

Related research

Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure. Besmira Nushi, Ece Kamar, Eric Horvitz; HCOMP 2018. pdf

Software Engineering for Machine Learning: A Case Study. Saleema Amershi, Andrew Begel, Christian Bird, Rob DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, Thomas Zimmermann; ICSE 2019. pdf

Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, Eric Horvitz; AAAI 2019. pdf

An Empirical Analysis of Backward Compatibility in Machine Learning Systems. Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, Eric Horvitz; KDD 2020. pdf

Understanding Failures of Deep Networks via Robust Feature Extraction. Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, Eric Horvitz. arXiv 2020. pdf

Translator announces Document Translation (Preview)

Krishna_Doss — Wed, 17 Feb 2021 21:30:00 GMT

We are announcing Document Translation, a new feature in Azure Translator service which enables enterprises, translation agencies, and consumers who require volumes of complex documents to be translated into one or more languages preserving structure and format in the original document. Document Translation is an asynchronous batch feature offering translation of large documents eliminating limits on input text size. It supports documents with rich content in different file formats including Text, HTML, Word, Excel, PowerPoint, Outlook Message, PDF, etc. It reconstructs translated documents preserving layout and format as present in the source.

Standard translation offerings in the market accept only plain text, or HTML, and limits count of characters in a request. Users translating large documents must parse the documents to extract text, split them into smaller sections and translate them separately. If sentences are split in an unnatural breakpoint it can lose the context resulting in suboptimal translations. Upon receipt of the translation results, the customer has to merge the translated pieces into the translated document. This involves keeping track of which translated piece corresponds to the equivalent section in the original document.

The problem gets complicated when customers want to translate complex documents having rich content. They convert the original file in variety of formats to either .html or .txt file format and reconvert translated content from html or txt files into original document file format. The transformation may result in various issues. The problem gets compounded when customer needs to translate a) large quantity of documents, b) documents in variety of file formats, c) documents while preserving the original layout and format, d) documents into multiple target languages.

Document Translation is an asynchronous offering to which the user makes a request specifying location of source and target documents, and the list of target output languages. Document Translation returns a job identifier enabling the user to track the status of the translation. Asynchronously, Document Translation pulls each document from the source location, recognizes the document format, applies right parsing technique to extract textual content in the document, translates the textual content into target languages. It then reconstructs the translated document preserving layout and format as present in the source documents, and stores translated document in a specified location. Document Translation updates the status of translation at the document level. Document Translation makes it easy for the customer to translate volumes of large documents in a variety of document formats, into a list of target languages thus eliminating all the challenges customers face today and improving their productivity.

Document Translation enables users to customize translation of documents by providing custom glossaries, a custom translation model id built using customer translator, or both as part of the request. Such customization retains specific terminologies and provides domain specific translations in translated documents.

“Translation of documents with rich formatting is a tricky business. We need the translation to be fluent and matching the context, while maintaining high fidelity in the visual appearance of complex documents. Document Translation is designed to address those goals, relieving client applications from having to disassemble and reassemble the documents after translation, making it easy for developers to build workflows that process full documents with a few simple steps.”, said Chris Wendt, Principal Program Manager.

To learn more about Translator and the Document Translation feature in the video below

References

Send your feedback to translator@microsoft.com

Hello, bot! Conversational AI on Microsoft Platform

shwars — Thu, 18 Feb 2021 18:23:54 GMT

During the pandemic, we all found ourselves in isolation, and relying more and more on effective electronic means of communication. The amount of digital conversations increased dramatically, and we need to rely on bots to help us handle some of those conversations. In this blog post, I give brief overview of conversational AI on Microsoft platform and show you how to build a simple educational bot to help students learn.

If you prefer video content, here is a great video to get you started, from our AI Developer Resources page:

Do We Need Bots, and When?

Many people believe that in the future we will be interacting with computers using speech, in the same way we interact between each other. While the future is still vague, we can already benefit from conversational interfaces in many areas, for example:

In user support, which has traditionally been based on interpersonal communication, automated chat-bots can solve a lot of routine problems for the users, leaving human specialists for solving only unusual cases.
During surgical operation, when hands-free interaction is essential. From personal experience, I find it personally more convenient to set morning alarm and “good night” music through voice assistant before going to sleep.
Automating some functions in interpersonal communication. My favorite example is a chat-bot that you can add to a group chat when organizing a party, and it will track how much money each of the participants spent on the preparation.

At the current state of development of conversational AI technologies, a chat bot will not replace a human, and it will not pass Turing test.

In practice, chat bots act as an advanced version of a command line, in which you do not need to know exact commands to perform an action.

Thus, successful bot applications will not try to pretend to be a human, because such behavior is likely to cause some user dissatisfaction in the future. It is one of the responsible conversational AI principles, which you need to consider when designing a bot.

Educational Bots

During the pandemic, one of the areas that is being transformed the most is education. We can envision educational bots that help student answer most common questions, or act as a virtual teaching assistant. In this blog post, I will show you how to create a simple assistant bot that will be able to handle several questions from the field of Geography.

Before we jump to this task, let’s talk about Microsoft conversational AI stack in general, and consider different development options.

Conversational AI Development Stack

When it comes to conversational AI, we can logically think of a conversational agent having two main parts:

Conversational interface handles passing messages from the user to the bot and back. It takes care about communication between user messaging agent (such as Microsoft Teams, Skype or Telegram) and our application logic, and includes code to handle request-response logic.
Intelligent backend adds some AI functionality to your bot, such are recognizing user’s phrases, or finding best possible answer.

A bot can exist without any intelligent backend, but it would not be smart. Still, bots like that are useful for automating simple tasks, such as form filling, or handling some pre-defined workflow.

Here I present slightly simplified view of the whole Bot Ecosystem, but this way it is easier to get the picture.

Conversational Interface: Microsoft Bot Framework and Azure Bot Service

At the heart of conversational interface is Microsoft Bot Framework - an open-source development framework (with source code available on GitHub), which contains useful abstractions for bot development. The main idea of Bot Framework is to abstract communication channel, and develop bots as web endpoints that asynchronously handle request-response communication.

Decoupling of bot logic and communication channel allows you to develop bot code once, and then connect it easily to different platforms, such as Skype, Teams or Telegram. Omnichannel bots are now made simple!

Bot Framework SDK supports primarily C#, Node.js, Python and Java, although C# or Node.js are highly recommended.

To host bots developed with Bot Framework on Azure, you use Azure Bot Service. It hosts bot logic itself (either as web application, or azure function), as well as allows you to declaratively define physical channels that your bot will be connected to. You can connect your bot to Skype or Telegram through Azure Portal with a few simple steps.

Intelligent Backend: LUIS and QnA Maker

Many modern bots support some form of natural language interaction. To do it, the bot needs to understand the user’s phrase, which is typically done through intent classification. We define a number of possible intents or actions that the bot can support, and then map an input phrase to one of the intents.

This mapping is typically done using a neural network trained on some dataset of sample phrases. To take away the complexity of training your own neural network model, Microsoft provides Language Understanding Intelligent Service, or LUIS, which allows you to train a model either through web interface, or an API.

In addition to intent classification, LUIS also performs named entity recognition (or NER). It can automatically extract some entities of well-known types, such as geolocations or references to date and time, and can learn to extract some user-defined entities as well.

Having entities extracted and intent correctly determined it should be much easier to program the logic of your bot. This is often done using slot filling technique: extracted entities from the user’s input populate some slots in a dictionary, and if some more values are required to perform the task - additional dialog is initiated to ask additional info from the user.

Another type of bot behavior that often comes up is the ability to find best matching phrase or piece of information in some table, i.e. do an intelligent lookup. It is useful if you want to provide FAQ-style bot that can answer user’s questions based on some database of answers, or if you just want to program chit-chat behavior with some common responses. To implement this functionality, you can use QnA Maker - a complex service, that encapsulates Azure Cognitive Search, and provides simple way to build question-answering functionality. You can index any existing FAQ document, or provide question-answer pairs through the web interface, and then hook up QnA maker to your bot with a few lines of code.

Bot Development: Composer, Power Virtual Agents or Code?

As I mentioned above, bots can be developed using your favorite programming language. However, this approach requires you to write some boilerplate code, understand asynchronous calls, and therefore has a significant learning curve. There are some simpler options that are good for a start!

It is recommended to start developing your bot using low-code approach through Bot Framework Composer - an interactive visual tool that allows you to design your bot by drawing dialog diagrams.

Composer integrates LUIS and QnA maker out of the box, so that you do not need to train those services through web interface first, and then worry about integrating them to your bot. From the same UI, you are able to specify events triggered by some user phrases, and dialogs that respond to them.

Another similar low-code option would be to use Power Virtual Agents (PVA), a tool from Power Platform family of tools for business automation. It would be especially useful if you are already familiar with Power Platform, and using any of its tools to enhance productivity. In this case, PWA will be a natural choice, and it will integrate nicely into all your data points and business processes. In short - Composer is a great low-code tool for developers, while PVA is more for business users.

Getting Started with Bot Development

Let me show you how we can start the development of a simple educational bot that will help K-12 students with their geography classes. We will develop a simple bot, which you can later host on Microsoft Azure and connect to most popular communication channels, such as Teams, Slack or Telegram. If you do not have an Azure account, you can get a free trial (or here, if you are a student).

To begin with, we will implement three simple functions in our bot:

Being able to tell a capital city for a country (What is the capital of Russia?).
Giving definitions of most useful terms, eg. answering a questions like What is a capital?
Support for simple chit-chat (How are you today?)

Those two functions cover two most important elements of our intelligent backend: QnA Maker (which can be used to implement the last two points) and LUIS.

Starting with Bot Composer

To begin development, you need to [install Bot Framework Composer][InstallComposer] - I recommend to do it as desktop application. Then, after starting it, click on New button, and chose starting template for your bot: QnA Maker and LUIS:

Once you do that, you will see the main screen of composer, with a list of triggers on the left, and the main pane to design dialogs:

Here, you can delete an unused trigger BuySurface (which is left over from the demo), and go to Welcome message to customize the phrase that the bot says to new user. The logic of Welcome Message trigger is a bit complex, you need to look for a box called Send a response, and change the message in the right pane.

The language used to define phrases is called Language generation, or lg. A few useful syntax rules to know:

A phrase starts with -. If want to chose from a number of replies, specify several phrases, and one of them will be selected randomly. For example: ```bash
Hello, I am geography helper bot!
Hey, welcome!
Hi, looking forward to chat with you! ```
Comments start with >
Some additional definitions start with @
You can use ${...} syntax for variable substitution (we will see an example of this later)

Connecting to Azure Services

To use intelligent backend, you need to create Azure resources for LUIS and QnA Maker and provide corresponding keys to Composer:

Create LUIS Authoring Resource, and make sure to remember the region in which it was created, and copy key from Keys and Endpoint page in Azure Portal.
Create QnA Maker Service, and copy corresponding key
In Composer, go to bot settings by pressing Project Settings button in the left menu (look for a wrench icon, or expand the menu if unsure). Under settings, fill in LUIS Authoring Key, LUIS region and QnA Maker Subscription key.

Starting the Bot

At this point, you can already start chatting with your bot. Click Start bot in the upper right corner, and let some time for the magic to happen. When starting a bot, Composer actually creates and trains underlying LUIS model, builds bot framework project, and starts local web server with a copy of the bot, ready to serve your requests.

To chat with a bot, click Test in emulator button (you need to have Bot Framework Emulator) installed for this to work). This automatically opens up the chat window with all required settings, and you can start talking to your bot right away.

Creating QnA Maker Knowledge base

Let’s start with creating term dictionary using QnA Maker. Click on QnA left menu, and then Create new KB.

Here you can either start from some existing data (provided as an URL to html, pdf, Word or Excel document), or start creating phrases from scratch.

In most of the cases, you would have a document to start with, but in our case we will start from scratch. After creating a knowledge base, you can click Add QnA Pair to add all question-answer combinations you need. Note that you can add several options of the question by using All alternative phrasing link.

In our case, I have added a how are you phrase (with several options), and a phrase to explain the meaning of the word capital.

Having added the phrases, we can start a bot and make sure that it correctly reacts to given phrases, or similar versions of those phrases - QnA maker does not require it to be an exact match, it looks for similar phrases to make a decision on which answer to provide.

Adding Specific Actions with LUIS

To give our bot an ability to give capitals of countries, we need to add some specific functionality to look for a capital, triggered by a certain phrase. We definitely do not want to type all 200+ countries and their capitals into QnA Maker!

A functionality to get information about a country is openly available via REST Countries API. For example, if we make GET request to https://restcountries.eu/rest/v2/name/Russia, we will get JSON response like this:

[ {"name":"Russian Federation",
   "topLevelDomain":[".ru"],
   "capital":"Moscow", ... } ]

To ask for a capital of a given country, a user will say something like What is a capital of Russia?, or I want to know the capital of Italy. This phrase intent can be recognized using LUIS, and the name of the country is also extracted.

To add LUIS trigger, from the Design page of the composer, select your bot dialog and press “…” next to it. You will see Add a trigger option in the drop-down box. Select it, and then chose Intent recognized as trigger type.

Intent Recognized is the most common trigger type. However, you can specify Dialog events, that allow you to structure part of the conversation as a separate dialog, or some conversational activities, such as Handoff to human.

Then, specify trigger phrases, using a variation of LG language. In our case, we will use the following:

- what is a capital of {country=Russia}?
- I want to know a capital of {country=Italy}.
- Give me a capital of {country=Greece}!

Here, we specify a number of trigger phrases starting with -, and we indicate that we want to extract part of the phrase as an entity country. LUIS will automatically train a model to extract entities based on the provided utterances, so make sure to provide a number of possible phrases.

There are some pre-defined entity types, such as datetimeV2, number, etc. Using pre-defined types is recommended, and entity type can be specified using @ <entity_type> <entity_name> notation. In our case, we can use geographyV2 entity type, which extracts geographic locations, including countries.

Once we have defined phrase recognizer, we need to add a block that will do the actual REST call and fetch information on the given country. Use use Send HTTP Request block, and specify the following parameters:

Method = GET
Url = https://restcountries.eu/rest/v2/name/${@country}. Here, ${@country} will be substituted with the name of the recognized country.
Result property = dialog.result
Response type = json

This would make the REST call to the API, and the result will be stored into dialog.result property. If we provided a valid country, json result will be automatically parsed, otherwise, invalid operation code will be recorded in dialog.result.statusCode - in our case, 404.

To test if the call was successful and define different logic based on the result, we insert Branch: If/Else block, and specify the following condition: = equals(dialog.result.statusCode,200). True condition will correspond to the left branch, and we will insert Send a response block there, with the following text:

- A capital of ${@country} is ${dialog.result.content[0].capital}

In case result code is not 200, the right branch will be executed, where we will insert an error message. Our final dialog should look like this:

Adding Preconfigured Chit-Chat Functionality

It would be nice if your bot could respond to more everyday phrases, such as How old are you?, or Do you enjoy being a bot. We can define all those phrases in QnA Maker, but that would take us quite some time to do so. Luckily, there is Project Personality Chat that contains a number of pre-defined QnA Maker knowledge bases for several languages, and for a number of personalities:

Professional
Friendly
Witty
Caring
Enthusiastic

You can grab a link to the knowledge base from here, then go to QnA Maker Portal, find your knowledge base, and add this URL link to your service:

Having done that, click Save and Train, and enjoy a talk to your bot! You can even try ask it about life, universe and everything!

Testing the Bot and Publishing to Azure

Now that our basic bot functionality is complete, we can test the bot in bot emulator:

Once the bot is running locally, we can deploy it to Azure right from the Composer. If you go to Publish from the left menu, you will be able to define Publishing profile for your bot. Select Define new publishing profile, and chose one of the following:

The most standard way to deploy is to use Azure Web App. Composer will only require you to provide Azure subscription and resource group name, and it will take care of creating all the required resources (including bot-specific LUIS/QnA Maker instances) automatically. It may take awhile, but it will save you a lot of time and hassle of doing manual deployment.

Once the bot is published to Azure, you can go to Azure portal and configure Channels through which you bot would be available to external world, such as Telegram, Microsoft Teams, Slack or Skype.

Conclusion

Creating a bot using Bot Composer seems like an easy thing to do. In fact, you can create quite powerful bots almost without any code! And you can also hook them to your enterprise endpoint using such features as HTTP REST APIs and OAuth authorization.

However, there are cases when you need to significantly extend bot functionality using code. In this case, you have several options:

Keep main bot authoring in Bot Composer, and develop Custom Actions in C#
Export complete bot code using Custom Runtime feature of Composer, which exports complete bot code in C# or Javascript, which you can then fully customize. This approach is not ideal, because you will lose the ability to maintain the source of your bot in Composer.
Write a bot from the beginning in one of the supported languages (C#, JS, Python or Java) using Bot Framework.

If you want to explore how the same Educational bot for Geography can be written in C#, check out this Microsoft Learn Module: Create a chat bot to help students learn with Azure Bot Service.

I am sure conversational approach to UI can prove useful in many cases, and Microsoft Conversational Platform offers you wide variety of tools to support all your scenarios.

Re: Introducing QnA Maker managed: now in public preview

Dan Nite — Wed, 17 Feb 2021 07:37:11 GMT

Hello,

When is available for ( west Europe ) Portugal ?

Re: Integrating AI: Best Practices and Resources to Get Started

Luigi Bruno — Mon, 15 Feb 2021 19:33:27 GMT

Thank you.

Computer Vision Read (OCR) API previews 73 human languages and new features on cloud and on-premise

sanjeev_jagtap — Mon, 15 Mar 2021 00:18:16 GMT

Overview

Businesses today are applying Optical Character Recognition (OCR) and document AI technologies to rapidly convert their large troves of documents and images into actionable insights. These insights power robotic process automation (RPA), knowledge mining, and industry-specific solutions. However, there are several challenges to successfully implementing these scenarios at scale.

The challenge

Your customers are global, and their content is global so your systems should also speak and read international languages. Nothing is more frustrating than not reaching your global customers due to lack of support for their native languages.

Secondly, your documents are large, with potentially hundreds and even thousands of pages. To complicate things, they have print and handwritten style text mixed into the same documents. To make matters worse, they have multiple languages in the same document, possibly even in the same line.

Thirdly, you are a business that’s trusted by your customers to protect their data and information. If your customers are in industries such as healthcare, insurance, banking, and finance, you have stringent data privacy and security needs. You need the flexibility to deploy your solutions on the world’s most trusted cloud or on-premise within your environment.

Finally, you should not have to choose between world-class AI quality, world languages support, and deployment on cloud or on-premise.

Computer Vision OCR (Read API)

Microsoft’s Computer Vision OCR (Read) technology is available as a Cognitive Services Cloud API and as Docker containers. Customers use it in diverse scenarios on the cloud and within their networks to help automate image and document processing.

What’s New

We are announcing Computer Vision's Read API v3.2 public preview as a cloud service and Docker container. It includes the following updates:

OCR for 73 languages including Simplified and Traditional Chinese, Japanese, Korean, and several Latin languages.
Natural reading order for the text line output.
Handwriting style classification for text lines.
Text extraction for selected pages for a multi-page document.
Available as a Distroless container for on-premise deployment.

First wave of language expansion

With the latest Read preview version, we are announcing OCR support for 73 languages, including Chinese Simplified, Chinese Traditional, Japanese, Korean, and several Latin languages, a 10x increase from the Read 3.1 GA version.

Thanks to Read’s universal model, to extract the text in these languages, use the Read API call without the optional language parameter. We recommend not using the language parameter if you are unsure of the language of the input document or image at run time.

The latest Read preview supports the following languages:

Read 3.2 Preview Languages

For example, once you have created a Computer Vision resource, the following curl code will call the Read 3.2 preview with the sample image.

Make the following changes in the command where needed:

Replace the value of <subscriptionKey> with your subscription key.
Replace the first part of the request URL (westcentralus) with the text in your own endpoint URL.

curl -v -X POST "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyze" -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription key>" --data-ascii "{\"url\":\"https://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Atomist_quote_from_Democritus.png/338px-Atomist_quote_from_Democritus.png\"}"

The response will include an Operation-Location header, whose value is a unique URL. You use this URL to query the results of the Read operation. The URL expires in 48 hours.

curl -v -X GET "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyzeResults/{operationId}" -H "Ocp-Apim-Subscription-Key: {subscription key}" --data-ascii "{body}"

Natural reading order output (Latin languages)

OCR services typically output text in a certain order in their output. With the new Read preview, choose to get the text lines in the natural reading order instead of the default left to right and top to bottom ordering. Use the new readingOrder query parameter with the “natural” value for a more human-friendly reading order output as shown in the following example.

The following visualization of the JSON formatted service response shows the text line order for the same document. Note the first column's text lines output in order before listing the second column and finally the third column.

OCR Read order example

For example, the following curl code sample calls the Read 3.2 preview to analyze the sample newsletter image and output a natural reading order for the extracted text lines.

curl -v -X POST "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyze?readingOrder=natural -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription key>" --data-ascii "{\"url\":\"https://docs.microsoft.com/en-us/microsoft-365-app-certification/media/dec01.png\"}"

The response will include an Operation-Location header, whose value is a unique URL. You use this URL to query the results of the Read operation.

curl -v -X GET "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyzeResults/{operationId}" -H "Ocp-Apim-Subscription-Key: {subscription key}" --data-ascii "{body}"

Handwriting style classification (Latin languages)

When you apply OCR on business forms and applications, it’s useful to know which parts of the form has handwritten text in them so that they can be handled differently. For example, comments and the signature areas of agreements typically contain handwritten text. With the latest Read preview, the service will classify Latin languages-only text lines as handwritten style or not along with a confidence score.

For example, in the following image, you see the appearance object in the JSON response with the style classified as handwriting along with a confidence score.

OCR handwriting style classification for text lines

The following code analyzes the sample handwritten image with the Read 3.2 preview.

curl -v -X POST "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyze -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription key>" --data-ascii "{\"url\":\"https://intelligentkioskstore.blob.core.windows.net/visionapi/suggestedphotos/2.png\"}"

The response will include an Operation-Location header, whose value is a unique URL. You use this URL to query the results of the Read operation.

curl -v -X GET "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyzeResults/{operationId}" -H "Ocp-Apim-Subscription-Key: {subscription key}" --data-ascii "{body}"

Extract text from select pages of a document

Many standard business forms have fillable sections followed by long informational sections that are identical between documents, and versions of those documents. At other times, you will be interested in applying OCR to specific pages of interest for business-specific reasons.

The following curl code sample calls the Read 3.2 preview to analyze the financial report PDF document with the pages input parameter set to the page range, "3-5".

curl -v -X POST "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyze?pages=3-5 -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription key>" --data-ascii "{\"url\":\"https://www.annualreports.com/HostedData/AnnualReports/PDF/NASDAQ_MSFT_2019.pdf\"}"

The response will include an Operation-Location header, whose value is a unique URL. You use this URL to query the results of the Read operation.

curl -v -X GET "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyzeResults/{operationId}" -H "Ocp-Apim-Subscription-Key: {subscription key}" --data-ascii "{body}"

The following JSON extract shows the resulting OCR output that extracted the text from pages 3, 4, and 5. You should see a similar output for your sample documents.

"readResults": [ { "page": 3, "angle": 0, "width": 8.5, "height": 11, "unit": "inch", "lines": [] }, { "page": 4, "angle": 0, "width": 8.5, "height": 11, "unit": "inch", "lines": [] }, { "page": 5, "angle": 0, "width": 8.5, "height": 11, "unit": "inch", "lines": [] } ]

On-premise option with Distroless container

The Read 3.2 preview OCR container provides:

All features from the Read cloud API preview
Distroless container release
Performance and memory enhancements

Install and run the Read containers to get started and find the recommended configuration settings.

Get Started

Create a Computer Vision resource in Azure.
Follow our SDK and REST API QuickStarts.
Learn more about OCR (Read) and Form Recognizer.
See the list of OCR supported languages.
Learn more about the Read containers and download them from Docker Hub.
Write to us at formrecog_contact@microsoft.com

Integrating AI: Best Practices and Resources to Get Started

Yonet — Thu, 11 Feb 2021 00:33:22 GMT

We use AI (Artificial Intelligence) integrated applications daily, from search engines optimized to find the most relevant content, to recommendation engines for streaming or shopping. During AI’s early years rising to popularity, improving applications with AI was only possible for companies with big budgets dedicated to research and experts, preventing companies that cannot effort an AI team to compete. Today AI is readily available for any product, without having to invest in research and development. There are open-source libraries that can help you train Machine Learning models like TensorFlow. With a fraction of the effort and the cost, pre-trained AI services are available to easily integrate into your applications, with APIs and UI based tools to train custom models for your specific use case. In Integrating AI series, I aim to help you decide if and how to integrate AI into your applications, get you started with Azure’s ready to use AI solutions, Cognitive Services and answer your most frequent questions when getting started.

Let’s start with these fundamental questions:

What are the problems you can solve with AI?
What do you need to know before starting to build your solution?
How do you measure the success of your new AI features?

What are the problems you can solve with AI?

AI is a groundbreaking technology but not a magical solution for everything. It is important to know if you are adding value or solving an actual user problem. There are complex products like Wikipedia and Reddit that have a lot of information but use crowdsourcing and simple search to cater to unique needs without the help of AI. To make an informed decision, you need to start with your users’ needs. What are the problems they face? Is there a process that you can automize like filling expense forms that can be automated with Form Recognizer service? Send voice messages to your customers with updates using Speech Services? Do they make complex choices while using your product that could be customized to your users with the use of Personalizer? Do you need to improve the usability of your application with voice interactions and Language Understanding? It is important to solve a real need for your users instead of assuming the solution that will be useful. User research is the best way to figure out the issues and a lot can be surfaced by user analytics. You can use Metrics Advisor AI service to detect anomalies and figure out future AI solutions as well.

Once you have a clear definition of the problem and define how to measure success, it is time to explore practical solutions. You can read about the Azure customer stories and learn from their methods and design process. For example, read about BBC's customer story before you read about the technical story of using Azure's Speech, Azure Bot Service and Language Understanding Services together to solve the customer needs they identified.

Most AI solutions can fall into two categories. The first major use case for AI is automating the mindless repetitive jobs. If the users of an expense report or a hiring application need to type in information from a form or a receipt to your system, it is easily automated by OCR (Optical Character Recognition). Similar automations are possible for close captioning, translation, classifying images and automizing alert messages.

The second category of AI solutions can be categorized as complex human decisions based on data. You could give your friends recommendations on what to watch next easily, knowing what they like, what they don’t like. For example, a streaming service with thousands of movies to choose from, cannot surface relevant content with simple filtering of the genres or release dates. It would take forever to choose what to watch by browsing unless you know the exact name of the movie. For a decision like recommendation among thousands or millions of results, AI might be better at recommending to your best friend, maybe even better than you over time. Understanding the language and intent of people is another example. A human can understand and classify a review as positive or negative easily. For machines to detect the same emotions, you must do more than detect certain words to get sentiment.

What do you need to know before starting to build your solution?

Some problems are easier to solve than others with AI. Experimenting with different tools to confirming your solutions is important. All the Cognitive services are easy to try out and here is how to do that:

AI Demos Website gives you a hand-on experience of Cognitive Services

You can also download the Intelligent Kiosk app to try out the demos on your local machine.

Once you create an Azure resource, you can see the code samples, API call examples and try out the Rest API end points directly on the Cognitive Services API Reference pages.
API descriptions

Code Samples

Request

Will your users love your solution?

Scaling an application and polishing the user experience takes most of the development time. It is better to try out features fast and adjust before making the investment in perfecting the wrong experience. You might assume an application flow that users are going to interact, but users can surprise you in their own creative ways of using your tools. Prototype your applications quickly and get user feedback early on.

Power platform is one of the tools that allows you to create mobile apps that integrates important AI capabilities without writing any code. With power platform, you can easily deploy and share your prototypes, without leaving the platform’s UI. After the free trial period, both training and using your AI models will cost but not as much as the development time of an actual app with AI and having to make major changes after the release. Check out some of the capabilities and use cases of AI Builder on Power Platform and how to train a custom vision model and creating a mobile app on Power Platform in this article.

There are other fast and easy options to add AI to your applications, without a big development investment, especially if you are adding the capabilities to an existing application. You can use a logic app to design an application on Azure platform to find twitter mentions of your brand and analyze the sentiment of the tweets. You can visualize the data on Power BI or your choice of visualization platform or tools.

Once you integrate your AI solution, you can make the new AI features to a limited group of users and compare the effectiveness of your solution with your non-AI features.

Start your learning journey on AI Developer resources and sign up for a free Azure Account .

Let us know the problems you are trying to solve and your specific use cases on the comments below.

Accelerate search index development with Visual Studio Code

DerekLegenzoff — Wed, 10 Feb 2021 21:34:00 GMT

Azure Cognitive Search provides developers with APIs and tools to make it easy to add a great search experience to your application. There are tools available in the portal to import data into a search index and SDKs to simplify the integration of search functionality into your code base.

However, sometimes you need something in between: simpler than code, but more powerful than the portal. In these cases, it’s common to interact directly with the REST APIs to quickly update an indexer, add a document, or perform other standard tasks. Tools like Postman are great for this but building out API calls from scratch can quickly become tedious. You wouldn’t write an API call from scratch to add a document to Azure Storage—you’d use Azure Storage Explorer—and we don’t want you to have to do that for search either.

With this in mind, we created the Visual Studio Code Extension for Azure Cognitive Search (Preview). The Visual Studio Code extension makes it easy to work with your search service using the full capabilities of the REST APIs while providing rich IntelliSense and snippets to help you. With the extension, you can create and update indexes and other components, add documents, search, and more. You’ll never need to struggle with remembering the correct syntax again.

Extension Functionality

The extension covers all the major REST API operations for Cognitive Search. Check out the examples below to see some of what’s possible and feel free to request additional functionality here.

Browse all your Azure Cognitive Search services

Get access to all your search services in one place. You can quickly see all your indexes, indexers, and other components.

Create new indexes, indexers, data sources, skillsets, and synonym maps

You can create a new index or other component just by editing the JSON and saving the file. You can then read, update, or delete these components at any time.

Take advantage of rich IntelliSense

The extension also includes IntelliSense to guide you as you’re building out your JSON. Instead of referencing external docs each time, you can see what parameters exist and what their allowed values are as you type.

In addition to IntelliSense, the extension provides snippets or templates for building more complex objects, such as data sources and skillsets, so that you have a good starting point.

Add or update documents in the search index

Adding or updating documents is something that’s not possible in the portal today. With the extension, you can quickly add a document, and it will even save you some time by creating a JSON template for you based on your index definition.

You can view or update existing documents too.

Query your search indexes

Finally, once you’ve added documents to your search service, you can also query from within the extension and view the results side by side. You can even add multiple queries or save the queries to a file to refer to them later.

Key use cases

If your security requirements mandate the use of Private Endpoints or IP Firewalls, you’ll find that some functionality is no longer available in the portal. For these cases, the extension is a great alternative to the portal for interacting with your indexes and the other components of your search service.

In other cases, if you find yourself constantly recreating indexes or making small tweaks to them or other search components, the extension can make it incredibly easy to make small updates such as adding a field to an index.

Get Started

Regardless of how you’re trying to use Cognitive Search, this extension will likely make your life easier. To get started today, download the extension, and follow the related quickstart. You’ll see just how quickly and easily you can get up and running with Cognitive Search using the Visual Studio Code Extension.

If you run into any issues or have any questions, please feel free to reach out to us at azuresearch_contact@microsoft.com or raise an issue on the extension’s GitHub repo.

Re: Build a natural custom voice for your brand

anandmicrosoft — Wed, 10 Feb 2021 18:31:28 GMT

Tried the ACC tool. Lots of Fun. LOL. Thrilled to use it shortly :)

How to use Cognitive Services and containers

hboelman — Thu, 11 Feb 2021 06:43:42 GMT

In this blog we are going to take a look at how we can run a selection of Cognitive Services in Docker compatible containers. This option of using these services can come in handy if you run into scenarios where your application can not connect to the cloud all the time or if you need more control over your data.

What are Cognitive Services

Azure Cognitive Services are cloud-based services that expose AI models through a REST API. These services enable you to add cognitive features, like object detection and speech recognition to your applications without having data science skills. By using the provided SDKs in the programming language of your choice you can create application that can see (Computer Vision), hear (Speech), speak (Speech), understand (Language), and even make decisions (Decision).

Cognitive Services in containers

Azure Cognitive Service in containers gives developers the flexibility in where to deploy and host the services that come with Docker containers and keeping the same API experience as when they where hosted in the Azure.

Using these containers gives you the flexibility to bring Cognitive Services closer to your data for compliance, security or other operational reasons.

What are containers
Containerization is an approach to software distribution in which an application or service, including its dependencies & configuration, is packaged together as a container image. With little or no modification, a container image can be deployed on a container host. Containers are isolated from each other and the underlying operating system, with a smaller footprint than a virtual machine. Containers can be instantiated from container images for short-term tasks, and removed when no longer needed.

When to use Cognitive Services in containers?

Running Cognitive Services in containers can be the solution for you if you have specific requirements or constraints making that make it impossible to run Cognitive services in Azure. The most common scenarios include connectivity and control over the data. If you are running Cognitive Services in Azure all the infrastructure is taken care of, running them in containers moves the infrastructure responsibility, like performance and updating the container, to you.

A case where you choose for container could be, if your connection to Azure is not stable enough. For instance if you have 1000's of document on-prem and you want to run OCR. If you use the Computer Vision OCR endpoint in the cloud you would need to send all the documents to the end point in azure, while if you run the container locally you only need to send the billing information every 15 minutes to Azure.

Features and benefits

Immutable infrastructure: Enable DevOps teams' to leverage a consistent and reliable set of known system parameters, while being able to adapt to change. Containers provide the flexibility to pivot within a predictable ecosystem and avoid configuration drift.

Control over data: Choose where your data gets processed by Cognitive Services. This can be essential if you can't send data to the cloud but need access to Cognitive Services APIs. Support consistency in hybrid environments – across data, management, identity, and security.

Control over model updates: Flexibility in versioning and updating of models deployed in their solutions. Portable architecture: Enables the creation of a portable application architecture that can be deployed on Azure, on-premises and the edge. Containers can be deployed directly to Azure Kubernetes Service, Azure Container Instances, or to a Kubernetes cluster deployed to Azure Stack. For more information, see Deploy Kubernetes to Azure Stack.

High throughput / low latency: Provide customers the ability to scale for high throughput and low latency requirements by enabling Cognitive Services to run physically close to their application logic and data. Containers do not cap transactions per second (TPS) and can be made to scale both up and out to handle demand if you provide the necessary hardware resources.

Scalability: With the ever growing popularity of containerization and container orchestration software, such as Kubernetes; scalability is at the forefront of technological advancements. Building on a scalable cluster foundation, application development caters to high availability.

Which services are available

Container support is currently available for a subset of Azure Cognitive Services, including parts of:

Group	Service	Documentation
Anomaly Detector	Anomaly Detector	Documentation
Computer Vision	Read OCR (Optical Character Recognition)	Documentation
	Spatial Analysis	Documentation
Form Recognizer	Form Recognizer	Documentation
Language Understanding	Language Understanding	Documentation
Speech	Custom Speech-to-text	Documentation
	Custom Text-to-speech	Documentation
	Speech-to-text	Documentation
	Text-to-speech	Documentation
	Neural Text-to-speech	Documentation
	Speech language detection	Documentation
Text Analytics	Key Phrase Extraction	Documentation
	Text language detection	Documentation
	Sentiment analysis	Documentation
Face	Face	Documentation

How to use Cognitive Services in containers

The use of the services in containers is exactly the same as if you would use them in Azure. The deployment of the container is the part that takes a bit of planning and research. The services are shipped in Docker Containers. This means that the containers can be deployed to any Docker compatible platform. This can be your local machine running Docker Desktop or a fully scalable Kubernetes installation in your on premise data center.

Generic workflow

Create the resource in Azure
Get the endpoint
Retrieve the API Key
Find the container for the service
Deploy the container
Use the container endpoint as you would use the API resource

Optional you can mount your own storage and connect Application Insights.

Tutorial: Run a Text to Speech container in an Azure Container Instance.

In this tutorial we are going to run a Cognitive Service Speech container in an Azure Container Instance and use the REST API to convert text into speech.

To run the code below you need an Azure Subscription. if you don’t have an Azure subscription you can get $200 credit for the first month. And have the Azure command-line interface installed. If you don't have the Azure CLI installed follow this tutorial.

1. Create a resource group

Everything in Azure always start with creating a Resource Group. A resource group is a resource that holds related resources for an Azure solution.

To create a resource group using the CLI you have to specify 2 parameters, the name of the group and the location where this group is deployed.

az group create --name demo_rg --location westeurope

2. Create Cognitive Service resource

The next resource that needs to be created is a Cognitive Services. To create this resource we need to specify a few parameters. Besides the name and resource group, you need to specify the kind of cognitive service you want to create. For our tutorial we are creating a 'SpeechServices' service.

az cognitiveservices account create \
    --name speech-resource \
    --resource-group demo_rg \
    --kind SpeechServices \
    --sku F0 \
    --location westeurope \
    --yes

3. Get the endpoint & API Key

If step 1 and 2 are successfully deployed we can extract the properties we need for when we are going to run the container in the next step. The 2 properties we need are the endpoint URL and the API key. The speech service in the container is using these properties to connect to Azure every 15 minutes to send the billing information.

To retrieve endpoint:

az cognitiveservices account show --name speech-resource --resource-group demo_rg  --query properties.endpoint -o json

To retrieve the API keys:

az cognitiveservices account keys list --name speech-resource --resource-group demo_rg

3. Deploy the container in an ACI

One of the easiest ways to run a container is to use Azure Container Instances. With one command in the Azure CLI you can deploy a container and make it accessible for the everyone.

To create an ACI it take a few parameters. If you want your ACI to be accessible from the internet you need to specify the parameter: '--dns-name-label'. The URL for the ACI will look like this: http://{dns-name-label}.{region}.azurecontainer. The dns-name-label property needs to be unique.

az container create 
    --resource-group demo_rg \
    --name speechcontainer \
    --dns-name-label <insert unique name> \
    --memory 2 --cpu 1 \
    --ports 5000 \
    --image mcr.microsoft.com/azure-cognitive-services/speechservices/text-to-speech:latest \
    --environment-variables \
        Eula=accept 
        Billing=<insert endpoint> 
        ApiKey=<insert apikey>

The deployment of the container takes a few minutes.

4. Validate that a container is running

The easiest way to validate if the container is running, is to use a browser and open the container homepage. To do this you first need to retrieve the URL for the container. This can be done using the Azure CLI with the following command.

az container show --name speechcontainer --resource-group demo_rg --query ipAddress.fqdn -o json

Navigate to the URL on port 5000. The URL should look like this: http://{dns-name-label}.{region}.azurecontainer.io:5000/

If everything went well you should see a screen like this:

5. Submit your first task

The Text to Speech service in the container is a REST endpoint. To use it we would need to create a POST request. There are many ways to do a POST request. For our tutorial we are going to use Visual Studio Code to do this.

Requirements:

Download and Install Visual Studio Code
Install a plugin called REST Client

If you have the Visual Studio Code with th REST Client installed create a file call: rest.http and copy past the code below in the file.

POST http://<dns-name-label>.<region>.azurecontainer.io:5000/speech/synthesize/cognitiveservices/v1  HTTP/1.1
Content-Type: application/ssml+xml
X-Microsoft-OutputFormat: riff-24khz-16bit-mono-pcm
Accept: audio/*

<speak version="1.0" xml:lang="en-US">
    <voice name="en-US-AriaRUS">
        The future we invent is a choice we make. 
        Not something that just happens.
    </voice>
</speak>

Change the name of the URL to the URL of your ACI.
Next click on the Send Request link (just above the URL)

On the right side of VS Code you should see the response of the API. In the top right corner you see "Save Response Body" click on the button and save the response as a .wav file. Now you can use any media player to play the response.

Learn more

Microsoft Learn

Microsoft Learn is a free, online training platform that provides interactive learning for Microsoft products and more.

For this blog we have created a custom Collection of Learn Modules covering all the topics in depth.

Blogs and articles

On Microsoft Docs

Build a natural custom voice for your brand

Qinying Liao — Thu, 04 Feb 2021 05:15:47 GMT

Custom Neural Voice is a Text-to-Speech (TTS) feature of Speech in Azure Cognitive Services that allows you to create a one-of-a-kind customized synthetic voice for your brand. Since its preview in September 2019, Custom Neural Voice has empowered organizations such as AT&T, Duolingo, Progressive, and Swisscom to develop branded speech solutions that delight users. (For more details, read the Innovation Stories blog).

Today, we are excited to announce that Custom Neural Voice is now generally available (GA). It is important to note that although Custom Neural Voice is GA from a technological standpoint, interested customers must apply and be approved to use it. Alternatively, developers can add TTS capabilities to their apps quickly by creating an Azure Speech instance and selecting from over 200 pre-built TTS and Neural TTS voices across 54 languages/locales.

In this blog, we’ll introduce how Custom Neural Voice works and share best practices in responsibly creating a highly natural brand voice for your apps. If you have questions, join us at our ‘Ask-Microsoft-Anything’ on Wednesday, 2/10 at 9AMPT. Add to Calendar.

Your voice, your brand

In a world where voice-based interactions are increasingly becoming the norm, your voice is your brand. A recognizable digital voice helps your customers connect with your brand in new ways.

In recent years we have seen increased interest from a broad range of companies across Media and Entertainment, Telecom, Automobile, Education, and Hospitality, who consider voice-based interactions from a range of devices like phones, speakers, TV/cable boxes, and cars as a key interaction point with their customers. These organizations are looking to have a consistent, branded experience delivered directly to their customers. To highlight one such example, below is an audio sample of the 'Flo' virtual chatbot from Progressive.

Voice Sample: 'Flo' from Progressive

Custom Neural Voice empowers people and organizations in many ways. The following scenarios are examples of use cases where customers find Custom Neural Voice particularly useful and valuable:

Customer Service Chatbots – Companies can automate their call center operation with conversational AI to answer calls from customers with a natural-sounding voice that conveys friendliness, empathy, and professionalism and other values that are important to companies. For example, Progressive is using Custom Neural Voice to enable their virtual version of Flo to help their customers with ‘everything from getting a free car insurance to general insurance questions’. Read the full story.

Voice Assistants – Companies developing smart assistants on appliances, cars, and homes can use Custom Neural Voice to create a unique synthetic voice that conveys the brand of the company, the persona of the assistant and a speaking style that enables the best experience for their target users. With Custom Neural Voice, Swisscom was able to create a multilingual voice assistant that sound human and unique to Swisscom and resonates with its audience. Read the full story.

Online Learning – Education providers can add speech to their learning material with a voice that is suitable for the subjects and the students, thereby improving the engagement of the students and the effectiveness of the learning. Duolingo is using the Custom Neural Voice capability to develop stylized voices for their virtual characters for their online learning experience. Learn more.

Audio Books – Content publishers can turn written content into audio that is spoken with a synthetic voice to make it more accessible to the global audience. With Custom Neural Voice, the content publishers can create one or more unique voices with natural reading styles that match the subject and context of the content as well as the preference of the listeners. The Beijing Hongdandan Visually Impaired Service Center is using the Custom Neural Voice capability to produce audiobooks based on the voice of Lina, a trainer at the organization who is familiar to the people who are blind in China.

Assistive Technology and Real-time Translations – Custom Neural Voice can be used in situations to assist people in need or improve accessibility. When used as an assistive technology, people with speech impairment could use the technology to enable them to communicate with others with a voice that sounds like them. Custom Neural Voice can used in other situations such as real-time translation allowing people to communicate with others in a foreign language in a familiar voice.

Public Service Announcement – Public service organizations can use Custom Neural Voice to create a voice that is suitable for public announcements, whether it is in an airport, a train terminal, or other venues. The use of synthetic voice provides the ability to generate announcements with dynamic content that cannot be recorded ahead of time.

Benefit of Custom Neural Voice

Traditionally, TTS requires a large volume of voice data—in the range of 10,000 lines or more—to produce a fluent voice model. Consequently, TTS models with fewer recorded lines tend to sound noticeably robotic.

With the innovation of deep neural networks and a powerful base model built with speech data from many different speakers, Neural TTS can 'learn' the way phonetics are combined in natural human speech rather than using classical programming or statistical methods.

Empowered with this technology, Custom Neural Voice enables users to build highly-realistic voices with just a small number of training audios. This new technology allows companies to spend a tenth of the effort traditionally needed to prepare training data while at the same time significantly increasing the naturalness of the synthetic speech output when compared to traditional training methods.

Listen to the samples created with Custom Neural Voice below. Or try more demos on the Speech Studio.

Language	Voice	Human	TTS (Custom Neural Voice)
Chinese (Mandarin, simplified)	Lina (Hongdandan)
English (Australia)	Thomas
English (United States)	Angela
French (France)	Zoe (Swisscom)
German (Germany)	Lara (Swisscom)

How it works

Custom Neural Voice is based on Neural TTS technology that creates a natural-sounding voice. The realistic and natural sounding voice of Custom Neural Voice can represent brands, personify machines, and allow users to interact with applications conversationally in a natural way.

The underlying Neural TTS technology used for Custom Neural Voice consists of three major components: Text Analyzer, Neural Acoustic Model, and Neural Vocoder. To generate natural synthetic speech from text, the text is first input into Text Analyzer, which provides output in the form of phoneme sequence. A phoneme is a basic unit of sound that distinguishes one word from another in a particular language. A sequence of phonemes defines the pronunciations of the words provided in the text. Then the phoneme sequence goes into the Neural Acoustic Model to predict acoustic features that define speech signals, such as the timbre, speaking style, speed, intonations, and stress patterns, etc. Finally, the Neural Vocoder converts the acoustic features into audible waves so that synthetic speech is generated.

Neural TTS voice models are trained using deep neural networks based on real voice recording samples. With the customization capability of Custom Neural Voice, you can adapt the Neural TTS engine to better fit your user scenarios. To create a custom neural voice, visit the Speech Studio to upload the recorded audio and corresponding scripts, train the model, and deploy the voice to a custom endpoint. Depending on the use case, Custom Neural Voice can be used to convert text into speech in real-time (e.g., used in a smart virtual assistant) or generate audio content offline (e.g., used as in audiobook or instructions in e-learning applications) with the text input provided by the user. This is made available through REST APIs, Speech SDK, or a no-code Audio Content Creation tool.

Building a Custom Neural Voice

As part of Microsoft’s commitment to responsible AI, we are designing and releasing Custom Neural Voice with the intention of protecting the rights of individuals and society, fostering transparent human-computer interaction, and counteract the proliferation of harmful deepfakes and misleading content. For this reason, we have limited the access and use of Custom Neural Voice. Submit an intake form here.

Microsoft requires every customer to obtain explicit written permission from the voice talent before creating a voice model (see Disclosure for Voice Talent). In addition, you must not use custom neural voice for certain prohibited use cases (see Code of Conduct) and must disclose the synthetic nature of the service to your users upon deployment of the custom voice model (see Disclosure Guidelines).

When preparing your recording script, make sure you include the following sentence to acquire the voice talent’s acknowledgement of using their voice data to create a TTS voice model and generate synthetic speech.

“I [state your first and last name] am aware that recordings of my voice will be used by [state the name of the company] to create and use a synthetic version of my voice.”

As a technical safeguard intended to prevent misuse of Custom Neural Voice services, Microsoft will use this recording to verify that the voice talent’s voice in the script matches the voice provided in the training data through the Speaker Verification technology. Read more about this process in the Data and Privacy document.

In the video below, we introduce how to use the Speech Studio to create a highly natural voice with your own data.

Creating a great custom voice requires careful quality control in each step, from voice design, data preparation, to the deployment of the voice model to your system. This docs page outlines in more detail the characteristics, limitations and the best practices in designing and building a custom neural voice.

Below are some key steps to take when creating a custom neural voice for your organization. (Note: this presumes you have applied and have been approved for use of Custom Neural Voice.)

Step 1: Persona design

First, design a persona of the voice that represents your brand using a persona brief document that defines elements such as the features of the voice, and the character behind the voice. This will help to guide the process of creating a custom voice model, including defining the scripts, selecting your voice talent, training and voice tuning.

Step 2: Script selection

Carefully select the recording script to represent the user scenarios for your voice. For example, you can use the phrases from bot conversations as your recording script if you are creating a customer service bot. Include different sentence types in your scripts, including statements, questions, exclamations, etc.

Step 3: Preparing training data

We recommend that the audio recordings be captured in a professional quality recording studio to achieve a high signal-to-noise ratio. The quality of the voice model heavily depends on your training data. Consistent volume, speaking rate, pitch, and consistency in expressive mannerisms of speech are required.

Common issues with recordings include speaking style mismatch (e.g., not in an ‘excited’ manner that you want to the voice to be), unnatural speed, unstable breaks, wrong pronunciation on words, etc. It is recommended that you work with a voice director to control the recording quality. Follow the recording guidance here.

Once the recordings are ready, follow the instructions here to prepare the training data in the right format.

Step 4: Testing

Prepare test scripts for your voice model that cover the different use cases for your apps. It’s recommended that you use scripts within and outside the training dataset so you can test the quality more broadly for different content.

Step 5: Tuning and adjustment

The style and the characteristics of the trained voice model depend on the style and the quality of the recordings from the voice talent used for training. However, several adjustments can be made using SSML (Speech Synthesis Markup Language) when you make the API calls to your voice model to generate synthetic speech. SSML is the markup language used to communicate with the TTS service to convert text into audio. The adjustments include change of pitch, rate, intonation, and pronunciation correction. If the voice model is built with multiple styles, SSML can also be used to switch the styles.

All of the SSML markups mentioned above can be passed directly to the API. We also provide an online tool, Audio Content Creation, that allows customers to fine-tune their audio output using a friendly UI.

Get started

Interested in building a custom neural voice? Check the languages supported. Sign up to Speech service on Azure and get started on the Speech Studio.

Besides the capability to customize TTS voice models, Microsoft offers over 200 neural and standard voices covering 54 languages and locales. With these Text-to-Speech voices, you can quickly add read-aloud functionality for a more accessible app design, or give a voice to chatbots to provide a richer conversational experiences to your users.

For more information:

Join us during our ‘Ask Microsoft Anything’ on Wed., Feb. 10^th (9amPT) (add to Calendar)
Add Text-to-Speech to your apps today
Apply for access to Custom Neural Voice
Learn more about responsible use of Custom Neural Voice
Try our demo to listen to existing neural voices

QnA with Azure Cognitive Search

nerajput — Wed, 03 Feb 2021 07:59:57 GMT

QnA + Azure Cognitive Search enables instant answers over your search results. Now, you do not need to spend time looking through your pile of documents to find the exact answer to your query. There will be an instant answer coming up for the user query from the most relevant documents present in your system. A solution where you can ingest your pile of documents and query over them to get the answer as well as related relevant documents to get more information.

This solution accelerator enables automatic bulk ingestion of documents for QnA processing via a Cognitive Search custom skill. The sample UI showcases the combined experience of instant answers to your questions as well as the list of relevant documents. Finally, the solution is easily deployed using a simple Deploy button, which sets up all necessary services in your Azure subscription

Benefits:

Converged search experience powering instant answer and relevant documents.
Search using natural language queries.
One-click deployment.
Saves end user time during search.
Flexibility to enhance and edit instant answers.

The solution combines the power of both Azure Cognitive Search and QnA Maker to extract question-answer pairs from your documents before storing them in the index. Once you deploy the solution, you get a single endpoint where for each end user query both the services will be called in parallel and you will get a combined result with an instant answer powered by QnA Maker along with the relevant documents coming from Azure Cognitive Search. Checkout the Cognitive Search Question Answering Solution Accelerator (github.com)

Architecture:

This solution accelerator contains the following artifacts:

ARM template to set up the solution.
Custom skill in Azure Cognitive Search, which ingests the data into QnA Maker.
User interface to view the results.

Live Demo Link:

You can view a live demo of this repo at the following link: https://aka.ms/qnaWithAzureSearchDemo

File Type Supported:

Currently instant answers will only be available for the file types supported by QnA Maker. By default, the logic in the Azure Cognitive Search service indexer also ingests only the following file types: .pdf,.docx,.doc,.xlsx,.xls,.html,.rtf,.txt,.tsv. You can change this by modifying the indexedFileNameExtensions property in the Indexer.json.

Tutorial:

NOTE: You need to have a GitHub account and Azure subscription to try out this solution.

Resource creation and deployment:

Click here to Deploy to Azure.
This will take you to the create blade where all the information will be pre-filled, as shown below. Click Review+ Create button to proceed.

Your deployment process will take 4-5 minutes to complete. Once completed you will land up on the following page:

Click on Deployment details to check all the resources that have been created.

Initialization:

To initialize the solution, click on the “Outputs” button on the left. Copy the “http trigger to initialize accelerator" value. Open a new browser tab and paste this URL into the browser. This will run for about a minute, and then you'll see a message indicating success or failure.
If the initialization is successful, then following message will appear:
Once the resources are initialized, you can access the portal through the “UI portal link” value in the Output tab.

Upload Documents:

You can upload the documents one by one through the UI portal, by going to the Upload tab.

You can also upload the documents in bulk, through a container.
- Go to your storage account.
- Click on Containers and select qna-container to upload the documents in bulk.
- Use the Upload tab and select the multiple files you want to ingest. It will take some time to index the documents and to extract the Question Answer pairs out of the documents.

Question Answer Enhancement:

Once the ingestion is complete, you can view all the Question Answer pairs extracted from the documents by clicking on “Knowledge Base”.
Play with your knowledge base!, You can also test for different queries using the Test Pane. Once you are satisfied with the experience, click on “Save and Train” and then “Publish” the changes to get these changes reflected on your main portal.

This solution has been specifically created for our customers to address the long-term standing ask to retrieve an instant answer from the relevant document. This solution currently covers the basic functionality, and we will keep adding more features based on user interaction and customer’s feedback. Please feel free to drop us a mail at search-qna-solution@microsoft.com to provide your valuable feedback.

Useful Links:

Re: How BERT is integrated into Azure automated machine learning

goroggy — Fri, 29 Jan 2021 19:36:36 GMT

Hi there,

with BERT we don’t technically need to train it as it’s pretrained on a large corpus of text.

It may be worth mentioning to which natural languages this statement holds.

Unified Neural Text Analyzer: an innovation to improve Neural TTS pronunciation accuracy

Qinying Liao — Thu, 28 Jan 2021 09:38:10 GMT

Introducing Unified Neural Text Analyzer: an innovation for Neural Text-to-Speech pronunciation accuracy improvement

This post is co-authored by Dongxu Han, Junwei Gan and Sheng Zhao

Neural Text-to-Speech (Neural TTS), part of Speech in Azure Cognitive Services, enables you to convert text to lifelike speech for more natural user interactions. Neural TTS has powered a wide range of scenarios, from audio content creation to natural-sounding voice assistants, for customers from all over the world. For example, BBC, Progressive and Motorola Solutions are using Azure Neural TTS to develop conversational interfaces for their voice assistants in English speaking locales. Swisscom and Poste Italiane are adopting neural voices in French, German and Italian to interact with their customers in the European market. Hongdandan, a non-profit organization, is adopting neural voices in Chinese to make their online library audible for the blind people in China.

In this blog, we introduce our latest innovation in the Neural TTS technology that helps to improve the pronunciation accuracy significantly: Unified Neural Text Analyzer.

What is text analyzer?

Neural TTS converts plain text into wave form via three modules: neural text analyzer, neural acoustic model and neural vocoder. Text analyzer converts plain text to pronunciations, acoustic model converts pronunciations to acoustic features and finally vocoder generates waveforms. Text analyzer is the first link of the entire TTS system with results directly affecting the acoustic model and vocoder. The correct pronunciation of a word or phrase is the basic expectation in TTS, which delivers the right information to use but it’s not always easy. For example, “live” should be read different in “We live in a mobile world” and “TV Apps and live streaming offerings from The Weather Network” depending on context. If TTS reads them incorrectly, the intelligibility and naturalness of the content will be significantly influenced. Thus, text analyzer is important to TTS.

Recent updates on Neural TTS include a major innovation to the text analyzer, called “UniTA” (Unified Neural Text Analyzer). UniTA is a unified text analyzer model, which seamlessly simplifies text analyzer workflow and reduces time latency in the runtime server. It adopts a multitask learning approach, jointly training all ambiguity models to solve context ambiguity and generate correct pronunciation and as a result reduces over 50% of pronunciation errors.

What are the challenges?

Generally, different natural languages have different linguistic grammar. In TTS, text analyzer needs to follow the same grammar of languages in order to generate correct pronunciations, which contains but isn’t limited to the following required grammar categories:

Word Segmentation is the process of dividing the written text into meaningful units, such as words. In English and many other languages using some form of the Latin alphabet, the space is a good approximation of a word divider. On the other hand, in languages such as Chinese or Japanese, there is no spacing in sentences. Different word segmentation results may cause different meanings and pronunciations.
Part-of-Speech Tagging is the process of marking up a word in a text as corresponding to a particular part of speech (such as noun, verb, adj, adv and so on), based on both its definition and its context.
Morphology is the progress of classifying words according to shared inflectional categories such as person (first, second, third), number (singular vs. plural), gender (masculine, feminine, neuter) and case (nominative, oblique, genitive) with a given lexeme.
Text Normalization is the process of transforming digits or symbols to their standard format for disambiguation, for example: “$200" would be normalized as "two hundred dollars”, “200M" would be normalized as "two hundred meters” or “two hundred million”.
Similar to Text Normalization, Abbreviation Expansion is the process of transforming non-standard words to their standard format for disambiguation, for example: “VI" would be normalized as "six”, “St" would be normalized as "Saint” or “street”.
Polyphone Disambiguation is the process of marking up polyphone word (heteronym word, which has one spelling but has more than one pronunciation and meaning) to its correct pronunciation based on its context.

Category	Example
Word Segmentation	[English] Nice to meet u:) --> Nice / to / meet / u / :) [Chinese] 在圣诞节纽约大都会有演出 --> 在 / 圣诞节 / 纽约 / 大 / 都会(du1 hui4) / 有 / 演出 [Chinese] 在圣诞节纽约大都会有演出 --> 在/ 圣诞节 / 纽约 / 大都(da4 dou1) / 会 / 有 / 演出
Part-of-Speech Tagging	[Noun, \| l ai v s \|] Many people have lost their lives since the cyclone because aid has not been able to be distributed. [Verb, \| l I v s \|] I also discovered the very angry raccoon that lives near my porch.
Morphology	[Singular] 1km --> one kilometer [Plural] 5km --> five kilometers
Text Normalization	[Fraction, nine out of ten] The O.S. Speed T1202 ups the ante for race-winning performance, resulting in a power plant that will dominate 9/10 scale competition. [Date, September tenth] 1st episode will air 9/10 with never before seen video of her birth!
Abbreviation Expansion	[Street] Oh man, biking from 24th St BART to the 29th St bikeshare station, that will be sweet. [Saint] We continue to ask anyone who was in the wider area near St Heliers School between 7.30am and 9am and witnessed any suspicious activity to contact police
Polyphone Disambiguation	[p r ih - z eh 1 n t] The prices will present the estimated discount utilizing the drug discount card. [p r eh 1 - z ax n t] But our present situation is not a natural one.

Most pronunciations are affected by these categories based on syntactic or semantic context, and these categories are all challenging disambiguation problems. The traditional TTS approach is a pipeline-based module called “text analyzer” with a series of models aimed at solving grammar disambiguation problems, which causes some of the following issues:

Complex model. Redundant models are built and optimized separately but implemented together in the traditional text analyzer, which causes pipeline long and complicated.
Error propagation. Accumulated errors caused by the models isolated would affect the final results.
High latency. Models run one by one in the traditional text analyzer which is pipeline-based. Time cost is high in the runtime server.

Compared to the traditional pipeline-based text analyzers, our Neural TTS proposes a Unified Neural Text Analyzer model (UniTA) to improve TTS pronunciation.

It builds a unified text analyzer model, which greatly simplifies the text analyzer workflow and reduces time latency in the runtime server.
It adopts a multitask learning approach, jointly training all ambiguity models to solve context ambiguity and generate the correct pronunciations, reducing pronunciation errors by over 50%.

How does UniTA improve pronunciations?

Firstly, UniTA converts the input text to word embedding vectors through a pre-trained model. Word embedding is a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from vocabulary are mapped to vectors of real numbers. Conceptually, it involves a mathematical embedding from a space with many dimensions per word to a continuous vector space with a much lower dimension. Pre-training models like XYZ-Code have demonstrated unprecedented effectiveness for learning universal language representations based on unlabeled corpus with the method achieving great success in many tasks like language understanding and language generation.

Secondly, a sequence tagging fine-tune strategy is adopted in the UniTA model. UniTA is designed as a typical word classification task, in which

Word Segmentation predicts word delimiter as word boundary or not.
Part-of-Speech (POS) predicts “noun”, “verb”, “adj” and so on to classify word part-of-speech.
Morphology predicts “singular”, “plural”, “masculine”, “feminine”, “neuter” and so on to classify word number, gender and case.
Text Normalization (TN) predicts candidate digits to “cardinal”, “date”, “time”, “stock” or other TN categories, and then an auxiliary component “TN Rule” helps convert digits to word form based on predicted category.
Abbreviation Expansion predicts candidate abbreviation word to its expanded form.
Polyphone disambiguation predicts polyphone words’ pronunciation. An auxiliary component, “Lexicon” is used here for achieving non-polyphone words’ pronunciations.

Different from the traditional text analyzer training models , UniTA adopts a multitask learning approach to jointly train all categories together including word segmentation, part-of-speech tagging, morphology, abbreviation expansion, text normalization and polyphone disambiguation. The multitask learning approach shares hidden layers’ information and jointly trains across different tasks, which has achieved state-of-art achievements on many NLP tasks. In UniTA, hidden information is also shared in models when training.

For example, the sentence “St. John had a 10-3 run to build its lead to 78-64 with 4:44 left.” in the training corpus is annotated as showed in the table below. “--” means there is no related tag in the category. In the word segmentation column, the phrase “10-3” is segmented as “10”, “-” and “3”; in the morphology column, the word “had” is annotated as “past tense”; in the text normalization column, “10-3” belongs to interpreting word “to” instead of “-“ while “4:44” belongs to the pattern using time format; In the abbreviation column, word “St.” is expanded as “Saint” rather than “Street”; and in the polyphone disambiguation column, the word “lead” is pronounced as [l i: d]. Actually, the word “lead” has two pronunciations, it is pronounced as [l i: d] when its POS is noun while pronounced as [l e d] when its POS is verb. This means the POS results and Polyphone results can share the inner information. In this way, multitask model improves UniTA accuracy.

Word	Word Segmentation	Part-of-Speech	Morphology	Text Normalization	Abbreviation	Polyphone disambiguation
*St.*	--	Noun	--	--	Saint	--
*John*	--	Noun	--	--	--	--
*had*	--	Verb	Past tense	--	--	--
a	--	Det	--	--	--	--
*10-3*	10 / - / 3	Num	--	numbers are predicted as “ten to three”	--	--
*run*	--	Noun	Singular	--	--	--
to	--	Particle	--	--	--	--
*build*	--	Verb	--	--	--	--
*its*	--	Det	--	--	--	--
*lead*	--	Noun	Singular	--	--	l i: d
to	--	Particle	--	--	--	--
*78-64*	78 / - / 64	Num	--	numbers are predicted as “seventy-eight to sixty-four”	--	--
*with*	--	Prep	--	--	--	--
*4:44*	4 / : / 44	Num	--	numbers are predicted as time format	--	--
*left*	--	Verb	Past participle	--	--	--
.	--	Symbol	--	--	--	--

UniTA model predicts categories’ results together in the neural TTS runtime service. The same as training, UniTA converts the plain texts to word embeddings and then the multitask sequence tagging model predicts all the categories’ results. Some auxiliary modules are embedded after fine-tuning categories to further improve pronunciations. Finally, the pronunciation results are generated from UniTA.

Here is the figure of the UniTA model structure in Neural TTS:

UniTA model diagram

Pronunciation accuracy improved with UniTA

Compared with the traditional TTS text analyzer, UniTA reduces over 50% of pronunciation errors in improving pronunciation accuracy. It is already used many neural voice languages such as English (United States), English (United Kingdom), Chinese (Mandarin, simplified), Russian (Russia), German (Germany), Japanese (Japan), Korean (Korea), Polish (Poland) and Finnish (Finland). Due to varying types of grammar in language, not all categories are suitable for every language. For example, Chinese and Japanese heavily depend on word segmentation and polyphone while these languages don’t need morphology or abbreviation expansion.

Here are some samples of the pronunciation improvement using UniTA.

Category	Language	Input text (target word bolded)	Previous pronunciation	Current pronunciation
Word Segmentation	Chinese (Mandarin, simplified)	太子与三殿下行过礼后坐了片刻就离开了。	“三殿 / 下行 / 过礼”	“三殿下 / 行过礼”
Word Segmentation	Chinese (Mandarin, simplified)	叶奎最终还是在剧痛下泄了气	“剧痛 / 下泄了气”	“剧痛下 / 泄了气”
Word Segmentation	German (Germany)	kulturform	kult+urform	kultur+form
Word Segmentation	Korean (Korea)	해외감염병	h̬ɛwɛg̥mjʌmbjʌŋ	h̬ɛwɛg̥mjʌmpjʌŋ
Morphology - case ambiguity	Russian (Russia)	Количество ударов по воротам (15 против 7) также говорит о преимуществе чемпионов мира	Семь	Семи
Abbreviation Expansion	English (United States)	Joined TX Army National Guard in 1979.	T.X.	Texas
Text Normalization	English (United States)	The Downtown Cabaret Theatre’s Main Stage Theatre division concludes its 2010/11 season with the Tony Award winning musical, in the heights by Lin-Manuel Miranda.	November 2010	2010 to 2011
Polyphone disambiguation	Chinese (Mandarin, simplified)	卓文君听琴后，理解了琴曲的含意，不由脸红耳热，心驰神往。	qu1	qu3
Polyphone disambiguation	English (United States)	I received a copy early in November, and read and contemplated it's provisions with great satisfaction.
Polyphone disambiguation	Japanese (Japan)	パッケージには、富士屋ホテルが発刊した「We Japanese」内の説明用の挿絵を採用。	うち (w u - ch i)	ない (n a - y i)

Hear how the Cortana voice pronounces each word accurately with UniTA.

Get started

With these updates, we’re excited to continue to power accurate, natural and intuitive voice experiences for customers world-wide. Azure Text-to-Speech service provides more than 200 voices in over 50 languages for developers all over the world.

Let us know how you are using or plan to use Neural TTS voices in this form. If you prefer, you can also contact us at mstts [at] microsoft.com. We look forward to hearing your experience and developing more compelling services together with you for the developers around the world.

For more information:

Try the demo
See our documentation
Check out our sample code

Get skilled on AI and ML – on your terms with Azure AI

Anand_Raman — Wed, 27 Jan 2021 21:37:16 GMT

Azure’s AI portfolio has options for every developer and data scientist, and we’re committed to empowering you to develop applications and machine learning models on your terms. Azure enables you to develop in your preferred language, environment, and machine learning framework, and allows you to deploy anywhere - to the cloud, on-premises, or the edge. We help improve your productivity regardless of your skill level, with code-first and low code/no code options which can help you accelerate the development process. We’re also devoted to empowering you with resources to help you get started with Azure AI and machine learning, grow your skills, and start building impactful solutions.

Announcing new AI & ML resource pages for developers and data scientists

Today we’re excited to announce new resources pages on Azure.com, with a rich set of content for data scientists and developers. Whether you’re new to AI and ML, or new to Azure, the videos, tutorials, and other content on these pages will help you get started.

Learn how your peers around the world are using Azure AI to develop AI and machine learning solutions on their terms to solve business challenges.
Grow your skills with curated learning journeys to help your skill up on Azure AI and Machine Learning in 30 days. Each learning journey has videos, tutorials, and hands-on exercises to help prepare you to pass a Microsoft certification in just 4 weeks. Upon completing the learning journey, you’ll be eligible to receive 50% off a Microsoft Certification exam.
Engage with our engineering teams and stay up to date with the latest innovations on our AI Tech Community, where you’ll find blogs, discussion forums, and more.

Pictured above: ML learning journey for developers and data scientists.

Register for the Azure AI Hackathon

Finally, put your skills to the test by entering the Azure AI Hackathon, which starts today and will run through March 22^nd, 2021. Winners will be announced in early April. The most innovative and impactful projects will win prizes up to $10,000 USD. We look forward to seeing what you build with Azure AI.

Get started today

Check out the pages to get started with your 30-day learning journey, and register for the hackathon:

Re: How to build a personal finance app using Azure

pspaulding1025 — Tue, 26 Jan 2021 11:00:27 GMT

How to build a voice-enabled grocery chatbot with Azure AI

wmendoza — Tue, 26 Jan 2021 00:30:09 GMT

Chatbots have become increasingly popular in providing useful and engaging experiences for customers and employees. Azure services allow you to quickly create bots, add intelligence to them using AI, and customize them for complex scenarios.

In this blog, we’ll walk through an exercise which you can complete in under two hours, to get started using Azure AI Services. This intelligent grocery bot app can help you manage your shopping list using voice commands. We’ll provide high level guidance and sample code to get you started, and we encourage you to play around with the code and get creative with your solution!

Features of the application:

Add or delete grocery items by dictating them to Alexa.

Easily access the grocery list through an app.
Check off items using voice commands; for example, “Alexa, remove Apples from my grocery list."
Ask Alexa to read the items you have in your grocery list.
Automatically organize items by category to help save time at the store.
Use any laptop or Web Apps to access the app and sync changes across laptop and phone.

Prerequisites:

If you don't have an Azure subscription, create a free account before you begin. If you have a subscription, log in to the Azure Portal.
Amazon Alexa account
Python 3.6 or above

Key components:

Azure Bot Service to develop bot and publish to Alexa channel.
Microsoft Bot Framework Emulator to test and debug bots using Bot Framework SDK.
Alexa skills to interact with the bot using voice commands via Amazon Alexa.
Language Understanding to help users interact with the bot with natural language, by enabling the bot to understand user intent.

Solution Architecture

App Architecture Description:

The user accesses the chatbot by invoking it as an Alexa skill.
User is authenticated with Azure Active Directory.
User interacts with the chatbot powered by Azure Bot Service; for example, user requests bot to add grocery items to a list.
Azure Cognitive Services process the natural language request to understand what the user wants to do. (Note: If you wanted to give your bot its own voice, you can choose from over 200 voices and 54 languages/locales. Try the demo to hear the different natural sounding voices.)
The bot adds or removes content in the database.

Another visual of the flow of data within the solution architecture is shown below.

Implementation

High level overview of steps involved in creating the app along with some sample code snippets for illustration:

We’ll start by creating an Azure Bot Service instance, and adding speech capabilities to the bot using the Microsoft Bot Framework and the Alexa skill. Bot Framework, along with Azure Bot Service, provides the tools required to build, test, deploy, and manage the end-to-end bot development workflow. In this example, we are integrating Azure Bot Service with Alexa, which can process speech inputs for our voice-based chatbot. However, for chatbots deployed across multiple channels, and for more advanced scenarios, we recommend using Azure’s Speech service to enable voice-based scenarios. Try the demo to listen to the over 200 high quality voices available across 54 languages and locales.

The first step in the process is to login into Azure portal and follow the steps here to create an Azure Bot Service resource and a web app bot. To add voice capability to the bot, click on channels to add Alexa (see the below snapshot) and note the Alexa Service Endpoint URI.

Azure Bot Service Channels

Next, we need to log into the Alexa Developer Console and create an Amazon Alexa skill. After creating the skill, we are presented with the interaction model. Replace the JSON Editor with the below example phrases.

{ "interactionModel": { "languageModel": { "invocationName": "Get grocery list", "intents": [ { "name": "AMAZON.FallbackIntent", "samples": [] }, { "name": "AMAZON.CancelIntent", "samples": [] }, { "name": "AMAZON.HelpIntent", "samples": [] }, { "name": "AMAZON.StopIntent", "samples": [] }, { "name": "AMAZON.NavigateHomeIntent", "samples": [] }, { "name": "Get items in the grocery", "slots": [ { "name": "name", "type": "AMAZON.US_FIRST_NAME" } ], "samples": [ "Get grocery items in the list", "Do I have bread in my list", ] } ], "types": [] } } }

Next, we’ll integrate the Alexa Skill with our Azure bot. We’ll need two pieces of information to do this: the Alexa Skill ID and the Alexa Service Endpoint URI. First, get the Skill ID either from the URl in the Alexa portal, or by going to the Alexa Developer Console and clicking “view Skill ID”. The skill ID should be a value like ‘amzn1.ask.skil.A GUID’. Then, get the Alexa Service Endpoint URI from the Azure portal, by going to the channels page of our Azure Web App Bot in the Azure portal, and clicking on Alexa to copy the Alexa Service Endpoint URI. Then integrate as shown:

Amazon Developer Console: After building the Alexa Skill, click on Endpoint and paste the Alexa Service Endpoint URI that we copied from the Azure portal and save the Endpoints.
Azure Portal: Go to the channels page of the Azure Bot, click on Alexa, and paste the Alexa Skill ID that we copied from the Alexa Developer Console.

Now, we’ll download and the bot locally for testing using the Bot Framework Emulator. Click on “Build” in the Azure Web Bot app to download the source code locally with Bot Framework Emulator. Modify app.py as below:
# Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. from http import HTTPStatus from aiohttp import web from aiohttp.web import Request, Response, json_response from botbuilder.core import ( BotFrameworkAdapterSettings, ConversationState, MemoryStorage, UserState, ) from botbuilder.core.integration import aiohttp_error_middleware from botbuilder.schema import Activity from config import DefaultConfig from dialogs import MainDialog, groceryDialog from bots import DialogAndWelcomeBot from adapter_with_error_handler import AdapterWithErrorHandler CONFIG = DefaultConfig() # Create adapter. # See https://aka.ms/about-bot-adapter to learn more about how bots work. SETTINGS = BotFrameworkAdapterSettings(CONFIG.APP_ID, CONFIG.APP_PASSWORD) # Create MemoryStorage, UserState and ConversationState MEMORY = MemoryStorage() USER_STATE = UserState(MEMORY) CONVERSATION_STATE = ConversationState(MEMORY) # Create adapter. # See https://aka.ms/about-bot-adapter to learn more about how bots work. ADAPTER = AdapterWithErrorHandler(SETTINGS, CONVERSATION_STATE) # Create dialogs and Bot RECOGNIZER = IntelligentGrocery(CONFIG) grocery_DIALOG = groceryDialog() DIALOG = MainDialog(RECOGNIZER, grocery_DIALOG) BOT = DialogAndWelcomeBot(CONVERSATION_STATE, USER_STATE, DIALOG) # Listen for incoming requests on /api/messages. async def messages(req: Request) -> Response: # Main bot message handler. if "application/json" in req.headers["Content-Type"]: body = await req.json() else: return Response(status=HTTPStatus.UNSUPPORTED_MEDIA_TYPE) activity = Activity().deserialize(body) auth_header = req.headers["Authorization"] if "Authorization" in req.headers else "" response = await ADAPTER.process_activity(activity, auth_header, BOT.on_turn) if response: return json_response(data=response.body, status=response.status) return Response(status=HTTPStatus.OK) APP = web.Application(middlewares=[aiohttp_error_middleware]) APP.router.add_post("/api/messages", messages) if __name__ == "__main__": try: web.run_app(APP, host="localhost", port=CONFIG.PORT) except Exception as error: raise error
Next, we’ll run and test the bot with Bot Framework Emulator. From the terminal, navigate to the code folder and run pip install -r requirements.txt to install the required packages to run the bot. Once the packages are installed, run python app.py to start the bot. The bot is ready to test as shown below:
}

Open the bot and add the below port number into the following URL.
Bot Framework Emulator view

Now we’re ready to add natural language understanding so the bot can understand user intent. Here, we’ll use Azure’s Language Understanding Cognitive Service (LUIS), to map user input to an “intent” and extract “entities” from the sentence. In the below illustration, the sentence “add milk and eggs to the list” is sent as a text string to the LUIS endpoint. LUIS returns the JSON seen on the right.
Language Understanding utterances diagram

Use the below template to create a LUIS JSON model file where we specify intents and entities manually. After the “IntelligentGrocery” app is created in the LUIS portal under “Import New App”, upload the JSON file with the below intents and entities.

{ "text": "access the groceries list", "intent": "Show", "entities": [ { "entity": "ListType", "startPos": 11, "endPos": 19, "children": [] } ] }, { "text": "add bread to the grocery list", "intent": "Add", "entities": [ { "entity": "ListType", "startPos": 23, "endPos": 29, "children": []

The above sample intents are for adding items and accessing the items in the grocery list. Now, it’s your turn to add additional intents to perform the below tasks, using the LUIS portal. Learn more about how to create the intents here.

Intents

Name	Description
CheckOff	Mark the grocery items as purchased.
Confirm	Confirm the previous action.
Delete	Delete items from the grocery list.

Once the intents and entities are added, we will need to train and publish the model so the LUIS app can recognize utterances pertaining to these grocery list actions.

Language Understanding (LUIS) Portal

After the model has been published in the LUIS portal, click ‘Access your endpoint Urls’ and copy the primary key, example query and endpoint URL for the prediction resource.
Language Understanding endpoint

Language Understanding (LUIS) Prediction view

Navigate to the Settings page in the LUIS portal to retrieve the App ID.
Application settings

Finally, test your Language Understanding model. The endpoint URL will be in the below format, with your own custom subdomain, and app ID and endpoint key replacing APP-ID, and KEY_ID. Go to the end of the URL and enter an intent; for example, “get me all the items from the grocery list”. The JSON result will identify the top scoring intent and prediction with a confidence score. This is a good test to see if LUIS can learn what should be predicted with the intent.

https://YOUR-CUSTOM-SUBDOMAIN.api.cognitive.microsoft.com/luis/prediction/v3.0/apps/APP-ID/slots/production/predict?subscription-key=KEY-ID&verbose=true&show-all-intents=true&log=true&query=YOUR_QUERY_HERE

Additional Ideas

We’ve now seen how to build a voice bot leveraging Azure services to automate a common task. We hope it gives you a good starting point towards building bots for other scenarios as well. Try out some of the ideas below to continue building upon your bot and exploring additional Azure AI services.

Add Google Home assistant as an additional channel to receive voice commands.
Add a PictureBot extension to your bot and add pictures of your grocery items. You will need to create intents that trigger actions that the bot can take, and create entities that require these actions. For example, an intent for the PictureBot may be “SearchPics”. This could trigger Azure Cognitive Search to look for photos, using a “facet” entity to know what to search for. See what other functionality you can come up with!
Use Azure QnA maker to enable your bot to answer FAQs from a knowledge base. Add a bit of personality using the chit-chat feature.
Integrate Azure Personalizer with your voice chatbot to enables the bot to recommend a list of products to the user, providing a personalized experience.
Include Azure Speech service to give your bot a custom, high quality voice, with 200+ Text to Speech options across 54 different locales/languages, as well as customizable Speech to Text capabilities to process voice inputs.
Try building this bot using Bot Framework Composer, a visual authoring canvas.

How to build an intelligent travel journal using Azure AI

maddybutzbach — Tue, 26 Jan 2021 00:26:41 GMT

AI capabilities can enhance many types of applications, enabling you to improve your customer experience and solve complex problems. With Azure Cognitive Services, you can easily access and customize industry-leading AI models, using the tools and languages of your choice.

In this blog, we’ll walk through an exercise which you can complete in under an hour, to get started using Azure AI Services. Many of us are dreaming of traveling again, and building this intelligent travel journal app can help you capture memories from your next trip, whenever that may be. We’ll provide high level guidance and sample code to get you started, and we encourage you to play around with the code and get creative with your solution!

Features of the application:

Capture voice memos, voice tag photos, and transcribe speech to text.
Automatically tag your photos based on key phrase extraction and analysis of text in pictures.
Translate tags and text into desired language.
Organize your memos by key phrase and find similar travel experiences you enjoyed with AI-powered search.

Prerequisites:

If you don't have an Azure subscription, create a free account before you begin. If you have a subscription, log in to the Azure Portal.
To run the provided sample code, you will need Visual Studio 2019 and .NET Core 3.1 or above (for FotoFly)
Refer to this tutorial for detailed guidance on how to publish a console app.

Key Azure technologies:

Speech Service batch transcription for speech to text transcription
Text Analytics for key phrase/intent extraction
Computer Vision for analyzing text in images
Translator to normalize tags/text into desired language.
Open Source FotoFly library for photo tagging. Alternatively, you can use blob metadata but functionality will be limited.
Azure Cognitive Search for AI-powered search.

NOTE: For more information refer to the “References.txt” file under respective folders within JournalHelper library project in the provided sample solution with this blog.

Solution Architecture

App Architecture Description:

User records a voice memo; for example, to accompany an image they’ve captured. The recorded file is stored in a file repository (alternatively, you could use a database).
The recorded voice memo (e.g. .m4a) is converted into desired format (e.g. .wav), using Azure’s Speech Service batch transcription capability.
The folder containing voice memos is uploaded to a Blob container.
Images are uploaded into a separate container for analysis of any text within the photos, using Azure Computer Vision.
Use Translator to translate text to different languages, as needed. This may be useful to translate foreign street signs, menus, or other text in images.
Extract tags from the generated text files using Text Analytics, and send tags back to the corresponding image file. Tags can be travel related (#milan, #sunset, #Glacier National Park), or based on geotagging metadata, photo metadata (camera make, exposure, ISO), and more.
Create a search indexer with Azure Cognitive Search, and use the generated index to search your intelligent travel journal.

Implementation

Sample code

The entire solution code is available for download at this link. Download/clone and follow instructions in ReadMe.md solution item for further setup.

Implementation summary

The sample is implemented using various client libraries and samples available for Azure Cognitive Services. All these services are grouped together into a helper library project named “journalhelper”. In the library we introduce a helper class to help with scenarios that combine various Cognitive Services to achieve desired functionality.

We use “.Net Core console app” as the front end to test the scenarios. This sample also uses another open source library (FotoFly), which is ported to .Net Core here, to access and edit image metadata.

High level overview of steps, along with sample code snippets for illustration:

Start by batch transcribing voice memos and extracting key tags from the text output. Group the input voice memos into a folder, upload them into an Azure Blob container or specify a list of their URls, and use batch transcription to get results back into the Azure Blob container, as well as a folder in your file system. The following code snippet illustrates how helper functions can be grouped together for a specific functionality. It combines local file system, Azure storage containers, and Cognitive Services speech batch transcription API.

Console.WriteLine("Uploading voice memos folder to blob container..."); Helper.UploadFolderToContainer( HelperFunctions.GetSampleDataFullPath(customSettings.SampleDataFolders.VoiceMemosFolder), customSettings.AzureBlobContainers.InputVoiceMemoFiles, deleteExistingContainer); Console.WriteLine("Branch Transcribing voice memos using containers..."); //NOTE: Turn the pricing tier for Speech Service to standard for this below to work. await Helper.BatchTranscribeVoiceMemosAsync( customSettings.AzureBlobContainers.InputVoiceMemoFiles, customSettings.AzureBlobContainers.BatchTranscribedJsonResults, customSettings.SpeechConfigSettings.Key, customSettings.SpeechConfigSettings.Region); Console.WriteLine("Extract transcribed text files into another container and folder, delete the intermediate container with json files..."); await Helper.ExtractTranscribedTextfromJsonAsync( customSettings.AzureBlobContainers.BatchTranscribedJsonResults, customSettings.AzureBlobContainers.InputVoiceMemoFiles, customSettings.AzureBlobContainers.ExtractedTranscribedTexts, HelperFunctions.GetSampleDataFullPath(customSettings.SampleDataFolders.BatchTranscribedFolder), true);

Next, create tags from the transcribed text. Sample helper function using the Text Analytics client library is listed below.

//text analytics public static void CreateTagsForFolderItems(string key, string endpoint, string batchTranscribedFolder, string extractedTagsFolder) { if (!Directory.Exists(batchTranscribedFolder)) { Console.WriteLine("Input folder for transcribed files does not exist"); return; } // ensure destination folder path exists Directory.CreateDirectory(extractedTagsFolder); TextAnalyticsClient textClient = TextAnalytics.GetClient(key, endpoint); var contentFiles = Directory.EnumerateFiles(batchTranscribedFolder); foreach(var contentFile in contentFiles { var tags = TextAnalytics.GetTags(textClient, contentFile).ConfigureAwait(false).GetAwaiter().GetResult(); // generate output file with tags string outFileName = Path.GetFileNameWithoutExtension(contentFile); outFileName += @"_tags.txt"; string outFilePath = Path.Combine(extractedTagsFolder, outFileName); File.WriteAllLinesAsync(outFilePath, tags).Wait() ; } }

The actual client library or service calls are made as shown:

static public async Task<IEnumerable<string>> GetTags(TextAnalyticsClient client, string inputTextFilePath) { string inputContent = await File.ReadAllTextAsync(inputTextFilePath); var entities = EntityRecognition(client, inputContent); var phrases = KeyPhraseExtraction(client, inputContent); var tags = new List<string>(); tags.AddRange(entities); tags.AddRange(phrases); return tags; }

Update tags to the photo/image file, using the open source FotoFly library. Alternatively, you can update the Blob metadata with these tags and include that in the search index, but the functionality will be limited to using Azure Blob storage.

string taggedPhotoFile = photoFile.Replace(inputPhotosFolder, OutPhotosFolder); File.Copy(photoFile, taggedPhotoFile, true); if (tags.Count > 0) { ImageProperties.SetPhotoTags(taggedPhotoFile, tags); }

Other useful functions to complete the scenario are:
1. Helper.ProcessImageAsync, and
2. Helper.TranslateFileContent

The first one can be used to extract text from images using OCR or regular text processing using Computer Vision. The second can detect the source language, translate using Azure’s Translator service into the desired output language, and then create more tags for an image file.

Finally, use Azure Cognitive Search to create an index from the extracted text files saved in the Blob container, enabling you to search for documents and create journal text files. For example, you can search for images by cities or countries visited, date, or even cuisines. You can also search for images by camera-related metadata or geolocation.

In this sample we have demonstrated simple built-in skillsets for entity and language detection. The solution can be further enhanced by adding additional data sources to process tagged images and their metadata, and adding additional information to the searches.

NOTE: The helper functions can be made more generic to take additional skillset input.

public static async Task CreateSearchIndexerAsync( string serviceAdminKey, string searchSvcUrl, string cognitiveServiceKey, string indexName, string jsonFieldsFilePath, string blobConnectionString, string blobContainerName ) { // Its a temporary arrangment. This function is not complete IEnumerable<SearchField> fields = SearchHelper.LoadFieldsFromJSonFile(jsonFieldsFilePath); // create index var searchIndex = await Search.Search.CreateSearchIndexAsync(serviceAdminKey, searchSvcUrl, indexName, fields.ToList()); // get indexer client var indexerClient = Search.Search.GetSearchIndexerClient(serviceAdminKey, searchSvcUrl); // create azure blob data source var dataSource = await Search.Search.CreateOrUpdateAzureBlobDataSourceAsync(indexerClient, blobConnectionString, indexName, blobContainerName); // create indexer // create skill set with minimal skills List<SearchIndexerSkill> skills = new List<SearchIndexerSkill>(); skills.Add(Skills.CreateEntityRecognitionSkill()); skills.Add(Skills.CreateLanguageDetectionSkill()); var skillSet = await Search.Search.CreateOrUpdateSkillSetAsync(indexerClient, indexName + "-skillset", skills, cognitiveServiceKey); var indexer = await Search.Search.CreateIndexerAsync(indexerClient, dataSource, skillSet, searchIndex); // wait for some time to have indexer run and load documents Thread.Sleep(TimeSpan.FromSeconds(20)); await Search.Search.CheckIndexerOverallStatusAsync(indexerClient, indexer); }

Finally, search documents and generate the corresponding journal files, utilizing the following functions:

Helper.SearchDocuments
Helper.CreateTravelJournal

Additional Ideas

In addition to the functionality described so far, there are many other ways you can leverage Azure AI to further enhance your intelligent travel journal and learn more advanced scenarios. We encourage you to explore some the following ideas to enrich your app:

Add real time voice transcription and store transcriptions in an Azure managed database, to correlate voice transcription with images in context.
Include travel tickets and receipts as images for OCR-based image analysis (Form Recognizer) and include them as journal artifacts.
Use multiple data sources for a given search index. We have simplified and only included text files to index in this sample, but you can include the tagged photos from a different data source for the same search index.
Add custom skills and data extraction for search indexer. Extract metadata from images and include as search content.
Extract metadata from video and audio content using Video Indexer.
Experiment with Language Understanding and generate more elaborate and relevant search content based on top scoring intents and entities. Sample keywords and questions related to current sample data are included in Objectives.docx solution item.
Build a consumer front-end app that stitches all of this together and displays the journal in a UI.

How to build a personal finance app using Azure

mernanashed — Tue, 26 Jan 2021 01:19:08 GMT

AI allows you to deliver breakthrough experiences in your apps. With Azure Cognitive Services, you can easily customize and deploy the same AI models that power Microsoft’s products, such as Xbox and Bing, using the tools and languages of your choice.

In this blog we will walk through an exercise that you can complete in under an hour and learn how to build an application that can be useful for you, all while exploring a set of Azure services. If you have ever wanted to get your financial transactions in order, look no further. With this exercise, we’ll explore how to quickly take a snap of a receipt from your phone and upload it for categorization, creating expense reports, and to gain insights to your spending. Remember, even though we’ll walk you through each step, you can always explore the sample code and get creative with your own unique solution!

Features of the application:

Snap a picture of your receipt and upload it using your smartphone
Extract relevant data from the images: Who issued the receipt? What was the total amount? What was purchased? All of this information can be effortlessly stored for exploration
Query the data: bring your receipts to life by extracting relevant and insightful information

Prerequisites

If you don't have an Azure subscription, create a free account before you begin. If you have a subscription, log in to the Azure Portal.
You will need to have python installed locally to run some of the samples.

Key Azure technologies:

Azure Form Recognizer scans image documents with optical character recognition and extracts text, key/value pairs, and tables from documents, receipts, and forms.
Form Recognizer’s prebuilt receipt model specifically extracts receipt data
Azure Blob Storage is used to store data
Azure Cognitive Search enriches the data by making it easily identifiable

Solution Architecture

App Architecture Description:

User uploads a receipt image from their mobile device
The uploaded image is verified and then sent to the Azure Form Recognizer to extract information
The image is analysed by the REST API within the Form Recognizer prebuilt receipt model
A JSON is returned that has both the text information and bounding box coordinates of the extracted receipt data
The resulting JSON is parsed and a simpler JSON is formed, saving only the relevant information needed
This receipt JSON is then stored in Azure Blob Storage
Azure Cognitive Search points directly to Azure Blob Storage and is used to index the data
The application queries this search index to extract relevant information from the receipts

Another visual of the flow of data within the solution architecture is shown below.

Now that we’ve explored the technology and services we’ll be using, let’s dive into building our app!

Implementation

To get started, data from receipts must be extracted; this is done by setting up the Form Recognizer service in Azure and connecting to the service to use the relevant API for receipts. A JSON is returned that contains the information extracted from receipts and is stored in Azure Blob Storage to be used by Azure Cognitive Search. Cognitive Search is then utilized to index the receipt data, and to search for relevant information.

High level overview of steps, along with sample code snippets for illustration:

Go to the Azure portal and create a new Form Recognizer resource. In the Create pane, provide the following information:

Name	A descriptive name for your resource.
Subscription	Select the Azure subscription which has been granted access.
Location	The location of your cognitive service instance. Different locations may introduce latency, but have no impact on the runtime availability of your resource.
Pricing Tier	The cost of your resource depends on the pricing tier you choose and your usage. For more information, see the API pricing details.
Resource Group	The Azure resource group that will contain your resource. You can create a new group or add it to a pre-existing group.

After Form Recognizer deploys, go to All Resources and locate the newly deployed resource. Save the key and endpoint from the resource’s key and endpoint page somewhere so you can access it later.
You can use the following Analyze Receipt API to start analyzing the receipt. Remember to replace <endpoint> & <subscription key> the values you saved earlier and replace <path to your receipt> with the local path to your scanned receipt image.
# Analyse script import json import time from requests import get, post # Endpoint URL endpoint = r"<endpoint url>" apim_key = "<subscription key>" post_url = endpoint + "/formrecognizer/v2.0/prebuilt/receipt/analyze" source = r"<path to your receipt>" headers = { # Request headers 'Content-Type': 'image/jpeg', 'Ocp-Apim-Subscription-Key': apim_key, } params = { "includeTextDetails": True } with open(source, "rb") as f: data_bytes = f.read() try: resp = post(url=post_url, data=data_bytes, headers=headers, params=params) if resp.status_code != 202: print("POST analyze failed:\n%s" % resp.text) quit() print("POST analyze succeeded:\n%s" % resp.headers) get_url = resp.headers["operation-location"] except Exception as e: print("POST analyze failed:\n%s" % str(e)) quit()
If you run this code and everything is as it should be, you'll receive a 202 (Success) response that includes an Operation-Location header, which the script will print to the console. This header contains an operation id that you can use to query the status of the asynchronous operation and get the results. In the following example value, the string after operations/ is the operation ID.

https://cognitiveservice/formrecognizer/v2.0/prebuilt/receipt/operations/54f0b076-4e38-43e5-81bd-b85b8835fdfb

Now you can call the Get Analyze Receipt Result API to get the Extracted Data.
# Get results. n_tries = 10 n_try = 0 wait_sec = 6 while n_try < n_tries: try: resp = get(url = get_url, headers = {"Ocp-Apim-Subscription-Key": apim_key}) resp_json = json.loads(resp.text) if resp.status_code != 200: print("GET Receipt results failed:\n%s" % resp_json) quit() status = resp_json["status"] if status == "succeeded": print("Receipt Analysis succeeded:\n%s" % resp_json) quit() if status == "failed": print("Analysis failed:\n%s" % resp_json) quit() # Analysis still running. Wait and retry. time.sleep(wait_sec) n_try += 1 except Exception as e: msg = "GET analyze results failed:\n%s" % str(e) print(msg) quit()

This code uses the operation id and makes another API call.

The JSON that is returned can be examined to get the required information - ‘readResults’ field will contain all lines of text that was decipherable, and the ‘documentResults’ field contains ‘key/value’ information for the most relevant parts of the receipt (e.g. the merchant, total, line items etc.)

The receipt image below,

resulted in the JSON from which we have extracted the following details:
MerchantName: THE MAD HUNTER TransactionDate: 2020-08-23 TransactionTime: 22:07:00 Total: £107.10

We will now create a JSON from all the data extracted from the analysed receipt. The structure of the JSON is shown below:
{ "id":"INV001", "user":"Sujith Kumar", "createdDateTime":"2020-10-23T17:16:32Z", "MerchantName":"THE MAD HUNTER", "TransactionDate":"2020-10-23", "TransactionTime":"22:07:00", "currency":"GBP", "Category":"Entertainment", "Total":"107.10", "Items":[ ] }

We can now save this JSON and build a search service to extract the information we want from it.

Before continuing onto step 8, you must have an Azure Storage Account with Blob storage.

We will now save the JSON files in an Azure Blob Storage container and use it as a source for the Azure Cognitive Search Service Index that we will create.
Sign-in to the Azure Portal and search for "Azure Cognitive Search" or navigate to the resource through Web > Azure Cognitive Search. Follow the steps to:

Choose a subscription
Set a resource group
Name the service appropriately
Choose a location
Choose a pricing tier for this service
Create your service
Get a key and URL endpoint

We will use the free Azure service, which means you can create three indexes, three data sources and three indexers. The dashboard will show you how many of each you have left. For this exercise you will create one of each.

In the portal, find the search service you created above and click Import data on the command bar to start the wizard. In the wizard, click on Connect to your data and specify the name, type, and connection information. Skip the ‘Enrich Content’ page and go to Customize Target Index.
For this exercise, we will use the wizard to generate a basic index for our receipt data. Minimally, an index requires a name and a fields collection; one of the fields should be marked as the document key to uniquely identify each document.

Fields have data types and attributes. The check boxes across the top are index attributes controlling how the field is used.

Retrievable means that it shows up in search results list. You can mark individual fields as off limits for search results by clearing this checkbox.
Key is the unique document identifier. It's always a string, and it is required.
Filterable, Sortable, and Facetable determine whether fields are used in a filter, sort, or faceted navigation structure.
Searchable means that a field is included in full text search. Only Strings are searchable.

Make sure you choose the following fields:

id
user
createdDateTime
MerchantName
TransactionDate
TransactionTime
Currency
Category
Total

Still in the Import data wizard, click Indexer > Name, and type a name for the indexer.

This object defines an executable process. For now, use the default option (Once) to run the indexer once, immediately.

Click Submit to create and simultaneously run the indexer.

Soon you should see the newly created indexer in the list, with status indicating "in progress" or success, along with the number of documents indexed.

The main service page provides links to the resources created in your Azure Cognitive Search service. To view the index you just created, click Indexes from the list of links.

Click on the index (azureblob-indexer in this case) from the list of links and view the index-schema.

Now you should have a search index that you can use to query the receipt data that’s been extracted from the uploaded receipts.

Click the search explorer

From the index drop down choose the relevant index. Choose the default API Version (2020-06-30) for this exercise.

In the search bar paste a query string (for eg. category='Entertainment')

You will get results as verbose JSON documents as shown below:

Now that you have built a query indexer and aimed it at your data you can now use it to build queries programmatically and extract information to answer some of the following questions:

How much did I spend last Thursday?
How much have I spent on entertainment over the last quarter?
Did I spend anything at ‘The Crown and Pepper’ last month?

Additional Ideas

In addition to the services and functionalities used throughout this exercise, there are numerous other ways you can use Azure AI to build in support for all kinds of receipts or invoices. For example, the logo extractor can be used to identify logos of popular restaurants or hotel chains, and the business card model can ingest business contact information just as easily as we saw with receipts.

We encourage you to explore some of the following ideas to enrich your application:

Search invoices for specific line items
Train the models to recognize different expense categories such as entertainment, supplies, etc.
Add Language Understanding (LUIS) to ask your app questions in natural language and extract formatted reports
Add Azure QnA Maker to your app and get insights such as how much you spent on entertainment last month, or other categories of insights you’d like to explore

Re: Introducing QnA Maker managed: now in public preview

aowens-jmt — Mon, 25 Jan 2021 18:50:41 GMT

I've sent a couple of emails to the team and have yet to hear. Wanted to make sure I was sending it to the proper email address. Thank you.

Re: QnA with Azure Cognitive Search

pchoudhari — Sun, 24 Jan 2021 09:00:36 GMT

, this is brilliant feedback. We will put this in our backlog.

Re: QnA with Azure Cognitive Search

CharlieDigital — Fri, 22 Jan 2021 15:53:34 GMT

There are a few gaps with Q&A at the moment which I hope the product team addresses.

The first is the inability to associate metadata with the content as it's being ingested from a file or URL source. For example, if you upload a PDF, it would be useful to be able to provide key-value pairs which are automatically set as metadata on the resultant index from that source. At current, the process requires re-processing each item in the index and evaluating the source and then assigning metadata to the item. This functionality already exists on the API as long as you are starting from a QnADTO object rather than a file. A simple example is that you may want to associate a version identifier to a document that is ingested so that each Q&A entry has a version metadata tag associated.

The second is to "downgrade" a response. Currently, you can kind of "promote" a response by adding an alternate phrasing to respond to a broader set of vocabularies, but there's no way to make a Q&A item less relevant.

It's a great service, but still has room for growth.

Re: Introducing QnA Maker managed: now in public preview

HesselW — Tue, 19 Jan 2021 09:17:55 GMT

Thanks. This would definitely help. I will give it a go.

Re: Introducing QnA Maker managed: now in public preview

nerajput — Mon, 18 Jan 2021 17:26:27 GMT

and Its great to hear that you are liking the new version of QnA Maker. We use two indexes per KB, only when you make the language setting KB specific instead of service specific, which is required to create your testing experience and relevance scores same as what you will see once published. In case you are creating KBs belonging to only one language in a service, then please don't use this setting when you are creating your first KB as this setting is allowed to be set only at the time of first KB creation once set you cannot update this setting. Please check: Language support - QnA Maker - Azure Cognitive Services | Microsoft Docs

Re: Introducing QnA Maker managed: now in public preview

nerajput — Mon, 18 Jan 2021 17:20:45 GMT

Please feel free to drop us a mail at qnamakersupport@microsoft.com

Re: Ignite 2020 Neural TTS updates: new language support, more voices and flexible deployment option

Qinying Liao — Wed, 27 Jan 2021 14:29:36 GMT

Hi Thank you for reporting the issue. We are investigating the cause and will fix if it's a bug.

If you have a support plan and you need technical help, you can create a support request.

For Issue type, select “Technical”.
For Subscription, select your subscription.
For Service, click My services, then select “Cognitive Services”.
For Summary, type a description of your issue.
For Problem type, select “Text to Speech”.
For problem subtype, select “accuracy of speech output”.

Re: Ignite 2020 Neural TTS updates: new language support, more voices and flexible deployment option

serhat1141 — Fri, 15 Jan 2021 17:04:24 GMT

Hi Sir, I am really enjoying the audio content creation so far, but I got to ask a question as there seems to be a problem within the audio content creation page. I am especially using neural voices, but for the last few days, I am adjusting the RATE first, then INTONATION to make the pronounciation better like real speech, but as I am ADJUSTING THE INTONATION, THE RATE JUST GOES TO THE BASE TO 1.00. And also, when adjusting the INTONATION, IF I HAVE A RATE SET BEFORE, IT DOES NOT PREVIEW THE ADJUSTED INTONATION. I have been having this problem for some 3-5 days, and it seems like some kind of n annoying problem as I can't create the voices I have been for the last few weeks. I would appreciate if you can correct this problem. Thank you so much.

Enhanced Table Extraction from documents with Form Recognizer

NetaH — Thu, 14 Jan 2021 19:03:46 GMT

Authors: Lei Sun, Neta Haiby, Cha Zhang, Sanjeev Jagtap

Documents containing tables pose a major hurdle for information extraction. Tables are often found in financial documents, legal documents, insurance documents, oil and gas documents and more. Tables in documents are often the most important part of the document but extracting data from tables in documents presents a unique set of challenges. Challenges include an accurate detection of the tabular region within an image, and subsequently detecting and extracting information from the rows and columns of the detected table, merged cells, complex tables, nested tables and more. Table extraction is the task of detecting the tables within the document and extracting them into a structured output that can be consumed by workflow applications such as robotic process automation (RPA) services, data analyst tools such as excel, databases and search services.

Customers often use manual processes for data extraction and digitization. However, with the new enhanced table extraction feature you can send a document (PDF or images) to Form Recognizer for extraction of all the information into a structured usable data at a fraction of the time and cost, so you can focus more time acting on the information rather than compiling it.

Table extraction challenges

Table extraction from a wide variety of document images is a challenging problem due to the heterogeneous table structures, diverse table contents, and erratic use of ruling lines. To name a few concrete examples, in financial reports and technical publications, some borderless tables may have complex hierarchical header structures, contain many multi-line, empty or spanned cells, or have large blank spaces between neighboring columns. In forms, some tables may be embedded in other more complex tabular objects (e.g., nested tables) and some neighboring tables may be very close to each other which makes it hard to determine whether they should be merged or not. In invoices, tables may have different sizes, e.g., some key-value pairs composed tables may contain only two rows/columns and some line-item tables may span multiple pages. Sometimes, some objects in document images like figures, graphics, code listings, structurally laid out text, or flow charts may have similar textures as tables, which poses another significant challenge for successful detection of tables and reduction of false alarms. To make matters worse, many scanned or camera-captured document images are of poor image quality, and tables contained in them may be distorted (even curved) or contain artifacts or noises. Existing table extraction solutions fall short of extracting tables from such document images with high accuracy, which has prevented workflow applications from effectively leveraging this technology.

Form Recognizer Table extraction

In recent years, the success of deep learning in various computer vision applications has motivated researchers to explore deep neural networks like convolutional neural networks (CNN) or graph neural networks (GNN) for detecting tables and recognizing table structures from document images. With these new technologies, the capability and performance of modern table extraction solutions have been improved significantly.

In the latest release of Form Recognizer, we created a state-of-the-art table extraction solution with cutting-edge deep learning technology. After validating that Faster/Mask R-CNN based table detectors are effective in detecting a variety of tables (e.g., bordered or borderless tables, tables embedded in other more complex tabular objects, and distorted tables) in document images robustly, we further proposed a new method to improve the localization accuracy of such detectors, and achieved state-of-the-art results on the ICDAR-2019 cTDaR table detection benchmark dataset by only using a lightweight ResNet18 backbone network (Table 1).

For the challenge of table recognition or table cell extraction, we leveraged existing CNN/GNN based approaches, which have proven to be robust to complex tables like borderless tables with complex hierarchical header structures and multi-line/empty/spanned cells. We further enhanced them to deal with distorted or even slightly curved tables in camera-captured document images, making the algorithm more widely applicable to different real-world scenarios. Figure 1 below shows a few examples to demonstrate such capabilities.

Easy and Simple to use

Try it out with the Form Recognizer Sample Tool.

Extracting tables from documents is as simple as 2 API calls, no training, preprocessing, or anything else needed. Just call the Analyze Layout operation with your document (image, TIFF, or PDF file) as the input and extracts the text, tables, selection marks, and structure of the document.

Step 1: The Analyze Layout Operation –

https://{endpoint}/formrecognizer/v2.1-preview.2/layout/analyze

The Analyze Layout call returns a response header field called Operation-Location. The Operation-Location value is a URL that contains the Result ID to be used in the next step.

Operation location -
https://cognitiveservice/formrecognizer/v2.1-preview.2/prebuilt/layout/analyzeResults/44a436324-fc4b-4387-aa06-090cfbf0064f

Step 2: The Get Analyze Layout Result Operation –

Once you have the operation location call the Get Analyze Layout Result operation. This operation takes as input the Result ID that was created by the Analyze Layout operation.

https://{endpoint}/formrecognizer/v2.1-preview.2/layout/analyzeResults/{resultId}

The output of the Get Analyze Layout Results will provide a JSON output with the extracted table – rows, columns, row span, col span, bounding box and more.

For example:

Get started

To get started create a Form Recognizer resource in the Azure Portal and try out your tables in the Form Recognizer Sample Tool. You can also use the Form Recognizer client library or REST API.
Note tables output is included in all parts of the Form Recognizer service – prebuilt, layout and custom in the JSON output pageResults section.
For additional questions please reach out to us at formrecog_contact@microsoft.com

Re: Introducing QnA Maker managed: now in public preview

manishagole96 — Wed, 30 Dec 2020 15:02:02 GMT

how to remove short answer after publishing qna maker KB?

Re: Introducing QnA Maker managed: now in public preview

amitc2021 — Tue, 29 Dec 2020 07:24:07 GMT

I agree with this is really problematic to waste one index for test per KB. If I were to create 50 actual KB's I would have to take a higher tier Search just for Text Indexes. Please think about it. I am already running QnA in 5 different languages and many more to be added next year.

Is it possible to bring this in GA sooner in Q1. It would save a huge lot of deal with managing several resources. Additionally West Europe region is missing in Preview, can it be added?

Just so you know I have adopted QnA Maker since it was in Preview in early 2018.

Re: Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

ozanyasindogan — Wed, 23 Dec 2020 10:51:19 GMT

Turkish Ahmet sounds like an evening news anchor in NTV news channel of Turkey, lol :)

Re: Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

Balaji Mishra — Sun, 20 Dec 2020 08:43:18 GMT

So great to see the huge list and choices, I clicked each one to get a feel of it!! One observation barring few dialects e.g., English(India, Ireland, Canada), Hindi, Telugu, Solvenian most of the other felt as if it's being spoken very fast as if a fight, or debate; is there a way to listen the same tad slower?

Re: Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

Abduxukur Abdurixit — Sat, 19 Dec 2020 19:17:48 GMT

Are you planning to include Uyghur language (ug-CN) soon? It is one of major languages in China and central Asian republics.

Re: Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

urido — Fri, 18 Dec 2020 14:56:40 GMT

does this mean Teams can transcribe meetings live in these languages? if not, is that expected soon?

Bot Framework Composer 1.3 is now available!

GaryPrettyMsft — Thu, 17 Dec 2020 10:42:45 GMT

This week, as the year draws to a close, we are excited to announce that Bot Framework Composer 1.3 is now available to download. Composer has come a long way since we made the product GA (generally available) at the Microsoft Build conference earlier this year and this is our biggest release yet, adding many significant capabilities and making building sophisticated bots and virtual assistants even easier!

New features to improve the developer experience and workflow

For developers who are working with Bot Framework Skills today, you will know that developing multiple bots locally that work together can sometimes be a challenge, especially when it comes to setting up debugging. In Composer 1.3, we have now added a multi-bot authoring and management experience to transform this scenario, adding the capability to create, manage and test multiple bots within a single project. With a single click, you can now start all local bots for debugging, enabling you to test your root (parent) bot, connected to one or more skills with no additional manual configuration needed.

Another significant enhancement is for the provisioning feature, which previously required developers to leave Composer and run a PowerShell script, copying back a resulting configuration into Composer. Now though, the provisioning process has been overhauled and users can now login to Azure, provision required resources and subsequently publish bots, all within the Composer environment!

Additionally, we have implemented a new settings experience, providing an improved interface, removing the need to manually edit the underlying JSON for common settings, whilst retaining the ability to make changes or add additional configuration manually if you need to.

Localization

In addition to the existing capability for developers to localize their bots, multilingual support has now been added to the Composer UI! You can now choose from a long list of available languages within the Application Settings pane to change the language displayed within Composer.

Preview features

As part of version 1.3, you can now choose to enable one or more preview features by choosing preview feature flags within the Composer settings page. These features are designed to give you early access and a chance to try what we are working on right now for future Composer releases. The following preview feature flags are now available.

Form Dialogs – Automatically generate a sophisticated dialog by simply providing the properties that you would like customers to provide as part of the conversation, with Composer then generating the appropriate dialog, language understanding (to enable dis-ambiguation and interruption scenarios) and bot responses (.lg files) assets.
Orchestrator – A new top-level recognizer which can help to arbitrate (dispatch) between multiple LUIS and QnA Maker models to ensure accurate routing of user requests to the appropriate language model or skill.
Package Manager – Developers can now discover and install packages from NuGet / NPM that contain re-usable assets, including dialogs, custom actions and .LG (language generation) files, that can be utilized by their bots. Once installed, assets contained within a package become available for use within a bot. Moving forward, we will provide guidance for how you can create and publish your own packages (including to internal feeds if desired), as well as making available a number of packages covering common scenarios that will ship with Composer.
Conversational core template – Built on the new package capabilities, surfaced via the preview of the Package Manager, we are developing a new component model for bot development using re-usable building blocks (packages). With this preview, users can create a bot using the new conversational core template which consists of a configurable runtime that can be extended with packages or importing additional skills.

Help us to improve Composer!

Within this release we have enabled the ability for users of Composer to opt in to sending usage information to us, to allow us to better understand how Composer is used. As we gather this telemetry, we can use it as an additional signal to help us prioritize our efforts in future releases and ensure we are focusing on the right features. You can help us by opting into providing usage data via the Data Collection section of the Composer settings page.

Finally, a huge thank you to all of our users for your support and feedback during 2020 - we are excited to bring more significant updates to you as we move into 2021. Happy Holidays to everyone from the entire Conversational AI team!

Re: Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

HotCakeX — Thu, 17 Dec 2020 08:21:42 GMT

Thank you!

Re: Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

GarfieldHe — Thu, 17 Dec 2020 02:26:28 GMT

thank you for your feedback! Persian is also considered in our backlog, please stay tune!

Re: Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

HotCakeX — Wed, 16 Dec 2020 18:59:22 GMT

Hi, I noticed there is no support for Persian language, it's spoken by roughly +100 millions people in the world, I see in the list languages less common are supported.

Re: Introducing QnA Maker managed: now in public preview

HesselW — Wed, 16 Dec 2020 14:47:43 GMT

Question for the team. I did migrate some bots in production to the preview. Works as a charm and takes away a lot of resources from the list.

I did notice that the preview adds a test index to each database. So for each knowledge base you add, two indexes are used used in de search service (in stead of using one testbk index in the old situation) Is this by design? And if so, what is the rationel behind it. This does have a very negative effect on operational costs.

Re: Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

GarfieldHe — Wed, 16 Dec 2020 02:06:08 GMT

thank you for your feedback! Belgium Dutch (nl-BE) is considered in future release, please stay tune!

Re: Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

backnext — Tue, 15 Dec 2020 16:34:36 GMT

The Dutch voices are typical voices for the Netherlands. In Flanders / Belgium we have the same language but very different pronunciation. A 'nl-BE' voice would be very welcome.

Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

GarfieldHe — Wed, 16 Dec 2020 06:46:34 GMT

This post was co-authored with Qinying Liao, Sheng Zhao, Gang Wang, Yueying Liu

Neural Text to Speech (Neural TTS), a powerful speech synthesis capability of Cognitive Services on Azure, enables you to convert text to lifelike speech which is close to human-parity. Since its launch, we have seen it widely adopted in a variety of scenarios by many Azure customers, from voice assistants to audio content creation. More and more customers are asking for richer and more diverse choices of synthetic voices for different use cases.

Today, we are excited to announce that Azure Neural TTS has added 51 new voices for a total of 129 neural voices across 54 languages/locales. With this release, we provide at least one male and one female voice for customers to choose in each language/locale. In total, Azure TTS now enables developers to reach millions more people with more than 200 voices available in standard and neural TTS.

What's new

Neural TTS has now been extended to support 51 new voices, which will bring to you the capability to have both male and female voices in each language for your apps. You can hear samples of the voices below, or try them with your own text in our demo.

46 new voices are generally available

In total 46 new voices are released across the 49 locales that are generally available in the Azure data centers/regions that support neural TTS (see the full list of Azure regions here).

Locale	Language	Gender	Voice	Sample audio
ar-EG	Arabic (Egypt)	Male	ShakirNeural	البركان هو أكثر ما في الطبيعــة إثارة للرهبة
ar-SA	Arabic (Saudi Arabia)	Male	HamedNeural	الناس مَعادن، تصدأ بالملل، وتتمدد بالأمل، وتنكمش بالألم
bg-BG	Bulgarian (Bulgaria)	Male	BorislavNeural	Шофьорът задължително трябва да вземе експерт за второ мнение, за да провери дали всички системи на автомобила работят нормално.
ca-ES	Catalan (Spain)	Male	EnricNeural	Les activitats docents tenen lloc al campus del Poblenou.
ca-ES	Catalan (Spain)	Female	JoanaNeural	L'artista està considerat com el pintor de les multituds.
cs-CZ	Czech (Czech)	Male	AntoninNeural	Opravdový zasvěcenec ví, že nejmocnějším tajemstvím je to, které nemá žádný obsah.
da-DK	Danish (Denmark)	Male	JeppeNeural	61 procent af de kandidatstuderende er kvinder.
de-AT	German (Austria)	Male	JonasNeural	Das ist das letzte lange Pfingstwochenende für Schülerinnen und Schüler.
de-CH	German (Switzerland)	Male	JanNeural	Eine Person, die sich bei Brandausbruch im oberen Stock aufgehalten hat, hat sich noch rechtzeitig in Sicherheit bringen können.
el-GR	Greek (Greece)	Male	NestorasNeural	Συγκλονιστικές εξελίξεις και ανατροπές στα επόμενα επεισόδια .
en-CA	English (Canada)	Male	LiamNeural	He had held the position since 2010.
en-IE	English (Ireland)	Male	ConnorNeural	Life is short, think before you talk.
en-IN	English (India)	Male	PrabhatNeural	Sometimes you can see snow on the mountains.
fi-FI	Finnish (Finland)	Male	HarriNeural	Yhtiö kertoi loppuvuoden tuloksestaan ennakkotietoja.
fi-FI	Finnish (Finland)	Female	SelmaNeural	Hevoset ovat uljaita ja nopeita eläimiä.
fr-CH	French (Switzerland)	Male	FabriceNeural	La Suisse comptera 5,6 millions (12%) de personnes actives en 2050.
he-IL	Hebrew (Israel)	Male	AvriNeural	הוא אמר שהמספרים מדאיגים בשל עצמם, אבל בכל הישיבות שלנו המסקנה היא שזה סימפטום למשהו רחב יותר.
hi-IN	Hindi (India)	Male	MadhurNeural	सिद्धार्थ ने भी शहनाज के साथ इस इवेंट की फोटो शेयर की है।
hr-HR	Croatian (Croatia)	Male	SreckoNeural	Video je pregledan gotovo 70 tisuća puta, a neki od obožavatelja su mu u komentarima pisali kako ih je motivirao.
hu-HU	Hungarian (Hungary)	Male	TamasNeural	A lakóhelyem nagyon komfortos.
id-ID	Indonesian (Indonesia)	Female	GadisNeural	Inflasi ringan terjadi apabila kenaikan harga berada di bawah angka 10% setahun.
ms-MY	Malay (Malaysia)	Male	OsmanNeural	Setiap individu perlu memakai topeng muka ketika berada di luar.
nb-NO	Norwegian (Bokmål, Norway)	Male	FinnNeural	Jansson forteller at den svenske øya tar imot rundt 8000 besøkende fra Norge årlig.
nb-NO	Norwegian (Bokmål, Norway)	Female	PernilleNeural	For en fantastisk forestilling!
nl-NL	Dutch (Netherlands)	Female	FennaNeural	De afstand tussen Rotterdam en Breda is ongeveer 45 km.
nl-NL	Dutch (Netherlands)	Male	MaartenNeural	Zij heeft haar studie al een tijdje geleden afgerond.
pl-PL	Polish (Poland)	Female	AgnieszkaNeural	To już nie będzie to samo, będzie drożej.
pl-PL	Polish (Poland)	Male	MarekNeural	Na wszelki wypadek sprawdź, czy coś cię jednak nie zaskoczy.
pt-PT	Portuguese (Portugal)	Male	DuarteNeural	Para a aprovação do exame, tenho de ter pelo menos 80% das respostas corretas.
pt-PT	Portuguese (Portugal)	Female	RaquelNeural	A minha mãe ensinou-me que devo ter respeito por todos, mas principalmente pelos mais velhos.
ro-RO	Romanian (Romania)	Male	EmilNeural	Actul normativ se axează pe instituirea de măsuri active, 41,5 % din salariul de bază la revenirea din șomaj tehnic.
ru-RU	Russian (Russia)	Male	DmitryNeural	Ранее посольство требовало от агентства опровержения статьи о количестве больничных коек в России.
ru-RU	Russian (Russia)	Female	SvetlanaNeural	Изменений в организме людей, попробовавших еду без приправ, не произошло.
sk-SK	Slovak (Slovakia)	Male	LukasNeural	Zápis 45 % je v skutočnosti iba skratka pre zlomok.
sl-SI	Slovenian (Slovenia)	Male	RokNeural	Zloraba bonov in dvigovanje cen turističnih storitev je nesprejemljivo ravnanje.
sv-SE	Swedish (Sweden)	Male	MattiasNeural	Båda lagen bjöd på riktigt bra hockey och skapade flera riktigt bra målchanser.
sv-SE	Swedish (Sweden)	Female	SofieNeural	Det fanns ingen trafik runt torget.
ta-IN	Tamil (India)	Male	ValluvarNeural	எவ்வளவு அருமையான பாடல் அது!
te-IN	Telugu (India)	Male	MohanNeural	అబ్బ, ఎంత పెద్ద భవనమో!
th-TH	Thai (Thailand)	Male	NiwatNeural	ธุรกิจขายอาหารเป็นธุรกิจที่ได้รับความนิยมมากที่สุด
tr-TR	Turkish (Turkey)	Male	AhmetNeural	Sosyal mesafeye büyük ölçüde riayet eden çocuklar, başta mahalle parkları olmak üzere sahiller ve oyun parklarında enerji attı.
vi-VN	Vietnamese (Vietnam)	Male	NamMinhNeural	Nhiệt độ hiện tại ở thành phố Hồ Chí Minh là 38 độ C.
zh-HK	Chinese (Cantonese, Traditional)	Female	HiuMaanNeural	抗疫舉措成為安全重啟經濟的重要一環。
zh-HK	Chinese (Cantonese, Traditional)	Male	WanLungNeural	隨着疫情緩和，愈來愈多人回到辦公室上班，但是很多人仍想留在家中工作（work from home）。
zh-TW	Chinese (Taiwanese Mandarin)	Female	HsiaoChenNeural	賭博的勝率應該不到50%。
zh-TW	Chinese (Taiwanese Mandarin)	Male	YunJheNeural	台北車站大廳能不能坐，連日引發正反意見。

5 new voices are in public preview

We have also added 5 male voices in the 5 low-resource languages that have been supported since November. These voices are available in public preview in three Azure regions: EastUS, SouthEastAsia and WestEurope.

Hear the samples below:

Locale	Language	Gender	Voice Name	Sample audio
et-EE	Estonian (Estonia)	Male	KertNeural	Ametlikku meetodit sellise pettuse avastamiseks ei olegi olemas.
ga-IE	Irish (Ireland)	Male	ColmNeural	Ritheadh próiseas comhairliúcháin faoin scéal sa bhfómhar.
lt-LT	Lithuanian (Lithuania)	Male	LeonasNeural	Aišku, anksčiau ar vėliau paaiškės tos priežastys.
lv-LV	Latvian (Latvia)	Male	NilsNeural	Aizvadīto gadu uzņēmums noslēdzis ar 6,3 miljonu eiro zaudējumiem.
mt-MT	Maltese (Malta)	Male	JosephNeural	Anki tfajjel tal-primarja jaf li l-popolazzjoni tikber fejn hemm il-prosperità.

With this release, we now support a total of 129 neural voices across 54 languages/locales. In addition, over 70 standard voices are available in 49 languages/locales. Visit Language support - Speech service - Azure Cognitive Services | Microsoft Docs for the full language and voice list.

Continuous voice quality improvement

In general, Neural TTS can convert text to lifelike speech, however there are nuances that can always be improved. For example, we have customers who have requested the ability to support a scenario where Katja, our de-DE neural voice, can pronounce English words in the context of a German sentence. This was valuable feedback, and we anticipate a similar need across languages.

For German, we observed that our users prefer the voice to handle an English word/phrase as close as the native English pronunciation. To enable a voice model to speak English as a second language, it is normally required that we collect the speech data of the same speaker speaking English besides his/her native language. This is a big challenge as we do not have sufficient multi-language speech data from our German voice talents. By leveraging cross-lingual capability of UNI-TTS, we are able to generate more English pronunciation data with the transferred voice from our German voice talent. Such data is used to improve the quality of the English word/phrase pronunciations for the German Katja voice, so Katja can pronounce English words in a more natural way.

The CMOS metric is used to measure the improvement of the English word pronunciation for Katja. Below table shows that the updated model is significantly better in pronouncing English words in the context of a German sentence.

Script	Old	New
Star Wars - Das Erwachen der Macht
Three Billboards outside Ebbing, Missouri.

This improvement has now been released to the Azure Neural TTS service for Katja. Moving forward, we’ll extend this capability to support more languages.

Tell us your experience!

By offering more voices across more languages and locales, we anticipate developers across the world will be able to build applications that change experiences for millions. Whether you’re building a voice-enabled chatbot or IoT device, an IVR solution, adding read-aloud features to your app, converting e-books to audio books, or even adding Speech to a translation app, you can make all these experiences natural sounding and fun with Neural TTS.

Get started

Add voice to your app in 15 minutes

Explore the available voices in this demo

Build a voice-enabled bot

Deploy Azure TTS voices on prem with Speech Containers

Build your custom voice

Re: Introducing QnA Maker managed: now in public preview

aowens-jmt — Thu, 10 Dec 2020 12:32:34 GMT

Whom can I reach out to, for questions and bugs?

Re: Introducing QnA Maker managed: now in public preview

nerajput — Mon, 07 Dec 2020 08:02:38 GMT

Glad that you liked our new offering. We are still working on the pricing and yes it will mostly be aligned to the current pricing model or even slightly simpler.

Thanks a lot. May be the issue is with the kind of structure/formatting those PDF files contains as our extraction currently works on the formatting structure of the semi-structured files. Could you please drop us a mail at qnamakerteam@microsoft.com so that we can debug this, we will be happy to help.

Re: Introducing QnA Maker managed: now in public preview

PeytonMcM — Thu, 03 Dec 2020 22:26:50 GMT

Looks great! We are trying to propose QnA maker in a customer's environment to help index a lot (hundreds) of documents. However, in our testing, QnA maker is having problems reading (ingesting) many of the pdf documents. Is this a common issue? Should we use Azure Search instead?

Meta-data driven key-value pairs extraction with Azure Form Recognizer

IssaghaBa — Mon, 30 Nov 2020 23:07:44 GMT

Most organizations are now aware of how valuable the forms (pdf, images, videos…) they keep in their closets are. They are looking for best practices and most cost-effective ways and tools to digitize those assets. By extracting the data from those forms and combining it with existing operational systems and data warehouses, they can build powerful AI and ML models to get insights from it to deliver value to their customers and business users.

With the Form Recognizer Cognitive Service, we help organizations to harness their data, automate processes (invoice payments, tax processing …), save money and time and get better accuracy.

Figure 1:Typical form

In my first blog about the automated form processing, I described how you can extract key-value pairs from your forms in real-time using the Azure Form Recognizer cognitive service. We successfully implemented that solution for many customers.

Often, after a successful PoC or MVP, our customers realize that, not only they need this real time solution but, they also have a huge backlog of forms they would like to ingest into their relational, NoSQL databases or data lake, in a batch fashion. They have different types of forms and they don’t want to build a model for each type. They are also looking for easy and quick way to ingest the new type of forms.

In this blog, we’ll describe how to dynamically train a form recognizer model to extract the key-value pairs of different type of forms and at scale using Azure services. We’ll also share a github repository where you can download the code and implement the solution we describe in this post.

The backlog of forms maybe in your on-premises environment or in a (s)FTP server. We assume that you were able to upload them into an Azure Data Lake Store Gen 2, using Azure Data Factory, Storage Explorer or AzCopy. Therefore, the solution we’ll describe here will focus on the data ingestion from the data lake to the (No)SQL database.

Our product team published a great tutorial on how to Train a Form Recognizer model and extract form data by using the REST API with Python. The solution described here demonstrates the approach for one model and one type of forms and is ideal for real-time form processing.

The value-add of the post is to show how to automatically train a model with new and different type of forms using a meta-data driven approach, in batch mode.

Below is the high-level architecture.

Figure 2: High Level Architecture

Azure services required to implement this solution

To implement this solution, you will need to create the below services:

Form Recognizer resource:

Form Recognizer resource to setup and configure the form recognizer cognitive service, get the API key and endpoint URI.

Azure SQL single database:

We will create a meta-data table in Azure SQL Database. This table will contain the non-sensitive data required by the Form Recognizer Rest API. The idea is, whenever there is a new type of form, we just insert a new record in this table and trigger the training and scoring pipeline.
The required attributes of this table are:

form_description: This field is not required as part of the training of the model the inference. It just to provide a description of the type of forms we are training the model for (example client A forms, Hotel B forms,...)

training_container_name: This is the storage account container name where we store the training dataset. It can be the same as scoring_container_name

training_blob_root_folder: The folder in the storage account where we’ll store the files for the training of the model.

scoring_container_name: This is the storage account container name where we store the files we want to extract the key value pairs from. It can be the same as the training_container_name

scoring_input_blob_folder: The folder in the storage account where we’ll store the files to extract key-value pair from.
model_id: The identify of model we want to retrain. For the first run, the value must be set to -1 to create a new custom model to train. The training notebook will return the newly created model id to the data factory and, using a stored procedure activity, we’ll update the meta data table with in the Azure SQL database.

Whenever you had a new form type, you need to reset the model id to -1 and retrain the model.

file_type: The supported types are application/pdf, image/jpeg, image/png, image/tif.

form_batch_group_id : Over time, you might have multiple forms type you train against different models. The form_batch_group_id will allow you to specify all the form types that have been training using a specific model.

Azure Key Vault:

For security reasons, we don’t want to store certain sensitive information in the parametrization table in the Azure SQL database. We store those parameters in Azure Key Vault secrets.

Below are the parameters we store in the key vault:

CognitiveServiceEndpoint: The endpoint of the form recognizer cognitive service. This value will be stored in Azure Key Vault for security reasons.

CognitiveServiceSubscriptionKey: The access key of the cognitive service. This value will be stored in Azure Key Vault for security reasons. The below screenshot shows how to get the key and endpoint of the cognitive service

Figure 3: Cognitive Service Keys and Endpoint

StorageAccountName: The storage account where the training dataset and forms we want to extract the key value pairs from are stored. The two storage accounts can be different. The training dataset must be in the same container for all form types. They can be in different folders.

StorageAccountSasKey : the shared access signature of the storage account

The below screen shows the key vault after you create all the secrets

Figure 4 : Key Vault Secrets

Azure Data Factory:

To orchestrate the training and scoring of the model. Using a look up activity, we’ll retrieve the parameters in the Azure SQL Database and orchestrate the training and scoring of the model using Databricks notebooks. All the sensitive parameters stored in Key vault will be retrieve in the notebooks.

Azure Data Lake Gen 2:

To store the training dataset and the forms we want to extract the key-values pairs from. The training and the scoring datasets can be in different containers but, as mentioned above, the training dataset must be in the same container for all form types.

Azure Databricks:

To implement the python script to train and score the model. Note that we could have used Azure functions.

Azure Key Vault:

To store the sensitive parameters required by the Form Recognizer Rest API.

The code to implement this solution is available in the following GitHub repository.

Additional Resources

Get started with deploying Form Recognizer –

Custom Model – extract text, tables and key value pairs

QuickStart: Train a Form Recognizer model and extract form data by using the REST API
QuickStart: Train a Form Recognizer model with labels using the sample labeling tool
Form Recognizer Sample Labeling Tool

Try it out: https://fott.azurewebsites.net/
Open Source project: https://github.com/microsoft/OCR-Form-Tools

Prebuilt receipts - extract data from USA sales receipts

Quickstart: Extract receipt data using the REST API

Layout - extract text and table structure (row and column numbers) from your documents

Quickstart: Extract layout data using the REST API

See What’s New

Re: Introducing QnA Maker managed: now in public preview

HesselW — Wed, 25 Nov 2020 12:41:04 GMT

You totally made my day. Just started a chatbot for a customer in need of three languages in one bot. This will help a lot. I already liked the fact that I don't have to manually change key's to link the qna service to another search service :flexed_biceps:

I do hope that product marketing keeps the price in line with the 'old' approach

Introducing Asynchronous APIs for Text Analytics and Text Analytics for Health

AshlyYeo — Tue, 24 Nov 2020 21:35:16 GMT

This post is co-authored with Sara Kandil

Today, we are announcing a preview of new asynchronous (batch) APIs for Text Analytics and Text Analytics for health, which enable developers to apply Natural Language Processing (NLP) to even more scenarios so they can identify key phrases, entities and even personally identifiable information (PII).

Asynchronous Analyze API for Text Analytics

Text Analytics is a generally available Azure Cognitive Service that lets you discover insights in text using Natural Language Processing (NLP). The service helps you identify key phrases and entities (people, place, organization, event, date among others), recognize text that contains personal information (PII) and analyze sentiment (positive, neutral, or negative).

To date, customers have been using Text Analytics by making synchronous calls to the service’s REST API, client library SDK, or by using containers to run Text Analytics in their own environment. Today, we are introducing a new preview Analyze operation for users to analyze larger documents asynchronously combining multiple Text Analytics features in one call. This gives customers the flexibility to analyze more information, at once, when their applications don’t need a synchronous response. The new asynchronous Analyze operation for Text Analytics supports individual documents of up to 125k characters, and up to 25 documents in a request.

The Analyze operation preview supports key phrase extraction, named entity recognition and PII recognition and is available in 5 Azure regions (West US 2, East US2, West Europe, North Europe, and Central US). Support for the rest of the Text Analytics capabilities and additional regions is coming soon.

Asynchronous Analyze API for Text Analytics for health

We are also introducing a new asynchronous hosted API for Text Analytics for health. As a refresher, early this year (July), we announced a preview of Text Analytics for health, a capability for the healthcare industry, trained to extract insights from medical data. With Text Analytics for health, users can:

Detect words and phrases mentioned in unstructured text as entities that are associated with semantic types in the healthcare and biomedical domain – such as diagnosis, medication name, symptom/sign, and more.
Link entities to medical ontologies and domain-specific coding systems (for example, the Unified Medical Language System), and extract meaningful connections between concepts mentioned in text (for example, finding the relationship between a medication name and the dosage associated with it.)
Detect negation of the different entities mentioned in the text.

Example of Text Analytics for health at work.

Previously, Text Analytics for health was only available for use via containers. This new API gives users the option to use the hosted service and avoid the heavy lifting of hosting containers unless they need to.

The hosted Text Analytics for health operation supports document sizes up to 5k characters and up to 10 documents in a single request. It is available for use in the West US 2, East US 2, Central US, North and West Europe regions.

In summary, Text Analytics is now more accessible with more ways to use the capabilities depending on your scenario. You can:

Call the synchronous endpoints to use the Text Analytics features.
Call the asynchronous Analyze API to process larger documents with multiple Text Analytics features in a single call.
Call the hosted asynchronous Text Analytics for health API if your dataset that is being analyzed has clinical and biomedical documents.
Use Text Analytics containers to host the endpoint in your own environments that meets your privacy and security requirements.

The new Text Analytics asynchronous APIs are available to use in Preview today. Please refer to our documentation to learn more and get started with these new APIs.

Re: Apps can now narrate what they see in the world as well as people do

Hao Hu — Mon, 23 Nov 2020 13:49:58 GMT

That‘s looks like more and more new functions release with REST API 3.1.

Re: Computer Vision for spatial analysis at the Edge

hemantkamalakar — Mon, 23 Nov 2020 09:18:17 GMT

I would also like to run the Spatial Analytics container on nvidia Jetson Nano dev kit - 4GB.

Re: Introducing QnA Maker managed: now in public preview

mikko1 — Fri, 20 Nov 2020 09:40:27 GMT

It's wonderful! I can deploy very quickly and manage resources very easily! Thanks Team!

Re: Introducing QnA Maker managed: now in public preview

ivanatilca — Thu, 19 Nov 2020 16:59:40 GMT

This is amazing !

Neural Text-to-Speech previews five new languages with innovative models in the low-resource setting

Qinying Liao — Thu, 19 Nov 2020 16:30:01 GMT

This post is co-authored with Xianghao Tang, Lihui Wang, Jun-Wei Gan, Gang Wang, Garfield He, Xu Tan and Sheng Zhao

Neural Text-to-Speech (Neural TTS), part of Speech in Azure Cognitive Services, enables you to convert text to lifelike speech for more natural user interactions. Neural TTS has powered a wide range of scenarios, from audio content creation to natural-sounding voice assistants, for customers from all over the world. For example, the BBC, Progressive and Motorola Solutions are using Azure Neural TTS to develop conversational interfaces for their voice assistants in English speaking locales. Swisscom and Poste Italiane are adopting neural voices in French, German and Italian to interact with their customers in the European market. Hongdandan, a non-profit organization, is using neural voices in Chinese to make their online books audible for the blind people in China.

By September 2020, we extended Neural TTS to support 49 languages/locales with 68 voices. At the same time, we continue to receive customer requests for more voice choices and more language support globally.

Today, we are excited to announce that Azure Neural TTS has extended its global support to five new languages: Maltese, Lithuanian, Estonian, Irish and Latvian, in public preview. At the same time, Neural TTS Container is generally available for customers who want to deploy neural voice models on-prem for specific security requirements.

Neural TTS previews 5 new languages

Five new voices and languages are introduced to the Neural TTS portfolio. They are: Grace in Maltese (Malta), Ona in Lithuanian (Lithuania), Anu in Estonian (Estonia), Orla in Irish (Ireland) and Everita in Latvian (Latvia). These voices are available in public preview in three Azure regions: EastUS, SouthEastAsia and WestEurope.

Hear samples of these voices, or try them with your own text in our demo.

Locale	Language	Voice name	Audio sample
mt-MT	Maltese (Malta)	“mt-MT-GraceNeural”	Fid-diskors tiegħu, is-Segretarju Parlamentari fakkar li dan il-Gvern daħħal numru ta’ liġijiet u inizjattivi li jħarsu lill-annimali.
lt-LT	Lithuanian (Lithuania)	“lt-LT-OnaNeural”	Derinti motinystę ir kūrybą išmokau jau po pirmojo vaiko gimimo.
et-EE	Estonian (Estonia)	“et-EE-AnuNeural”	Pese voodipesu kord nädalas või vähemalt kord kahe nädala järel ning ära unusta pesta ka kardinaid.
ga-IE	Irish (Ireland)	“ga-IE-OrlaNeural”	Tá an scoil sa mbaile ar oscailt arís inniu.
lv-LV	Latvian (Latvia)	“lv-LV-EveritaNeural”	Daži tumšās šokolādes gabaliņi dienā ir gandrīz būtiska uztura sastāvdaļa.

With these updates, Azure TTS service now supports 54 languages/locales with 78 neural voices and 77 standard voices available.

Behind the scenes: 10X faster voice building with the low resource setting.

The creation of a TTS voice model normally requires a large volume of training data, especially for extending to a new language, where sophisticated language-specific engineering is required. In this section, we introduce “LR-UNI-TTS”, a new Neural TTS production pipeline to create TTS languages where training data is limited, i.e., ‘low-resourced’. With this innovation, we are able to improve the Neural TTS locale development with 10x agility and support the five new languages quickly.

High resource vs. low resource

Traditionally, it can easily take more than 10 months to extend TTS service to support a new language due to the extensive language-specific engineering required. This includes collecting tens of hours of language-specific training data, and creating hand-crafted components like text analysis etc.. In many cases, one major challenge for supporting a new language is that such large volume of data is unavailable or hard to find, causing a language ‘low-resourced’ for TTS model building. To handle the challenge, Microsoft researchers have proposed an innovative approach, called LRSpeech, to handle the extremely low-resourced TTS development. It has been proved that LRSpeech has the capability to build good quality TTS in the low-resource setting, using multilingual pre-training, knowledge distillation, and importantly the dual transformation between text-to-speech (TTS) and speech recognition (SR).

How LR-UNI-TTS works

Built on top of LRSpeech and the multi-lingual multi-speaker transformer TTS model (called UNI-TTS), we have designed the offline model training pipeline and the online inference pipeline for the low-resource TTS. Three key innovations contribute to the significant agility gains with this approach.

First, by leveraging the parallel speech data (the pairing speech audios and the transcript) collected during the speech recognition development, the LR-UNI-TTS training pipeline greatly reduces the data requirements for refining the base model in the new language. Previously, the high-quality multi-speaker parallel data has been critical in extending TTS to support a new language. The TTS speech data is more difficult to collect as it requires the data to be clean, the speaker carefully selected, and the recording process well controlled to ensure the high audio quality.

Second, by applying the cross-lingual speaker transfer technology with the UNI-TTS pipeline, we are able to leverage the existing high-quality data in a different language to produce a new voice in the target language. This saves the effort to find a new professional speaker for the new languages. Traditionally, the high-quality parallel speech data in the target language is required, which easily takes months for the voice design, voice talent selection, and recording.

Lastly, the LR-UNI-TTS approach uses characters instead of phonemes as the input feature to the models, while the high-resource TTS pipeline is usually composed of a multi-step text analysis module that turns text into phonemes, costing long time to build.

Below figure describes the offline training pipeline for the low-resource TTS voice model.

Figure 1. The offline training pipeline for the low-resource TTS voice model.

In specific, at the offline training stage, we have leveraged a few hundred hours of the speech recognition data to further refine the UNI-TTS model. It can help the base model to learn more prosody and pronunciation patterns for the new locales. The speech recognition data is usually collected in daily environments using PC or mobile devices, unlike the TTS data which is normally collected in the professional recording studios. Although the SR data can be much lower-quality than the TTS data, we have found LR-UNI-TTS can benefit from such data effectively.

With this approach, the high-quality parallel data in the new language which is usually required for the TTS voice training becomes optional. If such high-quality parallel data is available, it can be used as the target voice in the new language. If no high-quality parallel data is available, we can also choose a suitable speaker from an existing but different language and transfer it into the new language through the cross-lingual speaker transfer-learning capability of UNI-TTS.

In the below chart, we describe the flow of the runtime.

Figure 2: The online inference pipeline for the low-resource TTS voice model.

At the runtime, a lightweight text analysis is designed to preprocess the text input with sentence separation and text normalization. Compared to the text analysis component of the high-resource language pipelines, this module is greatly simplified. For instance, it does not include the pronunciation lexicon or letter-to-sound rules which are used in high-resource languages. The normalized text characters are generated by the lightweight text analysis component. During this process, we also leverage the text normalization rules from the speech recognition development, which saves the overall cost a lot.

The other components are similar to the high-resource language pipelines. For example, the neural acoustic model uses the FastSpeech model to convert the character input into mel-spectrogram.

Finally, the neural vocoder HiFiNet is used to convert the mel-spectrogram into audio output.

Overall, using LR-UNI-TTS, a TTS model in a new language can be built in about one month, which is 10x faster than the traditional approaches.

In the next section, we share the quality measurement results for the voices built with LR-UNI-TTS.

Quality assessments

Similar to other TTS voices, the quality of the low-resource voices created in the new languages are measured using the Mean Opinion Score (MOS) tests and intelligibility tests. MOS is a widely recognized scoring method for speech naturalness evaluation. With MOS studies, participants rate speech characteristics such as sound quality, pronunciation, speaking rate, and articulation on a 5-point scale, and an average score is calculated for the report. Intelligibility test is a method to measure how intelligible a TTS voice is. With intelligibility tests, judges are asked to listen to a set of TTS samples and mark out the unintelligible words to them. Intelligibility rate is calculated using the percentage of the correctly intelligible words among the total number of words tested (i.e., the number of intelligible words/the total number of words tested * 100%). Normally a usable TTS engine needs to reach a score of > 98% for intelligibility.

Below table summarizes the MOS score and the intelligibility score of the five new languages created using LR-UNI-TTS .

Locale	Language (Region)	Average MOS	Intelligibility
mt-MT	Maltese (Malta)	3.59*	98.40%
lt-LT	Lithuanian (Lithuania)	4.35	99.25%
et-EE	Estonian (Estonia)	4.52	98.73%
ga-IE	Irish (Ireland)	4.62	99.43%
lv-LV	Latvian (Latvia)	4.51	99.13%

* Note: MOS scores are subjective and not directly comparable across languages. The MOS of the mt-MT voice is relatively lower but reasonable in this case considering that the human recordings used as the training data for this voice also gots a lower MOS.

As shown in the table, the voices created with the low resources available are highly intelligible and have achieved high or reasonable MOS scores among the native speakers.

It’s worth pointing out that due to the nature of the lightweight text analysis module for the runtime, the phoneme-based SSML tuning capabilities are not supported for the low-resource voice models, for example, the ‘phoneme’ and the ‘lexicon’ elements.

Coming next: extending Neural TTS to even more locales

LR-UNI-TTS has paved the way for us to extend Neural TTS to more languages for the global users more quickly. Most excitingly, LR-UNI-TTS can potentially be applied to preserve the languages that are disappearing in the world today, as pointed out in the guiding principles of XYZ-code.

With the five new languages released in public preview, we welcome user feedback as we continue to improve the voice quality. We are also interested to partner with passionate people and organizations to create TTS for more languages. Contact us (mstts[at]microsoft.com) for more details.

What’s more: Neural TTS Container GA

Together with the preview of these five new languages, we are happy to share that the Neural TTS Container is now GA. With Neural TTS Container, developers can run speech synthesis with the most natural digital voices in their own environment for specific security and data governance requirements. Learn more about how to install Neural TTS Container and visit the Frequently Asked Questions on Azure Cognitive Services Containers.

Get started

With these updates, we’re excited to be powering natural and intuitive voice experiences for more customers, supporting more flexible deployment. Azure Text-to-Speech service provides more than 150 voices in over 50 languages for developers all over the world.

For more information:

Try the TTS demo
See our documentation
Check out our sample code

How to operationalize more than 100 AI models in as little as 12 weeks using Azure Databricks

ClintonWFord-Databricks — Tue, 17 Nov 2020 14:00:00 GMT

Organizations are leveraging artificial intelligence (AI) and machine learning (ML) to derive insight and value from their data and to improve the accuracy of forecasts and predictions. In rapidly changing environments, Azure Databricks enables organizations to spot new trends, respond to unexpected challenges and predict new opportunities. Data teams are using Delta Lake to accelerate ETL pipelines and MLflow to establish a consistent ML lifecycle.

Solving the complexity of ML frameworks, libraries and packages

Customers frequently struggle to manage all of the libraries and frameworks for machine learning on a single laptop or workstation. There are so many libraries and frameworks to keep in sync (H2O, PyTorch, scikit-learn, MLlib). In addition, you often need to bring in other Python packages, such as Pandas, Matplotlib, numpy and many others. Mixing and matching versions and dependencies between these libraries can be incredibly challenging.

Figure 1. Databricks Runtime for ML enables ready-to-use clusters with built-in ML Frameworks

With Azure Databricks, these frameworks and libraries are packaged so that you can select the versions you need as a single dropdown. We call this the Databricks Runtime. Within this runtime, we also have a specialized runtime for machine learning which we call the Databricks Runtime for Machine Learning (ML Runtime). All these packages are pre-configured and installed so you don’t have to worry about how to combine them all together. Azure Databricks updates these every 6-8 weeks, so you can simply choose a version and get started right away.

Establishing a consistent ML lifecycle with MLflow

The goal of machine learning is to optimize a metric such as forecast accuracy. Machine learning algorithms are run on training data to produce models. These models can be used to make predictions as new data arrive. The quality of each model depends on the input data and tuning parameters. Creating an accurate model is an iterative process of experiments with various libraries, algorithms, data sets and models. The MLflow open source project started about two years ago to manage each phase of the model management lifecycle, from input through hyperparameter tuning. MLflow recently joined the Linux Foundation. Community support has been tremendous, with 250 contributors, including large companies. In June, MLflow surpassed 2.5 million monthly downloads.

Diagram: MLflow unifies data scientists and data engineers

Ease of infrastructure management

Data scientists want to focus on their models, not infrastructure. You don’t have to manage dependencies and versions. It scales to meet your needs. As your data science team begins to process bigger data sets, you don’t have to do capacity planning or requisition/acquire more hardware. With Azure Databricks, it’s easy to onboard new team members and grant them access to the data, tools, frameworks, libraries and clusters they need.

Alignment Healthcare

Alignment Healthcare, a rapidly growing Medicare insurance provider, serves one of the most at-risk groups of the COVID-19 crisis—seniors. While many health plans rely on outdated information and siloed data systems, Alignment processes a wide variety and large volume of near real-time data into a unified architecture to build a revolutionary digital patient ID and comprehensive patient profile by leveraging Azure Databricks. This architecture powers more than 100 AI models designed to effectively manage the health of large populations, engage consumers, and identify vulnerable individuals needing personalized attention—with a goal of improving members’ well-being and saving lives.

Building your first machine learning model with Azure Databricks

To help you get a feel for Azure Databricks, follow the code samples and videos in this blog post to build a simple model using sample data in Azure Databricks. Learn how to by attending an Azure Databricks event, watch how you can Turbocharge your business with Machine Learning, leverage this free Azure Databricks ML training module on MS Learn and join us at our next Azure Databricks Office Hours.

November 2020 – Conversational AI update

GaryPrettyMsft — Mon, 16 Nov 2020 21:21:35 GMT

We are excited to announce the November release of the Bot Framework SDK and Composer, driving the Microsoft Conversational AI platform forward and building on the announcements we made in September at Microsoft Ignite. Our November update sees new updates to the Bot Framework SDK and Bot Framework Composer, adding new capabilities for developers and improving integration with our key partners, including Power Virtual Agents and HealthBot.

Bot Framework v4.11

Version 4.11 of the Bot Framework SDK, including new releases for .NET, JavaScript, Python and Java (preview 7), along with updates to our tooling, including the CLI.

Following our quality focused 4.10 release, we continue to push on this area, including improvements to the commonly used typing and transcript logging middleware behavior and associated error handling.

For developers building solutions for Microsoft Teams, new support for meetings has been added, including the Meeting Participant API and meeting specific notifications.

We continue to reduce developer friction for Skill development, adding the ability to test Skills locally, using the Bot Framework Emulator, without requiring an App Id and password. Additional scenarios, such as interruption support when calling a Skill and the ability to update or delete activities from a Skill have also been added.

Skills support has now also been added to HealthBot, a cloud platform for virtual health bots and assistants built on Bot Framework, with solutions now able consume, or for themselves to be consumed as a Bot Framework Skill.

We’re also undertaking significant investments in automated testing in this area, with the opportunity for you to review and provide feedback on the current specifications for Functional Testing and the Test Runner.

Further improvements to our documentation include expanding content across Adaptive Dialogs, Skills, overall architecture topics, as well as adding reference documentation for the Java SDK preview.

Bot Framework Composer v1.2

A new release of Composer (v1.2) is now available. This release deepens integration with Power Virtual Agents (PVA), part of the Power Platform, with a new public preview of PVA integration with Bot Framework Composer.

Users of the no-code PVA platform were already able to extend their solutions by consuming Bot Framework Skills. Now, PVA solutions can be opened in Bot Framework Composer, using a deep-link from the PVA portal, extending them with more sophisticated capabilities and enabling the collaboration between business users and developers on the same project.

Try the new Power Virtual Agents integration with Bot Framework Composer today!

When ready, Composer developers can publish directly from Composer, using a pre-configured publishing profile, back into the PVA portal, with new PVA Topics added using Composer then shown alongside existing Topics and immediately ready for testing.

An upcoming release of Composer, expected in December, will add improved provisioning and publishing support and enhanced QnA Maker knowledgebase integration.

As part of the December release, users will also have the option to enable new preview capabilities through the addition of feature flags. The first preview features planned include Orchestrator integration, the new intent detection and arbitration (dispatch) technology that runs locally within your bot, along with Form Dialogs, enabling the rapid generation of intelligent slot-filling dialogs, including complex capabilities such as slot disambiguation.

Nightly builds of Composer are available (enabled via the Composer settings page) which allow you to try the latest updates as soon as they are available.

Re: Introducing QnA Maker managed: now in public preview

vikasgoyal — Thu, 12 Nov 2020 07:19:18 GMT

Congrats Team. Loved the deployment model without compromising data residency requirements.

Introducing QnA Maker managed: now in public preview

nerajput — Thu, 19 Nov 2020 05:55:03 GMT

QnA Maker is an Azure Cognitive Service that allows you to create a conversational layer over your data- in minutes. Today, we are announcing a new version of QnA Maker which advances several core capabilities like better relevance and precise answering, by introducing state-of-art deep learning technologies.

Illustrative representation of QnA Maker functionality.

Overview of new QnA Maker managed capabilities

Summary of new features introduced:

Deep learnt ranker with enhanced relevance of results across all supported languages.
Precise phrase/short answer extraction from answer passages.
Simplified resource management by reducing the number of resources deployed.
E2E region support for Authoring + Prediction.

Detailed description of the new features is further down in this article. Learn how to migrate to the new QnA Maker managed (Preview) knowledge base here.

QnA Maker managed (Preview) Architecture.

As per the architecture of QnA Maker managed (Preview), there will be only two resources: QnA Maker service for authoring and computation and Azure Cognitive Search for storage and L1 ranking. This has been done with an aim of simplifying the resource creation and management process. Now, customers need to manage only 2 resources instead of 5 different resources.
QnA Maker managed (Preview) also allows the user to do language setting specific to Knowledge Base.
Computation has been moved out of the user subscription, so there is no dependency on the customers for scaling and availability management. This allowed us to use SOTA deep learnt model for L2 ranker which enhances the L2 ranker horizontally across all the languages, so now we support all the 50+ languages with better and enhanced precision.
QnA Maker service will be available in multiple regions to give customers’ the flexibility to keep their end-to-end service in one region.
For inference logs and telemetry, the latest version will be using Azure Monitoring instead of App insights. To keep the experience seamless and easy to adopt all the APIs has been kept backward compatible. There is almost zero change in the management portal experience.

New features of QnA Maker managed (Preview)

This section talks about all the distinguishing features of QnA maker managed in detail.

Simplified Create Blade

Onboarding on QnA Maker managed (Preview), and resource creation has been kept quite simple. Now, you will see a checkbox with Managed, as shown below. As soon as you select the checkbox, the form will be updated with the required resources.

Precise Answering

Machine Reading Comprehension based answer span detection feature is most beneficial for the scenarios where the customers have big passages present as answer in their Knowledge Base. Currently, they put good amount of manual efforts in curating small/precise answers and ingest them in the Knowledge base.

The new features give them flexibility to either choose the precise answer or the answer passage, customers can take this decision based on the confidence score of the precise short answer and answer passage. Here are some examples to show how short answers can be useful:

Deep Learnt ranker

The new L2 ranker is based on Turing multilingual language model (T-ULRv2), a deep learning-based transformer model, which improves the precision of the service for all the languages. For any user query, the new L2 ranker model understands the semantics of the user query better and gives better aligned results. This model is not language specific and is targeted to improve the overall precision of all languages horizontally. Here are some examples to analyze the difference between the results of current service and QnA Maker managed (Preview) service:

Query

Current GA results

QnA Maker managed (Preview) results

Improvements in Preview

can someone ring me

I can tell you all about Wi-Fi calling, including the devices that support Wi-Fi calling and where you can get more information yourself. Feel free to ask me a question and I'll do what I can to answer it

Yes, you can make and receive calls using Wi-Fi calling. Pretty nifty, right?

The new L2 ranker can understand the relevance between “ring me” and “make and receive calls” and is returning more relevant result unlike the current GA, which has returned a generic answer.

can’t connect to mobile data

You'll be connected to Wi-Fi, so it'll only use your minutes and text allowances.

If you don't have mobile signal, it's no problem. With Three inTouch Wi-Fi Calling, you can call and text whenever you're on Wi-Fi in the UK, even without mobile signal.

The new L2 ranker is again able to understand the query better as its able to understand that mobile data is somewhere connected to mobile signals and hence giving better results based on the data present in the Knowledge Base than the current GA model.

E2E region support

With QnA Maker managed (Preview) our management service is no more limited to west-US region. We are offering end to end region support for:

South Central US
North Europe
Australia East.

Other hero regions will be added when we go GA.

Knowledge Base specific language setting

Now, customers can create Knowledge bases with different language setting within a service. This feature is beneficial for users who have multi-language scenarios and need to power the service for more than one language. In this case, there will be a test index specific to every Knowledge Base, so that the customer can verify how the service is performing specific to every language.

You can configure this setting only with the first Knowledge base of the service, once set the user will not be allowed to update the setting.

Pricing

Public preview of QnA Maker managed (Preview) will be free in all the regions (You only pay for the Azure Cognitive Search SKU). The standard pricing will be applicable when the service goes to GA by mid-2021.

References

Azure speaks your language: the 3 immediate benefits for your organization

dturchyn — Wed, 04 Nov 2020 15:48:28 GMT

The last several years brought exciting innovations in the field of Artificial Intelligence, especially when it comes to advancements in speech and language processing. Processing speech and making text and audio information searchable enables a diverse set of innovative applications, including helping researchers in searching for related papers, or building information graphs for predicting the best new drug candidates, or uncovering issues with products and services in near real time. For region like Central and Eastern Europe, which includes 30+ countries, most speaking their own language, support for local languages is a critical condition for implementing innovation. That’s why the recent (September 2020) Azure Speech services update has opened a whole new area of opportunity for our region.

With updated language support, most of the EU languages are now supported in Azure Speech services. For region which I am covering in my current role, it means that we now have support for all of our CEE EU languages (Polish, Bulgarian, Czech, Greek, Croatian, Hungarian, Romanian, Slovak, Slovenian, Estonian, Lithuanian, Latvian, Maltese) and Russian in Azure Speech and Translator services. Additionally, our speech generation models have also been updated, now leveraging the Neural TTS - a powerful speech synthesis capability, which enables to convert text to lifelike speech which is close to human-parity. Below you will find 3 benefits, how this might help you advance your products and services today.

Automatic generation of medical summary from spoken conversations between doctors and patients

First, analyzing speech data or generating speech enables you to extract insights from audio or video information, which otherwise would be unreachable for analytical systems. This might include data like customer support conversations or employee speech in videos or transcribing speech for field employees or doctors. Communicating with your customers with natural-sounding generated speech in your own language is another area of innovation, which enables scenarios from voice announcements to supporting people with visual impairments to building voice assistants. Is information the new currency? If you answer “yes” to this — why then would you have terabytes of currency sitting without you getting use of it? Now you can turn it into tangible cash-flow.

Azure Speech services are a sub-set of pre-built (but customizable) APIs for working with Speech. This includes transcribing spoken language into text for further analysis (Speech-to-Text) and generating naturally sounding speech form text input (Text-to-Speech). Azure Translator is another piece in the puzzle, which has also received major update for the languages, now translating text between 70+ languages.

Second, there are new scenarios enabled now by these pre-built AI models. Do you have that innovative idea for analysing customer conversations or augmenting your service with spoken messages in your local language? Often, these ideas were not realized due to the associated challenges like finding the right skilled people within your organization and investing into a project with unknown development cycle and returns. Now it is possible to build a realistic prototype app quickly to extract insights from your speech data, by calling the service through the API — in days, if not hours.

Analysing customer support conversations brings insights from priceless data, which is untapped without applying Speech processing

Third, this is one of those cloud services, which may work without sending your data to the cloud! Many of Azure Cognitive Services today may be deployed right within your own data center as containers. This means, that none of the actual data will be sent to the cloud, as even processing will happen locally. In this case, only billing information will be exchanged with Azure.

Interested enough to give it a try? If you are interested in learning more, you may request detailed information or virtual session on Azure Cognitive Services from our sales representatives (please specify whether you are looking for the session on Azure Cognitive services, or details of your specific projects where Speech services may be used). To read more or test Azure Speech services capabilities in your language, please refer to our Azure Speech Services Documentation.

Looking forward to the exciting results you will achieve in your business with the updated Azure Speech Services!

Re: Using GitHub Actions & Azure Machine Learning for MLOps

shikhaagrawal — Wed, 04 Nov 2020 13:12:46 GMT

Hi David

I am trying to follow this article but getting below error in github actions:

Microsoft REST Authentication Error: Get Token request returned http error: 
400 and server response: ***"error":"unauthorized_client",
"error_description":"AADSTS700016: Application with identifier '***' was not found in the directory '***'. 
This can happen if the application has not been installed by the administrator of the tenant or 
consented to by any user in the tenant. 
You may have sent your authentication request to the wrong tenant.\r\n
Trace ID: a4df3138-08b0-422c-88bb-5dd4eaa3c900\r\nCorrelation ID: f79fe3ff-a262-432c-ad93-ba189c0f88ba\r\nTimestamp: 2020-11-04 08:34:23Z",
"error_codes":[700016],"timestamp":"2020-11-04 08:34:23Z","trace_id":"a4df3138-08b0-422c-88bb-5dd4eaa3c900",
"correlation_id":"f79fe3ff-a262-432c-ad93-ba189c0f88ba","error_uri":"https://login.microsoftonline.com/error?code=700016"***

Azure Neural TTS upgraded with HiFiNet, achieving higher audio fidelity and faster synthesis speed

Qinying Liao — Thu, 19 Nov 2020 16:41:18 GMT

This post was co-authored with Jinzhu Li and Sheng Zhao

Neural Text to Speech (Neural TTS), a powerful speech synthesis capability of Cognitive Services on Azure, enables you to convert text to lifelike speech which is close to human-parity. Since its launch, we have seen it widely adopted in a variety of scenarios by many Azure customers, from voice assistants like the customer service bot like BBC and Poste Italiane, to audio content creation scenarios like Duolingo.

Voice quality, which includes the accuracy of pronunciation, the naturalness of prosody such as intonation and stress patterns, and the fidelity of audio, is the key reason that customers are migrating from the traditional TTS voices to neural voices. Today we are glad to share that we have upgraded our Neural TTS voices with a new-generation vocoder, called HiFiNet, which results much higher audio fidelity while significantly improving the synthesis speed. This is particularly beneficial to customers whose scenario relies on hi-fi audios or long interactions, including video dubbing, audio books, or online education materials.

What’s new?

Our recent updates on Azure Neural TTS voices include a major upgrading of the vocoder. The voice fidelity has been improved significantly and audio quality defects such as glitches and small noises are largely reduced. Our tests show that this new vocoder generates audios without hearable quality loss from the recordings of training data (more details are introduced later). In addition, it can synthesize speech much faster than our previous version of the product. All these benefits are achieved through a new-generation neural vocoder, called HiFiNet.

What is a vocoder and why does it matter?

Vocoder is a major component in speech synthesis, or text-to-speech. It turns an intermediate form of the audio, which is called acoustic feature, into audible waveform. Neural vocoder is a specific vocoder design which uses deep learning networks and is a critical module of Neural TTS.

Microsoft Azure Neural TTS consists of three major components in the engine: Text Analyzer, Neural Acoustic Model, and Neural Vocoder. To generate natural synthetic speech from text, first, text is input into Text Analyzer, which provides output in the form of phoneme sequence. A phoneme is a basic unit of sound that distinguishes one word from another in a particular language. Sequence of phonemes defines the pronunciations of the words provided in the text. Then the phoneme sequence goes into the Neural Acoustic Model to predict acoustic features, which defines speech signals, such as speaking style, speed, intonations, and stress patterns, etc. Finally, the Neural Vocoder converts the acoustic features into audible waves so the synthetic speech is generated.

The vocoder is critical to the final audio quality. In specific, it directly impacts the fidelity of a wave, including clearness, timbre, etc. Let’s hear the difference of the audio quality with samples generated using different neural vocoders based on the same acoustic features (recommended to listen with a high-quality headset).

Vocoder versions	2018 vocoder for real-time synthesis	2019 vocoder for real-time synthesis	2020 vocoder for real-time synthesis (HiFiNet)
“Top cinematographers weigh in on filmmaking in the age of streaming.”

With each vocoder update, the speech generated sounds clearer, voice less muffled and noises reduced. In the next section, we introduce how a HiFiNet vocoder is trained during the creation of a neural voice model.

How does HiFiNet work?

In Azure TTS system, neural voice models are trained using human voice recordings as training data with deep learning networks. As part of the training, a vocoder is built with the goal to generate high quality audio output close to the original recordings from the training data. In the meantime, it needs to run fast enough to produce at least 24,000 samples per seconds, i.e. with a sampling rate of 24khz, which is the default sampling rate of Azure Neural TTS voice models.

Leveraging the state-of-art research on vocoders, we designed the training pipeline for HiFiNet, the new-generation Neural TTS vocoder, and applied it to create neural voice models in Azure Neural TTS. This pipeline is built with one simple goal: produce machine-generated audio waves (synthetic speech) that is indistinguishable from its original waves (human recordings) in a high speed.

Below chart describes how the HiFiNet training pipeline works. With this pipeline, two key networks are trained: A Generator which is used to create audio (‘Generated Wave’), and a Discriminator which is used to identify the gap of the created audio from its training data (‘Real Wave’). The goal of the training is to make the Generator generate waves that the Discriminator can’t distinguish from the original real recordings.

Training pipeline of the HiFiNet Vocoder

First, the training pipeline uses the original human recording as input and extract the acoustic features. Then, the acoustic features are fed into the Generator module which generates waves, so we get two sets of waves: the original recordings as real waves, and the generated waves as fake waves. Next, the two sets of waves are fed into the Discriminator network to distinguish which are the real waves and which are the generated fake waves. This output from the Discriminator is used as feedback to help the Generator and Discriminator to learn better. As this training loop continues, the Generator becomes smarter to create indistinguishable fake waves, while the Discriminator gets smarter in making the right judgements. Finally, when the training reaches a point where Discriminator can’t distinguish the waves generated by the Generator from real waves, the vocoder is successfully trained. This vocoder is capable of producing audio outputs without noticeable quality loss compared to the original human recordings.

In the next section we describe the performance of HiFiNet vocoder.

What are the benefits?

HiFiNet significantly improves audio quality.

To understand the benefit of HiFiNet, we conducted a number of tests in many aspects which yielded positive results. Our tests show that the HiFiNet vocoder significantly improves the audio quality of the Neural TTS voice output, compared to our previous version of the product.

CMOS (Comparative Mean Opinion Score) is a well accepted method in the speech industry for comparing the voice quality of two TTS systems. A CMOS test is similar to an A/B testing, where participants listen to different pairs of audio samples generated by two systems and provide their subjective opinions on how A compares to B. Normally in one test, we recruit 30-60 anonymous testers with qualified language expertise to evaluate around 50 pairs of audio samples side by side. The result is reported as CMOS gap, which measures the average of the difference in the opinion score between the two systems. In the cases where the absolute value of a CMOS gap is <0.1, we claim system A and B are on par. When the absolute value of a CMOS gap is >=0.1, then one system is reported better than the other. If the absolute value of a CMOS gap is >=0.2, we say one system is significantly better than the other.

We have done hundreds of CMOS tests of HiFiNet compared to our last version vocoder, on 68 neural voices across 49 languages/locales. Our results show that HiFiNet is notably better than the previous production vocoder in Azure Neural TTS.

In general, the audio quality, especially the fidelity is obviously improved. On average, across all languages, the HiFiNet vocoder achieves a CMOS gain higher than 0.2 compared to the previous vocoder, which means the improvement is hearable for users.

In particular, HiFiNet also has better robustness than the previous version of vocoder. Audio defects are largely reduced in the generated waves with HiFiNet. Our tests show that with the previous production vocoder, in 100 test samples, our testers can hear about 10 defects like beep, click sound, fidelity loss. Although most of them are not obvious, it can still be annoying if it keeps happening in a long audio or multi-round voice interactions. Now, these defects are no longer reported with the HiFiNet audios, under the same test procedure with the same test sets.

With these advantages, we have updated the Neural TTS voices on Azure Cognitive Services with the new vocoder. Listen to the samples below to hear the difference. Or test the new voices using your own text with our online demo.

Language	Previous vocoder	HiFiNet	HiFiNet CMOS gain
English (US)			+0.122 (Better)
German			+0.193 (Better)
Chinese (Mandarin, Simplified)			+0.348 (Obviously Better)
Japanese			+0.465 (Obviously Better)

HiFiNet reaches human-parity audio fidelity.

In addition, we have conducted tests to compare the human recording audio quality and the computer-generated audio quality with HiFiNet. To make the comparison more accurate and more focused on the vocoder itself, we use the acoustic features extracted directly from human recordings instead of the TTS-predicted acoustic features so the acoustic differences are controlled and only the vocoder is evaluated in CMOS tests. Participants are asked to give their scores for different pairs of the generated waves and human recordings. Our result shows the CMOS gap of the audios produced by HiFiNet compared to human recordings is -0.05, which means the difference is hardly hearable and the audio quality is on par.

Hear how close the HiFiNet audio fidelity is to the human recordings with the samples below.

Language	Human recording	HiFiNet	HiFiNet CMOS gap
English (US)			+0.045 (on par)
Chinese (Mandarin, Simplified)			-0.054 (on par)

HiFiNet generates audios faster.

Real Time Factor (RTF) is used to measure the performance of vocoder. It is calculated as the time duration needed to generate the audio divided by the audio duration.

HiFiNet is a parallel vocoder so it can generate multiple samples at the same time. Here are some measurements of HiFiNet performance on both GPU and CPU devices.

With the output of 24khz sampling rate, on M60 GPU, through carefully optimized CUDA implementation, the vocoder RTF is around 0.01, which means the HiFiNet system can generate an audio 10 second-long in 0.1 second. This speed is almost 3x of our previous production vocoder.

On CPU machines, thanks to the highly-optimized ONNX runtime, the vocoder RTF is around 0.02 for 24khz sampling rate output.

With the performance improvement of HiFiNet, the end-to-end synthesis speed is about 2X as fast as our previous Neural TTS engine, which the audio quality is significantly improved at the same time.

What to expect next

Currently we support up to 24khz sampling rate on Azure Neural TTS service with 68 neural voice models available. In some highly sophisticated scenarios like audio dubbing, higher fidelity output like 48khz sampling rate makes a world of difference.

Below snippet from an audio spectrum shows the difference between 48hz sampling rate and 24khz. Audios with 48khz sampling rate get a higher frequency responding range which keeps more sophisticated details and nuances of the sound. Such high sampling rate creates challenges on both voice quality and inference speed.

24khz vs. 48khz: different frequency range

In our exploration, HiFiNet can handle both challenges well. According to our experiments, HiFiNet vocoder on 48khz sampling rate can be trained to achieve even higher quality with reasonable inference speed.

Hear the difference of the audio fidelity between the TTS output in 24khz and 48khs sampling rate, with a hi-fi speaker or headset.

Language	24khz HiFiNet	48khz HiFiNet
English (UK)
English (US)

The 48khz vocoder is now in private preview and can be applied to custom voices. Contact mstts [at] microsoft.com for details.

Create a custom voice with HiFiNet

The HiFiNet vocoder is also available in the Custom Neural Voice capability, enabling organizations to create a unique brand voice in multiple languages for their unique scenarios.

Learn more about the process for getting started with Custom Neural Voice.

Get started

With these updates, we’re excited to be powering more natural and intuitive voice experiences for global customers. Text to Speech has more than 70 standard voices in over 40 languages and locales in addition to our growing list of Neural TTS voices.

For more information:

Try the TTS demo
See our documentation
Check out our sample code

Re: Computer Vision for spatial analysis at the Edge

hussnain_ahmed — Tue, 27 Oct 2020 21:30:22 GMT

Is Tesla T4 GPU a recommendation or a mandatory requirement? Can we use Nvidia Jetson Series (TX2, Xavier NX etc.) as edge device for a single low frame rate video stream?

Re: Apps can now narrate what they see in the world as well as people do

boxinli — Tue, 20 Oct 2020 02:09:36 GMT

, if you are using the REST endpoint, please ensure your request URL uses the most recent (v3.1) version of the API. For example: https://{endpoint}/vision/v3.1/describe[?maxCandidates][&language]. If you are using the client library, please update to the latest version of the client library as well.

Re: Apps can now narrate what they see in the world as well as people do

HotCakeX — Fri, 16 Oct 2020 21:44:59 GMT

These are really amazing! I hope the marketing team make these new features and technologies known to people, because they deserve more attention.

Re: Apps can now narrate what they see in the world as well as people do

WahYuen — Thu, 15 Oct 2020 00:47:13 GMT

Hi,

Just wanted to clarify, are these new captioning abilities built into the existing APIs/REST Endpoints? (ie if we are already pulling out the captioning text, we should see the improvements?)

cheers

Apps can now narrate what they see in the world as well as people do

boxinli — Thu, 15 Oct 2020 00:21:12 GMT

How would you leverage technology capable of generating natural language image descriptions that are, in many cases, just as good or better than what a human could produce? What if that capability is just one cloud API call away? Would you create live scene captions for people who are blind or low vision to better understand the world around them, like Seeing AI?

With Azure Cognitive Services, you can now take advantage of state-of-the-art image captioning that has achieved human parity on captioning benchmarks thanks to advancements in the underlying AI model. Below are some examples showing how the improved model is more accurate than the old one:

Improved model: A trolley on a city street

Old model: a view of a city street

Improved model: A person using a microscope

Old model: A person sitting at a table using a laptop

Now, let us take a closer look at the technology and how to easily harness its power for your users.

Behind the Scenes of the Technology

The novel object captioning at scale (nocaps) challenge evaluates AI models on their abilities to generate image captions describing new objects that are not present in their training data. Microsoft’s Azure AI team pioneered the Visual Vocabulary (VIVO) pre-training technique that led to the industry first of surpassing human performance on the (nocaps) benchmark. Before we learn more about this innovation, we should understand Vision and Language Pre-training (VLP) first. It is a cross-modality (across vision and language) learning technique that uses large-scale image/sentence data pairs to train machine learning models capable of generating natural language captions for images. However, because visual concepts are learned from image/sentence pairs which are costly to obtain, it is difficult to train a broadly useful model with wide visual concept coverage. This is where VIVO pre-training comes in. It improves and extends VLP to allow rich visual concepts to be learned from easier to obtain image/word pairs (instead of sentence) to build a large-scale visual vocabulary. While natural language sentence generation is still trained with limited visual concepts, the resulting image caption is cleverly enriched by new objects from the large-scale visual vocabulary.

Figure 1: VIVO pre-training uses paired image-tag data to learn a rich visual vocabulary where image region features and tags of the same object are aligned. Fine-tuning is conducted on paired image-sentence data that only cover a limited number of objects (in blue). During inference, our model can generalize to describe novel objects (in yellow) that are learnt during VIVO pre-training.

Please see this MSR blog post to learn more about VIVO pre-training.

Try the Service in Your App

Imagine you would like to generate alternative text descriptions for images your users upload to your app. Azure Computer Vision Service with its much improved “describe image” (image captioning) capability can help. Let us take it for a spin.

We will be using Python client library to invoke the service in this blog post. Try these links if you prefer a different language or invoking the REST API directly.

Prerequisites

Python
An Azure subscription - create one for free
Once you have your Azure subscription, create a Computer Vision resource:
- Subscription: Pick the subscription you would like to use. If you just created a new Azure subscription, it should be an option in the dropdown menu.
- Resource group: Pick an existing one or create a new one.
- Region: Pick the region you would like your resource to be in.
- Name: Give your resource a unique name.
- Pricing tier: You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.
- Then click on “Review + create” to review your choices and click on “Create” again to deploy the resource

Once your resource is deployed, click “Go to resource.”

Click on “Keys and Endpoint” to get your subscription key and endpoint. You will be needing these for the code sample below.

Install the client

You can install the client library with:

pip install --upgrade azure-cognitiveservices-vision-computervision

Create and run the sample

Copy the following code into a text editor.
Optionally, replace the value of remote_image_url with the URL of a different image for which to generate caption.
Also, optionally, set useRemoteImage to FALSE and set local_image_path to the path of a local image for which to generate caption.
Save the code as a file with an .py extension. For example, describe-image.py.
Open a command prompt window.
At the prompt, use the python command to run the sample. For example, python describe-image.py.

import sys from azure.cognitiveservices.vision.computervision import ComputerVisionClient from msrest.authentication import CognitiveServicesCredentials # Best practice is to read this key from secure storage, # for this example we'll embed it in the code. subscription_key = "<your subscription key here>" endpoint = "<your endpoint here>" # Create the computer vision client computervision_client = ComputerVisionClient( endpoint, CognitiveServicesCredentials(subscription_key)) # Set to False if you want to use local image instead useRemoteImage = True if (useRemoteImage): # Get caption for a remote image, change to your own image URL as appropriate remote_image_url = "https://raw.githubusercontent.com/Azure-Samples/cognitive-services-sample-data-files/master/ComputerVision/Images/house.jpg" description_results = computervision_client.describe_image( remote_image_url) else: # Get caption for a local image, change to your own local image path as appropriate local_image_path = "<replace with local image path>" with open(local_image_path, "rb") as image: description_results = computervision_client.describe_image_in_stream( image) # Get the first caption (description) from the response if (len(description_results.captions) == 0): image_caption = "No description detected." else: image_caption = description_results.captions[0].text print("Description of image:", image_caption)

What’s that? Microsoft AI system describes images as well as people do

Learn more about other Computer Vision capabilities.