Azure AI articles

Localize your website with Microsoft Translator

Krishna_Doss — Thu, 22 Apr 2021 23:57:02 GMT

Web Localization and Ecommerce

Using Microsoft Azure Translator service, you can localize your website in a cost-effective way. With the advent of the internet, the world has become a much smaller place. Loads of information are stored and transmitted between cultures and countries, giving us all the ability to learn and grow from each other. Powered by advanced deep learning, Microsoft Azure Translator delivers fast and high-quality neural machine-based language translations, empowering you to break through language barriers and take advantage of all these powerful vehicles of knowledge and data transfer.

Research shows that 40% of internet users will never buy from websites in a foreign language[1]. Machine translation from Azure, supporting over 90 languages and dialects, helps you go to market faster and reach buyers in their native languages by localizing your web assets: from your marketing pages to user-generated content, and everything in-between.

Up to 95% of the online content that companies generate is available in only one language. This is because localizing websites, especially beyond the home page, is cost prohibitive outside of the top few markets. As a result, localized content seldom extends one or two clicks beyond a home page. However, with machine translation from Azure Translator Service, content that wouldn’t otherwise be localized can be, and now most of your content can reach customers and partners worldwide.

How to localize your website in a cost-effective way?

The first step is to understand the nature of your website content and classify them. It is critical as each of them needs different levels of localization. There are four types of content: a) static and dynamic, b) generated by you and posted by customer, c) sensitive like ‘Terms of Use’, d) part of UX elements.

Static content like about the organization, product or service description, user guides, terms of use, etc. can be translated once (or less frequently) offline into all required target languages. Translation results could be cached and served from your webserver. This could substantially reduce the cost of translation. Machine translation models which powers Azure Translator service are regularly updated to improve quality. Hence consider refreshing the translations once a quarter if not every month.

User generated content like customer reviews, information requests, etc. are dynamic in nature, not all of them requiring translations, and to be translated on need basis only. You could plan for an UX element in the webpage which could initiate translation on need basis. Target language for translation could be identified based on user browser language. Likewise, responses to customer could be translated back into the language of original request or comment.

Sensitive content like terms of use, company policies, are recommended to do a human review post-machine translation.

Text in UX elements of the webpage like menu, labels in forms, etc. are typically one or two words and have restricted space. Hence recommended to do a UX testing post translation for fit and finish. If necessitates look for alternate translation or human review.

Due to the speed and cost-effective nature that Azure Translator Service provides, you can easily test which localization option is optimal for your business and your users. For example, you may only have the budget to localize in dozens of languages and measure customer traffic in multiple markets in parallel. Using your existing web analytics, you will be able to decide where to invest in human translation in terms of markets, languages, or pages. For example, if the machine translated information passes a defined page view threshold, your system may trigger a human review of that content. In addition, you will still be able to maintain machine translation for other areas, to maintain reach.

By combining pure machine translation and paid translation resources, you can select different quality levels for the translations based on your business needs.

How to use Azure Translator service to translate static content

Pre-requisite:

Create an Azure subscription
Once you have an Azure subscription, create a Translator resource in the Azure portal.
Once Translator resource it created, go to the resource, and select ‘Keys and Endpoint’ which is used to connect your application to the Translator service.

Translating static webpage content:

Below code sample shows how to translate an element in the webpage. You could use it and iterate for each element in your webpage requiring translation.

import os, requests, uuid, json subscription_key = "YOUR_SUBSCRIPTION_KEY" endpoint = "https://api.cognitive.microsofttranslator.com" path = '/translate' constructed_url = endpoint + path params = { 'api-version': '3.0', 'to': ['de'], # target language 'textType': 'html' } headers = { 'Ocp-Apim-Subscription-Key': subscription_key, 'Content-type': 'application/json', 'X-ClientTraceId': str(uuid.uuid4()) } # You can pass more than one object in body. body = [{ "text": "The samples on this page use hard-coded keys and endpoints for simplicity. \ Remember to remove the key from your code when you're done, and \ never post it publicly. For production, consider using a secure way of \ storing and accessing your credentials. See the Cognitive Services security article \ for more information." }] request = requests.post(constructed_url, params=params, headers=headers, json=body) response = request.json() print (response[0]['translations'][0]['text']) # shows how to access the translated text from response

Localization is just a fraction of the things that you can do with Translator, so don't let the learning stop here. Check out recent new Translator features, additional doc links to dive deeper, and join the Translator Ask Microsoft Anything session on 4/27.

Get started:

Sign up for Azure trial
Join Translator engineering team on Ask Microsoft Anything on 4/27
Learn about Document Translation (Preview)
Create a language translator application with Unity and Azure Cognitive Services
Translator documentation

[1] CSA Research – Can’t Read, Won’t Buy – B2C Analyzing Consumer Language Preferences and Behaviors in 29 Countries https://insights.csa-research.com/reportaction/305013126/Marketing

Big data preparation in Azure Machine Learning – powered by Azure Synapse Analytics

Xun_Wang — Tue, 20 Apr 2021 16:00:00 GMT

Many customers who embark on a machine learning journey deal with big data, and need the power of distributed data processing engines to prepare their data for ML. By offering Apache Spark® (powered by Azure Synapse Analytics) in Azure Machine Learning (Azure ML), we are empowering customers to work on their end-to-end ML lifecycle including large-scale data preparation, featurization, model training, and deployment within Azure ML workspace without the need to switching between multiple tools for data preparation and model training. The ability to build the full ML lifecycle within Azure ML will reduce the time required for customers to iterate on a machine learning project which typically includes multiple rounds of data preparation and training.

With the preview of managed Apache Spark in Azure ML, customers can use Azure ML notebooks to connect to Spark pools in Azure Synapse Analytics, to do interactive data preparation using PySpark. Customers have the option to configure Spark sessions to quickly experiment and iterate on the data. Once ready, they can leverage Azure ML pipelines to automate their end-to-end ML workflow from data preparation to model deployment all in one environment, while maintaining their data and model lineage. Customers who prefer to train in the Spark environment can choose to install relevant libraries such as Spark MLlib, MMLSpark, etc. to complete their training on Spark pools.

Customers in preview will be able to benefit from the following key capabilities:

Reuse Spark pools from Azure Synapse workspace in Azure ML

Customers can leverage existing Spark pools from Azure Synapse Analytics (Azure Synapse) in Azure ML by just linking their Azure ML and Synapse workspaces via the Azure ML Studio, the Python SDK, or the ARM template. Customers just need to follow the widget in UI or leverage a few lines of code as described in the documentation here.

Once the workspaces are linked, customers can attach existing Spark pools into Azure ML workspace and can also register the supported linked services (data store sources).

Perform interactive data preparation via Spark magic from Azure ML notebooks

Customers can use Azure ML notebooks to start Spark sessions in PySpark via Spark Magic on attached Spark pools. Customers can register Azure ML datasets to load data from storage of choice. For data in Gen1 and Gen2, customers can use their own identities to authenticate access to data by leveraging AML datasets. The attached Spark pools can be used normally in Azure ML experiments, pipelines, and designer. More information on leveraging Spark Magic for data preparation on AML notebooks here

Productionize via Azure ML pipelines to orchestrate E2E ML steps including data preparation

After completing the interactive data preparation, customers can leverage Azure ML pipelines to automate data preparation on Apache Spark runtime as a step in the overall machine learning workflow. Customers can use the SynapseSparkStep for data preparation and choose either TabularDataset or FileDataset as input. Customers can also set up HDFSOutputDatasetConfig to generate the sparkstep output as a FileDataset, to be consumed by the following AzureML pipeline step. More details on How to use Apache Spark (powered by Azure Synapse) in your machine learning pipeline here.

Get started with big data preparation in Azure ML via Apache Spark powered by Azure Synapse

Get started by visiting our documentation and let us know your thoughts. We are committed to making the data preparation experience in Azure ML better for you!

Learn more about the Azure Machine Learning service and get started with a free trial.

Learn more about Azure Synapse big data preparation experience in Azure ML
Learn more about how to use Apache Spark in your machine learning pipelines
Learn more about Apache Spark
Learn more about Azure Synapse Analytics

Analyzing COVID Medical Papers with Azure and Text Analytics for Health

shwars — Tue, 13 Apr 2021 19:42:11 GMT

Automatic Paper Analysis

Automatic scientific paper analysis is fast growing area of studies, and due to recent improvements in NLP techniques is has been greatly improved in the recent years. In this post, we will show you how to derive specific insights from COVID papers, such as changes in medical treatment over time, or joint treatment strategies using several medications:

The main idea the approach I will describe in this post is to extract as much semi-structured information from text as possible, and then store it into some NoSQL database for further processing. Storing information in the database would allow us to make some very specific queries to answer some of the questions, as well as to provide visual exploration tool for medical expert for structured search and insight generation. The overall architecture of the proposed system is shown below:

We will use different Azure technologies to gain insights into the paper corpus, such as Text Analytics for Health, CosmosDB and PowerBI. Now let’s focus on individual parts of this diagram and discuss them in detail.

If you want to experiment with text analytics yourself - you will need an Azure Account. You can always get free trial if you do not have one. And you may also want to check out other AI technologies for developers

COVID Scientific Papers and CORD Dataset

The idea to apply NLP methods to scientific literature seems quite natural. First of all, scientific texts are already well-structured, they contain things like keywords, abstract, as well as well-defined terms. Thus, at the very beginning of COVID pandemic, a research challenge has been launched on Kaggle to analyze scientific papers on the subject. The dataset behind this competition is called CORD (publication), and it contains constantly updated corpus of everything that is published on topics related to COVID. Currently, it contains more than 400000 scientific papers, about half of them - with full text.

This dataset consists of the following parts:

Metadata file Metadata.csv contains most important information for all publications in one place. Each paper in this table has unique identifier cord_uid (which in fact does not happen to be completely unique, once you actually start working with the dataset). The information includes:
- Title of publication
- Journal
- Authors
- Abstract
- Data of publication
- doi
Full-text papers in document_parses directory, than contain structured text in JSON format, which greatly simplifies the analysis.
Pre-built Document Embeddings that maps cord_uids to float vectors that reflect some overall semantics of the paper.

In this post, we will focus on paper abstracts, because they contain the most important information from the paper. However, for full analysis of the dataset, it definitely makes sense to use the same approach on full texts as well.

What AI Can Do with Text?

In the recent years, there has been a huge progress in the field of Natural Language Processing, and very powerful neural network language models have been trained. In the area of NLP, the following tasks are typically considered:

Text classification / intent recognition: In this task, we need to classify a piece of text into a number of categories. This is a typical classification task. Sentiment Analysis; We need to return a number that shows how positive or negative the text is. This is a typical regression task. Named Entity Recognition (NER); In NER, we need to extract named entities from text, and determine their type. For example, we may be looking for names of medicines, or diagnoses. Another task similar to NER is keyword extraction.; Text summarization; Here we want to be able to produce a short version of the original text, or to select the most important pieces of text.; Question Answering; In this task, we are given a piece of text and a question, and our goal is to find the exact answer to this question from text.; Open-Domain Question Answering (ODQA); The main difference from previous task is that we are given a large corpus of text, and we need to find the answer to our question somewhere in the whole corpus.

In one of my previous posts, I have described how we can use ODQA approach to automatically find answers to specific COVID questions. However, this approach is not suitable for serious research.

To make some insights from text, NER seems to be the most prominent technique to use. If we can understand specific entities that are present in text, we could then perform semantically rich search in text that answers specific questions, as well as obtain data on co-occurrence of different entities, figuring out specific scenarios that interest us.

To train NER model, as well as any other neural language model, we need a reasonably large dataset that is properly marked up. Finding those datasets is often not an easy task, and producing them for new problem domain often requires initial human effort to mark up the data.

Pre-Trained Language Models

Luckily, modern transformer language models can be trained in semi-supervised manner using transfer learning. First, the base language model (for example, BERT) is trained on a large corpus of text first, and then can be specialized to a specific task such as classification or NER on a smaller dataset.

This transfer learning process can also contain additional step - further training of generic pre-trained model on a domain-specific dataset. For example, in the area of medical science Microsoft Research has pre-trained a model called PubMedBERT (publication), using texts from PubMed repository. This model can then be further adopted to different specific tasks, provided we have some specialized datasets available.

Text Analytics Cognitive Services

However, training a model requires a lot of skills and computational power, in addition to a dataset. Microsoft (as well as some other large cloud vendors) also makes some pre-trained models available through the REST API. Those services are called Cognitive Services, and one of those services for working with text is called Text Analytics. It can do the following:

Keyword extraction and NER for some common entity types, such as people, organizations, dates/times, etc.
Sentiment analysis
Language Detection
Entity Linking, by automatically adding internet links to some most common entities. This also performs disambiguation, for example Mars can refer to both the planet or a chocolate bar, and correct link would be used depending on the context.

For example, let’s have a look at one medical paper abstract analyzed by Text Analytics:

As you can see, some specific entities (for example, HCQ, which is short for hydroxychloroquine) are not recognized at all, while others are poorly categorized. Luckily, Microsoft provides special version of Text Analytics for Health.

Text Analytics for Health

Text Analytics for Health is a cognitive service that exposes pre-trained PubMedBert model with some additional capabilities. Here is the result of extracting entities from the same piece of text using Text Analytics for Health:

Currently, Text Analytics for Health is available as gated preview, meaning that you need to request access to use it in your specific scenario. This is done according to Ethical AI principles, to avoid irresponsible usage of this service for cases where human health depends on the result of this service. You can request access here.

To perform analysis, we can use recent version Text Analytics Python SDK, which we need to pip-install first:

pip install azure.ai.textanalytics==5.1.0b5

Note: We need to specify a version of SDK, because otherwise we can have current non-beta version installed, which lacks Text Analytics for Health functionality.

The service can analyze a bunch of text documents, up to 10 at a time. You can pass either a list of documents, or dictionary. Provided we have a text of abstract in txt variable, we can use the following code to analyze it:

poller = text_analytics_client.begin_analyze_healthcare_entities([txt]) res = list(poller.result()) print(res)

This results in the following object:

[AnalyzeHealthcareEntitiesResultItem(
  id=0, entities=[
     HealthcareEntity(text=2019, category=Time, subcategory=None, length=4, offset=20, confidence_score=0.85, data_sources=None, 
        related_entities={HealthcareEntity(text=coronavirus disease pandemic, category=Diagnosis, subcategory=None, length=28, offset=25, confidence_score=0.98, data_sources=None, related_entities={}): 'TimeOfCondition'}), 
     HealthcareEntity(text=coronavirus disease pandemic, category=Diagnosis, subcategory=None, length=28, offset=25, confidence_score=0.98, data_sources=None, related_entities={}), 
     HealthcareEntity(text=COVID-19, category=Diagnosis, subcategory=None, length=8, offset=55, confidence_score=1.0, 
        data_sources=[HealthcareEntityDataSource(entity_id=C5203670, name=UMLS), HealthcareEntityDataSource(entity_id=U07.1, name=ICD10CM), HealthcareEntityDataSource(entity_id=10084268, name=MDR), ...

As you can see, in addition to just the list of entities, we also get the following:

Enity Mapping of entities to standard medical ontologies, such as UMLS.
Relations between entities inside the text, such as TimeOfCondition, etc.
Negation, which indicated that an entity was used in negative context, for example COVID-19 diagnosis did not occur.

In addition to using Python SDK, you can also call Text Analytics using REST API directly. This is useful if you are using a programming language that does not have a corresponding SDK, or if you prefer to receive Text Analytics result in the JSON format for further storage or processing. In Python, this can be easily done using requests library:

uri = f"{endpoint}/text/analytics/v3.1-preview.3/entities/ health/jobs?model-version=v3.1-preview.4" headers = { "Ocp-Apim-Subscription-Key" : key } resp = requests.post(uri,headers=headers,data=doc) res = resp.json() if res['status'] == 'succeeded': result = t['results'] else: result = None

(We need to make sure to use the preview endpoint to have access to text analytics for health)

Resulting JSON file will look like this:

{"id": "jk62qn0z", "entities": [ {"offset": 24, "length": 28, "text": "coronavirus disease pandemic", "category": "Diagnosis", "confidenceScore": 0.98, "isNegated": false}, {"offset": 54, "length": 8, "text": "COVID-19", "category": "Diagnosis", "confidenceScore": 1.0, "isNegated": false, "links": [ {"dataSource": "UMLS", "id": "C5203670"}, {"dataSource": "ICD10CM", "id": "U07.1"}, ... ]}, "relations": [ {"relationType": "Abbreviation", "bidirectional": true, "source": "#/results/documents/2/entities/6", "target": "#/results/documents/2/entities/7"}, ...], }

Note: In production, you may want to incorporate some code that will retry the operation when an error is returned by the service. For more guidance on proper implementation of cognitive services REST clients, you can check source code of Azure Python SDK, or use Swagger to generate client code.

Using Cosmos DB to Store Analysis Result

Using Python code similar to the one above we can extract JSON entity/relation metadata for each paper abstract. This process takes quite some time for 400K papers, and to speed it up it can be parallelized using technologies such as Azure Batch or Azure Machine Learning. However, in my first experiment I just run the script on one VM in the cloud, and the data was ready in around 11 hours.

Having done this, we have now obtained a collection of papers, each having a number of entities and corresponding relations. This structure is inherently hierarchical, and the best way to store and process it would be to use NoSQL approach for data storage. In Azure, Cosmos DB is a universal database that can store and query semi-structured data like our JSON collection, thus it would make sense to upload all JSON files to Cosmos DB collection. This can be done using the following code:

coscli = azure.cosmos.CosmosClient(cosmos_uri, credential=cosmos_key) cosdb = coscli.get_database_client("CORD") cospapers = cosdb.get_container_client("Papers") for x in all_papers_json: cospapers.upsert_item(x)

Here, all_papers_json is a variable (or generator function) containing individual JSON documents for each paper. We also assume that you have created a Cosmos DB database called ‘CORD’, and obtained required credentials into cosmos_uri and cosmos_key variables.

After running this code, we will end up with the container Papers will all metadata. We can now work with this container in Azure Portal by going to Data Explorer:

Now we can use Cosmos DB SQL in order to query our collection. For example, here is how we can obtain the list of all medications found in the corpus:

-- unique medication names SELECT DISTINCT e.text FROM papers p JOIN e IN p.entities WHERE e.category='MedicationName'

Using SQL, we can formulate some very specific queries. Suppose, a medical specialist wants to find out all proposed dosages of a specific medication (say, hydroxychloroquine), and see all papers that mention those dosages. This can be done using the following query:

-- dosage of specific drug with paper titles SELECT p.title, r.source.text FROM papers p JOIN r IN p.relations WHERE r.relationType='DosageOfMedication' AND CONTAINS(r.target.text,'hydro')

You can execute this query interactively in Azure Portal, inside Cosmos DB Data Explorer. The result of the query looks like this:

[ { "title": "In Vitro Antiviral Activity and Projection of Optimized Dosing Design of Hydroxychloroquine for the Treatment of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)", "text": "400 mg" },{ "title": "In Vitro Antiviral Activity and Projection of Optimized Dosing Design of Hydroxychloroquine for the Treatment of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)", "text": "maintenance dose" },...]

A more difficult task would be to select all entities together with their corresponding ontology ID. This would be extremely useful, because eventually we want to be able to refer to a specific entity (hydroxychloroquine) regardless or the way it was mentioned in the paper (for example, HCQ also refers to the same medication). We will use UMLS as our main ontology.

--- get entities with UMLS IDs SELECT e.category, e.text, ARRAY (SELECT VALUE l.id FROM l IN e.links WHERE l.dataSource='UMLS')[0] AS umls_id FROM papers p JOIN e IN p.entities

Creating Interactive Dashboards

While being able to use SQL query to obtain an answer to some specific question, like medication dosages, seems like a very useful tool - it is not convenient for non-IT professionals, who do not have high level of SQL mastery. To make the collection of metadata accessible to medical professionals, we can use PowerBI tool to create an interactive dashboard for entity/relation exploration.

In the example above, you can see a dashboard of different entities. One can select desired entity type on the left (eg. Medication Name in our case), and observe all entities of this type on the right, together with their count. You can also see associated UMLS IDs in the table, and from the example above once can notice that several entities can refer to the same ontology ID (hydroxychloroquine and HCQ).

To make this dashboard, we need to use PowerBI Desktop. First we need to import Cosmos DB data - the tools support direct import of data from Azure.

Then we provide SQL query to get all entities with the corresponding UMLS IDs - the one we have shown above - and one more query to display all unique categories. Then we drag those two tables to the PowerBI canvas to get the dashboard shown above. The tool automatically understands that two tables are linked by one field named category, and supports the functionality to filter second table based on the selection in the first one.

Similarly, we can create a tool to view relations:

From this tool, we can make queries similar to the one we have made above in SQL, to determine dosages of a specific medications. To do it, we need to select DosageOfMedication relation type in the left table, and then filter the right table by the medication we want. It is also possible to create further drill-down tables to display specific papers that mention selected dosages of medication, making this tool a useful research instrument for medical scientist.

Getting Automatic Insights

The most interesting part of the story, however, is to draw some automatic insights from the text, such as the change in medical treatment strategy over time. To do this, we need to write some more code in Python to do proper data analysis. The most convenient way to do that is to use Notebooks embedded into Cosmos DB:

Those notebooks support embedded SQL queries, thus we are able to execute SQL query, and then get the results into Pandas DataFrame, which is Python-native way to explore data:

%%sql --database CORD --container Papers --output meds SELECT e.text, e.isNegated, p.title, p.publish_time, ARRAY (SELECT VALUE l.id FROM l IN e.links WHERE l.dataSource='UMLS')[0] AS umls_id FROM papers p JOIN e IN p.entities WHERE e.category = 'MedicationName'

 Here we end up with meds DataFrame, containing names of medicines, together with corresponding paper titles and publishing date. We can further group by ontology ID to get frequencies of mentions for different medications:
unimeds = meds.groupby('umls_id') \
              .agg({'text' : lambda x : ','.join(x), 
                    'title' : 'count', 
                    'isNegated' : 'sum'})
unimeds['negativity'] = unimeds['isNegated'] / unimeds['title']
unimeds['name'] = unimeds['text'] \
                  .apply(lambda x: x if ',' not in x 
                                     else x[:x.find(',')])
unimeds.sort_values('title',ascending=False).drop('text',axis=1)

This gives us the following table:

umls_id	title	isNegated	negativity	name
C0020336	4846	191	0.039414	hydroxychloroquine
C0008269	1870	38	0.020321	chloroquine
C1609165	1793	94	0.052426	Tocilizumab
C4726677	1625	24	0.014769	remdesivir
C0052796	1201	84	0.069942	azithromycin
…	…	…	…	…
C0067874	1	0	0.000000	1-butanethiol

From this table, we can select the top-15 most frequently mentioned medications:

top = { x[0] : x[1]['name'] for i,x in zip(range(15), unimeds.sort_values('title',ascending=False).iterrows()) }

To see how frequency of mentions for medications changed over time, we can average out the number of mentions for each month:

# First, get table with only top medications imeds = meds[meds['umls_id'].apply(lambda x: x in top.keys())].copy() imeds['name'] = imeds['umls_id'].apply(lambda x: top[x]) # Create a computable field with month imeds['month'] = imeds['publish_time'].astype('datetime64[M]') # Group by month medhist = imeds.groupby(['month','name']) \ .agg({'text' : 'count', 'isNegated' : [positive_count,negative_count] })

This gives us the DataFrame that contains number of positive and negative mentions of medications for each month. From there, we can plot corresponding graphs using matplotlib:
medh = medhist.reset_index()
fig,ax = plt.subplots(5,3)
for i,n in enumerate(top.keys()):
    medh[medh['name']==top[n]] \
    .set_index('month')['isNegated'] \
    .plot(title=top[n],ax=ax[i//3,i%3])
fig.tight_layout()

Visualizing Terms Co-Occurrence

Another interesting insight is to observe which terms occur frequently together. To visualize such dependencies, there are two types of diagrams:

Sankey diagram allows us to investigate relations between two types of terms, eg. diagnosis and treatment
Chord diagram helps to visualize co-occurrence of terms of the same type (eg. which medications are mentioned together)

To plot both diagrams, we need to compute co-occurrence matrix, which in the row i and column j contains number of co-occurrences of terms i and j in the same abstract (one can notice that this matrix is symmetric). The way we compute it is to manually select relatively small number of terms for our ontology, grouping some terms together if needed:

treatment_ontology = { 'C0042196': ('vaccination',1), 'C0199176': ('prevention',2), 'C0042210': ('vaccines',1), ... } diagnosis_ontology = { 'C5203670': ('COVID-19',0), 'C3714514': ('infection',1), 'C0011065': ('death',2), 'C0042769': ('viral infections',1), 'C1175175': ('SARS',3), 'C0009450': ('infectious disease',1), ...}

Then we define a function to compute co-occurrence matrix for two categories specified by those ontology dictionaries:
def get_matrix(cat1, cat2):
    d1 = {i:j[1] for i,j in cat1.items()}
    d2 = {i:j[1] for i,j in cat2.items()}
    s1 = set(cat1.keys())
    s2 = set(cat2.keys())
    a = np.zeros((len(cat1),len(cat2)))
    for i in all_papers:
        ent = get_entities(i)
        for j in ent & s1:
            for k in ent & s2 :
                a[d1[j],d2[k]] += 1
    return a

 Here get_entities function returns the list of UMLS IDs for all entities mentioned in the paper, and all_papers is the generator that returns the complete list of paper abstracts metadata.

To actually plot the Sankey diagram, we can use Plotly graphics library. This process is well described here, so I will not go into further details. Here are the results:

Plotting a chord diagram cannot be easily done with Plotly, but can be done with a different library - Chord. The main idea remains the same - we build co-occurrence matrix using the same function described above, passing the same ontology twice, and then pass this matrix to Chord:

def chord(cat): matrix = get_matrix(cat,cat) np.fill_diagonal(matrix,0) names = cat.keys() Chord(matrix.tolist(), names, font_size = "11px").to_html()

The results of chord diagrams for treatment types and medications are below:


Treatment types	Medications

Diagram on the right shows which medications are mentioned together (in the same abstract). We can see that well-known combinations, such as hydroxychloroquine + azitromycin, are clearly visible.

Conclusion

In this post, we have described the architecture of a proof-of-concept system for knowledge extraction from large corpora of medical texts. We use Text Analytics for Health to perform the main task of extracting entities and relations from text, and then a number of Azure services together to build a query took for medical scientist and to extract some visual insights. This post is quite conceptual at the moment, and the system can be further improved by providing more detailed drill-down functionality in PowerBI module, as well as doing more data exploration on extracted entity/relation collection. It would also be interesting to switch to processing full-text articles as well, in which case we need to think about slightly different criteria for co-occurrence of terms (eg. in the same paragraph vs. the same paper).

The same approach can be applied in other scientific areas, but we would need to be prepared to train a custom neural network model to perform entity extraction. This task has been briefly outlined above (when we talked about the use of BERT), and I will try to focus on it in one of my next posts. Meanwhile, feel free to reach out to me if you are doing similar research, or have any specific questions on the code and/or methodology.

Learn about Bot Framework Composer’s new authoring experience and deploy your bot to a telephone

KelvinChen — Wed, 14 Apr 2021 16:41:33 GMT

Customer expectations continue to increase, looking for immediate response and rapid issue resolution, across multiple channels 24/7. Nowhere is this more apparent than the contact center, with this landscape is driving the need for efficiencies, such as reducing call handling times and increasing call deflection rates – all whilst aiming to deliver a personalized and tailored customer experience.

To help respond to this need, we announced the public preview of the telephony channel for Azure Bot Service in February 2021, expanding the already significant number of touch points offered by the service, to include this increasingly critical method of communication.

Built on state-of-the-art speech services

The new telephony channel, combined with our Bot Framework developer platform, makes it easy to rapidly build always-available virtual assistants, or IVR assistants, that provide natural language intent-based call handling and the ability to handle advanced conversation flows, such as context switching and responding to follow up questions and still meeting the goal of reducing operational costs for enterprises.

This new capability combines several of our Azure and AI services, including our state-of-the-art Cognitive Speech Service, enabling fluid, natural-sounding speech that matches the patterns and intonation of human voices through Azure Text-to-Speech neural voices, with Azure Communications Services powering various calling capabilities. The channel also provides support for full duplex conversations and streaming audio over PSTN, support for DTMF, barge-in (allowing a caller to interrupt the virtual assistant) and more. Follow our roadmap and try out one of our samples on the Telephony channel GitHub repository.

Improving our Conversational AI SDK and tools for speech experiences

To compliment the introduction of the telephony channel and ensure our customers can create industry leading experiences, we have added new features to Bot Framework Composer, an open-source conversational authoring tool, featuring a visual canvas, built on top of the Bot Framework SDK, allowing you to extend and customize the conversation with code and pre-built components. Updates to Composer to support speech experiences include,

The ability to add tailored speech responses in seconds, either for a voice only or multi-modal (text and speech) agent.
Addition of global application settings for your bot, allowing you to set a consistent voice font to be used on speech enabled channels, including taking care of setting the required base SSML tags.
Authoring UI helpers that allow you to add additional common SSML (Speech Synthesis Markup Language) tags to control the intonation, speed and even the style of the voice used, including new styles available for our neural voice fonts, such as a dedicated Customer Service style.

Comprehensive Contact Center solution through Dynamics 365

Microsoft announced the expansion of Microsoft Dynamics 365 Customer Service omnichannel capabilities to include a new voice channel, that is built on this telephony channel infrastructure. With native voice, businesses receive seamless, end-to-end experiences within a single solution, ensuring consistent, personalized, and connected support across all channels of engagement. This new voice channel for Customer Service enables an all-in-one customer service solution without fragmentation or manual data integration required, and enables a faster time to value. Learn more here.

Get started building for telephony!

Sign up for Azure trial
Join us on live stream of AI Show on 4/16 11AM PDT
Sign up for conversational AI Ask Microsoft Anything (4/28)
To get started developing a virtual agent, that you can surface via the new telephony channel today, download Bot Framework Composer
Read more about the telephony channel preview, including documentation and samples, visit the Bot Framework telephony channel GitHub repository

Introducing Multivariate Anomaly Detection

Tony_Xing — Mon, 12 Apr 2021 15:11:56 GMT

Microsoft partners and customers have been building metrics monitoring solutions for AIOps and predictive maintenance, by leveraging the easy-to-use time-series anomaly detection Cognitive Service: Anomaly Detector. Because of its ability to analyze time-series individually, Anomaly Detector is benefiting the industry with its simplicity and scalability.

What's new

We are pleased to announce the new multi-variate capability of Anomaly Detector. The new multivariate anomaly detection APIs in Anomaly Detector further enable developers to easily integrate advanced AI of detecting anomalies from groups of metrics into their applications without the need for machine learning knowledge or labeled data. Dependencies and inter-correlations between different signals are now counted as key factors. The new feature protects your mission-critical systems and physical assets, such as software applications, servers, factory machines, spacecraft, or even your business, from failures with a holistic view.

Imagine 20 sensors from an auto engine generating 20 different signals, e.g., vibration, temperature, etc. The readings of those signals individually may not tell you much on system-level issues, but together, could represent the health of the engine. When the synergy of those signals turns odd, the multivariate anomaly detection feature can sense the anomaly like a seasoned floor expert. Moreover, the AI models are trained and customized for your data such that it understands your business. With the new APIs in Anomaly Detector, developers can now easily integrate the multivariate time-series anomaly detection capabilities as well as the interpretability of the anomalies into predictive maintenance solutions, or AIOps monitoring solutions for complex enterprise software, or business intelligence tools.

Customer love

“Medical device production demands unprecedented precision. For this reason, the Siemens Healthineers team uses Multivariate Anomaly Detector (MVAD) in medical device stress tests during the final inspection in the production. We found MVAD easy to use and work almost out of the box with promising performance. With the ready-to-use model, we don't need to develop a custom AD model, which ensures a short time to market. We plan to expand this technology also to other use cases. It is made easy due to good integration into our ML platform and processes.” - Dr. Jens Fürst, Head Digitalization and Automation at Siemens Healthineers

To better understand the health and condition of the aircraft and foresee and fix potential problems before they occur, Airbus deployed Anomaly Detector, part of Cognitive Services, to gather and analyze the telemetry data. It began as a proof of concept of the aircraft-monitoring application by loading telemetry data from multiple flights for analysis and model training. “Early tests have shown that for many cases, the out-of-the-box solution works beautifully, which helps us deploy our solutions faster. I would say that we save up to three months on development for our smaller use cases with Anomaly Detector.”
Marcel Rummens: Product Owner of Internal AI Platform, Airbus

AI horsepower

Time-series anomaly detection is an important research topic in data mining and has a wide range of applications in the industry. Efficient and accurate anomaly detection helps companies to monitor their key metrics continuously and alert for potential incidents on time. In many real-world applications like predictive maintenance and SpaceOps, multiple time-series metrics are collected to reflect the health status of a system. Univariate time-series anomaly detection algorithms can find anomalies for a single metric. However, it could be problematic in deciding whether the whole system is running normally. For example, sudden changes of a certain metric do not necessarily mean failures of the system. As shown in Figure 1, there are obvious boosts in the volume of TIMESERIES RECEIVED and DATA RECEIVED ON FLINK in the green segment, but the system is still in a healthy state as these two features share a consistent tendency. However, in the red segment, GC shows an inconsistent pattern with other metrics, indicating a problem in garbage collection. Consequently, it is essential to take the correlations between different time series into consideration in a multivariate time-series anomaly detection system.Fig.1

In this newly introduced feature, we productized a novel framework — MTAD-GAT (Multivariate Time-series Anomaly Detection via Graph Attention Network), to tackle the limitations of previous solutions. Our method considers each univariate time-series as an individual feature and tries to model the correlations between different features explicitly, while the temporal dependencies within each time-series are modeled at the same time. The key ingredients in our model are two graph attention layers, namely the feature-oriented graph attention layer and the time-oriented graph attention layer. The feature-oriented graph attention layer captures the causal relationships between multiple features, and the time-oriented graph attention layer underlines the dependencies along the temporal dimension. In addition, we jointly train a forecasting-based model and a reconstruction-based model for better representations of time-series data. The two models can be optimized simultaneously by a joint objective function.

The magic behind the scenes can be summarized as follows:

A novel framework to solve the multivariate time-series anomaly detection problem in a self-supervised manner. Our model shows superior performances on two public datasets and establishes state-of-the-art scores in the literature.
For the first time, we leverage two parallel graph attention (GAT) layers to learn the relationships between different time-series and timestamps dynamically. Especially, our model captures the correlations between different time-series successfully without any prior knowledge.
We integrate the advantages of both forecasting-based and reconstruction-based models by introducing a joint optimization target. The forecasting-based model focuses on single-timestamp prediction, while the reconstruction-based model learns a latent representation of the entire time-series.
Our network has good interpretability. We analyze the attention scores of multiple time-series learned by the graph attention layers, and the results correspond reasonably well to human intuition. We also show its capability of anomaly diagnosis.

Multivariate anomaly detection API overview

This new feature has a different workflow compared with the existing univariate feature. There are two phases to obtain the detection results, the training phase, and the inference phase. In the training phase, you need to provide some historical data to let the model learn past patterns. Then in the inference phase, you can call the inference API to acquire detection results of multivariate time-series in a given range.

APIs	Functionality
/multivariate/models	Create and train model using training data
/multivariate/models/{modelid}	Get model info including training status and parameters used in the model
multivariate/models[?$skip][&$top]	List models of a subscription
/multivariate/models/{modelid}/detect	Submit inference task with user's data, this is async
/multivariate/results/{resultid}	Get anomalies + root causes (the contribution scores of each variate for each incident)
multivariate/models/{modelId}	Delete an existing multivariate model according to the modelId
multivariate/models/{modelId}/export	Export Multivariate Anomaly Detection Model as Zip file

Get started!

Learning more from our documentation
QuickStarts: C#, Python, JavaScript, Java
Artificial Intelligence for developers

Supercharge Azure ML code development with new VS Code integration

Sid_Unnithan — Thu, 08 Apr 2021 15:25:26 GMT

This post is co-authored by Abe Omorogbe, Program Manager, Azure Machine Learning.

The Azure Machine Learning (Azure ML) team is excited to announce the release of an enhanced developer experience for ‘compute instance’ and ‘notebooks’ users, through a VS Code integration in the Azure ML Studio! It is now easier than ever to work directly on your Azure ML compute instances from within Visual Studio Code, , and with full access to a remote terminal, your favorite VS Code extensions, Git source control UI, and a debugger.

Bringing VS Code to Azure Machine Learning

The Azure Machine Learning and VS Code teams have been working in collaboration over the past couple of months to better understand user workflows for authoring, editing, and managing code files. The demand for VS Code became clear after speaking to a wide variety of users tasked with managing larger projects and operationalizing their models. Users were eager to continue working on their Azure ML compute resources and retain the development context initially defined through the Studio UI.

The first step to enabling a better editing experience for users was to evaluate what was currently used in VS Code. Users were familiar with extensions such as Remote-SSH and , the former used to connect to their remote compute and the latter to author notebook files. The advantage of using Jupyter, JupyterLab, or Azure ML notebooks was that they could be used for all compute instance types without requiring any additional configuration or networking changes.

To enable users to work against their compute instances without requiring SSH or additional networking changes, the Azure ML and VS Code teams built a Notebook-specific compute instance connect experience. The Azure ML extension was responsible for facilitating the connection between VS Code – Jupyter and the compute instance, taking care of authenticating on the user’s behalf. After a month or so of releasing this capability, it was clear that users were excited about connectivity without SSH and being able to work from directly within VS Code. However, working in the editor implied expectations around being able to use other VS code features such as the remote terminal, debugger, and language server. Users expressed their frustration with being limited to working in a single Notebook file, being unable to view files on the remote server, and not being able to use their preferred extensions.

VS Code Integration: Features

Learning from prior releases and talking to users led the Azure ML and VS code teams, to build a complete VS Code experience for compute instances without using SSH. Getting started with this experience is trivial – entry points have been integrated within the Azure ML Studio in both the Compute Instance and Notebooks tabs.

Studio UI Compute Entry Point

Studio UI Notebooks Entry Point

Through this VS Code integration customers will now have access to the following features and benefits:

Full integration with Azure ML file share and notebooks: All file operations in VS Code are fully synced with the Azure ML Studio. For example, if a user drags and drops files from their local machine into VS Code connected to Azure ML, all files will be synced and appear in the Azure ML Studio.
Git UI Experiences: Fully manage Git repos in Azure ML with the rich VS Code source control UI.
Notebook Editor: Seamlessly click out from the Azure ML notebooks and continue to work on notebooks in the new native VS code editor.
Debugging: Use the native debugging in VS Code to debug any training script before submitting it to an Azure ML cluster for batch training.
VS Code Terminal: Work in the VS Code terminal that is fully connected to the compute instance.
VS Code Extension Support: All VS Code extensions are fully supported in VS Code connected to the compute instance.
Enterprise Support: Work with VS Code securely in private endpoints without additional, complicated SSH and networking configuration. AAD credentials and RBAC are used to establish a secure connection to VNET/private link enabled Azure ML workspaces.

VS Code Integration: How it Works

Clicking out to VS Code will launch a desktop VS Code session which initiates a secondary remote connection to the target compute. Within the remote connection window, the Azure ML extension creates a WebSocket connection between your local VS Code client and the remote compute instance.

The connected window now provides you with:

Access to the mounted file share, with consistent syncing between what is seen in Jupyter* and the Azure ML Notebooks experience.
Access to the machine’s local SSD in case you would like to clone and manage repos outside of the shared file share.
The ability to manage repositories through the source control UI.
The ability to create, interact and debug running applications.
A remote terminal for executing commands directly against the remote compute.

Below is a high-level overview of the remote connection

Remote Connection Architecture Diagram (High-Level)

This new connect capability and direct integration in the Azure ML Studio creates a better-together experience between Azure ML and VS Code! When working on your machine learning projects you can get started with a notebook in the Azure ML Studio for early data prep and exploratory work, when you’re ready to start fleshing out the rest of your project, work on multiple file types, and use more advanced editing capabilities and VS Code extension, you can seamlessly transition over to working in VS Code. The retained context and file share usage enables you to move bi-directionally (from notebooks to VS Code and vice-versa) without requiring additional work.

Getting Started

You can initiate the connection to VS Code directly from the Studio UI through either the Compute Instance or Notebook pages. Alternatively, there are routes starting directly within VS Code if you would prefer. Given you have the Azure Machine Learning extension installed, you can find the compute instance in the tree view and right-click on it to connect. You can also invoke the command “Azure ML: Connect to compute instance” and follow the prompts to initiate the connection.

Azure ML extension command

Azure ML extension tree view context menu option

For more details on how you can get started with this experience, please take a look at our public documentation.

Both the Azure ML and VS Code extension teams are always looking for feedback on our current experiences and what we should work on next. If there is anything you would like us to prioritize, please feel free to suggest so via our GitHub repo; if you would like to provide more general feedback, please fill out our survey.

Eleven more languages are generally available for Azure Neural Text-to-Speech

GarfieldHe — Wed, 31 Mar 2021 15:21:04 GMT

This post is co-authored with Lihui Wang, Gang Wang, Xinfeng Chen, Qinying Liao, Garfield He and Sheng Zhao

Six new languages

12 voices from 6 brand-new languages, with one male and one female voice in each language are available now: Nia in ‘cy-GB’ Welsh (United Kingdom), Aled in ‘cy-GB’ Welsh (United Kingdom), Rosa in ‘en-PH’ English (Philippines), James in ‘en-PH’ English (Philippines), Charline in ‘fr-BE’ French (Belgium), Gerard in ‘fr-BE’ French (Belgium), Dena in ‘’nl-BE Dutch (Belgium), Arnaud in ‘nl-BE’ Dutch (Belgium), Polina in ‘uk-UA’ Ukranian (Ukraine), Ostap in ‘uk-UA’ Ukranian (Ukraine), Uzma in ‘ur-PK’ Urdu (Pakistan), and Asad in ‘ur-PK’ Urdu (Pakistan).

Hear the samples below or try them with your own text in our product demo on Azure.

Locale code	Language	Gender	Voice name	Audio sample
cy-GB	Welsh (UK)	Female	cy-GB-NiaNeural	Mae'r ysgol ar agor drwy'r wythnos.
cy-GB	Welsh (UK)	Male	cy-GB-AledNeural	Mae Bangor 8 milltir o Gaernarfon.
en-PH	English (Philippines)	Female	en-PH-RosaNeural	I need to buy a mineral water.
en-PH	English (Philippines)	Male	en-PH-JamesNeural	Let's meet tomorrow at 6 pm.
fr-BE	French (Belgium)	Female	fr-BE-CharlineNeural	On se voit pour dîner demain ?
fr-BE	French (Belgium)	Male	fr-BE-GerardNeural	Il existe 2 manières de participer.
nl-BE	Dutch (Belgium)	Female	nl-BE-DenaNeural	Hij is al urenlang online.
nl-BE	Dutch (Belgium)	Male	nl-BE-ArnaudNeural	Ik vond vele kabouters in hun tuin.
uk-UA	Ukrainian (Ukraine)	Female	uk-UA-PolinaNeural	У Києві завершили реставрацію Андріївської церкви.
uk-UA	Ukrainian (Ukraine)	Male	uk-UA-OstapNeural	Загалом було оновлено 4 395 км доріг.
ur-PK	Urdu (Pakistan)	Female	ur-PK-UzmaNeural	واہ! کیا ہی خوبصورت نظارہ ہے۔
ur-PK	Urdu (Pakistan)	Male	ur-PK-AsadNeural	سورج گرہن پاکستانی وقت کے مطابق شام 6 بج کر 34 منٹ پر شروع ہو گا۔

With this update, Azure TTS now supports 60 languages in total. Check out the figure below for more details or see the full language list.

Five preview languages now GA

Last November, we released 5 languages in preview with 10 voices for European locales. Now these languages become generally available in all Neural TTS regions/datacenters. Azure TTS now has full support for all 24 European languages.

Locale code	Language	Gender	Voice name	Audio samples
et-EE	Estonian (Estonia)	Female	et-EE-AnuNeural	Pese voodipesu kord nädalas või vähemalt kord kahe nädala järel ning ära unusta pesta ka kardinaid.
et-EE	Estonian (Estonia)	Male	et-EE- KertNeural	Ametlikku meetodit sellise pettuse avastamiseks ei olegi olemas.
ga-IE	Irish (Ireland)	Female	ga-IE- OrlaNeural	Tá an scoil sa mbaile ar oscailt arís inniu.
ga-IE	Irish (Ireland)	Male	ga-IE- ColmNeural	Ritheadh próiseas comhairliúcháin faoin scéal sa bhfómhar.
lt-LT	Lithuanian (Lithuania)	Female	lt-LT- OnaNeural	Derinti motinystę ir kūrybą išmokau jau po pirmojo vaiko gimimo.
lt-LT	Lithuanian (Lithuania)	Male	lt-LT- LeonasNeural	Aišku, anksčiau ar vėliau paaiškės tos priežastys.
lv-LV	Latvian (Latvia)	Female	lv-LV-EveritaNeural	Daži tumšās šokolādes gabaliņi dienā ir gandrīz būtiska uztura sastāvdaļa.
lv-LV	Latvian (Latvia)	Male	lv-LV- NilsNeural	Aizvadīto gadu uzņēmums noslēdzis ar 6,3 miljonu eiro zaudējumiem.
mt-MT	Maltese (Malta)	Female	mt-MT-GraceNeural	Fid-diskors tiegħu, is-Segretarju Parlamentari fakkar li dan il-Gvern daħħal numru ta’ liġijiet u inizjattivi li jħarsu lill-annimali.
mt-MT	Maltese (Malta)	Male	mt-MT- JosephNeural	Anki tfajjel tal-primarja jaf li l-popolazzjoni tikber fejn hemm il-prosperità.

How to integrate with the new voices/languages

Azure TTS now covers more languages of the world. Applications using Azure TTS can be easily updated to support coverage of additional countries. All the voices are available in the same API and SDK. Developers can just edit the voice and locale list in their applications to use these new voices without code logic modifications.

For instance, Microsoft Teams auto attendants lets people call your organization and navigate a menu system to speak to the right department, call queue, person, or an operator. It uses Azure TTS to render customized prompts as a call response. To better localize audio prompts for different countries, Teams has been integrated with the new TTS languages to serve more customers around the world.

Want more languages or voices?

If you find that the language which you are looking for is not supported by Azure TTS, reach out to your sales representative, or file a support ticket on Azure. We'd be happy to engage and discuss how to support the languages you need. You can also customize and create a brand voice with your speech data for your apps using the Custom Neural Voice feature.

Tell us your experiences!

By offering more voices across more languages and locales, we anticipate developers across the world will be able to build applications that change experiences for millions. Whether you are building a voice-enabled chatbot or IoT device, an IVR solution, adding read-aloud features to your app, converting e-books to audio books, or even adding Speech to a translation app, you can make all these experiences natural sounding and fun with Neural TTS.

Let us know how you are using or plan to use Neural TTS voices in this form. If you prefer, you can also contact us at mstts [at] microsoft.com. We look forward to hearing about your experience and look forward to developing more compelling services together with you for the developers around the world.

Get started

Add voice to your app in 15 minutes

Explore the available voices in this demo

Build a voice-enabled bot

Deploy Azure TTS voices on prem with Speech Containers

Build your custom voice

Azure Speech and Batch Ingestion

Panos Periorellis — Tue, 30 Mar 2021 20:21:49 GMT

Getting started with Azure Speech and Batch Ingestion Client

Batch Ingestion Client is as a zero-touch transcription solution for all your audio files in your Azure Storage. If you are looking for a quick and effortless way to transcribe your audio files or even explore transcription, without writing any code, then this solution is for you. Through an ARM template deployment, all the resources necessary to seamlessly process your audio files are set-up and set in motion.

Why do I need this?

Getting started with any API requires some amount of time investment in learning the API, understanding its scope, and getting value through trial and error. In order to speed up your transcription solution, for those of you that do not have the time to invest in getting to know our API or related best practices, we created an ingestion layer (a client for batch transcription) that will help you set-up a full blown, scalable and secure transcription pipeline without writing any code.

This is a smart client in the sense that it implements best practices and optimized against the capabilities of the Azure Speech infrastructure. It utilizes Azure resources such as Service Bus and Azure Functions to orchestrate transcription requests to Azure Speech Services from audio files landing in your dedicated storage containers.

Before we delve deeper into the set-up instructions, let us have a look at the architecture of the solution this ARM template builds.

The diagram is simple and hopefully self-explanatory. As soon as files land in a storage container, the Grid Event that indicates the complete upload of a file is filtered and pushed to a Service bus topic. Azure Functions (time triggered by default) pick up those events and act, namely creating Tx requests using the Azure Speech Services batch pipeline. When the Tx request is successfully carried out an event is placed in another queue in the same service bus resource. A different Azure Function triggered by the completion event starts monitoring transcription completion status and copies the actual transcripts in the containers from which the audio file was obtained. This is it. The rest of the features are applied on demand. Users can choose to apply analytics on the transcript, produce reports or redact, all of which are the result of additional resources being deployed through the ARM template. The solution will start transcribing audio files without the need to write any code. If -however- you want to customize further this is possible too. The code is available in this repo.

The list of best practices we implemented as part of the solution are:

Optimized the number of audio files included in each transcription with the view of achieving the shortest possible SAS TTL.
Round Robin around selected regions in order to distribute load across available regions (per customer request)
Retry logic optimization to handle smooth scaling up and transient HTTP 429 errors
Running Azure Functions economically, ensuring minimal execution cost

Setup Guide

The following guide will help you create a set of resources on Azure that will manage the transcription of audio files.

Prerequisites

An Azure Account as well as an Azure Speech key is needed to run the Batch Ingestion Client.

Here are the detailed steps to create a speech resource:

NOTE: You need to create a Speech Resource with a paid (S0) key. The free key account will not work. Optionally for analytics you can create a Text Analytics resource too.

Go to Azure portal
Click on +Create Resource
Type Speech and
Click Create on the Speech resource.
You will find the subscription key under Keys
You will also need the region, so make a note of that too.

To test your account, we suggest you use Microsoft Azure Storage Explorer.

The Project

Although you do not need to download or do any changes to the code you can still download it from GitHub:

git clone https://github.com/Azure-Samples/cognitive-services-speech-sdkcd cognitive-services-speech-sdk/samples/batch/transcription-enabled-storage

Make sure that you have downloaded the ARM Template from the repository.

Batch Ingestion Client Setup Instructions

Click on +Create Resource on Azure portal as shown in the following picture and type ‘ template deployment ’ on the search box.

2. Click on Create Button on the screen that appears as shown below.

3. You will be creating the relevant Azure resources from the ARM template provided. Click on click on the ‘Build your own template in the editor’ link and wait for the new screen to load.

You will be loading the template via the Load file option. Alternatively, you could simply copy/paste the template in the editor.

Saving the template will result in the screen below. You will need to fill in the form provided. It is important that all the information is correct. Let us look at the form and go through each field.

NOTE: Please use short descriptive names in the form for your resource group. Long resource group names may result in deployment error

First pick the Azure Subscription Id within which you will create the resources.
Either pick or create a resource group. [It would be better to have all the resources within the same resource group so we suggest you create a new resource group].
Pick a region [May be the same region as your Azure Speech key].

The following settings all relate to the resources and their attributes

Give your transcription enabled storage account a name [you will be using a new storage account rather than an existing one]. If you opt to use existing one then all existing audio files in that account will be transcribed too.

The following 2 steps are optional. Omitting them will result in using the base model to obtain transcripts. If you have created a Custom Speech model using Speech Studio, then:

Enter optionally your primary Acoustic model ID
Enter optionally your primary Language model ID

If you want us to perform Language identification on the audio prior to transcription you can also specify a secondary locale. Our service will check if the language on the audio content is the primary or secondary locale and select the right model for transcription.

Transcripts are obtained by polling the service. We acknowledge that there is a cost related to that. So, the following setting gives you the option to limit that cost by telling your Azure Function how often you want it to fire.

Enter the polling frequency [There are many scenarios where this would be required to be done couple of times a day]
Enter locale of the audio [you need to tell us what language model we need to use to transcribe your audio.]
Enter your Azure Speech subscription key and Locale information

NOTE: If you plan to transcribe large volume of audio (say millions of files) we propose that you rotate the traffic between regions. In the Azure Speech Subscription Key text box you can put as many keys separated by column ';'. In is important that the corresponding regions (Again separated by column ';') appear in the Locale information text box. For example if you have 3 keys (abc, xyz, 123) for east us, west us and central us respectively then lay them out as follows 'abc;xyz;123' followed by 'east us;west us;central us'

The rest of the settings related to the transcription request. You can read more about those in our docs.

Select a profanity option
Select a punctuation option
Select to Add Diarization [all locales]
Select to Add Word level Timestamps [all locales]

Do you need more than transcription? Do you need to apply Sentiment to your transcript? Downstream analytics are possible too, with Text Analytics Sentiment and Redaction being offered as part of this solution too.

If you want to perform Text Analytics please add those credentials.

Add Text analytics key
Add Text analytics region
Add Sentiment
Add data redaction

If you want to further analytics we could map the transcript json we produce to a DB schema.

Enter SQL DB credential login
Enter SQL DB credential password

You can feed that data to your custom PowerBI script or take the scripts included in this repository. Follow this guide for setting it up.

Press Create to trigger the resource creating process. It typically takes 1-2 mins. The set of resources are listed below.

If a Consumption Plan (Y1) was selected for the Azure Functions, make sure that the functions are synced with the other resources (see this for further details).

To do so, click on your StartTranscription function in the portal and wait until your function shows up:

Do the same for the FetchTranscription function

Important: Until you restart both Azure functions you may see errors.

Running the Batch Ingestion Client

Upload audio files to the newly created audio-input container (results are added to json-result-output and test-results-output containers). Once they are done you can test your account.

Use Microsoft Azure Storage Explorer to test uploading files to your new account. The process of transcription is asynchronous. Transcription usually takes half the time of the audio track to be obtained. The structure of your newly created storage account will look like the picture below.

There are several containers to distinguish between the various outputs. We suggest (for the sake of keeping things tidy) to follow the pattern and use the audio-input container as the only container for uploading your audio.

Customizing the Batch Ingestion Client

By default, the ARM template uses the newest version of the Batch Ingestion Client which can be found in this repository. If you want to customize this further clone the repo

To publish a new version, you can use Visual Studio, right click on the respective project, click publish and follow the instructions.

What to build next

Now that you’ve successfully implemented a speech to text scenario, you can build on this scenario. Take a look at the insights Text Analytics provides from the transcript like caller and agent sentiment, key phrase extraction and entity recognition. If you’re looking specifically to solve for Call centre transcription, review this docs page for further guidance

Microsoft named a Leader in 2021 Gartner Magic Quadrant for Cloud AI Developer Services

maddybutzbach — Fri, 19 Mar 2021 16:17:37 GMT

Gartner recently released their Magic Quadrant for 2021 Cloud AI Developer Services. Microsoft is in the Leaders quadrant and was positioned highest on the ability to execute axis. You can download a complimentary copy of the Magic Quadrant for Cloud AI Developer Services for the full report. In this post, we’ll look at why, we think, Microsoft was placed in the Leaders quadrant.

According to the report, “Gartner defines cloud AI developer services (CAIDS) as cloud-hosted or containerized services/models that allow development teams and business users to leverage artificial intelligence models via APIs, SDKs, or applications without requiring deep data science expertise.”

They specifically evaluated services with capabilities in language, vision, and automated machine learning. For Azure, this includes Azure Cognitive Services, Azure Machine Learning, and Microsoft’s conversational AI portfolio. For Power Platform, this includes AI Builder and Power Virtual Agents.

“Gartner believes that enterprise development teams will increasingly incorporate models built using AI and ML into applications. These services currently fall into three main functional areas: language, vision and automated machine learning (autoML). The language services include natural language understanding (NLU), conversational agent frameworks, text analytics, sentiment analysis and other capabilities. The vision services include image recognition, video content analysis and optical character recognition (OCR). The autoML services include automated tools that will let developers do data preparation, feature engineering, create models, deploy, monitor and manage models without having to learn data science.”

Azure AI enables you to develop AI applications on your terms, apply AI responsibly, and deploy mission-critical AI solutions.

Develop on your terms

Azure AI allows you to build AI applications in your preferred software development language and deploy in the cloud, on-premises, or at the edge. Azure provides options for data scientists and developers of all skill levels – no machine learning expertise required. See the Microsoft section of the Magic Quadrant for Cloud AI Developer Services.

Apply AI responsibly

Azure offers tools and resources to help you understand, protect, and control your AI solutions, including responsible ML toolkits, responsible bot development guidelines, tools to help you explain model behavior and test for fairness, and more. We never use your data to train our models, and we keep principles like inclusiveness, fairness, transparency, and accountability in mind at every stage of our AI research, development, and deployment. See the Microsoft section of the Magic Quadrant for Cloud AI Developer Services.

Deploy mission-critical solutions

Azure lets you access the same AI services that power products like Microsoft Teams and Xbox, and that are proven at global scale. Azure leads the industry when it comes to security, and we have the most comprehensive compliance coverage of any cloud service provider. We continue to innovate and our Microsoft Research team has made significant breakthroughs, most recently reaching human parity with image captioning. See the Microsoft section of the Magic Quadrant for Cloud AI Developer Services.

Whether you’re a professional developer or data scientist, or just getting started, we hope that you can use Azure AI services to build impactful AI-powered applications that solve complex problems and enhance customer experience.

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Microsoft. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

The Tenets of Knowledge Management Adoption

Sonia Ang — Fri, 19 Mar 2021 16:00:00 GMT

Knowledge Management Systems and Adoption Key Tenets:

Sonia M. Ang – CSA

In today’s competitive business environment organizations need a clear roadmap that aligns with their training needs and focuses on both short-and long-term objectives. Knowledge Management is a tool that can be implemented to identify and appeal to the training needs of the modern employee. A successful Enterprise Learning system goes beyond the organizational level and allows employees access to information and knowledge, thus creating better alignment at the enterprise level. A well-designed Knowledge Management System can break down barriers by providing partners, clients, and customers with not only essential information and robust training, but also opportunities to promote and inform your organization’s products and services.

Employee retention and customer satisfaction are essential to any organizations long-term success. When considering your organization’s ROI, the benefits of Enterprise Learning are threefold: retention (customers and employees), satisfaction, and improved profitability. Enterprise Learning can be leveraged to provide better development and training opportunities thus promoting a feeling of empowerment amongst your team. Enterprise Learning promotes efficiencies in training, in turn your organization will recognize the cost-saving benefits due to lower employee turnover and customer churn.

Enterprise Knowledge Management Adoption

1. Management Sponsorship and a COE

An executive sponsor provides that critical link between executive leadership and project management and helps support projects successfully to their completion at their expected performance.  Sponsorship of the Enterprise Knowledge Management Project will enhance your product base and create opportunities for your company in the rapidly advancing Knowledge Management sector.  Executive sponsorship will align with our company’s strategy to be experts in Knowledge Management.  Microsoft will be at the forefront of Knowledge Management as organizations rush to adopt more efficient and effective training strategies that better align to the modern worker's needs.

2. Beyond Training but Execution

Training programs at the company level are often too extensive in complexity and the amount of learning material can be downright overwhelming. Training professionals need to venture beyond traditional learning and utilize learning strategies that support the needs of today’s modern professionals. Microlearning, for example, provides a host of benefits to your organization in terms of increased learner participation, memorability of courses, and quick deployment with easy updates to your digital learning assets.

Knowledge Management allows for learning concepts to be extracted from larger training programs and utilized as checklists and instructional videos that are easily accessible at a moment’s notice. When learning material is successfully mined it allows the learning process to be refined, thus challenging concepts can be identified and made easier to process and understand.

3. Collaborative and Social Learning

How does your organization support collaboration, problem-solving and the co-creation of knowledge? A successful Enterprise Learning Strategy will push an organization forward by improving collaboration via robust communities and meaningful discussion forums. Building professional learning communities on platforms like Slack and Microsoft TEAMS can help break the walls down in organizations where ideas and knowledge may be siloed. In addition to collaborative discussion platforms that give all community members a voice, the development of Expert Finders can be an important catalyst for creating a robust culture of collaboration. Hidden ideas and knowledge will organically emerge from people in your organization that may hold previously unearthed niche expertise.

4. Where is the Data?

Generating data for analysis is the foundation of a robust Enterprise Learning strategy. The key to success is to build data collection directly into technical systems supported by a centralized knowledge repository. Data collection in the education sphere has traditionally focused on summative assessments like exams that are intended to measure a learner’s mastery of objectives. Using the specifications in Enterprise Learning provides another set of metrics by allowing an organization to track formative assessments, such as data and social learning activity. For example, these metrics allows an organization to collect new data, adding another layer of data to your knowledge repository to support the creation of more meaningful formative and summative assessments.

5. Reusable Content and Reproducibility

How do organizations move past the traditional content development models where bulky training manuals were the norm? Learning data and insights help organizations to build learning solutions that are reusable and provide 24/7 access to learning. Additionally, a Headless CMS allows your organization’s content creators to move away from the rigid templates that most traditional learning management systems utilize. This means content creators have more control over the quality of their content, and this streamlines the process of creating unique digital learning experiences for both your employees and customers. Perhaps your organization releases a short instructional video on company’s rules and regulations. This learning asset has value as both a stand-alone content object in a knowledge base as well as a learning module in a more extensive communications course. Content creators in your organization, ranging from instructional designers to marketing professionals, often create multiple versions of the same material. Creating content that is reusable and not redundant is more efficient as you are not reinventing the wheel. Additionally, you are not burdened with trying to maintain and keep up-to-date multiple versions of the same learning assets.

6. “Findability” of Learning Assets: Digitization and Technology

The successful implementation of KM tools can enhance the user experience through successful mining and classification of metadata. The power of KM is powerful in that it provides a taxonomy, ontology, and a finely tuned search system. A well-designed metadata strategy will take advantage of your learning assets which may include courses, webinars, professional learning communities (PLCs) and subject matter experts in your organization. This myriad of data exits across multiple systems in your organization. Using metadata that is contextualized and consistent will ensure that your data is findable. Taking it one-step further ontologies can be tapped to support a complete network of shareable and reusable knowledge across a domain for each unique user. 

Summary

The adoption of an Enterprise Management system can reframe your organization’s knowledge and learning infrastructure. Modern learners need to consume information quickly thus allowing them to efficiently apply and master essential skills and strategies. A well-engineered Enterprise Learning Plan that focuses on empowering your community will result in higher learner engagement, enhanced workplace skills and a robust culture-of-knowledge across your organization. Curiosity will drive success rather than traditional command and control training strategies. Empower your community by giving them the autonomy to self-direct their own learning. As a result, your organization will enjoy increased engagement, enhanced workforce skills and, in turn, a robust learning culture will grow and flourish

Extract Data from PDFs using Form Recognizer with Code or Without!

cassieview — Tue, 16 Mar 2021 16:43:56 GMT

Form Recognizer is a powerful tool to help build a variety of document machine learning solutions. It is one service however its made up of many prebuilt models that can perform a variety of essential document functions. You can even custom train a model using supervised or unsupervised learning for tasks outside of the scope of the prebuilt models! Read more about all the features of Form Recognizer here. In this example we will be looking at how to use one of the prebuilt models in the Form Recognizer service that can extract the data from a PDF document dataset. Our documents are invoices with common data fields so we are able to use the prebuilt model without having to build a customized model.

Sample Invoice:

After we take a look at how to do this with Python and Azure Form Recognizer, we will take a look at how to do the same process with no code using the Power Platform services: Power Automate and Form Recognizer built into AI Builder. In the Power Automate flow we are scheduling a process to happen every day. What the process does is look in the raw blob container to see if there is new files to be processed. If there is new files to be processed it gets all blobs from the container and loops through each blob to extract the PDF data using a prebuilt AI builder step. Then it deletes the processed document from the raw container. See what it looks like below.

Power Automate Flow:

Prerequisites for Python

Azure Account Sign up here!
Anaconda and/or VS Code
Basic programming knowledge

Prerequisites for Power Automate

Power Automate Account Sign up here!
No programming knowledge

Process PDFs with Python and Azure Form Recognizer Service

Create Services

First lets create the Form Recognizer Cognitive Service.

Go to portal.azure.com to create the resource or click this link.

Now lets create a storage account to store the PDF dataset we will be using in containers. We want two containers, one for the processed PDFs and one for the raw unprocessed PDF.

Create an Azure Storage Account
Create two containers: processed, raw

Upload data

Upload your dataset to the Azure Storage raw folder since they need to be processed. Once processed then they would get moved to the processed container.

The result should look something like this:

Create Notebook and Install Packages

Now that we have our data stored in Azure Blob Storage we can connect and process the PDF forms to extract the data using the Form Recognizer Python SDK. You can also use the Python SDK with local data if you are not using Azure Storage. This example will assume you are using Azure Storage.

Create a new Jupyter notebook in VS Code.
Install the Python SDK

!pip install azure-ai-formrecognizer --pre

Then we need to import the packages.

import os
from azure.core.exceptions import ResourceNotFoundError
from azure.ai.formrecognizer import FormRecognizerClient
from azure.core.credentials import AzureKeyCredential
import os, uuid
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__

Create FormRecognizerClient

Update the endpoint and key with the values from the service you created. These values can be found in the Azure Portal under the Form Recongizer service you created under the Keys and Endpoint on the navigation menu.

endpoint = "<your endpoint>"
key = "<your key>"

We then use the endpoint and key to connect to the service and create the FormRecongizerClient

form_recognizer_client = FormRecognizerClient(endpoint, AzureKeyCredential(key))

Create the print_results helper function for use later to print out the results of each invoice.

def print_result(invoices, blob_name):
    for idx, invoice in enumerate(invoices):
        print("--------Recognizing invoice {}--------".format(blob_name))
        vendor_name = invoice.fields.get("VendorName")
        if vendor_name:
            print("Vendor Name: {} has confidence: {}".format(vendor_name.value, vendor_name.confidence))
        vendor_address = invoice.fields.get("VendorAddress")
        if vendor_address:
            print("Vendor Address: {} has confidence: {}".format(vendor_address.value, vendor_address.confidence))
        customer_name = invoice.fields.get("CustomerName")
        if customer_name:
            print("Customer Name: {} has confidence: {}".format(customer_name.value, customer_name.confidence))
        customer_address = invoice.fields.get("CustomerAddress")
        if customer_address:
            print("Customer Address: {} has confidence: {}".format(customer_address.value, customer_address.confidence))
        customer_address_recipient = invoice.fields.get("CustomerAddressRecipient")
        if customer_address_recipient:
            print("Customer Address Recipient: {} has confidence: {}".format(customer_address_recipient.value, customer_address_recipient.confidence))
        invoice_id = invoice.fields.get("InvoiceId")
        if invoice_id:
            print("Invoice Id: {} has confidence: {}".format(invoice_id.value, invoice_id.confidence))
        invoice_date = invoice.fields.get("InvoiceDate")
        if invoice_date:
            print("Invoice Date: {} has confidence: {}".format(invoice_date.value, invoice_date.confidence))
        invoice_total = invoice.fields.get("InvoiceTotal")
        if invoice_total:
            print("Invoice Total: {} has confidence: {}".format(invoice_total.value, invoice_total.confidence))
        due_date = invoice.fields.get("DueDate")
        if due_date:
            print("Due Date: {} has confidence: {}".format(due_date.value, due_date.confidence))

Connect to Blob Storage

Now lets connect to our blob storage containers and create the BlobServiceClient. We will use the client to connect to the raw and processed containers that we created earlier.

# Create the BlobServiceClient object which will be used to get the container_client
connect_str = "<Get connection string from the Azure Portal>"
blob_service_client = BlobServiceClient.from_connection_string(connect_str)

# Container client for raw container.
raw_container_client = blob_service_client.get_container_client("raw")

# Container client for processed container
processed_container_client = blob_service_client.get_container_client("processed")

# Get base url for container.
invoiceUrlBase = raw_container_client.primary_endpoint
print(invoiceUrlBase)

HINT: If you get a "HttpResponseError: (InvalidImageURL) Image URL is badly formatted." error make sure the proper permissions to access the container are set. Learn more about Azure Storage Permissions here

Extract Data from PDFs

We are ready to process the blobs now! Here we will call list_blobs to get a list of blobs in the raw container. Then we will loop through each blob, call the begin_recognize_invoices_from_url to extract the data from the PDF. Then we have our helper method to print the results. Once we have extracted the data from the PDF we will upload_blob to the processed folder and delete_blob from the raw folder.

print("\nProcessing blobs...")

blob_list = raw_container_client.list_blobs()
for blob in blob_list:
    invoiceUrl = f'{invoiceUrlBase}/{blob.name}'
    print(invoiceUrl)
    poller = form_recognizer_client.begin_recognize_invoices_from_url(invoiceUrl)

    # Get results
    invoices = poller.result()

    # Print results
    print_result(invoices, blob.name)

    # Copy blob to processed
    processed_container_client.upload_blob(blob, blob.blob_type, overwrite=True)

    # Delete blob from raw now that its processed
    raw_container_client.delete_blob(blob)

Each result should look similar to this for the above invoice example:

The prebuilt invoices model worked great for our invoices so we don't need to train a customized Form Recognizer model to improve our results. But what if we did and what if we didn't know how to code?! You can still leverage all this awesomeness in AI Builder with Power Automate without writing any code. We will take a look at this same example in Power Automate next.

Use Form Recognizer with AI Builder in Power Automate

You can achieve these same results using no code with Form Recognizer in AI Builder with Power Automate. Lets take a look at how we can do that.

Create a New Flow

Log in to Power Automate
Click Create then click Scheduled Cloud Flow. You can trigger Power Automate flows in a variety of ways so keep in mind that you may want to select a different trigger for your project.
Give the Flow a name and select the schedule you would like the flow to run on.

Connect to Blob Storage

Click New Step
List blobs Step
- Search for Azure Blob Storage and select List blobs
- Select the ellipsis click Create new connection if your storage account isn't already connected
  - Fill in the Connection Name, Azure Storage Account name (the account you created), and the Azure Storage Account Access Key (which you can find in the resource keys in the Azure Portal)
  - Then select Create
- Once the storage account is selected click the folder icon on the right of the list blobs options. You should see all the containers in the storage account, select raw.

Your flow should look something like this:

Loop Through Blobs to Extract the Data

Click the plus sign to create a new step
Click Control then Apply to each
Select the textbox and a list of blob properties will appear. Select the value property
Next select add action from within the Apply to each Flow step.
Add the Get blob content step:
- Search for Azure Blob Storage and select Get blob content
- Click the textbox and select the Path property. This will get the File content that we will pass into the Form Recognizer.
Add the Process and save information from invoices step:
- Click the plus sign and then add new action
- Search for Process and save information from invoices
- Select the textbox and then the property File Content from the Get blob content section
Add the Copy Blob step:
- Repeat the add action steps
- Search for Azure Blob Storage and select Copy Blob
- Select the Source url text box and select the Path property
- Select the Destination blob path and put /processed for the processed container
- Select Overwrite? dropdown and select Yes if you want the copied blob to overwrite blobs with the existing name.
Add the Delete Blob step:
- Repeat the add action steps
- Search for Azure Blob Storage and select Delete Blob
- Select the Blob text box and select the Path property

The Apply to each block should look something like this:

Save and Test the Flow
- Once you have completed creating the flow save and test it out using the built in test features that are part of Power Automate.

This prebuilt model again worked great on our invoice data. However if you have a more complex dataset, use the AI Builder to label and create a customized machine learning model for your specific dataset. Read more about how to do that here.

Conclusion

We went over a fraction of the things that you can do with Form Recognizer so don't let the learning stop here! Check out the below highlights of new Form Recognizer features that were just announced and the additional doc links to dive deeper into what we did here.

Additional Resources

New Form Recognizer Features

What is Form Recognizer?

Quickstart: Use the Form Recognizer client library or REST API

Tutorial: Create a form-processing app with AI Builder

AI Developer Resources page

AI Essentials video including Form Recognizer

Model understanding with Azure Machine Learning

mithigpe — Tue, 16 Mar 2021 19:05:14 GMT

This post is co-authored by Mehrnoosh Sameki, Program Manager, Azure Machine Learning.

Overview

Model interpretability and fairness are part of the ‘Understand’ pillar of Azure Machine Learning’s Responsible ML offerings. As machine learning becomes ubiquitous in decision-making from the end-user utilizing AI-powered applications to the business stakeholders using models to make data-driven decisions, it is necessary to provide tools at scale for model transparency and fairness.

Explaining a machine learning model and performing fairness assessment is important for the following users:

Data scientists and model evaluators - At training time to help them to understand their model predictions and assess the fairness of their AI systems, enhancing their ability to debug and improve models.
Business stakeholders and auditors - To build trust with defined ML models and deploy them more confidently.

Customers like Scandinavian Airlines (SAS) and Ernst & Young (EY) put interpretability and fairness packages to the test to be able to deploy models more confidently.

SAS used interpretability to confidently identify fraud in its EuroBonus loyalty program. SAS data scientists could debug and verify model predictions using interpretability. They produced explanations about model behavior that gave stakeholders confidence in the machine learning models and assisted with meeting regulatory requirements.
EY utilized fairness assessment and unfairness mitigation techniques with real mortgage adjudication data to improve the fairness of loan decisions from having an accuracy disparity of 7 percent between men and women to less than 0.5 percent.

We are releasing enhanced experiences and feature additions for the interpretability and fairness toolkits in Azure Machine Learning, to empower more ML practitioners and teams to build trust with AI systems.

Model understanding using interpretability and fairness toolkits

These two toolkits can be used together to understand model predictions and mitigate unfairness. For this demonstration, we shall take a look at a loan allocation scenario. Let’s say that the label indicates whether each individual repaid a loan in the past. We will use the data to train a predictor to predict whether previously unseen individuals will repay a loan or not. The assumption is that the model predictions are used to decide whether an individual should be offered a loan.

Identify your model's fairness issues

Our revamped fairness dashboard can help uncover the harm of allocation which leads to the model unfairly allocating loans among different demographic groups. The dashboard can additionally uncover harm of quality of service which leads to a model failing to provide the same quality of service to some people as they do to others. Using the fairness dashboard, you can identify if our model treats different demographics of sex unfairly.

Dashboard configurations

When you first load the fairness dashboard, you need to configure it with desired settings, including:

selection of your sensitive demographic of choice (e.g., sex[1])
model performance metric (e.g., accuracy)
fairness metric (e.g., demographic parity difference).

Model assessment view

After setting the configurations, you will land on a model assessment view where you can see how the model is treating different demographic groups.

Our fairness assessment shows an 18.3% disparity in the selection rate (or demographic group difference). According to that insight, 18.3% more males are receiving qualifications for loan acceptance compared to females. Now that you’ve seen some unfairness indicators in your model, you can next use our interpretability toolkit to understand why your model is making such predictions.

Diagnose your model’s predictions

The new revamped interpretability dashboard greatly improves the user experience of the previous dashboard. In the loan allocation scenario, you can understand how a model treats female loan applicants differently than male loan applicants using the interpretability toolkit:

Dataset cohort creation: You can slice and dice your data into subgroups (e.g., female vs. male vs. unspecified) and investigate or compare your model’s performance and explanations across them.
Model performance tab: With the predefined female and male cohorts, we can observe the different prediction distributions between males and female cohorts, with females experiencing higher probability rates of being rejected for a loan.
Dataset explorer tab: Now that you have seen in the model performance tab how females are rejected at a higher rate than males, you can use the data explorer tab to observe the ground truth distribution between males and females. For males, the ground truth data is well balanced between those receiving a rejection or approval whereas, for females, the ground truth data is heavily skewed towards rejection thereby explaining how the model could come to associate the label ‘female’ with rejection.
Aggregate feature importance tab: Now we observe which top features contribute to the model’s overall prediction (also called global explanations) towards loan rejection. We sort our top feature importances by the Female cohort, which indicates that while the feature for “Sex” is the second most important feature to contribute towards the model’s predictions for individuals in the female cohort, they do not influence how the model makes predictions for individuals in the male cohort. The dependence plot for the feature “Sex” also shows that only the female group has positive feature importance towards the prediction of being rejected for a loan, whereas the model does not look at the feature “Sex” for males when making predictions.
Individual feature importance & What-If tab: Drilling deeper into the model’s prediction for a specific individual (also called local explanations), we look at the individual feature importances for only the Female cohort. We select an individual who is at the threshold of being accepted for a loan by the model and observe which features contributed towards her prediction of being rejected. “Sex” is the second most important feature contributing towards the model prediction for this individual. The Individual Conditional Expectation (ICE) plot calculates how a perturbation for a given feature value across a range can impact its prediction. We select the feature “Sex” and can see that if this feature had been flipped to male, the probability of being rejected is lowered drastically. We create a new hypothetical What-If point from this individual data point and switch only the “Sex” from female to male, and observe that without changing any other feature related to financial competency, the model now predicts that this individual will have their loan application accepted.

Once some potential fairness issues are observed and diagnosed, you can move to mitigate those unfairness issues.

Mitigate unfairness issues in your model

The unfairness mitigation part is powered by the Fairlearn open-source package which includes two types of mitigation algorithms: postprocessing algorithms (ThresholdOptimizer) and reduction algorithms (GridSearch, ExponentiatedGradient). Both operate as “wrappers” around any standard classification or regression algorithm. GridSearch, for instance, treats any standard classification or regression algorithm as a black box, and iteratively (a) re-weight the data points and (b) retrain the model after each re-weighting. After 10 to 20 iterations, this process results in a model that satisfies the constraints implied by the selected fairness metric while maximizing model performance. ThresholdOptimizer on the other hand takes as its input a scoring function that underlies an existing classifier and identifies a separate threshold for each group to optimize the performance metric, while simultaneously satisfying the constraints implied by the selected fairness metric.

The fairness dashboard also enables the comparison of multiple models, such as the models produced by different learning algorithms and different mitigation approaches. Bypassing the dominated models of GridSearch for instance, you can see the unmitigated model on the upper right side (with the highest accuracy and highest demographic parity difference) and can click on any of the mitigated models to observe them further. This allows you to examine trade-offs between performance and fairness.

Comparing results of unfairness mitigation

After applying the unfairness mitigation, we go back to the interpretability dashboard and compare the unmitigated model with the mitigated model. In the figure below, we see a more even probability distribution for the female cohort for the mitigated model on the right:

Revisiting the fairness assessment dashboard, we also see a drastic decrease in demographic parity difference from 18.8% (unmitigated model) to 0.412% (mitigated model):

Saving model explanations and fairness metrics to Azure Machine Learning Run History

Azure Machine Learning’s (AzureML) interpretability and fairness toolkits can be run both locally and remotely. If run locally, the libraries will not contact any Azure services. Alternatively, you can run the algorithms remotely on AzureML compute and log all the explainability and fairness information into AzurML’s run history via the AzureML SDK to save and share them with other team members or stakeholders in AzureML studio.

Azure ML’s Automated ML supports explainability for its best model as well as on-demand explainability for any other models generated by Automated ML.

Learn more

Explore this scenario and other sample notebooks in the Azure Machine Learning sample notebooks GitHub.

Learn more about the Azure Machine Learning service.

Learn more about Responsible ML offerings in Azure Machine Learning.

Learn more about interpretability and fairness concepts and see documentation on how-to guides for using interpretability and fairness in Azure Machine Learning.

Get started with a free trial of the Azure Machine Learning service.

[1] This dataset is from the 1994 US Census Bureau Database where “sex” in the data was limited to binary categorizations.

Advance Resource Access Governance for AML

mufy — Thu, 11 Mar 2021 09:28:08 GMT

Access control is a fundamental building block for enterprise customers, where protecting assets at various levels is absolutely necessary to ensure that only the relevant people with certain positions of authority are given access with different privileges. This is more so prevalent in machine learning, where data is absolutely essential in building ML models, and companies are highly cautious about how the data is accessed and managed, especially with the introduction of GDPR. We are seeing an increasing number of customers seeking for explicit control of not only the data, but various stages of the machine learning lifecycle, starting from experimentation and all the way to operationalization. Assets such as generated models, cluster creation and model deployment require to be governed to ensure that controls are in line with the company’s policy.

Azure traditionally provides Role-based Access Control [1], which helps to manage access to resources; who can access these and what they can access. This is primarily achieved via the concept of roles. A role defines a collection of permissions.

Existing Roles in AML

Azure Machine Learning provides three roles [3] for enterprise customers to provision as a coarse-grained access control, which is designed for simplicity in mind. The first role (Owner) has the highest level of privileges, that grants full control of the workspace. This is followed by a Contributor, which is a bit more restricted role that prevents users from changing role assignment. Reader having the most restrictive permissions and is typically read or view only (see figure 1 below).

Figure 1 - Existing AML roles

What we have found with the customers is that while Coarse-grained Access Control immensely simplifies the management of the roles, and works quite well with a small team, primarily working in the experimentation environment. However, when a company decides to operationalize the ML work, especially in the enterprise space, these roles become far too broad, and too simplistic. In the enterprise space, the deployment tends to have several stages (such as dev, test, pre-prod, prod, etc.), and require various skillset (data scientist, data engineer, etc.) with a greater control in each stage. For example, a Data Scientist may not operate in the production environment. A Data Engineer can only provision resources and should not have the ability to commission and decommission training clusters. Such governance policies are crucial for companies to be enforced and monitored to maintain integrity of their business and IT processes.

Unfortunately, such requirements cannot be captured with the existing roles. Enterprise needs a better mechanism to define policies for various assets in AML to satisfy their business specific requirements.

This is where the new exciting feature of advanced Role-based Access Control really shines. It is based on Fine-grained Access Control at component level (see figure 2) with a number of pre-built out of the box roles, plus the ability to create custom roles that can capture more complex governance access processes and enforce them.

Advance Fine-grained Role-based Access Control

The new advance Role-based Access Control feature of AML is really going to solve many of the enterprise problems around the ability to restrict or grant user permissions for various components. Azure AML currently defines 16 components with varying permissions.

Figure 2 - Components Level RBAC

Each component defines a list of actions such as read, write, delete, etc. These actions can then be amalgamated together to create a custom specific role. To illustrate this with an example of a list of actions currently available for a Datastore component (see Figure 3 below).

Figure 3 - Datastore Actions

A datastore along with Dataset are important concepts in Azure Machine Learning, since they provide access to various data sources, with lineage and tracking ability. Many enterprises have built global Datalake that contain terabytes of data which can contain highly sensitive information. Companies are quite protective of who can access these data, along with various business justifications for how these data are being accessed/used. It is therefore imperative that a tighter access control is mandated for a specific role, such as a Data Engineer to accomplish such a task.

Fortunately, AML advance access control provide custom roles. to cater for their company specific access control, which may be a hybrid of these roles. For such requirements, Azure caters for custom roles.

Custom Role

Custom role [4] allows creation of Fine-grained Access Control on various components, such as the workspace, datastore, etc.

Can be any combination of data or control plane actions that AzureML+AISC support.
Useful for creating scoped roles to a specific action like an MLOps Engineer

These controls are defined in a JSON policy definition, for example.

{ "Name": "Data Scientist", "IsCustom": true, "Description": "Can run experiment but can't create or delete datastore.", "Actions": ["*"], "NotActions": [ "Microsoft.MachineLearningServices/workspaces/*/delete", "Microsoft.MachineLearningServices/workspaces/ datastores/write", "Microsoft.MachineLearningServices/workspaces/ datastores /delete", “Microsoft.MachineLearningServices/workspaces/datastores/write”, "Microsoft.Authorization/*/write" ], "AssignableScopes": [ "/subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>" ] }

The above code defines a Data Scientist who can run an experiment but cannot create or delete a Datastore. This role can be created using the Azure CLI (az role definition create -role-definition filename), however, the CLI ML extension needs to be installed first.

Role Operation Workflow

In an organization, the following activities are to be undertaken by various role owners.

Sub admin comes in for an enterprise and requests Amlcompute quota
They create an RG and a workspace for a specific team, and also set workspace level quota
The team lead (aka workspace admin), comes in and starts creating compute within the quota that the sub admin defined for that workspace
Data Scientist comes in and uses the compute that workspace admin created for them (clusters or instances).

Roles for Enterprise

AML provides a single environment for doing end-to-end experimentation to operationalization. For a start-up this is really useful as they tend to operate in a very agile manner, where many iterations can happen in a short period of time and having the ability to quickly move from ideation to production really reduces their cycle time. Unfortunately, this may not be the case for the enterprise customers, where they would typically be using either two or three environments to carry out their production workload such as: Dev, QA and Prod.

Dev is used to do the experimentation, while QA is catered for satisfying various functional and non-functional requirements, followed by Prod for deployment into the production for consumer usage.

The environments would also have various roles to carry out different activities, such as Data Scientist, Data Engineer and MLOps Engineer (see figure 8 below).

Figure 8 - Enterprise Roles

A Data Scientist normally operates in the Dev environment and has full access to all the permissions related to carrying out experiments, such as provisioning training clusters, building models, etc. While some permissions are granted in the QA environment, primarily related model testing and performance, and very minimal access to the Prod environment, mainly telemetry (see below Table 1).

A Data Engineer on the other hand primarily operates in the Build and QA environment. The main focus is related to the data handling, such as data loading, doing some data wrangling, etc. They have restricted access in the Prod environment.

Table 1 - Role/environment Matrix

An MLOps Engineer has some permission in the Dev environment, but full permissions in the QA and Prod. This is because an MlOPs Engineer is tasked with building the pipeline, gluing things together, and ultimately deploying models in production.

The interesting part is how do all these roles and environments and other components fit together in Azure to provide the much-needed access governance for the enterprise customers.

Enterprise AML Roles Deployment

It is impressive for enterprises to be able to model these complex roles/environments mapping as shown in Table one. Fortunately these can be achieved in Azure using a combination of AD groups, roles and resource groups.

Figure 9 - Enterprise AML Roles Deployment

Fundamentally, Azure Active Directory groups play a major part in gluing all these components together to make it functional.

First step is to group the users specific to role(s) in a “Role AD group” for a given persona (DS, DE, etc.,). Then assign roles with various RBAC actions (Data Writer, MLContributor, etc.) to this AD group. All these users will now inherit the permissions specific to this role(s). Multiple AD groups will be created for different persona roles.

Separate AD groups (‘AD group for Environment’) are created for each environment (i.e. Dev, QA and Prod), the Role AD Groups are added to these Environment AD groups. This creates a mapping of users belonging to a specific role persona with given permissions to an environment.

The ‘AD group for Environment’ is then assigned to a resource group, which contains a specific AML Workspace. This ensures that the role permissions assigned to users will be enforced at the workspace level.

Summary

In this blog, we have discussed the new advance Role-based Access Control, and how it is being applied in a complex enterprise with various environments with different user personas.

The important point to note is the flexibility that comes with this new feature which can operate at any of the 16 AML components and be able to define Fine-grained Access Control for each through custom roles, and out of box four roles which should be sufficient for the majority of the customers.

References

[1] https://docs.microsoft.com/en-us/azure/role-based-access-control/overview

[2] https://azure.microsoft.com/en-gb/services/machine-learning/

[3] https://docs.microsoft.com/en-us/azure/machine-learning/concept-enterprise-security

[4] https://docs.microsoft.com/en-us/azure/role-based-access-control/custom-roles

Additional Links:

https://docs.microsoft.com/en-us/azure/machine-learning/how-to-assign-roles

co-author: @Nishank Gupt and @John Wu

Improving collaboration and productivity in Azure Machine Learning

abeomor — Wed, 10 Mar 2021 18:28:13 GMT

This post is co-authored by Sharon Xu Program Manager, Azure Notebooks.

Today we are very proud to announce the next set of productivity features and improvements for the notebook experience. Since we announced the GA release of Notebooks in Azure Machine Learning (Azure ML), we have learned a lot from our customers. Over the past few months, we have incrementally improved the notebook experience while simultaneously contributing back to the open source nteract project. The Azure ML team recently released a robust set of new functionalities designed to improve data scientist productivity and collaboration in Azure ML Notebooks.

Data scientist & Developer Productivity

We have spoken to several data scientists and developers to fully understand the additional features needed to improve productivity while developing machine learning projects. From feedback, we have found that users constantly needed the following enhancements to speed up their workflow: a clear indication that a cell has finished running, a way to templatize common code excerpts, a way to check variable contents, and more. The following list is a culmination of the most highly requested productivity features:

Cell Status Bar. The status bar located in each cell indicates the cell state: whether a cell has been queued, successfully executed, or run into an error. The status bar also displays the execution time of the last run.
Variable Explorer. provides a quick glance into the data type, size, and contents of your variables and dataframes, allowing for quicker and simpler debugging.

Figure 1: (1) Cell status bar (2) Variable explorer

Notebook snippets (preview). Common Azure ML code excerpts are now available at your fingertips. Navigate to the code snippets panel, accessible via the toolbar, or activate the in-code snippets menu using Ctrl + Space.

Figure 2 (1) Notebook snippets panel, showing all useful snippets

IntelliCode. IntelliCode provides intelligent auto-completion suggestions using an ML algorithm that analyzes the context of your notebook code. IntelliCode suggestions are designated with a star.

Figure 3: IntelliCode in Azure ML Notebooks

Keyboard shortcuts with full Jupyter parity. Azure ML now supports all the keyboard shortcuts available in Jupyter and more.
Table of Contents. For large notebooks, the Table of Contents panel then allows you to navigate to the desired section. The sections of the notebook are designated by the Markdown headers.
Markdown Side-by-side Editor in Notebooks. Within each notebook, the new side-by-side editor allows you to view the rendered results of your Markdown cells directly in your notebook editing.

Figure 4: (1) Table of content pane (2) Markdown side by side

Collaboration and Sharing

An increasing number of data scientists and developers are creating notebooks collaboratively and sharing these notebooks across their team We heard feedback that most users feel like they are missing adequate tools to edit notebooks simultaneously or share their notebooks with a broader audience. Users often resort to screen shares and calls to complete or present work within a notebook. We recently just release a few new features to help combat some of these issues:

Co-editing (preview). Co-editing makes collaboration easier than ever. The notebook can now be shared by sending the notebook URL, allowing multiple users to edit the notebook in real-time.

Figure 5: Live Co-editing in Azure ML

Export Notebook as Python, LaTeX or HTML. When you feel satisfied with the results from your notebook and ready to present to your colleagues, you can export the notebook to various formats for easy sharing. LaTeX, HTML, and .py are currently supported.

Figure 6: Export Notebooks as Python and more in Azure ML

Get Started Today

To begin using these features in Azure ML Notebooks, you will first need to create an Azure Machine Learning. Your Azure ML workspace serves as your one-stop-shop for all your machine learning needs, where you can create and share all your machine learning assets.

Once you have your workspace set up, you can get started using all the features in the Azure ML Notebooks experience. The notebooks experience aims to provide you with an integrated suite of data science tools. Users can start working with a highly productive and collaborative Jupyter notebook editor directly in their workspace as well as quickly access other ML assets such as experiment details, datasets, models, and more.

With the addition of this host of features, notebooks in Azure ML aims to improve every aspect of your development needs – collaboration, code editing, debugging. Give these features a try and leave your feedback. The feedback provided by our community is what drives us to improve and build new features. As we continue to push out new releases, keep an eye out, because the team has a few more exciting features coming out soon.

Integrating AI: Prototyping a No-Code solution with Power Apps

Yonet — Mon, 15 Mar 2021 21:14:46 GMT

You might have the cutting edge AI features but it is hard to know how useful it will be before letting your users beta test your prototype. You can build fast, deploy and deliver your app and iterate without writing any code, using AI Builder and Power Platform.

This article explains what Power Platform is, as well as go through a step by step process to create an application that detects objects from photos using Power Apps and AI Builder. Check out the video below to see the app we will build to detect different Mixed Reality Headsets such as HoloLens version 1 and 2 Augmented Reality and Virtual Reality headsets and their hand controllers.

What is Power Platform?

Power Platform is a set of tools, API's and SDK's that helps you analyze your data and build automations, applications and virtual agents with or without having to write any code.

What are Power Apps?

Power Apps, allows you to create applications with a drag and drop UI and easy integration of your data and 3rd party APIs through connectors.

A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows. For example, you can use the Twitter connector to get tweet data and visualize it in a dashboard or use the Twilio connector to send your users text messages without having to be an expert in Twitter or Twilio APIs or having to write a line of code.

Check out the list of connectors for Power Apps to see all the APIs that are available. Notice Power Automate or Logic App connectors might not be the same.

What is AI Builder?

AI Builder is one of the additional features of Power Apps. With AI Builder, you can add intelligence to your apps even if you have no coding or data science skills.

What are some of the use cases for AI Builder?

You can use pre-trained models to:

Detect objects from images
Analyze your customers' sentiment from feedback
Detect keywords from text
Extract specific information about your business from text

Is AI Builder the right choice?

Great question! There are so many tools out there and many ways to do the same thing. How do you know which one is the right solution before investing time and effort?

‌

I have a rule of thumb when I want to build something, use whatever is available and easy to use first. When your needs exceed what the tool you are using covers, look into another solution or building it yourself.

Use the tool easiest to get started when you are building your idea. When your needs exceed the capabilities of the tool you are using, find a solution that enables you. Don't invest in building things from scratch before you know it is worth it to do so.

For example, if you have an app idea, it is better to have a prototype running as easily as possible. You can test your ideas before investing your time into building custom designed UI or features. In our specific case, you can first prototype your app with the drag and drop UI of Power Apps and using prebuilt AI models. When your specific needs surface, such as recognizing a particular object or keyword, you can invest your time into creating your custom models to train for the object or keyword detection.

Can I use Power Apps and AI Builder for production?

Yes you can. As any tool that does things magically, AI Builder in Power Apps comes with a cost. That does not mean you can't try your ideas out for free.

What will my production app cost?

If you want to go to production with Power Apps, it is a good idea to consider the costs. Thankfully there is an app for that. AI Builder Calculator lets you input what AI tools you will need and how many users will be accessing your app's AI features and gives you the price it will cost you.

What are preview features?

AI Builder was released for public preview on June 10, 2019 in Europe and the United States. Preview release features are subject to change and may have restricted functionality before the official release for general availability. Preview releases are not meant for production use. You can try them out and influence the final product by giving feedback.

The General Availability (GA) release will occur in a phased manner, with some features remaining in preview status while others are released for GA. You can check out the release status on the AI Builder documentation.

What is Object Detection?

AI Builder Object detection is an AI model that you can train to detect objects in pictures. AI models usually require that you provide samples of data to train before you are able to perform predictions. Prebuilt models are pre-trained by using a set of samples that are provided by Microsoft, so they are instantly ready to be used in predictions.

Object detection can detect up to 500 different objects in a single model and support JPG, PNG, BMP image format or photos through the Power Apps control.

How to try out Object Detection capabilities?

You can try out and see how object detection works before having to create and accounts or apps yourself on the Azure Computer Vision page.

What can you do with Object Detection?

Object counting and inventory management
Brand logo recognition
Wildlife animal recognition

How to detect objects from images?

To start creating your AI model for your app, sign in to Power Apps and click on AI Builder on the left hand menu. Select Object Detection from the "Refine Model for your business needs" option.

Name your new AI model with a unique name. Select Common Objects and proceed to next section.

Name the objects that you are going to detect.

Upload images that contain the object you will detect. To start with you can upload 15 images for each object.

Make sure each object has approximately the same amount of images tagged. If you have more examples of one object, the training data will be likely to detect that object when it is not.
Tag your objects by selecting a square that your object is in and choosing the name of the object.

Once you are done, choose Done Tagging and Train. Training process will take some time.
If you choose to not use an image or clear any tags, you can do that at any time by going back to your model under the AI Builder on the left hand side menu and choose your model and choose edit.

AI Builder will give you a Performance score over 100 and a way to quickly test your model before publishing. You can edit your models and retrain to improve your performance. Next section will give you some best practices to improve your performance.

How to Improve Your Custom Model Performance?

Getting the best model performance for your business can be an iterative process. Results can vary depending on the customizations you make to the model, and the training data you provide.

‌

To help facilitate this process, AI Builder allows you to have multiple versions of your model so you can use your model and continue to improve it at the same time.

What are some best practices for training for object detection?

Use diverse images to train with all possible use cases. For example if you are training your data to detect a VR headset, use images of the headset used in different environments as well as the out of the box images. If you only train with images with people wearing the headset, your model would not recognize images of the same device when it is in its box.

Use images with a variety of backgrounds. Photos in context are better than photos in front of neutral backgrounds.

Use training images that have different lighting. For example, include images taken with flash, high exposure, and so on.

Use images of objects in varied sizes. Different sizing helps the model generalize better.
Use images taken from different angles. If all your photos are from a set of fixed cameras such as surveillance cameras, assign a different label to each camera. This can help avoid modeling unrelated objects such as lampposts as the key feature. Assign camera labels even if the cameras capture the same objects.

How to share your models?

By default, only you can see the models you create and publish. This feature allows you to test them and use them within apps and flows without exposing them.

If you want others to use your model, you can share it with specific users, groups, or your whole organization.

How to use your Custom Vision model in a Power App?

Once you are happy with your model's performance, you can add it to a new app by choosing Use model and New app.

You will be redirected to Power App editor and an Object Detection component that uses your model will be added automatically. In the editor, you can add new pages to navigate, design and customize your pages.

Once you are happy with the design, you can publish and share your app. You can use your new app by downloading Power Apps from Apple, Android or Microsoft stores. Once you sign in, your app will be listed in the mobile Power Apps.

What's next?

Now you have your app's prototype, you can add more features, get feedback and test your app.

Should I keep using my power app or rebuild it?

When your needs change, you can consider refactoring your application to a serverless backend and a custom built UI. If the app is working fine for you and your users, you can continue using and improving overtime using Power Apps.

‌

What would be the changes that require the upgrade? There are two possibilities for the changed requirements for your app:

Feature
Budget

How to create a custom feature for Power Apps?

Ready made tools are always limited to the features the product team decides to include. If you are writing custom code, you can add any feature that you need. Thankfully, for the features that are not implemented yet, it is always possible to author a custom connector that you can use with or without Power Apps.

A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows.

Check out the list of Power Apps Connectors and how to build a custom connector yourself.

How to compare costs for Power Apps and Logic Apps?

Once you start using your app, you will have a better idea about the number of users accessing AI capabilities and the number of images that you need to train. You can use AI Builder Cost Calculator and Logic App Cost Calculator to compare options. You can check any other service price through Azure Product Cost Calculator.

Additional Resources

Leave a comment below for your AI application use cases and the tutorials you would like to see.

Mask detection now available in preview via Azure Cognitive Services

vaparth — Mon, 08 Mar 2021 21:11:35 GMT

COVID-19 spread has changed our day-to-day life in unprecedented ways. Organizations around the world are taking action to, contain and help prevent further spread of the disease by using AI technologies like computer vision, help ensure the safety of the employees and customers.

Azure Cognitive Services now, provides Mask detection functionality, to assist application developers in building solutions that can help monitor and contain the spread. Mask detection can be deployed anywhere, from the cloud leveraging Face service, to the edge using Spatial analysis service.

Mask detection on the edge

Spatial analysis is, a capability of Computer Vision, part of Azure Cognitive Services. This capability understands people’s movements in a physical space by analyzing real-time video, significantly increasing efficiency, and providing valuable insights for enabling various scenarios including,

Counting people in a space for maximum occupancy
Understanding the distance between people for social distancing measures
Determining customer footfall such as in retail spaces
Determining wait time in a checkout line
Determining trespassing in protected areas

Spatial analysis can now detect whether a person is wearing a protective face covering or not. With this new capability, businesses can leverage insights to build applications that can measure safety and enhance compliance. For example, a business can aggregate data of percentage of people wearing masks in a physical space to improve compliance measures. To help ensure the safety of people working in a given space, mask detection can also be used to notify when a person may accidentally enter the space without a face mask.

Mask detection can be enabled for the following spatial analysis operations - personcount, personcrossingline and personcrossingpolygon. The classifier model can be enabled by configuring the ‘ENABLE_FACE_MASK_CLASSIFIER’ parameter to True, this is disabled by default. The attributes, face_mask or face_noMask, will be returned as metadata with confidence score for each person detected in the video stream.

Face mask and Person detection with Spatial analysis

Spatial analysis operations provide a real-time video analysis pipeline on new and existing RTSP cameras. The deployment of the spatial analysis container on edge devices is facilitated by Azure IoT Hub. When video is streamed and processed by spatial analysis, the container emits AI insight events about people’s movement which in turn are sent to Azure IoT Hub as IoT telemetry. From IoT Hub you can create various routes to other Azure services and build your business solutions.

Spatial analysis container deployment with Azure IoT

The events from each operation are egressed to Azure IoT Hub on JSON format. Sample JSON for an event output by cognitiveservices.vision.spatialanalysis-personcount operation.

{ "events": [ { "id": "3733eb36935e4d73800a9cf36185d5a2", "type": "personLineEvent", "detectionIds": [ "90d55bfc64c54bfd98226697ad8445ca" ], "properties": { "trackingId": "90d55bfc64c54bfd98226697ad8445ca", "status": "CrossLeft" }, "zone": "doorcamera" } ], "sourceInfo": { "id": "camera_id", "timestamp": "2020-08-24T06:06:53.261Z", "width": 608, "height": 342, "frameId": "1340", "imagePath": "" }, "detections": [ { "type": "person", "id": "90d55bfc64c54bfd98226697ad8445ca", "region": { "type": "RECTANGLE", "points": [ { "x": 0.491627341822574, "y": 0.2385801348769874 }, { "x": 0.588894994635331, "y": 0.6395559924387793 } ] }, "confidence": 0.9005028605461121, "metadata": { "attributes": { "face_Mask": 0.99 } } } ], "schemaVersion": "1.0" }

Learn how to build business applications with spatial analysis, follow these instructions to deploy a sample Azure Web Application that presents a live view of people counting events in a physical space. You can modify this app with other spatial analysis operations and make modifications based on the event output of the container.

Mask detection in the cloud

Mask detection is also available through the Face Detection cloud endpoint in Azure Cognitive Face API Service. This capability analyses images, detects one or more human faces along with attributes for each face in the image. Face mask attribute is available with the latest detection_03 model, along with additional attribute “noseAndMouthCovered” that provides insight about whether the mask covers both the nose and mouth.

To leverage the latest mask detection capability, users need to specify the detection model in the API request - assign the model version with the detectionModel parameter to detection_03. Refer to How to specify a detection model to learn more about the capabilities of each detection model and sample code to call it.

Face mask detection with Face Service

Detection_03 API response with face mask attribute:

{ "faceId": "eee58bd3-0b54-4f48-9a96-c9c60724ee80", "faceRectangle": { "top": 171, "left": 1212, "width": 79, "height": 125 }, "faceAttributes": { "mask": { "type": "faceMask", "noseAndMouthCovered": “true” } }, { "faceId": "2d83c3c1-7266-4b84-b47b-a65645368021", "faceRectangle": { "top": 364, "left": 600, "width": 66, "height": 80 }, "faceAttributes": { "mask": { "type": "faceMask", "noseAndMouthCovered": “true” } },

Responsible AI and Deployment Guide

Microsoft’s principled approach enables developers to build rich solutions while ensuring responsible use.

Responsible deployment recommendations for spatial analysis is provided in accordance with Microsoft Responsible AI Principles: fairness, reliability & safety, privacy & security, inclusiveness, transparency, and human accountability. For general guidelines and specific recommendations for height, angle, and camera-to-focal-point-distance, see Camera placement guide. And refer to Face API Transparency Note to get clear guidance on use of facial recognition to help ensure it fits your goals and achieve accurate results.

Get Started

Learn more with our documentation Spatial analysis, QuickStart: Face Service

Follow the tutorial to Create a People Counting Web App, Detect faces using the .NET SDK

Learn about Azure Stack Edge and Azure IoT Hub

Put AI into practice with Microsoft's Azure AI Hackathon

wmendoza — Mon, 08 Mar 2021 17:52:41 GMT

If you’ve been looking for a reason to get started with AI to solve a particular problem or use case, look no further! We invite you to put your skills to the test and apply Azure AI to a new or existing project. As you may have seen in an earlier post by Anand Raman, we have been hosting an Azure AI hackathon in which you can submit your project and be eligible to win prizes. Developers of all backgrounds and skill levels are welcome to join and submit any form of AI project, whether using Azure AI to enhance existing apps with pre-trained machine learning (ML) models with Cognitive Services or building your own custom ML models with Azure Machine Learning.

AI Hackathon homepage

If you’re interested in participating, visit the Azure AI Hackathon page to get started. The deadline is April 5^th so you still have time to build and submit a project! Use one or more of the following Azure AI services to build a new project or update an existing project: Azure Machine Learning, Azure Cognitive Services, Bot Framework and Azure Cognitive Search. Projects may integrate with other Azure services, open source technologies (including but not limited to frameworks, libraries, and APIs) and physical hardware of your choice.

If you’re looking for a little inspiration, below are a few examples of past winners:

2019 First Place– Trashé

Trashe Smarter Recycling solution

Submitted by Nathan Glover and Stephen Mott, Trashé is a SmartBin that aims to help people make more informed recycling decisions. While the idea is super impactful, it’s even more powerful when you see it in action- not just the intelligence, but the end-to-end scenario of how it can be applied in a real-world environment.

This team used many Azure services to connect the hardware, intelligence, and presentation layers—you can see this is a well-researched architecture that is reusable in multiple scenarios. Azure Custom Vision was a great choice in this case, enabling the team create a well performing model with very little training data. The more we recycle, the better the model will get. It was great to see the setup instructions included to help build unique versions of Trashé so users can contribute to helping the environment by recycling correctly within their local communities—this community approach is incredibly scalable.

2019 Second Place- AfriFarm

Niza Siwale’s app recognizes crop diseases from images using Azure Machine Learning service and publishes the findings so anyone can track disease breakouts. This also provides a real-time update for government agencies to act quickly and provide support to affected communities. As quoted by Niza, this project has an incredible reach to a possible 200 million farmers whose livelihoods depend on farming in Africa.

Creating a simple Android application where users can take photos of crops to analyze if each farmer is getting information when they need it, users can also contribute their own findings back to the community around them keeping everyone more informed and connected. Using the popular Keras framework along with the Azure Machine Learning service, this project built and deployed a good plant disease recognition model which could be called from the application. Any future work or improved versions of models can be monitored and deployed in a development cycle. From this, the progression of the model can be tracked over time.

2019 Third Place- Water Level Anomaly detector

Roy Kincaid’s project identifies drastic changes in water levels using an ultrasonic sensor that could be useful for detecting potential floods and natural disasters. This information can then be used to provide adequate warning for people to best prepare to major changes in their environment. Water Level Anomaly Detector could also be beneficial for long-term analysis of the effects of climate change. This is another great example of an end-to-end intelligent solution.

Roy is well skilled in the hardware and connection parts of this project, so it was brilliant to see the easy integration of the Anomaly Detector API from Azure Cognitive Services and to hear how quickly Roy could start using the service. Many IoT scenarios have a similar need for detecting rates and levels; in fact, Roy had hinted at coffee level detector in the future. In a world where we all want to do our part to help the environment, it’s a great example of how monitoring enables us to measure changes over time and be alerted when issues arise.

These are just 3 of the past winners and submissions. For more inspiration, visit our gallery of past submissions.

Resources to get started

Sign up for the Azure AI hackathon
Visit our AI for Developers resources page for tutorials and a curated 30-day learning journey
Visit our ML for Data Scientists resources page for tutorials and a curated 30-day learning journey

Form Recognizer now reads more languages, processes IDs and invoices, trains on tables, and more

christina-lee — Thu, 04 Mar 2021 23:59:38 GMT

Documents contain invaluable information powering core business processes. Extracting information from these documents with minimum manual intervention helps bolster organizational efficiency and productivity. As more and more processes and workflows get automated, the need for new features to help extract text and structures increases.

Today, we are excited to announce newest updates to Form Recognizer that will be available on March 15, 2021.

What’s New?

Form Recognizer v2.1 public preview 3 will be available on March 15, 2021, and it will include:

Extract data from invoices

Invoices are complex documents that vary in structure and contain data that is vital to organizations business processes. One of the most challenging tasks in extracting data from invoices is extracting data from invoices line items. The Form Recognizer Invoice API now supports line-item extraction, it also extracts now the full line item and its parts – description, amount, quantity, product ID, date and more. With a simple API \ SDK call you can extract all the data from your invoices – text, tables, key value pairs and line items.

Figure 1 Line items are extracted from invoices

Extract data from IDs

The new pre-built ID model enables customers to take worldwide passports and U.S. drivers license and return structured data representing the information available on the IDs. The new ID API extract the text and values of interest from IDs such as document number, last name, first name, date of expiration, country and more.

Figure 2 Pre-built ID model can extract information from passports and US drivers licenses

Supervised table labeling and training, empty-value labeling

In addition the Form Recognizer state of the art deep learning automatic table extraction capabilities it now also enables customer to train and label tables. This new release includes the ability to label line items/tables (dynamic and fixed) and train a custom model to extract key-value pairs and line items. Once a model is trained and documents are analyzed using this model the new line items will be extracted as part of the JSON output in the documentResults section.

Figure 3 Label tables in your training dataset

In addition to labeling tables, you can now label empty values and regions; if some documents in your training set do not have values for some fields, you can use this so that your model will know to extract values properly from analyzed documents.

Natural reading order, handwriting classification, and page selection

With this update, you can choose to get the text line outputs in the natural reading order instead of the default left-to-right and top-to-bottom ordering. Use the new readingOrder query parameter to “natural” value for a more human-friendly reading order output as shown in the following example. Note the first column’s text lines output in order before listing the second, and the third, column.

In addition, for Latin languages, Form Recognizer will classify Latin-languages only text lines as handwritten style or not and give a confidence score, as seen below.

Furthermore, when analyzing a multi-page PDF or TIFF, you can now specify which pages you want to analyze.

Pre-built Receipt model quality improvements

This new update includes a number of quality improvements for the pre-built Receipt model, especially around line item extraction.

Our Customers & Partners

AvidXchange has developed an account payable automation solution leveraging Form Recognizer. “By partnering with Microsoft, we’re able to deliver an accounts payable automation solution for the middle market that’s truly powered by machine learning,” said Chris Tinsley, Chief Technology Officer at AvidXchange. “Our customers will benefit from faster invoice processing times and increased accuracy so we can help ensure their suppliers are paid the right amount, at the right time.”

WEX has developed a tool to process Explanation of Benefits documents using Form Recognizer. Matt Dallahan, Senior Vice President of Product Management and Strategy, said “The technology is truly amazing. I was initially worried that this type of solution would not be feasible, but I soon realized that the Form Recognizer can read virtually any document with accuracy.”

GEP has developed an invoice processing solution for a client using Form Recognizer. “At GEP, we are seeing AI and automation make a profound impact on procurement and the supply chain. By combining our AI solution with Microsoft Form Recognizer, we automated the processing of 4,000 invoices a day for a client, saving them tens of thousands of hours of manual effort, while improving accuracy, controls and compliance on a global scale,” said Sarateudu Sethi, GEP’s Vice President of Artificial Intelligence.

“At Cross Masters, using cutting-edge AI technologies is not only a passion, it is an essential part of our work culture that requires continuous innovation. One of our latest success stories is automation of manual paperwork, required to process thousands of invoices. Thanks to Microsoft Form Recognizer’s AI engine we were able to develop a unique customized solution that provides to our clients market insights from large set of collected invoices. What we find the most convenient is human beating extraction quality and continuous introduction of new features, such as model composing or table labelling. This assures our client’s market advantage and helps our product to be the best-in-class solution” Jan Hornych, Head of Marketing Automation, Cross Masters

Try out Form Recognizer

To get started with Form Recognizer, please login to the Azure Portal to create a Form Recognizer resource. Once your resource is created, you can start exploring Form Recognizer, with the improvements mentioned above coming on March 15. You can learn more about Form Recognizer here.

Introducing semantic search: Bringing more meaningful results to Azure Cognitive Search

Luis Cabrera-Cordon — Fri, 05 Mar 2021 00:00:36 GMT

A few years ago, it became clear to our team that AI could bring value to our customers, from improvements in ingestion to data exploration. We knew we had a lot of these valuable assets around Microsoft, so our team set out on a mission to bring as much “intelligence” as we could to the product then known as Azure Search.

In the first phase of this mission, we took on "unsearchable" content; about 80% of business relevant data is in unstructured formats such as PDFs, PowerPoints, Word documents, JPEGs, CSVs, etc. We added AI powered enrichments to our ingestion process, enabling the ability to extract structure, insights and transform information from your data. These capabilities were well received by our customers, culminating in a product rebrand as "Azure Cognitive Search".

I am happy to announce that in our continuation of this journey, we are bringing state of the art AI capabilities to the “head” of our product, the core search sub-system. In partnership with the Bing team, we have integrated their semantic search investments (100s of development years and millions of dollars in compute time) into our query infrastructure, effectively enabling any developer to leverage this investment over searchable content that you own and manage. We believe semantic search on Azure Cognitive Search offers the best combination of search relevance, developer experience, and cloud service capabilities available on the market.

This post explains what new capabilities are available to you and how you can get started today. I would also encourage you to look at the post called “Bing’s AI behind semantic search” that goes deeper into the Bing technology that made semantic search possible.

Today, we are launching several exciting semantic search features in a public preview:

Semantic Ranking

Customers have grown accustomed to using natural language queries in web search engines, but these queries usually do not perform as well when using a traditional keyword-based retrieval approach with ranking only based on term frequencies. To demonstrate this, consider what happens when a customer types a query like “how to add a user in Office” in the Microsoft documentation. For this purpose, we loaded all the Microsoft documentation dataset into Azure Cognitive Search so that we could compare the results between the default lexical based ranking algorithm and the semantic ranking algorithm.

Traditional retrieval and ranking approach

The default ranker (BM25) uses words as discrete units and predicts relevance by using the frequencies of terms in the corpus. BM25 works well when searching for keywords, but it struggles to find the most relevant documents when issuing a natural language query.

Note that the results do meet the lexical frequency requirements. For instance, inspecting the top document “Troubleshoot user errors with Office Add-ins” shows that there are lot of mentions of terms like “office”, “user”, “add” and “how to” in the document – but unfortunately the article is not providing the information we meant to query for.

Semantics-based ranking

With the release of semantic search, now we can enable a ranking algorithm that will use deep neural networks to rank the articles based on how “meaningful” they are relative to the query. Internally, this is a ranker that is applied on top of the results returned by the BM25-based ranker. Using semantic search capabilities, these are the top results for our query:

I read the content of the top-document called “Add users and assign licenses at the same time”, and it is clear that this is exactly the document I need! Semantic search made this connection even though the title and content are not syntactically close to my query.

Semantic Answers

In the previous example, the title of the document by itself did not make it very easy for me to catch if that was a relevant document or not. I still had to read it to find the snippet in the documentation that told me how to add a user to Office.

The good news is that now you can also get semantic answers! It is one of my favorite features; it uses an AI model that extracts relevant passages from the top documents, and then ranks them on their likelihood of being an answer to the query. If we find a passage that has a high likelihood of answering the question, we will promote it as a semantic answer.

This is what it looks like, in this case. Note that we even leveraged a model from Bing to provide highlights for the most relevant section in the semantic answer.

Semantic Captions

Similarly, we can extract the most relevant section of each document returned so you can quickly skim through the results and see if they have the content that you care about; making it easier for you to triage the results briefly and go deeper into the ones that you think are relevant given your context.

Get started today!

Using semantic search is easy. After you sign up for the preview at http://aka.ms/semanticpreview, all you need to do is change your query parameters as part of the request as shown below. Note that there is no need to re-index any of your content!

Query parameter
queryType	Set to “semantic” to indicate that you would like semantic ranking and answers. Other values supported: “simple” and “full”.
searchFields	Ordered list of fields that semantic ranking should be applied on. If you have a title or a short field that describes your document, we recommend that to be your first field. Follow that by the url (if any), then the body of the document, and then any other relevant fields.
queryLanguage	“en-us” is the only supported value today. We will be adding more languages soon. Stay tuned.
speller	Set to “lexicon” if you would like spell correction to occur on the query terms. Otherwise set to “none”.
answers	Set to “extractive” if you would like to get extractive answers. Otherwise set to “none”.

Sample Query

POST https://[service name].search.windows.net/indexes/[index name]/docs/search?api-version=2020-06-30-preview { "search": " Where was Alan Turing born?", "queryType": "semantic", "searchFields": "title,url,body", "queryLanguage": "en-us", "speller": "lexicon", "answers": "extractive" }

Sample Response

{ "@search.answers": [ { "key": "a1234", "text": "Turing was born in Maida Vale, London, while his father, Julius…", "highlights": " Turing was born in Maida Vale, London , while …", "score": 0.87802511 } ], "value": [ { "@search.score": 51.64714, "@search.rerankerScore": 1.9928148165345192, "@search.captions": [ { "text": " Alan Mathison Turing, (born June 23, 1912, London, England—died June 7, 1954…", "highlights": " Alan Mathison Turing, (born June 23, 1912, London, England—died June…", } ], "id": "b5678", "body": "…" }, … ] }

Learn more about Semantic Search in our documentation.

I am personally super excited about these new capabilities, the efficiencies that they will bring to you, and the progression of our vision to bring the best AI capabilities at Microsoft to Azure developers!

Luis Cabrera – on behalf of the Azure Cognitive Search team

Customers & Partners

Case Study: PPL Electric Utilities Corporation, a utilities company, is working with Neudesic to create a web application with Azure Cognitive search to empower its field workers to find the most relevant information wherever they are.

Case Study: Howden teamed up with OrangeNXT to further improve Smart Records using key elements of their digitalNXT Search, a fully managed cloud solution powered by Azure Cognitive Search.

Call to Action

Preview sign-up form

Cognitive Search Team Ask Me Anything (March 10 2021)

Resources

Documentation

Mechanics Video

Bing science behind semantic search

Ombromanie: Creating Hand Shadow stories with Azure Speech and TensorFlow.js Handposes

jelooper — Thu, 25 Feb 2021 19:09:52 GMT

Have you ever tried to cast hand shadows on a wall? It is the easiest thing in the world, and yet to do it well requires practice and just the right setup. To cultivate your #cottagecore aesthetic, try going into a completely dark room with just one lit candle, and casting hand shadows on a plain wall. The effect is startlingly dramatic. What fun!

Even a tea light suffices to create a great effect

In 2020, and now into 2021, many folks are reverting back to basics as they look around their houses, reopening dusty corners of attics and basements and remembering the simple crafts that they used to love. Papermaking, anyone? All you need is a few tools and torn up, recycled paper. Pressing flowers? All you need is newspaper, some heavy books, and patience. And hand shadows? Just a candle.

This TikTok creator has thousands of views for their handshadow tutorials

But what's a developer to do when trying to capture that #cottagecore vibe in a web app?

High Tech for the Cottage

While exploring the art of hand shadows, I wondered whether some of the recent work I had done for body poses might be applicable to hand poses. What if you could tell a story on the web using your hands, and somehow save a video of the show and the narrative behind it, and send it to someone special? In lockdown, what could be more amusing than sharing shadow stories between friends or relatives, all virtually?

Hand shadow casting is a folk art probably originating in China; if you go to tea houses with stage shows, you might be lucky enough to view one like this!

A Show Of Hands

When you start researching hand poses, it's striking how much content there is on the web on the topic. There has been work since at least 2014 on creating fully articulated hands within the research, simulation, and gaming sphere:

MSR throwing hands

There are dozens of handpose libraries already on GitHub:

There are many applications where tracking hands is a useful activity:

• Gaming
• Simulations / Training
• "Hands free" uses for remote interactions with things by moving the body
• Assistive technologies
• TikTok effects :trophy:
• Useful things like Accordion Hands apps

One of the more interesting new libraries, handsfree.js, offers an excellent array of demos in its effort to move to a hands free web experience:

Handsfree.js, a very promising project

As it turns out, hands are pretty complicated things. They each include 21 keypoints (vs PoseNet's 17 keypoints for an entire body). Building a model to support inference for such a complicated grouping of keypoints has provenn challenging.

There are two main libraries available to the web developer when incorporating hand poses into an app: TensorFlow.js's handposes, and MediaPipe's. HandsFree.js uses both, to the extent that they expose APIs. As it turns out, neither TensorFlow.js nor MediaPipe's handposes are perfect for our project. We will have to compromise.

TensorFlow.js's handposes allow access to each hand keypoint and the ability to draw the hand to canvas as desired. HOWEVER, it only currently supports single hand poses, which is not optimal for good hand shadow shows.
MediaPipe's handpose models (which are used by TensorFlow.js) do allow for dual hands BUT its API does not allow for much styling of the keypoints so that drawing shadows using it is not obvious.

One other library, fingerposes, is optimized for finger spelling in a sign language context and is worth a look.

Since it's more important to use the Canvas API to draw custom shadows, we are obliged to use TensorFlow.js, hoping that either it will soon support multiple hands OR handsfree.js helps push the envelope to expose a more styleable hand.

Let's get to work to build this app.

Scaffold a Static Web App

As a Vue.js developer, I always use the Vue CLI to scaffold an app using vue create my-app and creating a standard app. I set up a basic app with two routes: Home and Show. Since this is going to be deployed as an Azure Static Web App, I follow my standard practice of including my app files in a folder named app and creating an api folder to include an Azure function to store a key (more on this in a minute).

In my package.json file, I import the important packages for using TensorFlow.js and the Cognitive Services Speech SDK in this app. Note that TensorFlow.js has divided its imports into individual packages:

"@tensorflow-models/handpose": "^0.0.6",
"@tensorflow/tfjs": "^2.7.0",
"@tensorflow/tfjs-backend-cpu": "^2.7.0",
"@tensorflow/tfjs-backend-webgl": "^2.7.0",
"@tensorflow/tfjs-converter": "^2.7.0",
"@tensorflow/tfjs-core": "^2.7.0",
...
"microsoft-cognitiveservices-speech-sdk": "^1.15.0",

Set up the View

We will draw an image of a hand, as detected by TensorFlow.js, onto a canvas, superimposed onto a video suppled by a webcam. In addition, we will redraw the hand to a second canvas (shadowCanvas), styled like shadows:

<div id="canvas-wrapper column is-half">
<canvas id="output" ref="output"></canvas>
    <video
        id="video"
        ref="video"
        playsinline
        style="
          -webkit-transform: scaleX(-1);
           transform: scaleX(-1);
           visibility: hidden;
           width: auto;
           height: auto;
           position: absolute;
         "
    ></video>
 </div>
 <div class="column is-half">
    <canvas
       class="has-background-black-bis"
       id="shadowCanvas"
       ref="shadowCanvas"
     >
    </canvas>
</div>

Load the Model, Start Keyframe Input

Working asynchronously, load the Handpose model. Once the backend is setup and the model is loaded, load the video via the webcam, and start watching the video's keyframes for hand poses. It's important at these steps to ensure error handling in case the model fails to load or there's no webcam available.

async mounted() {
    await tf.setBackend(this.backend);
    //async load model, then load video, then pass it to start landmarking
    this.model = await handpose.load();
    this.message = "Model is loaded! Now loading video";
    let webcam;
    try {
      webcam = await this.loadVideo();
    } catch (e) {
      this.message = e.message;
      throw e;
    }

    this.landmarksRealTime(webcam);
  },

Setup the Webcam

Still working asynchronously, set up the camera to provide a stream of images

async setupCamera() {
      if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
        throw new Error(
          "Browser API navigator.mediaDevices.getUserMedia not available"
        );
      }
      this.video = this.$refs.video;
      const stream = await navigator.mediaDevices.getUserMedia({
        video: {
          facingMode: "user",
          width: VIDEO_WIDTH,
          height: VIDEO_HEIGHT,
        },
      });

      return new Promise((resolve) => {
        this.video.srcObject = stream;
        this.video.onloadedmetadata = () => {
          resolve(this.video);
        };
      });
    },

Design a Hand to Mirror the Webcam's

Now the fun begins, as you can get creative in drawing the hand on top of the video. This landmarking function runs on every keyframe, watching for a hand to be detected and drawing lines onto the canvas - red on top of the video, and black on top of the shadowCanvas. Since the shadowCanvas background is white, the hand is drawn as white as well and the viewer only sees the offset shadow, in fuzzy black with rounded corners. The effect is rather spooky!

async landmarksRealTime(video) {
      //start showing landmarks
      this.videoWidth = video.videoWidth;
      this.videoHeight = video.videoHeight;

      //set up skeleton canvas
      this.canvas = this.$refs.output;
      ...

      //set up shadowCanvas
      this.shadowCanvas = this.$refs.shadowCanvas;
      ...

      this.ctx = this.canvas.getContext("2d");
      this.sctx = this.shadowCanvas.getContext("2d");

      ...

      //paint to main

      this.ctx.clearRect(0, 0, this.videoWidth, 
  this.videoHeight);
      this.ctx.strokeStyle = "red";
      this.ctx.fillStyle = "red";
      this.ctx.translate(this.shadowCanvas.width, 0);
      this.ctx.scale(-1, 1);

      //paint to shadow box

      this.sctx.clearRect(0, 0, this.videoWidth, this.videoHeight);
      this.sctx.shadowColor = "black";
      this.sctx.shadowBlur = 20;
      this.sctx.shadowOffsetX = 150;
      this.sctx.shadowOffsetY = 150;
      this.sctx.lineWidth = 20;
      this.sctx.lineCap = "round";
      this.sctx.fillStyle = "white";
      this.sctx.strokeStyle = "white";

      this.sctx.translate(this.shadowCanvas.width, 0);
      this.sctx.scale(-1, 1);

      //now you've set up the canvases, now you can frame its landmarks
      this.frameLandmarks();
    },

For Each Frame, Draw Keypoints

As the keyframes progress, the model predict new keypoints for each of the hand's elements, and both canvases are cleared and redrawn.

      const predictions = await this.model.estimateHands(this.video);

      if (predictions.length > 0) {
        const result = predictions[0].landmarks;
        this.drawKeypoints(
          this.ctx,
          this.sctx,
          result,
          predictions[0].annotations
        );
      }
      requestAnimationFrame(this.frameLandmarks);

Draw a Lifelike Hand

Since TensorFlow.js allows you direct access to the keypoints of the hand and the hand's coordinates, you can manipulate them to draw a more lifelike hand. Thus we can redraw the palm to be a polygon, rather than resembling a garden rake with points culminating in the wrist.

Re-identify the fingers and palm:

     fingerLookupIndices: {
        thumb: [0, 1, 2, 3, 4],
        indexFinger: [0, 5, 6, 7, 8],
        middleFinger: [0, 9, 10, 11, 12],
        ringFinger: [0, 13, 14, 15, 16],
        pinky: [0, 17, 18, 19, 20],
      },
      palmLookupIndices: {
        palm: [0, 1, 5, 9, 13, 17, 0, 1],
      },

...and draw them to screen:

    const fingers = Object.keys(this.fingerLookupIndices);
      for (let i = 0; i < fingers.length; i++) {
        const finger = fingers[i];
        const points = this.fingerLookupIndices[finger].map(
          (idx) => keypoints[idx]
        );
        this.drawPath(ctx, sctx, points, false);
      }
      const palmArea = Object.keys(this.palmLookupIndices);
      for (let i = 0; i < palmArea.length; i++) {
        const palm = palmArea[i];
        const points = this.palmLookupIndices[palm].map(
          (idx) => keypoints[idx]
        );
        this.drawPath(ctx, sctx, points, true);
      }

With the models and video loaded, keyframes tracked, and hands and shadows drawn to canvas, we can implement a speech-to-text SDK so that you can narrate and save your shadow story.

To do this, get a key from the Azure portal for Speech Services by creating a Service:

You can connect to this service by importing the sdk:

import * as sdk from "microsoft-cognitiveservices-speech-sdk";

...and start audio transcription after obtaining an API key which is stored in an Azure function in the /api folder. This function gets the key stored in the Azure portal in the Azure Static Web App where the app is hosted.

async startAudioTranscription() {
      try {
        //get the key
        const response = await axios.get("/api/getKey");
        this.subKey = response.data;
        //sdk

        let speechConfig = sdk.SpeechConfig.fromSubscription(
          this.subKey,
          "eastus"
        );
        let audioConfig = sdk.AudioConfig.fromDefaultMicrophoneInput();
        this.recognizer = new sdk.SpeechRecognizer(speechConfig, audioConfig);

        this.recognizer.recognized = (s, e) => {
          this.text = e.result.text;
          this.story.push(this.text);
        };

        this.recognizer.startContinuousRecognitionAsync();
      } catch (error) {
        this.message = error;
      }
    },

In this function, the SpeechRecognizer gathers text in chunks that it recognizes and organizes into sentences. That text is printed into a message string and displayed on the front end.

Display the Story

In this last part, the output cast onto the shadowCanvas is saved as a stream and recorded using the MediaRecorder API:

const stream = this.shadowCanvas.captureStream(60); // 60 FPS recording
      this.recorder = new MediaRecorder(stream, {
        mimeType: "video/webm;codecs=vp9",
      });
      (this.recorder.ondataavailable = (e) => {
        this.chunks.push(e.data);
      }),
        this.recorder.start(500);

...and displayed below as a video with the storyline in a new div:

      const video = document.createElement("video");
      const fullBlob = new Blob(this.chunks);
      const downloadUrl = window.URL.createObjectURL(fullBlob);
      video.src = downloadUrl;
      document.getElementById("story").appendChild(video);
      video.autoplay = true;
      video.controls = true;

This app can be deployed as an Azure Static Web App using the excellent Azure plugin for Visual Studio Code. And once it's live, you can tell durable shadow stories!

Try Ombromanie here. The codebase is available here

Take a look at Ombromanie in action:

Learn more about AI on Azure
Azure AI Essentials Video covering speech and language
Azure free account sign-up

Responsible Machine Learning with Error Analysis

besmiranushi — Thu, 18 Feb 2021 16:00:00 GMT

Website: ErrorAnalysis.ai

Github repository: https://github.com/microsoft/responsible-ai-widgets/

Machine Learning (ML) teams who deploy models in the real world often face the challenges of conducting rigorous performance evaluation and testing for ML models. How often do we read claims such as “Model X is 90% on a given benchmark.” and wonder what does this claim mean for practical usage of the model? In practice, teams are well aware that model accuracy may not be uniform across subgroups of data and that there might exist input conditions for which the model fails more often. Often, such failures may cause direct consequences related to lack of reliability and safety, unfairness, or more broadly lack of trust in machine learning altogether. For instance, when a traffic sign detector does not operate well in certain daylight conditions or for unexpected inputs, even though the overall accuracy of the model may be high, it is still important for the development team to know ahead of time about the fact that the model may not be as reliable in such situations.

Figure 1 - Error Analysis moves away from aggregate accuracy metrics, exposes the distribution of errors to developers in a transparent way, and enables them to identify & diagnose errors efficiently.

While there exist several problems with current model assessment practices, one of the most obvious is the usage of aggregate metrics to score models on a whole benchmark. It is difficult to convey a detailed story on model behavior with a single number and yet most of the research and leaderboards operate on single scores. At the same time, there may exist several dimensions of the input feature space that a practitioner may be interested in taking a deep dive and ask questions such as “What happens to the accuracy of the recognition model in a self-driving car when it is dark and snowing outside?” or “Does the loan approval model perform similarly for population cohorts across ethnicity, gender, age, and education?”. Navigating the terrain of failures along multiple potential dimensions like the above can be challenging. In addition, in the longer term, when models are updated and re-deployed frequently upon new data evidence or scientific progress, teams also need to continuously track and monitor model behavior so that updates do not introduce new mistakes and therefore break user trust.

To address these problems, practitioners often have to create custom infrastructure, which is tedious and time-consuming. To accelerate rigorous ML development, in this blog you will learn how to use the Error Analysis tool for:

Getting a deep understanding of how failure is distributed for a model.
Debugging ML errors with active data exploration and interpretability techniques.

The Error Analysis toolkit is integrated within the Responsible AI Widgets OSS repository, our starting point to provide a set of integrated tools to the open source community and ML practitioners. Not only a contribution to the OSS RAI community, but practitioners can also leverage these assessment tools in Azure Machine Learning, including Fairlearn & InterpretML and now Error Analysis in mid 2021.

If you are interested in learning more about training model updates that remain backward compatible with their previous selves by minimizing regress and new errors, you can also check out our most recent open source library and tool BackwardCompatibilityML.

Prerequisites

To install the Responsible AI Widgets “raiwidgets” package, in your python environment simply run the following to install the raiwidgets package from pypi. If you do not have interpret-community already installed, you will also need to install this for supporting the generation of model explanations.

pip install interpret-community pip install raiwidgets

Alternatively, you can also clone the open source repository and build the code from scratch:

git clone https://github.com/microsoft/responsible-ai-widgets.git

You will need to install yarn and node to build the visualization code, and then you can run:

yarn install yarn buildall

And install from the raiwidgets folder locally:

cd raiwidgets pip install –e .

For more information see the contributing guide.

If you intend to run repository tests, in the raiwidgets folder of the repository run:

pip install -r requirements.txt

Getting started

This post illustrates the Error Analysis tool by using a binary classification task on income prediction (>50K, <50K). The model under inspection will be trained using the tabular UCI Census Income dataset, which contains both numerical and categorical features such as age, education, number of working hours, ethnicity, etc.

We can call the error analysis dashboard using the API below, which takes in an explanation object computed by one of the explainers from the interpret-community repository, the model or pipeline, a dataset and the corresponding labels (true_y parameter):

ErrorAnalysisDashboard(global_explanation, model, dataset=x_test, true_y=y_test)

For larger datasets, we can downsample the explanation to fewer rows but run error analysis on the full dataset. We can provide the downsampled explanation, the model or pipeline, the full dataset, and then both the labels for the sampled explanation and the full dataset, as well as (optionally) the names of the categorical features:

ErrorAnalysisDashboard(global_explanation, model, dataset=X_test_original_full,true_y=y_test, categorical_features=categorical_features, true_y_dataset=y_test_full)

All screenshots below are generated using a LGBMClassifier with three estimators. You can directly run this example using the jupyter notebooks in our repository.

How Error Analysis works

1. Identification

Error Analysis starts with identifying the cohorts of data with a higher error rate versus the overall benchmark error rate. The dashboard allows for error exploration by using either an error heatmap or a decision tree guided by errors.

Error Heatmap for Error Identification

The view slices the data based on a one- or two-dimensional grid of input features. Users can choose the input features of interest for analysis. The heatmap visualizes cells with higher error with a darker red color to bring the user’s attention to regions with high error discrepancy. This is beneficial especially when the error themes are different in different partitions, which happens frequently in practice. In this error identification view, the analysis is highly guided by the users and their knowledge or hypotheses of what features might be most important for understanding failure.

Figure 2 - While the overall error rate for the dataset is 23.65%, the heatmap reveals that the error rates are visibly higher, up to 83%, for individuals with higher education. Error rates are also higher for males vs. females.

Decision Tree for Error Identification

Very often, error patterns may be complex and involve more than one or two features. Therefore, it may be difficult for developers to explore all possible combinations of features to discover hidden data pockets with critical failure. To alleviate the burden, the binary tree visualization automatically partitions the benchmark data into interpretable subgroups, which have unexpectedly high or low error rates. In other words, the tree leverages the input features to maximally separate model error from success. For each node defining a data subgroup, users can investigate the following information:

Error rate - a portion of instances in the node for which the model is incorrect. This is shown through the intensity of the red color.
Error coverage – a portion of all errors that fall into the node. This is shown through the fill rate of the node.
Data representation - number of instances in the node. This is shown through the thickness of the incoming edge to the node along with the actual total number of instances in the node.

Figure 3 – Decision tree that aims at finding failure modes by separating error instances from success instances in the data. The hierarchical error pattern here shows that while the overall error rate is 23.65% for the dataset, it can be as high as 96.77% for individuals who are married, have a capital gain higher than 4401, and a number of education years higher than 12.

Cohort definition and manipulation

To specialize the analysis and allow for deep dives, both error identification views can be generated for any data cohort and not only for the whole benchmark. Cohorts are subgroups of data that the user may choose to save for later use if they wish to come back to those cohorts for future investigation. They can be defined and manipulated interactively either from the heatmap or the tree. They can also be carried over to the next diagnostical views on data exploration and model explanations.

Figure 4 - Creating a new cohort for further investigation that focuses on individuals who are married and have capital gain lower than 4401.

2. Diagnosis

After identifying cohorts with higher error rates, Error Analysis enables debugging and exploring these cohorts further. It is then possible to gain deeper insights about the model or the data through data exploration and model interpretability.

Debugging the data

Data Explorer: Users can explore dataset statistics and distributions by selecting different features and estimators along the two axes of the data explorer. They can further compare the subgroup data stats with other subgroups or the overall benchmark data. This view can for instance uncover if certain cohorts are underrepresented or if their feature distribution is significantly different from the overall data, hinting therefore to the potential existence of outliers or unusual covariate shift.

Figure 5 - In figure 1 and 2, we discovered that for individuals with a higher number of education years, the model has higher failure rates. When we look at how the data is distributed across the feature “education_num” we can see that a) there are fewer instances for individuals with more than 12 years of education, and b) for this cohort the distribution between lower income (blue) and higher income (orange) is very different than for other cohorts. In fact, for this cohort there exist more people who have an income higher than 50K, which is not true for the overall data.

Instance views: Beyond data statistics, sometimes it is useful to merely just observe the raw data along with labels in a tabular or tile form. Instance views provide this functionality and divide the instances into correct and incorrect tabs. By eyeballing the data, the developer can identify potential issues related to missing features or label noise.

Debugging the model

Model interpretability is a powerful means for extracting knowledge on how a model works. To extract this knowledge, Error Analysis relies on Microsoft’s InterpretML dashboard and library. The library is a prominent contribution in ML interpretability lead by Rich Caruana, Paul Koch, Harsha Nori, and Sam Jenkins.

Global explanations

Feature Importance: Users can explore the top K important features that impact the overall model predictions (a.k.a. global explanation) for a selected subgroup of data or cohort. They can also compare feature importance values for different cohorts side by side. The information on feature importance or the ordering is useful for understanding whether the model is leveraging features that are necessary for the prediction or whether it is relying on spurious correlations. By contrasting explanations that are specific to the cohort with those for the whole benchmark, it is possible to understand whether the model behaves differently or in an unusual way for the selected cohort.

Dependence Plot: Users can see the relationship between the values of the selected feature to its corresponding feature importance values. This shows them how values of the selected feature impact model prediction.

Figure 6 - Global feature explanations for the income prediction model show that marital status and number of education years are the most important features globally. By clicking on each feature, it is possible to observe more granular dependencies. For example, marital statuses like “divorced”, “never married”, “separated”, or “widowed” contribute to model predictions for lower income (<50K). Marital status of “civil spouse” instead contributes to model predictions for higher income (>50K).

Local explanations

Global explanations approximate the overall model behavior. For focusing the debugging process on a given data instance, users can select any individual data points (with correct or incorrect predictions) from the tabular instance view to explore their local feature importance values (local explanation) and individual conditional expectation (ICE) plots.

Local Feature Importance: Users can investigate the top K (configurable K) important features for an individual prediction. Helps illustrate the local behavior of the underlying model on a specific data point.

Individual Conditional Expectation (ICE): Users can investigate how changing a feature value from a minimum value to a maximum value impacts the prediction on the selected data instance.

Perturbation Exploration (what-if analysis): Users can apply changes to feature values of the selected data point and observe resulting changes to the prediction. They can save their hypothetical what-if data points for further comparisons with other what-if or original data points.

Figure 7 - For this individual, the model outputs a wrong prediction, predicting that the individual earns less than 50K, while the opposite is true. With what-if explanations, it is possible to understand how the model would behave if one of the feature values changes. For instance, here we can see that if the individual were 10 years older (age changed from 32 to 42) the model would have made a correct prediction. While in the real world many of these features are not mutable, this sensitivity analysis is intended to further support practitioners with model understanding capabilities.

Other relevant tools

Error Analysis enables practitioners to identify and diagnose error patterns. The integration with model interpretability techniques testifies to the joint power of providing such tools together as part of the same platform. We are actively working towards integrating further considerations into the model assessment experience such as fairness and inclusion (via FairLearn) as well as backward compatibility during updates (via BackwardCompatibilityML).

Our team

The initial work on error analysis started with research investigations on methodologies for in-depth understanding and explanation of Machine Learning failures. Besmira Nushi, Ece Kamar, and Eric Horvitz at Microsoft Research are leading these efforts and continue to innovate with new techniques for debugging ML models. In the past year, our team was extended via a collaboration with the RAI tooling team in the Azure Machine Learning group as well as the Analysis Platform team in Microsoft Mixed Reality. The Analysis Platform team has invested several years of engineering work in building internal infrastructure and now we are making these efforts available to the community as open source as part of the Azure Machine Learning ecosystem. The RAI tooling team consists of Ilya Matiach, Mehrnoosh Sameki, Roman Lutz, Richard Edgar, Hyemi Song, Minsoo Thigpen, and Anup Shirgaonkar. They are passionate about democratizing Responsible AI and have several years of experience in shipping such tools for the community with previous examples on FairLearn, InterpretML Dashboard etc. We also received generous help and expertise along the way from our partners at Microsoft Aether Committee and Microsoft Mixed Reality: Parham Mohadjer, Paul Koch, Xavier Fernandes, and Juan Lema. All marketing initiatives, including the presentation of this blog, were coordinated by Thuy Nguyen.

Big thanks to everyone who made this possible!

Related research

Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure. Besmira Nushi, Ece Kamar, Eric Horvitz; HCOMP 2018. pdf

Software Engineering for Machine Learning: A Case Study. Saleema Amershi, Andrew Begel, Christian Bird, Rob DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, Thomas Zimmermann; ICSE 2019. pdf

Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, Eric Horvitz; AAAI 2019. pdf

An Empirical Analysis of Backward Compatibility in Machine Learning Systems. Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, Eric Horvitz; KDD 2020. pdf

Understanding Failures of Deep Networks via Robust Feature Extraction. Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, Eric Horvitz. arXiv 2020. pdf

Translator announces Document Translation (Preview)

Krishna_Doss — Wed, 17 Feb 2021 21:30:00 GMT

We are announcing Document Translation, a new feature in Azure Translator service which enables enterprises, translation agencies, and consumers who require volumes of complex documents to be translated into one or more languages preserving structure and format in the original document. Document Translation is an asynchronous batch feature offering translation of large documents eliminating limits on input text size. It supports documents with rich content in different file formats including Text, HTML, Word, Excel, PowerPoint, Outlook Message, PDF, etc. It reconstructs translated documents preserving layout and format as present in the source.

Standard translation offerings in the market accept only plain text, or HTML, and limits count of characters in a request. Users translating large documents must parse the documents to extract text, split them into smaller sections and translate them separately. If sentences are split in an unnatural breakpoint it can lose the context resulting in suboptimal translations. Upon receipt of the translation results, the customer has to merge the translated pieces into the translated document. This involves keeping track of which translated piece corresponds to the equivalent section in the original document.

The problem gets complicated when customers want to translate complex documents having rich content. They convert the original file in variety of formats to either .html or .txt file format and reconvert translated content from html or txt files into original document file format. The transformation may result in various issues. The problem gets compounded when customer needs to translate a) large quantity of documents, b) documents in variety of file formats, c) documents while preserving the original layout and format, d) documents into multiple target languages.

Document Translation is an asynchronous offering to which the user makes a request specifying location of source and target documents, and the list of target output languages. Document Translation returns a job identifier enabling the user to track the status of the translation. Asynchronously, Document Translation pulls each document from the source location, recognizes the document format, applies right parsing technique to extract textual content in the document, translates the textual content into target languages. It then reconstructs the translated document preserving layout and format as present in the source documents, and stores translated document in a specified location. Document Translation updates the status of translation at the document level. Document Translation makes it easy for the customer to translate volumes of large documents in a variety of document formats, into a list of target languages thus eliminating all the challenges customers face today and improving their productivity.

Document Translation enables users to customize translation of documents by providing custom glossaries, a custom translation model id built using customer translator, or both as part of the request. Such customization retains specific terminologies and provides domain specific translations in translated documents.

“Translation of documents with rich formatting is a tricky business. We need the translation to be fluent and matching the context, while maintaining high fidelity in the visual appearance of complex documents. Document Translation is designed to address those goals, relieving client applications from having to disassemble and reassemble the documents after translation, making it easy for developers to build workflows that process full documents with a few simple steps.”, said Chris Wendt, Principal Program Manager.

To learn more about Translator and the Document Translation feature in the video below

References

Send your feedback to translator@microsoft.com

Hello, bot! Conversational AI on Microsoft Platform

shwars — Thu, 18 Feb 2021 18:23:54 GMT

During the pandemic, we all found ourselves in isolation, and relying more and more on effective electronic means of communication. The amount of digital conversations increased dramatically, and we need to rely on bots to help us handle some of those conversations. In this blog post, I give brief overview of conversational AI on Microsoft platform and show you how to build a simple educational bot to help students learn.

If you prefer video content, here is a great video to get you started, from our AI Developer Resources page:

Do We Need Bots, and When?

Many people believe that in the future we will be interacting with computers using speech, in the same way we interact between each other. While the future is still vague, we can already benefit from conversational interfaces in many areas, for example:

In user support, which has traditionally been based on interpersonal communication, automated chat-bots can solve a lot of routine problems for the users, leaving human specialists for solving only unusual cases.
During surgical operation, when hands-free interaction is essential. From personal experience, I find it personally more convenient to set morning alarm and “good night” music through voice assistant before going to sleep.
Automating some functions in interpersonal communication. My favorite example is a chat-bot that you can add to a group chat when organizing a party, and it will track how much money each of the participants spent on the preparation.

At the current state of development of conversational AI technologies, a chat bot will not replace a human, and it will not pass Turing test.

In practice, chat bots act as an advanced version of a command line, in which you do not need to know exact commands to perform an action.

Thus, successful bot applications will not try to pretend to be a human, because such behavior is likely to cause some user dissatisfaction in the future. It is one of the responsible conversational AI principles, which you need to consider when designing a bot.

Educational Bots

During the pandemic, one of the areas that is being transformed the most is education. We can envision educational bots that help student answer most common questions, or act as a virtual teaching assistant. In this blog post, I will show you how to create a simple assistant bot that will be able to handle several questions from the field of Geography.

Before we jump to this task, let’s talk about Microsoft conversational AI stack in general, and consider different development options.

Conversational AI Development Stack

When it comes to conversational AI, we can logically think of a conversational agent having two main parts:

Conversational interface handles passing messages from the user to the bot and back. It takes care about communication between user messaging agent (such as Microsoft Teams, Skype or Telegram) and our application logic, and includes code to handle request-response logic.
Intelligent backend adds some AI functionality to your bot, such are recognizing user’s phrases, or finding best possible answer.

A bot can exist without any intelligent backend, but it would not be smart. Still, bots like that are useful for automating simple tasks, such as form filling, or handling some pre-defined workflow.

Here I present slightly simplified view of the whole Bot Ecosystem, but this way it is easier to get the picture.

Conversational Interface: Microsoft Bot Framework and Azure Bot Service

At the heart of conversational interface is Microsoft Bot Framework - an open-source development framework (with source code available on GitHub), which contains useful abstractions for bot development. The main idea of Bot Framework is to abstract communication channel, and develop bots as web endpoints that asynchronously handle request-response communication.

Decoupling of bot logic and communication channel allows you to develop bot code once, and then connect it easily to different platforms, such as Skype, Teams or Telegram. Omnichannel bots are now made simple!

Bot Framework SDK supports primarily C#, Node.js, Python and Java, although C# or Node.js are highly recommended.

To host bots developed with Bot Framework on Azure, you use Azure Bot Service. It hosts bot logic itself (either as web application, or azure function), as well as allows you to declaratively define physical channels that your bot will be connected to. You can connect your bot to Skype or Telegram through Azure Portal with a few simple steps.

Intelligent Backend: LUIS and QnA Maker

Many modern bots support some form of natural language interaction. To do it, the bot needs to understand the user’s phrase, which is typically done through intent classification. We define a number of possible intents or actions that the bot can support, and then map an input phrase to one of the intents.

This mapping is typically done using a neural network trained on some dataset of sample phrases. To take away the complexity of training your own neural network model, Microsoft provides Language Understanding Intelligent Service, or LUIS, which allows you to train a model either through web interface, or an API.

In addition to intent classification, LUIS also performs named entity recognition (or NER). It can automatically extract some entities of well-known types, such as geolocations or references to date and time, and can learn to extract some user-defined entities as well.

Having entities extracted and intent correctly determined it should be much easier to program the logic of your bot. This is often done using slot filling technique: extracted entities from the user’s input populate some slots in a dictionary, and if some more values are required to perform the task - additional dialog is initiated to ask additional info from the user.

Another type of bot behavior that often comes up is the ability to find best matching phrase or piece of information in some table, i.e. do an intelligent lookup. It is useful if you want to provide FAQ-style bot that can answer user’s questions based on some database of answers, or if you just want to program chit-chat behavior with some common responses. To implement this functionality, you can use QnA Maker - a complex service, that encapsulates Azure Cognitive Search, and provides simple way to build question-answering functionality. You can index any existing FAQ document, or provide question-answer pairs through the web interface, and then hook up QnA maker to your bot with a few lines of code.

Bot Development: Composer, Power Virtual Agents or Code?

As I mentioned above, bots can be developed using your favorite programming language. However, this approach requires you to write some boilerplate code, understand asynchronous calls, and therefore has a significant learning curve. There are some simpler options that are good for a start!

It is recommended to start developing your bot using low-code approach through Bot Framework Composer - an interactive visual tool that allows you to design your bot by drawing dialog diagrams.

Composer integrates LUIS and QnA maker out of the box, so that you do not need to train those services through web interface first, and then worry about integrating them to your bot. From the same UI, you are able to specify events triggered by some user phrases, and dialogs that respond to them.

Another similar low-code option would be to use Power Virtual Agents (PVA), a tool from Power Platform family of tools for business automation. It would be especially useful if you are already familiar with Power Platform, and using any of its tools to enhance productivity. In this case, PWA will be a natural choice, and it will integrate nicely into all your data points and business processes. In short - Composer is a great low-code tool for developers, while PVA is more for business users.

Getting Started with Bot Development

Let me show you how we can start the development of a simple educational bot that will help K-12 students with their geography classes. We will develop a simple bot, which you can later host on Microsoft Azure and connect to most popular communication channels, such as Teams, Slack or Telegram. If you do not have an Azure account, you can get a free trial (or here, if you are a student).

To begin with, we will implement three simple functions in our bot:

Being able to tell a capital city for a country (What is the capital of Russia?).
Giving definitions of most useful terms, eg. answering a questions like What is a capital?
Support for simple chit-chat (How are you today?)

Those two functions cover two most important elements of our intelligent backend: QnA Maker (which can be used to implement the last two points) and LUIS.

Starting with Bot Composer

To begin development, you need to [install Bot Framework Composer][InstallComposer] - I recommend to do it as desktop application. Then, after starting it, click on New button, and chose starting template for your bot: QnA Maker and LUIS:

Once you do that, you will see the main screen of composer, with a list of triggers on the left, and the main pane to design dialogs:

Here, you can delete an unused trigger BuySurface (which is left over from the demo), and go to Welcome message to customize the phrase that the bot says to new user. The logic of Welcome Message trigger is a bit complex, you need to look for a box called Send a response, and change the message in the right pane.

The language used to define phrases is called Language generation, or lg. A few useful syntax rules to know:

A phrase starts with -. If want to chose from a number of replies, specify several phrases, and one of them will be selected randomly. For example: ```bash
Hello, I am geography helper bot!
Hey, welcome!
Hi, looking forward to chat with you! ```
Comments start with >
Some additional definitions start with @
You can use ${...} syntax for variable substitution (we will see an example of this later)

Connecting to Azure Services

To use intelligent backend, you need to create Azure resources for LUIS and QnA Maker and provide corresponding keys to Composer:

Create LUIS Authoring Resource, and make sure to remember the region in which it was created, and copy key from Keys and Endpoint page in Azure Portal.
Create QnA Maker Service, and copy corresponding key
In Composer, go to bot settings by pressing Project Settings button in the left menu (look for a wrench icon, or expand the menu if unsure). Under settings, fill in LUIS Authoring Key, LUIS region and QnA Maker Subscription key.

Starting the Bot

At this point, you can already start chatting with your bot. Click Start bot in the upper right corner, and let some time for the magic to happen. When starting a bot, Composer actually creates and trains underlying LUIS model, builds bot framework project, and starts local web server with a copy of the bot, ready to serve your requests.

To chat with a bot, click Test in emulator button (you need to have Bot Framework Emulator) installed for this to work). This automatically opens up the chat window with all required settings, and you can start talking to your bot right away.

Creating QnA Maker Knowledge base

Let’s start with creating term dictionary using QnA Maker. Click on QnA left menu, and then Create new KB.

Here you can either start from some existing data (provided as an URL to html, pdf, Word or Excel document), or start creating phrases from scratch.

In most of the cases, you would have a document to start with, but in our case we will start from scratch. After creating a knowledge base, you can click Add QnA Pair to add all question-answer combinations you need. Note that you can add several options of the question by using All alternative phrasing link.

In our case, I have added a how are you phrase (with several options), and a phrase to explain the meaning of the word capital.

Having added the phrases, we can start a bot and make sure that it correctly reacts to given phrases, or similar versions of those phrases - QnA maker does not require it to be an exact match, it looks for similar phrases to make a decision on which answer to provide.

Adding Specific Actions with LUIS

To give our bot an ability to give capitals of countries, we need to add some specific functionality to look for a capital, triggered by a certain phrase. We definitely do not want to type all 200+ countries and their capitals into QnA Maker!

A functionality to get information about a country is openly available via REST Countries API. For example, if we make GET request to https://restcountries.eu/rest/v2/name/Russia, we will get JSON response like this:

[ {"name":"Russian Federation",
   "topLevelDomain":[".ru"],
   "capital":"Moscow", ... } ]

To ask for a capital of a given country, a user will say something like What is a capital of Russia?, or I want to know the capital of Italy. This phrase intent can be recognized using LUIS, and the name of the country is also extracted.

To add LUIS trigger, from the Design page of the composer, select your bot dialog and press “…” next to it. You will see Add a trigger option in the drop-down box. Select it, and then chose Intent recognized as trigger type.

Intent Recognized is the most common trigger type. However, you can specify Dialog events, that allow you to structure part of the conversation as a separate dialog, or some conversational activities, such as Handoff to human.

Then, specify trigger phrases, using a variation of LG language. In our case, we will use the following:

- what is a capital of {country=Russia}?
- I want to know a capital of {country=Italy}.
- Give me a capital of {country=Greece}!

Here, we specify a number of trigger phrases starting with -, and we indicate that we want to extract part of the phrase as an entity country. LUIS will automatically train a model to extract entities based on the provided utterances, so make sure to provide a number of possible phrases.

There are some pre-defined entity types, such as datetimeV2, number, etc. Using pre-defined types is recommended, and entity type can be specified using @ <entity_type> <entity_name> notation. In our case, we can use geographyV2 entity type, which extracts geographic locations, including countries.

Once we have defined phrase recognizer, we need to add a block that will do the actual REST call and fetch information on the given country. Use use Send HTTP Request block, and specify the following parameters:

Method = GET
Url = https://restcountries.eu/rest/v2/name/${@country}. Here, ${@country} will be substituted with the name of the recognized country.
Result property = dialog.result
Response type = json

This would make the REST call to the API, and the result will be stored into dialog.result property. If we provided a valid country, json result will be automatically parsed, otherwise, invalid operation code will be recorded in dialog.result.statusCode - in our case, 404.

To test if the call was successful and define different logic based on the result, we insert Branch: If/Else block, and specify the following condition: = equals(dialog.result.statusCode,200). True condition will correspond to the left branch, and we will insert Send a response block there, with the following text:

- A capital of ${@country} is ${dialog.result.content[0].capital}

In case result code is not 200, the right branch will be executed, where we will insert an error message. Our final dialog should look like this:

Adding Preconfigured Chit-Chat Functionality

It would be nice if your bot could respond to more everyday phrases, such as How old are you?, or Do you enjoy being a bot. We can define all those phrases in QnA Maker, but that would take us quite some time to do so. Luckily, there is Project Personality Chat that contains a number of pre-defined QnA Maker knowledge bases for several languages, and for a number of personalities:

Professional
Friendly
Witty
Caring
Enthusiastic

You can grab a link to the knowledge base from here, then go to QnA Maker Portal, find your knowledge base, and add this URL link to your service:

Having done that, click Save and Train, and enjoy a talk to your bot! You can even try ask it about life, universe and everything!

Testing the Bot and Publishing to Azure

Now that our basic bot functionality is complete, we can test the bot in bot emulator:

Once the bot is running locally, we can deploy it to Azure right from the Composer. If you go to Publish from the left menu, you will be able to define Publishing profile for your bot. Select Define new publishing profile, and chose one of the following:

The most standard way to deploy is to use Azure Web App. Composer will only require you to provide Azure subscription and resource group name, and it will take care of creating all the required resources (including bot-specific LUIS/QnA Maker instances) automatically. It may take awhile, but it will save you a lot of time and hassle of doing manual deployment.

Once the bot is published to Azure, you can go to Azure portal and configure Channels through which you bot would be available to external world, such as Telegram, Microsoft Teams, Slack or Skype.

Conclusion

Creating a bot using Bot Composer seems like an easy thing to do. In fact, you can create quite powerful bots almost without any code! And you can also hook them to your enterprise endpoint using such features as HTTP REST APIs and OAuth authorization.

However, there are cases when you need to significantly extend bot functionality using code. In this case, you have several options:

Keep main bot authoring in Bot Composer, and develop Custom Actions in C#
Export complete bot code using Custom Runtime feature of Composer, which exports complete bot code in C# or Javascript, which you can then fully customize. This approach is not ideal, because you will lose the ability to maintain the source of your bot in Composer.
Write a bot from the beginning in one of the supported languages (C#, JS, Python or Java) using Bot Framework.

If you want to explore how the same Educational bot for Geography can be written in C#, check out this Microsoft Learn Module: Create a chat bot to help students learn with Azure Bot Service.

I am sure conversational approach to UI can prove useful in many cases, and Microsoft Conversational Platform offers you wide variety of tools to support all your scenarios.

Computer Vision Read (OCR) API previews 73 human languages and new features on cloud and on-premise

sanjeev_jagtap — Mon, 15 Mar 2021 00:18:16 GMT

Overview

Businesses today are applying Optical Character Recognition (OCR) and document AI technologies to rapidly convert their large troves of documents and images into actionable insights. These insights power robotic process automation (RPA), knowledge mining, and industry-specific solutions. However, there are several challenges to successfully implementing these scenarios at scale.

The challenge

Your customers are global, and their content is global so your systems should also speak and read international languages. Nothing is more frustrating than not reaching your global customers due to lack of support for their native languages.

Secondly, your documents are large, with potentially hundreds and even thousands of pages. To complicate things, they have print and handwritten style text mixed into the same documents. To make matters worse, they have multiple languages in the same document, possibly even in the same line.

Thirdly, you are a business that’s trusted by your customers to protect their data and information. If your customers are in industries such as healthcare, insurance, banking, and finance, you have stringent data privacy and security needs. You need the flexibility to deploy your solutions on the world’s most trusted cloud or on-premise within your environment.

Finally, you should not have to choose between world-class AI quality, world languages support, and deployment on cloud or on-premise.

Computer Vision OCR (Read API)

Microsoft’s Computer Vision OCR (Read) technology is available as a Cognitive Services Cloud API and as Docker containers. Customers use it in diverse scenarios on the cloud and within their networks to help automate image and document processing.

What’s New

We are announcing Computer Vision's Read API v3.2 public preview as a cloud service and Docker container. It includes the following updates:

OCR for 73 languages including Simplified and Traditional Chinese, Japanese, Korean, and several Latin languages.
Natural reading order for the text line output.
Handwriting style classification for text lines.
Text extraction for selected pages for a multi-page document.
Available as a Distroless container for on-premise deployment.

First wave of language expansion

With the latest Read preview version, we are announcing OCR support for 73 languages, including Chinese Simplified, Chinese Traditional, Japanese, Korean, and several Latin languages, a 10x increase from the Read 3.1 GA version.

Thanks to Read’s universal model, to extract the text in these languages, use the Read API call without the optional language parameter. We recommend not using the language parameter if you are unsure of the language of the input document or image at run time.

The latest Read preview supports the following languages:

Read 3.2 Preview Languages

For example, once you have created a Computer Vision resource, the following curl code will call the Read 3.2 preview with the sample image.

Make the following changes in the command where needed:

Replace the value of <subscriptionKey> with your subscription key.
Replace the first part of the request URL (westcentralus) with the text in your own endpoint URL.

curl -v -X POST "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyze" -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription key>" --data-ascii "{\"url\":\"https://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Atomist_quote_from_Democritus.png/338px-Atomist_quote_from_Democritus.png\"}"

The response will include an Operation-Location header, whose value is a unique URL. You use this URL to query the results of the Read operation. The URL expires in 48 hours.

curl -v -X GET "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyzeResults/{operationId}" -H "Ocp-Apim-Subscription-Key: {subscription key}" --data-ascii "{body}"

Natural reading order output (Latin languages)

OCR services typically output text in a certain order in their output. With the new Read preview, choose to get the text lines in the natural reading order instead of the default left to right and top to bottom ordering. Use the new readingOrder query parameter with the “natural” value for a more human-friendly reading order output as shown in the following example.

The following visualization of the JSON formatted service response shows the text line order for the same document. Note the first column's text lines output in order before listing the second column and finally the third column.

OCR Read order example

For example, the following curl code sample calls the Read 3.2 preview to analyze the sample newsletter image and output a natural reading order for the extracted text lines.

curl -v -X POST "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyze?readingOrder=natural -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription key>" --data-ascii "{\"url\":\"https://docs.microsoft.com/en-us/microsoft-365-app-certification/media/dec01.png\"}"

The response will include an Operation-Location header, whose value is a unique URL. You use this URL to query the results of the Read operation.

curl -v -X GET "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyzeResults/{operationId}" -H "Ocp-Apim-Subscription-Key: {subscription key}" --data-ascii "{body}"

Handwriting style classification (Latin languages)

When you apply OCR on business forms and applications, it’s useful to know which parts of the form has handwritten text in them so that they can be handled differently. For example, comments and the signature areas of agreements typically contain handwritten text. With the latest Read preview, the service will classify Latin languages-only text lines as handwritten style or not along with a confidence score.

For example, in the following image, you see the appearance object in the JSON response with the style classified as handwriting along with a confidence score.

OCR handwriting style classification for text lines

The following code analyzes the sample handwritten image with the Read 3.2 preview.

curl -v -X POST "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyze -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription key>" --data-ascii "{\"url\":\"https://intelligentkioskstore.blob.core.windows.net/visionapi/suggestedphotos/2.png\"}"

The response will include an Operation-Location header, whose value is a unique URL. You use this URL to query the results of the Read operation.

curl -v -X GET "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyzeResults/{operationId}" -H "Ocp-Apim-Subscription-Key: {subscription key}" --data-ascii "{body}"

Extract text from select pages of a document

Many standard business forms have fillable sections followed by long informational sections that are identical between documents, and versions of those documents. At other times, you will be interested in applying OCR to specific pages of interest for business-specific reasons.

The following curl code sample calls the Read 3.2 preview to analyze the financial report PDF document with the pages input parameter set to the page range, "3-5".

curl -v -X POST "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyze?pages=3-5 -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription key>" --data-ascii "{\"url\":\"https://www.annualreports.com/HostedData/AnnualReports/PDF/NASDAQ_MSFT_2019.pdf\"}"

The response will include an Operation-Location header, whose value is a unique URL. You use this URL to query the results of the Read operation.

curl -v -X GET "https://westcentralus.api.cognitive.microsoft.com/vision/v3.2-preview.2/read/analyzeResults/{operationId}" -H "Ocp-Apim-Subscription-Key: {subscription key}" --data-ascii "{body}"

The following JSON extract shows the resulting OCR output that extracted the text from pages 3, 4, and 5. You should see a similar output for your sample documents.

"readResults": [ { "page": 3, "angle": 0, "width": 8.5, "height": 11, "unit": "inch", "lines": [] }, { "page": 4, "angle": 0, "width": 8.5, "height": 11, "unit": "inch", "lines": [] }, { "page": 5, "angle": 0, "width": 8.5, "height": 11, "unit": "inch", "lines": [] } ]

On-premise option with Distroless container

The Read 3.2 preview OCR container provides:

All features from the Read cloud API preview
Distroless container release
Performance and memory enhancements

Install and run the Read containers to get started and find the recommended configuration settings.

Get Started

Create a Computer Vision resource in Azure.
Follow our SDK and REST API QuickStarts.
Learn more about OCR (Read) and Form Recognizer.
See the list of OCR supported languages.
Learn more about the Read containers and download them from Docker Hub.
Write to us at formrecog_contact@microsoft.com

Integrating AI: Best Practices and Resources to Get Started

Yonet — Thu, 11 Feb 2021 00:33:22 GMT

We use AI (Artificial Intelligence) integrated applications daily, from search engines optimized to find the most relevant content, to recommendation engines for streaming or shopping. During AI’s early years rising to popularity, improving applications with AI was only possible for companies with big budgets dedicated to research and experts, preventing companies that cannot effort an AI team to compete. Today AI is readily available for any product, without having to invest in research and development. There are open-source libraries that can help you train Machine Learning models like TensorFlow. With a fraction of the effort and the cost, pre-trained AI services are available to easily integrate into your applications, with APIs and UI based tools to train custom models for your specific use case. In Integrating AI series, I aim to help you decide if and how to integrate AI into your applications, get you started with Azure’s ready to use AI solutions, Cognitive Services and answer your most frequent questions when getting started.

Let’s start with these fundamental questions:

What are the problems you can solve with AI?
What do you need to know before starting to build your solution?
How do you measure the success of your new AI features?

What are the problems you can solve with AI?

AI is a groundbreaking technology but not a magical solution for everything. It is important to know if you are adding value or solving an actual user problem. There are complex products like Wikipedia and Reddit that have a lot of information but use crowdsourcing and simple search to cater to unique needs without the help of AI. To make an informed decision, you need to start with your users’ needs. What are the problems they face? Is there a process that you can automize like filling expense forms that can be automated with Form Recognizer service? Send voice messages to your customers with updates using Speech Services? Do they make complex choices while using your product that could be customized to your users with the use of Personalizer? Do you need to improve the usability of your application with voice interactions and Language Understanding? It is important to solve a real need for your users instead of assuming the solution that will be useful. User research is the best way to figure out the issues and a lot can be surfaced by user analytics. You can use Metrics Advisor AI service to detect anomalies and figure out future AI solutions as well.

Once you have a clear definition of the problem and define how to measure success, it is time to explore practical solutions. You can read about the Azure customer stories and learn from their methods and design process. For example, read about BBC's customer story before you read about the technical story of using Azure's Speech, Azure Bot Service and Language Understanding Services together to solve the customer needs they identified.

Most AI solutions can fall into two categories. The first major use case for AI is automating the mindless repetitive jobs. If the users of an expense report or a hiring application need to type in information from a form or a receipt to your system, it is easily automated by OCR (Optical Character Recognition). Similar automations are possible for close captioning, translation, classifying images and automizing alert messages.

The second category of AI solutions can be categorized as complex human decisions based on data. You could give your friends recommendations on what to watch next easily, knowing what they like, what they don’t like. For example, a streaming service with thousands of movies to choose from, cannot surface relevant content with simple filtering of the genres or release dates. It would take forever to choose what to watch by browsing unless you know the exact name of the movie. For a decision like recommendation among thousands or millions of results, AI might be better at recommending to your best friend, maybe even better than you over time. Understanding the language and intent of people is another example. A human can understand and classify a review as positive or negative easily. For machines to detect the same emotions, you must do more than detect certain words to get sentiment.

What do you need to know before starting to build your solution?

Some problems are easier to solve than others with AI. Experimenting with different tools to confirming your solutions is important. All the Cognitive services are easy to try out and here is how to do that:

AI Demos Website gives you a hand-on experience of Cognitive Services

You can also download the Intelligent Kiosk app to try out the demos on your local machine.

Once you create an Azure resource, you can see the code samples, API call examples and try out the Rest API end points directly on the Cognitive Services API Reference pages.
API descriptions

Code Samples

Request

Will your users love your solution?

Scaling an application and polishing the user experience takes most of the development time. It is better to try out features fast and adjust before making the investment in perfecting the wrong experience. You might assume an application flow that users are going to interact, but users can surprise you in their own creative ways of using your tools. Prototype your applications quickly and get user feedback early on.

Power platform is one of the tools that allows you to create mobile apps that integrates important AI capabilities without writing any code. With power platform, you can easily deploy and share your prototypes, without leaving the platform’s UI. After the free trial period, both training and using your AI models will cost but not as much as the development time of an actual app with AI and having to make major changes after the release. Check out some of the capabilities and use cases of AI Builder on Power Platform and how to train a custom vision model and creating a mobile app on Power Platform in this article.

There are other fast and easy options to add AI to your applications, without a big development investment, especially if you are adding the capabilities to an existing application. You can use a logic app to design an application on Azure platform to find twitter mentions of your brand and analyze the sentiment of the tweets. You can visualize the data on Power BI or your choice of visualization platform or tools.

Once you integrate your AI solution, you can make the new AI features to a limited group of users and compare the effectiveness of your solution with your non-AI features.

Start your learning journey on AI Developer resources and sign up for a free Azure Account .

Let us know the problems you are trying to solve and your specific use cases on the comments below.

Accelerate search index development with Visual Studio Code

DerekLegenzoff — Wed, 10 Feb 2021 21:34:00 GMT

Azure Cognitive Search provides developers with APIs and tools to make it easy to add a great search experience to your application. There are tools available in the portal to import data into a search index and SDKs to simplify the integration of search functionality into your code base.

However, sometimes you need something in between: simpler than code, but more powerful than the portal. In these cases, it’s common to interact directly with the REST APIs to quickly update an indexer, add a document, or perform other standard tasks. Tools like Postman are great for this but building out API calls from scratch can quickly become tedious. You wouldn’t write an API call from scratch to add a document to Azure Storage—you’d use Azure Storage Explorer—and we don’t want you to have to do that for search either.

With this in mind, we created the Visual Studio Code Extension for Azure Cognitive Search (Preview). The Visual Studio Code extension makes it easy to work with your search service using the full capabilities of the REST APIs while providing rich IntelliSense and snippets to help you. With the extension, you can create and update indexes and other components, add documents, search, and more. You’ll never need to struggle with remembering the correct syntax again.

Extension Functionality

The extension covers all the major REST API operations for Cognitive Search. Check out the examples below to see some of what’s possible and feel free to request additional functionality here.

Browse all your Azure Cognitive Search services

Get access to all your search services in one place. You can quickly see all your indexes, indexers, and other components.

Create new indexes, indexers, data sources, skillsets, and synonym maps

You can create a new index or other component just by editing the JSON and saving the file. You can then read, update, or delete these components at any time.

Take advantage of rich IntelliSense

The extension also includes IntelliSense to guide you as you’re building out your JSON. Instead of referencing external docs each time, you can see what parameters exist and what their allowed values are as you type.

In addition to IntelliSense, the extension provides snippets or templates for building more complex objects, such as data sources and skillsets, so that you have a good starting point.

Add or update documents in the search index

Adding or updating documents is something that’s not possible in the portal today. With the extension, you can quickly add a document, and it will even save you some time by creating a JSON template for you based on your index definition.

You can view or update existing documents too.

Query your search indexes

Finally, once you’ve added documents to your search service, you can also query from within the extension and view the results side by side. You can even add multiple queries or save the queries to a file to refer to them later.

Key use cases

If your security requirements mandate the use of Private Endpoints or IP Firewalls, you’ll find that some functionality is no longer available in the portal. For these cases, the extension is a great alternative to the portal for interacting with your indexes and the other components of your search service.

In other cases, if you find yourself constantly recreating indexes or making small tweaks to them or other search components, the extension can make it incredibly easy to make small updates such as adding a field to an index.

Get Started

Regardless of how you’re trying to use Cognitive Search, this extension will likely make your life easier. To get started today, download the extension, and follow the related quickstart. You’ll see just how quickly and easily you can get up and running with Cognitive Search using the Visual Studio Code Extension.

If you run into any issues or have any questions, please feel free to reach out to us at azuresearch_contact@microsoft.com or raise an issue on the extension’s GitHub repo.

How to use Cognitive Services and containers

hboelman — Thu, 11 Feb 2021 06:43:42 GMT

In this blog we are going to take a look at how we can run a selection of Cognitive Services in Docker compatible containers. This option of using these services can come in handy if you run into scenarios where your application can not connect to the cloud all the time or if you need more control over your data.

What are Cognitive Services

Azure Cognitive Services are cloud-based services that expose AI models through a REST API. These services enable you to add cognitive features, like object detection and speech recognition to your applications without having data science skills. By using the provided SDKs in the programming language of your choice you can create application that can see (Computer Vision), hear (Speech), speak (Speech), understand (Language), and even make decisions (Decision).

Cognitive Services in containers

Azure Cognitive Service in containers gives developers the flexibility in where to deploy and host the services that come with Docker containers and keeping the same API experience as when they where hosted in the Azure.

Using these containers gives you the flexibility to bring Cognitive Services closer to your data for compliance, security or other operational reasons.

What are containers
Containerization is an approach to software distribution in which an application or service, including its dependencies & configuration, is packaged together as a container image. With little or no modification, a container image can be deployed on a container host. Containers are isolated from each other and the underlying operating system, with a smaller footprint than a virtual machine. Containers can be instantiated from container images for short-term tasks, and removed when no longer needed.

When to use Cognitive Services in containers?

Running Cognitive Services in containers can be the solution for you if you have specific requirements or constraints making that make it impossible to run Cognitive services in Azure. The most common scenarios include connectivity and control over the data. If you are running Cognitive Services in Azure all the infrastructure is taken care of, running them in containers moves the infrastructure responsibility, like performance and updating the container, to you.

A case where you choose for container could be, if your connection to Azure is not stable enough. For instance if you have 1000's of document on-prem and you want to run OCR. If you use the Computer Vision OCR endpoint in the cloud you would need to send all the documents to the end point in azure, while if you run the container locally you only need to send the billing information every 15 minutes to Azure.

Features and benefits

Immutable infrastructure: Enable DevOps teams' to leverage a consistent and reliable set of known system parameters, while being able to adapt to change. Containers provide the flexibility to pivot within a predictable ecosystem and avoid configuration drift.

Control over data: Choose where your data gets processed by Cognitive Services. This can be essential if you can't send data to the cloud but need access to Cognitive Services APIs. Support consistency in hybrid environments – across data, management, identity, and security.

Control over model updates: Flexibility in versioning and updating of models deployed in their solutions. Portable architecture: Enables the creation of a portable application architecture that can be deployed on Azure, on-premises and the edge. Containers can be deployed directly to Azure Kubernetes Service, Azure Container Instances, or to a Kubernetes cluster deployed to Azure Stack. For more information, see Deploy Kubernetes to Azure Stack.

High throughput / low latency: Provide customers the ability to scale for high throughput and low latency requirements by enabling Cognitive Services to run physically close to their application logic and data. Containers do not cap transactions per second (TPS) and can be made to scale both up and out to handle demand if you provide the necessary hardware resources.

Scalability: With the ever growing popularity of containerization and container orchestration software, such as Kubernetes; scalability is at the forefront of technological advancements. Building on a scalable cluster foundation, application development caters to high availability.

Which services are available

Container support is currently available for a subset of Azure Cognitive Services, including parts of:

Group	Service	Documentation
Anomaly Detector	Anomaly Detector	Documentation
Computer Vision	Read OCR (Optical Character Recognition)	Documentation
	Spatial Analysis	Documentation
Form Recognizer	Form Recognizer	Documentation
Language Understanding	Language Understanding	Documentation
Speech	Custom Speech-to-text	Documentation
	Custom Text-to-speech	Documentation
	Speech-to-text	Documentation
	Text-to-speech	Documentation
	Neural Text-to-speech	Documentation
	Speech language detection	Documentation
Text Analytics	Key Phrase Extraction	Documentation
	Text language detection	Documentation
	Sentiment analysis	Documentation
Face	Face	Documentation

How to use Cognitive Services in containers

The use of the services in containers is exactly the same as if you would use them in Azure. The deployment of the container is the part that takes a bit of planning and research. The services are shipped in Docker Containers. This means that the containers can be deployed to any Docker compatible platform. This can be your local machine running Docker Desktop or a fully scalable Kubernetes installation in your on premise data center.

Generic workflow

Create the resource in Azure
Get the endpoint
Retrieve the API Key
Find the container for the service
Deploy the container
Use the container endpoint as you would use the API resource

Optional you can mount your own storage and connect Application Insights.

Tutorial: Run a Text to Speech container in an Azure Container Instance.

In this tutorial we are going to run a Cognitive Service Speech container in an Azure Container Instance and use the REST API to convert text into speech.

To run the code below you need an Azure Subscription. if you don’t have an Azure subscription you can get $200 credit for the first month. And have the Azure command-line interface installed. If you don't have the Azure CLI installed follow this tutorial.

1. Create a resource group

Everything in Azure always start with creating a Resource Group. A resource group is a resource that holds related resources for an Azure solution.

To create a resource group using the CLI you have to specify 2 parameters, the name of the group and the location where this group is deployed.

az group create --name demo_rg --location westeurope

2. Create Cognitive Service resource

The next resource that needs to be created is a Cognitive Services. To create this resource we need to specify a few parameters. Besides the name and resource group, you need to specify the kind of cognitive service you want to create. For our tutorial we are creating a 'SpeechServices' service.

az cognitiveservices account create \
    --name speech-resource \
    --resource-group demo_rg \
    --kind SpeechServices \
    --sku F0 \
    --location westeurope \
    --yes

3. Get the endpoint & API Key

If step 1 and 2 are successfully deployed we can extract the properties we need for when we are going to run the container in the next step. The 2 properties we need are the endpoint URL and the API key. The speech service in the container is using these properties to connect to Azure every 15 minutes to send the billing information.

To retrieve endpoint:

az cognitiveservices account show --name speech-resource --resource-group demo_rg  --query properties.endpoint -o json

To retrieve the API keys:

az cognitiveservices account keys list --name speech-resource --resource-group demo_rg

3. Deploy the container in an ACI

One of the easiest ways to run a container is to use Azure Container Instances. With one command in the Azure CLI you can deploy a container and make it accessible for the everyone.

To create an ACI it take a few parameters. If you want your ACI to be accessible from the internet you need to specify the parameter: '--dns-name-label'. The URL for the ACI will look like this: http://{dns-name-label}.{region}.azurecontainer. The dns-name-label property needs to be unique.

az container create 
    --resource-group demo_rg \
    --name speechcontainer \
    --dns-name-label <insert unique name> \
    --memory 2 --cpu 1 \
    --ports 5000 \
    --image mcr.microsoft.com/azure-cognitive-services/speechservices/text-to-speech:latest \
    --environment-variables \
        Eula=accept 
        Billing=<insert endpoint> 
        ApiKey=<insert apikey>

The deployment of the container takes a few minutes.

4. Validate that a container is running

The easiest way to validate if the container is running, is to use a browser and open the container homepage. To do this you first need to retrieve the URL for the container. This can be done using the Azure CLI with the following command.

az container show --name speechcontainer --resource-group demo_rg --query ipAddress.fqdn -o json

Navigate to the URL on port 5000. The URL should look like this: http://{dns-name-label}.{region}.azurecontainer.io:5000/

If everything went well you should see a screen like this:

5. Submit your first task

The Text to Speech service in the container is a REST endpoint. To use it we would need to create a POST request. There are many ways to do a POST request. For our tutorial we are going to use Visual Studio Code to do this.

Requirements:

Download and Install Visual Studio Code
Install a plugin called REST Client

If you have the Visual Studio Code with th REST Client installed create a file call: rest.http and copy past the code below in the file.

POST http://<dns-name-label>.<region>.azurecontainer.io:5000/speech/synthesize/cognitiveservices/v1  HTTP/1.1
Content-Type: application/ssml+xml
X-Microsoft-OutputFormat: riff-24khz-16bit-mono-pcm
Accept: audio/*

<speak version="1.0" xml:lang="en-US">
    <voice name="en-US-AriaRUS">
        The future we invent is a choice we make. 
        Not something that just happens.
    </voice>
</speak>

Change the name of the URL to the URL of your ACI.
Next click on the Send Request link (just above the URL)

On the right side of VS Code you should see the response of the API. In the top right corner you see "Save Response Body" click on the button and save the response as a .wav file. Now you can use any media player to play the response.

Learn more

Microsoft Learn

Microsoft Learn is a free, online training platform that provides interactive learning for Microsoft products and more.

For this blog we have created a custom Collection of Learn Modules covering all the topics in depth.

Blogs and articles

On Microsoft Docs

Build a natural custom voice for your brand

Qinying Liao — Thu, 04 Feb 2021 05:15:47 GMT

Custom Neural Voice is a Text-to-Speech (TTS) feature of Speech in Azure Cognitive Services that allows you to create a one-of-a-kind customized synthetic voice for your brand. Since its preview in September 2019, Custom Neural Voice has empowered organizations such as AT&T, Duolingo, Progressive, and Swisscom to develop branded speech solutions that delight users. (For more details, read the Innovation Stories blog).

Today, we are excited to announce that Custom Neural Voice is now generally available (GA). It is important to note that although Custom Neural Voice is GA from a technological standpoint, interested customers must apply and be approved to use it. Alternatively, developers can add TTS capabilities to their apps quickly by creating an Azure Speech instance and selecting from over 200 pre-built TTS and Neural TTS voices across 54 languages/locales.

In this blog, we’ll introduce how Custom Neural Voice works and share best practices in responsibly creating a highly natural brand voice for your apps. If you have questions, join us at our ‘Ask-Microsoft-Anything’ on Wednesday, 2/10 at 9AMPT. Add to Calendar.

Your voice, your brand

In a world where voice-based interactions are increasingly becoming the norm, your voice is your brand. A recognizable digital voice helps your customers connect with your brand in new ways.

In recent years we have seen increased interest from a broad range of companies across Media and Entertainment, Telecom, Automobile, Education, and Hospitality, who consider voice-based interactions from a range of devices like phones, speakers, TV/cable boxes, and cars as a key interaction point with their customers. These organizations are looking to have a consistent, branded experience delivered directly to their customers. To highlight one such example, below is an audio sample of the 'Flo' virtual chatbot from Progressive.

Voice Sample: 'Flo' from Progressive

Custom Neural Voice empowers people and organizations in many ways. The following scenarios are examples of use cases where customers find Custom Neural Voice particularly useful and valuable:

Customer Service Chatbots – Companies can automate their call center operation with conversational AI to answer calls from customers with a natural-sounding voice that conveys friendliness, empathy, and professionalism and other values that are important to companies. For example, Progressive is using Custom Neural Voice to enable their virtual version of Flo to help their customers with ‘everything from getting a free car insurance to general insurance questions’. Read the full story.

Voice Assistants – Companies developing smart assistants on appliances, cars, and homes can use Custom Neural Voice to create a unique synthetic voice that conveys the brand of the company, the persona of the assistant and a speaking style that enables the best experience for their target users. With Custom Neural Voice, Swisscom was able to create a multilingual voice assistant that sound human and unique to Swisscom and resonates with its audience. Read the full story.

Online Learning – Education providers can add speech to their learning material with a voice that is suitable for the subjects and the students, thereby improving the engagement of the students and the effectiveness of the learning. Duolingo is using the Custom Neural Voice capability to develop stylized voices for their virtual characters for their online learning experience. Learn more.

Audio Books – Content publishers can turn written content into audio that is spoken with a synthetic voice to make it more accessible to the global audience. With Custom Neural Voice, the content publishers can create one or more unique voices with natural reading styles that match the subject and context of the content as well as the preference of the listeners. The Beijing Hongdandan Visually Impaired Service Center is using the Custom Neural Voice capability to produce audiobooks based on the voice of Lina, a trainer at the organization who is familiar to the people who are blind in China.

Assistive Technology and Real-time Translations – Custom Neural Voice can be used in situations to assist people in need or improve accessibility. When used as an assistive technology, people with speech impairment could use the technology to enable them to communicate with others with a voice that sounds like them. Custom Neural Voice can used in other situations such as real-time translation allowing people to communicate with others in a foreign language in a familiar voice.

Public Service Announcement – Public service organizations can use Custom Neural Voice to create a voice that is suitable for public announcements, whether it is in an airport, a train terminal, or other venues. The use of synthetic voice provides the ability to generate announcements with dynamic content that cannot be recorded ahead of time.

Benefit of Custom Neural Voice

Traditionally, TTS requires a large volume of voice data—in the range of 10,000 lines or more—to produce a fluent voice model. Consequently, TTS models with fewer recorded lines tend to sound noticeably robotic.

With the innovation of deep neural networks and a powerful base model built with speech data from many different speakers, Neural TTS can 'learn' the way phonetics are combined in natural human speech rather than using classical programming or statistical methods.

Empowered with this technology, Custom Neural Voice enables users to build highly-realistic voices with just a small number of training audios. This new technology allows companies to spend a tenth of the effort traditionally needed to prepare training data while at the same time significantly increasing the naturalness of the synthetic speech output when compared to traditional training methods.

Listen to the samples created with Custom Neural Voice below. Or try more demos on the Speech Studio.

Language	Voice	Human	TTS (Custom Neural Voice)
Chinese (Mandarin, simplified)	Lina (Hongdandan)
English (Australia)	Thomas
English (United States)	Angela
French (France)	Zoe (Swisscom)
German (Germany)	Lara (Swisscom)

How it works

Custom Neural Voice is based on Neural TTS technology that creates a natural-sounding voice. The realistic and natural sounding voice of Custom Neural Voice can represent brands, personify machines, and allow users to interact with applications conversationally in a natural way.

The underlying Neural TTS technology used for Custom Neural Voice consists of three major components: Text Analyzer, Neural Acoustic Model, and Neural Vocoder. To generate natural synthetic speech from text, the text is first input into Text Analyzer, which provides output in the form of phoneme sequence. A phoneme is a basic unit of sound that distinguishes one word from another in a particular language. A sequence of phonemes defines the pronunciations of the words provided in the text. Then the phoneme sequence goes into the Neural Acoustic Model to predict acoustic features that define speech signals, such as the timbre, speaking style, speed, intonations, and stress patterns, etc. Finally, the Neural Vocoder converts the acoustic features into audible waves so that synthetic speech is generated.

Neural TTS voice models are trained using deep neural networks based on real voice recording samples. With the customization capability of Custom Neural Voice, you can adapt the Neural TTS engine to better fit your user scenarios. To create a custom neural voice, visit the Speech Studio to upload the recorded audio and corresponding scripts, train the model, and deploy the voice to a custom endpoint. Depending on the use case, Custom Neural Voice can be used to convert text into speech in real-time (e.g., used in a smart virtual assistant) or generate audio content offline (e.g., used as in audiobook or instructions in e-learning applications) with the text input provided by the user. This is made available through REST APIs, Speech SDK, or a no-code Audio Content Creation tool.

Building a Custom Neural Voice

As part of Microsoft’s commitment to responsible AI, we are designing and releasing Custom Neural Voice with the intention of protecting the rights of individuals and society, fostering transparent human-computer interaction, and counteract the proliferation of harmful deepfakes and misleading content. For this reason, we have limited the access and use of Custom Neural Voice. Submit an intake form here.

Microsoft requires every customer to obtain explicit written permission from the voice talent before creating a voice model (see Disclosure for Voice Talent). In addition, you must not use custom neural voice for certain prohibited use cases (see Code of Conduct) and must disclose the synthetic nature of the service to your users upon deployment of the custom voice model (see Disclosure Guidelines).

When preparing your recording script, make sure you include the following sentence to acquire the voice talent’s acknowledgement of using their voice data to create a TTS voice model and generate synthetic speech.

“I [state your first and last name] am aware that recordings of my voice will be used by [state the name of the company] to create and use a synthetic version of my voice.”

As a technical safeguard intended to prevent misuse of Custom Neural Voice services, Microsoft will use this recording to verify that the voice talent’s voice in the script matches the voice provided in the training data through the Speaker Verification technology. Read more about this process in the Data and Privacy document.

In the video below, we introduce how to use the Speech Studio to create a highly natural voice with your own data.

Creating a great custom voice requires careful quality control in each step, from voice design, data preparation, to the deployment of the voice model to your system. This docs page outlines in more detail the characteristics, limitations and the best practices in designing and building a custom neural voice.

Below are some key steps to take when creating a custom neural voice for your organization. (Note: this presumes you have applied and have been approved for use of Custom Neural Voice.)

Step 1: Persona design

First, design a persona of the voice that represents your brand using a persona brief document that defines elements such as the features of the voice, and the character behind the voice. This will help to guide the process of creating a custom voice model, including defining the scripts, selecting your voice talent, training and voice tuning.

Step 2: Script selection

Carefully select the recording script to represent the user scenarios for your voice. For example, you can use the phrases from bot conversations as your recording script if you are creating a customer service bot. Include different sentence types in your scripts, including statements, questions, exclamations, etc.

Step 3: Preparing training data

We recommend that the audio recordings be captured in a professional quality recording studio to achieve a high signal-to-noise ratio. The quality of the voice model heavily depends on your training data. Consistent volume, speaking rate, pitch, and consistency in expressive mannerisms of speech are required.

Common issues with recordings include speaking style mismatch (e.g., not in an ‘excited’ manner that you want to the voice to be), unnatural speed, unstable breaks, wrong pronunciation on words, etc. It is recommended that you work with a voice director to control the recording quality. Follow the recording guidance here.

Once the recordings are ready, follow the instructions here to prepare the training data in the right format.

Step 4: Testing

Prepare test scripts for your voice model that cover the different use cases for your apps. It’s recommended that you use scripts within and outside the training dataset so you can test the quality more broadly for different content.

Step 5: Tuning and adjustment

The style and the characteristics of the trained voice model depend on the style and the quality of the recordings from the voice talent used for training. However, several adjustments can be made using SSML (Speech Synthesis Markup Language) when you make the API calls to your voice model to generate synthetic speech. SSML is the markup language used to communicate with the TTS service to convert text into audio. The adjustments include change of pitch, rate, intonation, and pronunciation correction. If the voice model is built with multiple styles, SSML can also be used to switch the styles.

All of the SSML markups mentioned above can be passed directly to the API. We also provide an online tool, Audio Content Creation, that allows customers to fine-tune their audio output using a friendly UI.

Get started

Interested in building a custom neural voice? Check the languages supported. Sign up to Speech service on Azure and get started on the Speech Studio.

Besides the capability to customize TTS voice models, Microsoft offers over 200 neural and standard voices covering 54 languages and locales. With these Text-to-Speech voices, you can quickly add read-aloud functionality for a more accessible app design, or give a voice to chatbots to provide a richer conversational experiences to your users.

For more information:

Join us during our ‘Ask Microsoft Anything’ on Wed., Feb. 10^th (9amPT) (add to Calendar)
Add Text-to-Speech to your apps today
Apply for access to Custom Neural Voice
Learn more about responsible use of Custom Neural Voice
Try our demo to listen to existing neural voices

QnA with Azure Cognitive Search

nerajput — Wed, 03 Feb 2021 07:59:57 GMT

QnA + Azure Cognitive Search enables instant answers over your search results. Now, you do not need to spend time looking through your pile of documents to find the exact answer to your query. There will be an instant answer coming up for the user query from the most relevant documents present in your system. A solution where you can ingest your pile of documents and query over them to get the answer as well as related relevant documents to get more information.

This solution accelerator enables automatic bulk ingestion of documents for QnA processing via a Cognitive Search custom skill. The sample UI showcases the combined experience of instant answers to your questions as well as the list of relevant documents. Finally, the solution is easily deployed using a simple Deploy button, which sets up all necessary services in your Azure subscription

Benefits:

Converged search experience powering instant answer and relevant documents.
Search using natural language queries.
One-click deployment.
Saves end user time during search.
Flexibility to enhance and edit instant answers.

The solution combines the power of both Azure Cognitive Search and QnA Maker to extract question-answer pairs from your documents before storing them in the index. Once you deploy the solution, you get a single endpoint where for each end user query both the services will be called in parallel and you will get a combined result with an instant answer powered by QnA Maker along with the relevant documents coming from Azure Cognitive Search. Checkout the Cognitive Search Question Answering Solution Accelerator (github.com)

Architecture:

This solution accelerator contains the following artifacts:

ARM template to set up the solution.
Custom skill in Azure Cognitive Search, which ingests the data into QnA Maker.
User interface to view the results.

Live Demo Link:

You can view a live demo of this repo at the following link: https://aka.ms/qnaWithAzureSearchDemo

File Type Supported:

Currently instant answers will only be available for the file types supported by QnA Maker. By default, the logic in the Azure Cognitive Search service indexer also ingests only the following file types: .pdf,.docx,.doc,.xlsx,.xls,.html,.rtf,.txt,.tsv. You can change this by modifying the indexedFileNameExtensions property in the Indexer.json.

Tutorial:

NOTE: You need to have a GitHub account and Azure subscription to try out this solution.

Resource creation and deployment:

Click here to Deploy to Azure.
This will take you to the create blade where all the information will be pre-filled, as shown below. Click Review+ Create button to proceed.

Your deployment process will take 4-5 minutes to complete. Once completed you will land up on the following page:

Click on Deployment details to check all the resources that have been created.

Initialization:

To initialize the solution, click on the “Outputs” button on the left. Copy the “http trigger to initialize accelerator" value. Open a new browser tab and paste this URL into the browser. This will run for about a minute, and then you'll see a message indicating success or failure.
If the initialization is successful, then following message will appear:
Once the resources are initialized, you can access the portal through the “UI portal link” value in the Output tab.

Upload Documents:

You can upload the documents one by one through the UI portal, by going to the Upload tab.

You can also upload the documents in bulk, through a container.
- Go to your storage account.
- Click on Containers and select qna-container to upload the documents in bulk.
- Use the Upload tab and select the multiple files you want to ingest. It will take some time to index the documents and to extract the Question Answer pairs out of the documents.

Question Answer Enhancement:

Once the ingestion is complete, you can view all the Question Answer pairs extracted from the documents by clicking on “Knowledge Base”.
Play with your knowledge base!, You can also test for different queries using the Test Pane. Once you are satisfied with the experience, click on “Save and Train” and then “Publish” the changes to get these changes reflected on your main portal.

This solution has been specifically created for our customers to address the long-term standing ask to retrieve an instant answer from the relevant document. This solution currently covers the basic functionality, and we will keep adding more features based on user interaction and customer’s feedback. Please feel free to drop us a mail at search-qna-solution@microsoft.com to provide your valuable feedback.

Useful Links:

Unified Neural Text Analyzer: an innovation to improve Neural TTS pronunciation accuracy

Qinying Liao — Thu, 28 Jan 2021 09:38:10 GMT

Introducing Unified Neural Text Analyzer: an innovation for Neural Text-to-Speech pronunciation accuracy improvement

This post is co-authored by Dongxu Han, Junwei Gan and Sheng Zhao

Neural Text-to-Speech (Neural TTS), part of Speech in Azure Cognitive Services, enables you to convert text to lifelike speech for more natural user interactions. Neural TTS has powered a wide range of scenarios, from audio content creation to natural-sounding voice assistants, for customers from all over the world. For example, BBC, Progressive and Motorola Solutions are using Azure Neural TTS to develop conversational interfaces for their voice assistants in English speaking locales. Swisscom and Poste Italiane are adopting neural voices in French, German and Italian to interact with their customers in the European market. Hongdandan, a non-profit organization, is adopting neural voices in Chinese to make their online library audible for the blind people in China.

In this blog, we introduce our latest innovation in the Neural TTS technology that helps to improve the pronunciation accuracy significantly: Unified Neural Text Analyzer.

What is text analyzer?

Neural TTS converts plain text into wave form via three modules: neural text analyzer, neural acoustic model and neural vocoder. Text analyzer converts plain text to pronunciations, acoustic model converts pronunciations to acoustic features and finally vocoder generates waveforms. Text analyzer is the first link of the entire TTS system with results directly affecting the acoustic model and vocoder. The correct pronunciation of a word or phrase is the basic expectation in TTS, which delivers the right information to use but it’s not always easy. For example, “live” should be read different in “We live in a mobile world” and “TV Apps and live streaming offerings from The Weather Network” depending on context. If TTS reads them incorrectly, the intelligibility and naturalness of the content will be significantly influenced. Thus, text analyzer is important to TTS.

Recent updates on Neural TTS include a major innovation to the text analyzer, called “UniTA” (Unified Neural Text Analyzer). UniTA is a unified text analyzer model, which seamlessly simplifies text analyzer workflow and reduces time latency in the runtime server. It adopts a multitask learning approach, jointly training all ambiguity models to solve context ambiguity and generate correct pronunciation and as a result reduces over 50% of pronunciation errors.

What are the challenges?

Generally, different natural languages have different linguistic grammar. In TTS, text analyzer needs to follow the same grammar of languages in order to generate correct pronunciations, which contains but isn’t limited to the following required grammar categories:

Word Segmentation is the process of dividing the written text into meaningful units, such as words. In English and many other languages using some form of the Latin alphabet, the space is a good approximation of a word divider. On the other hand, in languages such as Chinese or Japanese, there is no spacing in sentences. Different word segmentation results may cause different meanings and pronunciations.
Part-of-Speech Tagging is the process of marking up a word in a text as corresponding to a particular part of speech (such as noun, verb, adj, adv and so on), based on both its definition and its context.
Morphology is the progress of classifying words according to shared inflectional categories such as person (first, second, third), number (singular vs. plural), gender (masculine, feminine, neuter) and case (nominative, oblique, genitive) with a given lexeme.
Text Normalization is the process of transforming digits or symbols to their standard format for disambiguation, for example: “$200" would be normalized as "two hundred dollars”, “200M" would be normalized as "two hundred meters” or “two hundred million”.
Similar to Text Normalization, Abbreviation Expansion is the process of transforming non-standard words to their standard format for disambiguation, for example: “VI" would be normalized as "six”, “St" would be normalized as "Saint” or “street”.
Polyphone Disambiguation is the process of marking up polyphone word (heteronym word, which has one spelling but has more than one pronunciation and meaning) to its correct pronunciation based on its context.

Category	Example
Word Segmentation	[English] Nice to meet u:) --> Nice / to / meet / u / :) [Chinese] 在圣诞节纽约大都会有演出 --> 在 / 圣诞节 / 纽约 / 大 / 都会(du1 hui4) / 有 / 演出 [Chinese] 在圣诞节纽约大都会有演出 --> 在/ 圣诞节 / 纽约 / 大都(da4 dou1) / 会 / 有 / 演出
Part-of-Speech Tagging	[Noun, \| l ai v s \|] Many people have lost their lives since the cyclone because aid has not been able to be distributed. [Verb, \| l I v s \|] I also discovered the very angry raccoon that lives near my porch.
Morphology	[Singular] 1km --> one kilometer [Plural] 5km --> five kilometers
Text Normalization	[Fraction, nine out of ten] The O.S. Speed T1202 ups the ante for race-winning performance, resulting in a power plant that will dominate 9/10 scale competition. [Date, September tenth] 1st episode will air 9/10 with never before seen video of her birth!
Abbreviation Expansion	[Street] Oh man, biking from 24th St BART to the 29th St bikeshare station, that will be sweet. [Saint] We continue to ask anyone who was in the wider area near St Heliers School between 7.30am and 9am and witnessed any suspicious activity to contact police
Polyphone Disambiguation	[p r ih - z eh 1 n t] The prices will present the estimated discount utilizing the drug discount card. [p r eh 1 - z ax n t] But our present situation is not a natural one.

Most pronunciations are affected by these categories based on syntactic or semantic context, and these categories are all challenging disambiguation problems. The traditional TTS approach is a pipeline-based module called “text analyzer” with a series of models aimed at solving grammar disambiguation problems, which causes some of the following issues:

Complex model. Redundant models are built and optimized separately but implemented together in the traditional text analyzer, which causes pipeline long and complicated.
Error propagation. Accumulated errors caused by the models isolated would affect the final results.
High latency. Models run one by one in the traditional text analyzer which is pipeline-based. Time cost is high in the runtime server.

Compared to the traditional pipeline-based text analyzers, our Neural TTS proposes a Unified Neural Text Analyzer model (UniTA) to improve TTS pronunciation.

It builds a unified text analyzer model, which greatly simplifies the text analyzer workflow and reduces time latency in the runtime server.
It adopts a multitask learning approach, jointly training all ambiguity models to solve context ambiguity and generate the correct pronunciations, reducing pronunciation errors by over 50%.

How does UniTA improve pronunciations?

Firstly, UniTA converts the input text to word embedding vectors through a pre-trained model. Word embedding is a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from vocabulary are mapped to vectors of real numbers. Conceptually, it involves a mathematical embedding from a space with many dimensions per word to a continuous vector space with a much lower dimension. Pre-training models like XYZ-Code have demonstrated unprecedented effectiveness for learning universal language representations based on unlabeled corpus with the method achieving great success in many tasks like language understanding and language generation.

Secondly, a sequence tagging fine-tune strategy is adopted in the UniTA model. UniTA is designed as a typical word classification task, in which

Word Segmentation predicts word delimiter as word boundary or not.
Part-of-Speech (POS) predicts “noun”, “verb”, “adj” and so on to classify word part-of-speech.
Morphology predicts “singular”, “plural”, “masculine”, “feminine”, “neuter” and so on to classify word number, gender and case.
Text Normalization (TN) predicts candidate digits to “cardinal”, “date”, “time”, “stock” or other TN categories, and then an auxiliary component “TN Rule” helps convert digits to word form based on predicted category.
Abbreviation Expansion predicts candidate abbreviation word to its expanded form.
Polyphone disambiguation predicts polyphone words’ pronunciation. An auxiliary component, “Lexicon” is used here for achieving non-polyphone words’ pronunciations.

Different from the traditional text analyzer training models , UniTA adopts a multitask learning approach to jointly train all categories together including word segmentation, part-of-speech tagging, morphology, abbreviation expansion, text normalization and polyphone disambiguation. The multitask learning approach shares hidden layers’ information and jointly trains across different tasks, which has achieved state-of-art achievements on many NLP tasks. In UniTA, hidden information is also shared in models when training.

For example, the sentence “St. John had a 10-3 run to build its lead to 78-64 with 4:44 left.” in the training corpus is annotated as showed in the table below. “--” means there is no related tag in the category. In the word segmentation column, the phrase “10-3” is segmented as “10”, “-” and “3”; in the morphology column, the word “had” is annotated as “past tense”; in the text normalization column, “10-3” belongs to interpreting word “to” instead of “-“ while “4:44” belongs to the pattern using time format; In the abbreviation column, word “St.” is expanded as “Saint” rather than “Street”; and in the polyphone disambiguation column, the word “lead” is pronounced as [l i: d]. Actually, the word “lead” has two pronunciations, it is pronounced as [l i: d] when its POS is noun while pronounced as [l e d] when its POS is verb. This means the POS results and Polyphone results can share the inner information. In this way, multitask model improves UniTA accuracy.

Word	Word Segmentation	Part-of-Speech	Morphology	Text Normalization	Abbreviation	Polyphone disambiguation
*St.*	--	Noun	--	--	Saint	--
*John*	--	Noun	--	--	--	--
*had*	--	Verb	Past tense	--	--	--
a	--	Det	--	--	--	--
*10-3*	10 / - / 3	Num	--	numbers are predicted as “ten to three”	--	--
*run*	--	Noun	Singular	--	--	--
to	--	Particle	--	--	--	--
*build*	--	Verb	--	--	--	--
*its*	--	Det	--	--	--	--
*lead*	--	Noun	Singular	--	--	l i: d
to	--	Particle	--	--	--	--
*78-64*	78 / - / 64	Num	--	numbers are predicted as “seventy-eight to sixty-four”	--	--
*with*	--	Prep	--	--	--	--
*4:44*	4 / : / 44	Num	--	numbers are predicted as time format	--	--
*left*	--	Verb	Past participle	--	--	--
.	--	Symbol	--	--	--	--

UniTA model predicts categories’ results together in the neural TTS runtime service. The same as training, UniTA converts the plain texts to word embeddings and then the multitask sequence tagging model predicts all the categories’ results. Some auxiliary modules are embedded after fine-tuning categories to further improve pronunciations. Finally, the pronunciation results are generated from UniTA.

Here is the figure of the UniTA model structure in Neural TTS:

UniTA model diagram

Pronunciation accuracy improved with UniTA

Compared with the traditional TTS text analyzer, UniTA reduces over 50% of pronunciation errors in improving pronunciation accuracy. It is already used many neural voice languages such as English (United States), English (United Kingdom), Chinese (Mandarin, simplified), Russian (Russia), German (Germany), Japanese (Japan), Korean (Korea), Polish (Poland) and Finnish (Finland). Due to varying types of grammar in language, not all categories are suitable for every language. For example, Chinese and Japanese heavily depend on word segmentation and polyphone while these languages don’t need morphology or abbreviation expansion.

Here are some samples of the pronunciation improvement using UniTA.

Category	Language	Input text (target word bolded)	Previous pronunciation	Current pronunciation
Word Segmentation	Chinese (Mandarin, simplified)	太子与三殿下行过礼后坐了片刻就离开了。	“三殿 / 下行 / 过礼”	“三殿下 / 行过礼”
Word Segmentation	Chinese (Mandarin, simplified)	叶奎最终还是在剧痛下泄了气	“剧痛 / 下泄了气”	“剧痛下 / 泄了气”
Word Segmentation	German (Germany)	kulturform	kult+urform	kultur+form
Word Segmentation	Korean (Korea)	해외감염병	h̬ɛwɛg̥mjʌmbjʌŋ	h̬ɛwɛg̥mjʌmpjʌŋ
Morphology - case ambiguity	Russian (Russia)	Количество ударов по воротам (15 против 7) также говорит о преимуществе чемпионов мира	Семь	Семи
Abbreviation Expansion	English (United States)	Joined TX Army National Guard in 1979.	T.X.	Texas
Text Normalization	English (United States)	The Downtown Cabaret Theatre’s Main Stage Theatre division concludes its 2010/11 season with the Tony Award winning musical, in the heights by Lin-Manuel Miranda.	November 2010	2010 to 2011
Polyphone disambiguation	Chinese (Mandarin, simplified)	卓文君听琴后，理解了琴曲的含意，不由脸红耳热，心驰神往。	qu1	qu3
Polyphone disambiguation	English (United States)	I received a copy early in November, and read and contemplated it's provisions with great satisfaction.
Polyphone disambiguation	Japanese (Japan)	パッケージには、富士屋ホテルが発刊した「We Japanese」内の説明用の挿絵を採用。	うち (w u - ch i)	ない (n a - y i)

Hear how the Cortana voice pronounces each word accurately with UniTA.

Get started

With these updates, we’re excited to continue to power accurate, natural and intuitive voice experiences for customers world-wide. Azure Text-to-Speech service provides more than 200 voices in over 50 languages for developers all over the world.

Let us know how you are using or plan to use Neural TTS voices in this form. If you prefer, you can also contact us at mstts [at] microsoft.com. We look forward to hearing your experience and developing more compelling services together with you for the developers around the world.

For more information:

Try the demo
See our documentation
Check out our sample code

Get skilled on AI and ML – on your terms with Azure AI

Anand_Raman — Wed, 27 Jan 2021 21:37:16 GMT

Azure’s AI portfolio has options for every developer and data scientist, and we’re committed to empowering you to develop applications and machine learning models on your terms. Azure enables you to develop in your preferred language, environment, and machine learning framework, and allows you to deploy anywhere - to the cloud, on-premises, or the edge. We help improve your productivity regardless of your skill level, with code-first and low code/no code options which can help you accelerate the development process. We’re also devoted to empowering you with resources to help you get started with Azure AI and machine learning, grow your skills, and start building impactful solutions.

Announcing new AI & ML resource pages for developers and data scientists

Today we’re excited to announce new resources pages on Azure.com, with a rich set of content for data scientists and developers. Whether you’re new to AI and ML, or new to Azure, the videos, tutorials, and other content on these pages will help you get started.

Learn how your peers around the world are using Azure AI to develop AI and machine learning solutions on their terms to solve business challenges.
Grow your skills with curated learning journeys to help your skill up on Azure AI and Machine Learning in 30 days. Each learning journey has videos, tutorials, and hands-on exercises to help prepare you to pass a Microsoft certification in just 4 weeks. Upon completing the learning journey, you’ll be eligible to receive 50% off a Microsoft Certification exam.
Engage with our engineering teams and stay up to date with the latest innovations on our AI Tech Community, where you’ll find blogs, discussion forums, and more.

Pictured above: ML learning journey for developers and data scientists.

Register for the Azure AI Hackathon

Finally, put your skills to the test by entering the Azure AI Hackathon, which starts today and will run through March 22^nd, 2021. Winners will be announced in early April. The most innovative and impactful projects will win prizes up to $10,000 USD. We look forward to seeing what you build with Azure AI.

Get started today

Check out the pages to get started with your 30-day learning journey, and register for the hackathon:

How to build a voice-enabled grocery chatbot with Azure AI

wmendoza — Tue, 26 Jan 2021 00:30:09 GMT

Chatbots have become increasingly popular in providing useful and engaging experiences for customers and employees. Azure services allow you to quickly create bots, add intelligence to them using AI, and customize them for complex scenarios.

In this blog, we’ll walk through an exercise which you can complete in under two hours, to get started using Azure AI Services. This intelligent grocery bot app can help you manage your shopping list using voice commands. We’ll provide high level guidance and sample code to get you started, and we encourage you to play around with the code and get creative with your solution!

Features of the application:

Add or delete grocery items by dictating them to Alexa.

Easily access the grocery list through an app.
Check off items using voice commands; for example, “Alexa, remove Apples from my grocery list."
Ask Alexa to read the items you have in your grocery list.
Automatically organize items by category to help save time at the store.
Use any laptop or Web Apps to access the app and sync changes across laptop and phone.

Prerequisites:

If you don't have an Azure subscription, create a free account before you begin. If you have a subscription, log in to the Azure Portal.
Amazon Alexa account
Python 3.6 or above

Key components:

Azure Bot Service to develop bot and publish to Alexa channel.
Microsoft Bot Framework Emulator to test and debug bots using Bot Framework SDK.
Alexa skills to interact with the bot using voice commands via Amazon Alexa.
Language Understanding to help users interact with the bot with natural language, by enabling the bot to understand user intent.

Solution Architecture

App Architecture Description:

The user accesses the chatbot by invoking it as an Alexa skill.
User is authenticated with Azure Active Directory.
User interacts with the chatbot powered by Azure Bot Service; for example, user requests bot to add grocery items to a list.
Azure Cognitive Services process the natural language request to understand what the user wants to do. (Note: If you wanted to give your bot its own voice, you can choose from over 200 voices and 54 languages/locales. Try the demo to hear the different natural sounding voices.)
The bot adds or removes content in the database.

Another visual of the flow of data within the solution architecture is shown below.

Implementation

High level overview of steps involved in creating the app along with some sample code snippets for illustration:

We’ll start by creating an Azure Bot Service instance, and adding speech capabilities to the bot using the Microsoft Bot Framework and the Alexa skill. Bot Framework, along with Azure Bot Service, provides the tools required to build, test, deploy, and manage the end-to-end bot development workflow. In this example, we are integrating Azure Bot Service with Alexa, which can process speech inputs for our voice-based chatbot. However, for chatbots deployed across multiple channels, and for more advanced scenarios, we recommend using Azure’s Speech service to enable voice-based scenarios. Try the demo to listen to the over 200 high quality voices available across 54 languages and locales.

The first step in the process is to login into Azure portal and follow the steps here to create an Azure Bot Service resource and a web app bot. To add voice capability to the bot, click on channels to add Alexa (see the below snapshot) and note the Alexa Service Endpoint URI.

Azure Bot Service Channels

Next, we need to log into the Alexa Developer Console and create an Amazon Alexa skill. After creating the skill, we are presented with the interaction model. Replace the JSON Editor with the below example phrases.

{ "interactionModel": { "languageModel": { "invocationName": "Get grocery list", "intents": [ { "name": "AMAZON.FallbackIntent", "samples": [] }, { "name": "AMAZON.CancelIntent", "samples": [] }, { "name": "AMAZON.HelpIntent", "samples": [] }, { "name": "AMAZON.StopIntent", "samples": [] }, { "name": "AMAZON.NavigateHomeIntent", "samples": [] }, { "name": "Get items in the grocery", "slots": [ { "name": "name", "type": "AMAZON.US_FIRST_NAME" } ], "samples": [ "Get grocery items in the list", "Do I have bread in my list", ] } ], "types": [] } } }

Next, we’ll integrate the Alexa Skill with our Azure bot. We’ll need two pieces of information to do this: the Alexa Skill ID and the Alexa Service Endpoint URI. First, get the Skill ID either from the URl in the Alexa portal, or by going to the Alexa Developer Console and clicking “view Skill ID”. The skill ID should be a value like ‘amzn1.ask.skil.A GUID’. Then, get the Alexa Service Endpoint URI from the Azure portal, by going to the channels page of our Azure Web App Bot in the Azure portal, and clicking on Alexa to copy the Alexa Service Endpoint URI. Then integrate as shown:

Amazon Developer Console: After building the Alexa Skill, click on Endpoint and paste the Alexa Service Endpoint URI that we copied from the Azure portal and save the Endpoints.
Azure Portal: Go to the channels page of the Azure Bot, click on Alexa, and paste the Alexa Skill ID that we copied from the Alexa Developer Console.

Now, we’ll download and the bot locally for testing using the Bot Framework Emulator. Click on “Build” in the Azure Web Bot app to download the source code locally with Bot Framework Emulator. Modify app.py as below:
# Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. from http import HTTPStatus from aiohttp import web from aiohttp.web import Request, Response, json_response from botbuilder.core import ( BotFrameworkAdapterSettings, ConversationState, MemoryStorage, UserState, ) from botbuilder.core.integration import aiohttp_error_middleware from botbuilder.schema import Activity from config import DefaultConfig from dialogs import MainDialog, groceryDialog from bots import DialogAndWelcomeBot from adapter_with_error_handler import AdapterWithErrorHandler CONFIG = DefaultConfig() # Create adapter. # See https://aka.ms/about-bot-adapter to learn more about how bots work. SETTINGS = BotFrameworkAdapterSettings(CONFIG.APP_ID, CONFIG.APP_PASSWORD) # Create MemoryStorage, UserState and ConversationState MEMORY = MemoryStorage() USER_STATE = UserState(MEMORY) CONVERSATION_STATE = ConversationState(MEMORY) # Create adapter. # See https://aka.ms/about-bot-adapter to learn more about how bots work. ADAPTER = AdapterWithErrorHandler(SETTINGS, CONVERSATION_STATE) # Create dialogs and Bot RECOGNIZER = IntelligentGrocery(CONFIG) grocery_DIALOG = groceryDialog() DIALOG = MainDialog(RECOGNIZER, grocery_DIALOG) BOT = DialogAndWelcomeBot(CONVERSATION_STATE, USER_STATE, DIALOG) # Listen for incoming requests on /api/messages. async def messages(req: Request) -> Response: # Main bot message handler. if "application/json" in req.headers["Content-Type"]: body = await req.json() else: return Response(status=HTTPStatus.UNSUPPORTED_MEDIA_TYPE) activity = Activity().deserialize(body) auth_header = req.headers["Authorization"] if "Authorization" in req.headers else "" response = await ADAPTER.process_activity(activity, auth_header, BOT.on_turn) if response: return json_response(data=response.body, status=response.status) return Response(status=HTTPStatus.OK) APP = web.Application(middlewares=[aiohttp_error_middleware]) APP.router.add_post("/api/messages", messages) if __name__ == "__main__": try: web.run_app(APP, host="localhost", port=CONFIG.PORT) except Exception as error: raise error
Next, we’ll run and test the bot with Bot Framework Emulator. From the terminal, navigate to the code folder and run pip install -r requirements.txt to install the required packages to run the bot. Once the packages are installed, run python app.py to start the bot. The bot is ready to test as shown below:
}

Open the bot and add the below port number into the following URL.
Bot Framework Emulator view

Now we’re ready to add natural language understanding so the bot can understand user intent. Here, we’ll use Azure’s Language Understanding Cognitive Service (LUIS), to map user input to an “intent” and extract “entities” from the sentence. In the below illustration, the sentence “add milk and eggs to the list” is sent as a text string to the LUIS endpoint. LUIS returns the JSON seen on the right.
Language Understanding utterances diagram

Use the below template to create a LUIS JSON model file where we specify intents and entities manually. After the “IntelligentGrocery” app is created in the LUIS portal under “Import New App”, upload the JSON file with the below intents and entities.

{ "text": "access the groceries list", "intent": "Show", "entities": [ { "entity": "ListType", "startPos": 11, "endPos": 19, "children": [] } ] }, { "text": "add bread to the grocery list", "intent": "Add", "entities": [ { "entity": "ListType", "startPos": 23, "endPos": 29, "children": []

The above sample intents are for adding items and accessing the items in the grocery list. Now, it’s your turn to add additional intents to perform the below tasks, using the LUIS portal. Learn more about how to create the intents here.

Intents

Name	Description
CheckOff	Mark the grocery items as purchased.
Confirm	Confirm the previous action.
Delete	Delete items from the grocery list.

Once the intents and entities are added, we will need to train and publish the model so the LUIS app can recognize utterances pertaining to these grocery list actions.

Language Understanding (LUIS) Portal

After the model has been published in the LUIS portal, click ‘Access your endpoint Urls’ and copy the primary key, example query and endpoint URL for the prediction resource.
Language Understanding endpoint

Language Understanding (LUIS) Prediction view

Navigate to the Settings page in the LUIS portal to retrieve the App ID.
Application settings

Finally, test your Language Understanding model. The endpoint URL will be in the below format, with your own custom subdomain, and app ID and endpoint key replacing APP-ID, and KEY_ID. Go to the end of the URL and enter an intent; for example, “get me all the items from the grocery list”. The JSON result will identify the top scoring intent and prediction with a confidence score. This is a good test to see if LUIS can learn what should be predicted with the intent.

https://YOUR-CUSTOM-SUBDOMAIN.api.cognitive.microsoft.com/luis/prediction/v3.0/apps/APP-ID/slots/production/predict?subscription-key=KEY-ID&verbose=true&show-all-intents=true&log=true&query=YOUR_QUERY_HERE

Additional Ideas

We’ve now seen how to build a voice bot leveraging Azure services to automate a common task. We hope it gives you a good starting point towards building bots for other scenarios as well. Try out some of the ideas below to continue building upon your bot and exploring additional Azure AI services.

Add Google Home assistant as an additional channel to receive voice commands.
Add a PictureBot extension to your bot and add pictures of your grocery items. You will need to create intents that trigger actions that the bot can take, and create entities that require these actions. For example, an intent for the PictureBot may be “SearchPics”. This could trigger Azure Cognitive Search to look for photos, using a “facet” entity to know what to search for. See what other functionality you can come up with!
Use Azure QnA maker to enable your bot to answer FAQs from a knowledge base. Add a bit of personality using the chit-chat feature.
Integrate Azure Personalizer with your voice chatbot to enables the bot to recommend a list of products to the user, providing a personalized experience.
Include Azure Speech service to give your bot a custom, high quality voice, with 200+ Text to Speech options across 54 different locales/languages, as well as customizable Speech to Text capabilities to process voice inputs.
Try building this bot using Bot Framework Composer, a visual authoring canvas.

How to build an intelligent travel journal using Azure AI

maddybutzbach — Tue, 26 Jan 2021 00:26:41 GMT

AI capabilities can enhance many types of applications, enabling you to improve your customer experience and solve complex problems. With Azure Cognitive Services, you can easily access and customize industry-leading AI models, using the tools and languages of your choice.

In this blog, we’ll walk through an exercise which you can complete in under an hour, to get started using Azure AI Services. Many of us are dreaming of traveling again, and building this intelligent travel journal app can help you capture memories from your next trip, whenever that may be. We’ll provide high level guidance and sample code to get you started, and we encourage you to play around with the code and get creative with your solution!

Features of the application:

Capture voice memos, voice tag photos, and transcribe speech to text.
Automatically tag your photos based on key phrase extraction and analysis of text in pictures.
Translate tags and text into desired language.
Organize your memos by key phrase and find similar travel experiences you enjoyed with AI-powered search.

Prerequisites:

If you don't have an Azure subscription, create a free account before you begin. If you have a subscription, log in to the Azure Portal.
To run the provided sample code, you will need Visual Studio 2019 and .NET Core 3.1 or above (for FotoFly)
Refer to this tutorial for detailed guidance on how to publish a console app.

Key Azure technologies:

Speech Service batch transcription for speech to text transcription
Text Analytics for key phrase/intent extraction
Computer Vision for analyzing text in images
Translator to normalize tags/text into desired language.
Open Source FotoFly library for photo tagging. Alternatively, you can use blob metadata but functionality will be limited.
Azure Cognitive Search for AI-powered search.

NOTE: For more information refer to the “References.txt” file under respective folders within JournalHelper library project in the provided sample solution with this blog.

Solution Architecture

App Architecture Description:

User records a voice memo; for example, to accompany an image they’ve captured. The recorded file is stored in a file repository (alternatively, you could use a database).
The recorded voice memo (e.g. .m4a) is converted into desired format (e.g. .wav), using Azure’s Speech Service batch transcription capability.
The folder containing voice memos is uploaded to a Blob container.
Images are uploaded into a separate container for analysis of any text within the photos, using Azure Computer Vision.
Use Translator to translate text to different languages, as needed. This may be useful to translate foreign street signs, menus, or other text in images.
Extract tags from the generated text files using Text Analytics, and send tags back to the corresponding image file. Tags can be travel related (#milan, #sunset, #Glacier National Park), or based on geotagging metadata, photo metadata (camera make, exposure, ISO), and more.
Create a search indexer with Azure Cognitive Search, and use the generated index to search your intelligent travel journal.

Implementation

Sample code

The entire solution code is available for download at this link. Download/clone and follow instructions in ReadMe.md solution item for further setup.

Implementation summary

The sample is implemented using various client libraries and samples available for Azure Cognitive Services. All these services are grouped together into a helper library project named “journalhelper”. In the library we introduce a helper class to help with scenarios that combine various Cognitive Services to achieve desired functionality.

We use “.Net Core console app” as the front end to test the scenarios. This sample also uses another open source library (FotoFly), which is ported to .Net Core here, to access and edit image metadata.

High level overview of steps, along with sample code snippets for illustration:

Start by batch transcribing voice memos and extracting key tags from the text output. Group the input voice memos into a folder, upload them into an Azure Blob container or specify a list of their URls, and use batch transcription to get results back into the Azure Blob container, as well as a folder in your file system. The following code snippet illustrates how helper functions can be grouped together for a specific functionality. It combines local file system, Azure storage containers, and Cognitive Services speech batch transcription API.

Console.WriteLine("Uploading voice memos folder to blob container..."); Helper.UploadFolderToContainer( HelperFunctions.GetSampleDataFullPath(customSettings.SampleDataFolders.VoiceMemosFolder), customSettings.AzureBlobContainers.InputVoiceMemoFiles, deleteExistingContainer); Console.WriteLine("Branch Transcribing voice memos using containers..."); //NOTE: Turn the pricing tier for Speech Service to standard for this below to work. await Helper.BatchTranscribeVoiceMemosAsync( customSettings.AzureBlobContainers.InputVoiceMemoFiles, customSettings.AzureBlobContainers.BatchTranscribedJsonResults, customSettings.SpeechConfigSettings.Key, customSettings.SpeechConfigSettings.Region); Console.WriteLine("Extract transcribed text files into another container and folder, delete the intermediate container with json files..."); await Helper.ExtractTranscribedTextfromJsonAsync( customSettings.AzureBlobContainers.BatchTranscribedJsonResults, customSettings.AzureBlobContainers.InputVoiceMemoFiles, customSettings.AzureBlobContainers.ExtractedTranscribedTexts, HelperFunctions.GetSampleDataFullPath(customSettings.SampleDataFolders.BatchTranscribedFolder), true);

Next, create tags from the transcribed text. Sample helper function using the Text Analytics client library is listed below.

//text analytics public static void CreateTagsForFolderItems(string key, string endpoint, string batchTranscribedFolder, string extractedTagsFolder) { if (!Directory.Exists(batchTranscribedFolder)) { Console.WriteLine("Input folder for transcribed files does not exist"); return; } // ensure destination folder path exists Directory.CreateDirectory(extractedTagsFolder); TextAnalyticsClient textClient = TextAnalytics.GetClient(key, endpoint); var contentFiles = Directory.EnumerateFiles(batchTranscribedFolder); foreach(var contentFile in contentFiles { var tags = TextAnalytics.GetTags(textClient, contentFile).ConfigureAwait(false).GetAwaiter().GetResult(); // generate output file with tags string outFileName = Path.GetFileNameWithoutExtension(contentFile); outFileName += @"_tags.txt"; string outFilePath = Path.Combine(extractedTagsFolder, outFileName); File.WriteAllLinesAsync(outFilePath, tags).Wait() ; } }

The actual client library or service calls are made as shown:

static public async Task<IEnumerable<string>> GetTags(TextAnalyticsClient client, string inputTextFilePath) { string inputContent = await File.ReadAllTextAsync(inputTextFilePath); var entities = EntityRecognition(client, inputContent); var phrases = KeyPhraseExtraction(client, inputContent); var tags = new List<string>(); tags.AddRange(entities); tags.AddRange(phrases); return tags; }

Update tags to the photo/image file, using the open source FotoFly library. Alternatively, you can update the Blob metadata with these tags and include that in the search index, but the functionality will be limited to using Azure Blob storage.

string taggedPhotoFile = photoFile.Replace(inputPhotosFolder, OutPhotosFolder); File.Copy(photoFile, taggedPhotoFile, true); if (tags.Count > 0) { ImageProperties.SetPhotoTags(taggedPhotoFile, tags); }

Other useful functions to complete the scenario are:
1. Helper.ProcessImageAsync, and
2. Helper.TranslateFileContent

The first one can be used to extract text from images using OCR or regular text processing using Computer Vision. The second can detect the source language, translate using Azure’s Translator service into the desired output language, and then create more tags for an image file.

Finally, use Azure Cognitive Search to create an index from the extracted text files saved in the Blob container, enabling you to search for documents and create journal text files. For example, you can search for images by cities or countries visited, date, or even cuisines. You can also search for images by camera-related metadata or geolocation.

In this sample we have demonstrated simple built-in skillsets for entity and language detection. The solution can be further enhanced by adding additional data sources to process tagged images and their metadata, and adding additional information to the searches.

NOTE: The helper functions can be made more generic to take additional skillset input.

public static async Task CreateSearchIndexerAsync( string serviceAdminKey, string searchSvcUrl, string cognitiveServiceKey, string indexName, string jsonFieldsFilePath, string blobConnectionString, string blobContainerName ) { // Its a temporary arrangment. This function is not complete IEnumerable<SearchField> fields = SearchHelper.LoadFieldsFromJSonFile(jsonFieldsFilePath); // create index var searchIndex = await Search.Search.CreateSearchIndexAsync(serviceAdminKey, searchSvcUrl, indexName, fields.ToList()); // get indexer client var indexerClient = Search.Search.GetSearchIndexerClient(serviceAdminKey, searchSvcUrl); // create azure blob data source var dataSource = await Search.Search.CreateOrUpdateAzureBlobDataSourceAsync(indexerClient, blobConnectionString, indexName, blobContainerName); // create indexer // create skill set with minimal skills List<SearchIndexerSkill> skills = new List<SearchIndexerSkill>(); skills.Add(Skills.CreateEntityRecognitionSkill()); skills.Add(Skills.CreateLanguageDetectionSkill()); var skillSet = await Search.Search.CreateOrUpdateSkillSetAsync(indexerClient, indexName + "-skillset", skills, cognitiveServiceKey); var indexer = await Search.Search.CreateIndexerAsync(indexerClient, dataSource, skillSet, searchIndex); // wait for some time to have indexer run and load documents Thread.Sleep(TimeSpan.FromSeconds(20)); await Search.Search.CheckIndexerOverallStatusAsync(indexerClient, indexer); }

Finally, search documents and generate the corresponding journal files, utilizing the following functions:

Helper.SearchDocuments
Helper.CreateTravelJournal

Additional Ideas

In addition to the functionality described so far, there are many other ways you can leverage Azure AI to further enhance your intelligent travel journal and learn more advanced scenarios. We encourage you to explore some the following ideas to enrich your app:

Add real time voice transcription and store transcriptions in an Azure managed database, to correlate voice transcription with images in context.
Include travel tickets and receipts as images for OCR-based image analysis (Form Recognizer) and include them as journal artifacts.
Use multiple data sources for a given search index. We have simplified and only included text files to index in this sample, but you can include the tagged photos from a different data source for the same search index.
Add custom skills and data extraction for search indexer. Extract metadata from images and include as search content.
Extract metadata from video and audio content using Video Indexer.
Experiment with Language Understanding and generate more elaborate and relevant search content based on top scoring intents and entities. Sample keywords and questions related to current sample data are included in Objectives.docx solution item.
Build a consumer front-end app that stitches all of this together and displays the journal in a UI.

How to build a personal finance app using Azure

mernanashed — Tue, 26 Jan 2021 01:19:08 GMT

AI allows you to deliver breakthrough experiences in your apps. With Azure Cognitive Services, you can easily customize and deploy the same AI models that power Microsoft’s products, such as Xbox and Bing, using the tools and languages of your choice.

In this blog we will walk through an exercise that you can complete in under an hour and learn how to build an application that can be useful for you, all while exploring a set of Azure services. If you have ever wanted to get your financial transactions in order, look no further. With this exercise, we’ll explore how to quickly take a snap of a receipt from your phone and upload it for categorization, creating expense reports, and to gain insights to your spending. Remember, even though we’ll walk you through each step, you can always explore the sample code and get creative with your own unique solution!

Features of the application:

Snap a picture of your receipt and upload it using your smartphone
Extract relevant data from the images: Who issued the receipt? What was the total amount? What was purchased? All of this information can be effortlessly stored for exploration
Query the data: bring your receipts to life by extracting relevant and insightful information

Prerequisites

If you don't have an Azure subscription, create a free account before you begin. If you have a subscription, log in to the Azure Portal.
You will need to have python installed locally to run some of the samples.

Key Azure technologies:

Azure Form Recognizer scans image documents with optical character recognition and extracts text, key/value pairs, and tables from documents, receipts, and forms.
Form Recognizer’s prebuilt receipt model specifically extracts receipt data
Azure Blob Storage is used to store data
Azure Cognitive Search enriches the data by making it easily identifiable

Solution Architecture

App Architecture Description:

User uploads a receipt image from their mobile device
The uploaded image is verified and then sent to the Azure Form Recognizer to extract information
The image is analysed by the REST API within the Form Recognizer prebuilt receipt model
A JSON is returned that has both the text information and bounding box coordinates of the extracted receipt data
The resulting JSON is parsed and a simpler JSON is formed, saving only the relevant information needed
This receipt JSON is then stored in Azure Blob Storage
Azure Cognitive Search points directly to Azure Blob Storage and is used to index the data
The application queries this search index to extract relevant information from the receipts

Another visual of the flow of data within the solution architecture is shown below.

Now that we’ve explored the technology and services we’ll be using, let’s dive into building our app!

Implementation

To get started, data from receipts must be extracted; this is done by setting up the Form Recognizer service in Azure and connecting to the service to use the relevant API for receipts. A JSON is returned that contains the information extracted from receipts and is stored in Azure Blob Storage to be used by Azure Cognitive Search. Cognitive Search is then utilized to index the receipt data, and to search for relevant information.

High level overview of steps, along with sample code snippets for illustration:

Go to the Azure portal and create a new Form Recognizer resource. In the Create pane, provide the following information:

Name	A descriptive name for your resource.
Subscription	Select the Azure subscription which has been granted access.
Location	The location of your cognitive service instance. Different locations may introduce latency, but have no impact on the runtime availability of your resource.
Pricing Tier	The cost of your resource depends on the pricing tier you choose and your usage. For more information, see the API pricing details.
Resource Group	The Azure resource group that will contain your resource. You can create a new group or add it to a pre-existing group.

After Form Recognizer deploys, go to All Resources and locate the newly deployed resource. Save the key and endpoint from the resource’s key and endpoint page somewhere so you can access it later.
You can use the following Analyze Receipt API to start analyzing the receipt. Remember to replace <endpoint> & <subscription key> the values you saved earlier and replace <path to your receipt> with the local path to your scanned receipt image.
# Analyse script import json import time from requests import get, post # Endpoint URL endpoint = r"<endpoint url>" apim_key = "<subscription key>" post_url = endpoint + "/formrecognizer/v2.0/prebuilt/receipt/analyze" source = r"<path to your receipt>" headers = { # Request headers 'Content-Type': 'image/jpeg', 'Ocp-Apim-Subscription-Key': apim_key, } params = { "includeTextDetails": True } with open(source, "rb") as f: data_bytes = f.read() try: resp = post(url=post_url, data=data_bytes, headers=headers, params=params) if resp.status_code != 202: print("POST analyze failed:\n%s" % resp.text) quit() print("POST analyze succeeded:\n%s" % resp.headers) get_url = resp.headers["operation-location"] except Exception as e: print("POST analyze failed:\n%s" % str(e)) quit()
If you run this code and everything is as it should be, you'll receive a 202 (Success) response that includes an Operation-Location header, which the script will print to the console. This header contains an operation id that you can use to query the status of the asynchronous operation and get the results. In the following example value, the string after operations/ is the operation ID.

https://cognitiveservice/formrecognizer/v2.0/prebuilt/receipt/operations/54f0b076-4e38-43e5-81bd-b85b8835fdfb

Now you can call the Get Analyze Receipt Result API to get the Extracted Data.
# Get results. n_tries = 10 n_try = 0 wait_sec = 6 while n_try < n_tries: try: resp = get(url = get_url, headers = {"Ocp-Apim-Subscription-Key": apim_key}) resp_json = json.loads(resp.text) if resp.status_code != 200: print("GET Receipt results failed:\n%s" % resp_json) quit() status = resp_json["status"] if status == "succeeded": print("Receipt Analysis succeeded:\n%s" % resp_json) quit() if status == "failed": print("Analysis failed:\n%s" % resp_json) quit() # Analysis still running. Wait and retry. time.sleep(wait_sec) n_try += 1 except Exception as e: msg = "GET analyze results failed:\n%s" % str(e) print(msg) quit()

This code uses the operation id and makes another API call.

The JSON that is returned can be examined to get the required information - ‘readResults’ field will contain all lines of text that was decipherable, and the ‘documentResults’ field contains ‘key/value’ information for the most relevant parts of the receipt (e.g. the merchant, total, line items etc.)

The receipt image below,

resulted in the JSON from which we have extracted the following details:
MerchantName: THE MAD HUNTER TransactionDate: 2020-08-23 TransactionTime: 22:07:00 Total: £107.10

We will now create a JSON from all the data extracted from the analysed receipt. The structure of the JSON is shown below:
{ "id":"INV001", "user":"Sujith Kumar", "createdDateTime":"2020-10-23T17:16:32Z", "MerchantName":"THE MAD HUNTER", "TransactionDate":"2020-10-23", "TransactionTime":"22:07:00", "currency":"GBP", "Category":"Entertainment", "Total":"107.10", "Items":[ ] }

We can now save this JSON and build a search service to extract the information we want from it.

Before continuing onto step 8, you must have an Azure Storage Account with Blob storage.

We will now save the JSON files in an Azure Blob Storage container and use it as a source for the Azure Cognitive Search Service Index that we will create.
Sign-in to the Azure Portal and search for "Azure Cognitive Search" or navigate to the resource through Web > Azure Cognitive Search. Follow the steps to:

Choose a subscription
Set a resource group
Name the service appropriately
Choose a location
Choose a pricing tier for this service
Create your service
Get a key and URL endpoint

We will use the free Azure service, which means you can create three indexes, three data sources and three indexers. The dashboard will show you how many of each you have left. For this exercise you will create one of each.

In the portal, find the search service you created above and click Import data on the command bar to start the wizard. In the wizard, click on Connect to your data and specify the name, type, and connection information. Skip the ‘Enrich Content’ page and go to Customize Target Index.
For this exercise, we will use the wizard to generate a basic index for our receipt data. Minimally, an index requires a name and a fields collection; one of the fields should be marked as the document key to uniquely identify each document.

Fields have data types and attributes. The check boxes across the top are index attributes controlling how the field is used.

Retrievable means that it shows up in search results list. You can mark individual fields as off limits for search results by clearing this checkbox.
Key is the unique document identifier. It's always a string, and it is required.
Filterable, Sortable, and Facetable determine whether fields are used in a filter, sort, or faceted navigation structure.
Searchable means that a field is included in full text search. Only Strings are searchable.

Make sure you choose the following fields:

id
user
createdDateTime
MerchantName
TransactionDate
TransactionTime
Currency
Category
Total

Still in the Import data wizard, click Indexer > Name, and type a name for the indexer.

This object defines an executable process. For now, use the default option (Once) to run the indexer once, immediately.

Click Submit to create and simultaneously run the indexer.

Soon you should see the newly created indexer in the list, with status indicating "in progress" or success, along with the number of documents indexed.

The main service page provides links to the resources created in your Azure Cognitive Search service. To view the index you just created, click Indexes from the list of links.

Click on the index (azureblob-indexer in this case) from the list of links and view the index-schema.

Now you should have a search index that you can use to query the receipt data that’s been extracted from the uploaded receipts.

Click the search explorer

From the index drop down choose the relevant index. Choose the default API Version (2020-06-30) for this exercise.

In the search bar paste a query string (for eg. category='Entertainment')

You will get results as verbose JSON documents as shown below:

Now that you have built a query indexer and aimed it at your data you can now use it to build queries programmatically and extract information to answer some of the following questions:

How much did I spend last Thursday?
How much have I spent on entertainment over the last quarter?
Did I spend anything at ‘The Crown and Pepper’ last month?

Additional Ideas

In addition to the services and functionalities used throughout this exercise, there are numerous other ways you can use Azure AI to build in support for all kinds of receipts or invoices. For example, the logo extractor can be used to identify logos of popular restaurants or hotel chains, and the business card model can ingest business contact information just as easily as we saw with receipts.

We encourage you to explore some of the following ideas to enrich your application:

Search invoices for specific line items
Train the models to recognize different expense categories such as entertainment, supplies, etc.
Add Language Understanding (LUIS) to ask your app questions in natural language and extract formatted reports
Add Azure QnA Maker to your app and get insights such as how much you spent on entertainment last month, or other categories of insights you’d like to explore

Enhanced Table Extraction from documents with Form Recognizer

NetaH — Thu, 14 Jan 2021 19:03:46 GMT

Authors: Lei Sun, Neta Haiby, Cha Zhang, Sanjeev Jagtap

Documents containing tables pose a major hurdle for information extraction. Tables are often found in financial documents, legal documents, insurance documents, oil and gas documents and more. Tables in documents are often the most important part of the document but extracting data from tables in documents presents a unique set of challenges. Challenges include an accurate detection of the tabular region within an image, and subsequently detecting and extracting information from the rows and columns of the detected table, merged cells, complex tables, nested tables and more. Table extraction is the task of detecting the tables within the document and extracting them into a structured output that can be consumed by workflow applications such as robotic process automation (RPA) services, data analyst tools such as excel, databases and search services.

Customers often use manual processes for data extraction and digitization. However, with the new enhanced table extraction feature you can send a document (PDF or images) to Form Recognizer for extraction of all the information into a structured usable data at a fraction of the time and cost, so you can focus more time acting on the information rather than compiling it.

Table extraction challenges

Table extraction from a wide variety of document images is a challenging problem due to the heterogeneous table structures, diverse table contents, and erratic use of ruling lines. To name a few concrete examples, in financial reports and technical publications, some borderless tables may have complex hierarchical header structures, contain many multi-line, empty or spanned cells, or have large blank spaces between neighboring columns. In forms, some tables may be embedded in other more complex tabular objects (e.g., nested tables) and some neighboring tables may be very close to each other which makes it hard to determine whether they should be merged or not. In invoices, tables may have different sizes, e.g., some key-value pairs composed tables may contain only two rows/columns and some line-item tables may span multiple pages. Sometimes, some objects in document images like figures, graphics, code listings, structurally laid out text, or flow charts may have similar textures as tables, which poses another significant challenge for successful detection of tables and reduction of false alarms. To make matters worse, many scanned or camera-captured document images are of poor image quality, and tables contained in them may be distorted (even curved) or contain artifacts or noises. Existing table extraction solutions fall short of extracting tables from such document images with high accuracy, which has prevented workflow applications from effectively leveraging this technology.

Form Recognizer Table extraction

In recent years, the success of deep learning in various computer vision applications has motivated researchers to explore deep neural networks like convolutional neural networks (CNN) or graph neural networks (GNN) for detecting tables and recognizing table structures from document images. With these new technologies, the capability and performance of modern table extraction solutions have been improved significantly.

In the latest release of Form Recognizer, we created a state-of-the-art table extraction solution with cutting-edge deep learning technology. After validating that Faster/Mask R-CNN based table detectors are effective in detecting a variety of tables (e.g., bordered or borderless tables, tables embedded in other more complex tabular objects, and distorted tables) in document images robustly, we further proposed a new method to improve the localization accuracy of such detectors, and achieved state-of-the-art results on the ICDAR-2019 cTDaR table detection benchmark dataset by only using a lightweight ResNet18 backbone network (Table 1).

For the challenge of table recognition or table cell extraction, we leveraged existing CNN/GNN based approaches, which have proven to be robust to complex tables like borderless tables with complex hierarchical header structures and multi-line/empty/spanned cells. We further enhanced them to deal with distorted or even slightly curved tables in camera-captured document images, making the algorithm more widely applicable to different real-world scenarios. Figure 1 below shows a few examples to demonstrate such capabilities.

Easy and Simple to use

Try it out with the Form Recognizer Sample Tool.

Extracting tables from documents is as simple as 2 API calls, no training, preprocessing, or anything else needed. Just call the Analyze Layout operation with your document (image, TIFF, or PDF file) as the input and extracts the text, tables, selection marks, and structure of the document.

Step 1: The Analyze Layout Operation –

https://{endpoint}/formrecognizer/v2.1-preview.2/layout/analyze

The Analyze Layout call returns a response header field called Operation-Location. The Operation-Location value is a URL that contains the Result ID to be used in the next step.

Operation location -
https://cognitiveservice/formrecognizer/v2.1-preview.2/prebuilt/layout/analyzeResults/44a436324-fc4b-4387-aa06-090cfbf0064f

Step 2: The Get Analyze Layout Result Operation –

Once you have the operation location call the Get Analyze Layout Result operation. This operation takes as input the Result ID that was created by the Analyze Layout operation.

https://{endpoint}/formrecognizer/v2.1-preview.2/layout/analyzeResults/{resultId}

The output of the Get Analyze Layout Results will provide a JSON output with the extracted table – rows, columns, row span, col span, bounding box and more.

For example:

Get started

To get started create a Form Recognizer resource in the Azure Portal and try out your tables in the Form Recognizer Sample Tool. You can also use the Form Recognizer client library or REST API.
Note tables output is included in all parts of the Form Recognizer service – prebuilt, layout and custom in the JSON output pageResults section.
For additional questions please reach out to us at formrecog_contact@microsoft.com

Bot Framework Composer 1.3 is now available!

GaryPrettyMsft — Thu, 17 Dec 2020 10:42:45 GMT

This week, as the year draws to a close, we are excited to announce that Bot Framework Composer 1.3 is now available to download. Composer has come a long way since we made the product GA (generally available) at the Microsoft Build conference earlier this year and this is our biggest release yet, adding many significant capabilities and making building sophisticated bots and virtual assistants even easier!

New features to improve the developer experience and workflow

For developers who are working with Bot Framework Skills today, you will know that developing multiple bots locally that work together can sometimes be a challenge, especially when it comes to setting up debugging. In Composer 1.3, we have now added a multi-bot authoring and management experience to transform this scenario, adding the capability to create, manage and test multiple bots within a single project. With a single click, you can now start all local bots for debugging, enabling you to test your root (parent) bot, connected to one or more skills with no additional manual configuration needed.

Another significant enhancement is for the provisioning feature, which previously required developers to leave Composer and run a PowerShell script, copying back a resulting configuration into Composer. Now though, the provisioning process has been overhauled and users can now login to Azure, provision required resources and subsequently publish bots, all within the Composer environment!

Additionally, we have implemented a new settings experience, providing an improved interface, removing the need to manually edit the underlying JSON for common settings, whilst retaining the ability to make changes or add additional configuration manually if you need to.

Localization

In addition to the existing capability for developers to localize their bots, multilingual support has now been added to the Composer UI! You can now choose from a long list of available languages within the Application Settings pane to change the language displayed within Composer.

Preview features

As part of version 1.3, you can now choose to enable one or more preview features by choosing preview feature flags within the Composer settings page. These features are designed to give you early access and a chance to try what we are working on right now for future Composer releases. The following preview feature flags are now available.

Form Dialogs – Automatically generate a sophisticated dialog by simply providing the properties that you would like customers to provide as part of the conversation, with Composer then generating the appropriate dialog, language understanding (to enable dis-ambiguation and interruption scenarios) and bot responses (.lg files) assets.
Orchestrator – A new top-level recognizer which can help to arbitrate (dispatch) between multiple LUIS and QnA Maker models to ensure accurate routing of user requests to the appropriate language model or skill.
Package Manager – Developers can now discover and install packages from NuGet / NPM that contain re-usable assets, including dialogs, custom actions and .LG (language generation) files, that can be utilized by their bots. Once installed, assets contained within a package become available for use within a bot. Moving forward, we will provide guidance for how you can create and publish your own packages (including to internal feeds if desired), as well as making available a number of packages covering common scenarios that will ship with Composer.
Conversational core template – Built on the new package capabilities, surfaced via the preview of the Package Manager, we are developing a new component model for bot development using re-usable building blocks (packages). With this preview, users can create a bot using the new conversational core template which consists of a configurable runtime that can be extended with packages or importing additional skills.

Help us to improve Composer!

Within this release we have enabled the ability for users of Composer to opt in to sending usage information to us, to allow us to better understand how Composer is used. As we gather this telemetry, we can use it as an additional signal to help us prioritize our efforts in future releases and ensure we are focusing on the right features. You can help us by opting into providing usage data via the Data Collection section of the Composer settings page.

Finally, a huge thank you to all of our users for your support and feedback during 2020 - we are excited to bring more significant updates to you as we move into 2021. Happy Holidays to everyone from the entire Conversational AI team!

Azure Neural Text-to-Speech updates: 51 new voices added to the portfolio

GarfieldHe — Wed, 16 Dec 2020 06:46:34 GMT

This post was co-authored with Qinying Liao, Sheng Zhao, Gang Wang, Yueying Liu

Neural Text to Speech (Neural TTS), a powerful speech synthesis capability of Cognitive Services on Azure, enables you to convert text to lifelike speech which is close to human-parity. Since its launch, we have seen it widely adopted in a variety of scenarios by many Azure customers, from voice assistants to audio content creation. More and more customers are asking for richer and more diverse choices of synthetic voices for different use cases.

Today, we are excited to announce that Azure Neural TTS has added 51 new voices for a total of 129 neural voices across 54 languages/locales. With this release, we provide at least one male and one female voice for customers to choose in each language/locale. In total, Azure TTS now enables developers to reach millions more people with more than 200 voices available in standard and neural TTS.

What's new

Neural TTS has now been extended to support 51 new voices, which will bring to you the capability to have both male and female voices in each language for your apps. You can hear samples of the voices below, or try them with your own text in our demo.

46 new voices are generally available

In total 46 new voices are released across the 49 locales that are generally available in the Azure data centers/regions that support neural TTS (see the full list of Azure regions here).

Locale	Language	Gender	Voice	Sample audio
ar-EG	Arabic (Egypt)	Male	ShakirNeural	البركان هو أكثر ما في الطبيعــة إثارة للرهبة
ar-SA	Arabic (Saudi Arabia)	Male	HamedNeural	الناس مَعادن، تصدأ بالملل، وتتمدد بالأمل، وتنكمش بالألم
bg-BG	Bulgarian (Bulgaria)	Male	BorislavNeural	Шофьорът задължително трябва да вземе експерт за второ мнение, за да провери дали всички системи на автомобила работят нормално.
ca-ES	Catalan (Spain)	Male	EnricNeural	Les activitats docents tenen lloc al campus del Poblenou.
ca-ES	Catalan (Spain)	Female	JoanaNeural	L'artista està considerat com el pintor de les multituds.
cs-CZ	Czech (Czech)	Male	AntoninNeural	Opravdový zasvěcenec ví, že nejmocnějším tajemstvím je to, které nemá žádný obsah.
da-DK	Danish (Denmark)	Male	JeppeNeural	61 procent af de kandidatstuderende er kvinder.
de-AT	German (Austria)	Male	JonasNeural	Das ist das letzte lange Pfingstwochenende für Schülerinnen und Schüler.
de-CH	German (Switzerland)	Male	JanNeural	Eine Person, die sich bei Brandausbruch im oberen Stock aufgehalten hat, hat sich noch rechtzeitig in Sicherheit bringen können.
el-GR	Greek (Greece)	Male	NestorasNeural	Συγκλονιστικές εξελίξεις και ανατροπές στα επόμενα επεισόδια .
en-CA	English (Canada)	Male	LiamNeural	He had held the position since 2010.
en-IE	English (Ireland)	Male	ConnorNeural	Life is short, think before you talk.
en-IN	English (India)	Male	PrabhatNeural	Sometimes you can see snow on the mountains.
fi-FI	Finnish (Finland)	Male	HarriNeural	Yhtiö kertoi loppuvuoden tuloksestaan ennakkotietoja.
fi-FI	Finnish (Finland)	Female	SelmaNeural	Hevoset ovat uljaita ja nopeita eläimiä.
fr-CH	French (Switzerland)	Male	FabriceNeural	La Suisse comptera 5,6 millions (12%) de personnes actives en 2050.
he-IL	Hebrew (Israel)	Male	AvriNeural	הוא אמר שהמספרים מדאיגים בשל עצמם, אבל בכל הישיבות שלנו המסקנה היא שזה סימפטום למשהו רחב יותר.
hi-IN	Hindi (India)	Male	MadhurNeural	सिद्धार्थ ने भी शहनाज के साथ इस इवेंट की फोटो शेयर की है।
hr-HR	Croatian (Croatia)	Male	SreckoNeural	Video je pregledan gotovo 70 tisuća puta, a neki od obožavatelja su mu u komentarima pisali kako ih je motivirao.
hu-HU	Hungarian (Hungary)	Male	TamasNeural	A lakóhelyem nagyon komfortos.
id-ID	Indonesian (Indonesia)	Female	GadisNeural	Inflasi ringan terjadi apabila kenaikan harga berada di bawah angka 10% setahun.
ms-MY	Malay (Malaysia)	Male	OsmanNeural	Setiap individu perlu memakai topeng muka ketika berada di luar.
nb-NO	Norwegian (Bokmål, Norway)	Male	FinnNeural	Jansson forteller at den svenske øya tar imot rundt 8000 besøkende fra Norge årlig.
nb-NO	Norwegian (Bokmål, Norway)	Female	PernilleNeural	For en fantastisk forestilling!
nl-NL	Dutch (Netherlands)	Female	FennaNeural	De afstand tussen Rotterdam en Breda is ongeveer 45 km.
nl-NL	Dutch (Netherlands)	Male	MaartenNeural	Zij heeft haar studie al een tijdje geleden afgerond.
pl-PL	Polish (Poland)	Female	AgnieszkaNeural	To już nie będzie to samo, będzie drożej.
pl-PL	Polish (Poland)	Male	MarekNeural	Na wszelki wypadek sprawdź, czy coś cię jednak nie zaskoczy.
pt-PT	Portuguese (Portugal)	Male	DuarteNeural	Para a aprovação do exame, tenho de ter pelo menos 80% das respostas corretas.
pt-PT	Portuguese (Portugal)	Female	RaquelNeural	A minha mãe ensinou-me que devo ter respeito por todos, mas principalmente pelos mais velhos.
ro-RO	Romanian (Romania)	Male	EmilNeural	Actul normativ se axează pe instituirea de măsuri active, 41,5 % din salariul de bază la revenirea din șomaj tehnic.
ru-RU	Russian (Russia)	Male	DmitryNeural	Ранее посольство требовало от агентства опровержения статьи о количестве больничных коек в России.
ru-RU	Russian (Russia)	Female	SvetlanaNeural	Изменений в организме людей, попробовавших еду без приправ, не произошло.
sk-SK	Slovak (Slovakia)	Male	LukasNeural	Zápis 45 % je v skutočnosti iba skratka pre zlomok.
sl-SI	Slovenian (Slovenia)	Male	RokNeural	Zloraba bonov in dvigovanje cen turističnih storitev je nesprejemljivo ravnanje.
sv-SE	Swedish (Sweden)	Male	MattiasNeural	Båda lagen bjöd på riktigt bra hockey och skapade flera riktigt bra målchanser.
sv-SE	Swedish (Sweden)	Female	SofieNeural	Det fanns ingen trafik runt torget.
ta-IN	Tamil (India)	Male	ValluvarNeural	எவ்வளவு அருமையான பாடல் அது!
te-IN	Telugu (India)	Male	MohanNeural	అబ్బ, ఎంత పెద్ద భవనమో!
th-TH	Thai (Thailand)	Male	NiwatNeural	ธุรกิจขายอาหารเป็นธุรกิจที่ได้รับความนิยมมากที่สุด
tr-TR	Turkish (Turkey)	Male	AhmetNeural	Sosyal mesafeye büyük ölçüde riayet eden çocuklar, başta mahalle parkları olmak üzere sahiller ve oyun parklarında enerji attı.
vi-VN	Vietnamese (Vietnam)	Male	NamMinhNeural	Nhiệt độ hiện tại ở thành phố Hồ Chí Minh là 38 độ C.
zh-HK	Chinese (Cantonese, Traditional)	Female	HiuMaanNeural	抗疫舉措成為安全重啟經濟的重要一環。
zh-HK	Chinese (Cantonese, Traditional)	Male	WanLungNeural	隨着疫情緩和，愈來愈多人回到辦公室上班，但是很多人仍想留在家中工作（work from home）。
zh-TW	Chinese (Taiwanese Mandarin)	Female	HsiaoChenNeural	賭博的勝率應該不到50%。
zh-TW	Chinese (Taiwanese Mandarin)	Male	YunJheNeural	台北車站大廳能不能坐，連日引發正反意見。

5 new voices are in public preview

We have also added 5 male voices in the 5 low-resource languages that have been supported since November. These voices are available in public preview in three Azure regions: EastUS, SouthEastAsia and WestEurope.

Hear the samples below:

Locale	Language	Gender	Voice Name	Sample audio
et-EE	Estonian (Estonia)	Male	KertNeural	Ametlikku meetodit sellise pettuse avastamiseks ei olegi olemas.
ga-IE	Irish (Ireland)	Male	ColmNeural	Ritheadh próiseas comhairliúcháin faoin scéal sa bhfómhar.
lt-LT	Lithuanian (Lithuania)	Male	LeonasNeural	Aišku, anksčiau ar vėliau paaiškės tos priežastys.
lv-LV	Latvian (Latvia)	Male	NilsNeural	Aizvadīto gadu uzņēmums noslēdzis ar 6,3 miljonu eiro zaudējumiem.
mt-MT	Maltese (Malta)	Male	JosephNeural	Anki tfajjel tal-primarja jaf li l-popolazzjoni tikber fejn hemm il-prosperità.

With this release, we now support a total of 129 neural voices across 54 languages/locales. In addition, over 70 standard voices are available in 49 languages/locales. Visit Language support - Speech service - Azure Cognitive Services | Microsoft Docs for the full language and voice list.

Continuous voice quality improvement

In general, Neural TTS can convert text to lifelike speech, however there are nuances that can always be improved. For example, we have customers who have requested the ability to support a scenario where Katja, our de-DE neural voice, can pronounce English words in the context of a German sentence. This was valuable feedback, and we anticipate a similar need across languages.

For German, we observed that our users prefer the voice to handle an English word/phrase as close as the native English pronunciation. To enable a voice model to speak English as a second language, it is normally required that we collect the speech data of the same speaker speaking English besides his/her native language. This is a big challenge as we do not have sufficient multi-language speech data from our German voice talents. By leveraging cross-lingual capability of UNI-TTS, we are able to generate more English pronunciation data with the transferred voice from our German voice talent. Such data is used to improve the quality of the English word/phrase pronunciations for the German Katja voice, so Katja can pronounce English words in a more natural way.

The CMOS metric is used to measure the improvement of the English word pronunciation for Katja. Below table shows that the updated model is significantly better in pronouncing English words in the context of a German sentence.

Script	Old	New
Star Wars - Das Erwachen der Macht
Three Billboards outside Ebbing, Missouri.

This improvement has now been released to the Azure Neural TTS service for Katja. Moving forward, we’ll extend this capability to support more languages.

Tell us your experience!

By offering more voices across more languages and locales, we anticipate developers across the world will be able to build applications that change experiences for millions. Whether you’re building a voice-enabled chatbot or IoT device, an IVR solution, adding read-aloud features to your app, converting e-books to audio books, or even adding Speech to a translation app, you can make all these experiences natural sounding and fun with Neural TTS.

Get started

Add voice to your app in 15 minutes

Explore the available voices in this demo

Build a voice-enabled bot

Deploy Azure TTS voices on prem with Speech Containers

Build your custom voice

Meta-data driven key-value pairs extraction with Azure Form Recognizer

IssaghaBa — Mon, 30 Nov 2020 23:07:44 GMT

Most organizations are now aware of how valuable the forms (pdf, images, videos…) they keep in their closets are. They are looking for best practices and most cost-effective ways and tools to digitize those assets. By extracting the data from those forms and combining it with existing operational systems and data warehouses, they can build powerful AI and ML models to get insights from it to deliver value to their customers and business users.

With the Form Recognizer Cognitive Service, we help organizations to harness their data, automate processes (invoice payments, tax processing …), save money and time and get better accuracy.

Figure 1:Typical form

In my first blog about the automated form processing, I described how you can extract key-value pairs from your forms in real-time using the Azure Form Recognizer cognitive service. We successfully implemented that solution for many customers.

Often, after a successful PoC or MVP, our customers realize that, not only they need this real time solution but, they also have a huge backlog of forms they would like to ingest into their relational, NoSQL databases or data lake, in a batch fashion. They have different types of forms and they don’t want to build a model for each type. They are also looking for easy and quick way to ingest the new type of forms.

In this blog, we’ll describe how to dynamically train a form recognizer model to extract the key-value pairs of different type of forms and at scale using Azure services. We’ll also share a github repository where you can download the code and implement the solution we describe in this post.

The backlog of forms maybe in your on-premises environment or in a (s)FTP server. We assume that you were able to upload them into an Azure Data Lake Store Gen 2, using Azure Data Factory, Storage Explorer or AzCopy. Therefore, the solution we’ll describe here will focus on the data ingestion from the data lake to the (No)SQL database.

Our product team published a great tutorial on how to Train a Form Recognizer model and extract form data by using the REST API with Python. The solution described here demonstrates the approach for one model and one type of forms and is ideal for real-time form processing.

The value-add of the post is to show how to automatically train a model with new and different type of forms using a meta-data driven approach, in batch mode.

Below is the high-level architecture.

Figure 2: High Level Architecture

Azure services required to implement this solution

To implement this solution, you will need to create the below services:

Form Recognizer resource:

Form Recognizer resource to setup and configure the form recognizer cognitive service, get the API key and endpoint URI.

Azure SQL single database:

We will create a meta-data table in Azure SQL Database. This table will contain the non-sensitive data required by the Form Recognizer Rest API. The idea is, whenever there is a new type of form, we just insert a new record in this table and trigger the training and scoring pipeline.
The required attributes of this table are:

form_description: This field is not required as part of the training of the model the inference. It just to provide a description of the type of forms we are training the model for (example client A forms, Hotel B forms,...)

training_container_name: This is the storage account container name where we store the training dataset. It can be the same as scoring_container_name

training_blob_root_folder: The folder in the storage account where we’ll store the files for the training of the model.

scoring_container_name: This is the storage account container name where we store the files we want to extract the key value pairs from. It can be the same as the training_container_name

scoring_input_blob_folder: The folder in the storage account where we’ll store the files to extract key-value pair from.
model_id: The identify of model we want to retrain. For the first run, the value must be set to -1 to create a new custom model to train. The training notebook will return the newly created model id to the data factory and, using a stored procedure activity, we’ll update the meta data table with in the Azure SQL database.

Whenever you had a new form type, you need to reset the model id to -1 and retrain the model.

file_type: The supported types are application/pdf, image/jpeg, image/png, image/tif.

form_batch_group_id : Over time, you might have multiple forms type you train against different models. The form_batch_group_id will allow you to specify all the form types that have been training using a specific model.

Azure Key Vault:

For security reasons, we don’t want to store certain sensitive information in the parametrization table in the Azure SQL database. We store those parameters in Azure Key Vault secrets.

Below are the parameters we store in the key vault:

CognitiveServiceEndpoint: The endpoint of the form recognizer cognitive service. This value will be stored in Azure Key Vault for security reasons.

CognitiveServiceSubscriptionKey: The access key of the cognitive service. This value will be stored in Azure Key Vault for security reasons. The below screenshot shows how to get the key and endpoint of the cognitive service

Figure 3: Cognitive Service Keys and Endpoint

StorageAccountName: The storage account where the training dataset and forms we want to extract the key value pairs from are stored. The two storage accounts can be different. The training dataset must be in the same container for all form types. They can be in different folders.

StorageAccountSasKey : the shared access signature of the storage account

The below screen shows the key vault after you create all the secrets

Figure 4 : Key Vault Secrets

Azure Data Factory:

To orchestrate the training and scoring of the model. Using a look up activity, we’ll retrieve the parameters in the Azure SQL Database and orchestrate the training and scoring of the model using Databricks notebooks. All the sensitive parameters stored in Key vault will be retrieve in the notebooks.

Azure Data Lake Gen 2:

To store the training dataset and the forms we want to extract the key-values pairs from. The training and the scoring datasets can be in different containers but, as mentioned above, the training dataset must be in the same container for all form types.

Azure Databricks:

To implement the python script to train and score the model. Note that we could have used Azure functions.

Azure Key Vault:

To store the sensitive parameters required by the Form Recognizer Rest API.

The code to implement this solution is available in the following GitHub repository.

Additional Resources

Get started with deploying Form Recognizer –

Custom Model – extract text, tables and key value pairs

QuickStart: Train a Form Recognizer model and extract form data by using the REST API
QuickStart: Train a Form Recognizer model with labels using the sample labeling tool
Form Recognizer Sample Labeling Tool

Try it out: https://fott.azurewebsites.net/
Open Source project: https://github.com/microsoft/OCR-Form-Tools

Prebuilt receipts - extract data from USA sales receipts

Quickstart: Extract receipt data using the REST API

Layout - extract text and table structure (row and column numbers) from your documents

Quickstart: Extract layout data using the REST API

See What’s New

Introducing Asynchronous APIs for Text Analytics and Text Analytics for Health

AshlyYeo — Tue, 24 Nov 2020 21:35:16 GMT

This post is co-authored with Sara Kandil

Today, we are announcing a preview of new asynchronous (batch) APIs for Text Analytics and Text Analytics for health, which enable developers to apply Natural Language Processing (NLP) to even more scenarios so they can identify key phrases, entities and even personally identifiable information (PII).

Asynchronous Analyze API for Text Analytics

Text Analytics is a generally available Azure Cognitive Service that lets you discover insights in text using Natural Language Processing (NLP). The service helps you identify key phrases and entities (people, place, organization, event, date among others), recognize text that contains personal information (PII) and analyze sentiment (positive, neutral, or negative).

To date, customers have been using Text Analytics by making synchronous calls to the service’s REST API, client library SDK, or by using containers to run Text Analytics in their own environment. Today, we are introducing a new preview Analyze operation for users to analyze larger documents asynchronously combining multiple Text Analytics features in one call. This gives customers the flexibility to analyze more information, at once, when their applications don’t need a synchronous response. The new asynchronous Analyze operation for Text Analytics supports individual documents of up to 125k characters, and up to 25 documents in a request.

The Analyze operation preview supports key phrase extraction, named entity recognition and PII recognition and is available in 5 Azure regions (West US 2, East US2, West Europe, North Europe, and Central US). Support for the rest of the Text Analytics capabilities and additional regions is coming soon.

Asynchronous Analyze API for Text Analytics for health

We are also introducing a new asynchronous hosted API for Text Analytics for health. As a refresher, early this year (July), we announced a preview of Text Analytics for health, a capability for the healthcare industry, trained to extract insights from medical data. With Text Analytics for health, users can:

Detect words and phrases mentioned in unstructured text as entities that are associated with semantic types in the healthcare and biomedical domain – such as diagnosis, medication name, symptom/sign, and more.
Link entities to medical ontologies and domain-specific coding systems (for example, the Unified Medical Language System), and extract meaningful connections between concepts mentioned in text (for example, finding the relationship between a medication name and the dosage associated with it.)
Detect negation of the different entities mentioned in the text.

Example of Text Analytics for health at work.

Previously, Text Analytics for health was only available for use via containers. This new API gives users the option to use the hosted service and avoid the heavy lifting of hosting containers unless they need to.

The hosted Text Analytics for health operation supports document sizes up to 5k characters and up to 10 documents in a single request. It is available for use in the West US 2, East US 2, Central US, North and West Europe regions.

In summary, Text Analytics is now more accessible with more ways to use the capabilities depending on your scenario. You can:

Call the synchronous endpoints to use the Text Analytics features.
Call the asynchronous Analyze API to process larger documents with multiple Text Analytics features in a single call.
Call the hosted asynchronous Text Analytics for health API if your dataset that is being analyzed has clinical and biomedical documents.
Use Text Analytics containers to host the endpoint in your own environments that meets your privacy and security requirements.

The new Text Analytics asynchronous APIs are available to use in Preview today. Please refer to our documentation to learn more and get started with these new APIs.

Neural Text-to-Speech previews five new languages with innovative models in the low-resource setting

Qinying Liao — Thu, 19 Nov 2020 16:30:01 GMT

This post is co-authored with Xianghao Tang, Lihui Wang, Jun-Wei Gan, Gang Wang, Garfield He, Xu Tan and Sheng Zhao

Neural Text-to-Speech (Neural TTS), part of Speech in Azure Cognitive Services, enables you to convert text to lifelike speech for more natural user interactions. Neural TTS has powered a wide range of scenarios, from audio content creation to natural-sounding voice assistants, for customers from all over the world. For example, the BBC, Progressive and Motorola Solutions are using Azure Neural TTS to develop conversational interfaces for their voice assistants in English speaking locales. Swisscom and Poste Italiane are adopting neural voices in French, German and Italian to interact with their customers in the European market. Hongdandan, a non-profit organization, is using neural voices in Chinese to make their online books audible for the blind people in China.

By September 2020, we extended Neural TTS to support 49 languages/locales with 68 voices. At the same time, we continue to receive customer requests for more voice choices and more language support globally.

Today, we are excited to announce that Azure Neural TTS has extended its global support to five new languages: Maltese, Lithuanian, Estonian, Irish and Latvian, in public preview. At the same time, Neural TTS Container is generally available for customers who want to deploy neural voice models on-prem for specific security requirements.

Neural TTS previews 5 new languages

Five new voices and languages are introduced to the Neural TTS portfolio. They are: Grace in Maltese (Malta), Ona in Lithuanian (Lithuania), Anu in Estonian (Estonia), Orla in Irish (Ireland) and Everita in Latvian (Latvia). These voices are available in public preview in three Azure regions: EastUS, SouthEastAsia and WestEurope.

Hear samples of these voices, or try them with your own text in our demo.

Locale	Language	Voice name	Audio sample
mt-MT	Maltese (Malta)	“mt-MT-GraceNeural”	Fid-diskors tiegħu, is-Segretarju Parlamentari fakkar li dan il-Gvern daħħal numru ta’ liġijiet u inizjattivi li jħarsu lill-annimali.
lt-LT	Lithuanian (Lithuania)	“lt-LT-OnaNeural”	Derinti motinystę ir kūrybą išmokau jau po pirmojo vaiko gimimo.
et-EE	Estonian (Estonia)	“et-EE-AnuNeural”	Pese voodipesu kord nädalas või vähemalt kord kahe nädala järel ning ära unusta pesta ka kardinaid.
ga-IE	Irish (Ireland)	“ga-IE-OrlaNeural”	Tá an scoil sa mbaile ar oscailt arís inniu.
lv-LV	Latvian (Latvia)	“lv-LV-EveritaNeural”	Daži tumšās šokolādes gabaliņi dienā ir gandrīz būtiska uztura sastāvdaļa.

With these updates, Azure TTS service now supports 54 languages/locales with 78 neural voices and 77 standard voices available.

Behind the scenes: 10X faster voice building with the low resource setting.

The creation of a TTS voice model normally requires a large volume of training data, especially for extending to a new language, where sophisticated language-specific engineering is required. In this section, we introduce “LR-UNI-TTS”, a new Neural TTS production pipeline to create TTS languages where training data is limited, i.e., ‘low-resourced’. With this innovation, we are able to improve the Neural TTS locale development with 10x agility and support the five new languages quickly.

High resource vs. low resource

Traditionally, it can easily take more than 10 months to extend TTS service to support a new language due to the extensive language-specific engineering required. This includes collecting tens of hours of language-specific training data, and creating hand-crafted components like text analysis etc.. In many cases, one major challenge for supporting a new language is that such large volume of data is unavailable or hard to find, causing a language ‘low-resourced’ for TTS model building. To handle the challenge, Microsoft researchers have proposed an innovative approach, called LRSpeech, to handle the extremely low-resourced TTS development. It has been proved that LRSpeech has the capability to build good quality TTS in the low-resource setting, using multilingual pre-training, knowledge distillation, and importantly the dual transformation between text-to-speech (TTS) and speech recognition (SR).

How LR-UNI-TTS works

Built on top of LRSpeech and the multi-lingual multi-speaker transformer TTS model (called UNI-TTS), we have designed the offline model training pipeline and the online inference pipeline for the low-resource TTS. Three key innovations contribute to the significant agility gains with this approach.

First, by leveraging the parallel speech data (the pairing speech audios and the transcript) collected during the speech recognition development, the LR-UNI-TTS training pipeline greatly reduces the data requirements for refining the base model in the new language. Previously, the high-quality multi-speaker parallel data has been critical in extending TTS to support a new language. The TTS speech data is more difficult to collect as it requires the data to be clean, the speaker carefully selected, and the recording process well controlled to ensure the high audio quality.

Second, by applying the cross-lingual speaker transfer technology with the UNI-TTS pipeline, we are able to leverage the existing high-quality data in a different language to produce a new voice in the target language. This saves the effort to find a new professional speaker for the new languages. Traditionally, the high-quality parallel speech data in the target language is required, which easily takes months for the voice design, voice talent selection, and recording.

Lastly, the LR-UNI-TTS approach uses characters instead of phonemes as the input feature to the models, while the high-resource TTS pipeline is usually composed of a multi-step text analysis module that turns text into phonemes, costing long time to build.

Below figure describes the offline training pipeline for the low-resource TTS voice model.

Figure 1. The offline training pipeline for the low-resource TTS voice model.

In specific, at the offline training stage, we have leveraged a few hundred hours of the speech recognition data to further refine the UNI-TTS model. It can help the base model to learn more prosody and pronunciation patterns for the new locales. The speech recognition data is usually collected in daily environments using PC or mobile devices, unlike the TTS data which is normally collected in the professional recording studios. Although the SR data can be much lower-quality than the TTS data, we have found LR-UNI-TTS can benefit from such data effectively.

With this approach, the high-quality parallel data in the new language which is usually required for the TTS voice training becomes optional. If such high-quality parallel data is available, it can be used as the target voice in the new language. If no high-quality parallel data is available, we can also choose a suitable speaker from an existing but different language and transfer it into the new language through the cross-lingual speaker transfer-learning capability of UNI-TTS.

In the below chart, we describe the flow of the runtime.

Figure 2: The online inference pipeline for the low-resource TTS voice model.

At the runtime, a lightweight text analysis is designed to preprocess the text input with sentence separation and text normalization. Compared to the text analysis component of the high-resource language pipelines, this module is greatly simplified. For instance, it does not include the pronunciation lexicon or letter-to-sound rules which are used in high-resource languages. The normalized text characters are generated by the lightweight text analysis component. During this process, we also leverage the text normalization rules from the speech recognition development, which saves the overall cost a lot.

The other components are similar to the high-resource language pipelines. For example, the neural acoustic model uses the FastSpeech model to convert the character input into mel-spectrogram.

Finally, the neural vocoder HiFiNet is used to convert the mel-spectrogram into audio output.

Overall, using LR-UNI-TTS, a TTS model in a new language can be built in about one month, which is 10x faster than the traditional approaches.

In the next section, we share the quality measurement results for the voices built with LR-UNI-TTS.

Quality assessments

Similar to other TTS voices, the quality of the low-resource voices created in the new languages are measured using the Mean Opinion Score (MOS) tests and intelligibility tests. MOS is a widely recognized scoring method for speech naturalness evaluation. With MOS studies, participants rate speech characteristics such as sound quality, pronunciation, speaking rate, and articulation on a 5-point scale, and an average score is calculated for the report. Intelligibility test is a method to measure how intelligible a TTS voice is. With intelligibility tests, judges are asked to listen to a set of TTS samples and mark out the unintelligible words to them. Intelligibility rate is calculated using the percentage of the correctly intelligible words among the total number of words tested (i.e., the number of intelligible words/the total number of words tested * 100%). Normally a usable TTS engine needs to reach a score of > 98% for intelligibility.

Below table summarizes the MOS score and the intelligibility score of the five new languages created using LR-UNI-TTS .

Locale	Language (Region)	Average MOS	Intelligibility
mt-MT	Maltese (Malta)	3.59*	98.40%
lt-LT	Lithuanian (Lithuania)	4.35	99.25%
et-EE	Estonian (Estonia)	4.52	98.73%
ga-IE	Irish (Ireland)	4.62	99.43%
lv-LV	Latvian (Latvia)	4.51	99.13%

* Note: MOS scores are subjective and not directly comparable across languages. The MOS of the mt-MT voice is relatively lower but reasonable in this case considering that the human recordings used as the training data for this voice also gots a lower MOS.

As shown in the table, the voices created with the low resources available are highly intelligible and have achieved high or reasonable MOS scores among the native speakers.

It’s worth pointing out that due to the nature of the lightweight text analysis module for the runtime, the phoneme-based SSML tuning capabilities are not supported for the low-resource voice models, for example, the ‘phoneme’ and the ‘lexicon’ elements.

Coming next: extending Neural TTS to even more locales

LR-UNI-TTS has paved the way for us to extend Neural TTS to more languages for the global users more quickly. Most excitingly, LR-UNI-TTS can potentially be applied to preserve the languages that are disappearing in the world today, as pointed out in the guiding principles of XYZ-code.

With the five new languages released in public preview, we welcome user feedback as we continue to improve the voice quality. We are also interested to partner with passionate people and organizations to create TTS for more languages. Contact us (mstts[at]microsoft.com) for more details.

What’s more: Neural TTS Container GA

Together with the preview of these five new languages, we are happy to share that the Neural TTS Container is now GA. With Neural TTS Container, developers can run speech synthesis with the most natural digital voices in their own environment for specific security and data governance requirements. Learn more about how to install Neural TTS Container and visit the Frequently Asked Questions on Azure Cognitive Services Containers.

Get started

With these updates, we’re excited to be powering natural and intuitive voice experiences for more customers, supporting more flexible deployment. Azure Text-to-Speech service provides more than 150 voices in over 50 languages for developers all over the world.

For more information:

Try the TTS demo
See our documentation
Check out our sample code

How to operationalize more than 100 AI models in as little as 12 weeks using Azure Databricks

ClintonWFord-Databricks — Tue, 17 Nov 2020 14:00:00 GMT

Organizations are leveraging artificial intelligence (AI) and machine learning (ML) to derive insight and value from their data and to improve the accuracy of forecasts and predictions. In rapidly changing environments, Azure Databricks enables organizations to spot new trends, respond to unexpected challenges and predict new opportunities. Data teams are using Delta Lake to accelerate ETL pipelines and MLflow to establish a consistent ML lifecycle.

Solving the complexity of ML frameworks, libraries and packages

Customers frequently struggle to manage all of the libraries and frameworks for machine learning on a single laptop or workstation. There are so many libraries and frameworks to keep in sync (H2O, PyTorch, scikit-learn, MLlib). In addition, you often need to bring in other Python packages, such as Pandas, Matplotlib, numpy and many others. Mixing and matching versions and dependencies between these libraries can be incredibly challenging.

Figure 1. Databricks Runtime for ML enables ready-to-use clusters with built-in ML Frameworks

With Azure Databricks, these frameworks and libraries are packaged so that you can select the versions you need as a single dropdown. We call this the Databricks Runtime. Within this runtime, we also have a specialized runtime for machine learning which we call the Databricks Runtime for Machine Learning (ML Runtime). All these packages are pre-configured and installed so you don’t have to worry about how to combine them all together. Azure Databricks updates these every 6-8 weeks, so you can simply choose a version and get started right away.

Establishing a consistent ML lifecycle with MLflow

The goal of machine learning is to optimize a metric such as forecast accuracy. Machine learning algorithms are run on training data to produce models. These models can be used to make predictions as new data arrive. The quality of each model depends on the input data and tuning parameters. Creating an accurate model is an iterative process of experiments with various libraries, algorithms, data sets and models. The MLflow open source project started about two years ago to manage each phase of the model management lifecycle, from input through hyperparameter tuning. MLflow recently joined the Linux Foundation. Community support has been tremendous, with 250 contributors, including large companies. In June, MLflow surpassed 2.5 million monthly downloads.

Diagram: MLflow unifies data scientists and data engineers

Ease of infrastructure management

Data scientists want to focus on their models, not infrastructure. You don’t have to manage dependencies and versions. It scales to meet your needs. As your data science team begins to process bigger data sets, you don’t have to do capacity planning or requisition/acquire more hardware. With Azure Databricks, it’s easy to onboard new team members and grant them access to the data, tools, frameworks, libraries and clusters they need.

Alignment Healthcare

Alignment Healthcare, a rapidly growing Medicare insurance provider, serves one of the most at-risk groups of the COVID-19 crisis—seniors. While many health plans rely on outdated information and siloed data systems, Alignment processes a wide variety and large volume of near real-time data into a unified architecture to build a revolutionary digital patient ID and comprehensive patient profile by leveraging Azure Databricks. This architecture powers more than 100 AI models designed to effectively manage the health of large populations, engage consumers, and identify vulnerable individuals needing personalized attention—with a goal of improving members’ well-being and saving lives.

Building your first machine learning model with Azure Databricks

To help you get a feel for Azure Databricks, follow the code samples and videos in this blog post to build a simple model using sample data in Azure Databricks. Learn how to by attending an Azure Databricks event, watch how you can Turbocharge your business with Machine Learning, leverage this free Azure Databricks ML training module on MS Learn and join us at our next Azure Databricks Office Hours.

November 2020 – Conversational AI update

GaryPrettyMsft — Mon, 16 Nov 2020 21:21:35 GMT

We are excited to announce the November release of the Bot Framework SDK and Composer, driving the Microsoft Conversational AI platform forward and building on the announcements we made in September at Microsoft Ignite. Our November update sees new updates to the Bot Framework SDK and Bot Framework Composer, adding new capabilities for developers and improving integration with our key partners, including Power Virtual Agents and HealthBot.

Bot Framework v4.11

Version 4.11 of the Bot Framework SDK, including new releases for .NET, JavaScript, Python and Java (preview 7), along with updates to our tooling, including the CLI.

Following our quality focused 4.10 release, we continue to push on this area, including improvements to the commonly used typing and transcript logging middleware behavior and associated error handling.

For developers building solutions for Microsoft Teams, new support for meetings has been added, including the Meeting Participant API and meeting specific notifications.

We continue to reduce developer friction for Skill development, adding the ability to test Skills locally, using the Bot Framework Emulator, without requiring an App Id and password. Additional scenarios, such as interruption support when calling a Skill and the ability to update or delete activities from a Skill have also been added.

Skills support has now also been added to HealthBot, a cloud platform for virtual health bots and assistants built on Bot Framework, with solutions now able consume, or for themselves to be consumed as a Bot Framework Skill.

We’re also undertaking significant investments in automated testing in this area, with the opportunity for you to review and provide feedback on the current specifications for Functional Testing and the Test Runner.

Further improvements to our documentation include expanding content across Adaptive Dialogs, Skills, overall architecture topics, as well as adding reference documentation for the Java SDK preview.

Bot Framework Composer v1.2

A new release of Composer (v1.2) is now available. This release deepens integration with Power Virtual Agents (PVA), part of the Power Platform, with a new public preview of PVA integration with Bot Framework Composer.

Users of the no-code PVA platform were already able to extend their solutions by consuming Bot Framework Skills. Now, PVA solutions can be opened in Bot Framework Composer, using a deep-link from the PVA portal, extending them with more sophisticated capabilities and enabling the collaboration between business users and developers on the same project.

Try the new Power Virtual Agents integration with Bot Framework Composer today!

When ready, Composer developers can publish directly from Composer, using a pre-configured publishing profile, back into the PVA portal, with new PVA Topics added using Composer then shown alongside existing Topics and immediately ready for testing.

An upcoming release of Composer, expected in December, will add improved provisioning and publishing support and enhanced QnA Maker knowledgebase integration.

As part of the December release, users will also have the option to enable new preview capabilities through the addition of feature flags. The first preview features planned include Orchestrator integration, the new intent detection and arbitration (dispatch) technology that runs locally within your bot, along with Form Dialogs, enabling the rapid generation of intelligent slot-filling dialogs, including complex capabilities such as slot disambiguation.

Nightly builds of Composer are available (enabled via the Composer settings page) which allow you to try the latest updates as soon as they are available.

Introducing QnA Maker managed: now in public preview

nerajput — Thu, 19 Nov 2020 05:55:03 GMT

QnA Maker is an Azure Cognitive Service that allows you to create a conversational layer over your data- in minutes. Today, we are announcing a new version of QnA Maker which advances several core capabilities like better relevance and precise answering, by introducing state-of-art deep learning technologies.

Illustrative representation of QnA Maker functionality.

Overview of new QnA Maker managed capabilities

Summary of new features introduced:

Deep learnt ranker with enhanced relevance of results across all supported languages.
Precise phrase/short answer extraction from answer passages.
Simplified resource management by reducing the number of resources deployed.
E2E region support for Authoring + Prediction.

Detailed description of the new features is further down in this article. Learn how to migrate to the new QnA Maker managed (Preview) knowledge base here.

QnA Maker managed (Preview) Architecture.

As per the architecture of QnA Maker managed (Preview), there will be only two resources: QnA Maker service for authoring and computation and Azure Cognitive Search for storage and L1 ranking. This has been done with an aim of simplifying the resource creation and management process. Now, customers need to manage only 2 resources instead of 5 different resources.
QnA Maker managed (Preview) also allows the user to do language setting specific to Knowledge Base.
Computation has been moved out of the user subscription, so there is no dependency on the customers for scaling and availability management. This allowed us to use SOTA deep learnt model for L2 ranker which enhances the L2 ranker horizontally across all the languages, so now we support all the 50+ languages with better and enhanced precision.
QnA Maker service will be available in multiple regions to give customers’ the flexibility to keep their end-to-end service in one region.
For inference logs and telemetry, the latest version will be using Azure Monitoring instead of App insights. To keep the experience seamless and easy to adopt all the APIs has been kept backward compatible. There is almost zero change in the management portal experience.

New features of QnA Maker managed (Preview)

This section talks about all the distinguishing features of QnA maker managed in detail.

Simplified Create Blade

Onboarding on QnA Maker managed (Preview), and resource creation has been kept quite simple. Now, you will see a checkbox with Managed, as shown below. As soon as you select the checkbox, the form will be updated with the required resources.

Precise Answering

Machine Reading Comprehension based answer span detection feature is most beneficial for the scenarios where the customers have big passages present as answer in their Knowledge Base. Currently, they put good amount of manual efforts in curating small/precise answers and ingest them in the Knowledge base.

The new features give them flexibility to either choose the precise answer or the answer passage, customers can take this decision based on the confidence score of the precise short answer and answer passage. Here are some examples to show how short answers can be useful:

Deep Learnt ranker

The new L2 ranker is based on Turing multilingual language model (T-ULRv2), a deep learning-based transformer model, which improves the precision of the service for all the languages. For any user query, the new L2 ranker model understands the semantics of the user query better and gives better aligned results. This model is not language specific and is targeted to improve the overall precision of all languages horizontally. Here are some examples to analyze the difference between the results of current service and QnA Maker managed (Preview) service:

Query

Current GA results

QnA Maker managed (Preview) results

Improvements in Preview

can someone ring me

I can tell you all about Wi-Fi calling, including the devices that support Wi-Fi calling and where you can get more information yourself. Feel free to ask me a question and I'll do what I can to answer it

Yes, you can make and receive calls using Wi-Fi calling. Pretty nifty, right?

The new L2 ranker can understand the relevance between “ring me” and “make and receive calls” and is returning more relevant result unlike the current GA, which has returned a generic answer.

can’t connect to mobile data

You'll be connected to Wi-Fi, so it'll only use your minutes and text allowances.

If you don't have mobile signal, it's no problem. With Three inTouch Wi-Fi Calling, you can call and text whenever you're on Wi-Fi in the UK, even without mobile signal.

The new L2 ranker is again able to understand the query better as its able to understand that mobile data is somewhere connected to mobile signals and hence giving better results based on the data present in the Knowledge Base than the current GA model.

E2E region support

With QnA Maker managed (Preview) our management service is no more limited to west-US region. We are offering end to end region support for:

South Central US
North Europe
Australia East.

Other hero regions will be added when we go GA.

Knowledge Base specific language setting

Now, customers can create Knowledge bases with different language setting within a service. This feature is beneficial for users who have multi-language scenarios and need to power the service for more than one language. In this case, there will be a test index specific to every Knowledge Base, so that the customer can verify how the service is performing specific to every language.

You can configure this setting only with the first Knowledge base of the service, once set the user will not be allowed to update the setting.

Pricing

Public preview of QnA Maker managed (Preview) will be free in all the regions (You only pay for the Azure Cognitive Search SKU). The standard pricing will be applicable when the service goes to GA by mid-2021.

References

Azure speaks your language: the 3 immediate benefits for your organization

dturchyn — Wed, 04 Nov 2020 15:48:28 GMT

The last several years brought exciting innovations in the field of Artificial Intelligence, especially when it comes to advancements in speech and language processing. Processing speech and making text and audio information searchable enables a diverse set of innovative applications, including helping researchers in searching for related papers, or building information graphs for predicting the best new drug candidates, or uncovering issues with products and services in near real time. For region like Central and Eastern Europe, which includes 30+ countries, most speaking their own language, support for local languages is a critical condition for implementing innovation. That’s why the recent (September 2020) Azure Speech services update has opened a whole new area of opportunity for our region.

With updated language support, most of the EU languages are now supported in Azure Speech services. For region which I am covering in my current role, it means that we now have support for all of our CEE EU languages (Polish, Bulgarian, Czech, Greek, Croatian, Hungarian, Romanian, Slovak, Slovenian, Estonian, Lithuanian, Latvian, Maltese) and Russian in Azure Speech and Translator services. Additionally, our speech generation models have also been updated, now leveraging the Neural TTS - a powerful speech synthesis capability, which enables to convert text to lifelike speech which is close to human-parity. Below you will find 3 benefits, how this might help you advance your products and services today.

Automatic generation of medical summary from spoken conversations between doctors and patients

First, analyzing speech data or generating speech enables you to extract insights from audio or video information, which otherwise would be unreachable for analytical systems. This might include data like customer support conversations or employee speech in videos or transcribing speech for field employees or doctors. Communicating with your customers with natural-sounding generated speech in your own language is another area of innovation, which enables scenarios from voice announcements to supporting people with visual impairments to building voice assistants. Is information the new currency? If you answer “yes” to this — why then would you have terabytes of currency sitting without you getting use of it? Now you can turn it into tangible cash-flow.

Azure Speech services are a sub-set of pre-built (but customizable) APIs for working with Speech. This includes transcribing spoken language into text for further analysis (Speech-to-Text) and generating naturally sounding speech form text input (Text-to-Speech). Azure Translator is another piece in the puzzle, which has also received major update for the languages, now translating text between 70+ languages.

Second, there are new scenarios enabled now by these pre-built AI models. Do you have that innovative idea for analysing customer conversations or augmenting your service with spoken messages in your local language? Often, these ideas were not realized due to the associated challenges like finding the right skilled people within your organization and investing into a project with unknown development cycle and returns. Now it is possible to build a realistic prototype app quickly to extract insights from your speech data, by calling the service through the API — in days, if not hours.

Analysing customer support conversations brings insights from priceless data, which is untapped without applying Speech processing

Third, this is one of those cloud services, which may work without sending your data to the cloud! Many of Azure Cognitive Services today may be deployed right within your own data center as containers. This means, that none of the actual data will be sent to the cloud, as even processing will happen locally. In this case, only billing information will be exchanged with Azure.

Interested enough to give it a try? If you are interested in learning more, you may request detailed information or virtual session on Azure Cognitive Services from our sales representatives (please specify whether you are looking for the session on Azure Cognitive services, or details of your specific projects where Speech services may be used). To read more or test Azure Speech services capabilities in your language, please refer to our Azure Speech Services Documentation.

Looking forward to the exciting results you will achieve in your business with the updated Azure Speech Services!

Azure Neural TTS upgraded with HiFiNet, achieving higher audio fidelity and faster synthesis speed

Qinying Liao — Thu, 19 Nov 2020 16:41:18 GMT

This post was co-authored with Jinzhu Li and Sheng Zhao

Neural Text to Speech (Neural TTS), a powerful speech synthesis capability of Cognitive Services on Azure, enables you to convert text to lifelike speech which is close to human-parity. Since its launch, we have seen it widely adopted in a variety of scenarios by many Azure customers, from voice assistants like the customer service bot like BBC and Poste Italiane, to audio content creation scenarios like Duolingo.

Voice quality, which includes the accuracy of pronunciation, the naturalness of prosody such as intonation and stress patterns, and the fidelity of audio, is the key reason that customers are migrating from the traditional TTS voices to neural voices. Today we are glad to share that we have upgraded our Neural TTS voices with a new-generation vocoder, called HiFiNet, which results much higher audio fidelity while significantly improving the synthesis speed. This is particularly beneficial to customers whose scenario relies on hi-fi audios or long interactions, including video dubbing, audio books, or online education materials.

What’s new?

Our recent updates on Azure Neural TTS voices include a major upgrading of the vocoder. The voice fidelity has been improved significantly and audio quality defects such as glitches and small noises are largely reduced. Our tests show that this new vocoder generates audios without hearable quality loss from the recordings of training data (more details are introduced later). In addition, it can synthesize speech much faster than our previous version of the product. All these benefits are achieved through a new-generation neural vocoder, called HiFiNet.

What is a vocoder and why does it matter?

Vocoder is a major component in speech synthesis, or text-to-speech. It turns an intermediate form of the audio, which is called acoustic feature, into audible waveform. Neural vocoder is a specific vocoder design which uses deep learning networks and is a critical module of Neural TTS.

Microsoft Azure Neural TTS consists of three major components in the engine: Text Analyzer, Neural Acoustic Model, and Neural Vocoder. To generate natural synthetic speech from text, first, text is input into Text Analyzer, which provides output in the form of phoneme sequence. A phoneme is a basic unit of sound that distinguishes one word from another in a particular language. Sequence of phonemes defines the pronunciations of the words provided in the text. Then the phoneme sequence goes into the Neural Acoustic Model to predict acoustic features, which defines speech signals, such as speaking style, speed, intonations, and stress patterns, etc. Finally, the Neural Vocoder converts the acoustic features into audible waves so the synthetic speech is generated.

The vocoder is critical to the final audio quality. In specific, it directly impacts the fidelity of a wave, including clearness, timbre, etc. Let’s hear the difference of the audio quality with samples generated using different neural vocoders based on the same acoustic features (recommended to listen with a high-quality headset).

Vocoder versions	2018 vocoder for real-time synthesis	2019 vocoder for real-time synthesis	2020 vocoder for real-time synthesis (HiFiNet)
“Top cinematographers weigh in on filmmaking in the age of streaming.”

With each vocoder update, the speech generated sounds clearer, voice less muffled and noises reduced. In the next section, we introduce how a HiFiNet vocoder is trained during the creation of a neural voice model.

How does HiFiNet work?

In Azure TTS system, neural voice models are trained using human voice recordings as training data with deep learning networks. As part of the training, a vocoder is built with the goal to generate high quality audio output close to the original recordings from the training data. In the meantime, it needs to run fast enough to produce at least 24,000 samples per seconds, i.e. with a sampling rate of 24khz, which is the default sampling rate of Azure Neural TTS voice models.

Leveraging the state-of-art research on vocoders, we designed the training pipeline for HiFiNet, the new-generation Neural TTS vocoder, and applied it to create neural voice models in Azure Neural TTS. This pipeline is built with one simple goal: produce machine-generated audio waves (synthetic speech) that is indistinguishable from its original waves (human recordings) in a high speed.

Below chart describes how the HiFiNet training pipeline works. With this pipeline, two key networks are trained: A Generator which is used to create audio (‘Generated Wave’), and a Discriminator which is used to identify the gap of the created audio from its training data (‘Real Wave’). The goal of the training is to make the Generator generate waves that the Discriminator can’t distinguish from the original real recordings.

Training pipeline of the HiFiNet Vocoder

First, the training pipeline uses the original human recording as input and extract the acoustic features. Then, the acoustic features are fed into the Generator module which generates waves, so we get two sets of waves: the original recordings as real waves, and the generated waves as fake waves. Next, the two sets of waves are fed into the Discriminator network to distinguish which are the real waves and which are the generated fake waves. This output from the Discriminator is used as feedback to help the Generator and Discriminator to learn better. As this training loop continues, the Generator becomes smarter to create indistinguishable fake waves, while the Discriminator gets smarter in making the right judgements. Finally, when the training reaches a point where Discriminator can’t distinguish the waves generated by the Generator from real waves, the vocoder is successfully trained. This vocoder is capable of producing audio outputs without noticeable quality loss compared to the original human recordings.

In the next section we describe the performance of HiFiNet vocoder.

What are the benefits?

HiFiNet significantly improves audio quality.

To understand the benefit of HiFiNet, we conducted a number of tests in many aspects which yielded positive results. Our tests show that the HiFiNet vocoder significantly improves the audio quality of the Neural TTS voice output, compared to our previous version of the product.

CMOS (Comparative Mean Opinion Score) is a well accepted method in the speech industry for comparing the voice quality of two TTS systems. A CMOS test is similar to an A/B testing, where participants listen to different pairs of audio samples generated by two systems and provide their subjective opinions on how A compares to B. Normally in one test, we recruit 30-60 anonymous testers with qualified language expertise to evaluate around 50 pairs of audio samples side by side. The result is reported as CMOS gap, which measures the average of the difference in the opinion score between the two systems. In the cases where the absolute value of a CMOS gap is <0.1, we claim system A and B are on par. When the absolute value of a CMOS gap is >=0.1, then one system is reported better than the other. If the absolute value of a CMOS gap is >=0.2, we say one system is significantly better than the other.

We have done hundreds of CMOS tests of HiFiNet compared to our last version vocoder, on 68 neural voices across 49 languages/locales. Our results show that HiFiNet is notably better than the previous production vocoder in Azure Neural TTS.

In general, the audio quality, especially the fidelity is obviously improved. On average, across all languages, the HiFiNet vocoder achieves a CMOS gain higher than 0.2 compared to the previous vocoder, which means the improvement is hearable for users.

In particular, HiFiNet also has better robustness than the previous version of vocoder. Audio defects are largely reduced in the generated waves with HiFiNet. Our tests show that with the previous production vocoder, in 100 test samples, our testers can hear about 10 defects like beep, click sound, fidelity loss. Although most of them are not obvious, it can still be annoying if it keeps happening in a long audio or multi-round voice interactions. Now, these defects are no longer reported with the HiFiNet audios, under the same test procedure with the same test sets.

With these advantages, we have updated the Neural TTS voices on Azure Cognitive Services with the new vocoder. Listen to the samples below to hear the difference. Or test the new voices using your own text with our online demo.

Language	Previous vocoder	HiFiNet	HiFiNet CMOS gain
English (US)			+0.122 (Better)
German			+0.193 (Better)
Chinese (Mandarin, Simplified)			+0.348 (Obviously Better)
Japanese			+0.465 (Obviously Better)

HiFiNet reaches human-parity audio fidelity.

In addition, we have conducted tests to compare the human recording audio quality and the computer-generated audio quality with HiFiNet. To make the comparison more accurate and more focused on the vocoder itself, we use the acoustic features extracted directly from human recordings instead of the TTS-predicted acoustic features so the acoustic differences are controlled and only the vocoder is evaluated in CMOS tests. Participants are asked to give their scores for different pairs of the generated waves and human recordings. Our result shows the CMOS gap of the audios produced by HiFiNet compared to human recordings is -0.05, which means the difference is hardly hearable and the audio quality is on par.

Hear how close the HiFiNet audio fidelity is to the human recordings with the samples below.

Language	Human recording	HiFiNet	HiFiNet CMOS gap
English (US)			+0.045 (on par)
Chinese (Mandarin, Simplified)			-0.054 (on par)

HiFiNet generates audios faster.

Real Time Factor (RTF) is used to measure the performance of vocoder. It is calculated as the time duration needed to generate the audio divided by the audio duration.

HiFiNet is a parallel vocoder so it can generate multiple samples at the same time. Here are some measurements of HiFiNet performance on both GPU and CPU devices.

With the output of 24khz sampling rate, on M60 GPU, through carefully optimized CUDA implementation, the vocoder RTF is around 0.01, which means the HiFiNet system can generate an audio 10 second-long in 0.1 second. This speed is almost 3x of our previous production vocoder.

On CPU machines, thanks to the highly-optimized ONNX runtime, the vocoder RTF is around 0.02 for 24khz sampling rate output.

With the performance improvement of HiFiNet, the end-to-end synthesis speed is about 2X as fast as our previous Neural TTS engine, which the audio quality is significantly improved at the same time.

What to expect next

Currently we support up to 24khz sampling rate on Azure Neural TTS service with 68 neural voice models available. In some highly sophisticated scenarios like audio dubbing, higher fidelity output like 48khz sampling rate makes a world of difference.

Below snippet from an audio spectrum shows the difference between 48hz sampling rate and 24khz. Audios with 48khz sampling rate get a higher frequency responding range which keeps more sophisticated details and nuances of the sound. Such high sampling rate creates challenges on both voice quality and inference speed.

24khz vs. 48khz: different frequency range

In our exploration, HiFiNet can handle both challenges well. According to our experiments, HiFiNet vocoder on 48khz sampling rate can be trained to achieve even higher quality with reasonable inference speed.

Hear the difference of the audio fidelity between the TTS output in 24khz and 48khs sampling rate, with a hi-fi speaker or headset.

Language	24khz HiFiNet	48khz HiFiNet
English (UK)
English (US)

The 48khz vocoder is now in private preview and can be applied to custom voices. Contact mstts [at] microsoft.com for details.

Create a custom voice with HiFiNet

The HiFiNet vocoder is also available in the Custom Neural Voice capability, enabling organizations to create a unique brand voice in multiple languages for their unique scenarios.

Learn more about the process for getting started with Custom Neural Voice.

Get started

With these updates, we’re excited to be powering more natural and intuitive voice experiences for global customers. Text to Speech has more than 70 standard voices in over 40 languages and locales in addition to our growing list of Neural TTS voices.

For more information:

Try the TTS demo
See our documentation
Check out our sample code

Apps can now narrate what they see in the world as well as people do

boxinli — Thu, 15 Oct 2020 00:21:12 GMT

How would you leverage technology capable of generating natural language image descriptions that are, in many cases, just as good or better than what a human could produce? What if that capability is just one cloud API call away? Would you create live scene captions for people who are blind or low vision to better understand the world around them, like Seeing AI?

With Azure Cognitive Services, you can now take advantage of state-of-the-art image captioning that has achieved human parity on captioning benchmarks thanks to advancements in the underlying AI model. Below are some examples showing how the improved model is more accurate than the old one:

Improved model: A trolley on a city street

Old model: a view of a city street

Improved model: A person using a microscope

Old model: A person sitting at a table using a laptop

Now, let us take a closer look at the technology and how to easily harness its power for your users.

Behind the Scenes of the Technology

The novel object captioning at scale (nocaps) challenge evaluates AI models on their abilities to generate image captions describing new objects that are not present in their training data. Microsoft’s Azure AI team pioneered the Visual Vocabulary (VIVO) pre-training technique that led to the industry first of surpassing human performance on the (nocaps) benchmark. Before we learn more about this innovation, we should understand Vision and Language Pre-training (VLP) first. It is a cross-modality (across vision and language) learning technique that uses large-scale image/sentence data pairs to train machine learning models capable of generating natural language captions for images. However, because visual concepts are learned from image/sentence pairs which are costly to obtain, it is difficult to train a broadly useful model with wide visual concept coverage. This is where VIVO pre-training comes in. It improves and extends VLP to allow rich visual concepts to be learned from easier to obtain image/word pairs (instead of sentence) to build a large-scale visual vocabulary. While natural language sentence generation is still trained with limited visual concepts, the resulting image caption is cleverly enriched by new objects from the large-scale visual vocabulary.

Figure 1: VIVO pre-training uses paired image-tag data to learn a rich visual vocabulary where image region features and tags of the same object are aligned. Fine-tuning is conducted on paired image-sentence data that only cover a limited number of objects (in blue). During inference, our model can generalize to describe novel objects (in yellow) that are learnt during VIVO pre-training.

Please see this MSR blog post to learn more about VIVO pre-training.

Try the Service in Your App

Imagine you would like to generate alternative text descriptions for images your users upload to your app. Azure Computer Vision Service with its much improved “describe image” (image captioning) capability can help. Let us take it for a spin.

We will be using Python client library to invoke the service in this blog post. Try these links if you prefer a different language or invoking the REST API directly.

Prerequisites

Python
An Azure subscription - create one for free
Once you have your Azure subscription, create a Computer Vision resource:
- Subscription: Pick the subscription you would like to use. If you just created a new Azure subscription, it should be an option in the dropdown menu.
- Resource group: Pick an existing one or create a new one.
- Region: Pick the region you would like your resource to be in.
- Name: Give your resource a unique name.
- Pricing tier: You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.
- Then click on “Review + create” to review your choices and click on “Create” again to deploy the resource

Once your resource is deployed, click “Go to resource.”

Click on “Keys and Endpoint” to get your subscription key and endpoint. You will be needing these for the code sample below.

Install the client

You can install the client library with:

pip install --upgrade azure-cognitiveservices-vision-computervision

Create and run the sample

Copy the following code into a text editor.
Optionally, replace the value of remote_image_url with the URL of a different image for which to generate caption.
Also, optionally, set useRemoteImage to FALSE and set local_image_path to the path of a local image for which to generate caption.
Save the code as a file with an .py extension. For example, describe-image.py.
Open a command prompt window.
At the prompt, use the python command to run the sample. For example, python describe-image.py.

import sys from azure.cognitiveservices.vision.computervision import ComputerVisionClient from msrest.authentication import CognitiveServicesCredentials # Best practice is to read this key from secure storage, # for this example we'll embed it in the code. subscription_key = "<your subscription key here>" endpoint = "<your endpoint here>" # Create the computer vision client computervision_client = ComputerVisionClient( endpoint, CognitiveServicesCredentials(subscription_key)) # Set to False if you want to use local image instead useRemoteImage = True if (useRemoteImage): # Get caption for a remote image, change to your own image URL as appropriate remote_image_url = "https://raw.githubusercontent.com/Azure-Samples/cognitive-services-sample-data-files/master/ComputerVision/Images/house.jpg" description_results = computervision_client.describe_image( remote_image_url) else: # Get caption for a local image, change to your own local image path as appropriate local_image_path = "<replace with local image path>" with open(local_image_path, "rb") as image: description_results = computervision_client.describe_image_in_stream( image) # Get the first caption (description) from the response if (len(description_results.captions) == 0): image_caption = "No description detected." else: image_caption = description_results.captions[0].text print("Description of image:", image_caption)

What’s that? Microsoft AI system describes images as well as people do

Learn more about other Computer Vision capabilities.

Real-time Inference on NVIDIA GPUs in Azure Machine Learning (Preview)

gopalv — Mon, 05 Oct 2020 16:00:00 GMT

AI today is about scale: models with billions of parameters used by millions of people. Azure Machine Learning is built to support your delivery of AI-powered experiences at scale. With our notebook-based authoring experience, our low-code and no-code training platform, our responsible AI integrations, and our industry-leading ML Ops capabilities, we give you the ability to develop large machine learning models easily, responsibly, and reliably.

One key component of employing AI in your business is model serving. Once you have trained a model and assessed it per responsible machine learning principles, you need to quickly process requests for predictions, for many users at a time. While serving models on general-purpose CPUs can work well for less complex models serving fewer users, those of you with a significant reliance on real-time AI predictions have been asking us how you can leverage GPUs to scale more effectively.

That is why today, we are partnering with NVIDIA to announce the availability of the Triton Inference Server in Azure Machine Learning to deliver cost-effective, turnkey GPU inferencing.

There are three components to serving an AI model at scale: server, runtime, and hardware. This new Triton server, together with ONNX Runtime and NVIDIA GPUs on Azure, complements Azure Machine Learning’s support for developing AI models at scale by giving you the ability to serve AI models to many users cheaply and with low latency. Below, we go into detail about each of the three components to serving AI models at scale.

Server

Triton Inference Server in Azure Machine Learning can, through server-side mini batching, achieve significantly higher throughput than can a general-purpose Python server like Flask.

Triton can support models in ONNX, PyTorch, TensorFlow, and Caffe2, giving your data scientists the freedom to explore any framework of interest to them during training time.

Runtime

For even better performance, serve your models in ONNX Runtime, a high-performance runtime for both training (in preview) and inferencing.

Numbers Courtesy of NVIDIA

ONNX Runtime is used by default when serving ONNX models in Triton, and you can convert PyTorch, TensorFlow, and Scikit-learn models to ONNX.

Hardware

NVIDIA Tesla T4 GPUs in Azure provide a hardware-accelerated foundation for a wide variety of models and inferencing performance demands. The NC T4 v3 series is a new, lightweight GPU-accelerated VM, offering a cost-effective option for customers performing real-time or small batch inferencing who may not need the throughput afforded by larger GPU sizes such as the V100-powered ND v2 and NC v3-series VMs, and desire a wider regional deployment footprint.

The new NCasT4_v3 VMs are currently available for preview in the West US 2 region, with 1 to 4 NVIDIA Tesla T4 GPUs per VM, and will soon expand in availability with over a dozen planned regions across North America, Europe and Asia.

To learn more about NCasT4_v3-series virtual machines, visit the NCasT4_v3-series documentation.

Easy to Use

Using Triton Inference Server with ONNX Runtime in Azure Machine Learning is simple. Assuming you have a Triton Model Repository with a parent directory triton and an Azure Machine Learning deploymentconfig.json, run the commands below to register your model and deploy a webservice.

az ml model register -n triton_model -p triton --model-framework=Multi az ml model deploy -n triton-webservice -m triton_model:1 --dc deploymentconfig.json --compute-target aks-gpu

Next Steps

In this blog, you have seen how Azure Machine Learning can enable your business to serve large AI models to many users simultaneously. By bringing together a high-performance inference server, a high-performance runtime, and high-performance hardware, we give you the ability to serve many requests per second at millisecond latencies while saving money.

To try this new offering yourself:

Sign up for an Azure Machine Learning trial
Clone our samples repository on GitHub
Read our documentation
Be sure to let us know what you think

You can also request access to the new NCasT4_v3 VM series (In Preview) by applying here.

Azure Machine Learning's native support for MLflow

shivani-patel — Fri, 02 Oct 2020 16:31:23 GMT

Azure Machine Learning service expands support for MLflow (Public Preview)

Background

Many data scientists start their machine learning projects using Jupyter notebooks or editors like Visual Studio Code. To ensure models can be used in production, it is essential to systematically track all aspects of an ML workflow, such as the data, environment, code, and models produced. These challenges with reproducibility can become complex when working in a hybrid cloud environment – but are mitigated if both environments conform to open standards.

AzureML’s support for MLflow

Azure ML now supports managing the end to end machine learning lifecycle using open MLflow standards, enabling existing workloads to seamlessly move from local execution to the intelligent cloud & edge. Azure Machine Learning has expanded support for running machine learning workflows to train, register and deploy models via native integration (API compatibility) with MLflow.

Let’s walk through some of the latest enhancements to the Azure ML and MLflow interoperability.

MLflow Projects

MLflow Projects provide a way to organize and describe your code to enable other data scientists or automated tools to run it. Any local directory or Git repository can be treated as an MLflow project. You can enable MLflow's tracking URI and logging API, collectively known as MLflow Tracking, to connect your MLflow experiments and Azure Machine Learning. You can submit your MLflow experiments locally or remotely using MLflow Projects with full tracking support in AzureML by setting the project backend to “azureml”.

A project includes the following:

Conda environment specification (conda.yaml)
Any .py or .sh file in the project can be an entry point, with no parameters explicitly declared. When you run the command with a set of parameters, MLflow passes each parameter on the command line using --key <value> syntax.
You specify more options by adding an MLproject file, which is a text file in YAML syntax. An example MLproject file looks like this:

name: tutorial conda_env: conda.yaml entry_points: main: parameters: alpha: float l1_ratio: {type: float, default: 0.1} command: "python train.py {alpha} {l1_ratio}"

Here’s an example set up for a local run. I’ve set the backend to “azureml” to get all the tracking support and error logging from Azure ML. The backend config object is used to store necessary information such as the compute target, local managed environment or a system managed environment.

local_env_run = mlflow.projects.run(uri=".", parameters={"alpha":0.3}, backend = "azureml", use_conda=False, backend_config = {"USE_CONDA": False})

In the image below you can see that Azure ML automatically tags the run with MLflow related metadata for visibility and logs the git info.

You can then log and visualize your run metrics in Azure Machine Learning Studio or the MLflow Experimentation UI.

You can see the same metrics in the Azure ML studio and MLflow UI.

MLflow Model Registry and Deployment

With the new support for the MLflow model format, it becomes even easier to track and deploy models on Azure ML. You can register models from local files or a run and use it to make predictions online or in batch mode. By deploying models as a web service, you can apply the Azure Machine Learning monitoring and data drift detection functionalities to your production models. Let's look at an example:

From the MLflow project run, you can see the output model from the projects run is registered following the MLflow model schema.

The MLmodel file contains all the model details and metadata.

If you want to register, containerize, and deploy the model, you can now do that in one step. Using the mlflow.azureml.deploy() Python SDK method, AzureML will register the model in AzureML, build the docker container and deploy it to the chosen target. The deployed service will also retain the MLflow metadata as tags as show in the image below.

With the continuous support for MLflow, Azure ML is committed to being interoperable with Open source standards providing flexibility for users to work on-prem or on the cloud. To get more details about the Mlflow and Azure ML integration check out the following links:

How to use MLflow with Azure Machine Learning

MLflow and Azure Machine Learning notebook examples

Framework Specific notebooks

Simplify and accelerate AI for the entire data science team with Azure Machine Learning designer

Lu_Zhang — Wed, 30 Sep 2020 12:33:44 GMT

At Microsoft Ignite, we announced the general availability of Azure Machine Learning designer, the drag-and-drop workflow capability in Azure Machine Learning studio which simplifies and accelerates the process of building, testing, and deploying machine learning models for the entire data science team, from beginners to professionals. We launched the preview in November 2019, and we have been excited with the strong customer interest. We listened to our customers and appreciated all the feedback. Your responses helped us reach this milestone. Thank you.

“By using Azure Machine Learning designer we were able to quickly release a valuable tool built on machine learning insights, that predicted occupancy in trains, promoting social distancing in the fight against Covid-19.” - Steffen Pedersen, Head of AI and advanced analytics, DSB (Danish State Railways)

Artificial intelligence (AI) is gaining momentum in all industries. Enterprises today are adopting AI at a rapid pace with different skill sets of people, from business analysts, developers, data scientists to machine learning engineers. The drag-and-drop experience in Azure Machine Learning designer can help your entire data science team to speed up machine learning model building and deployment. Specially, it is tailored for:

Data scientists who are more familiar with visual tools than coding.
Users who are new to machine learning and want to learn it in an intuitive way.
Machine learning experts who are interested in rapid prototyping.
Machine learning engineers who need a visual workflow to manage model training and deployment.

Connect and prepare data with ease

Azure Machine Learning designer is fully integrated with Azure Machine Learning dataset service for the benefit of versioning, tracking and data monitoring. You can import data by dragging and dropping a registered dataset from the asset library, or connecting to various data sources including HTTP URL, Azure blob, Azure Data Lake, Azure SQL or upload from a local file with Import Data module . You can use right click to preview and visualize the data profile, and preprocess data using a rich set of built-in modules for data transformation and feature engineering.

Build and train models with no-code/low-code

In Azure Machine Learning designer, you can build and train machine learning models with state-of-the art machine learning and deep learning algorithms, including those for traditional machine learning, computer vision, text analytics, recommendation and anomaly detection. You can also use customized Python and R code to build your own models. Each module can be configured to run on different Azure Machine Learning compute clusters so data scientists don’t need to worry about the scaling limitation and can focus on their training work.

Validate and evaluate model performance

You can evaluate and compare your trained model performance with a few clicks using the built-in evaluate model modules, or use execute Python/R script modules to log the customized metrics/images. All metrics are stored in run history and can be compared among different runs in the studio UI.

Root cause analysis with immersed debugging experience

While interactively running machine learning pipelines, you can always perform quick root cause analysis using the graph search and navigation to quickly nailed down to the failed step, preview logs and outputs for debugging and troubleshooting without losing context of the pipeline, and find snapshots to trace scripts and dependencies used to run the pipeline.

Deploy models and publish endpoints with a few clicks

Data scientists and machine learning engineers can deploy models for real-time and batch inferencing as versioned REST endpoints to their own environment. You don’t need to worry about the deep knowledge of coding, model management, container services, etc., as scoring files and the deployment image are automatically generated with a few clicks. Models and other assets can also be registered in the central registry for MLOps tracking, lineage, and automation.

Get started today

Get started today with your new Azure free trial, and learn more about Azure Machine Learning designer.

Leveling-up Local Experiment Runs with the VS Code AML Extension

Sid_Unnithan — Tue, 29 Sep 2020 22:57:18 GMT

Hey AzML community! The VS Code team is excited to announce version 0.6.15 of the AzML extension, with a brand new way for you to validate your scripts, environments, and datasets before submitting to a remote cluster.

If you'd like to follow along with the blog post and try out the new features, you can install the extension here!

Gaining confidence in your experiment runs

Experiencing a sense of worry or anxiety when submitting a remote experiment is common and expected. It's hard to predict how the training script you've been working very hard on is going to behave once it runs on your remote target. Many of you have expressed pain in not:

Knowing whether the environment you want to use will correctly work with your training script.
Knowing whether your datasets are materialized and being referenced correctly.
Having the confidence to submit your remote experiment and context-switch to another project you're working on.

The VS Code AzML extension team has been working hard over the past few weeks to bring a new capability to alleviate your pains: running a local experiment with an interactive debugging session.

Interactive Debugging with the AML Extension

You might be asking yourself - how is this different from my running my training script in VS Code? Here are some key differences:

The AzML service always uses an environment when submitting a remote run. These environments are materialized as Docker containers. When running a local experiment, the AzML extension will build the same Docker image and same Docker container that's used when running remotely.
Running a Python script normally assumes that you've taken care of data materialization and access. When running remotely, you're recommended to use AzML Datasets giving you the advantage of working with helper functions and configuration options. The extension enables you to configure a local run and work with Datasets the same way in which you would remotely, helping you validate that your dataset is being used correctly.
The extension streamlines setting up an optional debug session when running your experiment. This allows you to set breakpoints and step through your code with ease.
The extension has tightly coupled components of the debugging experience, like the debug console, with your experiment. Expressions you evaluate or print to the console will be written in your 70_driver_log.txt.

Running a local experiment is straightforward and closely resembles the extension's current functionality for submitting a remote run. Here's a summary of the steps for submitting a run.

Right-click on an experiment node in the tree view and choose the Run Experiment option.
Pick the local run option and choose whether you want to debug.
Create a new run configuration or pick a previous created one. The rest of the steps assume the former.
Pick an environment and dataset for your training.
(Only when debugging) Add the debugpy package to your environment. Debugpy is required when running an interactive debug session.
Validate the final configuration options and submit your run.
(Optional) If you've chosen to debug, start the debugger via the prompt or from your run node.

Local Experiment Submission with AML Extension

Congratulations! If you've followed the above steps you've successfully submitted a local experiment and can now confidently proceed to submit a remote run.

For more detailed step-by-step instructions you can follow our docs here.

Feedback

We're working hard to further improve your run experience from within VS Code, with focus on the following scenarios:

Debugging a single-node remote run on AmlCompute targets.
Streamlining submitting a remote run after succeeding locally.
Streamlining running a local debug experiment from a failed remote run.

If there's anything that you would like us to prioritize, please feel free to let us know on Github.

If you would like to provide feedback on the overall extension, please feel free to do so via our survey.

Microsoft named a leader in Forrester’s Notebook-based Predictive Analytics & Machine Learning Wave

Nishant Thacker — Fri, 02 Oct 2020 05:39:56 GMT

Forrester Research recently released their Wave report for Notebook-based Predictive Analytics and Machine Learning. Microsoft Azure is named a leader in this Wave, receiving the highest score possible in the ability to execute criteria and rated highest in the strategy category. You can download a complimentary copy of The Forrester Wave™ for Notebook-based Predictive Analytics and Machine Learning Solutions, Q3 2020 report here. In this post we’ll look at why Forrester rated Microsoft Azure as a leader.

Forrester evaluated Azure Machine Learning, recognizing its ‘full suite of enterprise PAML capabilities, from centralized model registries to hyperparameter tuning and modular model training and deployment pipelines.’ Forrester gave Azure Machine Learning the highest possible score in 13 evaluation metrics, the most in the report.

“The major cloud vendors have long had a gap in offering a comprehensive PAML platform that meets the full set of enterprise data science team needs, to the detriment of bewildered customers who have had to build or find their own solutions. Microsoft has filled that gap and then some.”

- The Forrester Wave™: Notebook-based Predictive Analytics and Machine Learning, Q3 2020

Let’s take a closer look at the capabilities that Azure Machine Learning features:

Collaboration and productivity – Azure Machine Learning offers customers a collaborative workspace with a dedicated notebook-based machine learning experience along with integrated compute (CPU and GPU) environments with support for all open source tools, frameworks and libraries. The Azure Machine Learning SDK, offered for both Python and R, also makes it simple to for people using tools like Jupyter, VS Code or any other notebook environment/IDE, to collaborate and reap the benefits of Azure Machine Learning. This makes data scientists productive from day 1.

Comprehensive coverage of the ML lifecycle – Azure Machine Learning helps with every step of the ML lifecycle. From data preparation and modelling, through to deployment and monitoring, every aspect of machine learning is carefully optimized with features like data labelling, hyperdrive, pipelines, drift monitoring and responsible ML toolkits. Azure Machine Learning also brings optimizers for popular frameworks and libraries to ensure that the training process runs optimally.

Enterprise readiness – Azure Machine Learning is a service that enables operationalization of ML models irrespective of how stringent the criteria is. Offering a best-in-class MLOps experience, Azure Machine Learning is equipped to help implement a robust deployment pipeline with automated monitoring and retraining capabilities. Azure Machine Learning also offers the security and governance features like private link, customer managed keys, VNet, and role-based access control (RBAC).

Ecosystem – Azure Machine Learning is part of growing ecosystem of services with Azure’s Data & AI offerings. It integrates natively with services like Azure Synapse Analytics, Azure Databricks and Power BI, to offer customers the flexibility to leverage the engine of their choice, like Apache Spark™ and/or SQL, for data wrangling and model scoring. It also brings a strong partner ecosystem and a dedicated certification accompanied by self-paced learning courses on Microsoft Learn and Udacity.

We feel Azure Machine Learning is the best environment for any organization that is building an ML practice with a code-first approach. Be it with notebooks or IDEs, the Azure Machine Learning Studio and the accompanying SDK, makes the Azure Machine Learning capabilities omnipresent across developers and data scientists tools and offers the best way to do secure, managed and scalable data science.

“Microsoft provides coding data scientists with all the bells and whistles.”

- The Forrester Wave™: Notebook-based Predictive Analytics and Machine Learning, Q3 2020

Although we doubled down on a lot of the notebook-based capabilities, Azure Machine Learning offers even more to the developer data scientist and/or citizen data scientist community. Capabilities like automated ML and designer, which we’ve recently made generally available, offer an experience where users can build machine learning models without knowing the intricacies of how frameworks and algorithms work. They can let the platform do the heavy lifting for them.

Microsoft’s mission is to empower every person and every organization on the planet to achieve more. With Azure Machine Learning we’re trying to accomplish this for the data science community.

Computer Vision for spatial analysis at the Edge

AdinaTru — Tue, 22 Sep 2020 15:00:52 GMT

Today businesses use manual processes to understand their physical spaces and meet business requirements such as maximizing revenue for store layouts, compliance, worker safety in manufacturing plants, and more. These manual processes are occurring infrequently and through inefficient methods where employees manually count people entering stores and visually monitor social distancing requirements.

Computer Vision, an Azure Cognitive Service, is introducing the spatial analysis feature to meet the needs of businesses across a variety of industries. Spatial analysis uses Computer Vision AI on real-time video and offers the ability to understand people’s movements in a physical space, significantly increasing efficiency and resolution of customer data. Equipped with this net new knowledge, employee time can be spent on high value experiences, to maximize return on investment, including efficient product placement, shelf stocking, sanitation, and focusing on improving the customer experience.

With AI Edge container support, spatial analysis provides the flexibility to solve current challenges around data control, privacy, and network intensive video AI workloads. Spatial analysis brings the power of the Intelligent Edge and the Azure cloud to create powerful business applications at scale.

Spatial analysis runs on the Intelligent Edge

Understanding the physical world using video is very complex, requiring deployments of video pipelines, hardware, AI models, and processing insights, all of which need to be connected into the existing infrastructure and aggregated into useful views. Organizations want simple deployments that can be hooked into their existing cameras as well as using new cameras that can be quickly installed to get immediate business value. The goal of this post is to demonstrate how enterprise applications can use the spatial analysis container on the edge, and how to use the Azure cloud to create powerful business applications.

Customer Story

RXR Realty needed a way to integrate new safety measures for tenants after its buildings reopened for business during the COVID-19 pandemic. The RxWell™ solution built by RXR is a comprehensive, public-health–based, data-driven program that merges physical and digital assets to help keep employees informed and supported during the “new abnormal” and beyond. RXR leveraged the Computer Vision spatial analysis capabilities of Cognitive Services to count the number of people in a specific zone and calculate the distance between each person. RxWell™ securely runs the models at the edge in Docker containers on Azure Stack Edge hardware. The resulting AI insights are sent to the cloud, where they’re used for real-time alerting and historical trending. To learn more about the RxWell™ solution with spatial analysis see this article.

Spatial analysis container for Azure IoT Edge deployments

Computer Vision for spatial analysis is a collection of AI operations using AI models. These operations are the building blocks that enable scenarios including:

Counting people in a space for maximum occupancy
Understanding the distance between people for social distancing measures
Determining footfall such as in retail spaces
Understanding dwell time such as in front of a retail display or other designated location
Determining wait time in a queue
Determining when people are in a forbidden zone such as near industrial equipment
Determining trespassing in protected areas

This list portrays a few examples of how spatial analysis can be utilized in a variety of scenarios. In addition to these scenarios, real-time messaging can power customer workflows including Teams, Power Apps, Power Automate, and saved events can enable powerful reports with Power BI.

Developers, enterprise solution providers, and customers with development capabilities can use these AI operations to build edge or cloud solutions, with no Machine Learning experience required. The ready-to-use, high-quality AI Models for spatial analysis operations are trained to understand people movement across a wide variety of scenarios, camera types, angles, and lighting conditions.

Spatial analysis operations implement a real-time video pipeline to connect to new and existing RTSP cameras, including Closed Circuit TV systems. The deployment of the spatial analysis container on edge devices is facilitated by Azure IoT Hub. You're in control of your edge hardware located on your premises. The recommended edge device is Azure Stack Edge with the Nvidia T4 GPU. Other edge devices may be used if an Nvidia T4 GPU is available.

Spatial analysis container deployment with Azure IoT

You need to install the camera devices in your physical space following the manufacturer’s instructions. The edge device hosting the container needs to be able to connect to these cameras over the RTSP protocol and stream video. Camera placement is an important step in the deployment process. For general guidelines and specific recommendations for height, angle, and camera-to-focal-point-distance, see Camera placement article.

When video is streamed and processed by the spatial analysis AI models, the container emits AI Insight events about people’s movement which in turn are sent to Azure IoT Hub as IoT telemetry. From IoT Hub you can create various routes to other Azure services and build your business solution. You may also decide to process the AI Insights locally on the edge device.

Live Video Analytics with spatial analysis

The spatial analysis container container can be deployed side by side with the Live Video Analytics container. In this configuration, the Live Video Analytics container will stream the live video from the RTSP cameras, and it will invoke spatial analysis for AI processing. The Live Video Analytics and spatial analysis containers running on Azure IoT Edge can be configured to enable rich video analysis and recording of video clips locally or to Azure Blob Storage.

Azure IoT deployment for Live Video Analytics and spatial analysis containers

Building business applications with spatial analysis AI insights

You can build various business applications using the spatial analysis AI insights for people movement. Follow these instructions to deploy a sample Azure Web Application that presents a live view of people counting events in a physical space. The AI insights events emitted by the spatial analysis container are being sent to Azure IoT Hub and subsequently to the Web Application, which in this case implements a visualization with a chart that updates the people count in real time as shown below. You can further modify this app with other spatial analysis operations and make modifications based on the event output of the container. When it comes to business decisions centered around social distancing requirements, determining store layouts, adding team members to reduce wait times, or improving customer satisfaction, spatial analysis makes it easy to gain these valuable insights.

Person count visualization in a web application

Responsible AI & Innovation

Microsoft is releasing Computer Vision for spatial analysis together with responsible deployment guidance grounded in user and societal research.

Microsoft developed the Responsible Deployment recommendations by applying many of the responsible innovation best practices in collaboration with customers to uncover deployment recommendations for spatial analysis in accordance with Microsoft Responsible AI Principles: fairness, reliability & safety, privacy & security, inclusiveness, transparency and human accountability.

Microsoft’s principled approach enables developers to build rich solutions while upholding the human dignity and the needs of everyone impacted by the technology.

Get Started Today

Learn more with Computer Vision for spatial analysis documentation

Follow the tutorial to Create a People Counting Web App

Get started with Azure Stack Edge

Learn about Azure IoT Hub

Automatically detect audio language with the Speech Language Detection Container

katerinaprastakou — Wed, 14 Apr 2021 23:46:59 GMT

We are excited to announce the release of the Speech Language Detection Container for Public Preview!

The Speech Language Detection feature is used to determine the most likely language match for a given audio where the language is not already known. By doing so, it unlocks our Speech-to-Text service to a vast number of scenarios and helps eliminate the language barrier.

Since the release of Speech Language Detection as an online service on Azure Cognitive Services, we have watched you enable new scenarios along with our Speech-to-Text and Translation services that open new doors for productivity and accessibility. Multilingual meetings, call center conversations, voicemails, and video streams can now capture every word for captioning and analytical insights. Spoken machine translation can automatically determine the source language without manual selection. And recommendation systems can better promote video content users can understand.

Control over your data

Perhaps you have wanted to explore with the Speech Language Detection feature before but were limited because of data restrictions. This could be due to data regulations, not wanting to or being able to load all your data into the cloud. By using the container version, you can now use the Speech Language Detection feature with complete control over your data.

Control over your throughput

Have you had to deal with a weak network connection or disconnected environments leading to high latency? With the container, you can scale for high throughput, low latency, requirements by enabling Cognitive Services to run in Azure Kubernetes Service physically close to your application logic and data.

Portable architecture

You can create a portable application architecture that can be deployed in the cloud, on-premises and the edge. This allows you the flexibility to add or remove containers very easily and adapt for what your project needs.

How It Works

The process is quite simple:

Client runs container with audio file or stream
Container receives the request
Container runs language detection model
Container returns to client the highest language match

Conceptually depicted below:

The result to the client will ultimately look like this:

Get started with installing the Speech Language Detection container

Learn more about how to download and run Speech service containers, including Speech Language Detection by visiting our documentation. Let us know your thoughts or new features you would like to see on this container!

For more information you can also explore:

Computer Vision Read (OCR) API previews new languages and docker containers

sanjeev_jagtap — Mon, 15 Mar 2021 00:20:11 GMT

Overview

Businesses today are racing to convert their scanned paper documents, digital files, and even on-screen content into actionable insights. These insights power knowledge mining, business process automation, and accessibility for everyone regardless of the source of content, location of users, and the language and medium of communication.

Optical Character Recognition (OCR) is the foundational technology that drives the digitization of content today by extracting text from images, documents, and screens. There are several OCR technology providers that provide this capability as services, tools, and solutions, both in the cloud and for deployment within your environment.

However, there are several challenges to successfully implementing OCR at scale.

Challenges

Text extraction quality

To extract text with high accuracy from the diverse content types, formats, and mediums, your OCR should be of the highest out-of-the-box quality, work on a variety of content textures, fonts, and styles, and be easy to integrate by using cloud APIs and SDKs.

Cloud and on-premise compliance

If you are a business that serves customers in healthcare, insurance, banking, or other verticals with additional data privacy and security requirements, you typically require not just secure online access but the flexibility to deploy within your network to ensure that the personal data does not leave your network.

Mixed languages

Your customers and users are global so your OCR should also support international languages and locales. These documents most likely contain text in multiple languages that are impossible to identify as you are scanning them at scale.

Handwritten text

Finally, your documents and forms have both print and handwritten text. To combat this challenge, you should use a technology that seamlessly handles both styles of text in the same document.

Computer Vision Read (OCR)

Microsoft’s Computer Vision OCR (Read) capability is available as a Cognitive Services Cloud API and as Docker containers. Customers use it in diverse scenarios on the cloud and within their networks to solve the challenges listed in the previous section.

The following figure illustrates the high-level flow of the OCR process. The input is your document or image. The service extracts the text and converts it into a structured JSON response that includes the extracted text lines and words with their bounding boxes and confidence scores. You integrate with the service or the containers with a simple API that’s described next.

OCR (Read) Cloud API overview

At its core, the OCR process breaks it down into two operations. You use the Read operation to submit your image or document. That starts an asynchronous process that you poll with the Get Read Results operation.

Read operation

Call the Read operation to extract the text. The call returns with a response header field called Operation-Location. The Operation-Location value is a URL that contains the Operation ID to be used in the next step.

See the full Read OCR REST API QuickStart in Python for the following code snippets.

text_recognition_url = endpoint + "/vision/v3.0/read/analyze" # Set image_url to the URL of an image that you want to recognize. image_url = "https://raw.githubusercontent.com/MicrosoftDocs/azure-docs/master/articles/cognitive-services/Computer-vision/Images/readsample.jpg" headers = {'Ocp-Apim-Subscription-Key': subscription_key} data = {'url': image_url} response = requests.post( text_recognition_url, headers=headers, json=data) response.raise_for_status() # Extracting text requires two API calls: One call to submit the # image for processing, the other to retrieve the text found in the image. # Holds the URI used to retrieve the recognized text. operation_url = response.headers["Operation-Location"]

Get Read results operation

Call the Get Read Results operation until it returns with a completed status. This operation takes as input the operation ID that was created by the Read operation. It returns a JSON response that contains a status field with the following possible values.

# The recognized text isn't immediately available, so poll to wait for completion. analysis = {} poll = True while (poll): response_final = requests.get( response.headers["Operation-Location"], headers=headers) analysis = response_final.json() print(json.dumps(analysis, indent=4)) time.sleep(1) if ("analyzeResult" in analysis): poll = False if ("status" in analysis and analysis['status'] == 'failed'): poll = False polygons = [] if ("analyzeResult" in analysis): # Extract the recognized text, with bounding boxes. polygons = [(line["boundingBox"], line["text"]) for line in analysis["analyzeResult"]["readResults"][0]["lines"]]

New Cloud API and Container releases

During Ignite 2020, we are announcing new cloud service and container releases.

New features in Read 3.1 preview (cloud and container)

The new Read 3.1 preview for cloud and containers adds these capabilities:

1. Support for Simplified Chinese and Japanese languages. See all supported languages.

The following images show Simplified Chinese and Japanese text lines extracted respectively, along with their bounding boxes (locations).

OCR (Read) Simplified Chinese

OCR (Read) Japanese

2. Indicating whether the appearance of text is Handwriting or Print style, along with a confidence score (Latin languages only).

The following image shows that the handwritten style text is not only correctly extracted but its appearance is recognized as handwriting style along with a confidence score for more granular control by the invoking business process.

OCR (Read) Print vs Handwritten Classification

3. Ability to extract text for selected pages or page range within multi-page documents.

The following image shows the JSON response with only the selected pages (201 and 202) when the Read operation was called with a pages query parameter of “201,202" for a 500 page document.

{ "status": "succeeded", "createdDateTime": "2020-09-08T19:23:19Z", "lastUpdatedDateTime": "2020-09-08T19:23:35Z", "analyzeResult": { "version": "3.1.0", "readResults": [ { "page": 201, "angle": 0, "width": 8.2639, "height": 11.6944, "unit": "inch", "language": "", "lines": [...] }, { "page": 202, "angle": 0, "width": 8.2639, "height": 11.6944, "unit": "inch", "language": "", "lines": [...] } ] } }

This preview version of the Read API supports English, Dutch, French, German, Italian, Japanese, Portuguese, Simplified Chinese, and Spanish languages.

Read 3.1 cloud API preview

We are releasing the new Read 3.1 preview cloud API with the following features:

Support for Simplified Chinese and Japanese
Print vs. handwriting appearance for each text line with confidence scores
Extract text from only selected page(s) from a large multi-page document

MIcrosoft Cognitive Services

See the Read API overview to learn more.

Read 3.0 and Read 3.1 container previews

We are announcing two new container releases.

docker containers

The Read 3.0 container preview

The Read 3.0 container preview is the on-premise version of the Read 3.0 Cloud API that’s generally available (GA) today.

The Read 3.0 container preview is a significant upgrade to the Read 2.0 container preview that’s available today. Major features include:

Enhanced accuracy based on updated deep learning models
Support for Dutch, English, French, German, Italian, Portuguese, and Spanish
Support for multiple languages within the same document
Single operation for both documents and images
Support for significantly larger documents and images
Confidence scores with a full range of 0 to 1 instead of labels (“low” only), , for granular visibility.
Support for mixed model documents with print and handwritten style text

The Read 3.1 container preview

The Read 3.1 container preview is the on-premise version of the same Read 3.1 API preview features we reviewed in the previous section. Moving forward, Read 3.1 and newer versions will add expanded language coverage and enhancements.

The Read 3.1 container preview includes everything that Read 3.0 has and adds the following additional capabilities covered in the previous section:

Support for Simplified Chinese and Japanese
Print vs. handwriting appearance for each text line with confidence scores
Extract text from only selected page(s) from a large multi-page document
Unified code and architecture ensures the container will be in step with cloud API releases in the future.

Get started with containers

Learn how to install and run the Read containers to get started and find the recommended configuration settings. If you are using Read 2.0 containers today, a migration guide is available for help along the way.

When to deploy which version

Can’t wait? Deploy the Read 3.0 container preview knowing that this release is on track for general availability (GA) soon. Want more languages and enhancements? Deploy the Read 3.1 container preview if you can wait a tad longer for more features.

Deploy the Read container of your choice

What customers say

Instabase

Instabase is a technology platform for business productivity applications that can be deployed in the cloud or on-premises.

Instabase logo

"Microsoft's Read OCR container technology provides a powerful option that our customers can leverage to read text from documents, which Instabase uses to generate understanding, without data ever leaving their firewall, ensuring data privacy and security. This is essential for our banking and enterprise customers." - Justin Herlick, Product Manager, Instabase

GE Aviation

While mandatory for regulatory compliance, assembling complete back-to-birth aircraft maintenance record is an expensive, time-consuming, and unreliable process. GE Aviation’s Digital Solutions Group built the AirVault solution to solve the challenge.

GE Aviation logo

“It can be a huge task across an airline's entire fleet to record and then easily retrieve evidence of maintenance activity or compliance to an Airworthiness Directive. A lot of what gets archived, things like parts receipts, vendor airworthiness certificates or maintenance records are still paper-based, and these documents can contain both handwriting as well as printed text. Microsoft’s Computer Vision OCR technology helped us to greatly enhance our full text word search capability during the conversion of paper documents to digital format as well as documents that were never printed in the first place with scale, speed, and accuracy.” - Nate Hicks, Sr. Product Group Leader at GE Aviation Digital Solutions

NHS Business Services Authority

UK’s NHS Business Services Authority (NHS BSA) is a Special Health Authority and an arm's length body of the Department of Health and Social Care (DHSC).

NHS Business Authority logo

“We believe that we can do so much more by using AI to read our documentation, to read more fields on that, and to read handwritten info, and to use that AI engine to deliver better taxpayer value, to deliver better outcomes, and deliver better patient safety.” - Michael Brodie, Chief Executive, NHS BSA

Read the case study

Get Started

Create a Computer Vision resource in Azure and follow one of our QuickStarts.
Learn more about OCR (Read) and Form Recognizer.
Learn more about the Read containers and download them from Docker Hub.
Write to us at formrecog_contact@microsoft.com

Accelerate self-paced learning at the edge with Speech Containers

Phani_Mutyala — Mon, 28 Sep 2020 16:31:01 GMT

We are pleased to announce that Speech to Text and Text to Speech containers from Azure Cognitive Services are now Generally Available (GA). Using these containers, customers can build a speech application architecture that is optimized for both robust cloud capabilities and edge locality.

With Speech to Text in containers, businesses across industries have unlocked new productivity gains and insights by enabling real-time and batch transcription of audio streams into text. With Text to Speech customers can enable applications, tools, or devices to convert text into human-like synthesized speech.

Organizations ranging from banking, telecom, aerospace and defense leverage speech containers to solve great business needs including: call center transcription & analytics, self-paced learning tools, and intelligent kiosks. Azure is the only cloud provider enabling customers with full flexibility of running artificial intelligence on their own terms, whether on-premises or or at the edge.

The goal of this post is to show how our customers leverage containers to solve AI needs at the edge.

Airbus is an international leader in the aerospace sector. They design, manufacture and deliver industry-leading commercial aircraft, helicopters, military transports, satellites and launch vehicles, as well as providing data services, navigation, secure communications, urban mobility and other solutions for customers on a global scale.

With Azure Cognitive Services, Airbus advances its aerospace operations, specifically their pilot training chatbots to harness the speech capabilities to engage and educate pilot staff. By integrating Azure AI speech and transcription capabilities, Airbus was able to engage and educate pilot staff with most up to date detail and safe practices. Watch this video for more details:

The Problem and Customer Pain Point

Airbus trains tens of thousands of commercial aircraft and military pilots annually. The customer pain point in pilot training is driven by the complexity of modern commercial and military aircraft. In recent years aircraft complexity has increased at such a rate that the scope of knowledge required for operating aircraft reliably and safely has increased exponentially.

The volume of training material amounting to a pilot training course content is rapidly approaching levels which are becoming difficult for trainees to retain with acceptable levels of recall and accuracy.

How bad is the pain?

The average pilot conversion course for an experienced pilot converting to a new aircraft platform exceeds more than 7000 pages of printed documentation. This content must be reviewed, committed to memory and recalled with very high levels of accuracy not only during the 10 to 12 week duration of the conversion training course, but throughout the entire operational life of the pilot. The Airbus pilot training chatbot has been developed on an enterprise chatbot platform and is being enhanced with Azure speech service capabilities.

Pilot training chatbot and how it is integrated with Azure speech containers

The objective of the pilot training chatbot is to provide pilot trainees with an alternative method for review, revision and self-paced learning. The pilot training chatbot is not designed to replace human flight instructors but rather looks to extend the coverage and access to their existing instructor knowledge base and supplements already developed standardized training methods. The chatbot is used to test knowledge areas for recall and accuracy.

Technical Challenges - The technical challenge for this project was to not depend on any public cloud services as, although initially focusing on civil aircraft, the projects aims to support military and governmental aircraft types as well.

This requires a disconnected and, in some cases, air gapped style of deployment. The heart of the chatbot is implemented in an on premise enterprise chatbot platform. It has a powerful conversation engine; however, but doesn't include speech technologies. However these can be integrated into the conversation using the APIs of speech technology services.

The challenge to integrate a voice interface was addressed using the Cognitive Services (speech) containers. The containers have been deployed on a kubernetes cluster running in a secured environment, ensuring flexibility and ease of deployment.

The chatbot connects via API to the container using the Speech SDK. It forwards the user's speech input to convert to text output using the Azure on premise speech to text container and responds to the user in either text or voice, depending on the user’s settings. In case the user chooses the full speech mode it will vocalize answers by using the Text to Speech API container to receive an audio file, which is sent to the user’s device for playback. Since all APIs are in the same environment, latency is no issue and the interaction feels natural and fast.

The following graph gives a short overview of the communication flow:

Solution Overview - Speech to Text Integration

The UI for the pilot training chatbot is a JavaScript Web UI incorporating HTML chat window display and controls. Once the Speech to Text container was deployed within the Kubernetes environment, the next step was to integrate Speech to Text into the chatbot Web UI.

For this step, the sample JavaScript code from the Azure Cognitive Service Speech SDK proved to be a really useful resource. It provides code templates for Airbus to update and get started with connectivity to the STT Container, using the Microsoft Speech SDK JavaScript library.

Once connectivity to the STT Kubernetes service was established, configured and tested with the STT settings and Speech SDK, all the required functions transferred directly into the chatbot JavaScript code. Utilizing the Speech SDK examples for the initial testing and configuration before integration into the target UI really saved time to achieve the final goal of integration of Speech capabilities for the chatbot.

The recognizeonceAsync function was used as an example from the sample, as it requires a single utterance transcription, after which to stops to await reply from the Pilot Training chatbot.

The continuous recognitionAsync function was not required but can be used if transcribing continuously until explicitly stopped is needed, e.g. dictation uses cases.

reco.recognizeOnceAsync( function (result) { window.console.log(result); statusDiv.innerHTML += "(continuation) Reason: " + SpeechSDK.ResultReason[result.reason]; switch (result.reason) { case SpeechSDK.ResultReason.RecognizedSpeech: statusDiv.innerHTML += " Text: " + result.text; break; case SpeechSDK.ResultReason.NoMatch: var noMatchDetail = SpeechSDK.NoMatchDetails.fromResult(result); statusDiv.innerHTML += " NoMatchReason: " + SpeechSDK.NoMatchReason[noMatchDetail.reason]; break; case SpeechSDK.ResultReason.Canceled: var cancelDetails = SpeechSDK.CancellationDetails.fromResult(result); statusDiv.innerHTML += " CancellationReason: " + SpeechSDK.CancellationReason[cancelDetails.reason]; if (cancelDetails.reason === SpeechSDK.CancellationReason.Error) { statusDiv.innerHTML += ": " + cancelDetails.errorDetails; } break; } statusDiv.innerHTML += "\r\n"; phraseDiv.innerHTML = result.text + "\r\n"; sdkStopRecognizeOnceAsyncBtn.click();

Text to Speech Integration

Airbus utilized the Azure Cognitive Service Speech SDK to get started with Text to Speech as soon as the container was deployed on their Kubernetes Infrastructure. The team selected Hazel UK as the voice for the initial tests with speech synthesis due to the clarity of pronunciation, but neural speech synthesis is of huge interest to try the future.

For the speech synthesis within the Chat UI, the application calls the Text to Speech container’s Rest API directly without using the Speech SDK library, as it requires fewer configurations for the service call in comparison to Speech to Text.

For longer responses from the chatbot (more than 100 characters they set a lower prosody rate of 0.9 ( <prosody rate="0.9"> ). This is so as to ensure the response is not too fast for the Pilot, enabling them to have time to process the response from the chatbot. In addition, we replace end of sentence full stops and commas with longer pauses ('<break time="600ms"/>'). Again this allows the Pilot sufficient time to process the reply of the chatbot.

For example, the sentence: “MTOW is an abbreviation for Maximum Takeoff Weight, which defines the maximum weight at which a pilot is allowed to attempt to take off, due to structural or other limits.”

Would translate to:

“<say-as interpret-as="characters">MTOW</say-as> is an abbreviation for Maximum Takeoff Weight, <break time="600ms"/> which defines the maximum weight at which a pilot is allowed to attempt to take off, due to structural or other limits.”

The word “MTOW” is spoken as individual letters and an empathized pause is included after the first comma.

Measuring key results - As an initial launch platform, the Pilot Training chat bot is targeting the Airbus A330 MRTT type, a military tanker refueling version of the standard Airbus A330 airliner.

Results and feedback have been extremely positive with both the instructor and trainee communities eagerly offering recommendations for content enhancements to supplement existing course material. Users feel the combination of AI and speech capability will bring about an enhanced learning experience, making for more efficient aircraft operation and safer skies.

Takeaways from this solution implementation

Azure Speech SDK really accelerated the integration of speech technologies with the pilot training chatbot, saving many hours of development. Documentation and developer samples are easy to follow and are available in a variety of programming languages.
Standard speech to text containers brings both accuracy and the speed of inference.
Speech container’s pronunciation is clear and professional.
Containers allowed Airbus to use the latest technology with reasonable effort in even the strictest and most regulated environments, boosting their time to market and providing a clear differentiator compared to other AI providers.

Get Started...! Learn more about speech containers and deploy solutions at the edge

Deploying your first container is about a 2-minute read, you basically create a resource at Azure portal, download image, run container with environmental variables. Here's a document to help you get started on running containers.

Containers available from Azure Speech Service are:

Speech Service Documentation

Cognitive Services containers
Get Started, learn more and take advantage of Azure Cognitive Services containers to build intelligent applications today and learn more.

Introducing Metrics Advisor - A new Cognitive Service

Tony_Xing — Tue, 22 Sep 2020 14:40:56 GMT

It is key to stay on top of the status of the physical assets, products, services, and business through data intelligence for companies and organizations which are embracing digital transformation. The way they are doing this is by extracting the key metrics which are proxies to those assets and monitoring the metrics 24X7. And if there is anything wrong detected, they would like to know immediately and act on that to prevent the small issues from becoming customer-impacting incidents. This becomes difficult when the data volume is huge, therefore identifying objects, groups of objects, events, or event patterns that deviate from the expected or norm with scale is needed.

We are pleased to announce the preview of the Metrics Advisor, part of Azure Cognitive Services to address the need for metrics intelligence. The service ingests data from various sources, using machine learning to automatically find anomalies from sensors, products, and business metrics, and provides diagnostics insights. Metrics Advisor goes beyond simple Anomaly Detection by providing developers an out-of-the-box platform of multi-dimensional metric data ingestion, anomaly detection, and automatic model customization through user feedback powered by reinforcement learning. The capabilities of the pipeline can be easily used by developers to build predictive maintenance, AIOps (artificial intelligence for IT operations), and business metric monitoring solutions.

Overview

Metrics Advisor leverages sophisticated mathematical techniques — machine learning and other advanced analytics — to precisely detect more-subtle anomalies, provide earlier notice of likely future anomalies, and streamline the design and development of systems that detect (and even act on) anomalies. Built on the Anomaly Detector, it includes the capabilities to ingest data from various standard data sources, use the data to build out models, model tuning, and feedback-based model customization behind the scenes. Last but not least root cause analysis with advanced insights & recommend actions.

What Can You Do with Metrics Advisor

Let me show you what kind of problems you can solve with the Metrics Advisor. Imagine you are the someone responsible for company Contoso’s e-commerce website. To ensure both the business and services are in good health, many important business metrics and infrastructure metrics are generated and onboarded to Metrics Advisor, e.g., DAU (daily active users), CPU usage, web page latency, database throughput…
At a given date, DAU anomaly was detected on the all-up metric aggregated by all regions and all channels and you got a notification.

At this moment, automated diagnostics info was available on the portal or via APIs. It turned out that the leading contributors to this anomaly were from the region of the United States and the channel of Direct. So the investigation and mitigation should focus on those areas to start with.

Were there any other underlying issues you should look into? By checking out the Metrics graph which depicts the metrics dependency across the infrastructure metrics, it became obvious that the MySQL problem caused the latency issue of the web app, which propagated to impact the DAU of specific region and channel.

You as the engineering lead of the Contoso e-commerce website, can easily get automated insight within a few minutes and identify the potential root cause. All the operations can be done via portal or APIs if you would like to embed this capability into your org’s own experience. After Ignite 2020, we will launch SDKs to ease your coding with the APIs.

Magic Behind the Scenes

Model selection framework & tuning

Firstly, the time-series anomaly detection task is challenging because of the complex characteristics of time-series, which are messy, stochastic, and often without proper labels. This prohibits training supervised models because of lack of labels and a single model hardly fits different time series.
We present an automated model selection framework to automatically find the most suitable detection model with proper parameters for the incoming data. The model selection layer is extensible as it can be updated without too much effort when a new detector is available to the service. Finally, we incorporate a customized tuning algorithm to flexibly filter anomalies to meet customers’ criteria. Experiments on real-world datasets show the effectiveness of our solution.
As shown in the pipeline below, the incoming series is first processed by a set of transformations and feature extractors. Then in the automated model selection phase, Model Selector takes the extracted features as input and outputs the anomaly detection model that best fits the input data. Each anomaly detection model is associated with a Parameter Estimator, which is used to compute related parameters. Next, our service uses the selected model and its corresponding parameters to detect anomalies of the input data and obtains a preliminary anomaly detection result. Lastly, tuned parameters are applied to obtain a customized anomaly detection result.

Model customization through user feedback

Tuning is one way to customize the model to users’ business and dataset. We are also leveraging user feedback as human knowledge to adapt the model/parameters to better serve and fit customers’ data and business.

The motivation is that different customers have different service patterns and anomaly definitions. It is complicated for customers to tune the model directly to be adapted to their scenarios. To solve this problem, we provide a feedback mechanism in the Metrics Advisor. Customers can obtain more accurate detection results by providing confirmation on these results which are fed into an end-to-end framework based on Reinforcement Learning (RL) to learn customer feedbacks.

Adaptive root cause analysis

Root cause analysis consists of two major processes,

Automation part: From the nature of the metrics, such as hierarchy and distribution, using machine learning to generate an analysis report to find out the most likely root causes in an incident.
Online Learning part: From the data topology, feedbacks, and interactions of the incident owner, inference the severity of an incident to make sure the alerts are actionable and reduce ignorable information in analysis reports.

The learning part will ingest strategies into the automation part to improve the quality of the analysis report and alerts.

While there are a bunch of automation technologies are available for root cause analysis, Metrics Advisor takes customization into account. Because the Metrics Advisor is a general cloud service but customers’ scenarios are diverse, similar incidents might mean differently for different customers, or for the same service in different stages. The capabilities of learning from customers’ feedbacks and implicitly being tuned to be more actionable are the biggest advantage of Metrics Advisor. To implement that, the Metrics Advisor creatively combines online learning technologies and incident representation to achieve a self-evolvable Root Cause Analysis.

Get Started

Learning more with our documentation
Create a free Metrics Advisor resource
Onboard your first dataset with our QuickStarts

References

Ignite 2020 Neural TTS updates: new language support, more voices and flexible deployment options

Qinying Liao — Fri, 25 Sep 2020 10:54:41 GMT

Ignite 2020 Neural Text-to-Speech updates: new language support, more voices and flexible deployment options

This post was co-authored by Garfield He, Melinda Ma, Yueying Liu and Yinhe Wei

Neural Text to Speech (Neural TTS), a powerful speech synthesis capability of Cognitive Services on Azure, enables you to convert text to lifelike speech which is close to human-parity. Since its launch, we have seen it widely adopted in a variety of scenarios by many Azure customers, from voice assistants to audio content creation. We continue to push the envelope to enable more developers to add natural-sounding voices to their applications and solutions.

Today, we are happy to announce a series of updates to Neural TTS that extends its reach globally and allows developers to deploy it anywhere the data resides. This includes new languages available, new voices with rich personas, and on-prem deployment through docker containers.

**18 new languages/locales supported**

Neural TTS has now been extended to support 18 new languages/locales. They are Bulgarian, Czech, German (Austria), German (Switzerland), Greek, English (Ireland), French (Switzerland), Hebrew, Croatian, Hungarian, Indonesian, Malay, Romanian, Slovak, Slovenian, Tamil, Telugu and Vietnamese.

You can hear samples of these voices below.

Locale	Language	Gender	Voice	Sample
bg-BG	Bulgarian	Female	Kalina	Архитектурното културно наследство в България е в опасност.
cs-CZ	Czech	Female	Vlasta	Policisté většinou chodí v uniformě a jsou označeni hodnostmi.
de-AT	German (Austria)	Female	Ingrid	Ab Herbst werden Lehrer, die sich dafür interessieren, eigens ausgebildet.
de-CH	German (Switzerland)	Female	Leni	Dreizehn Millionen Liter mehr als im Vorjahr.
el-GR	Greek	Female	Athina	Για να βρεις ποιος σε εξουσιάζει, απλώς σκέψου ποιος είναι αυτός που δεν επιτρέπεται να κριτικάρεις .
en-IE	English (Ireland)	Female	Emily	Now we have seventy members and two dragon boats.
fr-CH	French (Switzerland)	Female	Ariane	Chaque équipe jouera donc 5 matchs de 20 minutes dans sa poule.
he-IL	Hebrew (Israel)	Female	Hila	הכל פתוח במאבק על המקום האחרון לפלייאוף העליון של ליגת העל בכדורגל.
hr-HR	Croatian	Female	Gabrijela	Idemo na pobjedu u Maksimiru, pred našem publikom dat ćemo sto posto.
hu-HU	Hungarian	Female	Noemi	A macska felmászott a tetőre és leugrott.
id-ID	Indonesian	Male	Ardi	Inflasi dapat digolongkan menjadi empat golongan, yaitu inflasi ringan, sedang, berat, dan hiperinflasi.
ms-MY	Malay	Female	Yasmin	Beg berkenaan dibawa ke hospital untuk menjalankan proses pengenalan.
ro-RO	Romanian	Female	Alina	Temperaturile maxime se vor încadra între 15 şi 23 de grade Celsius.
sk-SK	Slovak	Female	Viktoria	Kúzelné miesta nájdete aj za jej hranicami, v malebnej prírode.
sl-SI	Slovenian	Female	Petra	Predlagani zakon vključuje tudi načrt nadaljnjega ukrepanja.
ta-IN	Tamil	Female	Pallavi	உச்சிமீது வானிடிந்து வீழுகின்ற போதினும், அச்சமில்லை அச்சமில்லை அச்சமென்பதில்லையே
te-IN	Telugu	Female	Shruti	అందం ముఖంలో ఉండదు. సహాయం చేసే మనసులో ఉంటుంది
vi-VN	Vietnamese	Female	HoaiMy	Hà Nội là thủ đô của Việt Nam.

With these new voices, Microsoft Azure Neural TTS supports 49 languages/locales in total.

14 additional voices released to enrich the variety

Customers use TTS for different scenarios and their requirements for voice personas can vary. To provide more options to developers, we continue to create more voices in each language. Besides the extension to support new locales, we’ve announced 14 new voices to enrich the variety in the existing languages.

Hear samples of these voices below.

Locale	Language	Gender	Voice	Sample
de-DE	German	Male	Conrad	Je würziger das Fleisch, desto würziger und kräftiger sollte auch der Wein sein.
en-AU	English (Australia)	Male	William	They have told me nothing, and probably cannot tell me anything to the purpose.
en-GB	English (UK)	Male	Ryan	Today’s temperature was a record 26.5 degrees Celsius.
en-US	English (US)	Female	Jenny	For example, we place a session cookie on your computer each time you visit our Website.
es-ES	Spanish (Spain)	Male	Alvaro	Dos helicópteros medicalizados tuvieron que acudir al lugar a rescatar a los heridos.
es-MX	Spanish (Mexico)	Male	Jorge	El niño mencionó que si pudiera caminar, pediría un balón para poder patearlo o una cuerda para poder saltar.
fr-CA	French (Canada)	Male	Jean	Ce jour tant attendu arrive enfin!
fr-FR	French (France)	Male	Henri	Jusqu'ici, nous vous avons toujours fait confiance et accordé le bénefice du doute.
it-IT	Italian	Female	Isabella	I gel igienizzanti sono aumentati di prezzo.
it-IT	Italian	Male	Diego	Domani preparerò dei biscotti con le gocce di cioccolato.
ja-JP	Japanese	Male	Keita	キャッシュレス決済を利用して、支払いを簡単にする。
ko-KR	Korean	Male	InJoon	규모가 더욱 확대되었다.
pt-BR	Portuguese (Brazil)	Male	Antonio	O que você quer ganhar de presente de natal?
th-TH	Thai	Female	Premwadee	วิกฤตแบบนี้บริษัทยิ่งต้องการคนที่พร้อมเผชิญปัญหา

With these updates, Microsoft Azure Text-to-Speech service offers 68 neural voices. Hear all these neural voices saying 'Thank you' in 49 languages/locales in the video below.

Across standard and neural TTS capabilities, we now offer 140+ voices in total. Check the 70+ standard voices.

More than 15 speaking styles available in en-US and zh-CN voices

Today, we’re building upon our Neural TTS capabilities in English (US) and Chinese (CN) with new voice styles. By default, the Text-to-Speech service synthesizes text using a neutral speaking style. With neural voices, you can adjust the speaking style to express different emotions like cheerfulness, empathy, and calm, or optimize the voice for different scenarios like customer service, newscasting and voice assistant that fit your need.

With the English (US) new voice, Jenny, which is created with a friendly, warm and comforting voice persona focusing on conversational scenarios, we provide additional speaking styles including chatbot, customer service, and assistant.

You can hear the different speaking styles in Jenny’s voice below:

Style	Style description	Sample
General	Expresses a neutral tone and available for general use	Valentino Lazaro scored a late winner for Austria to deny Northern Ireland a first Nations League point.
Chat	Expresses a casual and relaxed tone in conversation	Oh, well, that's quite a change from California to Utah.
Customer service	Expresses a friendly and helpful tone for customer support	Okay, great. In the meantime, see if you can reach out to Verizon and let them know your issue. And Randy should be calling you back shortly.
Assistant	Expresses a warm and relaxed tone for digital assistants	United States spans 2 time zones. In Nashville, it's 9:45 PM.

A new speaking style is also available for the en-US male voice, Guy. Guy’s newscast style can be a great choice for a male voice that can read professional and news related content.

In addition, 10 new speaking styles are available with our zh-CN voice, Xiaoxiao. These new styles are optimized for audio content creators and intelligent bot developers to create more engaging interactive audios that express rich emotions.

You can hear the new speaking styles in Xiaoxiao’s voice below:

Calm	Affectionate	Angry
那，那我再问你，你之前有养过宠物嘛？	老公，把灯打开好吗，好黑呀，我很怕。	没想到，我们八年的感情真的完了！
Disgruntled	Fearful	Gentle
这你都不明白吗？真是个榆木脑袋。	先生，你没事吧？要不要我叫医生过来？	我今天运气特别好,如果没有遇到您,还不知道会怎么样呢！
Cheerful	Serious	Sad
太好了，恭喜你顺利通过考核。	不要恋战，等待时机，随时准备突围。	没想到，你居然是这么一个无情无义的的人！

For the Chinese voice Xiaoxiao, the intensity (‘style degree’) of speaking style can be further adjusted to better fit your use case. You can specify a stronger or softer style with 'style degree' to make the speech more expressive or subdued.

没想到，你居然是这么一个无情无义的的人！
Sad=0.5	Sad=1.0
Sad=1.5	Sad=2.0

The style degree can be adjusted from 0.01 to 2 inclusive. The default value is 1 which means the predefined style intensity will be applied. The minimum unit is 0.01, which softens the style with a flatter tone. The value of 2 is the highest, which makes the style intensity obviously stronger than the default.

The SSML snippet below illustrates how the 'style degree' attribute is used to change the intensity of a speaking style.

<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis"

xmlns:mstts="https://www.w3.org/2001/mstts" xml:lang="zh-CN">

<mstts:express-as style="sad" styledegree="2">

快走吧，路上一定要注意安全，早去早回。

</mstts:express-as>

</voice>

</speak>

The 'style degree' feature currently only applies to the Chinese voice Xiaoxiao and will come to more languages and voices later soon.

Check SSML for the details on how to use these speaking styles, together with other rich voice tuning capabilities.

Neural TTS Container is in public preview with 16 voices available in 14 languages

We have launched Neural TTS Container in public preview, as we are seeing a clear trend towards a future powered by the intelligent cloud and intelligent edge. With Neural TTS Container, developers can run speech synthesis with the most natural digital voices in their own environment for specific security and data governance requirements. Their Speech apps are portable and scalable with greater consistency whether they run on the edge or in Azure.

Currently 14 languages/locales are supported with 16 voices in Neural TTS Containers, as listed below.

Locale	Voice
de-de	KatjaNeural
en-au	NatashaNeural
en-ca	ClaraNeural
en-gb	LibbyNeural
en-gb	MiaNeural
en-us	AriaNeural
en-us	GuyNeural
es-es	ElviraNeural
es-mx	DaliaNeural
fr-ca	SylvieNeural
fr-fr	DeniseNeural
it-it	ElsaNeural
ja-jp	NanamiNeural
ko-kr	SunHiNeural
pt-br	FranciscaNeural
zh-cn	XiaoxiaoNeural

To get started, fill out and submit the request form to request access to the container. Currently Neural TTS Containers are gated and only approved for enterprises (EA customers) and Microsoft partners, and to an extent only for qualified customers.

Azure Cognitive Services Containers including Neural TTS Containers aren't licensed to run without being connected to the metering / billing endpoint. You must enable the containers to communicate billing information with the billing endpoint at all times. Cognitive Services containers don't send customer data, such as the image or text that's being analyzed, to Microsoft. Queries to the container are billed at the pricing tier of the Azure resource that's used for the ApiKey.

Here are the steps of how to install and run the container:

Make sure your machine to host the container meets the hardware requirements.
Get the container image with docker pull. For all the supported locales and corresponding voices of the neural text-to-speech container, please see Neural Text-to-speech image tags.
Run the container with docker run.
Validate that the container is running.
Query the container’s endpoint. Take AriaNeural voice for example, you can run below HTTP post method to get the TTS output audio:

curl -s -v -X POST http://localhost:5000/speech/synthesize/cognitiveservices/v1 \

-H 'Accept: audio/*' \

-H 'Content-Type: application/ssml+xml' \

-H 'X-Microsoft-OutputFormat: riff-24khz-16bit-mono-pcm' \

-d '<speak version="1.0" xml:lang="en-US"><voice name="en-US-AriaNeural">This is a test, only a test.</voice></speak>' > output.wav

Learn more about Container support in Cognitive Services and visit the Frequently Asked Questions on Azure Cognitive Services Containers.

Get started

With these updates, we’re excited to be powering natural and intuitive voice experiences for more customers globally with flexible deployment options. For more information, visit below.

Try the TTS demo
See our documentation
Check out our sample code
Learn about Speech containers

Ignite 2020 - Conversational AI updates

GaryPrettyMsft — Tue, 22 Sep 2020 15:26:11 GMT

In the 6 months since Microsoft Build 2020, where we made exciting steps forward, such as the GA availability of Bot Framework Composer and the Virtual Assistant Solution Accelerator, we have continued to drive the Conversational AI platform forward - improving the developer experience and meeting the needs of our enterprise customers. Azure Bot Service now handles 2.5 billion messages per month, double the rate announced at Build, with over 525,000 registered developers.

Our updates for Ignite 2020 include a new release of Bot Framework Composer, a public preview of Orchestrator, providing language understanding arbitration / decision making, optimized for conversational AI applications and version 4.10 of the Bot Framework SDK.

New release of Bot Framework Composer

Bot Framework Composer v.1.1.1, released earlier this month, has added a number of significant features to the application, including the addition of creation and management capabilities for QnA Maker knowledgebases. Now, as with the existing integration for LUIS apps, QnA pairs can be added / edited from within Composer, improving overall productivity by removing the need to use a separate portal for these tasks.

QnA Maker integration in Composer

The ability to build bots that target multiple languages has been added, with a user able to produce appropriate LU (language understanding) and LG (language generation) assets in seconds to target one or more alternative locales.

Other enhancements in this release include automatic generation of manifests when developing Bot Framework Skills, the addition of Intellisense for text editing and preview support for a JavaScript bot runtime.

We also continue to deepen our integration with other key partners within Microsoft and starting this fall, users of Power Virtual Agents will be able to create custom dialogs and directly add them to Power Virtual Agents bots. These dialogs can be saved, hosted, and executed together with Power Virtual Agents bot content, providing a simpler way to extend bot capabilities with custom code.

The next release of Composer, later this year, will feature further QnA Maker integration, improvements to the authoring canvas and the ability to easily re-use assets built with Composer between projects.

Additionally, we are updating the list of pre-built skills, that we released as part of Virtual Assistant Solution Accelerator 1.0, to be based on Bot Framework Composer and adaptive dialogs.

Get started with Composer today at http://aka.ms/bfcomposer.

Orchestrator public preview!

Conversational AI applications today are built using multiple technologies to fulfil various language understanding needs, such as LUIS and QnA Maker, as well as often being composed of multiple skills, with each fulfilling a specific conversation topic. Orchestrator answers a critical need for language understanding arbitration and decision making, to route incoming user request to an appropriate skill or to dispatch to a specific sub-component within a bot.

Orchestrator is a transformer-based solution, which is heavily optimized for conversational AI applications and runs locally within a bot. You can find more details and try the Orchestrator public preview by visiting https://aka.ms/bf-orchestrator.

Later this year, we plan to introduce a preview of Orchestrator support within Composer and the Virtual Assistant Solution Accelerator.

Bot Framework SDK 4.10 released

Version 4.10 of the Bot Framework SDK is now available, adding several new supporting features for our key partners. These included enhanced support for building skills for Power Virtual Agents and adding adaptive dialog and additional lifecycle event support for Microsoft Teams.

A core focus of this release was on quality across the entire stack, covering key pillars of documentation, customer supportability, customer feature requests, code quality and improvements to our internal team agility. Almost 600 GitHub issues were resolved as part of the release, across our SDK languages (C#, JavaScript, Python, Java) and our tools, including accessibility improvements for WebChat.

See the August 2020 release notes for more details on v4.10 of the SDK.

Azure Bot Service

We are excited to announce that the Alexa channel for Azure Bot Service (ABS), which went entered public preview at Build 2020, is now generally available!

In response to feedback from customers, ABS now has support for WhatsApp, allowing you to surface your bot on the popular chat app, alongside the existing channels available via ABS. Built in partnership with InfoBip, the WhatsApp adapter can be added to your bot within minutes. Get started with WhatsApp integration for Bot Framework at https://aka.ms/bfwhatsapp.

As part of our commitment to customer privacy and security, ABS has introduced support for Azure Lockbox. Lockbox enables approval flows and audits when support engineers require access to customer data and, additionally, we will soon add customer managed encryption keys.

Azure Bot Service now has expanded channel support within the Azure US Government region and, looking ahead, we will be adding a preview of Adaptive Cards 2.0 and SSO (single sign-on) support for the Teams and WebChat channels.

Ignite 2020 sessions

Conversational AI Customer and Employee Virtual Assistants
Darren Jefford, Group Program Manager, Conversational AI

Building Bots with Power Virtual Agents and extending them with Microsoft Bot Framework
Marina Kolomiets - Senior Program Manager, Power Virtual Agents

Power your VS Code Notebooks with AzML compute instances!

Sid_Unnithan — Thu, 03 Sep 2020 22:30:00 GMT

Hey AzML community! The VS Code team is excited to announce version 0.6.14 of the AzML extension, with added support for Dataset creation from existing Datastores and an awesome new feature that enables you to power-up your local Jupyter Notebooks.

If you'd like to follow-along with the blog post and try out the new features, you can install the extension here.

Powering your VS Code Notebooks with an AzML Compute instance

Many of you have praised the Jupyter Notebook integration in VS Code as it's become an integral part of your data-science workflow. You've also exclaimed that sometimes you want to run your Notebooks against more powerful machines without having to deal with SSH and connecting to remote servers.

The AzML extension now provides a highly streamlined way of connecting your local Jupyter Notebooks to a compute instance. We're authenticating using your Azure credentials and eliminating previous manual steps for connecting to the remote server.

It's extremely easy to get started - you can simply click on the "Jupyter server: " button in the Notebook toolbar or invoke the "Azure ML: Connect to compute instance Jupyter server" command.

Invoke remote Jupyter server connection

In case you're unable to see the "Azure ML: Compute Instances" list option, it's likely that you don't have the AzML extension installed. It might take a little bit of time for the quick-pick options to show up as you may be activating the AzML and Python extensions for the first time.

Following the extension's guided prompts, you can select from a list of compute instances in your workspace. You can also create a new instance by simply providing a name and a VM size.

As a friendly reminder, if you don't have any workspaces you can create one via the "Azure ML: Create Workspace" command.

Select or create a new compute instance

Once you've selected a compute instance, you will be prompted to reload your VS Code window. After reloading the window and reopening your Notebook, you must run a cell to initiate the compute instance connection.

Run a cell to connect to the compute instance

Voila! Your local Jupyter Notebook is now running against your AzML compute instance. You gain all the benefits of using Notebooks in VS Code, coupled with the benefits of running against a more powerful remote compute.

For more detailed step-by-step instructions you can follow our docs.

Creating a Dataset from an existing Datastore

With our 0.6.12 (May) release, the AzML extension added support for creating datasets from directly within VS Code. Up until now you could only create a dataset using local files or a web URL. The extension now allows you to use an existing datastore to create a dataset. Following the guided prompts, you can choose from a list of registered datastores and then the absolute path to your data.

Create a dataset from an existing datastore

Feedback

The new Notebook and compute instance integration is still in its preleminary phase and we're actively working on supporting a broader set of scenarios:

Viewing and interacting with your remote server's filesystem.
Mounting a dataset onto the compute instance.
Supporting remote Jupyter server UI changes in the new native notebook editor.

If there's anything that you would like us to prioritize, please feel free to let us know on Github.

If you would like to provide feedback on the overall extension, please feel free to do so via our survey.

Accelerating AI for COVID-19 on Microsoft Azure Machine Learning using Clara Imaging from NVIDIA NGC

Krishna_Anumalasetty — Sun, 11 Oct 2020 18:43:42 GMT

This post was co-authored by Krishna Anumalasetty, Tom Drabas, Nalini Chandhi from Microsoft and Abhilash Somasamudramath, Manuel Reyes Gomez, Brad Genereaux, Akhil Docca from NVIDIA

The following three videos will walk you through the major steps outlined in this blog and will help you in implementing the solution.

Figure 1. CT-Scan Images Classified for COVID-19 Probability

COVID-19 has fundamentally transformed the world we live in. As the scientific community across the globe unites in the face of this pandemic, it is crucial to enable researchers to collaborate and leverage tools that will speed up detection and drug discovery for COVID-19. The power of AI in radiological medical imaging is helping with faster detection, segmentation, and notifications.

Leveraging AI for healthcare applications is challenging for many reasons. The complicated science required to build and train deep learning neural networks, setup and maintenance of the supporting infrastructure required to develop, deploy, and manage these applications at scale, pose barriers towards addressing our most pressing healthcare challenges with AI.

Cloud computing has enabled researchers with easy access to scalable, on-demand infrastructure to get their AI applications up and running quickly. In the medical imaging space, the real need is for a platform that combines the power of NVIDIA GPUs with a secure environment that also allows easy access to AI software. NVIDIA Clara™ is a full-stack GPU-accelerated healthcare framework accelerating the use of AI for medical research and is available on the NVIDIA NGC Catalog.

Azure Machine Learning

Azure Machine Learning (Azure ML) empowers developers, data scientists, machine learning engineers, and AI engineers to build, train, deploy, and manage machine learning models. It is an open platform with built-in support for open-source tools and frameworks, such PyTorch, SciKit Learn and, TensorFlow along with numerous Integrated Developer Environments (IDEs), supporting key languages like Python and R.

Azure ML abstracts the set-up, installation and configuration of the machine learning environment, saving you the hassle of infrastructure management by taking care of the underlying technicalities. This enables domain experts, such as healthcare researchers and developers, to build mission-critical AI solutions faster and easier. Whether the project requires image classification, object detection, speech analysis, or natural language processing. Azure ML streamlines AI-powered solution development.

Productivity is also boosted by accelerating your training jobs with Azure’s global infrastructure. Enhance your workflow by scaling out to multi-node compute clusters or scaling up to powerful GPU-enabled machines. Combining this with end-to-end AI lifecycle management through industry-leading MLOps means data science teams can collaborate better and get to production quicker. And, with more than 60 compliance certifications including FedRAMP High, HIPAA and DISA IL5, plus configurable security features, Azure ML allows you to create a trusted working environment.

NVIDIA NGC Catalog and Clara

A key component of the NVIDIA AI ecosystem is the NGC Catalog. It is a software hub of GPU-optimized AI, HPC, and data analytics software built to simplify and accelerate end-to-end workflows. With over 150 enterprise-grade containers, 100+ models, and industry-specific SDKs that can be deployed on-premise, cloud or at the edge, the NGC Catalog enables data scientists and developers to build best-in-class solutions, gather insights, and deliver business value faster. Every single asset in the NGC Catalog is validated for performance, quality, and security by NVIDIA, providing you with the confidence needed to deploy within the Azure ML environment.

The deep learning containers in the NGC Catalog are updated and fine-tuned monthly to enable maximum performance through software-driven optimizations that augment GPU hardware acceleration. These performance improvements are made to libraries and runtimes to extract maximum performance from NVIDIA GPUs. Pre-trained models from the NGC Catalog help speed up the application building process. You can find more than 100 pre-trained models across a wide array of applications such as image analysis, natural language processing, speech processing and recommendation systems. The models are curated and tuned to perform optimally on NVIDIA GPUs for maximum performance. By applying transfer learning, you can create custom models by retraining it against your own data.

NVIDIA Clara is one many artifacts available in the NGC Catalog. Clara is a healthcare framework that comprises of full-stack GPU-accelerated libraries, SDKs, and reference applications to create secure, and scalable healthcare applications.

The Clara family of application frameworks include:

Clara Imaging: Application frameworks for accelerating the development and deployment of AI based medical imaging workflows in radiology and pathology, as well as in some medical instruments
Clara Parabricks: Computational framework supporting genomic analysis in DNA and RNA
Clara Guardian: Application framework and partner ecosystem that simplifies the development and deployment of smart sensors within the hospital with multimodal AI

Clara Imaging, the focus of this blog, offers easy-to-use, domain-optimized tools to create high-quality, labeled datasets, collaborative techniques to train robust AI models, and end-to-end software for scalable and modular AI deployments. It consists of two essential elements – Clara Train and Clara Deploy:

Clara Train is a framework that includes two main libraries; AI-Assisted Annotation (AIAA), which enables medical viewers to rapidly create annotated datasets suitable for training, and a Training Framework, a TensorFlow based framework to kick start AI development with techniques like transfer learning, federated learning, and AutoML.
Clara Deploy provides a container-based development and deployment framework for multi-AI, multi-domain workflows in smart hospitals for imaging, genomics, and signal processing workloads. It leverages Kubernetes to enable developers and data scientists to define a multi-staged container-based pipeline.

Figure 2. The entire Clara pipeline that includes Clara train and deploy

The entire Clara Framework can be easily accessed from the Clara portfolio page. In addition to Clara Train and Deploy, reference models and pipelines are also available for download. Applying transfer learning, developers can create new models with their own custom dataset.

Figure 3. Various types of pre-trained models that cover both 2D and 3D segmentation and classification for different use cases

NGC-AzureML Quick Launch Toolkit

A ready-to-use Jupyter notebook was created to showcase the fine-tuning of a pre-trained COVID-19 CT Scan Classification model from the NGC Catalog.

To help automate the deployment, we have developed the NGC-AzureML Quick Launch toolkit that leverages the Azure ML SDK and creates the necessary compute and software resources needed to run Machine Learning applications. The Azure ML SDK uses Azure Machine Learning Compute Clusters, which require their own quota, and can be created using the same mechanism as the process followed to setup a quota for Azure VMs.

Azure Machine Learning Workspace is a prerequisite and a video on setting up the Azure Machine Learning workspace can be found at Setting up Azure Machine Learning Workspace. The video that walks through on how to install the AzureML-NGC toolkit described in this section can be found at Installing the AzureML-NGC Toolkit

The toolkit can be used to launch any asset from the NGC Catalog, by just changing a config file. In our example, the toolkit takes relevant assets for Clara, but you can customize it to pull assets related to other uses cases such as computer vision, natural language understanding, and more.

The toolkit automates the steps outlined below:

An Azure instance with NVIDIA GPUs is configured with the right NVIDIA libraries and drivers
Setting the desired NGC Catalog image (Clara Train SDK in this example) onto the Azure instance
Uploading additional material: model(s) (the pre-trained COVID-19 CT Scan Classifier in this case), auxiliary code and the corresponding datasets from the NGC Catalog to the Azure instance
Loading the ready-to-use Jupyter notebook from the NGC Catalog that contains the application
Installing JupyterLab the Azure instance and accessible locally to run the ready-to-use Jupyter notebook

To set up the AzureML environment, you only need to run two commands in the command line interface (CLI):

Install azureml-ngc-tools

First, install the NGC-AzureML Quick Launch Toolkit on the local machine, via Pip:

pip install azureml-ngc-tools

Configure azure_config.json

This file contains the Azure credentials and desired choice of instance type. A ready-to-use template for this example to complete can be downloaded here, which should be edited with user credentials.

An example of how the azure_config.json file might look is the following:

{ "azureml_user": { "subscription_id": "ab221ca4-f098-XXXXXXXXX-5073b3851e68", "resource_group": "TutorialTestA", "workspace_name": "TutorialTestA1", "telemetry_opt_out": true }, "aml_compute" { "ct_name":"clara-ct", "exp_name":"clara-exp", "vm_name":"Standard_NC12s_v3", "admin_name": "clara", "min_nodes":0, "max_nodes":1, "vm_priority": "dedicated", "idle_seconds_before_scaledown":300, "python_interpreter":"/usr/bin/python", "conda_packages":["matplotlib","jupyterlab"], "environment_name":"clara_env", "docker_enabled":true, "user_managed_dependencies":true, "jupyter_port":9000 } }

When the above file is used with the azureml-ngc-tools command, an Azure Machine Learning Compute Cluster named “clara-ct” is created using a node from the “Standard_NC12s_v3” VM size. Follow this link to learn more about the other specifications.

This file lists references to all the NGC Catalog assets that are to be pre-installed in the Azure ML environment to achieve the specific use-case. In this example, the additional resources are the ready-to-use Jupyter notebook and the associated data files. A ready-to-use config file lists the various assets needed for the application below, can be downloaded here. No additional modifications are required for this file.

The ngc_config.json file looks like:

{ "base_dockerfile":"nvcr.io/nvidia/clara-train-sdk:v3.0", "additional_content": { "download_content": true, "unzip_content": true, "upload_content": true, "list":[ { "url":"https://api.ngc.nvidia.com/v2/models/nvidia/med/clara_train_covid19_3d_ct_classification/versions/1/zip", "filename":"clara_train_covid19_3d_ct_classification_1.zip", "localdirectory":"clara/experiments/covid19_3d_ct_classification-v2", "computedirectory":"clara/experiments/covid19_3d_ct_classification-v2", "zipped":true }, { "url":"https://api.ngc.nvidia.com/v2/resources/nvidia/med/getting_started/versions/1/zip", "filename":"clarasdk.zip", "localdirectory":"clara", "computedirectory":"clara", "zipped":true }, { "url":"https://api.ngc.nvidia.com/v2/resources/nvidia/azuremlclarablogquicklaunch/versions/example/zip", "filename":"claractscanexample.zip", "localdirectory":"clara/claractscanexample", "computedirectory":"clara", "zipped":true } ] } }

When used with the azureml-ngc-tools command, the “nvcr.io/nvidia/clara-train-sdk:v3.0” container is pulled from the NGC Catalog and loaded onto the Azure ML Compute Cluster. Additionally, three resources are downloaded and unzipped into the local environment (with the names "filename" and at the relative directories "localdirectory" provided) and then loaded into the Compute Cluster at the provided location ("computedirectory").

Run azureml-ngc-tools

Once the two configuration files are ready, run the azureml-ngc-tools command on the local machine to provision the instance:

azureml-ngc-tools --login azure_config.json --app ngc_config.json

Refer to the following screenshot:

Figure 4. AzureML instance being setup by NGC-AzureML Quick Launch Toolkit

The command creates a Compute Cluster "clara-ct" using vmSize :"Standard_NC12s_v3,” then it creates AzureML environment, "clara_env,” with base image “nvcr.io/nvidia/clara-train-sdk:v3.0.” The container is downloaded and subsequently cached after the first use, making it easier for you to reuse the container the next time.

Next, the additional NGC Catalog content is downloaded and unzipped locally, which is then uploaded to the Compute Cluster. The user should be able to see the newly created Compute Cluster, "clara-ct", in the Azure Portal under the specified workspace, "TutorialTestA1" in this example and tabs Compute ->Compute clusters. This should look like the following:

Figure 5. Console view of AzureML Environment Setup by NGC-AzureML Quick Launch Toolkit

Additional Clara introductory content such as examples of other pre-trained model, datasets and Jupyter notebooks are also downloaded and unzipped locally at the relative directories provided in the “ngc_config.json” file. That content is also uploaded to the Compute Cluster, visible as follows:

Figure 6. Assets from the NGC Catalog pre-loaded onto Azure ML by NGC-AzureML Quick Launch toolkit

The command creates and launches JupyterLab in the Compute Cluster and forwards the port to the local machine so that the Jupyter Lab is accessed on your local machine. A URL is generated that contains the link to JupyterLab running on the Azure ML instance setup, along with the assets specified in the “ngc_app.json” file to be used on your local machine.

Figure 7. Direct URL to fully setup and configured Azure ML JupyterLab for this particular use-case

Once the you have completed your work on the JupyterLab, the Compute Cluster can be stopped by simply entering CTRL+C on the terminal.

The JupyterLab can be launched by copying and pasting the URL into a browser window, as follows:

Figure 8. URL opened on any browser window launches ready-to-use Azure ML JupyterLab

Note that it starts at the workspace root folder for the provided workspace.

To access the content uploaded by the NGC-AzureML Quick Launch Toolkit, you should navigate to the “workspaceblobstore/clara/” folder. All your relevant content will be now available in the session:

Figure 9. Overview of NGC assets pre-loaded onto your AzureML JupyterLab for this use-case

Fine-Tuning a Model with Clara Imaging and Azure ML

A video that walks through on how fine tune a model described in this section can be found at Fine Tuning the pre-trained model. To demonstrate how to build and deploy AI for medical imaging using Clara Imaging and Azure ML, we will fine-tune a pre-trained COVID-19 CT Scan Classification model and optimize it for inference with a custom dataset. This pre-trained model was developed by NVIDIA Clara researchers in collaboration with the NIH, which had a repository of CT radiological images from around the world. The NVIDIA pre-trained model reports an accuracy of over 90% on classification of COVID vs. Non-COVID findings from chest CT scans. More information about the training methodology and results achieved by this pre-trained model are contained in a white paper here.

The model uses two inputs-- a CT scan image, and a lung segmentation image. A computerized tomography (CT) scan is a 3D medical imaging procedure that uses computer-processed combinations of many X-ray measurements taken from different angles to produce cross-sectional (tomographic) slices of images. The data needs to be preprocessed, first by converting to Hounsfield units and then rotated to a prescribed orientation, before it can be used for training. A multitude of other pre-trained models developed by NVIDIA Clara for various healthcare applications are made available for free in the NGC Catalog. (N.B. These models are provided for research purposes only.)

The following images show one slice of the stack of images that comprise of the patient’s CT and the corresponding mask:

Figure 10. Example CT-Scan Slice with the corresponding mask (on the right) to focus the model on the lungs

The classifier produces a probability between zero and one, indicating whether the patient has or has not been infected with COVID-19. However, such predictions require a high degree of accuracy and in most cases, the CT scans used for training may have different characteristics from the original data set used to build the base model. Thus, a model will need to be fine-tuned to achieve the necessary accuracy. NVIDIA Clara includes a fine-tuning mechanism to efficiently adapt the pre-trained model to your dataset, with minimal effort, while increasing the accuracy of your model.

The hospital data for the blog is simulated using 40 labelled data from two sources:

CT-scans from COVID patients on the COVID-19 CT scans Kaggle Database
CT-scans from non-COVID CT-scans from The Cancer Imaging Archive.

Running the CovidCT-ScanClassifier.ipynb Jupyter Notebook

Figure 11. The Jupyter notebook for this blog example walks through these steps

Step 1: Set Up

The ready-to-use Jupyter notebook first describes the mechanics of how the pre-trained model is built, and how it can be fine-tuned by introducing the concept of a MMAR (Medical Model ARchive). The dataset from Kaggle is downloaded (by previously obtaining the user’s Kaggle key) and the input examples are examined and visualized. The new data is then indexed in a way that Clara can refer to it for different tasks, such as inferring if the patient has COVID-19 or not, fine tuning, and/or deploying the model. Refer to the CovidCT-ScanClassifier.ipynb Jupyter notebook to see how all these tasks are done with more details.

The data set that we are using to fine-tune the model consists of 40 samples studies. To test the current accuracy of the model, we’ll use three data points from the set. We’ll exclude these three data points when we eventually fine-tune the model.

The Jupyter notebook separates those example as seen here:

Figure 12. Slicing the reference dataset for training and testing

Step 2 Classify/Infer on Test Data

Use the infer.sh command to check the accuracy of the model by using the test data.

The MMAR from the base model has the original configuration files used to train the base model with. Those files need to be adapted to the new data before the infer.sh command is run. The Jupyter notebook will execute all the modifications needed to the required files.

Once the configuration files have been adapted, the infer command is run on the test data.

Inspecting the inference results

The Jupyter Notebook then retrieves the probabilities produced by the “new_infer.sh” command and estimates the predicted labels (1:COVID, 0:NO COVID). Those labels are then used along the true labels to compute the testing examples average precision score using the sklearn.metrics.average_precision_score function.

Figure 13. Testing the pre-trained model for performance without fine-tuning on the new reference dataset

Notice that the average precision (0.833) is not as high as expected (.90 or 90% or higher) because some instances of the new data have either not been preprocessed in Hounsfield units or haven’t be rotated to a specific orientation. We can now improve the accuracy of the model by fine-tuning it with the full data set.

Step 3 Fine Tune

The train_finetune.sh command executes the fine-tuning mechanism where the original script and its configuration files are adapted to point to the new training data.

Execute new_train_finetune.sh Command

Figure 14. Fine-tuning the Clara COVID-19 CT Scan Classification pre-trained model with training set of reference dataset

Once the training is complete, you can view the training accuracy, training loss, mean accuracy, and the time it took for the fine-tuning to finish. You can decide to either export the model or continue fine-tuning the model. By default, the resulting model is not automatically saved nor exported. The trained model can be exported using the export.sh command. This produces the frozen graphs necessary for the model to be used by other applications, such as Clara Deploy. The Jupyter notebook sets up the new model so that it can be used to classify the test data.

Step 4: Re-test Reclassifying Inference with fine-tuned model

The Jupyter notebook creates a new infer.sh command that points to the full data set and to the new fine-tuned model and then executes the command.

The next step is to compute the Average COVID-19 Classification Precision across all testing examples. The Jupyter Notebook then retrieves the probabilities produced by the “finetuned_infer.sh” command and estimates the predicted labels with 1 (high likelihood of infection) or 0 (low likelihood of infection).

Those labels are then used along the true labels to compute the testing examples’ average precision score using the sklearn.metrics.average_precision_score function.

Figure 15. Testing the fine-tuned model for performance on the testing slice of data from the new reference dataset

Notice that the average precision is now much higher, so the fine-tuning mechanism has succeeded to fine tune the model to account for the peculiarities of the new data.

Deploying the Model

After the AI model is exported and checked, the Medical Model Archive MMAR can then be connected into research workflow pipelines. These pipelines contain operators for the various phases of pre-transforms and inference. The results are consumable by a medical imaging ecosystem. (e.g. a DICOM-SR, a secondary capture image with burnt-in results, an HL7 or FHIR message) The imaging pipeline can be constructed using Clara Deploy SDK, which uses building blocks called operators. Clara Deploy SDK comes with many ready-made pipelines for getting started.

Understanding and mapping the architecture in any given environment is important to know when constructing this pipeline. For example, simulating the connectivity from an image manager like a PACS by using transformation frameworks to transform the data for the model, is a good first step. Subsequent to the first step, inference tasks follow, and then delivering the outputs in the format accepted by systems and devices at medical institutions.

Figure 16. A sample deployment workflow using the Clara Deploy application framework

When thinking about the results, it is important to consider the types of data being produced and the systems that it will be deployed on. For instance, is it a classification result (presence or absence of a disease, like COVID-19), a DICOM-SR for display within a PACS, or a FHIR Observation object. If it is a segmentation result (identifying a lesion or nodule), creating a DICOM Segmentation object may be appropriate. These are examples of the types of objects consumable by the medical imaging ecosystem and the architecture of the environment is important to know when constructing this pipeline.

Summary

We have shown you how you get started with Clara Train on AzureML for radiological CT images, using a pre-trained AI COVID-19 classification model. This is one example that can be built with the software from the NGC Catalog and AzureML. Beyond just building healthcare centric application, you can use the containers, models, and SDKs from the NGC Catalog to build applications across other use cases, such as conversational AI, recommendation systems, and many more.

Get started today with NVIDIA Clara from the NVIDIA NGC Catalog on Azure ML.

Dealing with Imbalanced Data in AutoML

Arjun-Singh — Tue, 01 Sep 2020 16:00:00 GMT

What is Imbalanced Data?

For a given classification problem, if the classes/targets within the dataset are not represented equally, then the dataset is said to be imbalanced. The classes with a higher representation are called majority classes, while the ones with lower representation are called minority classes. Sometimes imbalance exists due to limited availability of data for certain classes, while in other cases it could be attributed to the nature of the dataset. For instance, if there is a spam classification dataset, we’d expect more non-spam emails than spam emails. Similarly, for fraud detection, we would expect the fraudulent transactions to occur only rarely.

It is harder for learning models to generate predictions for the minority classes because the model has access to limited training examples representing those classes, making it difficult for it to learn to distinguish these classes. Most ML algorithms for classification perform well when all classes have a roughly equal distribution and hence it becomes important to address the problem of class imbalance.

How do we detect class imbalance?

While there is no fixed definition of what constitutes an imbalance, generally it is detected using one of the following:

Ratio of the samples in the least populated class to the overall number of samples in the dataset. Let’s call this Ratio-1.
Ratio of the samples in the least populated class to the samples in the most populated class. Let’s call this Ratio-2.

Let’s evaluate both methods using a couple of sample cases:

For a four-class problem with data distributed among labels like this: {'a': 20, 'b': 20, 'c': 20, 'd': 200}, Ratio-1 is 7.7%, while Ratio-2 is 10%.
For a four-class problem with data distributed among labels like this: {'a': 20, 'b': 200, 'c': 200, 'd': 200}, Ratio-1 is 3.2%, while Ratio-2 is 10%.

As seen in the above examples, Ratio-2 remains consistent, even if the constitution of the other classes in the overall data changes.

Hence, we utilize Ratio-2 as the indicator of imbalance in AutoML and imbalance is detected when this ratio is lower than 20%.

How to address the problem of Imbalance?

Ideally one would prefer collecting more data for the minority class(es). However, often that is quite expensive or may not be feasible due to the nature of the problem (e.g., fraud or spam detection).

Therefore, we leverage the following methods for dealing with imbalanced data within AutoML:

Using weights for class balancing: this feature gets automatically applied in AutoML if it improves performance on a subset of the user’s data (more details in later sections)
Using metrics that are sensitive to imbalance: users can pick relevant metrics based on our recommendations.
Leveraging the sample weights provided directly by the user via the AutoML config

Additionally, users can apply over/under sampling to rebalance the data before feeding it to AutoML. Methods #1 and #3 will impact the ML algorithm’s cost function in the same manner (covered in the next section), but the difference is that in method #1 AutoML will automatically do it for you. This feature of Class Balancing using weights is the focus of this blog and we shall elucidate that next.

Weights for Class Balancing:

Without actually over-sampling the minority classes or under-sampling the majority classes, we can simply apply weights to the samples belonging to a class, in the inverse proportion of the number of samples representing that class (Fig 1 elaborates on this calculation). The intent is that the underrepresented classes would have a higher weight than the overrepresented ones.

We leverage this method because it allows us to apply balancing without increasing the size of the dataset, and hence without modifying memory requirements. Additionally, even if the classes are well balanced, applying this method would simply apply uniform weights, and in theory this method could be applied for all datasets.

To understand the impact of applying weights, let’s review the cost function J(θ) for a Logistic Regression classifier below. Here m is the number of training samples, x and y are the features and labels respectively, θ refers to the model parameters.

When we use the Weights for Class Balancing, the above cost function is modified to apply the class weight corresponding to every training sample, as shown below, where c(i) refers to the class weight for the i^thtraining sample.

Using metrics sensitive to imbalance:

For dealing with class imbalance we recommend using AUC-weighted as the optimization metric for AutoML runs. There are several benefits, as follows:

Any macro average will independently calculate the average for every class and then take the overall average, thus treating all classes equally; whereas a weighted average will calculate the contribution of every class based on the relative number of samples representing that class, and hence this is more robust to imbalance. Hence, we choose a weighted metric.
Additionally, AUC-weighted is threshold invariant since it measures the area under the (ROC) curve over all possible classification thresholds by aggregating them.
We also report several other metrics sensitive to class imbalance, such as F1, Precision, Recall and Matthew’s Correlation Coefficient.

Analysis and Experimentation:

After selecting the solution and the metric, we performed A/B tests on a variety of datasets with the following setup:

Treatment Group: using weights for treating class imbalance
Control Group: with no class balancing
Significance Level (Type 1 Error probability) preselected as 5%
H₀(Null Hypothesis): There is no impact on the performance of AutoML after applying weight-balancing
Ha (Alternative Hypothesis): Weight balancing improves the performance of AutoML
One tailed t-tests were performed by leveraging the sample statistics like sample standard deviation from multiple runs performed on the same datasets

Extensive A/B testing with this setup demonstrated that we could leverage subsampled data reliably to determine the effectiveness of our class balancing solution. Running A/B tests on subsampled data rather than the full data enabled us to reduce the overall execution time. These tests utilize sample statistics computed using binomial distribution and some heuristics. This helps us decide if balancing the classes using weights would improve the performance and the complete flow is described in the graphic below.

Additionally, AutoML’s Guardrails feature informs the user if their data exhibits imbalance and if the class balancing solution was applied.

Figure 1: Flow chart describing the handling of imbalanced data within AutoML

If you’re getting started today with Microsoft Azure’s Automated Machine Learning, here are a couple of helpful links:

https://azure.microsoft.com/en-us/services/machine-learning/automatedml/

https://docs.microsoft.com/en-us/azure/machine-learning/concept-automated-ml#how-automated-ml-works

Contributors:

Arjun Singh – Data & Applied Scientist II

Anup Shirgaonkar – Principal Data & Applied Scientist

Improve remote learning with speech-enabled apps powered by Azure Cognitive Services

Qinying Liao — Fri, 28 Aug 2020 08:20:30 GMT

Improve remote learning with speech-enabled apps powered by Azure Cognitive Services

This post was co-authored by Melissa Ma, Yueying Liu, Anny Dow and Sheng Zhao

Online learning has grown rapidly over the last couple of months as schools and organizations adapt to new ways of connecting and methods of education. Speech technology can play a significant role in making distance learning more engaging and accessible to students of all backgrounds. With Azure Cognitive Services, developers can quickly add speech capabilities to applications, bringing online learning to life.

Enhancing language fluency with pronunciation assessment

One key element in language learning is improving pronunciation skills. For new language learners, practicing pronunciation and getting timely feedback is essential to becoming a more fluent speaker. In the current environment, online language learning and the ability to practice anytime, anywhere, has become even more important.

At the Build conference in May, we announced the preview of the pronunciation assessment capability, powered by Speech to Text.

The pronunciation assessment capability evaluates speech pronunciation and gives speakers feedback on the accuracy and fluency of spoken audio, allowing users to benefit from:

Highly accurate evaluations - Provides consistent and accurate evaluation results using a machine learning-based approach that correlates highly with speech assessments conducted by native experts. The pronunciation assessment model was trained with 100,000+ hours of speech data from native English speakers and is highly robust. It assesses three dimensions of pronunciation: accuracy, fluency and completeness. Pronunciation assessment can provide evaluations at multiple levels of granularity, returning accuracy scores for specific phonemes, words, sentences, or even whole articles.
Ability to account for inserted and omitted words – Enables rich configuration parameters to support flexibility in using the API. Using NLP techniques and EnableMiscue setting, pronunciation assessment can detect errors such as extra, missing, or repeated words—when compared to reference text—to assist in more accurate scoring. This is particularly useful for longer paragraphs of text.
Real-time streaming - Supports streaming upload on audio files for immediate feedback.

With pronunciation assessment, language learners can practice, get instant feedback, and improve their pronunciation. Online learning solution providers or educators can use the capability to evaluate pronunciation of multiple speakers in real-time. Pronunciation assessment currently supports the English language.

Educational organizations, like the Tomorrow Advancing Life (TAL) Education Group, are already building applications using pronunciation assessment to help students practice language learning remotely.

“Effectively and efficiently teaching accurate pronunciation to students of different levels is a big challenge, both in class and outside of class. The Speech service’s pronunciation assessment capability provides a powerful solution to address this challenge. We’ve been highly impressed by the robustness of pronunciation assessment and its ability to deal with noisy environments, and how well it correlates with pronunciation evaluations conducted by our teachers.”

- Xiangyu Hu, AI Scientist of Tomorrow Advancing Life (TAL) Education Group

Learn how you can get started with the pronunciation assessment using our tutorial video and download source code from Github to try out.

Developing interactive courses with Text to Speech

Another way that Speech technology can support better online learning experiences is through Text to Speech, a Speech service feature that converts text to lifelike speech. Educators can create interactive materials with highly expressive and humanlike voices using Neural Text to Speech (Neural TTS), now available in 36 voices with 31 languages. (Learn about our most recent languages here.)

With Neural TTS, developers can add natural-sounding voice to learning materials, for scenarios like slide narration. Neural TTS can also be used for reading aloud any content, facilitating new ways for students to interact with material as well as increasing accessibility for students with learning differences. Educational organizations can also use Neural TTS to create AI-powered virtual “teachers” that interact with students to make online courses more engaging.

Experience the Neural Voices with the new Edge browser

With the Custom Neural Voice capability, online learning solution providers can further create interactive learning experiences for their students in a voice that represents their brand, or develop unique voices for different characters. For example, Duolingo, one of the world’s most popular language learning apps, is creating unique voices for different characters used in the lessons.

Using SSML or the Audio Content Creation tool, users can further finetune audio characteristics like speaking rate, pitch, and pronunciation to fit their scenarios—no code required. Neural TTS also supports different speaking styles—like cheerfulness and empathy—making it easier to bring audiobooks to life. Recently we have just added 10 new voice styles, available in Chinese (Xiaoxiao voice) and will be expanded to other languages. With these new styles, online education solution providers can create more engaging interactive courses that express rich emotions.

To learn more about Audio Content Creation, watch the video tutorial.

To learn more and get started adding speech to your educational applications, check out our resources below:

Pronunciation Assessment

Try out our demo
Learn more with our documentation
Check out easy-to-deploy samples
Watch the video introduction and video tutorial

Text to Speech

Check out our demo
Learn more with our documentation
Follow the QuickStart
Learn more about responsible deployment of Custom Neural Voice
Video tutorial for the Audio Content Creation tool

How Language Understanding enables voice commands for Dictate in Word

AliciaEP — Tue, 25 Aug 2020 21:17:33 GMT

People are interacting with technology using voice input more than ever before. Dictation in productivity apps such as Word empower people to conquer the blank page. Dictating is a fast and easy way to get your thoughts on the page during brainstorming, outlining, and authoring content.

Today, Office announced voice commanding on web and mobile as the next step to make dictation more powerful:

Prior to having voice commands, dictation only allowed people to add basic text with speech-to-text and punctuation with phrases like “new line” or “period”. Now, without leaving dictation or having to switch to keyboard and mouse, you can seamlessly use voice commands to accomplish tasks such as correcting text, light formatting, and making lists using natural language. Phrases like “delete that”, “bold last word”, “start list” are now among many new commands and more will be added regularly.

When the Office dictation team was looking for a good way to understand the natural spoken language commands, they reached out to the Azure Language Understanding team. Following that, an exciting new collaboration formed. Language Understanding in Azure Cognitive Services enables the team to build custom natural language models that interpret a users’ natural speech instead of using a strict command language- all without having to be a machine learning expert. With Language Understanding, you can create custom language models with UI-experience with confidence.

This is just one of the many examples of how Microsoft products are powered by cutting edge AI technology in Azure AI. For example, suggested replies in Outlook and PowerPoint Designer each rely on Azure Machine Learning to make recommendations on quick replies and recommended design layouts, respectively. Additionally, Microsoft Teams uses Speech for live captioning of meetings, while Xbox uses Personalizer to make personalized recommendations for Xbox gamers.

Any developer can extend apps with rich AI-powered capabilities irrespective of machine learning expertise. Azure Cognitive Services offers pre-built AI models and simple customization capabilities that help you infuse capabilities of vision, speech, language and decision.

Train and Score Hundreds of Thousands of Models in Parallel

Sam_Istephan — Wed, 05 Aug 2020 16:33:23 GMT

Abstract

With the Azure Machine Learning service, the training and scoring of hundreds of thousands of models with large amounts of data can be completed efficiently leveraging pipelines where certain steps like model training and model scoring run in parallel on large scale out compute clusters. In order to help organizations get a head start on building such pipelines, the Many Models Solution Accelerator has been created. The Many Models Solution Accelerator provides two primary examples, one using custom machine learning and the other using AutoML. Give it a try today!

Executive Overview

Many executives are looking to Machine Learning to improve their business. With the reliance on a more digital world, the amount of data generated is increasing faster than ever. Further, companies are purchasing 3^rd party datasets to combine with internal data to gain further insight and make better predictions. In order to make better predictions, sophisticated machine learning models are being built that leverages this large pool of data. Further, as companies are expanding to do business in a variety of markets and environments, general machine learning model no longer suffice, instead, more specific machine learning models are needed. In this case, "general machine learning models" refers to the granularity for which this model was built, for example, building a demand forecast model for a product at the country level versus at the city level which would be a "specific machine learning model." Building more specific machine learning models can easily results in building hundreds of thousands of specific models instead of a handful of general models. Combining large datasets with the need of building hundreds of thousands of more specific machine learning models is not a trivial task. Doing so requires very large compute power. In fact, this task can greatly benefit from parallel compute power where multiple compute instances are working simultaneously to build the machine learning models in parallel. Once those models are trained, leveraging them to score large amounts of data presents the same problem characteristics which again, leveraging a compute cluster where multiple compute instances are making predictions using the machine learning models simultaneously can greatly reduce the time required to do so. With the Azure Machine Learning service, the training and scoring of hundreds of thousands of models with large amounts of data can be completed efficiently leveraging pipelines where certain steps like model training and model scoring run in parallel on large scale out compute clusters. In order to help organizations get a head start on building such pipelines, the Many Models Solution Accelerator has been created. The Many Models Solution Accelerator provides two primary examples, one using custom machine learning and the other using AutoML.

In Azure Machine Learning, AutoML automates the building of the most common categories of Machine Learning models in a very robust and sophisticated manner. For example, a very common machine learning problem is demand forecasting. Having a more accurate demand forecast can increase revenues and reduce waste. Traditionally, many statistical methods have been used to do just this. However, more modern techniques leverage Machine Learning including Deep Learning techniques to provide a more accurate demand forecast. Further, the demand forecast can be improved by moving from forecasting a broader scope (general machine learning model) to forecasting a more granular scope (specific machine learning model). Doing so means, for example, instead of building one forecast for each Product at the country level, building a forecast for each product at the city. Moving to more specific models results in building hundreds of thousands of forecasts using large amounts of data which as discussed above can be solved using the Many Models Solution Accelerator.

Technical Overview

As data scientist move from building a handful of general machine learning models to hundreds of thousands of more specific machine learning models (i.e. geography or product scope), the need to perform the model training and model scoring tasks require parallel compute power to finish in a timely manner. In the Azure Machine Learning service SDK, this is accomplished using Pipelines and specifically a ParallelRunStep which runs on a multi node Compute Clusters. The data scientist provides the ParallelRunStep a custom script, an input dataset, a compute cluster and the amount of parallelism they would like. This concept can be applied to a custom python script and to Automated Machine Learning (AutoML).

Automated Machine Learning (AutoML) uses over ten algorithms (including deep learning algorithms) with varying hyperparameters to build Classification, Regression and Forecasting models. Further, Pipelines automate the invocation of AutoML across multiple nodes using ParallelRunStep to train the models in parallel as well as to batch score new data. Pipelines can be scheduled to run within Azure Machine Learning or invoked using their REST endpoint from various Azure services (i.e. Azure Data Factory, Azure DevOps, Azure Functions, Azure Logic Apps, etc). When invoked, the Parallel Pipelines run on Compute Clusters within Azure Machine Learning. The Compute clusters can be scaled up and out to perform the training and scoring. Each node in a compute clusters can be have Terabytes of RAM, over a 100 cores, and multiple GPUs. Finally, the scored data can be stored in an a datastore in Azure, such as Azure DataLake Gen 2, and then copied to a specific location for an application to consume the results.

In order to provide a jump start in leveraging Pipelines with the new ParallelRunStep, the Many Models Solution Accelerator has been created. This solution accelerator showcases both a custom python script and an AutoML script.

Major Components

The main components of the Many Models Solution Accelerator includes an Azure Machine Learning Workspace, a Pipeline, a ParallelRunStep, a Compute Target, a Datastore, and a Python Script File as depicted in Figure 1, below.

Figure 1. The architecture of a Pipeline with a ParallelRunStep

For an overview of getting started with Azure Machine Learning service, please see the blog article MLonAzure: Getting Started with Azure Machine Learning service for the Data Scientist

For an overview of Pipelines, please see blog article, MLonAzure: Azure Machine Learning service Pipelines

Major Steps

1. Prerequisites

An Azure Subscription along with an Azure Machine Learning Workspace are needed to get started. To get started with Azure Machine Learning, see Getting Started with Azure Machine Learning services
Within the Azure Machine Learning, a Compute Instance needs to be created to serve as the Data Scientist’s workstation. Using the Compute Instance, clone the Many Models Solution Accelerator Github repository.

2. Data Prep

Data needs to be split into multiple files (.csv or .parquet) for each group that a model is to be created for. Each file must contain one or more entire time series for the give group.
The data must be placed in Azure Storage (e.g. ADL Gen 2, Blob Storage). The storage will then be registered as a Datastore from which two FileDatasets will be registered, one pointing to the folder containing the training data and the other to the folder containing the data to be scored.
For example, to build a forecast model for each brand within a store, the training sales data would be split to create files StoreXXXX_BrandXXXX.

3. Model Training

The solution accelerator showcases model training with a custom python script and with AutoML which are orchestrated using a Pipeline. Please see solution-accelerator-manymodels-customscript and solution-accelerator-manymodels-AutoML. Putting it all together results in the architecture depicted in Figure 2, below.

Figure 2: Solution Accelerator Model Training

3. a. Pipeline

The solution accelerator leverages the Pipeline object to train the model. Specifically, a ParallelRunStep is used which requires a configuration parameters, ParallelRunConfig

ParallelRunConfig has many parameters, below are the typical ones used for the Many Models Solution Accelerator. For a complete list of ParallelRunConfig parameters, please see the ParallelRunConfig Class.

Parameter	Explanation
environment	Provides the configurations for the Python Environment
Entry_script	This is a Python file (.py extension only) that will run in parallel. Note that the Many Models Solution Accelerator contains a custom Entry_script that leverages AutoML and one that leverages custom code.
Compute_target	The AML ComputeCluster to run the step on.
Node_count	The number of nodes to use within the training cluster. Scale this number to a higher number to increase parallelism.
Process_count_per_node	The number of cores that will be used within each node
mini_batch_size	For FileDatasets it’s the number of files that are used at a time, for Tabular Datasets it’s the number data size that will be processed at a given time.
Run_inovcation_timeout	The overall allowed time for the parallel run step

3. b. Training script with AutoML

The solution accelerator showcases using AutoML Forecasting. AutoML has many parameters, below are the typical ones used for doing a Forecasting task within the Many Models Solution Accelerator. For a complete list of AutoML Config parameters, please see the AutoMLConfig Class.

automl_settings = { "task" : 'forecasting', "primary_metric" : 'normalized_root_mean_squared_error', "iteration_timeout_minutes" : 10, "iterations" : 15, "experiment_timeout_hours" : 1, "label_column_name" : 'Quantity', "n_cross_validations" : 3, "verbosity" : logging.INFO, "debug_log": 'DebugFileName.txt', "time_column_name": 'WeekStarting', "max_horizon" : 6, "group_column_names": ['Store', 'Brand'], "grain_column_names": ['Store', 'Brand'] }

Parameter	Explanation
task	The type of AutoML Task: Classification, Regression, Forecasting
primary_metric	The metric that AutoML should optimize based on
Iteration_timeout_minutes	How long each of the number of “iterations” can run for
Iterations	Number of models that should be tried (combinations of various Algorithms + various Hyperparameters)
Experiment_timeout_hours	How long the overall AutoML Experiment can take. Note: The experiment may might timeout before all iterations are complete.
label_column_name	The column that is being predicted
n_cross_validations	Number of cross validations that should take place within the training dataset
verbosity	Log details
debug_log	Location for the debug log
time_column_name	The name of the Time column, note that the training dataset can have multiple time series
max_horizon	How far how the forecast will go
group_column_names	The names of columns used to group your models. For timeseries, the groups must not split up individual time-series. That is, each group must contain one or more whole time-series.
grain_column_names	The column names used to uniquely identify timeseries in data that has multiple rows with the same timestamp.

4. Model Forecasting

The solution accelerator showcases model forecasting with a custom python script and with AutoML which are orchestrated using a Pipeline. Please see solution-accelerator-manymodels-customscript and solution-accelerator-manymodels-AutoML. Putting it all together results in the architecture depicted in Figure 3, below.

Figure 3: Solution Accelerator Model Scoring

5. Automation

In order to automate the solution, the training and scoring pipelines must be published and a PipelineEndPoint must be created. Once that’s done, the PipelineEndpoint can then be invoked from Azure Data Factory. Specifically, the Azure Machine Learning Pipeline Activity is used. Note that the training and scoring pipelines can be collapsed into one pipelines if the training and scoring occur consecutively.

Next Steps

Azure Machine Learning Documentation

Many Models Solution Accelerator

Many Models Solution Accelerator Video

Azure Data Factory: Azure Machine Learning Pipeline Activity

MLOnAzure GitHub: Getting started with Pipelines

MLonAzure Blog: Getting Started with Azure Machine Learning for the Data Scientist

MLonAzure Blog: Azure Machine Learning service Pipelines

MLonAzureBlog: Azure Machine Learning Pipeline - ParallelRunStep

MLonAzureBlog: Video Walkthrough of ParallelRunStep

LinkedIn Group: Machine Learning on Azure

Re-ranking Cognitive Search results with Machine Learning for better search relevance

Luis Cabrera-Cordon — Wed, 29 Jul 2020 15:44:08 GMT

Are you looking for ways to fine-tune your model relevance? Sometimes developers create a customized ranking model to re-rank the results returned by Azure Cognitive Search. This allows them to use application-specific context as part of that model. To help facilitate this, Azure Cognitive Search is introducing a new query parameter called featuresMode. When this parameter is set, the response will contain information used to compute the search score of retrieved documents, which can be leveraged to train a re-ranking model using a Machine Learning approach.

We have created a new sample and tutorial that walks you through the learning to rank process end-to-end, with steps for designing, training, testing, and consuming a ranking model. The tutorial shows you how to extract features using the featuresMode parameter and train a ranking model to increase total search relevance as measured by the offline NDCG metric.

For customers who are less familiar with machine learning, a learn-to-rank method re-ranks top results based on a machine learning model. The re-ranking process can incorporate clickthrough data or domain expertise as a reflection of what is truly relevant to users. The is a visualization of the components of a learn-to-rank method used in the tutorial.

Legend	Description
Data	The articles and search statistics that reside in Azure Blob storage.
Search Index	Azure Cognitive Search ingests the data into a search index.
Re-ranker	Queries against the index produce scores and scoring features that are used to train a machine learning model based on labels derived from clickthrough data. After the model is trained, you can use it to re-rank your documents.
Judgement labels	To train the machine learning model, you need to have labeled data that contains signal for what documents are most relevant for different queries. One way to do this is to collect clickthrough data to understand which documents are most popular. Another mechanism may be to find human judges to label the most relevant documents.

The featuresMode parameter is currently in preview and can be accessed through the Azure Cognitive Search REST APIs.

Sample Request

POST https://[service name].search.windows.net/indexes/[index name]/docs/search?api-version=[api-version]

Content-Type: application/json

api-key: [admin or query key]

Request Body

{

"search": ".net core",

"featuresMode": "enabled",

"select": "title_en_us, description_en_us",

"searchFields": "body_en_us,description_en_us,title_en_us,apiNames,urlPath,searchTerms, keyPhrases_en_us",

"scoringStatistics": "global"
}

Sample Response

{

"value": [

{

"@search.score": document_score (if a text query was provided),

"@search.highlights": {

field_name: [ subset of text, ... ],

...

"@search.features": {

"field_name_1": {

"uniqueTokenMatches": 1.0,

"similarityScore": 0.29541412,

"termFrequency": 2

"field_name_2": {

"uniqueTokenMatches": 3.0,

"similarityScore": 1.75345345,

"termFrequency": 6

...

]
}

If you are interested in this new capability, contact us at azuresearchrelevance@microsoft.com

References

Search Ranking Tutorial Github
FeaturesMode REST API Reference

Azure Machine Learning studio - a web interface for managing the machine learning lifecycle

Tzvi Keisar — Wed, 15 Jul 2020 18:51:52 GMT

Machine learning is a complex and task heavy art, be it cleaning data, creating new models, deploying models, managing a model repository, or automating the entire CI/CD pipeline for machine learning.

As more companies embark on the journey of machine learning in everything they do, Microsoft Azure Machine Learning provides them with enterprise-grade capabilities to accelerate the machine learning lifecycle and empowers developers and data scientists of all skill levels to build, train, deploy, and manage models responsibly and at scale.

Azure Machine Learning studio is the web user interface of Azure Machine Learning, enabling data scientists to complete their end-to-end machine learning lifecycle, from cleaning and labeling data, to training and deploying models using cloud scalable compute, in a single enterprise-ready tool.

We are excited to announce that Azure Machine Learning studio is now generally available worldwide, supporting 18 languages and over 30 locales!

Azure Machine Learning studio caters to all skill levels, with authoring tools such as the automated machine learning user interface to train and deploy models in a click of a button, and the drag and drop designer to create ML pipelines using a visual interface. All resources and assets created during the ML process – notebooks, models, pipelines, are all available for team collaboration under one roof.

With this release, studio is even more comprehensive and easy to use

Notebooks: Intellisense, checkpoints, tabs, editing without compute, updated file operations, improved kernel reliability, and many more. Read more about Azure machine learning studio notebooks here.

Notebooks are integrated into Azure Machine Learning studio

Experimentation: Compare multiple runs graphically using an improved charting visualization experience including chart smoothing, displaying aggregated data and more.

Charts and metrics for tracking and analyzing runs

Security: Granular Role Based Access Controls (RBAC) are now supported (in preview) out of the box for the most common actions in your studio workspace. Specific actions or controls will now be hidden based on your role assignment automatically as setup by your IT Admins.

Compute: Compute instance has tons of improvements in quality, reliability, availability, provisioning latency, and user experience:

New enterprise readiness and administrator capabilities:

REST API and CLI support to help automate creation and management of compute instance
ARM template support for provisioning compute instance with sample template documented and downloadable from UI
Ability for admin to create compute instance on behalf of other users and assign to them through ARM template and REST API. Data scientists do not need to have create/delete RBAC permissions and can access Jupyter, JupyterLab, RStudio, use compute instance from integrated notebooks, and can start/stop/restart compute instances (this is in preview).
Validating user subnet NSG rules in virtual network for improved compute instance creation.
Encryption in transit using TLS 1.2

More information available in the updated compute creation panel

Designer (preview): Improved performance and reliability. Updates to user experience and new features:

New graph engine, with new-style modules. Modules have colored side bars to show the status and can be resized.
New asset library, to split Datasets, Modules, Models into 3 tabs
Output setting. Enable user to set module output datastores.
New modules:
Computer Vision: Support image dataset preprocessing, and train PyTorch models (ResNet/DenseNet), and score for image classification
Recommendation: Support Wide&Deep recommender

New style to Modules in the drag-and-drop Designer

Data Labeling: Create, manage, and monitor labeling projects directly inside the studio web experience. Coordinate data, labels, and team members to efficiently manage labeling tasks. Supports image classification, either multi-label or multi-class, and object identification with bounding boxes.

The machine learning assisted labeling feature (Preview) lets you trigger automatic machine learning models to accelerate the labeling task.

Learn more about Azure Machine Learning data labeling in this blog post.

Data labeling updated style and machine learning assisted labeling

Fairlearn (preview): Azure Machine Learning is used for managing the artifacts in your model training and deployment process.

With the new fairness capabilities, users can store and track their models’ fairness (disparity) insights in Azure Machine Learning studio, easily share their models’ fairness learnings among different stakeholders. Beyond logging fairness insights within Azure Machine Learning run history, users can load Fairlearn’s visualization dashboard in studio to interact with mitigated or original models’ predictions and fairness insights, select a pleasant model, and register/deploy the model for scoring time.

Fairlearn visualization now available as preview in the studio

Automated machine learning user interface (preview) Automated machine learning is the process of automating the time-consuming, iterative tasks of machine learning model development to enable non data scientists to operationalize their machine learning models.

The new Data Guardrails helps fix and alert users of potential data issues. The model details tab includes key information around the best model and the run. There is more control over which visualizations are generated - choose a metric of interest and visualizations pertaining to that metric will display.

Data guardrails in automated machine learning will alert for issues in the data and even fix some of them

Continuing the journey together

Our customers inspire us to continue the journey, building together experiences that make machine learning easier to use, productive, and fun!

Send us your feedback

Use the feedback panel to share your thoughts with us

Feedback panel to share your thoughts

Log in to Azure Machine Learning studio
Learn more about Azure Machine Learning service.
See the latest product announcements.

Using VS Code to enhance your machine learning experience

Sid_Unnithan — Wed, 08 Jul 2020 22:30:00 GMT

Hey AML community! The VS Code team is excited to present new capabilities we've added to the Azure Machine Learning (AML) extension. From version 0.6.12 onwards we've introduced UI changes and ways to help you manage Datastores, Datasets, and Compute instances all from directly within your favourite editor!

We're guessing many of you may be reading about this extension for the first time - don't worry, we're here to explain!

The extension is a companion tool to the AML service. It provides a guided experience to help you create and manage your AML resources from directly within VS Code. The extension aims to streamline tasks such as running experiments, creating compute targets, and managing environments, without requiring the context-switch from the editor to the browser. With an easy-to-navigate tree view you can work across all your workspaces and interact with your core AML assets using single-click commands.

If you'd like to learn more and experiment with the extension you can install it here and try the getting started docs here!

Datastore Integration

One of the new features we released is the support for Datastore registration. The extension currently supports Azure Blob Storage and Azure File Share datastore types. We've designed a set of streamlined input options to enable faster registrations, such as automatic retrieval of your Account Key credentials to authenticate against the storage account.

Dataset Integration

The extension also supports creating Tabular and File datasets from local files or web URLs.

Create a Tabular or File Dataset via the extension tree view

Once you've created a Tabular dataset, you can use the extension to preview your data from directly within the editor. In the case of parquet data, the extension may require a profile run before previewing

Preview tabular dataset and filter rows.

Via the extension, you can use your datasets during training without having to write extra AML SDK code. Right before submitting, you're shown a partial run configuration which abstracts the complexities of referencing your datasets through an estimator. In the configuration, you just need to input the script parameter and attach mechanism you want to use for File datasets, and the named input you'd like for Tabular datasets.

"datasets": { // file dataset input "mnist-ds": { "version": 1, "scriptParam": "--data-folder", "attachMechanism": "Mount" }, // tabular dataset input "titanic-ds": { "version": 1, "namedInput": "titanic_ds" } }

Compute Instance Integration

Creating and managing compute instances has never been easier! You can view all your workspace's compute instances and start/stop/restart them through commands in the tree. With a small number of clicks, you can create an SSH-enabled compute instance from directly within VS Code. Upon creating an SSH-enabled compute instance, you can follow our in-editor documentation to easily connect to your compute via the VS Code Remote SSH extension.

Manage compute instances and connect to them via SSH

UI Changes

Something we've been hearing for a long time is how the extension UI differs from the Azure ML Studio. In the previous photos you may have already noticed the highly consistent design in the extension tree view. We've updated each node with Studio-equivalent icons and have renamed/reordered them where appropriate.

Feedback

As mentioned throughout the blog post, many of the newly released features are in their preliminary phases and we're actively working to support a broader set of scenarios that are consistent with the Azure ML Studio and SDK experiences. Here are some of the scenarios we're actively working on:

Running your Notebooks in VS Code directly on an AML compute instance.
Building and working in Docker containers from an AML environment.
Creating datasets from an existing blob or file-based datastore.
Using AML environments when deploying an endpoint.

If there's anything that you would like us to prioritize, please feel free to let us know on Github!

If you're an existing user of the extension and would like to provide feedback, please feel free to do so via our survey.

Accelerate extraction of text, data and structure from your documents with Form Recognizer

pracjain — Thu, 09 Jul 2020 16:20:11 GMT

This blog has been authored by Neta Haiby (Principal PM, Azure AI) and Prachi Jain (PMM, Azure AI)

Documents are prevalent and often contain vital information that are essential to drive business outcomes; however, extracting data quickly and accurately for processing is often a challenge for so many organizations. Manual extraction can take up long processing cycles, cause errors and inefficiencies. Hence, extracting text and structure from documents with Form Recognizer helps tackle these challenges swiftly and boost productivity.

We are excited to announce the general availability (GA) release of Form Recognizer. You can now extract text, tables, and key value pairs quickly and accurately from documents. It will support multi-page documents (Images, PDFs and Tiff files) and extract a structured representation output of the document and its contents.

Form Recognizer comprises the following:

1. Layout

Detects and extracts text and tables structure extraction:

How to use and get started

You can use Form Recognizer Layout to recognize tables, text lines and words in documents, without needing to train a model. To get started you can use the following:

Extract the layout of a document, using the StartRecognizeContentFromUri method in the Form Recognizer client library.
Follow this QuickStart to extract the layout of the a document using the REST API.

2. Pre-built

These are pre-trained models for common scenarios that extract value of interest from documents. The pre-built receipts model that extracts data from receipts is in general availability today.

How to use and get started

You can use Form Recognizer to extract common fields from receipts, using a pre-trained receipt model. To get started you can use the following:

Extract data from receipts using the StartRecognizeReceiptsFromUri method in the Form Recognizer client library.
Follow this QuickStart to extract data from receipts using the REST API.

3. Custom

This custom service lets you train with your own data to learn the structure of your documents in an intelligent way with unsupervised and supervised learning

How to use and get started

You can train custom models tailored to your own documents. A trained model can output structured data that includes the text, tables, and key value pairs relationships in the original form document. After you train the model, you can test it and eventually use it to reliably extract data from more forms according to your needs. To get started you can use the following:

Train a custom model without labels and analyze your data using the method Custom Form Model in the Form Recognizer client library or using the REST API.
Train a custom model with labels and analyze your data follow this QuickStart and try out Form Recognizer sample labeling tool located here.

Code examples

These code snippets show you how to do the following tasks with the Form Recognizer client library for .NET:

Authenticate the client

Extract text and tables from documents using Layout

Recognize receipts

Train a custom model

Analyze forms with a custom model

Manage your custom models

How partners have built solutions with Form Recognizer

" Automation Anywhere has expanded the capabilities of IQ Bot to include Form Recognizer with an easy-to-use, “IQ Bot Forms” solution which combines the power of Microsoft Cognitive Services with IQ Bot and RPA to accelerate the end-to-end processing of complex documents. Use cases supported include Driver's Licenses, Insurance Claims and Tax Forms. This highly secure solution, comprises Automation Anywhere RPA with native Intelligent Document Processing (IDP) and Azure Cognitive Services Computer Vision API and Form Recognizer." Shobhana Viswanathan, Director of Business Development, Automation Anywhere

“Blue Prism has always been committed to creating a connected-RPA platform that makes it easy for our customers to consume the best in AI and machine learning technologies. As part of this commitment to innovation, we recently released a Form Recognizer API skill that gives our customers the power to quickly add deep-learning algorithms, advanced machine learning, and key value pair extraction to any Blue Prism process.” Colin Redbond - Senior Vice President – Emerging Technologies at Blue Prism

“Icertis’ suite of AI technologies use machine learning to help understand the contract, its obligations and its environment better. Form Recognizer is an important tool in that arsenal that helps identify structured data in forms quickly and accurately. With a very simple training interface, it empowers the Icertis Contract Management platform users to effectively incorporate AI in their day-to-day processes while ensuring that their data is safe and protected – important steps in our vision of making contracting simple, yet powerful.” Monish Darda, CTO and Co-founder, Icertis

“With the power of forms recognizer, Neudesic was able to create a simple interface for business users to extract data from multiple unique document sets, each containing complex data structures and dozens of data points, including tables. Users simply provide sample documents and label their data - no need to understand how Form Recognizer, or the other powerful Cognitive Services involved, should be applied, dramatically simplifying how AI can be applied to their processes.” Ken Kuzdas, Artificial Intelligence and Process Automation Lead, Neudesic

“UiPath remains committed to an open Platform and building integrations with partner AI services so you can automate document processing using your service of choice. The UiPath Activity Pack for Microsoft Azure Form Recognizer make it easy to automate tasks that involve document data – for example, reading invoices, timesheets, tables, and reports. By combining AI-powered document extraction services from Microsoft with the industry-leading UiPath Enterprise RPA Platform, you can extract data from any document using the service of your choice – and easily leverage this data in your automated processes.” Brandon Brown, Director Integrations and Solutions Delivery,UiPath

Independent benchmark testing results

Cazton,a top leader in IT and Software Consulting, Training and Recruiting services across United States, Canada and Europe performed an independent study comparing available cloud offerings for recognizing form data in the cloud and concluded that Azure Form Recognizer does a fantastic job in creating a viable solution with just five sample documents. It performs end-to-end Optical Character Recognition (OCR) on handwritten as well as digital documents with an amazing accuracy score and in just three seconds.

Chander Dhall, CEO of Cazton quotes that “I am impressed with Microsoft's focus on creating artificial intelligence powered solutions that have practical uses in the enterprise.”

New in this release

In this release we are introducing the following new features:

1. Enhanced security features

Bring your own key
Form Recognizer automatically encrypts your data when persisted it to the cloud. Form Recognizer encryption protects your data and to help you to meet your organizational security and compliance commitments. By default, your subscription uses Microsoft-managed encryption keys. However, you can now also manage your subscription with your own encryption keys. Customer-managed keys (CMK), also known as bring your own key (BYOK), offer greater flexibility to create, rotate, disable, and revoke access controls. You can also audit the encryption keys used to protect your data. Learn more here
Private endpoints
Enables you on a virtual network (VNet) to securely access data over a Private Link

2. Better Accuracy

Table enhancements and Extraction enhancements
This feature includes extraction enhancements, accuracy improvements and table extractions enhancements, specifically, the capability to learn tables headers and structures in custom train without labels.
Currency support
Helps detection and extraction of global currency symbols

3. Extended Availability

Azure Gov
Form Recognizer is available in 22 commercial regions and also in Azure Gov

Get started

To get started create a Form Recognizer resource in the Azure Portal and follow one of our quick starts to extract data from your documents.
To learn more about Form Recognizer and the rest of the Azure AI ecosystem, please visit our website and read the documentation.
For additional questions please reach out to us at formrecog_contact@microsoft.com

Easily add voice commands to your apps with Custom Commands

Vishesh Oberoi — Wed, 08 Jul 2020 18:19:08 GMT

Voice-enabled assistants—that enable users to search, ask questions, complete tasks, and much more—have been gaining increasing momentum and becoming integrated into consumer’s daily lives. Voice enables more seamless, natural interfaces, providing a more intuitive way of interacting with technology. In our current environment, voice and contactless experiences will play an increasingly important role, with the United States alone already seeing a 20 percent increase in preference for contactless operations (McKinsey 2020).

We saw some of this vision at Microsoft Build 2020 earlier this year where CTO Kevin Scott discussed emergent trends on the path to reshape software development, including the convergence of physical and digital worlds. One example centered around voice tech for food delivery, in which Boston Dynamics’ Spot robot completed curb-side deliveries using voice interaction.

We are committed to enabling developers and designers to build innovative voice-enabled solutions. To help make it easier to build voice commanding applications, today, we’re excited to announce the general availability of Custom Commands. Custom Commands is a capability of Speech in Azure Cognitive Services, that streamlines the process for creating task-oriented voice applications, providing a unified authoring experience with relatively lower complexity, helping you focus on building the best solution for your voice commanding scenarios.

Voice applications such as voice assistants listen to users and take an action in response. They involve transcribing the user's speech, taking action on the text using natural language processing, and using voice to respond with text-to-speech. Custom Commands brings together the best of Speech and Language in Azure Cognitive Services—Speech to Text for speech recognition, Language Understanding for capturing spoken entities with speech adaptation, and voice response with Text to Speech, to accelerate the addition of voice capabilities to your apps iteratively and with low-code authoring experience.

Custom Commands is best suited for task completion or command-and-control scenarios that have a well-defined set of variables. In addition to the voice-activated delivery example with Boston Dynamics’ Spot, Custom Commands supports solutions in a variety of verticals including hospitality, automotive and retail. For example, you can build in-room voice-controlled experiences for your guests, enable in-vehicle communication and entertainment systems, or manage store inventory with an ambient smart speaker.

Building Voice Assistants

Speech in Azure Cognitive Services provides solutions for building voice assistants that are tailored for your use case. Custom Commands streamlines the process for creating voice-enabled apps for simple task completion (with a fixed vocabulary and defined set of variables).

For some scenarios, you may also need a solution that handles more complex conversational interactions. For flexible voice assistants designed for open-ended conversational scenarios, Direct Line Speech enables you to build a robust solution that is optimized for voice-in, voice-out interaction with bots.

Here’s a sample reference architecture for an end-to-end voice assistant supported by Custom Commands:

Customization and Extensibility

With Custom Commands, our goal is to simplify the process of creating a unique voice-first experience that reflects your brand. You can configure multiple commands for commanding or task completion, add parameters and conditions to a particular task before completion, or configure interaction rules to handle confirmation prompts or one-step correction to help disambiguate.

Publish your app and integrate it with any client app using the Speech SDK. You can follow our documentation to integrate using the Speech SDK for C#, or build your own using our Speech SDK, which is available in multiple languages on various platforms.

It is very easy to integrate your app with the Speech SDK. Start by specifying the applicationId, subscriptionKey and region.

// Your application id const string customCommandsApplicationId = "YourApplicationId"; // Your subscription key const string customSubscriptionKey = "YourSpeechSubscriptionKey"; // The subscription service region. const string region = "YourServiceRegion"; var customCommandsConfig = CustomCommandsConfig.FromSubscription(customCommandsApplicationId, customSubscriptionKey, region);

Then configure your client app to receive activity from the Custom Commands app.

// Implement event handlers connector.ActivityReceived += (sender, activityReceivedEventArgs) => { ... };

Once you’ve published your app, consider adding a Custom Keyword to your app. Custom Keyword allows your product to be voice activated with a word or short phrase (for example, “Hey Cortana” is the keyword for the Cortana assistant).

Keywords generated using Custom Keyword can be easily integrated with your device or application via the Speech SDK. Note that audio only starts streaming to the cloud (for verification that the user said the keyword) after the keyword has been detected locally on the user’s device.

// Start listening for keyword var model = KeywordRecognitionModel.FromFile("YourKeywordModelFileName"); connector.StartKeywordRecognitionAsync(model);

Finally, bring your app to life with natural-sounding voices using Text to Speech. You can either use one of our 100+ out-of-the-box voices, or create a custom voice for your brand.

In addition, we provide comprehensive support for your development workflow, including the ability to import/export your app and integrate with continuous deployment pipelines.

We’re excited to see what you’ll build with Custom Commands.

Get started today

Get started today and check out our demos at https://speech.microsoft.com/customcommands

Learn more with our documentation: Custom Commands documentation

Follow the Quickstart: Create a voice assistant using Custom Commands

Check out easy-to-deploy samples: Voice Assistants GitHub repository

See the video tutorial: Building Voice Assistants using Custom Commands

Introducing Text Analytics for Health

hadasb — Wed, 08 Jul 2020 18:04:11 GMT

Hadas Bitran, Group Manager, Microsoft Healthcare

The healthcare industry is overwhelmed with data. Much of this healthcare data is in the form of unstructured text, such as doctor’s notes, medical publications, electronic health records, clinical trials protocols, medical encounter transcripts and more. Healthcare organizations, providers, researchers, pharmaceutical companies, and others face an incredible challenge in trying to identify and draw insights from all that information. Unlocking insights from this data has massive potential for improving healthcare services and patient outcomes.

Today, we are excited to introduce Text Analytics for health, a new preview feature of Text Analytics in Azure Cognitive Services that enables developers to process and extract insights from unstructured medical data. Trained on a diverse range of medical data—covering various formats of clinical notes, clinical trials protocols, and more—the health feature is capable of processing a broad range of data types and tasks, without the need for time-intensive, manual development of custom models to extract insights from the data.

Uncover deep insights and relationships in medical data

With Text Analytics for health, users can detect words and phrases mentioned in unstructured text as entities that can be associated with semantic types in the healthcare and biomedical domain, such as diagnosis, medication name, symptom/sign, examinations, treatments, dosage, and route of administration. (For full list of health entity types and relationships, see the documentation.) In addition, users can extract more than 100 types of personally identifiable information (PII), including protected health information (PHI), in unstructured text.

Text Analytics also links entities to medical ontologies and domain-specific coding systems (for example, the Unified Medical Language System), and identifies meaningful connections between concepts mentioned in text (for example, finding the relationship between a medication name and the dosage associated with it).

The meaning of medical content is also highly affected by modifiers, such as negation, which can have a critical implication if misdiagnosed. For example, it is important for healthcare professionals to determine when a patient “has not been diagnosed with something” or “ does not experience a certain symptom.” The health feature supports negation detection for the different entities mentioned in text.

Speed time to healthcare insights

Text Analytics for health enables researchers, data analysts, medical professionals and ISVs in the healthcare and biomedical space to unlock a wide range of scenarios—like producing analytics on historical medical data and creating prediction models, matching patients to clinical trials, or assisting in clinical quality reviews.

In response to the COVID-19 pandemic, Micosoft partnered with the Allen Institute for AI and leading research groups to prepare the COVID-19 Open Research Dataset, a free resource of over 47,000 scholarly articles for use by the global research community. With Cognitive Search and Text Analytics, we developed the COVID-19 search engine, which enables researchers to more quickly evaluate and gain insights from the overwhelming amount of information about COVID-19.

Learn more about using AI to mine unstructured research papers to fight COVID-19.

We are working closely with organizations such as the University College London (UCL), which is conducting reviews of medical research reports.

“One of our focuses as a research group is undertaking systematic reviews across a range of policy areas,” says Professor James Thomas at UCL, and Director of the EPPI-Centre’s Reviews Facility for the Department of Health, England. “We have been partnering with engineers at Microsoft and data scientists to build a ‘living’ reviews system – that automatically identifies relevant research for reviews as they are published. Text Analytics for health provides a powerful tool for extracting insights from clinical literature, with rich support for a wide range of healthcare terminology so that we can more quickly and accurately identify relevant information.”

At Microsoft, our goal within healthcare is to empower people and organizations to address the complex challenges facing the healthcare industry today, working closely with our customers and partners to bring healthcare solutions to life. We’re excited to make Text Analytics for health available in support of this mission.

Get started with Text Analytics for health

Text Analytics for health is currently available in containers. With containers, you can deploy resources in your own development environment that meets your specific security and data governance requirements.

The container provides REST-based query prediction endpoint APIs. Below is an example API request and response body:

For more resources:

Check out the Text Analytics documentation
See Text Analytics in action with the Covid-19 search engine demo
Contact us to try Text Analytics for health
To learn more about healthcare solutions using Azure, visit our website

Neural Text to Speech extends support to 15 more languages with state-of-the-art AI quality

Qinying Liao — Wed, 08 Jul 2020 14:19:53 GMT

Neural Text to Speech extends support to 15 more languages with state-of-the-art AI quality

This post was co-authored by Sheng Zhao, Jie Ding, Anny Dow, Garfield He and Lei He.

Neural Text to Speech, part of Speech in Azure Cognitive Services, enables you to convert text to lifelike speech for more natural interfaces. Neural Text to Speech (Neural TTS) enables a wide range of scenarios, from audio content creation to natural-sounding voice assistants. Companies like the BBC and Motorola Solutions are using Text to Speech in Azure to develop conversational interfaces for their voice assistants.

To make it possible for more developers to add natural-sounding voices to their applications and solutions, today, we’re building on our language support with 15 new Neural TTS voices along with significant voice quality improvements.

Language support extended with 15 new voices

Our new Neural TTS voices include: Salma in Arabic (Egypt), Zariyah in Arabic (Saudi Arabia), Alba in Catalan (Spain), Christel in Danish (Denmark), Neerja in English (India), Noora in Finnish (Finland), Swara in Hindi (India), Colette in Dutch (Netherland), Zofia in Polish (Poland), Fernanda in Portuguese (Portugal), Dariya in Russian (Russia), Hillevi in Swedish (Sweden), Achara in Thai (Thailand), HiuGaai in Chinese (Cantonese, Traditional) and HsiaoYu in Chinese (Taiwanese Mandarin).

Hear samples of the voices, or try them with your own text in our demo.

Locale code	Language	Voice name	Audio sample
ar-EG	Arabic (Egypt)	“ar-EG-SalmaNeural”	أتساءل ماذا يمكن ان يحدث لجسمك عندما تأكل الزنجبيل كل يوم لمدة شهر؟
ar-SA	Arabic (Saudi Arabia)	“ar-SA-ZariyahNeural”	لديك نصف ساعة فقط؟
ca-ES	Catalan (Spain)	“ca-ES-AlbaNeural”	L'obra és el retrat d'un moment històric de mobilització popular.
da-DK	Danish (Denmark)	“da-DK-ChristelNeural”	Halvfjerds procent af din krop består af vand
en-IN	English (India)	“en-IN-NeerjaNeural”	How about coming to the barbecue at the tennis club?
fi-FI	Finnish (Finland)	“fi-FI-NooraNeural”	Tavoitteena on lisätä kohtuuhintaisten vuokra-asuntojen määrää kasvukeskuksissa.
hi-IN	Hindi (India)	“hi-IN-SwaraNeural”	‘आयरन’ शब्द किस खेल से सम्बन्धित है ?
nl-NL	Dutch (Netherland)	“nl-NL-ColetteNeural”	Alle oceanen zijn met elkaar verbonden en vormen samen één grote massa zout water.
pl-PL	Polish (Poland)	“pl-PL-ZofiaNeural”	Wyjazd z Poznania planujemy o godzinie czwartej rano.
pt-PT	Portuguese (Portugal)	“pt-PT-FernandaNeural”	Amanhã vai estar tanto calor que vou à praia.
ru-RU	Russian (Russia)	“ru-RU-DariyaNeural”	В качестве примера он привел искусственный интеллект, беспилотную технику, генетику, медицину и образование.
sv-SE	Swedish (Sweden)	“sv-SE-HilleviNeural”	Ett kul och intressant avsnitt även för dig som inte var på plats!
th-TH	Thai (Thailand)	“th-TH-AcharaNeural”	เขาทำด้วยหัวใจบริสุทธิ์และต้องการให้ความยุติธรรมแก่ประชาชน
zh-HK	Chinese (Cantonese, Traditional)	“zh-HK-HiuGaaiNeural”	了解該等基金的三大特點,有助投資者作出更明智的選擇。
zh-TW	Chinese (Taiwanese Mandarin)	“zh-TW-HsiaoYuNeural”	如果一個人從一所優秀大學畢業，可能意味他有能力做大事。

Text-to-speech quality is measured by Mean Opinion Score (MOS), a widely-recognized scoring method for speech quality evaluation. For MOS studies, participants rate speech characteristics such as sound quality, pronunciation, speaking rate, and articulation on a 5-point scale. According to several MOS tests we have done (n>50 for each study), the average MOS score for the 15 new Neural TTS voices is above 4.1, about +0.5 higher than the scores for standard (non-neural) voices.

See the full language list for Neural TTS and standard voices.

Voice quality and performance improved with state-of-the-art neural speech synthesis models

Neural TTS initially achieved near-human-parity on sentence reading using a recurrent neural network (RNN) based sequence-to-sequence model. Inspired by the Transformer model—a powerful sequence-to-sequence modeling architecture that advanced the state-of-the-art in neural machine translation (NMT), Microsoft researchers piloted the Transformer and FastSpeech models on Neural TTS and saw significant improvements in performance and efficiency. The Transformer TTS model is based on the auto-regressive Transformer structure, which can produce speech output in the quality close to the actual human voices with 5x less training time. FastSpeech is a new text-to-speech model that improves speech synthesis speed, accuracy, and controllability.

New neural voice model creation based on teacher-student transfer learning

Multi-lingual and multi-speaker TTS recordings are first used to train a transformer base model. To scale TTS development for many locales and voices, it is vital to have a highly agile development process. We built a "transformer teacher model" with 3,000+ hours of speech data from hundreds of speakers in 50+ languages/locales —about 50x of a typical single language multi-speaker model.

By using around 2 hours of a target speaker’s data, we can now adapt the multi-lingual multi-speaker transformer teacher model to generate a new high quality model for the speaker that sounds very similar to the original recording. Then we can use a “finetuned teacher” to generate training data with rich content coverage to train a FastSpeech “student” for deployment that achieves the same quality as its finetuned teacher.

With this powerful multi-lingual model, we are also able to take the voice samples from one speaker in one language as input and transfer the voice into another target language, without losing quality.

With the Transformer and FastSpeech models, key improvements include:

Quality enhancements: The new models achieved significant MOS improvements over the previous robust LSTM-based Neural TTS models in our platform. For example, we did a side-by-side comparison on de-DE Kajta voice; the new model shows +0.4 comparative MOS gain over the baseline.

Higher performance: With the new models, users can get high quality Neural TTS output with faster response time. FastSpeech “students” have 10X inference speedup on mel-spectrogram generation using M60 GPUs compared to our previous production systems. Neural TTS can run 40% faster on a Kubernetes GPU Pod. We can also run Neural TTS on CPU with 0.06 RTF (Real Time Factor), which means 1 second of audio can be generated in 60ms on a Kubernetes CPU Pod.

Language-specific improvements

When developing Neural TTS for new languages, there are also language-specific challenges that need to be addressed to ensure high voice quality and performance.

For example, to make synthetic speech sound humanlike, it is critical to get pitch accents right. Japanese (ja-JP) poses challenges for speech synthesis because of its complicated pitch accents. However, most end-to-end TTS systems cannot perform well on pitch accents; we found that about 60% of production system's problems in Japanese synthesis are related to intonation and accents.

Language-specific pitch accent prediction model

We built a transformer model to predict and account for pitch accent related features. The accent model predicts accent phrase boundaries and accent type information, and these accent features are introduced into the acoustic model. The teacher model and student model will use the accent features in training and synthesis.

With the pitch accent features, the voice quality improves significantly. Our MOS test shows that the new ja-JP voice, Nanami, has a +0.3 improvement in MOS score compared to the previous production system. This method is also applicable to other languages with pitch accents.

Here are some samples:

Text	Sample of the old model without pitch accent support	Sample of new model with pitch accent support
1日2食に切り替える予定だ。
被災地には僕らの番組のため今も毎週のように行っています。

Create a custom voice with Neural TTS technology

The latest technical advancements in Neural TTS are also available in the Custom Neural Voice capability, enabling organizations to create a unique brand voice in multiple languages with 5-10X less data.

Learn more about the process for getting started with Custom Neural Voice.

Get started

With these updates, we’re excited to be powering natural and intuitive voice experiences for more customers. Text to Speech on Azure has more than 70 standard voices in over 40 languages and locales in addition to our growing list of Neural TTS voices. .

For more information:

Try the TTS demo
See our documentation
Check out our sample code

Accelerate labeling productivity by using AML Data Labeling

Vijai_Kannan — Tue, 07 Jul 2020 04:31:02 GMT

Labeled data is critical to training supervised learning models. Higher volumes and more accurate labeled data contribute to more accurate models but labeling data has traditionally been time-intensive and error-prone.

With Data Labeling in Azure Machine Learning, you now have a central place to create, manage, and monitor labeling projects. You can now manage data labeling projects seamlessly from within the studio web experience to generate and manage tasks reducing the back-and-forth of labelling data offline. With AML Data Labeling, you can load and label data and be ready to train in minutes.

To increase productivity and decrease costs for a given task, the Assisted Machine Learning labeling feature allows you to leverage automatic machine learning models to accelerate labeling by clustering like objectives and automatically prelabeling data when the underlying model has reached high confidence. This feature is available for image classification (multi-class or multi-label) and Object detection tasks, in Enterprise edition workspaces.

Data Labeling in Azure Machine learning now includes below capabilities:

Image Classification Multi-Class

This project type helps you to categorize an image when you want to apply only a single class from a set of classes to an image.

Image Classification Multi-label

This project type allows you to categorize an image when you want to apply one or more labels from a set of classes to an image. For instance, a photo of a dog might be labeled with both dog and land.

Object Identification (Bounding Box)

Use this project type when you want to assign a class and a bounding box to each object within an image. If your project is of type "Object Identification (Bounding Boxes)," you'll specify one or more bounding boxes in the image and apply a tag to each box. Images can have multiple bounding boxes, each with a single tag.

Assisted machine learning

The machine assisted labeling lets you trigger automatic machine learning models to accelerate the labeling task. At the beginning of your labeling project, the images are shuffled into a random order to reduce potential bias. However, any biases that are present in the dataset will be reflected in the trained model. For example, if 80% of your images are of a single class, then approximately 80% of the data used to train the model will be of that class. This training does not include active learning.

Enabling ML assisted labeling consists of two phases:

Clustering
Prelabeling

The exact number of labeled images necessary to start assisted labeling is not a fixed number. This can vary significantly from one labeling project to another. ML Assisted Labeling uses a technique called Transfer Learning, and the pre-labeling will be triggered when sufficient confidence is achieved which varies based on the dataset.

Since the final labels still rely on input from the labeler, this technology is sometimes called human in the loop labeling.

Clustering

After a certain number of labels are submitted manually, the machine learning model for image classification starts to group together similar images. These similar images are presented to the labelers on the same screen to speed up manual tagging. Clustering is especially useful when the labeler is viewing a grid of 4, 6, or 9 images.

The clustering phase does not appear for object detection models.

Prelabeling

After enough image labels are submitted, a classification model is used to predict image tags. Or an object detection model is used to predict bounding boxes. The labeler now sees pages that contain predicted labels already present on each image. For object detection, predicted boxes are also shown. Accuracy will vary depending images, labels, the domain, and other factors. With Pre-Labeling, you can review the predictions before committing the labels.

Once a machine learning model has been trained on your manually labeled data, the model is evaluated on a test set of manually labeled images to determine its accuracy at a variety of different confidence thresholds. This evaluation process is used to determine a confidence threshold above which the model is accurate enough to show pre-labels. The model is then evaluated against unlabeled data. Images with predictions more confident than this threshold are used for pre-labeling.

Resources

Learn more about the Azure Machine Learning service.

Get started with a free trial of the Azure Machine Learning service.

Gain deeper insights from customer reviews using Opinion Mining

raymondlag — Wed, 01 Jul 2020 16:07:46 GMT

Product reviews, social media posts, and discussion forums are a major source of information for businesses for analyzing customer feedback. Natural Language Processing (NLP), and specifically sentiment analysis, provides an automated way to categorize this feedback into positive, neutral, and negative categories. Using this information, businesses can identify trends in customer sentiment, find what drives customer satisfaction, and react to negative feedback.

With the release of the new Opinion Mining feature in Text Analytics, we provide a new tool for analyzing user-generated content. Opinion Mining is an implementation of aspect-based sentiment analysis which goes beyond identifying the sentiment to provide more insights from text data. Using this new tool, businesses can extract customers’ sentiment and opinions around specific aspects or attributes of a product or service.

For example, in a review comment regarding a hotel stay such as “I liked the pool, but the staff was unfriendly”, Opinion Mining would identify the following:

Aspect	Opinion	Sentiment
Pool	Liked	Positive
Staff	Unfriendly	Negative

Aggregating this data can highlight trends and provide deeper understanding of customer sentiment, breaking down the data into specific areas the business can dive into.

Image showing multiple aspects and opinions from hotel reviews

Opinion Mining can add greater depth to understanding of customer sentiment and provide a more granular view of the data in voice of customer, customer feedback analytics, and call center analytics solutions. Customer feedback that mentions a competitor’s product features can be used to perform competitive analysis. And combined with customer purchase history and profile data from CRM systems, it can provide a powerful way to analyze and guide product design and launch, brand strategy, and advertising and marketing campaigns.

Get started

For more information on how to use sign up and use Opinion Mining, see the documentation.

COVID-19 “Back to Work” Solution Template using Microsoft Healthcare Bot, Azure and Teams

nikitapitliya — Wed, 05 Aug 2020 00:26:10 GMT

Update: This blog was last edited on 8/4/2020. This blog outlines using Healthcare Bot for COVID-19 Back to Work use case with authentication and data persistence in Azure API for FHIR or Azure SQL Database. The template in Healthcare Bot's template catalog has been updated to a simple version without data persistence (not covered in this blog).

What is the COVID-19 “Back to Work” Solution Template?

As countries worldwide seek to re-open their economies by relaxing “stay-at-home” orders, many employers are considering how to prepare their facilities and employees for the return to physical workplace. To do this as safely as possible, it is critical to monitor employees for common COVID-19 symptoms and provide a simple way for affected employees to physically return once they are cleared to do so. Microsoft has developed a special solution template to enable employers worldwide to easily create and deploy Microsoft technologies to scale and automate those critical steps for return to the workplace. We call it the “Back-to-Work” solution template. In this blog, we will discuss the solution and how you can use it to empower a safer return to the workplace for your organization.

Key Elements and Steps

Our COVID-19 Back-to-Work solution template is built on Microsoft Healthcare Bot service and the Microsoft Azure platform. Some key elements include:

Symptom tracking for essential workers who have been exposed to COVID (physicians, nurses, ancillary staff, volunteers) (Please refer to Microsoft Privacy Principles for data collection during COVID-19 crisis)
Alignment with CDC guidelines*
Organizational configurable documentation of information such as symptoms, occupational exposure type, and testing status
Allows systems to determine when employees will be safe to return to patient care activities, the workplace, or campus
Supported by Microsoft’s industry leading compliance, privacy, and security portfolio (link)

*CDC guidelines are rapidly evolving. Microsoft does not guarantee to keep this solution updated

The solution template involves three steps:

Reference Architecture of Microsoft platform support

The COVID-19 Back-to-Work solution template is an ACCELERATOR KIT to help you quickly build and deploy a custom solution for your organization. Organizations across all industries, such as healthcare, education, retail, manufacturing, and financial services can use this highly customizable template. Microsoft’s platform provides the necessary capabilities by combining our Healthcare Bot service with the Azure platform and Microsoft Teams as shown below:

With the Microsoft Healthcare Bot service at its core, the deployed solution will allows you to instrument the health bot in multiple UI channels including a Website, Microsoft Teams, Twilio, Facebook and Telegram. (We plan to release configuration for building a mobile native app soon). The COVID-19 Back-to-Work logic asks a set of questions on COVID-19 exposure, symptoms and lab tests. Based on responses, each individual is then directed to either stay at home or is cleared to enter the workplace. Information from your preferred data store can be easily visualized using Power BI, thanks to the integrated Power BI model available with the solution.

Using Microsoft Healthcare Bot service to write user responses to Azure API for FHIR is our primary recommendation for healthcare organizations to provide data interoperability from different health systems. Details on using the healthcare bot with Azure API for FHIR is provided in Additional Resources section below.

COVID-19 Back to Work Talk-Track

Watch the video below to get a quick walk-through of the solution overview.

Typical end-to-end Workflow

One typical end-to-end flow can be summarized as:

Company A’s IT Administrator configures a time-based trigger to send email notifications to all employees required to take daily screening assessment
Users click on the bot link, register themselves and take the day’s screening assessment
User responses are captured and stored in Company A’s Azure tenant in the chosen backend (Azure API for FHIR or Azure SQL Database or Azure Cosmos Database)
IT Administrator can view metrics on
- Users who took assessment vs missed/left incomplete
- Users who were asked to stay home vs who were cleared to enter the workplace
- Predict when a large employee-base is coming back to workplace and make necessary arrangements to ensure their safe return
- Users get an automated email notification every day. Users click on the bot link, login and complete the day’s screening assessment.

For a brief demo of this end-to-end workflow, watch the video below.

Additional Resources

To use the Accelerator Kit with Azure SQL Database persistence and rest of the building blocks, a GitHub repository is available at https://github.com/microsoft/covid19-BackToWork
To use Back-to-Work template with Azure API for FHIR, refer step-by-step instruction guide
For a quick start guide on Healthcare Bot service, refer step-by-step quickstart

Stay tuned for more updates on the solution. Thanks for reading and let us know how we can help. Thank you!

Nikita Pitliya , Microsoft Senior Solution Architect

Han Zhang, Microsoft Cloud Solution Architect

Julian Soh , Microsoft Principal Solution Architect

Joe Karasha , Microsoft Senior Solution Architect

Greg Beaumont , Microsoft Senior Technical Specialist

Srini Surendranath , Microsoft WW Customer Lead

Using GitHub Actions & Azure Machine Learning for MLOps

David_Aronchick — Wed, 27 May 2020 01:18:21 GMT

Background

Like Gitops, Machine Learning Operations (or MLOps) can make significant improvements in accelerating how data scientists can impact organizational needs. A well-implemented MLOps process not only speeds the time from code to production, but also provides ownership, lineage and historical information, critical for understanding the performance of any machine learning model. Critical to this process is a CI/CD system that understands the elements of ML natively as well as stays in sync with any code or data changes, no matter what platform organizations need them to run on.

Unfortunately, many data scientists are still forced to implement MLOps manually. Oftentimes CI/CD platforms are powerful, but quite generic, requiring the implementation of “ML aware” through custom code. And, worse, these platforms often require separating the actions from the code, leading to difficulty in debugging and hard to reproduce caching issues. Our goal is to both give these data scientists tools that are easy to implement and use.

GitHub Actions and Azure Machine Learning To The Rescue

Today, we’re proud to announce a series of GitHub Actions designed to allow people to implement MLOps with just a few configuration settings, but they are flexible enough to support even complicated workflows. Now, by just checking in your code or opening a pull request, you can kick off an entire ML pipeline, recording all information about the process, and updating that model from the actions.

The first five functions we have published are:

aml-workspace - Login action to login / connect with Azure Machine Learning
aml-compute - Create Compute action to create compute for Azure Machine Learning will allow you to create a new compute target on Azure Machine Learning
aml-run - Train action for training machine learning models using Azure Machine Learning
aml-registermodel - Save a model to Azure Machine Learning
aml-deploy - Deploy action to deploy your model on Azure Machine Learning and creates a real-time endpoint for use in other systems.

NOTE: Though these are all Azure Machine Learning functions, GitHub Actions for MLOps support any cloud.

These actions are based on DevOps principles and practices that increase the efficiency of workflows. For example, continuous integration, delivery, and deployment. We have applied these principles to the machine learning process with the goal of:

Faster experimentation and development of models
Faster deployment of models into production
Quality assurance

Not only that, because the entire MLOps as a service is hosted and run on behalf of the users, it frees up time for the ML Engineers to focus on more business critical issues. Additionally, workflows can be updated and added on the back end without the users even knowing, making maintenance of these pipelines even easier.

To show you end-to-end what this would like, we have a short video:

Walk-through

Let’s walk you through how you would implement something like this.

Using GitHub Actions and Azure Machine Learning

First, you’ll need some initial setup variables. These include:

Azure subscription
Contributor access to the Azure subscription
Access to GitHub Actions

If you don’t have an Azure subscription, create a free account before you begin. Try the free or paid version of Azure Machine Learning today.

Second, create your own repository from the template. You can do this"Use this template" button in the repo: https://aka.ms/ml-template

Third, you’ll need a service principal with contributor rights to a resource group (either new or existing). To create a new one on Azure, use the Azure CLI on your computer and execute the following command to generate the required credentials:

# Replace {service-principal-name}, {subscription-id} and {resource-group} with your 
# Azure subscription id and resource group name and any name for your service principle
az ad sp create-for-rbac --name {service-principal-name} \
--role contributor \
--scopes /subscriptions/{subscription-id}/resourceGroups/{resource-group} \
--sdk-auth

This will generate the following JSON output:

{
"clientId": "<GUID>",
"clientSecret": "<GUID>",
"subscriptionId": "<GUID>",
"tenantId": "<GUID>",
(...)
}

Add this JSON output as a secret with the name AZURE_CREDENTIALS in your GitHub repository:

Please follow this link for more details.

Next, modify the parameters in the /.cloud/.azure/workspace.json file in your repository, so that the GitHub Actions create or connect to the desired Azure Machine Learning workspace.

Once you save your changes to the file, the predefined GitHub workflow that trains and deploys a model on Azure Machine Learning gets triggered. Check the actions tab to view if your actions have successfully run.

Now that you have a running pipeline, you can start modifying the code in the code folder so that the pipeline uses your custom code.

With just a few configuration settings, you can move from zero to an entire code & GitHub Action driven workflow. In addition to the above actions, we are also publishing two templates that include code and workflow definitions for an end to end ML/AI lifecycle.

Simple template repository: ml-template-azure
Go to this template and follow the getting started guide to set up an ML Ops process within minutes and learn how to use the Azure Machine Learning GitHub Actions in combination. This template demonstrates a very simple process for training and deploying machine learning models.
Advanced template repository: aml-template
This template demonstrates how approval processes can be included in the process and how training and deployment workflows can be split. It also shows how workflows (e.g. deployment) can be triggered by pull requests. More enhancements will be added to this template in the future to make it more enterprise-ready.

Please dive into either repo and let us know if there’s anything we can do to help you achieve your goals with MLOps.

Further, though we’ve implemented the first version using Azure Machine Learning, the platform is flexible enough to support most deployment platforms, both on-prem and on any cloud. Just clone our template repo and customize on your own. And make sure to publish your actions to the GitHub Marketplace so that others can use it!

Finally, we very much want to build a community around these actions - please join us at https://aka.ms/ml-template (for the standard template) or https://aka.ms/ml-template-advanced (for the advanced template) to file issues, pull requests and comments about what we can do better. Thank you so much!

-- Zander, Marvin, Pulkit & Dave

Batch Inference in Azure Machine Learning

tracych — Tue, 26 May 2020 16:49:06 GMT

Today, we are announcing the general availability of Batch Inference in Azure Machine Learning service, a new solution called ParallelRunStep that allows customers to get inferences for terabytes of structured or unstructured data using the power of the cloud. ParallelRunStep provides parallelism out of the box and makes it extremely easy to scale fire-and-forget inference to large clusters of machines, thereby increasing development productivity and decreasing end-to-end cost.

Batch inference is now being widely applied to businesses, whether to segment customers, forecast sales, predict customer behaviors, predict maintenance, or improve cyber security. It is the process of generating predictions on a high volume of instances without the need of instant responses. The predictions are stored and accessible for further usage. Often times, data scientists and engineers want to generate many predictions at once. But it’s challenging to take advantage of scalable compute resource to parallelize the large workload to achieve this. For example, how to partition the large amounts of input data? How to distribute and coordinate workloads across a cluster of machines? How to consolidate the output results? How to manage the machine cluster to avoid unnecessary cost? What if a task fails or machine dies?

ParallelRunStep, the new solution we provide for batch inference, handles all these for you. It simplifies scaling up and out large machine learning workloads so data scientists and engineers can spend less time developing computer programs and focus on business objectives.

ParallelRunStep includes capabilities that enable parallel processing and easy management of your large workloads:

Managed stack with out-of-the-box parallelism
Resiliency with failure tolerance
Fully composable with Azure Machine Learning Pipelines
Flexible design for a variety of workloads

Managed stack with out-of-the-box parallelism

ParallelRunStep is a managed stack with out-of-the-box parallelism. You only need to provide full data inputs, scoring script, and necessary configures, the left will be taken care of.

Partition input data

Let’s start with input data partitioning. You have 10K invoices to be extracted using your form recognizer model. To parallelize the workload, you want to combine 10 invoices as a mini batch and send to one machine for execution. You will have 1000 mini batches in total. In another use case, you have a 5GB csv file containing customer data to predict churn score per each record. You need to split the big file into smaller ones for parallel processing. For example, 500 files with the size of 10MB for each. With ParallelRunStep, you can easily achieve both above partition strategy.

ParallelRunStep accepts data inputs through Azure Machine Learning datasets. Dataset is a resource for exploring, transforming, and managing data. Partition your data by setting the mini batch size and leveraging the two types of dataset.

FileDataset represents single or multiple files with any format. For the invoice extraction case, use FileDataset and define mini batch size as 10 will automatically partition the workload into 1000 mini batches.

TabularDataset represents data in a tabular format by parsing the provided file or list of files. It can be created from csv, tsv, parquet files, SQL query results, etc. For the churn score calculation case, use TabularDataset and specify mini batch size as 10MB to get 500 mini batch workloads created.

Distribute workloads to managed machine cluster

After the entire input data is partitioned into multiple mini batches, ParallelRunStep distributes the mini batch workloads to a managed Azure Machine Learning compute cluster.

The Azure Machine Learning compute cluster is created and managed by Azure Machine Learning. It can be auto scaled each time you run a job. Such autoscaling ensures that machines are shut down when your job is completed to save your cost. It supports for both CPU and GPU resources. You can also choose low priority virtual machines to save resources for other latency sensitive jobs and reduce your costs further.

One mini batch workload will be sent to one compute node for execution. The more compute nodes you use, the more parallelism you will get. Execution efficiency for each mini batch workload is determined by the power of the compute node.

Consolidate the results

Processed results from each mini batch workload are collected, stored, and made available to you for further analysis. You will also get a job run summary and a performance report.

Resiliency with failure tolerance

ParallelRunStep is a resilient and highly available solution. While the system manages the strategy, you also have the control of when to timeout your job, how many times to retry and how many errors to tolerant.

Timeout setting is provided per mini batch workload invocation. You can tune this timeout properly to fail fast and avoid waiting forever in case of failures.

Automatic retry is embedded.The system by default retries three times for each mini batch workload if it fails and you can customize the max retry count. Even with several failed mini batch workloads, you can still get the partial results from other successful mini batch workloads.

You can control how many errors to tolerant through the error threshold configure. It’s defined as the number of files or records processing errors the entire job can tolerant. When the error threshold is reached, your job will be terminated. You can also choose to ignore all errors and allow your job to process all inputs.

Fully composable in Azure Machine Learning Pipelines

ParallelRunStep is available through Azure Machine Learning pipelines. Pipelines are constructed from multiple steps, which are distinct computational units in the pipeline. ParallelRunStep is one of such steps. Existing Azure ML Pipeline customers can easily add or switch to ParallelRunStep to run batch inference.

A pipeline is reusable after it is designed and published. Defining the input dataset or parameters as the type PipelineParameter gives you the ability to use dynamic input for each run, and fine tune your pipeline for better performance.

You can run a pipeline on recurring schedule based on elapsed time, or on data changes. For example, if you want to get a monthly customer churn score, you can create a time-based schedule to kick off your batch inference job every month.

Flexible design for a variety of workloads

ParallelRunStep is flexibly designed for a variety of workloads. It’s not just for batch inference, but also other workloads which necessitate parallel processing, e.g. training many models concurrently, or processing large amount of data.

Get started

Get started with Azure Machine Learning for free today!

Learn more:

Build the best search experience in your application with new capabilities in Azure Cognitive Search

pracjain — Wed, 10 Jun 2020 20:20:41 GMT

This blog has been authored by Vinod Kurpad (Principal PM, Azure Cognitive Search) and Prachi Jain (PMM, Azure AI)

Azure Cognitive Search is a cloud service that enables developers with APIs and tools to build rich search experiences over a variety of content in web, mobile, and enterprise applications. This week at //build, we announced new capabilities in Azure Cognitive Search making it easier for developers to build search experience and customize with new skills to deliver more relevant results to their users.

Improved Development Experience
Debug sessions is a new portal preview feature in Cognitive Search, that gives you a rich IDE like experience for refining a skillset and fixing issues within an AI enrichment pipeline, As visualized in Skills Graph, you can explore enrichments across all nodes within and enrichment tree and evaluate each skill invocation.

Debug Sessions allow you to modify skills as well as inspect inputs and outputs of each step.

“Debug session skill graph”

Debug sessions help you with three kinds of issues:

Skillset issue: expressions, paths and type mismatches within your skillset
Skill failures: Applies mostly to custom skills, with the ability to generate a request that you can debug locally
Data inconsistencies: Handle scenarios where a specific document fails, for example if your data source now contains a document in a different language from what you had configured that another skill does not recognize.
Debug sessions also help you start small and incrementally build more complex skillsets.

This capability is now in preview and you can sign up for the preview here, watch to learn more and follow the step by step guide.

More intelligent than ever

We have continued to grow our catalog of skills, with new additions:

• PII skill identifies and redacts personally identifiable information, such as social security numbers, email addresses, credit card numbers, drivers’ licenses and many more entities. You can find the complete list of entities and languages supported here.
• Translation skill identifies the language of the text in the document and translate it into a target language. The translation skill supports a variety of languages ensuring coverage over most scenarios.
• Document extraction skill can inserted at any point within the skillset. In the past, document extraction was implicit in the beginning of the enrichment pipeline, now it can be configured within the pipeline. This enables scenarios like working with encrypted files.
• Azure Machine Learning Skill (AML) makes discovering and consuming a model built within AML simple and intuitive. Endpoint discovery, authentication, and schema validation are some of the key benefits.
The AML skill is a preview feature that you can sign up to use.

Now your Azure Machine Learning skills can be automatically detected when you edit your skills from the portal.

“AML Skill : Adding the AML skill to skillset"

More relevant search results

We are bringing new capabilities that help you deliver better search results

The introduction of a new BM25 based ranking algorithm, that in our tests increased Normalized Discounted Cumulative Gain (NDCG) by about 5 points! This generates more intuitive results that align with user expectations. You can test this algorithm today.
Mechanisms to provide more consistent results to users even in a world of shards and replicas that are constantly changing. We have introduced the ability to specify a query session (to reduce changes across the session) as well as the ability to request the scoring statistics to be computed based on global statistics (across shards). Learn more about scoring statistics.
For more granular control on ranking your search results, those of you who are more data science inclined, we have exposed some of the statistics computed for indexing purposes that you can use as an input into a “Learning to Rank” model that you create. You can then invoke this model to override the defaults to re-rank your search results. Get index statistics as part of your query.

More secure than ever

Now that we have encryption for data in transit and at rest, our next security advancements are in the areas of endpoint protection and access control. The latest security features include support for accessing a search service over a private endpoint, limiting access to specific IP ranges, and limiting access to only clients in a virtual network. In addition, we are also announcing preview support for AD managed identities. You can register a search service with Active Directory, and then grant read access to that identity from the Azure data sources you index from. Watch to learn more.

Now you can assign Azure Cognitive Search a managed identity that can be given “rights” to read a data source

It is great to how customers like PwC, have built a capability to automatically identify obligations for US consumer financial regulations on Azure Cognitive Search via their Regulatory Obligation Identifier Skill, saving significant manual effort for their customers searching for documents that contain regulation content. Learn more about their solution.

Get Started:

Build 2020 - Introducing Bot Framework Virtual Assistant 1.0

GaryPrettyMsft — Thu, 21 May 2020 09:14:12 GMT

Customers and partners have an increasing need to deliver advanced conversational assistant experiences tailored to their brand, personalized to their users, and made available across a broad range of canvases and devices. The Virtual Assistant Solution Accelerator answers this need and, with v1.0 released at Build 2020, is now generally available!

The solution accelerator is open source in GitHub and provides you with a set of core foundational capabilities and full customization over the end user experience - including the name, voice, and personality of your assistant – whilst not sacrificing control over privacy and data.

You can get started in minutes and extend rapidly, using pre-built reusable conversational Skills which cover common assistant use-cases, or develop your own skills using comprehensive end-to-end tooling such as Bot Framework Composer.

This article will provide a high-level overview of the Virtual Assistant Solution Accelerator, providing a good grasp of the key concepts. Over the coming weeks we will release additional articles, each providing a focused deep dive into each area.

Virtual Assistant Core

The Virtual Assistant Core is the foundation of your solution, built on top of the latest Bot Framework SDK and integrated with Cognitive Services to provide the core assistant experience, such as Language Understanding, which is used for natural language understanding (NLU). Key features include.

Common dialog implementations - for common assistant requirements, such as introduction, on-boarding experience, and handling situations where there is a need to hand off the conversation to a human. These base implementations include the language understanding models (.LU files) for recognizing user intents to trigger them (e.g. “I need to speak to a human”).
FAQ and Personality - allowing the bot to answer user questions, from FAQs made available from a QnA Maker knowledgebase, including taking advantage of the new multi-turn feature, in addition to making use of the Chit Chat personalities provided by the service, giving your assistant the ability to respond to common ‘small talk’, making it more engaging. Pre-built data sets are provided for professional, friendly, witty, caring and enthusiastic personalities and they are fully customisable.
Complex conversational capabilities, including interruption and context switching – interacting via natural language can be complex, but the Virtual Assistant handles common scenarios with ease, such as the ability for a user to switch context or interrupt their conversation, such as to a different skill, escalating to a human, asking for contextual help, going back to an earlier step or cancelling their current flow completely.
Multi-locale Language Generation (LG) support – The solution takes advantage of LG files (also made generally available at Build), which allow for more natural and dynamic responses. Using LG, you can provide variations for each of your responses, meaning that a conversation doesn’t feel static to a user who is regularly interacting with the assistant. You are also able to access state and in-memory data, allowing you to customize responses based on context. LG files are available in English, Spanish, French, German, Italian and Chinese, with the ability to easily add additional locales if required. Multi-locale support also extends to Language Understanding (LU) assets.
Speech support - Speech-first experiences can be enabled without any custom-code, responding to the evolving change in user behavior towards multi-modal experiences on a broad range of platforms and devices.
Telemetry, Logging and Analytics - A telemetry pipeline for Virtual Assistant, leveraging both PowerBI and Azure Application Insights. This enables you to quickly and easily understand how your assistant is being used by users and gain actionable insights to make tangible improvements. Automated logging of transcripts can also be enabled, allowing for deeper analysis at a later date or passing conversation history to a human agent when handing off. An explicit mechanism is also available to ask users for their feedback when they complete a scenario using the assistant.

Skills

Several Skills, covering common assistant scenarios, are available to plug-in to your assistant immediately – rapidly increasing the capability of your solution without the need to expend custom development effort. However, as with the core, Skills are fully customisable and make use of the same assets (dialogs, LU and LG files), allowing you to easily tailor them to suit your specific requirements.

The following Skills are currently available and they are pre-integrated with services such as the Microsoft Graph.

Calendar – providing calendar, meeting room booking and meeting management capabilities for users. E.g. “book a meeting with Darren tomorrow at 2pm at the Hyatt”. Using the Microsoft Graph, this skill is able to correctly search for an identify contacts without users needing to explicitly use their full name or email address.

Email – ability to compose, search, read, delete, and reply to email by interacting via natural language, connecting to an Office 365 or Google mailbox.
To do – provides task management capabilities to your assistant, allowing users to add, search, delete tasks and mark them as complete when they are done. As with the Calendar and Email Skills, Microsoft Graph integration is built in, bringing synchronization of tasks across platforms such as Microsoft ToDo, Planner and Outlook.
Point of Interest – users can find points of interest and directions by taking advantage of the integration with Azure Maps and FourSquare.

The following experimental / preview skills are also available.

Hospitality – allowing for experiences such as managing reservations, check out, and amenity requests.
IT Service Management (ITSM) – provides a basic skill that provides ticket and knowledge base related capabilities, with support for ServiceNow built-in.
Music – Features artists and playlist lookup for the popular music service Spotify. Playback information is then signalled back to the device through Events enabling native device playback.

Channels and Clients

It is crucial that you surface your assistant on the channels being used by your users, meeting them where they are. Via Azure Bot Service (ABS), you can connect your assistant to any channel currently supported by that service, including Microsoft Teams, web chat, Facebook Messenger, Slack and the new preview channel for Alexa Skills. Beyond the channels currently supported by ABS, you can also take advantage of available Bot Framework adapter implementations, allowing your assistant to accept request directly from other platforms such as WhatsApp, RingCentral, Google Assistant and Zoom.

We also recognize the need to be able to have devices such as phones, tablets, and other IOT devices (e.g. Cars, Alarm Clocks, etc.) as interfaces to interact with their users. To simplify this, a base Android application is available, including the following capabilities:

Can be set as default assistant on the device
Speech support via the Direct Line Speech service, including the ability to open and close the mic on the device
Ability to render Adaptive Cards
Consume events and engage with events from the local Android OS (navigation, phone dialer, etc.)
UI supporting threaded conversation
Light and dark mode support and easy customization of colors
Much more…

Assistant Samples

As we continue to grow our Virtual Assistant capabilities, as well as providing the ability to start with just the Core component, which does incorporate any pre-configured skills, we have seen the value in providing sample implementations for specific verticals, combining appropriate skills and channels to further accelerate the development of assistants within those industries.

The following assistant samples are currently available.

Enterprise Assistant - Microsoft has assembled a typical configuration of a Virtual Assistant that covers scenarios often required by our Enterprise customers implementing internal facing productivity assistants. This sample provides an implementation of a Virtual Assistant that includes pre-configured capabilities such as weather, news, calendar, to do, and ITSM. Single sign on support (SSO) for Azure Active Directory is also included.
Hospitality Assistant - A typical configuration of a Virtual Assistant that is targeted at the hospitality industry. This sample will provide an implementation of a Virtual Assistant that includes actions such as event information, POI finding, weather, news, hospitality (via the hospitality skill detailed above) etc.

Getting started and documentation

We have overhauled the documentation for Virtual Assistant, with a dedicated site, making it even easier to find information about the Virtual Assistant, its capabilities, customization and deployment. This site will always contain up to date information regarding the latest version of the solution, including steps you can take to migrate to the new releases, ensuring your assistant remains up to date.

To get started today building your own Virtual Assistant, you can find dedicated articles, for both C# and TypeScript, detailing the steps you need to take to get up and running within minutes, with end-to-end scripts to configure your assistant and deploy all of the required Azure resources.
We also provide sample implementations for both continuous integration (CI) and continuous deployment (CD) scenarios within Azure DevOps for both C# and TypeScript.

Looking to the future, now that v1.0 is generally available, we are focused on providing the ability for you to take advantage of the other significant capabilities within the Bot Framework eco-system at Build 2020, such as the ability to develop a virtual assistant (and its connected Skills) using Bot Framework Composer. A preview demonstrating Composer integration and the improved developer experience that comes with taking advantage of the new declarative dialog model that underpins it, can be found at https://aka.ms/bfskillsbuildpreview.

We are excited about hitting this milestone and can’t wait to see the solutions you build, to improve the experiences of your customers!

Introducing Reinforcement Learning on Azure Machine Learning

keijik — Wed, 20 May 2020 04:28:00 GMT

We are excited to announce the preview of Reinforcement Learning on Azure Machine Learning. Reinforcement learning is an approach to machine learning to train agents to make a sequence of decisions. This technique has gained popularity over the last few years as breakthroughs have been made to teach reinforcement learning agents to excel at complex tasks like playing video games. There are many practical real-world use cases as well, including robotics, chemistry, online recommendations, advertising and more.

What is reinforcement learning?

In reinforcement learning, the goal is to train an agent policy that outputs actions based on the agent’s observations of its environment. Actions result in further observations and rewards for taking the actions. In reinforcement learning, the full reward for policy actions may take many steps to obtain. Learning a policy involves many trial-and-error runs of the agent interacting with the environment and improving its policy.

The new reinforcement learning support in Azure Machine Learning service enables data scientists to scale training to many powerful CPU or GPU enabled VMs using Azure Machine Learning compute clusters which automatically provision, manage, and scale down these VMs to help manage your costs.

Learning reinforcement learning with Minecraft

We use reinforcement learning in many ways at Microsoft to improve our products and services. For example, Office uses reinforcement learning to improve the suggestions it makes to users in its apps. To help you get started with reinforcement learning you should check out sample notebooks to train an agent to navigate a lava maze in Minecraft using Azure Machine Learning.

The agent’s goal is to navigate a maze and get to the blue exit tile by walking along solid tiles. If the agent wanders off the solid tiles, it falls into lava and must start over again. Each maze map is randomly generated so the agent must learn to generalize to handle different conditions.

The first step for a data scientist is to develop the training script. The training script for the Minecraft sample is on Github. A typical experience involves iterative development using a combination of local or cloud hosted notebooks, and development tools such as Visual Studio Code or PyCharm. Azure Machine Learning Compute Instance is a cloud hosted Jupyter notebook server that enables rapid iteration using cloud resources.

Once a data scientist creates a Python training script, expanding training to multiple nodes is simple. After creating compute clusters in Azure Machine Learning Studio UI or by using Python SDK calls, the data scientist submits an agent training job using the Azure Machine Learning ReinforcementLearningEstimator. The following example sets up a training configuration to run Minecraft on 8 worker compute nodes to collect training data.

worker_config = WorkerConfiguration( compute_target=cpu_cluster, node_count=8, environment=cpu_minecraft_environment) estimator = ReinforcementLearningEstimator( source_directory='files', entry_script='minecraft_train.py', compute_target=gpu_cluster, environment=gpu_minecraft_environment, worker_configuration=worker_config, max_run_duration_seconds=2 * 60 * 60, shm_size=1024 * 1024 * 1024 * 30) run = experiment.submit(estimator)

Azure Machine Learning automatically allocates compute nodes in the compute target, loads them with container images containing Minecraft and simulation code, and starts running the training script. After training completes, compute nodes automatically deallocate based on user policy to avoid incurring extra charges.

Data scientists track progress of training with multiple methods, including Tensorboard, within the Jupyter notebook, and on Azure Machine Learning Studio. Here we show how the training reward increases over time in Azure Machine Learning Studio.

After training is completed, the agent can be evaluated to see how well it performs. In the animation below, the agent is seen successfully navigating the maze! Training this agent takes around 90 minutes using the configuration in the Minecraft code sample.

Training Agents on Azure Machine Learning

Azure Machine Learning customers are applying Reinforcement Learning on Azure Machine Learning to industrial and other applications. We are seeing Azure Machine Learning customers train reinforcement learning agents on up to 512 cores or running their training over multiple days. In practice, it can take millions of trial runs to train an agent. These trial runs happen automatically, rapidly in parallel and the system continuously learns and improves. Azure Machine Learning uses the Ray framework to distribute reinforcement learning training to support large scale training

To train agents on Azure Machine Learning, data scientists use standard machine learning tools including the Azure Machine Learning Python SDK, the Azure Machine Learning Studio UI to monitor and manage progress, and the command line interface. Azure Machine Learning simplifies running reinforcement learning on remote compute clusters, including tracking experiment results in Tensorboard and Azure Machine Learning Studio.

Get started with reinforcement learning

Learn more about Azure Machine Learning and get started with a free trial.

You can train your own reinforcement learning agents using Azure Machine Learning using the following resources.

How to: reinforcement learning with Azure Machine Learning
Github samples
AI Show: introductory video
Video interview with Katja Hofmann, Director of Game Intelligence Group, Microsoft Research

We look forward to hearing your feedback!

Bringing IntelliSense, collaboration and more to Jupyter notebooks with Azure Machine Learning

abeomor — Thu, 21 May 2020 03:37:31 GMT

This post is co-authored by Maxim Lukiyanov, Principal PM Manager, Azure Machine Learning.

Today, we are pleased to announce the release of an enhanced notebook editor in the Azure Machine Learning Studio. This update enables members of Azure ML workspace to edit, share and collaborate on the notebooks in the same environment that contains their ML experiments, metrics, models, datasets, and more. With the new Studio Notebooks, Data Scientists are one link away from collaboration. We believe this simplicity of sharing will be especially welcome in today’s world of mandatory remote work. The new notebook editor is based on open source nteract project and provides full compatibility with standard Jupyter. The editor also brings some best-in-class code editor features from VS Code that our customers know and love. For the first time, Data Scientists can use advanced features like full IntelliSense and inline error highlighting directly in their Jupyter notebooks.

Getting Started

Here’s how to get started using the new Studio Notebooks experience. Azure Machine Learning workspace is a one-stop-shop for all my Machine Learning needs.
In this workspace, users can easily share all my machine learning assets with teammates. In the notebook experience, users can browse all their files and the files of others on my team. Making it extremely simple to collaborate. Users can start working with a Jupyter Notebook directly in my workspace and have easy access to any of my Experiment details, datasets, models and more.

Live Collaboration Experiences (Coming Soon)

Currently, there are not many ways for data scientists to share notebooks and experiments with their team members in a secure fashion. Even when sharing is successful, it is still difficult to collaborate with others and ensure an experiment will work seamlessly with another user’s computer setup. The whole process can be cumbersome. Data scientists should have access to a tool that can make the authoring and collaboration experiences as easy as possible while ensuring the use of a compute that supports the needs for a specific experiment.

Coming soon, users will be able to collaborate instantly with colleagues using Microsoft Office’s Fluid Framework for seamless a co-editing experience. The Fluid Framework is the same technology Office uses for collaborations, so users can expect similar collaboration functionality and feature as the Microsoft Office Suite. Users will be able to pair debug and unblock teammates easier by being able to live co-edit a notebook.

Built-In Notebook IntelliSense and Improved Editor

When a Data scientist is writing code cell in a notebook, most code is usually error-prone due to typos, syntax errors, or using the wrong function name. Most of these issues are due to the lack of common code editor features like code suggestion or syntax highlighting.
With Studio Notebooks, users will get a code editor experience in a Notebook, every code cell is powered by VSCode’s Monaco editor. When a user starting writing in a code cell, they can use first-in-class code editor features such as IntelliSense, inline error highlighting and code suggestions, variable highlighting, multi-line select, and more. These features that will help boost productivity for anyone typing code, and are even more impactful within a notebook canvas, fitting well into the user’s fast-paced and iterative workflow.

Inline Compute Controls and Notebook Kernel Switching

Occasionally when training a model, a data scientist might need more powerful computing resources or might need to write some code in a language other than Python. Changing compute or installing a need kernel can sometimes be a time-intensive process. With Studio Notebooks this process just got much more efficient. Users can easier control their Azure Machine Learning Compute resource directly from the notebooks. The Notebook toolbar has inline controls to start, stop and create a new compute. Each compute in the dropdown also has added details for all created computes. Backed by the power of the Azure Cloud, users can easily spin up a new GPU or CPU Compute Instance that meets your computing needs, right inside the notebooks.

While using a notebook, users will also be able to add new kernels to the notebook editor and quickly switch between different kernels (Python and R kernel are available by default). Learn more about adding new kernels here.

Automated Notebook Cleanup with Gather (Coming Soon)

When experimenting and prototyping in a notebook, it can often become busy as a user explores different approaches. After eventually reaching the desired result a user would then need to manually curate the cells involved with this specific flow. This task can be laborious and error-prone, leaving users without a strong approach for aggregating related cells. With the Gather feature, users can now easily clean up notebooks with, Gather uses an automated dependency analysis of your notebook, ensuring the essential code is kept, but removing any irrelevant pieces.

R Support

R is an extremely popular language in the Data science community and many Data Scientist use R in their experiment workflow. With this new notebook editor, the R language is supported by default in any notebook via the R Kernel. You can easily switch to the R Kernel and run R code directly in any notebook.

iPyWidget Support

Often data scientist uses widgets to communicate concepts and ideas more effectively in Jupyter notebooks. In the Studio Notebook experience users can create richer notebooks with iPywidgets. The Studio Notebook fully supports all standard ipywidgets. You can now make more captivating and interactive notebooks directly in the Azure Machine Learning Studio.

Try it out yourself

Click ‘Notebooks’ in the Azure Machine Learning Studio and start building Jupyter notebooks with minimum setup. Most popular machine learning packages and the Azure Machine Learning Python SDK come pre-configured with any attached compute instance. You can check out the documentation and video for more tips on how to use Studio Notebooks.

We are excited for everyone to try out this new Studio Notebooks experience and become more productive creating ML experiments with the Azure Machine Learning Studio! We would also love to hear about your experience on the new and upgraded version so please send us your feedback!

Build predictive maintenance, conversational user interface and powerful analytics at the edge

Phani_Mutyala — Tue, 19 May 2020 15:00:03 GMT

With Azure Cognitive Services in containers businesses across industries have unlocked new productivity gains and insights. Organizations ranging from manufacturing, legal to financial services have transformed their processes and customer experiences as a result. Azure is the only cloud provider supporting customers giving full flexibility to run AI in their own terms (on-prem, cloud or at the edge).

In general, developers need to have right data science skills to build, train and run models, but Cognitive Services brings AI within reach of every developer with pre-built and customizable AI that can be embedded into apps with the programming languages of your own choice. With AI container support, we took one step ahead empowering developers to deploy hybrid solutions. This solves current challenges with data control, privacy, and network intensive workloads.

The goal of this post is to show several examples of how enterprise applications leverage containers to solve AI needs at the edge. Learn more about how customers built solutions using Cognitive Services Containers

Use case #1 – Anomaly detection in factory equipment using predictive maintenance solution
Tibco - TIBCO, a strategic partner of Microsoft, has a market offering that incorporates two containerized Cognitive Services into their Anomaly Detection solution.

TIBCO unlocks the potential of data assets for making faster, smarter decisions with their Connected Intelligence Platform. The platform allows you to seamlessly connect to any application or data source; unify data for greater access, trust, and control; and confidently predict outcomes at scale in real-time.

The TIBCO anomaly detection solution includes Azure Cognitive Services container deployment with text mining and root cause analysis. Anomaly detection and analysis provides value across nearly every industry including energy, financial fraud and risk, algorithmic insurance, connected vehicles, healthcare and insurance claims, and manufacturing fault detection and yield optimization.

This blog focuses on manufacturing, identifying anomalies for asset management. This specific example uses machine learning techniques to detect anomalies, understand root cause from related text data, and alert case managers when sensor readings are deviating from expected patterns. This enables operators to implement condition-based maintenance interventions before equipment failure, and to prevent costly manufacturing process shutdown.

Reference Architecture:

Solution overview:

Manufacturing, energy, mining and power plant customers use TIBCO Data Science to detect anomalies on historical data from various facilities at scale. This TIBCO Data Science workflow produces a portable analytics model object that the maintenance engineer can call remotely, and also bring to the engineering site and invoke using TIBCO Spotfire locally.

TIBCO Spotfire calls the Azure Containerized Cognitive services using a Spotfire data function via a python API. As the maintenance engineer detects anomalies onsite, Spotfire’s brush-linked visual analytics and data science tooling runs a root cause anomaly analysis, After the anomalies are detected, root cause factors are identified for a specific time window as part of the analysis. These analyses are combined with results from the Text Analytics containerized service using key phrase extraction to determine the recommended actions to be taken.

Let’s see how this solution built in few steps:

Phase 1 - Anomaly Detection on historical data: Here is the TIBCO Data Science platform which is used for detecting anomalies across all the manufacturing sites.

What you see in the image below is power plant data. The workflow consists of multiple steps including data pre-processing, transformation of time-series data, filtering and ultimately calling the MS Azure services from a TIBCO Data Science operator designed to invoke the services.

Phase 2 – Detecting Anomalies at remote site: Once the maintenance engineer is at a remote location that may not be connected to the internet, the first step is to perform anomaly detection analysis using the Spotfire equipment maintenance dashboard by selecting different features as shown below.

The user selects a response metric and granularity, and the anomalies are detected based on historical data collected at the particular site. As shown in the image below, the upper section shows the original data readings along with the expected values provided from the containerized service, and the bottom section shows the difference between the original and expected values over time. The red markers are anomalies detected by the Anomaly Detection routine.

Phase 3 – Performing root cause analysis: The site engineer investigates anomalies from a certain time window to perform a root cause analysis. On the image below (bottom left portion), key driving factors indicate what factors are contributing to the anomalies in production per minute for the time window selected by an engineer using the TIBCO Spotfire analytics application.

At this point, the site engineer digs deeper to understand the maintenance action items performed prior to these anomalies occurring by getting insights from log data using key phrase extraction from the Text Analytics container as indicated in the image below in the top section (Get Insights).

From here, TIBCO Spotfire generates a recommendation for next steps. In the example shown, the ‘BaroPressure’ sensor was restarted 5 times vs. being recommended for replacement just once. So, the equipment has to be replaced to avoid any further shutdowns.

In summary, initially detecting anomalies in power plant data across all sites on historical data using TIBCO Data Science. This produced a portable analytics model object that the maintenance engineer can take to the remote site. At the site, all the recommended actions by leveraging containerized cognitive services from Azure combined with decision intelligence embedded in TIBCO Spotfire using root cause analysis.

This use case is just an illustrative example and one of the many ways how customer’s use TIBCO’s analytics and data science capabilities to solve some of the most complex business problems such as pricing and promotion optimization, supply chain analytics, fraud detection, yield enhancement, and real-time process control.

To learn more about TIBCO’s anomaly detection solution, visit Anomaly Detection at TIBCO.

Use case #2 – Organizing unstructured data and surface insight in Power BI dashboard

Wilson Allen is a systems integrator with deep analytic experience serving the legal. Given the confidentiality of the work involved, the data must stay local. Law firms use Wilson Allen’s Proforma Tracker to manage their pre-bill processes. Wilson is using cognitive services to recognize text, optimize legal templates and process them for billing, translate and ensure they are in compliance with regulatory laws as well as enabling editing via hand writing and capturing this digitally.

Architecture-wise, Wilson Allen is using Azure’s AI in the form of containerized Cognitive Services to bring form-based data into firm’s data ecosystem and to create meta data from text-based data, using ML techniques to establish patterns, causality relationships, and to render it all in a common data model without the traditional data warehouse visualized through Power BI reporting.

Using Form Recognizer, they are also extracting historic sets of HR PDF forms (intake, change, termination) to create a database for master data and analytic purposes. Additionally, they are using Sentiment Analysis for marrying structured and unstructured client feedback data then surfaced in Power BI.

Containers used: Form Recognizer, Text Analytics (Sentiment Analysis)

Use case #3 – Build conversational user interface to help retailers manage inventory better, help bankers improve support experience, provide healthcare customer with better guidance in manage appointment and track efficiency.

Insight - Insight’s Kiosk Framework leverages the Cognitive Services containers to create a functional experience for the customer with a minimal connectivity requirement. Using the Azure IoT Edge runtime and IoT hub, the framework can be managed remotely and securely allowing the customer to change the “personality” of the kiosk remotely based on the business requirements and constraints around image capture, responses and content. It also allows the customer to manage the deployment and management of this framework at scale across locations.

Containers used: Face Recognition, Speech to Text, Language Understanding

Get Started...! Learn more and deploy AI at the edge with Cognitive Services

Deploying your first container is about a 2-minute read. You basically create a resource at Azure portal, download image, run container with environmental variables. Here's a guide help you get started on running containers.

Best Practices

Let’s look at some best practices that might be helpful for you to run containers:

Container security

Security should be a primary focus whenever you're developing applications. The importance of security is a metric for success. When you're architecting a software solution that includes Cognitive Services containers, it's vital to understand the limitations and capabilities available to you. For more information about network security, see Configure Azure Cognitive Services virtual networks.

The diagram below illustrates the default and non-secure approach:

As an alternative and secure approach, consumers of Cognitive Services containers could augment a container with a front-facing component, keeping the container endpoint private. Let's consider a scenario where we use Istio as an ingress gateway. Istio supports HTTPS/TLS and client-certificate authentication. In this scenario, the Istio front end exposes the container access, presenting the client certificate that is whitelisted beforehand with Istio.

Nginx is another popular choice in the same category. Both Istio and Nginx act as a service mesh and offer additional features including things like load-balancing, routing, and rate-control.

Container networking

Cognitive Services containers are required to submit metering information for billing purposes. Failure to allow list various network channels that the Cognitive Services containers rely on will prevent the container from working.

Allow list Cognitive Services domains and ports

The host should allow list port 443 and the following domains:

*.cognitive.microsoft.com
*.cognitiveservices.azure.com

Disable deep packet inspection

Deep packet inspection (DPI) is a type of data processing that inspects in detail the data being sent over a computer network, and usually takes action by blocking, re-routing, or logging it accordingly.

Disable DPI on the secure channels that the Cognitive Services containers create to Microsoft servers. Failure to do so will prevent the container from functioning correctly.

Get started today by going to Azure Cognitive Services to build intelligent applications that span Azure and the edge. For more information, please refer to container documentation page.

ONNX Runtime Training Technical Deep Dive

SherlockNoMad — Thu, 29 Oct 2020 17:00:30 GMT

Author: Sherlock Huang, AI Frameworks, Microsoft

This post is co-authored by Cheng Tang, Jesse Benson, Kaarthik Sivashanmugam and Alexey Svyatkovskiy

Today we announced the preview for new training feature in ONNX Runtime (ORT). This blog explains how we have been using it to accelerate training for large transformer models. ONNX Runtime Training is integrated with PyTorch so that existing training code can be directly accelerated for training.

In this paper, we will describe some of the key aspects of ORT design and implementation that enable us to achieve the distributed training performance improvements. We will also use BERT-L pre-training as the benchmark to illustrate the performance of ORT training. Finally, we will present a case study of training GPT-2 model for code autocompletion feature in Visual Studio IntelliCode.

Design and Implementation

ONNX Runtime Training is built on the same open sourced code as the popular inference engine for ONNX models. Figure 1 shows the high-level architecture for ONNX Runtime’s ecosystem. ORT is a common runtime backend that supports multiple framework frontends, such as PyTorch and Tensorflow/Keras. It makes use of the Execution Provider interface to perform computation on different hardware. This enables us to build hardware-agnostic, graph-level optimizations that are extensible across different platforms, as well as hardware specific optimization targeting platforms like NVIDIA GPU. We have also implemented additional optimizations, outlined below, to expedite training for large transformer models.

Figure 1. ONNX Runtime High Level Architecture

Static Graph Optimizations

Machine learning models are commonly abstracted as computation graphs. The computation graph used by deep learning frameworks could be either static or dynamic. In the current implementation, ORT has a view of the entire static computation graph. This makes it possible to enable many common graph optimization techniques, such as constant folding, redundant operation elimination, and operator fusion. They are first applied on the forward computation graph before auto differentiation engine builds the backward graph. As ORT has the global knowledge of data dependencies, it only builds the minimal gradient graph that is needed for targeted weights. Consequently, activation tensors that are not needed for backward computation are automatically dropped after use. With a minimal training graph, it ensures that only essential computation is performed and memory consumption is minimized.

Memory Usage Optimizations

Over the last few years, the size of deep learning models has been growing rapidly. GPU memory consumption has become a limiting factor for large model training. ORT has made conscious efforts to preserve and reuse memory whenever possible. For example, ORT reuses the same buffer segments throughout a series of operations, including gradient accumulation, gradient scaling adjustment, allreduce communication and weight update computation (if the optimizer allows). ORT also tries to perform in-place operations if the source tensor is no longer consumed elsewhere in the computation graph. ORT’s kernel implementation also tries to minimize the use of scratch buffers, such as avoid using some memory intensive cuDNN functions, and reusing output buffer as scratch buffer if possible. As a result, ORT can train BERT with 2x the batch size as PyTorch. This enables us to utilize the GPU resources more efficiently, resulting in better performance on the same model and the ability to train larger models.

ZeRO Stage 1 Integration

Zero Redundancy Optimizer (ZeRO) is a memory optimization technique from Microsoft Research. ZeRO is used to save GPU memory consumption by eliminating duplicated states across workers during distributed training. ZeRO has three main optimization stages. Currently, ONNX Runtime implemented Stage 1 of ZeRO. ZeRO Stage 1, known as the optimizer state partitioning, allows ORT to shard the optimizer states, including 1^st and 2^nd order moments (and fp32 copy of weights in mixed precision mode), across multiple workers with no extra communication overhead. With ZeRO, ORT can further boost batch size or train a larger model. In BERT-L pre-training, ZeRO allows batch size to further grow from 148 to 168 for phase 1 and from 23 to 27 for phase 2 in a 32GB V100. Distributed checkpointing is also introduced, as model persistent state is distributed across multiple workers. ZeRO can be enabled with a config flag.

Native Mixed Precision Training Support

Unlike PyTorch’s dependency on NVIDIA Apex extension, ORT has implemented its own support for mixed precision training. Mixed precision training can be enabled with a config flag – no other code change needed. Under the hood, ORT converts the static computation graph into mixed precision mode through a series of graph transformations, i.e. running most of the computations in fp16 while keeping some numerically sensitive computation in fp32. ORT supports dynamic loss scaling by automatically inserting the computation nodes for loss scaling into the graph.

Highly Scaleable Distributed Training

ORT seeks to build a unified highly scaleable distributed training framework for hybrid parallelism, including a mixed of data and model parallelisms. ORT supports data parallelism, which is the most popular distributed training mode adopted by many internal teams. We are enhancing ORT to fully support training extremely large models (>100 billion parameters). It has an experimental implementation of Megatron-style horizontal parallelism and we are actively developing to support pipeline parallelism, such as PipeDream.

CUDA Kernel Optimizations

ORT has introduced highly optimized CUDA kernels for some key operations including Reductions, Dropout and Softmax. In addition, we have also introduced a few key operator fusions with fused kernels for LayerNormalization, Gelu and their gradients, as well as Lamb Optimizer.

Using ORT with PyTorch Training Code

ONNX Runtime has the capability to train existing PyTorch models through its optimized backend. For this, we have introduced a python API for PyTorch, called ORTTrainer, which can be used to switch the training backend for PyTorch models (instance of torch.nn.Module) to ORT. This requires some changes from the user, such as replacing the PyTorch optimizer, and optionally, setting flags to enable additional features such as mixed-precision training. Under the hood, as shown in Figure 2, ORTTrainer first converts the PyTorch model to ONNX format through the PyTorch-ONNX exporter. Next, ORT backend takes over and applies graph optimizations, builds a training graph, performs transformations on it as needed (e.g. mixed-precision transformation), and sets up the graph elements needed for distributed training. In this design, while all the computation-intensive workload is offloaded onto the ORT backend, users can still enjoy the rich PyTorch frontend utilities, such as data loading, checkpointing , and easy specification of loss functions.

Figure 2. Workflow for converting an PyTorch model into an ORT training graph

It is important to note that the current API is experimental and expected to see significant changes in the near future. A new version of the API is under active development. Our goal is to improve the interface to provide more seamless integration with PyTorch training that requires minimal changes in users’ training code, introduce new features, and present a more flexible API to cover advanced scenarios. Please refer to the training examples for more details.

Benchmarking Training Acceleration with ONNX Runtime

We now present the performance evaluation of BERT-L pre-training with ONNX Runtime in a 4-node DGX-2 cluster. In AzureML, we also reproduced the pre-training convergence for BERT-Large using sample from NVIDIA’s DeepLearningExamplesle’s repo. We also validated fine tuning accuracy with SQuAD benchmarks.

Benchmarking on DGX-2

We compared PyTorch and ORT’s BERT-L training performance on 4 NVIDIA DGX-2 machines (each with 16x 32GB V100) interconnected with InfiniBand. PyTorch’s result was obtained with NGC 20.03-py3 docker image following Nvidia’s recipe. ORT’s result was obtained following the same recipe, except that ORT used bigger local batch sizes. As described above, ORT is able to run at a 2x batch size of PyTorch’s. ORT ran at a local batch size of 128 and 16 for phase 1 and 2 respectively, whereas PyTorch ran at batch size of 64 and 8. The effective global batch size remained unchanged in both cases. Overall, ORT achieved throughput improvement of 11.32% and 14.61% for phase 1 and 2. The total time to train was reduces by 11.16%, from 17.74 hours to 15.76 hours.

Table 1. Time to train on 4 NVIDIA DGX-2 machines
	PyTorch 1.5 with NGC 20.03-py3	PyTorch 1.5 with ONNX Runtime	% Gain with ONNX Runtime
Phase 1 Throughput (ex/sec)	11522.1	12826.2	11.32%
Phase 2 Throughput (ex/sec)	2150.0	2464.1	14.61%
Phase 1 time (hours)	11.12	9.99	10.16%
Phase 2 time (hours)	6.62	5.77	12.84%
Total time (hours)	17.74	15.76	11.16%

BERT-L Pre-training on AzureML

We performed BERT-L pre-training on 8x ND40rs_v2 cluster (each with 8x 32GB V100) interconnected with InfiniBand in AzureML. We used the same Nvidia’s recipe, expect that we doubled the local batch size in the same way we mentioned above. Mixed precision mode and LAMB optimizer was used throughout the training. As the end of phase 2, we achieved the training loss of 1.31. The end-to-end training time was 18.32 hours.

Table 2. Time to train on Azure ML with 8x ND40rs_v2
	PyTorch 1.5 with ONNX Runtime
Phase 1 Throughput (ex/sec)	10751.4
Phase 2 Throughput (ex/sec)	2223.7
Phase 1 Time (hours)	11.92
Phase 2 Time (hours)	6.40
Total Time (hours)	18.32

Figure 3 shows a loss curve produced in a typical pre-training run. Phase 1 ends with a loss value around 1.4 after 7038 steps. Phase 2 continues with a jump of loss due to switch of sequence length, and it finally decrease to a loss value around 1.3.

Figure 3. ORT BERT-L pre-training loss curves

The pretrained model is then further finetuned on SQuAD dataset. Both full precision or mixed precision finetuning result in satisfactory Exact Match and F1 scores.

Table 3. BERT-L fine-tuning result on SQuAD Dataset
Accuracy Metrics		Finetuning - FP32	Finetuning - mixed precision
Exact Match %		84.63	84.81
F1 score %	91.15		91.32

A Case Study with Visual Studio using GPT-2 Medium

Microsoft Visual Studio uses ONNX Runtime to accelerate pre-training a 24-layer GPT-2 Medium model to power code autocompletion in the IntelliCode of Visual Studio. Intellicode serves as a universal programming language compiler, effectively generating syntactically correct code in multiple programming languages, capable of completing an entire line of code in a couple of keystrokes. The training dataset for this task comprises over 1.2 billion lines of source code in Python, C#, JavaScript and TypeScript programming language from 52000 top-starred projects in GitHub. We treat the source code data as a sequence of tokens corresponding to the output of a lexical analyzer.

The training was performed in a DGX-2 cluster. As we use a large sequence length of 1024, the memory usage is very intensive and PyTorch is only able to fit a batch size of 2 on the 32GB V100. ORT achieved 15.8% higher throughput under the identical local batch. As ORT is more memory efficient and able to run at a bigger batch size of 3, it delivered an overall 20.5% of the throughput improvement. As a result, the overall training time is reduced from 202 hours to 168 hours (with 1.2 x higher throughput). The final evaluation metric also achieved the same production shipping bar.

Table 4. GPT-2 medium pre-training performance.
	Batch size / GPU	Throughput (ex/sec)	Time to train (hours)
PyTorch	2	48.7	202
PyTorch + ORT	2	56.4	174
Pytorch + ORT	3	58.7	160

Conclusion

Today, we announced the preview of training support in ONNX Runtime with a focus on large scale computation intensive transformer models. We have demonstrated that, on a 4 DGX-2 cluster, ONNX Runtime can achieve a throughput gain of 11.32% and 14.61% for BERT-L phase 1 and 2 pre-training over PyTorch. The total training time was reduced by 11.16%, from 17.74 hours to 15.76 hours. ONNX Runtime is able to train BERT-L at a 2x batch size as PyTorch. We have shown a similar 20.5% speedup on a GPT-2 model, saving 34 hours in total training time. ONNX Runtime Training is integrated with PyTorch so that existing PyTorch training code can be directly accelerated for transformer models training.

Get Started

As a part of the announcement on using ONNX Runtime for training, we have released a Docker image with ORT and made available a repo at https://github.com/microsoft/onnxruntime-training-examples that will host examples for ORT training. The first recipe available in this repo will help you get started with ORT for BERT pretraining in Azure Machine Learning service or NVIDIA DGX-2 and see the speedup in action. This recipe shows how to use ONNX Runtime training with BERT pretraining implementation in PyTorch. You can use this example either with the two datasets used in the original implementation or with your custom dataset to pretrain a BERT model and get the performance improvements with ORT reported in this blog. We are planning to add more examples for transformer models and other models. We also welcome your contribution to this repo and feedback to improve ORT training capabilities and experience.

Training deep learning models at scale in Azure

Chris Lauren — Wed, 20 May 2020 16:19:13 GMT

This post was co-authored by @Chris Lauren , @Ian Finder , @David_Aronchick , Maxim Lukiyanov, Gopi Kumar

Microsoft, like much of the tech industry, has adopted deep learning to power features across our business and accelerate the value we can provide for users. We use deep learning models to improve user productivity and provide innovative experiences in Office, power code autocompletion in Visual Studio, improve search results in Bing, predictively optimize availability in Azure compute among many other scenarios across the company.

Part of what allowed us to move quickly to embrace this world changing technology was investing in the infrastructure and services necessary to increase our data scientists and machine learning engineers’ productivity in a cloud-first, highly scalable manner. The combination of our investments, tooling and experience has allowed us to make big advancements in the efficiency of our teams. When we launched Azure Machine Learning to GA 18 months ago, our goal was to bring our productive and powerful environment to users of all skill levels who want to leverage machine learning and accelerate the practice of data science.

Today, any ML developer can use Azure to experiment with innovative models, run distributed model training and deploy trained models quickly to infuse more intelligence into their business applications. The same machine learning platform, tools and powerful compute are part of Microsoft’s AI at Scale initiative that is enabling the next generation of AI capabilities

Let’s take a deeper look at the Azure AI infrastructure and the Azure Machine Learning service which together comprise the machine learning platform we use to power our AI innovations across Bing, Office, Teams and more.

How Microsoft uses AI at global scale to increase our user's productivity

One example of how Microsoft uses the Azure AI infrastructure and Azure Machine Learning service is the new “suggested replies” feature in Outlook. When you receive an email that can be answered with a quick response, Outlook suggests three responses that you can use to reply quickly, reducing the time and effort involved in replying to an email. This feature is powered by large scale deep learning natural language processing (NLP) model trained in Azure on powerful GPUs using Azure Machine Learning.

The Microsoft Outlook "Suggested Replies" feature uses Azure Machine Learning to train deep learning models at scale

The Outlook team uses Azure Machine Learning pipelines to process their data and train their models on a recurring basis in a repeatable manner. During the model training, the team uses GPU pools available in Azure. Once the model is created, data scientists can compare the model performance with previous models and evaluate which approaches perform better at recommending relevant suggested replies. Additionally, by using accelerated training with ONNX Runtime they were able to get up to an additional 45% improvement in training performance with minimal changes to their existing model training code. This increased the team's productivity even further by enabling more frequent machine learning experiments.

Training models at this scale and frequency would not be possible without the scalable Azure AI infrastructure and Azure Machine Learning service.

Powerful Azure AI infrastructure

Azure offers best-in-class infrastructure for AI workloads of all sizes, supporting GPU acceleration for popular frameworks like TensorFlow, PyTorch, and others from a single GPU, up to the flagship NDv2 VM offering eight 32 GB NVIDIA V100 GPUs with NVLink, as well as cluster-level 100 Gigabit InfiniBand EDR with out-of-box NCCL2 support to allow jobs to transparently harness the power of close to 1,000 GPUs concurrently.

Azure's virtualization technology harnesses features of the underlying hardware to offer nearly identical performance and architectural behavior to bare metal, along with the security and manageability benefits of virtual machines. Even at the driver level, Azure VMs employ standard NVIDIA device drivers, and the same Mellanox OFED InfiniBand RDMA drivers a customer might use on-premise, ensuring a rapid and seamless lift for existing AI workloads to begin leveraging Azure.

Going forward, Azure will continue to invest in the promise of distributed training and interconnects with the latest technologies to deliver higher bandwidth and scale to larger clusters of future GPU products, like NVIDIA’s new A100, with the same commitment to standard communication methods used by workloads such as NCCL2.

Azure Machine Learning service

Azure Machine Learning service runs on top of the Azure AI Infrastructure and provides a complete solution to manage the end to end machine learning lifecycle: preparing data, building/training models, deploying the models to cloud or the edge, and monitoring models performance to determine whether to retrain them on new data to improve over time.

Machine learning lifecycle using Azure Machine Learning service

Training large scale deep learning models on a budget

Using the latest GPU hardware at scale can be expensive. However, Azure Machine Learning makes it easy to minimize infrastructure costs to meet AI development budgets. AML has cost management and budget controls which enable your teams to share AML compute clusters which provision powerful GPU and CPU VMs on demand to train large scale models and turns them off when they are not being used. Additionally, you can further control costs using role based access control and quota management.

Pre-built environments for machine learning frameworks

Data scientists spend a lot of time preparing environments which contain combinations of open source software libraries. These libraries are tested individually, but as data scientists create their own software environment (often in a docker container) they must test the different versions of these libraries on their own to resolve version conflicts. This is a time consuming process which does not directly accrue value to training great models.

Azure ML reduces this complexity by providing pre-built environments for popular machine learning frameworks, and their popular distributed training flavors. Among supported frameworks are standard PyTorch, TensorFlow, their native distributed training backends, popular distributed training framework Horovod, and variety of communications protocols such as MPI, NCCL or Gloo. Azure Machine Learning makes it easy to explore new distributed models or take existing model from research community and run it as is on Azure compute with minimum or no modifications.

Additionally, Azure Machine Learning makes it easy to collaborate with other data scientists on your team using our new preview integrated Jupyter Notebooks. This further increases data scientists productivity and leverages nteract to bring Notebooks right into the AML Studio UI.

Azure Machine Learning's new preview integrated Jupyter notebooks will soon offer collaboration capabilities

Scalable experimentation platform

Building new machine learning models requires iterative experimentation; sometimes it takes hundreds of iterations over the model design, algorithms and hyperparameters to achieve optimal performance. Some experiments that appear promising initially may yield poor results and researchers will have to step back and reassess results from the previous experiments.

As data scientists experiment in this way, it becomes increasingly important to share experiment results easily, reproduce any experiment reliably and collaborate with their team using a platform that is able to scale to handle large models that take days to train and produce GBs of output metrics per run.

Azure ML provides a fully managed, highly scalable machine learning platform. ML developers can organize their model training runs into experiments, track with each run all of the relevant parameters and metrics of the model, versions of training data used, source code, git commit, hyperparameters and more. With all the relevant data tracked by default, Azure ML makes it easy to compare the experiment runs and determine which produced the best model to deploy.

Get started with Azure Machine Learning for free today!

Learn more:

Build 2020 - Conversational AI updates

GaryPrettyMsft — Thu, 21 May 2020 18:06:19 GMT

Conversational AI updates that help you build sophisticated and personalized experiences

Now, more than ever, developers need to respond to the rapidly increasing demand from customers for support and accurate information - meeting them where they are – any time of the day and on an expanding range of platforms and devices. Within just the last few weeks, Azure AI has met unprecedented demand, underpinning over 1500 Covid-19 related bots via the Microsoft Health Bot service alone, in addition to the over 1.25 billion messages per month already handled by Azure Bot Service.

As part of our key updates for Build 2020, we continue to improve the developer experience and answer the evolving needs of enterprises looking to implement conversational experiences, both employee and customer facing. Significant announcements include the general availability (GA) of Bot Framework Composer, an integrated development tool for building conversational experiences, and the Virtual Assistant solution, an open source solution for building a branded virtual assistant. Azure Bot Service brings a public preview of Alexa integration along with new capabilities for the Language Understanding, Speech and QnA Maker Cognitive Services, including the general availability for container support.

Bot Framework Composer is now GA!

Now generally available, Bot Framework Composer is a new open source, visual authoring canvas for developers to design and build conversational experiences. Composer focuses the bot creation process more on conversation design and less on the scaffolding required to begin building awesome bots. Composer easily brings together the common components required to build bots such as the ability to define Language Understanding models, integrate with QnA Maker and build sophisticated composition of bot replies using Language Generation.

Composer also supports building Bot Framework Skills (bots that can perform a set of tasks for another bot allowing for re-usability and componentizing bot solutions as their complexity and surface area increases. Skills built with Composer can be consumed by other bots built with Composer or using the Bot Framework SDK, as well as from Power Virtual Agents.

Find out more and get started with Composer at https://aka.ms/bfcomposer.

Bot Framework SDK v4.9

Build 2020 sees the release of version 4.9 for the JavaScript, C# and Python SDKs and our commitment, to meet developers where they are continues with the release of Bot Framework Java SDK Preview 4. This latest preview brings the ability to build bots for Microsoft Teams, aligning with the existing capabilities in the JS, C# and Python SDKs, including conversation bots, messaging extensions, and broad API and event coverage for the platform.

We are also excited to make Adaptive Dialogs generally available! Adaptive Dialogs, which underpin the dialog design and management in Composer, enable developers to dynamically update conversation flow based on context and events. This is especially useful when dealing with more sophisticated conversation requirements, such as context switches and interruptions (link to adaptive docs?). Bot Framework Skills can now also leverage Adaptive Dialogs.

Also available in an early preview is new Generated Dialog tools. These new tools can automatically create robust Bot Framework Composer assets from JSON or JSON Schema that implement best practices like out-of-order slot filling, ambiguity resolution, help, cancel, correction and list manipulation.

Other significant updates include a Developer Preview of Single Sign-On (SSO) capabilities in Microsoft Teams, answering a common requirement from our customers and, ultimately, reducing friction for end users. A new Health Check API allows for monitoring of bots in Production environments.

For more details of all of the changes in this latest release, see the version 4.9 release notes.

Azure Bot Service

The Direct Line App Service Extension is now generally available and enables customers to have even greater control over how data is stored and transmitted within their bot using Direct Line or Webchat. Often customers in industries such as banking, medical, legal and others deploy their solution into Virtual Networks (VNETS) which provides networking isolation capabilities. With the Direct Line App Service Extension (Direct Line-ASE), they can now deploy their bot inside the VNET and connect directly to their users’ clients rather than data passing through shared cloud infrastructure. In addition, Direct Line-ASE uses web sockets for communication between client and bot which can reduce latency as well.

A common scenario, repeatedly encountered by our customers, is the need for human handoff as part of a conversation where it is most appropriate, or a customer explicitly asks to speak to a human. Implementing such scenarios within your own bot could, historically, be complex and we are aiming to reduce the implementation time from weeks to minutes by making pre-built integrations for popular customer service platforms available, including LivePerson and Microsoft Omnichannel. Plus, if an existing integration does not already exist developing your own is now much easier with Microsoft now providing common patterns, backed by updates to the Bot Framework SDK and protocol.

Whilst Azure Bot Service already provides a broad range of channels, customer requirements continue to evolve leading to demand for additional integrations. As part of responding to these demands, we are pleased to announce the public preview of a new channel for Amazon Alexa Skills, allowing you to build a bot that targets the popular home assistant platform, alongside the existing channels you already build for today. For more details on configuring the new Alexa channel preview, see the updated Bot Framework docs.

Virtual Assistant 1.0 now generally available

Now generally available, Virtual Assistant Solution Accelerator, has now reached version 1.0. Virtual Assistant allows developers to quickly stand up a fully functional Virtual Assistant that can be modified to be their own unique experience.

Virtual Assistant now has been fully moved over to Bot Framework Skills. Virtual Assistant Sample Skills have been moved into their own GitHub repository to allow for easier update to the Virtual Assistant Core as well as the Sample Skills that developers have used in their implementation. Virtual Assistant adds all of the items below together to provide the best starting point for developers looking to quickly provide a Bot that has the core components needed to be able to scale and work right out of the box.

Virtual Assistant has also added new capabilities as we move beyond v1.0 to allow developers to see how to leverage Bot Framework Composer to create skills for Virtual Assistant. This allows developers to unlock the power of Adaptive Dialogs. Virtual Assistant has added 3 new Preview Skills that are Composer / Adaptive versions (Calendar, To Do, Who).

Read a deeper overview of version 1.0 and get started with Virtual Assistant at http://aka.ms/VirtualAssistant.

Speech, Language Understanding and QnA Maker

Azure Cognitive Services brings AI within reach of every developer—without requiring machine-learning expertise. At Build 2020, we made several announcements related to new features and improvements across the Cognitive Services used within the Conversational AI eco-system.

The Speech service is broadening language coverage and updating Speech to Text and neural Text to Speech with significant accuracy improvements. Additional new capabilities such as custom commands and pronunciation assessment are making it easier for customers to embed advanced speech capabilities into their solutions.

The Language Understanding service has released a major update to the portal, with a dramatically improved labeling experience, making it easier than ever to build apps and bots that can understand the complex language people tend to use. For example, somebody ordering a pizza might say, “I want two large chicken deep-pan pizzas, a medium pizza with olives and a side of fries.”. This is a complex order, but using the new machine learned entity labelling and decomposition allows you to extract actionable data with ease (full order, quantities, toppings, modifiers and sides). The user has used two different language structures within the same order. This new portal makes it easier to break apart complex requests into related parts.

In addition, Language Understanding, as well as Text Analytics, can now be deployed from the cloud to the edge, with containers support for both services now generally available! For more detail, read the Language Understanding blog.

Bot Framework Orchestrator enters private preview and provides a transformer model based orchestration capability optimized for Conversational AI. This capability helps deliver improved accuracy of Skill based routing critical to more sophisticated conversational experiences, reduced latency and a multi-label classifier enabling multiple intents to be identified from utterances and processed individually. Moving forwards this capability will replace our current Dispatch capability.

The QnA Maker service also receives an update to the editing experience for QnA Knowledgebases, with the addition of rich text editor support, as well as the addition of Role Based Access Control (RBAC) allowing for greater control and governance of knowledgebase management.

Conversational AI Build sessions and on-demand videos

We encourage you to find out more about our announcements via our Build sessions, either live or on demand afterwards. We also already have a number of on-demand content available now.

Build Breakout sessions

Conversational AI powered Customer and Employee Virtual Assistants

1st session Wednesday May 20th - 2:00 - 2:30 pm PST. Check the link below for all session times.

https://mybuild.microsoft.com/sessions?q=INT139

Accelerate bot development in Power Virtual Agents

1st session Tuesday May 19th - 3:00 - 3:30 pm PST. Check the link below for all session times.

https://mybuild.microsoft.com/sessions?q=INT155

Deploying Voice Assistants for driverless vehicles

https://mybuild.microsoft.com/sessions/e5f46ac9-65f4-4e50-a3d4-f76a046ffd51

On demand content

Bot Framework Composer: Bot Framework’s new collaborative Conversational AI development environment
https://youtu.be/r9WQPSaLnaU

Use the Efficiency of Low-Code with the Extensibility to Azure to Design World-Class Chatbots

https://youtu.be/oJWJA-U4-m8

Conversational AI and human agents working together

https://youtu.be/Z0IDiekbOp4

Author rich content in QnA Maker knowledge base and enable role based sharing

https://www.youtube.com/watch?v=lB9klrdrqOk

New features in Language Understanding

https://youtu.be/Zu6cYF7y9os

Self-Driving Vehicle Systems in a Post COVID-19 World

https://mybuild.microsoft.com/sessions/6abe0cd4-ee29-4f6b-abdf-521fce76f54e

Build 2020 - Language Understanding (LUIS) new portal, tools and container support

AliciaEP — Thu, 21 May 2020 19:37:44 GMT

We’ve listened to the feedback from our customers on how to create more accurate models while making the service even easier to use with several core Language Understanding enhancements. In addition based you your feedback we’re containers generally available in June. Finally, for developers that want to integrate Language Understanding into their CI/CD and release management pipelines, we’re previewing a sample repository template.

I cover these enhancements in more detail on the AI Show with Seth Juarez, but have also captured the key points below.

Core language understanding enhancements

Upgraded machine learned entities replacing composite and simple entities

We’ve introduced the ability to add sub-entities to machine learned entities, going up to 5 levels deep. This replaces composite entities and gives you more power to recognize more sophisticated entities, reuse them across your application and even recognize multiple actions in a single utterance. In addition, this top-down thinking to build a schema is more natural than the bottom up thinking that was needed when creating composites.

If you have an application that used the old composite entities, you can easily upgrade that app to use the updated machine learned entities to take advantage of this new functionality. This upgrade is seamless for you – you do not need to re-label any of your entities, and no changes are needed in your code. The upgrade experience creates a new version of your application for you to give you the option of testing it separately.

Not only can you build and recognize more sophisticated entities, but an added benefit of sub-entities is that it can improve your model’s accuracy by using entities as features.

Updated portal

Another piece of feedback we received is you can lose context when updating your entities and features while labeling. To address this, we’ve added the new entity palette (pictured below) which allows you to see all the ML entities and list entities you’ve created while you’re labeling new utterances. You can also edit your entities and add or edit features while labeling utterances.

Screen shot of the new portal entity palette.

Improved labeling tools

We’ve listened to your feedback about difficulties with labeling utterances and there are several changes to the interface in the portal to make this interaction easier. Now you can label entities from both the new entity palette or inline.

When you use the entity palette you can label a child node and the parent will be inferred, and when you label the parent it will automatically merge. Choose the entity labeler tool, then select the entity you want to label for and highlight it in the utterance.

Labeling with the new entity palette experience.

For customers that prefer inline labeling, we have improved that as well. Inline labeling supports labeling entities in any order with a cascading menu.

Labeling with enhanced inline labeling experience.

In addition, predictions are shown with a dotted line when a new utterance is added. If all the predictions are correct for a new entity, you can confirm them all in one click.

Viewing entity predictions and confirming to label.

Normalized word forms

We have addressed customer issues around recognizing variations of a word, for example, changing 'flight' to 'flights' would show very different results for intent predictions. To solve this, we've added a setting called 'Normalize word forms' that will help your model recognize plurals of a word automatically and generalize better. Currently available in English only, go to your application's settings in the Manage pane to turn it on.

Change from constraints to required features

If you were using constraints before, we’ve changed this functionality slightly. You can still constrain the output of a machine learned entity. Now you add a required feature to a machine learned entity, to ensure that entity won’t be predicted without the presence of the required feature.

Container support

A frequent customer request has been support for docker containers. It’s been preview and as of June 1, you can deploy and host Language Understanding anywhere using the GA of docker containers. When hosting the service in a container you have the flexibility to scale as much as you need without any limitations on TPS and you can use Language Understanding in scenarios where you don’t wish to send data to the cloud.

Dev ops sample

For developers that want to integrate Language Understanding into their CI/CD and release management pipelines, we’re previewing a sample repository template. This template enables you to develop a Language Understanding application while following DevOps engineering practices that adhere to software engineering fundamentals around source control, testing, CI/CD and release management. You can customize it for use with your own project. Learn more and try it out here.

Get started with Language Understanding today.

Watch the AI Show with Seth Juarez to see these enhancements in more detail.

Data Scientists – How you can stay productive while working remotely

samkemp — Mon, 18 May 2020 17:31:00 GMT

With COVID-19 continuing to impact people and countries around the world and data science teams everywhere are now working remotely, we will be running a series of blogs to help data science teams be productive in the current environment. This blog focusses on how Azure Machine Learning can foster collaboration and productivity when working remotely.

You may have lost access to a powerful workstation or server located at the office where you would normally execute your training jobs and collaborating with other data scientists in your team may have become harder.

On Azure we offer you two options to help stay productive while working remotely - Azure Machine Learning and the Data Science Virtual Machine. Azure Machine Learning is designed to get you up and running quickly by using built-in notebooks with the choice of your compute that we manage on your behalf. Or if you prefer, managing our own VMs, you can use Data Science Virtual Machine (DSVM) which comes pre-configured and up-to-date ML packages, deep learning frameworks and GPU drivers.

We recommend Azure Machine Learning for data science teams because it provides a fully manged collaborative development environment that is not offered by the Data Science Virtual Machine. Furthermore, Azure Machine Learning separates the compute from your notebooks by automatically mounting a cloud-based file store to host your notebooks. Simply put, this means that you can have different compute sizes without having to move files between machines – for example, you can develop & test some PyTorch code on a CPU compute instance and then switch the compute to a GPU machine to run the code. This architecture also means that you can delete a compute instances without losing your work.

Below is a table that outlines the key differences between these two options to help you decide which is the most appropriate for you.

	Azure Machine Learning A fully managed low hassle way to get up-and-running. Has built-in security and collaboration.	Data Science Virtual Machine Unmanaged machine learning workstation.
Recommended for	Data science teams and individual data scientists looking for a collaborative environment to accelerate their overall machine learning process	Individual data scientists that need a friction-free, pre-configured data science environment
Built-in Collaboration	Yes	No
Language Support	Python and R	Python, R, Julia, SQL, C#, Java, Node.js, F#
Operating System	Linux	Linux and Windows
Pre-Configured GPU	Yes	Yes
Pre-Configured Frameworks	Scikit, Tensorflow, PyTorch	Scikit, Tensorflow, PyTorch, Spark (Standalone), Keras, CNTK, MXNet, Chainer, Caffe, Caffe2, Theano
Hosted Notebooks (notebooks separated from compute)	Yes	No
Share notebooks with a link	Yes	No
Built-in SSO for Jupyterlab	Yes	No
Pre-configured Tools	Jupyter(lab) and RStudio	Linux: Jupyter(lab), RStudio Windows: Jupyter(lab), RStudio, VSCode, Visual Studio CE, Pycharm, Juno, PowerBI, SSMS, H20, LightGBM, Rattle, Vowpal Wabbitt, Weka, XGBoost, Apache Drill, Microsoft Office

Over the next few sections we will show you how to get started with a Compute Instance or DSVM.

Getting started with Azure Machine Learning’s managed notebooks and compute

Firstly, you will need to create an Azure Machine Learning workspace. To create a workspace, you need an Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the Azure Machine Learning for free today.

1. Sign in to the Azure portal by using the credentials for your Azure subscription.

2. In the upper-left corner of Azure portal, select + Create a resource.

3. Use the search bar to find Machine Learning.

4. Select Machine Learning.

5. In the Machine Learning pane, select Create to begin.

6. Provide the following information to configure your new workspace:

Field	Description
Workspace name	Enter a unique name that identifies your workspace. In this example, we use docs-ws. Names must be unique across the resource group. Use a name that's easy to recall and to differentiate from workspaces created by others. The workspace name is case-insensitive.
Subscription	Select the Azure subscription that you want to use.
Resource group	Use an existing resource group in your subscription or enter a name to create a new resource group. A resource group holds related resources for an Azure solution. In this example, we use docs-aml.
Location	Select the location closest to your users and the data resources to create your workspace.
Workspace edition	Select Basic or Enterprise. This workspace edition determines the features to which you'll have access and pricing. Learn more about Basic and Enterprise edition offerings.

7. When you're finished configuring the workspace, select Review + Create.

8. Review the settings and make any additional changes or corrections. When you're satisfied with the settings, select Create.

9. To view the new workspace, select Go to resource.

Adding team members to the workspace

To add team members to the workspace, navigate to Azure Machine Learning resource in the Azure portal and click on Access Control followed by Add

Click on Add Role Assignment and select an appropriate role assignment (e.g. Contributor, Reader, etc) and then search for the user or group to add (by name or email address).

Once the workspace is provisioned and team members are added, you can access the Azure Machine Learning Studio – an immersive experience for managing the end-to-end machine learning lifecycle in a browser: https://ml.azure.com

You will see the following:

You can create, view, edit and execute your notebooks in Azure Machine Learning Studio (https://ml.azure.com) by selecting Notebooks left-hand menu.

You will see each team member has their own directory to store their notebooks and code. To create a new notebook, click on the File+ button. Provide a filename and select the file type to be a Python Notebook - for example:

You will need to create compute to edit the file. To do this click on the + New Compute button articulated below:

This will take you to a New Compute Instance blade when you can enter the name of your compute and VM machine size (there are CPU and GPU machines available). You can the edit the files within Azure Machine Learning Studio:

Alternatively, you can open click on the Jupyter dropdown and select Jupyter(lab). This will take you to Jupyter(lab)

R users can leverage either Jupyter or have the option of RStudio. To navigate to RStudio head back to the Azure Machine Learning Studio (https://ml.azure.com) and click on Compute, which will bring up the Compute Instances blade where you will see your compute instance, click on RStudio

This will authenticate you into an RStudio Instance and you will see all your cloud-based notebook and code files.

Collaboration

Azure Machine Learning provides a shared file system for all users in the workspace, which allows team members to:

share/edit each other’s code
get help
get their code reviewed by a team lead

In addition, the compute instance comes pre-installed with Git - to clone a Git repository into this file share, we recommend that you create a Compute Instance & open a terminal. Once the terminal is opened, you have access to a full Git client and can clone and work with Git via the Git CLI experience.

We recommend that you clone the repository into your user’s directory so that others will not make collisions directly on your working branch.

You can clone any Git repository you can authenticate to (GitHub, Azure Repos, BitBucket, etc.)

For a guide on how to use the Git CLI, read the git handbook.

Accessing your data on a Compute Instance

If your data is on your local machine, then you can upload this Azure Machine Learning and consume this from any compute instance. To do this head to studio (https://ml.azure.com) and select Datasets from the left-hand menu:

Click on +Create dataset > from local files. Choose a name for your dataset and a dataset type - there are two types, which provide different capabilities:

A Tabular dataset represents data in a tabular format by parsing the provided file or list of files. This provides you with the ability to materialize the data into a Pandas or Spark DataFrame.
A File dataset references a single or multiple files in your datastores or public URLs. This provides you with the ability to download or mount the files to your compute.

In our case we are going to upload the Iris dataset, which is Tabular. Click next. On the next screen (Datastore and file selection) you select a cloud based datastore to upload the file to -AzureML automatically creates a cloud Datastore called workspaceblobstore for you when it is provisioned.

Click Next. On the following screen (Settings and preview) Azure Machine Learning will automatically detect the file type and parse the dataset into a table – make any necessary changes to header row, etc and click Next. Confirm the schema is correct and click Next followed by Create. You will see that the data has been loaded into a cloud store and is registered as an asset in Azure Machine Learning workspace:

If you click on the name of the Dataset it will bring up the details. When you click on the Consume tab, you will see something like the following:

Copy the Sample usage code into a cell in your own notebook. When you execute that code block you will see the following:

Notice that Azure Machine Learning will render the data file into a pandas data frame for you. Other team members in the workspace will also be able to access the data.

If your data already exists in the Azure Cloud (Blob, Azure Data Lake, Azure SQL DB/Postgres/MySQL) you can register that datastore in Azure Machine Learning workspace and access data from it. To do this click on Datastores in Azure Machine Learning Studio > + New Datastore > choose a datastore name and select the type. Complete the credentials to access the store (Azure Machine Learning will store these credentials automatically in a secure KeyVault). Follow the same process as above to create a dataset but instead of choosing a local file choose From datastore.

Enterprise security

Azure Machine Learning has comprehensive built-in enterprise security features such as:

VNET Support
RBAC
Private Link Support
Authentication
Monitoring

Full details can be gleaned from the documentation.

How to Create your Data Science Virtual Machine

To create a Data Science Virtual Machine instance:

Go to the Azure portal You might be prompted to sign in to your Azure account if you're not already signed in.
Find the virtual machine listing by typing in "data science virtual machine" and selecting "Data Science Virtual Machine - Windows 2019” for Windows or "Data Science Virtual Machine- Ubuntu 18.04" for a Linux-based DSVM.
Select the Create button at the bottom.
You should be redirected to the "Create a virtual machine" blade.
Fill in the Basics tab:

Subscription: If you have more than one subscription, select the one on which the machine will be created and billed. You must have resource creation privileges for this subscription.
Resource group: Create a new group or use an existing one.
Virtual machine name: Enter the name of the virtual machine. This is how it will appear in your Azure portal.
Location: Select the datacenter that's most appropriate. For fastest network access, it's the datacenter that has most of your data or is closest to your physical location. Learn more about Azure Regions.
Image: Leave the default value.
Size: This should auto-populate with a size that is appropriate for general workloads. Read more about Windows VM sizes in Azure.
Username: Enter the administrator username. This is the username you will use to log into your virtual machine, and need not be the same as your Azure username.
Password: Enter the password you will use to log into your virtual machine.

Select Review + create.
Review+create

Verify that all the information you entered is correct.
Select Create.

How to access the Data Science Virtual Machine

If you provisioned a Windows DSVM follow the steps listed to connect to your Azure-based virtual machine. Use the admin account credentials that you configured in the Basics step of creating a virtual machine.

You're ready to start using the tools that are installed and configured on the VM. Many of the tools can be accessed through Start menu tiles and desktop icons.

If you provisioned an Ubuntu DSVM, then you can access the VM in one of three ways:

SSH for terminal sessions
X2Go for graphical sessions
JupyterHub and JupyterLab for Jupyter notebooks

Follow the guidance on the how to access an Ubuntu DSVM page for further details on how to access using these methods.

We hope the guidance provided in this blog will you get started

CustomVision: Accelerating a model with ONNX Runtime on a CPU, GPU or Movidius Neural Compute Stick

KevinSaye — Mon, 22 Jun 2020 10:59:10 GMT

While I have written before about the speed of the Movidius: Up and running with a Movidius container in just minutes on Linux, there were always challenges “compiling” models to run on that ASIC. Since that blog, Intel has been fast at work with OpenVINO and Microsoft has been contributing to ONNX. Combining these together, we can now use something as simple as a Model created in http://customvision.ai at the edge running in OpenVINO for acceleration.

Confused? The following block diagram shows the relationship:

On just standard hardware and with a RTSP camera, I am able to score 3.4 frames per second on 944 x 480 x 24 bit images with version 1 of the Compute Stick or 6.2 frames per second with version 2. While I can get something close to this using CPU, the Movidius frees the CPU and allows multiple “calling applications” where the CPU performance is limited to just one.

OpenVINO Tag	Hardware	FPS from RTSP	FPS Scored	CPU Average	Memory
CPU_FP32	4 @ Atom 1.60 GHz (E3950)	25	3.43	300% (of 400%)	451 MB
GPU_FP16	Intel® HD Graphics 505 on E3950	25	6.3	70% (of 400%)	412 MB
GPU_FP32	Intel® HD Graphics 505 on E3950	25	5.5	75% (of 400%)	655 MB
MYRIAD_FP16	Neural Compute Stick	25	3.6	20% (of 400%)	360 MB
MYRIAD_FP16	Neural Compute Stick version 2	25	6.2	30% (of 400%)	367 MB

More info here: https://software.intel.com/content/www/us/en/develop/articles/get-started-with-neural-compute-stick.html and most of this work is based off this reference implementation: https://github.com/Azure-Samples/onnxruntime-iot-edge/blob/master/README-ONNXRUNTIME-OpenVINO.md.

Similar to the reference implantation above, I base this approach using a Docker Container, allowing this to be portable and deployed as an Azure IoT Edge Module. Note, while OpenVINO, ONNX and Movidius are supported on Windows, exposing the hardware to a container is only supported on Linux.

Step 1: in CustomVision.AI, create and train a model, then export it as ONNX. Note 6/22/2020: Use the “General (compact)” and not the “General (compact) [S1]”, as the second Domain currently does not work as expected.

Step 2: Using Git, clone the following repository: https://github.com/ksaye/CustomVisionWithMovidius.git

Modify line 67 of the DockerFile to reflect the URL of your exported ONNX zip file.

Step 3: On Linux with Docker or Moby installed at a command prompt where you cloned the repository, run as shown below. This will take a while.

docker build --rm -t customvisionwithmovidius --network host .

Step 4: On the host Linux PC, run the following commands to ensure that the application has access to the USB or Integrated MyriadX ASIC:

sudo usermod -a -G users "$(whoami)"

sudo echo SUBSYSTEM=="usb", ATTRS{idProduct}=="2150", ATTRS{idVendor}=="03e7", GROUP="users", MODE="0660", ENV{ID_MM_DEVICE_IGNORE}="1" > /etc/udev/rules.d/97-myriad-usbboot.rules
sudo echo SUBSYSTEM=="usb", ATTRS{idProduct}=="2485", ATTRS{idVendor}=="03e7", GROUP="users", MODE="0660", ENV{ID_MM_DEVICE_IGNORE}="1" >> /etc/udev/rules.d/97-myriad-usbboot.rules
sudo echo SUBSYSTEM=="usb", ATTRS{idProduct}=="f63b", ATTRS{idVendor}=="03e7", GROUP="users", MODE="0660", ENV{ID_MM_DEVICE_IGNORE}="1" >> /etc/udev/rules.d/97-myriad-usbboot.rules

sudo udevadm control --reload-rules

sudo udevadm trigger

sudo ldconfig

Step 5: With the image built, create an instance running the following command to create the container, start the container and monitor the log files. Note this web service listens on port 87 by default.

docker create --net=host --privileged -v /dev:/dev --name customvision customvisionwithmovidius && docker start customvision && docker logs -f customvision

Step 6: To send an image to the web service, simply run the following curl command, replacing 1.jpg for your image.

curl -X POST http://127.0.0.1:87/image -F imageData=@1.jpg

Looking at the screen, I see the Myriad was found, and the ONNX model was supported and loaded.

Because I have a process sending images to the web service as fast as it will take them, note that we see below multiple inferences per second.

Azure IoT Edge:

While the goal of this blog was to show the foundational capabilities, the sample code can be adapted to run at the intelligent (Azure IoT) Edge, as shown here: https://github.com/Azure-Samples/onnxruntime-iot-edge/blob/master/README-ONNXRUNTIME-OpenVINO.md. Once you have the foundation, the possibilities are endless!

Summary:

With the integration of ONNX and OpenVINO, Microsoft and Intel have really expanded the hardware acceleration platform and have made it easy for developers to adopt. While in the past you might have approached challenges with raw CPU power, you can actually get better performance and not at the expense of other processes by adding a cost effective hardware accelerator, like Intel's MyriadX.

Add to this the simplicity of http://www.customvision.ai and you can build some amazingly fast AI solutions at the edge at a fraction of the hardware and software cost. Combining IoT Edge and have have amazing speed, simplified AI models managed at the Intelligent Edge.

Get started with automating form processing to enable organizations’ productivity

NetaH — Wed, 13 May 2020 19:23:55 GMT

This blog has been authored by Neta Haiby (Principal PM, Form Recognizer) and Prachi Jain (PMM, Azure AI)

Healthcare organizations, hospitals, government agencies are at the forefront tackling COVID-19 responses. As the fight for the pandemic continues, so does the challenges in extracting and processing information, providing quick responses, and maintaining efficiency in processes.

Forms are everywhere and various scenarios require data extraction from forms like unemployment claims, employee sick leave, loan and mortgage applications, COVID-19 relief paperwork, clinical forms and more. Extracting data from these forms today is mostly manual which takes long processing cycles. Automating the data extraction enables companies to quicken the processing time enabling productivity and helps saves cost.

As an example, we have analyzed a credit card authorization form. A credit card authorization form allows a 3rd party to make a payment by using a person’s or companies written consent and credit card information. This can either be for a 1-time charge or recurring (weekly, monthly, etc.) and is used in by insurance companies, on-boarding new patients and more. Automatic extraction of the data from these forms enables companies to speed up on-boarding new customers and reduces the processing time.

The following credit card authorization forms contains fictitious content for illustrative purposes.

To extract data from forms Form Recognizer enables you to get started with 5 forms to train a custom model and label the values of interest to extract the data you need.

Form Recognizer enables you to extract the values of interest from these forms. With Form Recognizer Sample Labeling Tool you can easily and quickly label the values of interest such as name, souse name, phone, address, transcript required and more and train a model to extract the data. You then can use this model to analyze all incoming forms and automatically extract the data as part of your workflow or Robotic Process Automation solution.

How to Label, Train and Analyze Forms –

You will need a set of at least six forms of the same type (same structure \ format). You'll use this data to train the model and test a form.
Go to the Azure Portal and create a Form Recognizer resource if you don’t already have one.
Label your forms using the Form Recognizer Sample Labeling tool. You can use the try out site here or deploy it locally or in the cloud (How to deploy Form Recognizer Sample Labeling Tool). To create a new project see the Train with Labels Quick Start Guide – Setup input data. For more information on how to label forms see also the Train with Labels QuickStart guide – Label Your Forms.
Train a custom model, click the Train icon on the left pane to open the Training page. Then click the Train button to begin training the model. For more information on how to train a custom model see also the Train with Labels QuickStart guide – Train a Custom Model. You can also train a custom model using the Form Recognizer Train Custom Model API.

5. Analyze a Form using your custom model, Click on the Predict (light bulb) icon on the left to test your model. Upload a form document that you haven't used in the training process. Then click the Predict button on the right to get key/value predictions for the form. The tool will apply tags in bounding boxes and will report the confidence of each tag. You can also analyze using the Form Recognizer Analyze Form API.

Learn more about how customers have built solutions for COVID-19 for data extraction using Microsoft Computer Vision and Form Recognizer –

Ernst & Young

EY US developed an innovative Paycheck Protection Program (PPP) Loan Forgiveness Platform using Microsoft cloud and Form Recognizer. This enables banks at this critical juncture in the US economic recovery to efficiently meet the increasing demands across the end-to-end lending process required by the unique provisions outlined under the CARES Act. Learn more here.

Automation Anywhere

Automation Anywhere has developed a solution to help accelerate the reporting and processing of many complex forms, including CRFs and E28.This highly secure solution, comprised of Automation Anywhere RPA with native Intelligent Document Processing (IDP) and Azure Cognitive Services Computer Vision API and Form Recognizer. Learn more here.

Additional Resources

Get started with deploying Form Recognizer –

Custom Model – extract text, tables and key value pairs

QuickStart: Train a Form Recognizer model and extract form data by using the REST API
QuickStart: Train a Form Recognizer model with labels using the sample labeling tool
Form Recognizer Sample Labeling Tool
- Try it out: https://fott.azurewebsites.net/
- Open Source project: https://github.com/microsoft/OCR-Form-Tools

Prebuilt receipts - extract data from USA sales receipts

Quickstart: Extract receipt data using the REST API

Layout - extract text and table structure (row and column numbers) from your documents
- Quickstart: Extract layout data using the REST API
See What’s New

MLOps is Not Enough

charleswm — Mon, 22 Jun 2020 19:18:59 GMT

MLOps is Not Enough

The Need for an End-to-End Data Science Lifecycle Process

If you've ever worked on (or with) a data science team, you know that consistently delivering value can be frustrating (to put it nicely). There are so many places where things can go wrong and projects can fail. It has almost become a cliché to talk about the high failure rates of data science projects. However, given the demonstrated value that AI and Data Science have shown across industries, it's a problem that needs to be solved. There's just too much value to leave on the table. The division between successful companies and those who fall behind will be largely influenced by the success of their data science capabilities.

In response to this, it seems almost everyone is jumping on the MLOps train, and with good reason. MLOps has finally given us a way to consistently deploy, monitor, and retrain our models at scale. It's becoming clear that MLOps will be a required component of any successful data science team. So why do I say it’s not enough?

It's my view that MLOps on its own won't deliver. It will be transformational for making your existing models more robust, easier to retrain and monitor, etc. But what about new projects and new models? MLOps starts with a model, which means you've already found a model that works and now you want to enter the MLOps loop. Train, register, deploy, monitor, retrain, repeat.

The reality is that most teams and organizations struggle getting to that point consistently. Beyond deployment difficulties and risks, there are several other key areas where things go wrong:

Solving the wrong problems
Building models that don't map well to business processes
Bad assumptions about the data or a mismatch in population
Converting the results of your experimentation into a production ready model

Figure 1: An Internet Famous MLOps Diagram (with annotations)

We've seen all of these kill data science projects well before teams got to the stage where they'd even think about deployment. The good news is that while data science is experimental in nature, it's not random, which means we can identify ways to account for these common patterns.

Introducing the Data Science Lifecycle Process

In a previous article, I talked about the need for teams to create processes that cover the end-to-end data science process. We knew that MLOps would be a critical component, but based on our experience working with many data science teams, we still felt that there was a gap in the process when it came from going for the ideation phase to the point where you had a model you were ready to build and deploy. We dubbed what we came up with the Data Science Lifecycle Process (lovingly referred to as the DSLP).

We’re happy to announce that we’ve open-sourced this process so that every data science team can start improving their processes immediately. We’ve documented the process and created issue templates and repos and it’s all available on GitHub in the DSLP repo.

The DSLP is designed to break down the siloes between data scientists, developers, IT, and the business. Data science projects are cross-functional by nature. This means we need to bridge the gap between the (often ad-hoc) experimental workflows of data scientists and the more systematic approach of engineering teams. We've attempted to do this by creating a branching strategy, issue templates, and workflow patterns that establish clear boundaries and handoff points from the model development process to the implementation and deployment process. With a clear pivot point, it becomes easy to apply all the best parts of MLOps to the implementation and deployment process, while still giving data scientists the flexibility they need in the problem framing, experimentation, and development parts of the process.

Figure 2: The Phases of an ML Project and the Roles Involved

The need for a process like this is likely apparent to anyone who has worked on delivering data science projects in an enterprise environment. The friction between data science teams and just about everyone else is generally pretty high and leads to a lot of throwing things over the wall. It's not good folks.

Feedback on the DSLP So Far

As we developed the DSLP, we worked with several teams to test how well the process performed on real data science projects. We spoke with Cameron Vetter (an ML Engineer) and Carolyn Olsen (a Data Scientist) from Octavian Technology Group about the challenges they've seen enterprise data science teams face and how the DSLP addresses many of them.

Carolyn’s perspective:

In my experience, there are two places where existing data science processes really break down. The first is that the traditional software development processes don’t fit data science well, because data science is such a non-linear process. Workflows can quickly spread out like a hydra’s head. After a few weeks of work, data scientists may struggle to replicate exactly what they did along the way, or can get lost down analytical rabbit holes. DSLP makes non-linear data science processes focused and reproducible, by linking exploration and modeling experiments artifacts directly with Issues describing exactly what they’re meant to accomplish and results.

The second struggle many data scientists have is the pain of building a great model then seeing it “sit on a shelf,” never getting into production. Like agile project management, DSLP helps keep work focused on business goals, increasing likelihood of stakeholder buy-in. It also facilitates hand-off from data scientists to the engineers getting the model into production, by giving data scientists a structured way to hand off code and documentation.

Cameron’s perspective:

Data Science projects often have a disconnect between the engineers and data scientists. These two groups work in vastly different ways, and often struggle to sync their efforts. Engineers usually work within SDLC processes using them to align their teams towards the same goal. Data Scientists tend to be more experimental in their work. A Data Scientist will often go down a path and completely abandon it, starting down a new path many times during a project.

This experimental nature often leads Data Scientists to follow an ad-hoc process, making it difficult to hand off their work to ML engineers. By the time the work is handed off to the engineers, the Data Scientists are unable to explain why certain decisions were made around modeling, data shaping, and data enhancement. This can lead to a lot of throw-over-the-wall deployments where engineers are making decisions without understanding the how or the why behind what they are implementing.

The DSLP adds process to this experimental phase and does it within familiar SDLC tools that the engineers are comfortable with. This allows engineers to use the documented issues combined with the branching strategy to understand the flow of what happened prior to the hand off. This understanding will impact how this model is brought to production. This enables them to collaboratively iterate with the Data Scientists as they productionize the model.

What’s Next?

We’re going to continue building out this process as we continue to work on the projects we do with our customers and partners. We’re sure that what we’ve built isn’t perfect, but from what we’ve seen it can create a major positive impact on data science teams. Try it for yourself by implementing the branching strategy and using the issue templates on your next project. Keep on the look-out for more content from us as we continue to develop, document, and evangelize this process.

Feedback is welcome and as an open-source initiative we hope to create a vibrant community over time. If you want to learn more, test it out, or engage with us on implementing or improving this process, feel free to open an issue on GitHub or email us at dslp@microsoft.com.

https://github.com/dslp/dslp

How to build chatbots that deliver better customer experiences and help support a surge in inquiries

wmendoza — Fri, 15 May 2020 20:20:02 GMT

How to build chatbots that deliver better customer experiences and help support a surge in inquiries

This blog has been authored by Jim Lewallen (Principal PM, Conversational AI) and Will Mendoza (Senior PMM, Azure AI)

Organizations globally are seeing a significant increase in demand from customers looking for support and accurate information. As a result, developers are building chatbots that address a range of scenarios- from simple to sophisticated, to better serve customers and communities. An example of a simple scenario is informational Q&A chatbots that help answer frequently asked questions. Sophisticated scenarios can include branded virtual assistants for your organization that make people more productive by assisting with common tasks like scheduling meetings or making a reservation.

Before diving into how to build these types of chatbots, let us quickly walk through and define some of the key Azure AI components that you will need to build a chatbot.

Key Azure AI services and tools

Bot Framework is the open source SDK and tools for developers to design, build and test chatbots. If you want full control of your chatbot, including building your own language models, you will want to start here.
Azure Bot Service is the cloud service through which developers can host a chatbot in Azure, and quickly connect to popular channels such as Teams, Skype, Slack, email, and webchat, as well as community adapters for other channels like Alexa and Google Assistant ecosystems.
Azure Cognitive Services are a comprehensive family of AI services that enable you to build intelligent applications. Common examples of Cognitive Services for chatbots include Language Understanding to understand the meaning of utterances from users and QnA Maker to convert FAQ documents into conversational question and answer experiences.

Simple informational Q&A chatbot

One of the first types of chatbots you can build to get started quickly are simple informational Q&A bots. These chatbots can be used to alleviate strained resources that are answering the same basic questions. By implementing these types of chatbots, organizations can scale to more easily answer frequently asked questions in a cost effective manner, while enabling specialists to handle more nuanced requests. For example, Accenture built such a chatbot to help onboard new joiners in an organization who had the same common onboarding requests. Additionally, UNSW Sydney created a question chatbot to better engage with students and more quickly answers questions students might have. Other examples of common use cases include IT help desk password resets and customer service FAQs.

Image of the UNSW Sydney Question bot in Teams.

For this type of chatbot, you will need:

QnA Maker
Azure Bot Service

QnA Maker is the easiest way to build a chatbot in Azure. As described in the section above, QnA Maker will help you to quickly convert information in documents like FAQ pages and product manuals into a question and answer conversational experience. If you already have an FAQ document or page, you can build this experience in minutes within the QnA Maker portal, with the ability to answer common questions.

Additionally, you can easily add a personality type to handle small talk to handle odd questions like “Who made you?” or “Where are you from?” in a tone that is consistent with your brand. Once you have tested and re-trained the service, you can deploy QnA Maker to Azure Bot Service, and publish to Teams, Slack, or other popular channels.

Image of QnAMaker.ai portal experience with chit-chat.

Get Started:

Quickstart guide to walk you through how to build the bot.
If you prefer video tutorials, here is a guided video.

Transactional support chatbot

Another common type of chatbot is one that can help customers not only self-serve with answers to frequently asked questions, but also to retrieve information such as the status of a package or update a record in a system such as an insurance plan. Again, rather than take up limited resources chasing down answers, organizations are automating this capability so customers can self-serve. Developers at Jet.com built a customer service chatbot to help them scale to meet the growing customer service needs.

Example of a Jet.com bot interaction.

For this type of chatbot, you will need:

Bot Framework
Azure Bot Service
Azure Cognitive Services (e.g. Language Understanding)

These transactional, self-service chatbots can be built with the following components: Bot Framework, Azure Bot Service to build the chatbot engine and Azure Cognitive Services like Language Understanding to build the ability to understand utterances (e.g. inquiries, or requests from users) or QnA Maker to easily answer commonly asked questions.

Get started:

Quick start guide to begin with Bot Framework SDK.
For a more visual experience, use the Bot Framework Composer Quick start guide.
View the video tutorial on Bot Framework Composer
Note: Download Bot Framework Composer directly for Windows, Mac, or Linux

Image of the Bot Framework Composer tool. Download directly for Windows, Mac, or Linux.

Branded virtual assistant

Finally, some developers want to build their own custom virtual assistant capable of delivering personalized experiences enabling users to make a wide range of inquiries or requests across many canvases and even supporting the use of voice. La Liga and Vodafone are a couple of examples of organizations that have built their own custom voice assistants to better engage with their fans or customers.

Click the above image for a video on how La Liga built their own virtual assistant.

For this type of chatbot, you will need:

Virtual Assistant solution accelerator (which brings together key Azure services required)

To help simplify the steps required to solve for this scenario, we have made the Virtual Assistant solution accelerator available in our Github repository. The Virtual Assistant solution accelerator is an Azure Resource Management (ARM) template that orchestrates the deployment of the core Azure services required to deploy a custom virtual assistant.

Get Started:

Virtual Assistant solution accelerator template available on Github
Quick start guide for written tutorial
Watch this video for additional guidance

Bringing it all together

In short, as customer inquiry volumes surge, pressure is being put on organizations to continue to deliver great customer experiences and solve the needs of their customers. To help meet this need, developers are building informational chatbots to answer FAQs, transactional chatbots to allow customers to self-serve, or even context-aware branded virtual assistants for improved customer experiences.

We hope this blog has helped outline how you can bring together Azure AI offerings such as Azure Bot Service, Bot Framework and Cognitive Services to build chatbots that better serve your customers and communities.

Additional resources:

For more details on building your first chatbot, you can download this developer guide.

Finetune neural text-to-speech output with advanced customization features

Melinda Ma — Fri, 01 May 2020 16:30:29 GMT

This post was co-authored by @Qinying Liao, Yueying Liu, Sheng Zhao, @Anny Dow , Bohan Li and Jun-wei Gan

Neural Text to Speech (TTS) converts text to lifelike speech for more natural interfaces. With natural-sounding speech that matches the stress patterns and intonation of human voices, neural TTS significantly reduces listening fatigue when users are interacting with AI systems.

Common use cases for neural TTS include, but are not limited to, voice assistants, connected cars, smart-home devices, and various e-learning systems as well as reading apps. While neural TTS provides you a set of voices that already sound natural and human-like, you may still want to modify the speech properties to make voices better fit your scenario and context.

A wide range of fine-tuning features are available through Speech Synthesis Markup Language (SSML) and a code-free Audio Content Creation tool for you to adapt TTS output, such as adding or removing a pause/break, changing the pronunciation, adjusting the speaking rate, volume, pitch and more.

In this article, we’ll deep dive into the latest advanced features that can help you adapt the intonation and stress patterns of neural TTS output as well as define custom lexicon for your applications.

Control the prosody of your neural TTS output

Prosody, as one of the SSML elements, can be used to specify changes to pitch, contour, range, rate, duration, and volume for the TTS output, making your audio result easier to follow.

We are glad to share that the adjustments around contour, breaks/pauses and speaking rates of neural TTS are smoothly supported today. Now you can easily tailor the prosody of your TTS output using SSML or the Audio Content Creation tool.

Pitch contour

Pitch contour represents changes in pitch at specified times in speech output. tuning the pitch contour, you can make the intonation of your synthesized output sound different. For example, you can use it to emphasize different parts of your sentence or change the tone to make it sound more natural.

Here are some examples of adjusting pitch contour with SSML.

Original

Tuned

I never said he stole your money

<prosody contour="(11%, +65%) (60%, -43%) (80%, -34%)">

I never said he stole your money.

</prosody>

That's how you pronounce it ?

<prosody contour="(60%, -11%) (85%, +85%)">

That's how you pronounce it ?

</prosody>

Breaks/pauses

You can insert pauses (or breaks) between words or adjust pauses automatically added by the neural voices

Original

Tuned

Now 50 years after the event, he may finally have an answer.

Now <break time="100ms" />50 years after the event, he may finally have an answer.

通过语音合成技术，我们可以创造出不同风格的智能语音。

通过语音合成技术，我们可以<mstts:ttsbreak strength="none" />创造出不同风格的智能语音。

Adjust rate

Rate indicates the speed at which text is read aloud. You can adjust the speed of a whole sentence or a part of a sentence read by neural voices.

Original

Tune in SSML

Sometimes somebody will bring something that you really like.

Sometimes somebody will bring something that you <prosody rate="-51.00%">really </prosody>like.

Adjust neural voice prosodies through the audio content creation tool

Besides SSML, we also offer an easy-to-use Audio Content Creation tool to help you fine-tune TTS output. Paste or upload your text in the audio content creation tool, specify the voice you want to use, and then adjust the voice parameters in the tuning panel. You can switch your view to check the SSML format generated along with your adjustments and use the SSML in your code, or generate audio directly from the tool for your further use.

See below for a demo showing how prosody is adjusted using the code-free tool.

Define lexicon for your neural TTS output

Sometimes TTS does not pronounce words accurately in the way you want, such as a company or person’s name. To improve pronunciation, you can define the reading of these entities in SSML using the <phoneme> and tags. However, defining multiple entities one by one during speech synthesis can be time-consuming. The new custom lexicon capability makes this process easier.

With custom lexicon, simply specify the reading of entities in a list stored as an .xml or .pls file, provide a web link for your list and refer to this list in SSML. The right pronunciation will be applied to all specified custom words at once.

Here is a sample:

For your scenario, you may want to adjust the pronunciations of “BTW,” “Alki Beach” and “Jean” from the default TTS. Hear the differences in the samples below

Script

Default reading

Applied custom lexicon

BTW, we will arrive Alki Beach probably 8:00 tomorrow morning.

Could you help leave a message to Jean Pierre for me?

This is how the custom lexicon list is defined for the above sample:

You can upload the list online and put it in a data store like Azure Blob Storage.

During speech synthesis, use below SSML to refer to the list and apply custom lexicon to the input text. Speech synthesis will then reflect your defined pronunciations in the output all at once.

<lexicon uri="http://www.example.com/customlexicon.xml"/> BTW, we will arrive Alki beach probably 8:00 tomorrow morning. Could you help leave a message to Jean Pierre for me?

For more information about custom lexicon, please see our documentation.

Get started

Since the release of our Neural TTS less than two years ago, this field has advanced rapidly. New research models including Transformer TTS and FastSpeech have been proposed and improved the state of art. With these research innovations, we’ve not only improved the controllability of the neural voice output, but also made the synthesized speech more robust and largely improved the performance of neural TTS.

Get started with Text to Speech on Azure today.

Running ML.NET + Notebooks in Azure Machine Learning Studio

gopalv — Wed, 22 Apr 2020 20:20:49 GMT

Time Series Forecasting in ML.NET and Notebooks in Azure ML Studio

In this sample, learn how to run time series forecasting in a Jupyter notebook. We will read in data from a csv file, do some exploratory plots, fit a regression model, and fit a more sophisticated Singular Spectrum Analysis (SSA) forecaster.

Download the source code

Access the GitHub repo and copy the “clone” link in order to run this tutorial on your own machine.

Prerequisites

Install C# Kernel

Note: These instructions only apply if you intend to run this notebook in Azure Machine Learning. You can also run this notebook on your local machine by following the instructions at the dotnet interactive GitHub repo

Go to ml.azure.com. Select your subscription and machine learning workspace.
Open up the "Notebooks" tab on the lefthand side of the page
Create a compute instance if you have not already, or select an existing one from the dropdown menu.
Open a notebook file with an extension of .ipynb
Select the Terminal button at the top right.
Follow the instructions here to register a Microsoft product key and install .NET Core 3.1.
Install dotnet interactive by running dotnet tool install -g --add-source "https://dotnet.myget.org/F/dotnet-try/api/v3/index.json" dotnet-interactive
Create a symlink between the installed location of dotnet interactive and your local bin directory: sudo ln -s /home/azureuser/.dotnet/tools/dotnet-interactive /usr/local/bin/dotnet-interactive
Set your dotnet root directory: export DOTNET_ROOT=$(dirname $(realpath $(which dotnet)))
Install the jupyter kernel: dotnet interactive jupyter install
Verify the installation by running jupyter kernelspec list. You should see ".net-fsharp" and ".net-csharp" listed as kernels.

Install Mkl on Ubuntu Linux

If you are running ML.NET for the first time on an Ubuntu Linux machine (like Azure Machine Learning notebooks), please follow these instructions to download the required dependencies.

Start visualizing data

Great! We’re now set up to run ML.NET in Azure ML Integrated Notebooks. Let’s begin by visualizing our data, using the XPlot library. Notice how the data display a sinusoidal pattern, but there’s also a good amount of noise.

Compute an engineered feature

As we mentioned, the data display a sinusoidal pattern, so let’s use that intuition to fit a regression model with an engineered feature. Specifically, let’s fit a model using a cosine function as our independent variable. Below, consider how well a cosine model can mimic the periodicity of our original model. The only things that are wrong are the distance between crests and troughs of each wave (the “amplitude”) and the y-intercept of the wave. Luckily, linear regression can give us these values.

Fit a linear regression model

Let’s try fitting a model using our engineered features from the previous step. Because the input data are so nicely sinusoidal, this model actually works quite well. It has a Mean Absolute Error (MAE) of 1.997 and a Root Mean Squared Error (RMSE) of 2.574. Let’s see if we can do better.

Use ML.NET’s SSA Forecasting Transformer

ML.NET’s SSAForecastingTransformer can fit a forecasting model on our original data, without our having to provide it with engineered features. Most of the required parameters are based on the amount of data you have and the amount of time in the future you expect to predict. The only tricky one is the “windowSize” parameter, which should be set to be twice the length of the maximum expected seasonality in the data. For example, if you have data that is collected once per day in an environment that shows both monthly and yearly seasonality, you should set windowSize to be twice the length of the year, or 730. See the example notebook for more details on the other parameters.

Notice that the SSA Forecasting Transformer gives us not only a lower MAE and RMSE of 1.963 and 2.491, respectively, but also gives us 95% confidence bounds.

Predict future values

So we’ve found our model of interest, now let’s use it to predict the future! We can simply retrain the model on all of the data, and then use CreateTimeSeriesEngine to get a predictor, and then call Predict() to predict points up to the horizon we specified during training.

Next steps

In this notebook, you learned how to do time series forecasting in ML.NET with Jupyter notebooks. We initially used linear regression with an engineered feature, but we were able to improve performance by relying on ML.NET's SSA forecaster.

To learn more about C# and Jupyter Notebooks, check out this GitHub repo.

To see another example of using ML.NET in Jupyter, check out this blog.

To learn about using DataFrames in C#, check out this blog.

To get started with Model Builder in Visual Studio, try this getting started tutorial.

Open-Source Repository of Forecasting Best Practices for Accelerating Solution Development

chenhuihu — Tue, 14 Apr 2020 23:30:20 GMT

Chenhui Hu, Vanja Paunic, Hong Ooi, Tao Wu, Wee Hyong Tok

Time series forecasting is one of the most important topics in data science. Imagine that you are a business owner, you might want to predict different sorts of future events to make better decisions and optimize your resource allocation. Typical examples of time series forecasting use cases are retail sales forecasting, package shipment delay forecasting, energy demand forecasting, and financial forecasting. As you can see, forecasting is everywhere! Given its ubiquitous nature and wide-ranging business applications, we have developed an open-source forecasting repo that puts world-class models and forecasting best practices in the hands of data scientists and industry experts – i.e., you!

Figure 1: Visualization of training and testing iterations of a sales forecasting scenario using LightGBM model

Forecasting Best Practices and Solution Accelerators

This repository provides examples of building forecasting solutions presented as Python Jupyter notebooks, R markdown files, and a library of utility functions. Our goal is to help you as a data scientist or machine learning engineer with varying levels of knowledge in forecasting:

Learn best practices for the development of forecasting solutions in a variety of languages.
Leverage recent advances in forecasting algorithms to build high-performance solutions and operationalize them.
Accelerate the solution development process for real-world forecasting problems. With the provided examples, you will be able to significantly reduce the “time to market” by simplifying the experience from defining the business problem to the development of solutions by orders of magnitude.

In the repository, you will find state-of-the-art (SOAT) forecasting models using traditional machine learning and deep learning approaches. Implementations of SOTA models in this release are centered around retail sales forecasting and are written in Python and R, two of the most popular programming languages in the forecasting domain. To enable high-throughput forecasting scenarios, we have included notebooks for forecasting multiple time series with distributed training techniques such as Ray in Python, the parallel package in R, and multi-threading in LightGBM. The following is a quick summary of forecasting models covered in this repository.

Model	Language	Description
Auto ARIMA	Python	Auto Regressive Integrated Moving Average (ARIMA) model that is automatically selected
Linear Regression	Python	Linear regression model trained on lagged features of the target variable and external features
LightGBM	Python	Gradient boosting decision tree implemented with LightGBM package for high accuracy and fast speed
DilatedCNN	Python	Dilated Convolutional Neural Network that captures long-range temporal flow with dilated causal connections
Mean Forecast	R	Simple forecasting method based on historical mean
ARIMA	R	ARIMA model without or with external features
ETS	R	Exponential Smoothing algorithm with additive errors
Prophet	R	Automated forecasting procedure based on an additive model with non-linear trends and Tidyverts framework

The repository also comes with Azure Machine Learning (Azure ML) themed notebooks and best practices recipes to accelerate the development of scalable, production-grade forecasting solutions on Azure. You will find the following examples for forecasting with Azure AutoML as well as tuning and deploying a forecasting model on Azure.

Method	Language	Description
Azure AutoML	Python	Azure ML service that automates model development process and identifies the best machine learning pipeline
HyperDrive	Python	Azure ML service for tuning hyperparameters of machine learning models in parallel on cloud
Azure ML Web Service	Python	Azure ML service for deploying a model as a web service on Azure Container Instance

Developing an accurate forecasting solution can be a complex and time-consuming process. We hope the forecasting repo will help shorten your development cycle.

To Learn More and Contribute

For more information, please visit: https://github.com/microsoft/forecasting

Contributions from open-source community are always welcome! Please feel free to check our contribution guide if you would like to contribute to the content and bring in the latest SOTA algorithms.

Deploying your COVID-19 Healthcare Bot - Everything you need to get started

Ganesh-R — Wed, 22 Apr 2020 03:33:05 GMT

Public health organizations, healthcare providers, hospitals and others on the frontline of COVID-19 response have had to act quickly to support the sudden spike in inquiries from patients and constituents looking to get answers to a common set of requests such as up-to-date outbreak information, assess symptoms and risk factors for people worried about infection, and suggest a next course of action. Many of these organizations have expressed concerns with being able to support the volumes of inquiries, and consequently have been using the Microsoft Healthcare Bot to help provide critical information to their patients.

Microsoft’s Healthcare Bot is a scalable Azure-based SaaS solution that empowers Microsoft customers and partners to build and deploy compliant, AI-powered health agents, allowing them to offer their users intelligent, personalized access to health-related information and interactions through a natural conversation experience. It is one solution that uses AI to help the CDC and other frontline organizations to provide help to those who need it.

The Healthcare Bot can easily be customized to suit an organizations scenarios and protocols. To assist in the rapid deployment of COVID-19 specific bots Microsoft has made available a set of COVID-19 templates that customers can use and modify:

COVID-19 Risk Assessment
COVID-19 Frequently Asked Questions
COVID-19 Worldwide metrics
COVID-19 Clinical Triage

Deployment of COVID-19 Healthcare Bot in your environment

To help you deploy your COVID-19 healthcare bot, Microsoft has created a Reference architecture, deployment template and supporting How to videos and guides.

Reference Architecture

The reference architecture provides guidance on a High Availability deployment of the Healthcare Bot and associated Azure services across 2 regions.

Note: The architecture can also be deployed in a single region, if you choose to deploy in a single region it is recommended that you model and estimate your peak traffic expectations to ensure that a single region deployment is appropriate for your situation.

Deployment Template

To assist in deploying the reference architecture we have developed an ARM template for you to use. The step by step instruction to deploy and configure the reference architecture can be found here:

Deploy Microsoft Health Bot Reference Architecture

To then set up your Health Bot – follow the instruction in the Quick Start: Setting Up Your COVID-19 Health Bot

If you are ready to deploy and would like assistance:

Contact your account team for a quick demo and/ or alignment of resources.
Speak to one of our Health Bot Partners who can help you deploy and customize your own COVID-19 Health Bot.

Additional Resources

Thanks for reading and let us know how else we can help!

Mark Perry, Microsoft Director - Customer Success

Ganesh Radhakrishnan, Microsoft Cloud Solution Architect

Introducing new voice styles in Azure Cognitive Services

Melinda Ma — Thu, 02 Apr 2020 22:34:35 GMT

This post was co-authored by , , Yueying Liu, and Peter Pan.

Neural TTS enables fluid, natural-sounding speech that matches the patterns and intonation of human voices, helping developers bring their solutions to life.

Today, we’re building upon our Neural Text to Speech (Neural TTS) capabilities in Azure Cognitive Services with new voice styles. With the new styles—newscast, customer service, and digital assistant—developers can tailor the voice of their apps and services to fit their brand or unique scenario.

Built on a powerful base model, our neural TTS voices are very natural, reliable, and expressive. Through transfer learning, the neural TTS model can learn different speaking styles from various speakers, enabling nuanced voices.

In addition to our new voice styles optimized for specific scenarios, we are also releasing new emotion styles. These styles allow you to adjust voices to express different emotions to fit the context, like cheerfulness or empathy. Let’s dive in.

Introducing Newscast, Customer Service, and Digital Assistant styles

Newscast

With neural TTS voices in the newscast style, your users can enjoy listening to news or articles in a professional tone that reflects what you might hear on TV or radio newscasts.

Hear Aria's (English – Female) and Xiaoxiao’s (Chinese – Female) voices in the newscast style:

Text

Newscast style

Default

Heavy snow and strong winds hammered parts of the central U.S. on Thursday and began moving into the Great Lakes region, knocking out power to tens of thousands of people and creating hazardous travel conditions a day after pummeling Colorado.

现今，大批企业以数字化转型为战略目标，数字化转型可赋能企业重构竞争环境、满足客户期望、增强服务运营。为了真正实现“ being digital ”, 许多企业将人工智能视作实现数字化转型目标的首选技术工具之一。

Check out the newscast style in the Bing mobile app. When you search news with the voice search feature, you can hear news briefs using Aria’s newscast style voice.

You can also check out Xiaoxiao’s newscast style voice, which has been adopted in WeChat through the Microsoft Listening Docs app. In Microsoft Listening Docs, users can hear Xiaoxiao’s voice read out multiple document types such as Word, PowerPoint, Excel, as well as images. Users can easily generate audio content for online trainings, news podcasts and more, and share with their social circles.

Customer Service

The customer service style features a friendly and engaging tone and is suitable for scenarios involving customer support, such as an individual checking into their flight, making a restaurant reservation, or reporting a claim.

Hear Aria's and Xiaoxiao’s voices in the customer service style:

Text

Customer Service style

Default

Alright, it's going to be right in front of your door, within 30 minutes. Thanks for calling Pizza Loco! Have a great night!

客服：您好，欢迎致电智慧银行，我是您的智能客服晓晓，请问有什么可以帮您？

客户：你好，我想调整信用卡的额度。

客服：嗯，请稍等，我查询一下状态。请问您要调整到多少额度？

客户：帮我调到三万人民币吧。

客服：好的，已经给您变更成功，稍后您会收到短信提醒。

客户：好的，谢谢。

客服：感谢您的来电，祝您生活愉快，再见。

Digital Assistant

Many customers have been using neural TTS voices for their digital assistant solutions. We are introducing two styles in this area: a chat style for more casual, conversational bots, and a more professional style for scenarios such as in-car digital assistants.

The chat style features a conversational tone, simulating casual dialogue.

Hear Aria’s voice in the chat style:

Style

Text

Chat style

Default

Chat

Oh, well that's quite a change from California to Utah.

The assistant style features a friendly and helpful tone, which is suitable in scenarios such as smart speakers or in-car assistants. Use the digital assistant voice to hear the weather forecast, search for information, navigate directions, set reminders, and more.

Hear Xiaoxiao’s voice in the assistant style:

Text

Assistant style

Default

没听到你说话，请再说一次。

现在听的是：FM88.8，江苏音乐台的节目，滴滴叭叭早上好。

Bringing new emotions to Neural Text to Speech

To enable you to build nuanced voices for your unique scenario, Neural Text to Speech also offers different emotion styles. You can access cheerful and empathetic styles for Aria’s voice, lyrical style for Xiaoxiao’s voice—which sounds heartfelt and is optimized to read prose or poetry, and cheerful style for Francisca’s voice (Brazilian Portuguese).

Hear the new styles below:

Style	Text	Style	Default
Cheerful	Great, I hope she will like it!
Cheerful	A canadense postou uma música nova no seu perfil oficial do Twitter.
Empathetic	I want to let you know that you’re loved. I know things are hard right now and it’s OK. You don’t have to do this alone
Lyrical	大家晚上好，我是晓晓。在每一个夜晚来临的时候，我都在这里陪你入睡。忙碌的一天又过去了，现在的你是窝在沙发上看着窗外发呆，还是倒了一杯咖啡继续解决白天没有做完的工作呢？时间过得真快呀，在学校里咬着早餐上课，和同学们嬉戏打闹的日子，仿佛就在昨天。但一转眼，我们都穿着西装变成了大人。

These new voice styles are also available for customized brand voices through our Custom Neural Voice capability, allowing you to build a unique voice that can also benefit from our new scenario and emotion styles. As part of Microsoft's commitment to designing AI responsibly, we have developed guidelines for customers in using Custom Neural Voice, in alignment with Microsoft's principles for responsible innovation in AI. Learn more about the process for getting started with Custom Neural Voice here.

Get Started

Get started with the new neural TTS voice styles available in Azure Cognitive Services. Check out our documentation to learn more.

Cognitive Services adds Brazilian Portuguese to Neural Text to Speech

Qinying Liao — Thu, 02 Apr 2020 13:51:51 GMT

Cognitive Services adds Brazilian Portuguese to Neural Text to Speech

This post was co-authored by Sheng Zhao, Anny Dow, Edward Un, Yueying Liu, Garfield He and Yang Zheng.

(voiced by Neural TTS)

Neural Text to Speech (Neural TTS) converts text to lifelike speech for more natural interfaces. With natural-sounding speech that matches the stress patterns and intonation of human voices, neural TTS significantly reduces listening fatigue when users are interacting with AI systems, enabling scenarios from audiobooks to voice assistants.

Brazilian Portuguese neural voice now available

We’re excited to share that we are expanding our available neural TTS voices with Francisca, our new Brazilian Portuguese (pt-BR) voice. Francisca features the same human-like natural prosody of the other neural TTS voices on Azure — Guy (American English Male), Jessa (American English Female), Katja (German Female), Elsa (Italian Female), and Xiaoxiao (Mandarin Chinese Female).

With a powerful base model created using a large volume of speech samples, we were able to build Francisca’s voice from much less training data than it would require otherwise. The neural TTS base model learns different speaking styles from multiple speakers, and through transfer learning, can easily adapt its style to a target speaker. Like other neural voices, Francisca can generate realistic speech waveforms for a given text input, matching the patterns of stress and intonation transitions in spoken language seamlessly.

Besides the capability to synthesize speech, developers can also tailor the voice for different scenarios with different voice styles using the neural TTS. For example, the new pt-BR voice can also speak with a “cheerful” tone. The “cheerful” style can be used to express an emotion that is positive and happy. This is particularly useful in chat bot scenarios. You can adjust the speaking styles easily with the <mstts:express-as> element in SSML.

We conducted MOS (Mean Opinion Score) studies to evaluate the naturalness of Francisca. In a crowd-sourcing test with more than 60 native speakers, we examined 30 audios produced by Francisca in the neutral style and another 30 in the cheerful style. Overall impressions were rated on a 1-5 Likert scale, with naturalness in rhythm variations, pitch variations, stresses, pauses, and intelligibility considered. Human speech and a pt-BR voice from another cloud service provider (company X) were used as benchmarks. Results showed very positive feedback on Francisca in both the neutral (4.44) and cheerful (4.38) styles.

Figure 1. MOS comparison of Francisca with human speech and company X

Hear what Francisca sounds like.

Example 1: Francisca (neutral)

Example 2: Francisca (cheerful)

High fidelity and controllable output

Like other neural voices, Francisca is created using 24khz sampling rate. You can maximize the fidelity of neural voice outputs with 24khz related formats:

raw-24khz-16bit-mono-pcm
riff-24khz-16bit-mono-pcm
audio-24khz-160kbitrate-mono-mp3
audio-24khz-96kbitrate-mono-mp3
audio-24khz-48kbitrate-mono-mp3

For scenarios where lower sampling rate is required, for example playing back for phone calls, Francisca and other neural voices can also be easily sampled down with a lower bit rate. Learn more about the output format supported.

Hear text aloud with Read Aloud in the new Edge browser

Neural TTS is powering Microsoft services at scale. The Francisca voice is now supported in the new Microsoft Edge, enabling you to anytime, anywhere with natural voices.

Figure 2. Neural TTS in Edge Read-Aloud

Edge Read Aloud also makes it easy to follow along with text, supporting the output of word boundaries so each word being read out is simultaneously highlighted in the UI. This is an essential feature for immersive reading scenarios. To build your own Read Aloud apps, check out SynthesisWordBoundaryEventAsync function in our sample codes.

Create a custom voice in Brazilian Portuguese

The same transfer learning technology is now shipped in the the Custom Neural Voice capability, enabling organizations to create their one-of-a-kind digital voices with 5X less data while still delivering high-fidelity audio outputs.

With Brazilian Portuguese (pt-BR) added to the family, seven locales are now supported in the custom neural voice online training portal - American English (en-US), British English (en-UK), Indian English (en-IN), German, French, Chinese (zh-CN) and Brazilian Portuguese (pt-BR). More locales are available through customer engagement. Submit a request to create your custom voice using the neural TTS technology.

Get started

With these updates, we’re excited to be powering natural and intuitive voice experiences. Text to Speech has more than 75 standard voices in over 45 languages and locales in addition to our growing list of neural voices. Learn more about how you can get started.

How BERT is integrated into Azure automated machine learning

eric-cb — Tue, 03 Mar 2020 16:54:10 GMT

Introduction

We’re introducing the BERT deep learning architecture for text data to Azure Automated ML. This model usually performs much better than older machine learning techniques that rely on bag of words-style features for text classification. BERT, which is both a neural net architecture and a particular transfer learning technique, has had a huge impact on large and small companies (example use cases include Microsoft, Google, Stitch Fix). Since Automated ML uses a BERT model that has already been pretrained on a large corpus of text, the user of Automated ML doesn't need very much training data to see a lot of benefit (even ~ 100s of rows are okay in some circumstances), which can be very valuable if labeled data is hard or expensive to acquire. We’ve implemented BERT in automated ML in such a way that it is first fine-tuned on the dataset the user provides, and then automated ML uses the embeddings from the fine-tuned BERT model as text features for other ML algorithms like logistic regression or LightGBM. Our implementation of BERT uses the popular transformers repository (paper, github).

What does deep learning for text data do that older techniques don’t do?

Deep learning for text data gives you more accurate models compared to bag of words-based approaches to handling text data. Bag of words-style features used for text in Automated ML include unigrams, bigrams, and tri-character grams. Another text feature sometimes used in automated ML are static pretrained word embeddings. It’s not big news anymore that some deep learning architecture outperforms older “shallower” learning techniques, but what really grabbed our attention was seeing how much BERT tends to outperform bag of words type approaches on small training data (100s of rows) as compared to larger data of 10000 rows or more. In general, we’ve found that BERT can often get the same accuracy as the bag of words approach with only 1/10 of the data! This ability to learn from small data can really benefit your product if, as is often the case, labeled data is difficult or expensive to get. To illustrate this, we ran Automated ML model with and without BERT on a four class news dataset (called AG News) and plot the learning curve in Figure 1. To illustrate the value of pretraining (both through BERT and pretrained word embeddings), we also trained a logistic regression model with unigram and bigram features as a simple baseline. Notably, automated ML with BERT achieves 94.7% accuracy on AG News when trained with 120k rows, which would put it at 4th place on this leaderboard for ag news as of this writing. To ensure that training does not take too long and to avoid GPU memory issues, automated ML uses a smaller BERT model (called bert-base-uncased) that will run on any Azure GPU VM.

Figure 1: Learning curves in for bag of words style features vs BERT features for the AG News dataset.

What's so special about BERT that makes it so much better than bag of words?

Theoretically BERT has a big advantage over bag of words methods in that BERT is sensitive to word order and long range word interdependencies (e.g. the meaning of "it" might depend on a particular word 10s of words to the left of "it"), but we were curious how this difference plays out with real world data. So we examined how well BERT vs bag of words does in our hold out set for this news classification dataset. After looking at many examples and the predictions from BERT versus simpler methods, we did notice a pattern: When a news article is misclassified by bag of words methods, it often contains one to a few words that shift the meaning of the entire document. To illustrate this, here’s two examples that we constructed and fed into the models trained on the ag news dataset.

1. “The two players were evenly matched, but the first player's skill with the joystick pushed her over the top.”

2. "The two players were evenly matched, but the first player's skill with the hockey stick pushed her over the top. "

The sentence with "hockey stick" is easy to classify as being about "sports", and indeed the bag of words approach and BERT correctly classify it. The sentence with "joystick" is harder to classify because it has a lot of words ordinarily associated with sports, but it's actually in the "science & tech" category. For the "joystick" sentence, BERT correctly predicts that it's in the "science & tech" category, while the bag of words-based model incorrectly predicts the "joystick" sentence as being about sports. What’s particularly interesting about this example is how just one word, “joystick”, or phrase, “hockey stick”, dramatically changes the meaning of the document from being about video games to being about physical sports. This is why the bag of words approach fails for the "joystick" sentence, the preponderance of strongly “sports”-like words pushes the prediction to sports. BERT, on the other hand, doesn’t just model words as a static collection of distinct things (aka “bag of words”), but rather contains sophisticated mechanisms that ensure the features of one word both depend on that word’s position in a document and also depend on the other words in the document around it. This way it can know that a “player” in this document is actually a video game “player” by virtue of the presence of “joystick” elsewhere in the document. This sophistication in BERT, we hypothesize, is the reason BERT isn’t fooled into thinking a video game competition text snippet like example 1 is about physical sports.

How Is BERT integrated into Automated ML?

BERT is used in the featurization layer of Automated ML. In this layer we detect if a column contains free text or other types of data like timestamps or simple numbers and we featurize accordingly. For BERT we fine-tune/train the model by utilizing the user-provided labels, then we output document embeddings (for BERT these are the final hidden state associated with the special [CLS] token) as features alongside other features like timestamp-based features (e.g. day of week) or numbers that many typical datasets have. Please see Figure 2 for a schematic of this.

Figure 2. How a dataset with columns of data consisting of a timestamp, a count of some sort, two text columns, and class labels get featurized by BERT in Azure Automlated ML. The blue blocks represents the data, which starts out as the raw inputs, and eventually get transformed into predictions, and the gray elements represent the machine learning pipeline.

It’s worth noting that with BERT we don’t technically need to train it as it’s pretrained on a large corpus of text. However, there’s no good way to get document embeddings unless BERT is fine tuned. One way to see what fine-tuning does to embeddings is to visualize BERT generated document embeddings using a t-SNE plot (this visualization method places points close together in 2 dimensions according to the probability they are close together in the original ~ few hundred dimensional embeddings space). We created two 2D t-SNE plots: one where BERT has been trained on 1% of a dataset vs another BERT model that was trained on the full dataset. Each point represents a document, and its color is the ground-truth class label of that document. Both of these models use the same four class text dataset. In the 1% case you can see that the embeddings don’t display much class structure since most points belong to one blob. On the right where BERT was trained on the full dataset the class structure is much more obvious. So it’s now apparent that fine-tuning BERT is quite important if you want BERT to generate embeddings that “know” about the particularities of a dataset!

Figure 3: BERT document embeddings (coming the final hidden state of the special [CLS] token). Note there is not much structure when BERT is trained on small fraction of a four class dataset, but on the full dataset the four classes are clearly present.

Conclusion

With the gains of BERT for training data both big and small in mind, we conclude with a recommendation for when you should use BERT vs when you might want to use a bag of words-based model. If you need predictions to be very fast (like < few milliseconds per prediction) and/or you want to perform predictions on a CPU, then you should not use BERT and should stick with bag of words-based models. In the end, you will need to make a choice regarding this trade-off between the fast inference time of bag of words-type models and the high accuracy of BERT.

So how can I try BERT in Azure Automated ML?

To get started with Azure automated machine learning, you can read our docs here. If you're comfortable with python, you can jump right into our Jupyter notebook that illustrates BERT. The main thing to keep in mind is that to benefit from BERT, you need to

Have a classification dataset with 1 or more columns of text (you can have other columns of e.g. categorical data, but BERT will only train on the text column(s) as indicated by Figure 2)
Select an Azure VM with a GPU (e.g. "STANDARD_NC6"). If you're creating the compute using the python sdk, then see our BERT notebook for how select a GPU. If you're using Azure Machine Learning Studio (i.e. you're using Automated ML from a browser-based UI), when prompted to select vm size, be sure to select a GPU-enabled vm size and when selecting task type, check the "enable deep learning" checkbox.

Once your automated ML run is complete, you can use your trained automl model to do inference on a GPU or a CPU Azure VM, just note that performing inference on a GPU will be much faster.

Contributors (alphabetical order):

Eric Clausen-Brown – Senior Data & Applied Scientist

Zubin Pahuja – Software Engineer

Anup Shirgaonkar – Principal Data & Applied Scientist

Arjun Singh – Intern Data & Applied Scientist

Azure and OpenCV partner on Deep Learning with PyTorch course

Courtney_Luk — Thu, 05 Mar 2020 16:45:04 GMT

OpenCV (Open Source Computer Vision Library) is the leading open source library and community for computer vision, image processing, and machine learning. The library has been used for real-time tasks such as monitoring mine equipment in China, detecting intrusions through surveillance video in Israel, and product label inspection in factories around the world.

The non-profit organization is dedicated to advancing the beneficial uses of computer vision within society. As part of this mission, OpenCV has committed itself to creating online courses in AI to educate a global workforce. These courses are geared towards anyone interested in learning AI and designed to equip students with practical skills for solving real-world problems and help them stand out in the job market.

Microsoft shares OpenCV's commitment to making an AI education accessible for everyone. Today, we're excited to announce that Azure AI is partnering with OpenCV to offer 100 hours of free GPU credit to all students enrolled in their Deep Learning with PyTorch course that will be taught on Azure.

"OpenCV.org is honored to receive 100 hours of free GPU time by Microsoft on its Azure Platform for our students enrolled in the Deep Learning with PyTorch course. The notebook based Azure platform is easy for beginners and very flexible for AI practitioners. Not only is Microsoft leading AI research, it is helping make AI education accessible to students around the world while promoting open standards. We are thankful for Microsoft's commitment to the open source community."

-Satya Mallick, CEO of OpenCV

Deep Learning with PyTorch will first provide students with a theoretical understanding of simple neural nets and then gradually move to explore Deep Neural Nets and Convolutional Neural Networks. It will then give them hands-on experience for successfully training Deep Neural Networks, which requires a significant amount of time and compute. By providing students with GPU credit for their coursework, we hope to provide them with the best learning experience that sets them up for success.

Along with this hands-on education, OpenCV's Deep Learning with PyTorch course will provide students with a discussion forum where they can collaborate with one another on questions and challenges and get help from dedicated mentors. Upon completion of the course, students will receive a certification that recognizes them as OpenCV certified AI practitioners.

The Deep Learning with PyTorch course was originally only open to those who committed through OpenCV's AI Courses Kickstarter, but registration has been re-opened to anyone interested starting today (February 24). Course registration will remain open for the next 8 days on OpenCV.org's course site.

(Nearly) Everything you need to know about computer vision in one repo

jiata — Tue, 03 Mar 2020 17:53:06 GMT

This post was co-authored by @JS Tan, @Patrick Buehler, and @Jun Ki Min

In recent years, we've seen extraordinary growth in Computer Vision, with applications in image understanding, search, mapping, semi-autonomous or autonomous vehicles and many more.

The ability for models to understand actions in a video, a task that was unthinkable just a few years ago, is now something that we can achieve with relatively high accuracy and in near real-time.

Action Recognition

However, the field is not particularly welcoming for newcomers. Without prior experience or guidance, building an accurate classifier can easily take weeks. Unless you're ready to spend a long-time learning computer vision, it's extremely hard to master the basics, let alone begin to explore some of the cutting-edge technologies in the field. Even for computer vision experts, building a quick Proof of Concept (POC) can be nontrivial and could easily end up taking many days to put together.

At Microsoft, we have been working for many years on diverse Computer Vision solutions for our customers and collected our learnings into our new public Microsoft repository: https://github.com/microsoft/ComputerVision-recipes.

The goal of this repository is to provide examples and best practice guidelines for building computer vision systems on Azure, and to share this with the open-source community. More specifically, our goal was to create a repository that will help us to provide solutions rapidly to the community and to customers that we work with, or with on-boarding new team members who may have expertise in data science, but not specifically in computer vision. From mastering some of the most common scenarios in the field, like image classification, object detection, and image similarity, to exploring cutting edge scenarios like activity recognition and crowd counting, this repo will guide you through building models, fine-tuning them, and using them in real-world scenarios.

We're kicking off our repo with 5 scenarios:

Scenario	Support	Description
Classification	Base	Image Classification is a way to learn and predict the category of a given image. (Ex: Is the picture of a ‘dog’ or a ‘cat’?)
Similarity	Base	Image Similarity is a way to compute a similarity score given a pair of images. Given an image, it allows you to identify the most similar images in a dataset. (Ex: This picture of a dog is the most like which of the following images of animals?)
Detection	Base	Object Detection is a supervised machine learning technique that allows you to detect where on a given image an object of interest is. (Ex: Where in the image are there animals?)
Action Recognition	Contrib	Action Recognition is used to identify in video footage what actions are performed and at what respective start/end times. (Ex: When is there someone drinking in the video?)
Crowd Counting	Contrib	Crowd Counting is a use-case that leverages supervised machine learning techniques to count the number of people in an image – this applies to both low-crowd-density (e.g. less than 50 people) and high-crowd-density (e.g. thousands of people). (Ex. How many pedestrians are in this image of a street?)

Rather than creating implementations from scratch, we draw from popular state-of-the-art libraries (e.g. fast.ai and torchvision), and we build additional utility around loading image data, optimizing models, and evaluating models. In addition, we aim to answer the frequently asked questions, try to explain the deep learning intuitions, and highlight common pitfalls.

Whether you are an expert in computer vision or just getting your hands wet, we believe this repository offers something for you. For the beginner, this repo will guide you through building a state-of-the-art model and help you develop an intuition for the craft. For the experts, this repository can quickly get you to a strong baseline model which is easy to extend using custom Python/PyTorch code. In addition, the repository also aims to provide support with 1) the full data science process, and 2) the tooling to succeed on Azure.

We hope that these examples and utilities will make it easier and faster for developers to create custom vision applications.

The Data Science Process

The Computer Vision Recipes GitHub repository shows you how to approach the five key steps of the data science process and provides utilities to enrich each of the steps:

Data preparation - Prepare and load your data.
Modeling - Build models using deep learning algorithms.
Evaluating – Evaluate your model. Depending on the metric you’re interested in optimizing, you may want to explore different methods of evaluation.
Model selection and optimization - Tune and optimize hyperparameters to get the highest performing model. Because Computer Vision models are often computationally costly, we show you how to seamlessly scale your parameter tuning into Azure.
Operationalizing - Operationalize models in a production environment on Azure by deploying it onto Kubernetes.

Inside the computer vision recipes repo, we have added a lot of utility to support common tasks such as loading datasets in the format expected by different algorithms, splitting training/test data, and evaluating model outputs.

Azure Machine Learning

This computer vision repository also has deep integration with the Azure Machine Learning service to complement your work locally. We provide code examples on how you can optionally and easily scale your training into the cloud, and how you can deploy your models for production workloads.

Azure Cognitive Services

Note that for certain computer vision problems, you may not need to build your own models. Instead, pre-built or easily customizable solutions exist which do not require any custom coding or machine learning expertise.

Vision Services are a set of pre-trained REST APIs which can be called for image tagging, OCR, video analytics, and more. These APIs work out of the box and require minimal expertise in machine learning but have limited customization capabilities. See the various demos available to get a feel for the functionality (e.g. Computer Vision).
Custom Vision is a SaaS service to train and deploy a model as a REST API given a user-provided training set. All steps including image upload, annotation, and model deployment can be performed using either the UI or a Python SDK. Training image classification or object detection models can be achieved with minimal machine learning expertise. The Custom Vision offers more flexibility than using the pre-trained cognitive services APIs but requires the user to bring and annotate their own data.

Before using the Computer Vision repository, we strongly recommend evaluating if these can sufficiently solve your problem.

Scenario Example: Object Detection

To give you a sense of how you can use our repo to build a state of the art (SOTA) model, here is a preview of how simple it is to create an Object Detection model. Of course, you can go much deeper and add custom PyTorch code, but getting started is as simple as this:

1. Load your data

The first step is to load your data – we help you do this with a simple object that automatically parses your data and the annotations:

from utils_cv.detection.data import DetectionLoader data = DetectionLoader("path/to/data")

2. Train/fine-tune your model

Then we create a 'learner' object that helps you manage and train your model. By default, it will use torchvision's Faster R-CNN model. But you can easily switch it out.

from utils_cv.detection.model import DetectionLearner detector = DetectionLearner(data) detector.fit()

3. Evaluate

Finally, lets evaluate our model using the built-in helper functions. We can look at the precision and recall curves to give us a sense of how our model is performing.

from utils_cv.detection.plot import plot_pr_curves eval = detector.evaluate() plot_pr_curves(eval)

As we continue to build out of repository, we will be looking for new computer vision scenarios to unlock. Feel free to reach out to cvbp@microsoft.com or post an issue if you wish to see us cover a scenario.

Accelerating Distributed Training in Azure Machine Learning service using SR-IOV

Ravi_Kolli — Wed, 11 Dec 2019 18:13:06 GMT

Author: Ravi Shankar Kolli

This post is co-authored by Mathew Salvaris, Aashna Garg, Vaibhav Jain, Reyhan Patia, Caghan Demirci, Alex Sutton

Today’s state of the art deep learning models like BERT require distributed multi machine training to reduce training time from weeks to days. Interconnect is one of the key components to reduce communication overhead and achieve good scaling efficiency in distributed multi machine training.

Azure Machine Learning users can now speed up their training time by taking advantage of the Azure Virtual Machines with SR-IOV and InfiniBand support. In September 2018, Azure introduced the NC, ND, and H-series of VMs dedicated InfiniBand networks. All RDMA-enabled sizes are capable of leveraging that network using Intel MPI. SR-IOV stands for “single root input/output virtualization” which optimizes sharing of PCI Express devices in a system with virtual machines. In Azure, SR-IOV for InfiniBand enables near bare-metal performance for any MPI library.

MPI, or message-passing interface, is a communication library commonly used for distributed training between GPUs on many systems. Nvidia’s NCCL software uses MPI to make distributed training easier in deep learning frameworks like PyTorch and TensorFlow.

Azure now supports using any MPI library with SR-IOV enabled VM families such as NCv3, NDv2, and HC or HB for HPC applications. Older GPU hardware with InfiniBand such as NCv2 and NDv1 will be updated for SR-IOV in 2020.

Intel MPI version 5.x will continue to be supported as will all subsequent Intel MPI versions. In addition, all other MPIs supported by the Open Fabric Enterprise Distribution (OFED), OpenMPI, and Nvidia’s NCCL2 library, providing optimized performance for GPUs are supported.

These enhancements will provide customers with higher InfiniBand bandwidth, lower latencies, and most importantly, better distributed application performance. Infiniband connectivity provides higher throughput and lower latencies compared to the ethernet based connection. SR-IOV enables communication over an Infiniband network using any flavor of MPI. A reference implementation of Bert in Azure Machine Learning using SR-IOV and Infiniband can be found on Github.

Throughput Improvement in BERT

SR-IOV and Infiniband provided up to 75% improvement in the throughput of BERT Large model. When SR-IOV is enabled, throughput improves to about 28 sequences/second/GPU which is 75% better than the baseline. Below charts show the throughput improvement of BERT large pretraining on 16 Azure StandardNC24s_v3 VMs. Model is in PyTorch and used Torch.Distributed and Open MPI for multi-node training. Note that the below charts do not reflect the best throughput of BERT on Azure.

Throughput Improvement in ResNet

In order to observe the improvements in speed for PyTorch we ran a selection of ResNet models from Torchvision on synthetic data at full precision. This allowed us to estimate the throughput without having to worry about IO overhead. Below we can see figures for clusters with SR-IOV enabled vs those that didn’t have SR-IOV. We were using NC24rs_v3 vms each equipped with 4 V100 GPUs. Therefore, when we report 8 GPUs it is across 2 nodes and 16 is across 4. We can see that across models and GPU configurations SR-IOV offers 2-3 times improvement over No SR-IOV.

In the figures below the number reported in the center of the bar is the scaling efficiency on RDMA-enabled VMs. As we can see for Horovod and DistributeDataParallel both using NCCL the scaling efficiency is over 90% across all three models with the performance almost doubling with the doubling of GPUs.

Summary

SR-IOV yielded significant throughput improvements to distributed multi machine training. Bert large throughput increased by 75% with SR-IOV and certain Resnet models were faster by about 2-3x with SR-IOV. Throughput also scaled linearly on Resnet models as the number of NC24rs_v3 nodes scaled from 1 to 2, 4 and 8 instances.

Stay tuned for our next blog on scaling distributed deep learning training on Azure NDv2 VMs. These VMs feature 8 NVIDIA Tesla V100 NVLINK interconnected GPUs, 32GB HBM2 memory per GPU and 100Gbps EDR Infiniband interconnect.

Get started with Distributed Deep Learning training on Azure Machine Learning. Report any implementation issues or observed throughput improvements of SR-IOV on Azure Machine Learning at Stack Overflow.