Choosing the Right Language Analyzer for Azure AI Search

By Abhishree Shetty, Alec Berntson, Lihang Li

A language analyzer is a specific type of text analyzer that performs lexical analysis using the linguistic rules of the target language. It applies language-specific rules to convert raw text chunks into searchable terms, handling aspects like word boundaries, word forms, and compound words.

If you build search experiences on Azure AI Search, the analyzer you choose directly affects the quality of keyword search results. Azure AI Search offers Lucene and Microsoft language analyzers covering 50+ languages, but there has been no systematic guidance on which to use or whether they matter at all. 

We compared Lucene, Microsoft, and no analyzer across 20 languages and 3 datasets. This post covers what we found, how customers use analyzers today, why results differ between Lucene and Microsoft, and how to pick the right one for your content. 

TL;DR 

• Always use a language analyzer. +15% NDCG on average, up to +120% for morphologically complex languages.
• Lucene and Microsoft are both strong. Differences are small; Microsoft has broader language coverage.
• No latency penalty. Analyzer choice does not affect query search performance.
• Some languages have a clear preference.  For example, German and Spanish consistently favor Lucene, while Chinese favors Microsoft. Read the post for complete per-language breakdown. 

Table of Contents

What Does a Language Analyzer Do?

By default, Azure AI Search uses a standard tokenizer that splits text on whitespace and punctuation and lowerscases everything. It has no understanding of language. A language analyzer adds linguistic intelligence to this process: 

Segmentation — Identifying word boundaries in languages that lack whitespace: 

서울에서 맛있는 음식점 추천해주세요 → 서울 / 맛있 / 음식점 / 추천 

Lemmatization — Reducing inflected words to their base form: 

fuimos, vamos, yendo → ir 

Compound decomposition — Breaking compound words into meaningful parts: 

Krankenhausverwaltungssystem → Kranken + haus + verwaltung + system 

Azure AI Search supports the following analyzer options: 

Table 1: Language analyzer options in Azure AI Search

 

Here is how the three built-in options processes text across different languages: 

 Table 2: How each analyzer tokenizes the same input text 

The tokens in Table 2 were generated using Azure AI Search Analyze API. You can try it yourself with your own text. 

Does It Even Matter? 

Out of 31 language–dataset combinations we tested, the standard tokenizer (None) never won. In every case, a language analyzer delivered better results. 

Table 3: Average NDCG@10 across all 31 language–dataset combinations

Improvements vary by language. Some see massive lifts; others see modest gains: 

Table 4: Languages with the largest retrieval quality lift from using a language analyzer

Morphologically complex languages (Finnish, Korean, Polish) benefit the most — lifts exceed +70%. CJK languages also see large gains because they lack whitespace word boundaries. But the lift extends well beyond these: Arabic (+34%), Spanish (+55%), French (+18%), and Turkish (+14%) all see double-digit improvements. Even English on well-formatted content shows a +5.9% lift (MIML). On enterprise support content, the lift grows to +14.1%. 

Key takeaway: Always use a language analyzer.

What Do Customers Actually Do Today? 

We examined analyzer usage across Azure AI Search indexes. 

Table 5: Language analyzer adoption metrics

Among customers who do use an analyzer, behavior largely aligns with study findings: 

• French: Customers favor Lucene, consistent with our results.
• Korean: Customers favor Microsoft, also consistent.
• English: Customers lean Microsoft (68.4%). Our study shows Microsoft wins 2 of 3 datasets for English, consistent with this preference. 

Key takeaway: Most customers today don't use a language analyzer.

Lucene vs Microsoft — The Full Picture 

Table 6: NDCG@10 wins by analyzer across all 31 combinations

Microsoft wins more combinations overall, but the differences are often small. Here are some examples: 

• English: Microsoft 0.9413 vs Lucene 0.9383 — difference of 0.0030 

• Chinese: Microsoft 0.5668 vs Lucene 0.5646 — difference of 0.0022 

From the Support dataset: 

• Danish: Microsoft 0.3044 vs Lucene 0.3028 — difference of 0.0016 

• Korean: Lucene 0.3252 vs Microsoft 0.3218 — difference of 0.0034 

There is no meaningful quality difference between these two. 

Language coverage 

Microsoft analyzers support 50 languages; Lucene supports ~35. For 20+ languages, Microsoft is the only option: Croatian, Estonian, Gujarati, Hebrew, Hindi, Icelandic, Kannada, Latvian, Lithuanian, Malayalam, Malay, Marathi, Punjabi, Serbian, Slovak, Slovenian, Tamil, Telugu, Ukrainian, Urdu, and Vietnamese. 

Corpus type effects 

We define two content categories: 

• Clean text: Well-formatted, edited content — academic papers, Wikipedia articles, curated knowledge bases. Minimal typos, consistent structure. 

• Noisy text: User-generated or operational content — support tickets, short queries, product forums. Frequent typos, mixed languages, informal grammar. 

The choice between Lucene and Microsoft depends on which type your content resembles: 

Table 7: Analyzer performance by content type

On clean data, Lucene has an edge (+7.6% average NDCG over Microsoft on MIML). On noisy data, Microsoft wins decisively — 13 of 18 languages, with +12.4% higher average NDCG than Lucene.

Under the Hood — Why Results Differ 

The core difference is their approach to morphological analysis. 

Lucene 

Lucene language analyzers reduce words to short stems using rule-based algorithms (Porter, Snowball). By collapsing inflected forms to a common root, stemming increases recall — a query for attach will match documents containing attachments, attached, or attaching. This is particularly effective on clean, well-structured text where broader matching helps surface relevant content. 

The tradeoff is precision. On noisy text, stemming can be too aggressive: 

Table 8: Examples of Lucene stemming behavior

Microsoft 

Microsoft analyzers preserve the original term and add normalized variants — for example, settings yields both settings and setting. This gives recall benefits similar to stemming while keeping the original form available for precise matching. 

Microsoft analyzers also: 

• Preserve proper nouns: Windows stays Windows rather than being stemmed
• Decompose compounds: Swedish snabbstart → snabb + start
• Retain more stopwords: Where Lucene removes functional words like in, not, and, Microsoft keeps more of them 

Comparing on noisy text 

Consider a typical support query: "how to fix Windows settings not responding after update" 

• None (9 tokens): how, to, fix, windows, settings, not, responding, after, update
• Lucene (7 tokens): how, fix, window, set, respond, after, updat — drops to, not, windows; collapses settings → set and update → updat
• Microsoft (10 tokens): how, fix, window, windows, setting, settings, respond, responding, after, update — keeps originals and adds normalized forms 

Lucene loses not (negation context) and settings becomes the generic set. Microsoft preserves the original terms and adds variants, keeping the query's meaning intact. 

On clean text, Lucene's stemming is mostly beneficial — broader matching helps recall. On noisy text, the same aggressiveness can compound the problem, which is why Microsoft tends to perform better on noisy data. 

Is There a Latency Cost during Search? 

Language analyzers run at both index time and query time, but the heavier processing happens during indexing. At query time, all three options produce identical p50 and p95 latency. We verified this across multiple languages under concurrent load. Your choice of analyzer will not affect search performance. 

Practical Recommendations 

Decision framework 

Figure 1: Decision framework for choosing a language analyzer in Azure AI Search

Language-specific overrides 

For three languages, our data is consistent across all datasets tested: 

Table 9: Languages with a consistent analyzer preference across all datasets tested

For other languages, the best choice depends on your content type — Lucene tends to win on clean data, Microsoft on noisy data.
Note: These results are based on our three test datasets. We recommend validating your own data using the Analyze API. 

Key Takeaways 

1. Always configure a language analyzer. The biggest gain comes from using any analyzer vs None (+15% average). The Lucene-vs-Microsoft difference is secondary. 
2. Both Lucene and Microsoft are safe choices. Always-Lucene captures 95.0% of optimal NDCG; Always-Microsoft captures 96.7%. 
3. There is no latency penalty. Analyzer choice does not affect query performance. 
4. Use per-field analyzers for multilingual indexes. Each language field should use its own analyzer for best results. 
5. Microsoft is the broader default. 50 languages, meaning-preserving normalization, and the only option for 20+ languages. 

Appendix A: Datasets and Methodology 

Datasets 

We evaluated across three datasets chosen for their diversity in content type, language coverage, and query difficulty: 

Table 10: Datasets used in this study

Hole filling 

In all three datasets, top-10 retrieved documents often include unjudged items ("holes"). We used GPT-4o to fill these holes: for each unjudged document, the model rates relevance on a 0–4 scale (ABYSMAL to IDEAL). Documents rated ≥ 2 (FAIR) are counted as relevant. This ensures NDCG@10 is computed over a complete set of judgments rather than penalizing unjudged documents. 

Metric 

We use NDCG@10 (Normalized Discounted Cumulative Gain at rank 10) as the primary metric throughout this post. It measures ranking quality — whether the most relevant documents are ranked highest in the top-10 results. 

Experiment setup 

• 93 experiments: 31 language–dataset combinations × 3 analyzer types (None, Lucene, Microsoft)
• Search service: Azure AI Search (BM25 retrieval, top-10 results)
• Evaluation: Per-query metrics with paired statistical significance testing (t-test + Wilcoxon signed-rank) 

Appendix B: Detailed Results Per Dataset 

MIML (6 languages, 12,970 queries) 

Table 11: MIML: NDCG@10 per language and analyzer

MIRACL (7 languages, 1,750 queries) 

Table 12: MIRACL: NDCG@10 per language and analyzer

Support (18 languages, 16,178 queries) 

Table 13: Support: NDCG@10 per language and analyzer