Home
%3CLINGO-SUB%20id%3D%22lingo-sub-1106493%22%20slang%3D%22en-US%22%3EAnnouncing%20the%20Python%20Custom%20Skills%20Toolkit%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1106493%22%20slang%3D%22en-US%22%3E%3CH1%20id%3D%22toc-hId-380668535%22%20id%3D%22toc-hId-380668535%22%3EAnnouncing%20the%20Python%20Custom%20Skills%20Toolkit%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAre%20you%20working%20with%20Azure%20Cognitive%20Search%20and%20need%20to%20create%20custom%20skills%3F%20Are%20you%20in%20need%20to%20do%20some%20basic%20operations%20within%20your%20enrichment%20pipeline%20but%20prefer%20to%20use%20Python%3F%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FRodrigossz%2FPyhton-Custom-Skills-Toolkit%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EThis%3C%2FA%3E%20GitHub%20repository%20may%20be%20a%20good%20help%20for%20you.%20It%20is%20a%20collection%20of%20Azure%20Functions%20written%20in%20Python%20for%20special%20characters%20removal%2C%20dates%20extraction%2C%20filtering%2C%20and%20more.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20you%20want%20to%20know%20more%20about%20Azure%20Functions%20for%20Python%20in%20Azure%20Cognitive%20Search%20projects%2C%20and%20why%20to%20use%20it%2C%20click%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fai-customer-engineering-team%2Fusing-azure-functions-for-python-to-create-a-cognitive-search%2Fba-p%2F850106%22%20target%3D%22_blank%22%20rel%3D%22noopener%22%3Ehere%3C%2FA%3E%20to%20read%20our%20previous%20blog%20post%20about%20it.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId--1426785928%22%20id%3D%22toc-hId--1426785928%22%3EWhat%20is%20this%20Toolkit%3F%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIt%20is%20a%20collection%20of%20useful%20python%20functions%20ready%20to%20be%20deployed%20as%20Azure%20Cognitive%20Search%20custom%20skills.%20The%20skills%20can%20be%20used%20as%26nbsp%3B%3CSTRONG%3Etemplates%20or%20starting%20points%3C%2FSTRONG%3E%26nbsp%3Bfor%20your%20own%20custom%20skills%2C%20or%20they%20can%20be%20deployed%20and%20used%20as%20they%20are%20if%20they%20happen%20to%20meet%20your%20requirements.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAll%20code%20was%20written%20for%20Azure%20Functions%20in%20Python%203.7%2C%20to%20address%20specific%20projects%20requirements.%20The%20functions%20are%20turbo-charged%20with%20detailed%20comments%20for%20easy%20understanding%20and%20customization.%20They%20also%20include%20limitations%20and%20restrictions.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20content%20will%20help%20you%20to%20prepare%20your%20development%20environment%2C%20as%20well%20as%20the%20tests%20and%20the%20deployment.%20Best%20practices%20and%20lessons%20learned%20are%20also%20included.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId-1060726905%22%20id%3D%22toc-hId-1060726905%22%3EWhat%20are%20Custom%20Skills%3F%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20Custom%20Web%20API%20skill%20allows%20you%20to%20extend%20AI%20enrichment%20by%20calling%20out%20to%20a%20Web%20API%20endpoint%20providing%20custom%20operations.%20Like%20built-in%20skills%2C%20a%20Custom%20Web%20API%20skill%20has%20inputs%20and%20outputs.%20Depending%20on%20the%20inputs%2C%20your%20Web%20API%20receives%20a%20JSON%20payload%20when%20the%20indexer%20runs%2C%20and%20outputs%20a%20JSON%20payload%20as%20a%20response%2C%20along%20with%20a%20success%20status%20code.%20To%20learn%20more%20about%20Custom%20Skills%2C%20click%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsearch%2Fcognitive-search-custom-skill-interface%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehere%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F165220iA415BFEBF24EB11C%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22clipboard_image_0.png%22%20title%3D%22clipboard_image_0.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EFigure%201%3A%20How%20custom%20skills%20are%20integrated%20into%20the%20reference%20architecture%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId--746727558%22%20id%3D%22toc-hId--746727558%22%3EWhat%20is%20offered%3F%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CTABLE%20style%3D%22width%3A%20623px%3B%22%20width%3D%22624%22%3E%0A%3CTHEAD%3E%0A%3CTR%3E%0A%3CTD%20width%3D%2281%22%20style%3D%22width%3A%2080px%3B%22%3E%3CP%3E%3CSTRONG%3ESkill%3C%2FSTRONG%3E%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22544%22%20style%3D%22width%3A%20543px%3B%22%3E%3CP%3E%3CSTRONG%3EWhen%20to%20Use%3C%2FSTRONG%3E%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3C%2FTHEAD%3E%0A%3CTBODY%3E%0A%3CTR%3E%0A%3CTD%20width%3D%2281%22%20style%3D%22width%3A%2080px%3B%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FRodrigossz%2FPyhton-Custom-Skills-Toolkit%2Fblob%2Fmaster%2Fcode%2Fdates-extractor.md%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EDates%20Extractor%3C%2FA%3E%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22544%22%20style%3D%22width%3A%20543px%3B%22%3E%3CP%3EExtracts%20dates%20from%20string.%20Differentiates%20itself%20from%20the%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsearch%2Fcognitive-search-skill-entity-recognition%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EEntity%20Extraction%20built-in%20skill%3C%2FA%3E%26nbsp%3Bby%20generating%20dates%20in%20yyyy-mm-dd%20format.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%2281%22%20style%3D%22width%3A%2080px%3B%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FRodrigossz%2FPyhton-Custom-Skills-Toolkit%2Fblob%2Fmaster%2Fcode%2Fstrings-merger.md%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EStrings%20Merger%3C%2FA%3E%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22544%22%20style%3D%22width%3A%20543px%3B%22%3E%3CP%3EMerges%202%20strings.%20Differentiates%20itself%20from%20the%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsearch%2Fcognitive-search-skill-textmerger%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EText%20Merger%20built-in%20skill%3C%2FA%3E%26nbsp%3Bby%20allowing%20you%20to%20merge%20any%202%20strings%2C%20not%20only%20the%20content%20with%20the%20OCR%20text%20extracted%20from%20images.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%2281%22%20style%3D%22width%3A%2080px%3B%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FRodrigossz%2FPyhton-Custom-Skills-Toolkit%2Fblob%2Fmaster%2Fcode%2Fstrings-cleaner.md%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EStrings%20Cleaner%3C%2FA%3E%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22544%22%20style%3D%22width%3A%20543px%3B%22%3E%3CP%3ERemoves%20special%20characters%20from%20strings%2C%20returning%20a%20string%20clean%20of%20those%20values.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%2281%22%20style%3D%22width%3A%2080px%3B%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FRodrigossz%2FPyhton-Custom-Skills-Toolkit%2Fblob%2Fmaster%2Fcode%2Fcsv-filter.md%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3ECSV%20Filter%3C%2FA%3E%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22544%22%20style%3D%22width%3A%20543px%3B%22%3E%3CP%3ERemoves%20the%20csv%20file%20values%20from%20the%20input%2C%20returning%20a%20string%20clean%20of%20those%20values.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%2281%22%20style%3D%22width%3A%2080px%3B%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FRodrigossz%2FPyhton-Custom-Skills-Toolkit%2Fblob%2Fmaster%2Fcode%2Fcsv-lookup.md%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3ECSV%20Lookup%3C%2FA%3E%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22544%22%20style%3D%22width%3A%20543px%3B%22%3E%3CP%3EExtracts%20the%20csv%20file%20values%20that%20were%20found%20in%20the%20input%20string%2C%20returning%20an%20array%20of%20strings.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20align%3D%22center%22%20style%3D%22width%3A%2080px%3B%22%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FRodrigossz%2FPyhton-Custom-Skills-Toolkit%2Fblob%2Fmaster%2Fcode%2Fstrings-distinct.md%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3EStrings%20Distinct%3C%2FA%3E%3C%2FTD%3E%0A%3CTD%20align%3D%22left%22%20style%3D%22width%3A%20543px%3B%22%3ERemoves%20duplicated%20elements%20from%20the%20input%20array.%20Useful%20when%20you%20are%20extracting%20entities%20or%20key%20phrases%20per%20page%2C%20and%20some%20values%20are%20present%20in%20multiple%20pages.%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3C%2FTBODY%3E%0A%3C%2FTABLE%3E%0A%3CH1%20id%3D%22toc-hId-1740785275%22%20id%3D%22toc-hId-1740785275%22%3EAdditional%20Links%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EACE%20Team%20-%20Python%20Custom%20Skills%20Toolkit%26nbsp%3B-%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FRodrigossz%2FPyhton-Custom-Skills-Toolkit%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EGitHub%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3EACE%20Team%20-%20Knowledge%20Mining%26nbsp%3BAccelerator%26nbsp%3B-%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Faka.ms%2Fkma%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eaka.ms%2Fkma%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3EACE%20Team%20-%20Knowledge%20Mining%20Bootcamp%26nbsp%3B%E2%80%93%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Faka.ms%2Fkmb%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eaka.ms%2Fkmb%3C%2FA%3E%26nbsp%3B%3C%2FLI%3E%0A%3CLI%3EACE%20Team%20-%20Knowledge%20Mining%20blog%20posts%26nbsp%3B%E2%80%93%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FAI-Customer-Engineering-Team%2Fbg-p%2FAICustomerEngineeringTeam%2Flabel-name%2FKnowledge%2520Mining%22%20target%3D%22_blank%22%20rel%3D%22noopener%22%3Eaka.ms%2FACE-Blog%3C%2FA%3E%26nbsp%3B%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1106493%22%20slang%3D%22en-US%22%3E%3CP%3EAre%20you%20working%20with%20Azure%20Cognitive%20Search%20and%20need%20to%20create%20custom%20skills%3F%20Do%20you%20prefer%20to%20work%20with%20Python%3F%20This%20blog%20post%20is%20for%20you.%3C%2FP%3E%3C%2FLINGO-TEASER%3E
Microsoft

Announcing the Python Custom Skills Toolkit

 

Are you working with Azure Cognitive Search and need to create custom skills? Are you in need to do some basic operations within your enrichment pipeline but prefer to use Python?

 

This GitHub repository may be a good help for you. It is a collection of Azure Functions written in Python for special characters removal, dates extraction, filtering, and more.

 

If you want to know more about Azure Functions for Python in Azure Cognitive Search projects, and why to use it, click here to read our previous blog post about it.

 

What is this Toolkit?

 

It is a collection of useful python functions ready to be deployed as Azure Cognitive Search custom skills. The skills can be used as templates or starting points for your own custom skills, or they can be deployed and used as they are if they happen to meet your requirements.

 

All code was written for Azure Functions in Python 3.7, to address specific projects requirements. The functions are turbo-charged with detailed comments for easy understanding and customization. They also include limitations and restrictions.

 

The content will help you to prepare your development environment, as well as the tests and the deployment. Best practices and lessons learned are also included.

 

What are Custom Skills?

 

The Custom Web API skill allows you to extend AI enrichment by calling out to a Web API endpoint providing custom operations. Like built-in skills, a Custom Web API skill has inputs and outputs. Depending on the inputs, your Web API receives a JSON payload when the indexer runs, and outputs a JSON payload as a response, along with a success status code. To learn more about Custom Skills, click here.

 

clipboard_image_0.png

Figure 1: How custom skills are integrated into the reference architecture

 

What is offered?

 

Skill

When to Use

Dates Extractor

Extracts dates from string. Differentiates itself from the Entity Extraction built-in skill by generating dates in yyyy-mm-dd format.

Strings Merger

Merges 2 strings. Differentiates itself from the Text Merger built-in skill by allowing you to merge any 2 strings, not only the content with the OCR text extracted from images.

Strings Cleaner

Removes special characters from strings, returning a string clean of those values.

CSV Filter

Removes the csv file values from the input, returning a string clean of those values.

CSV Lookup

Extracts the csv file values that were found in the input string, returning an array of strings.

Strings Distinct Removes duplicated elements from the input array. Useful when you are extracting entities or key phrases per page, and some values are present in multiple pages.

Additional Links

 

  • ACE Team - Python Custom Skills Toolkit - GitHub
  • ACE Team - Knowledge Mining Accelerator - aka.ms/kma
  • ACE Team - Knowledge Mining Bootcamp – aka.ms/kmb 
  • ACE Team - Knowledge Mining blog posts – aka.ms/ACE-Blog