Express Provisioning of Azure-SSIS Integration Runtime

Published Jun 24 2019 11:43 PM 4,157 Views
Microsoft

As we continue to improve customer experience in using Azure-SSIS Integration Runtime (SSIS IR), we are excited to announce that SSIS IR can now be provisioned within 5 minutes!

 

In the past, provisioning SSIS IR took at least 20-30 minutes.  This is because the underlying nodes/Virtual Machines (VMs) would have to be created and prepared just in time from scratch, leading to the prolonged 20-30 minutes duration.   This has been a customer pain point, especially when demoing/trying out SSIS IR as a Proof of Concept (PoC) that needs to be done quickly.

 

Now SSIS IR creation can leverage pools of standby VMs that have been created and prepared in advance.  The standby VM types and quantities are chosen based on historical data.  When there is a provisioning request for SSIS IR, its VMs are allocated from a pool of standby VMs.  As a result, the provisioning time is now reduced by around 80%.

 

At present, our express provisioning can support all SSIS IRs outside a Virtual Network (VNet), with or without custom setups that can be configured following this documentation.  Due to current technical limitations, standby VMs cannot be leveraged to shorten the provisioning time of SSIS IRs joining/inside a VNet.

 

Based on our recent monitoring, more than 95% of SSIS IR creations outside a VNet have benefited from our express provisioning and excluding any custom setup time, more than 95% of them have been provisioned in under 5 minutes.

 

We hope you will find this improvement useful.  Please do not hesitate to contact us if you have any feedbacks/questions/issues and we will follow up ASAP.  Thank you as always for your support.

18 Comments
New Contributor

Hi,

 

This is great news, is this going to work all types of VM's or only for a subset?

Microsoft

@edkreuk72 All VM size are supported, we reserved VM size and the number of VM based on history data, VM size and the number of VM is also keeping adjusted based on usage.

Frequent Visitor

Today I started a runtime service, and it's taken between 20 and 30 minutes.

What am I doing wrong?

Regular Visitor

@Michelon777 Do you use vNet? We do not support express provision for IR with vNet. Also express provision hit rate is 95%+, but there is some chance some provision cannot benefit express provision, we are continuing improve the algorithm to try achieve higher hit rate. If you can send me the subscription ID/ data factory name/IR name, I can check from backend why your IR is not benefit from express provision.

Occasional Visitor

Hi,

 

For the last few days my SSIS IR has been taking around 30 minutes to start up. Previously it was taking less than 5 minutes.  Are there any known performance issues at the moment or anything I can do to improve performance myself?

 

Thanks,

Nic

Microsoft

@nicnzl can you open a ticket via Azure portal so we can check more? We are working to continue improving the hit rate.

Microsoft

@nicnzl there was a livesite issue that caused the express provisioning stop working when you have a custom setup script enabled. This issue has already been resolved. 

Regular Visitor

Hi,

My SSIS-IR is also taking 20 - 30 minutes to startup.  It references a custom URI for a DLL created in-house that is registered in the GAC for reference during the ETL.  We are not on a VNET.  Any help would be greatly appreciated.

Thank you,

Paula

New Contributor

@Paula-6967, Did the SSIS-IR started before faster or has always be around 20-30 minutes to startup. Did you try to start it without the the custom DDL registering? 

Regular Visitor

Hi,

Yes, it has started faster before.  But recently, within the last month or 2, it has been much slower.  I have not tried without the custom DLL because the IR cannot be used by our ETL without it.  Would you like me to try the startup without the DLL just for timing to let you know?

Thanks!

New Contributor

@Paula-6967 yes that's is a option. Or just create a new one and see if then will start faster. With this way you can delete easily afterwards or @VincentHe do you have another option?

Regular Visitor

Hi,

The SSIS-IR did started in less than 10 minutes without the custom DLL.  To not use the DLL would require a major redesign of the ETL.  Obviously, it would be our preference to be able to have the startup be responsive with the DLL included.  If you say that is not going to be possible, I will go back to management and ask which they prefer, redesign or wait.  I appreciate your help.

Thanks,

Paula

New Contributor

@Tim.Chen and @VincentHe any suggestion on this question?

Regular Visitor

@edkreuk72 @Paula-6967 I will check from backend log to see the overall hit rate?
In the meanwhile, can you open a ticket via Azure portal so we can check more? 

 

Regular Visitor

Hi @VincentHe021 ,

Thank you for checking the logs.  I'm not able to create a ticket in the portal because we do not have a support plan at this time.  I am able to begin the creation process but when reaching the details section am guided to asking the community or purchasing a plan.   Is there anything else I could do instead?

 

Thanks for the help.  It is appreciated.

Paula

Regular Visitor

@Paula-6967 can you let me know the region you are using? Or send me the subscription id ? So I can check that region and adjust the cached computer resource number to improve the hit rate. thanks

Visitor

Could you please advise as to when you will have a solution that reduces startup time within a VNet? Given that this is best practice for security concerns, this is something that would be expected for a secure and responsive user experience.

Microsoft

@nigelmeakins , this is being worked on now. Tentatively it should go to public early next year. 

%3CLINGO-SUB%20id%3D%22lingo-sub-719058%22%20slang%3D%22en-US%22%3EExpress%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-719058%22%20slang%3D%22en-US%22%3E%3CP%3EAs%20we%20continue%20to%20improve%20customer%20experience%20in%20using%20Azure-SSIS%20Integration%20Runtime%20(SSIS%20IR)%2C%20we%20are%20excited%20to%20announce%20that%20SSIS%20IR%20can%20now%20be%20provisioned%20within%205%20minutes!%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20the%20past%2C%20provisioning%20SSIS%20IR%20took%20at%20least%2020-30%20minutes.%26nbsp%3B%20This%20is%20because%20the%20underlying%20nodes%2FVirtual%20Machines%20(VMs)%20would%20have%20to%20be%20created%20and%20prepared%20just%20in%20time%20from%20scratch%2C%20leading%20to%20the%20prolonged%2020-30%20minutes%20duration.%20%26nbsp%3B%26nbsp%3BThis%20has%20been%20a%20customer%20pain%20point%2C%20especially%20when%20demoing%2Ftrying%20out%20SSIS%20IR%20as%20a%20Proof%20of%20Concept%20(PoC)%20that%20needs%20to%20be%20done%20quickly.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENow%20SSIS%20IR%20creation%20can%20leverage%20pools%20of%20standby%20VMs%20that%20have%20been%20created%20and%20prepared%20in%20advance.%26nbsp%3B%20The%20standby%20VM%20types%20and%20quantities%20are%20chosen%20based%20on%20historical%20data.%26nbsp%3B%20When%20there%20is%20a%20provisioning%20request%20for%20SSIS%20IR%2C%20its%20VMs%20are%20allocated%20from%20a%20pool%20of%20standby%20VMs.%26nbsp%3B%20As%20a%20result%2C%20the%20provisioning%20time%20is%20now%20reduced%20by%20around%2080%25.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAt%20present%2C%20our%20express%20provisioning%20can%20support%20all%20SSIS%20IRs%20outside%20a%20Virtual%20Network%20(VNet)%2C%20with%20or%20without%20custom%20setups%20that%20can%20be%20configured%20following%20this%20%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fdocs.microsoft.com%252Fazure%252Fdata-factory%252Fhow-to-configure-azure-ssis-ir-custom-setup%26amp%3Bdata%3D02%257C01%257Cdashe%2540microsoft.com%257Cd4764ae3d44a4daa6e8c08d6f3cc6f5c%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C636964458011331142%26amp%3Bsdata%3DhnEG2BomVmDiigBzltAs8P%252B1cc8X7%252BJ%252FvGsA2Qa%252Bbuw%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Edocumentation%3C%2FA%3E.%26nbsp%3B%20Due%20to%20current%20technical%20limitations%2C%20standby%20VMs%20cannot%20be%20leveraged%20to%20shorten%20the%20provisioning%20time%20of%20SSIS%20IRs%20joining%2Finside%20a%20VNet.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EBased%20on%20our%20recent%20monitoring%2C%20more%20than%2095%25%20of%20SSIS%20IR%20creations%20outside%20a%20VNet%20have%20benefited%20from%20our%20express%20provisioning%20and%20excluding%20any%20custom%20setup%20time%2C%20more%20than%2095%25%20of%20them%20have%20been%20provisioned%20in%20under%205%20minutes.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20hope%20you%20will%20find%20this%20improvement%20useful.%26nbsp%3B%20Please%20do%20not%20hesitate%20to%20contact%20us%20if%20you%20have%20any%20feedbacks%2Fquestions%2Fissues%20and%20we%20will%20follow%20up%20ASAP.%26nbsp%3B%20Thank%20you%20as%20always%20for%20your%20support.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-723582%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-723582%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThis%20is%20great%20news%2C%20is%20this%20going%20to%20work%20all%20types%20of%20VM's%20or%20only%20for%20a%20subset%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-724632%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-724632%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F296550%22%20target%3D%22_blank%22%3E%40edkreuk72%3C%2FA%3E%26nbsp%3BAll%20VM%20size%20are%20supported%2C%20we%20reserved%20VM%20size%20and%20the%20number%20of%20VM%20based%20on%20history%20data%2C%20VM%20size%20and%20the%20number%20of%20VM%20is%20also%20keeping%20adjusted%20based%20on%20usage.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-985124%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-985124%22%20slang%3D%22en-US%22%3E%3CP%3EToday%20I%20started%20a%20runtime%20service%2C%20and%20it's%20taken%20between%2020%20and%2030%20minutes.%3C%2FP%3E%3CP%3EWhat%20am%20I%20doing%20wrong%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-985965%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-985965%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F445381%22%20target%3D%22_blank%22%3E%40Michelon777%3C%2FA%3E%26nbsp%3BDo%20you%20use%20vNet%3F%20We%20do%20not%20support%20express%20provision%20for%20IR%20with%20vNet.%20Also%20express%20provision%20hit%20rate%20is%2095%25%2B%2C%20but%20there%20is%20some%20chance%20some%20provision%20cannot%20benefit%20express%20provision%2C%20we%20are%20continuing%20improve%20the%20algorithm%20to%20try%20achieve%20higher%20hit%20rate.%20If%20you%20can%20send%20me%20the%20subscription%20ID%2F%20data%20factory%20name%2FIR%20name%2C%20I%20can%20check%20from%20backend%20why%20your%20IR%20is%20not%20benefit%20from%20express%20provision.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1447282%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1447282%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EFor%20the%20last%20few%20days%20my%20SSIS%20IR%20has%20been%20taking%20around%2030%20minutes%20to%20start%20up.%20Previously%20it%20was%20taking%20less%20than%205%20minutes.%26nbsp%3B%20Are%20there%20any%20known%20performance%20issues%20at%20the%20moment%20or%20anything%20I%20can%20do%20to%20improve%20performance%20myself%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%2C%3C%2FP%3E%3CP%3ENic%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1447572%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1447572%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F691874%22%20target%3D%22_blank%22%3E%40nicnzl%3C%2FA%3E%26nbsp%3Bcan%20you%20open%20a%20ticket%20via%20Azure%20portal%20so%20we%20can%20check%20more%3F%20We%20are%20working%20to%20continue%20improving%20the%20hit%20rate.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1461692%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1461692%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F691874%22%20target%3D%22_blank%22%3E%40nicnzl%3C%2FA%3E%26nbsp%3Bthere%20was%20a%20livesite%20issue%20that%20caused%20the%20express%20provisioning%20stop%20working%20when%20you%20have%20a%20custom%20setup%20script%20enabled.%20This%20issue%20has%20already%20been%20resolved.%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1771729%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1771729%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3EMy%20SSIS-IR%20is%20also%20taking%2020%20-%2030%20minutes%20to%20startup.%26nbsp%3B%20It%20references%20a%20custom%20URI%20for%20a%20DLL%20created%20in-house%20that%20is%20registered%20in%20the%20GAC%20for%20reference%20during%20the%20ETL.%26nbsp%3B%20We%20are%20not%20on%20a%20VNET.%26nbsp%3B%20Any%20help%20would%20be%20greatly%20appreciated.%3C%2FP%3E%3CP%3EThank%20you%2C%3C%2FP%3E%3CP%3EPaula%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1772085%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1772085%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F829219%22%20target%3D%22_blank%22%3E%40Paula-6967%3C%2FA%3E%2C%20Did%20the%20SSIS-IR%20started%20before%20faster%20or%20has%20always%20be%20around%2020-30%20minutes%20to%20startup.%20Did%20you%20try%20to%20start%20it%20without%20the%20the%20custom%20DDL%20registering%3F%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1772111%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1772111%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3EYes%2C%20it%20has%20started%20faster%20before.%26nbsp%3B%20But%20recently%2C%20within%20the%20last%20month%20or%202%2C%20it%20has%20been%20much%20slower.%26nbsp%3B%20I%20have%20not%20tried%20without%20the%20custom%20DLL%20because%20the%20IR%20cannot%20be%20used%20by%20our%20ETL%20without%20it.%26nbsp%3B%20Would%20you%20like%20me%20to%20try%20the%20startup%20without%20the%20DLL%20just%20for%20timing%20to%20let%20you%20know%3F%3C%2FP%3E%3CP%3EThanks!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1772183%22%20slang%3D%22en-US%22%3ERe%3A%20Express%20Provisioning%20of%20Azure-SSIS%20Integration%20Runtime%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1772183%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F829219%22%20target%3D%22_blank%22%3E%40Paula-6967%3C%2FA%3E%26nbsp%3Byes%20that's%20is%20a%20option.%20Or%20just%20create%20a%20new%20one%20and%20see%20if%20then%20will%20start%20faster.%20With%20this%20way%20you%20can%20delete%20easily%20afterwards%20or%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F357402%22%20target%3D%22_blank%22%3E%40VincentHe%3C%2FA%3E%26nbsp%3Bdo%20you%20have%20another%20option%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Jun 24 2019 11:55 PM
Updated by: