Optimizing the utilization of PowerShell processing through multi-threading techniques such as runspaces and multiple jobs can be done in several ways. These techniques can often be intricate and demand significant effort to implement, particularly for individuals new to PowerShell. This blog post will present an alternative approach to multi-threading utilizing a simple parameter, without the complexities generally associated with creating and managing of runspaces and multiple jobs in PowerShell.
What is the switch?
The “Parallel” switch, introduced in PowerShell 7, allows script blocks to execute concurrently for each piped input object. This feature is particularly advantageous for bulk operations where tasks can be performed simultaneously. By utilizing this capability, one can significantly reduce processing time and improve efficiency.
This functionality requires PowerShell 7. You can follow this article to upgrade to PS v7. Older versions of PowerShell, such as version 5, do not support the Parallel switch.
How to use this?
The Parallel switch is available only when you write a ForEach-Object loop.
The ‘foreach’ which is an alias of the ForEach-Object loop is not supported currently.
For example:
$data | ForEach-Object -Parallel { script block }
The Parallel switch will open PowerShell instances hidden in the background but connected to the currently open instance and run the script block simultaneously in those instances. Any errors or outputs will be displayed in the currently open parent instance. The number of parallel instances to be opened can be defined with the ThrottleLimit switch trailing behind the script block. The default ThrottleLimit is 5 and the PowerShell will open 5 parallel session if the ThrottleLimit switch is not defined.
$data | ForEach-Object -Parallel { script block } -ThrottleLimit 5
Considerations
- Please note that Parallel sessions run in their own isolated spaces and do not share dependencies amongst themselves. Therefore, you will need to define all necessary elements, such as authentication tokens, and modules (if required), inside the script block.
- Each parallel session will use the available system resources and thus the throttle limit should be set to the number of available cores on the system.
- Bear in mind that this parallelising feature in PowerShell reduces the overall time taken for a script to run serially. Thus, this it is useful for tasks that are independent of previous executions, such as bulk operations, specially as it reduces overall time and effort taken for the bulk operation.
A real-life example
Problem: A colleague requested assistance in adding trusted domains and email addresses to 15,000 mailboxes. Processing of each object takes ~5 seconds to complete, resulting in a total of ~75,000 seconds (~20.8 hours).
Solution: The following code was provided to complete the task in less than 5 hours:
$users = ipcsv C:\input_file_location.csv
$users | ForEach-Object -Parallel {
$conn = Get-Mailbox -Identity valid-test-account@domain.com
if ($conn -eq $null) {
Connect-ExchangeOnline -UserPrincipalName exo-admin-account@domain.com -UseMultithreading:$true }
Set-MailboxJunkEmailConfiguration -Identity $_.identity -TrustedSendersAndDomains @{Add="no-reply@sharepointonline.com","noreply@yammer.com","noreply@planner.office365.com" } -ThrottleLimit 5
Disconnect-ExchangeOnline -Confirm:$false Script explanation:
The following line would create a variable called $users and import the data from csv file located at the location “C:\input_file_location.csv”:
$users = ipcsv C:\input_file_location.csvAbout the following block... As parallel sessions are opened independently, the Auth token from one session cannot be transferred to another session due to the token’s secure design. Therefore, the command to connect to Exchange Online (EXO) must be placed within the parallel loop so that the loop gets logged in to EXO. However, this approach will initiate as many PowerShell connections to EXO as there are users in the $users variable. This can result in the account being flagged in Risky Sign-Ins, with subsequent connections after the first ten being blocked.
An effective way to address this is to implement a conditional login to check if the session is already connected and connect only if it is not. By using a simple Get-Mailbox command for any test user in the environment to verify that the session is connected, and storing the result in $conn, the parallel session will only connect to EXO if $conn is null. This approach ensures that only one connection is made for each parallel session and addresses the risk of the account being flagged for Risky Sign-Ins.
Further, using the SkipLoadingCmdletHelp switch prevents the help file to be downloaded and occupy memory resources which can overburden the system. This is an optional switch and is required only if using Exchange Online PowerShell module older than version 3.7. This switch is not available in module ver. 3.7 and above. See this post for more information.
$users | ForEach-Object -Parallel {
$conn = Get-Mailbox -Identity valid-test-account@domain.com
if ($conn -eq $null) {
Connect-ExchangeOnline -UserPrincipalName exo-admin-account@domain.com -SkipLoadingCmdletHelp}Sharing data between the parent loop and the parallel sessions can be achieved using the $using technique; however, this will be covered in the next blog post!
Thank you for reading!
Abhijeet Kowale and Indraneel Roy
Microsoft
