Script Optimization

[beno_low]

Hello Peeps,

I have a script that read in 2 files.

File A contains FDQN

File B contains HOSTNAME + ENV

The pseudo code is as follows:

For each line in File A

     Split HOST from FDQN

             For each line in File B

                    if match HOST to HOSTNAME, set env = ENV

Using the above logic, i have 2 loops, outerloop A and innerLoop B, the time taken is close to 8 hours, as the number of entries is close to 27000 in each file (27000 x 27000 loops).

How can i make this faster?

[#2]

@beno_low 

Hello beno_low,

You didn’t attach a example input file, so I was not able to test the script below. But I think you can speedup your script al lot by using your RAM instead of reading and writing to disk in each loop. Try something like the code below. I moved the Get-Content $args[1] in line 25 and load that into the $cmdb_content var. The Out-file at line 94 is also moved outside your primairy loop. 

WARNING: You are using $env in your script, this is not a best practice since Powershell is also using $Env to store enviroment variables! More info: https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment...

##############################################################################
#   Filename : gen_host_env.ps1
#   Usage : .\gen_host_env CS_host CMDB_host
#   Both input files must be in text format
#   Output : 



$Final_Result = @()
#$Header = 'CMDB_Host,CMDB_Env,CS_Host,CS_Env,CS_FQDN,L1_Domain,L2_Domain'+"`r`n"
#Write-Host $Header
'CMDB_Host,CMDB_Env,CS_Host,CS_Env,CS_FQDN,L1_Domain,L2_Domain' | Out-File -Encoding ascii -Append .\crowdstrike_host.txt

$cmdb_content = Get-Content $args[1]
$output = @()

foreach ($fqdn in (Get-Content $args[0])){
	
	$comp = $fqdn.Split(".")
    $hname = $comp[0]
    $l1 = $comp[1]
    $l2 = $comp[2]
    $l3 = $comp[3]
    $l4 = $comp[4]
    $l5 = $comp[5]
    $l6 = $comp[6]
    
    foreach ($cmdb_entry in $cmdb_content) {
        $cmdb_split = $cmdb_entry.Split(",")
        $cmdb_host = $cmdb_split[0]
        $cmdb_env = $cmdb_split[1]
        #Write-Host $cmdb_host

        if ($hname -imatch $cmdb_host) {
            $env_frm_cmdb = $cmdb_env
            break
        }
        else {
            $cmdb_host = 'NO match'
            $env_frm_cmdb = 'undetermined'
        
        }
    } 

    #Write-Host $hname
    #Write-Host $l1
 
 #### First Part of FQDN match criteria hostname.prod|pat|dev|.xyz###########     
    if ($l1 -match 'prod'){
        $env = 'prod' }
    
    elseif ($l1 -match 'splunk'){
        $env = 'prod' }

    elseif ($l1 -match 'corp'){
        $env = 'prod' }
                
    elseif ($l1 -match 'dev') {
        $env = 'dev'}
    
    elseif ($l1 -match 'englab') {
        $env = 'dev'}

    elseif ($l1 -match 'sit') {
        $env = 'sit'}

    elseif ($l1 -match 'pat') {
        $env = 'pat'}
  
  #### Second Part of FQDN match criteria hostname.abc.prod|pat|dev|.xyz###########  
    elseif ($l2 -match 'prod'){
        $env = 'prod' }
        
    elseif ($l2 -match 'dev') {
        $env = 'dev'}

    elseif ($l2 -match 'sit') {
        $env = 'sit'}

    elseif ($l2 -match 'pat') {
        $env = 'pat'}

############ Hadoop Server match env starts here #####################
    elseif ($l2 -match 'hadoop') {
        if ($l1 -match 'c01') { $env = 'prod' }
        elseif ($l1 -match 'c02') { $env = 'dev' }
        elseif ($l1 -match 'c03') { $env = 'pat' }
        elseif ($l1 -match 'c2#') { $env = 'sit' }
        elseif ($l1 -match 'c5#') { $env = 'sit' }
    }

    else {
        $env = 'undetermine'}

    $Final_Result += $cmdb_host+","+$env_frm_cmdb+","+$hname+","+$env+","+$fqdn+","+$l1+","+$l2+"`r`n"
    #Write-Host $Final_Result
    $output += $cmdb_host+","+$env_frm_cmdb+","+$hname+","+$env+","+$fqdn+","+$l1+","+$l2 
    }

$output | Out-File -Encoding ascii -Append .\crowdstrike_host.txt

Good luck!

Grtz, 

Manfred de Laat

[#3]

@Manfred101 

Thanks, Manfred. Yes, it helps to reduce the time from 8 hours down to 2 hours approximately.