array limitations?
Are there limitations to array sizes? I'm trying to get the line count of very large .txt files ( so far the largest file is 782MB) and also look for duplicates. The script works with small files as a test but when I try to run it against a larger file it doesn't finish and doesn't error. I'm monitoring CPU & memory and powershell_ise.exe where I am running it, is using 25% CPU and a memory working set of 880,568. Overall CPU on the VM is running around 27% and memory 50%, so it isn't topping out physical resources. The script does a few arrays to capture folders & subfolders and check subfolder names and then does counts of the .txt files found in the subfolders. This is a snippet of the section in question. ForEach ($sCARDHIST in $aCARDHISTFILES) { $sCARDHIST1 = $sCARDHIST.FullName ### get line counts in txt ### $sCARDHISTLINES = (Get-Content $sCARDHIST1).length ### Look for duplicates in file ### $aDUPS = Get-Content $sCARDHIST1 | group-Object | Where-Object {$_.Count -gt 1} | Select -ExpandProperty Name If ($aDUPS.count -gt 0 ) { echo "Duplicates found in $sCARDHIST1" >>$sFLAGFILE Write-Host "" Write-Host "=================================================" Write-Host "duplicate records found in $sCARDHIST1" Write-Host "=================================================" ForEach ($sDUP in $aDUPS) { FuncLogWrite "$sDUP" Write-Host "$sDUP" } } } Does anyone have information on array limitations or is there a better way in powershell to look for duplicates and line counts?