csv file comparison

%3CLINGO-SUB%20id%3D%22lingo-sub-2433916%22%20slang%3D%22en-US%22%3Ecsv%20file%20comparison%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2433916%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20All%2C%3CBR%20%2F%3EI%20am%20comparing%20two%20csv%20files%20based%20on%20USERNAME%20column%20of%20both%20files%2C%20kindly%20see%20my%20script%20below%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3ECompare-Object%20%24SourceFile%20%24TargetFile%20-Property%20%22USERNAME%22%20-PassThru%20%7C%20Where-Object%20%7B%24_.SideIndicator%20-eq%20%22%3D%26gt%3B%22%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3Ethis%20is%20working%20perfectly%20fine%20from%20the%20past%20few%20years.%3C%2FP%3E%3CP%3E%3CBR%20%2F%3ETill%20last%20week%20both%20files%20had%20around%2070%2C000%20records%20in%20each%2C%20so%20this%20comparison%20took%202%20hours%20to%20complete%2C%3CBR%20%2F%3ERecently%20these%20file(source%2C%20target)%20records%20has%20grown%20like%208%2C00%2C000%2C%20now%20this%20comparison%20is%20taking%20around%2048%20hours%20and%20some%20times%20i%20am%20getting%26nbsp%3B%20below%20error.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3E%2B%20CategoryInfo%20%3A%20NotSpecified%3A%20(%3A)%20%5BWhere-Object%5D%2C%20OutOfMemoryException%3C%2FSTRONG%3E%3CBR%20%2F%3E%3CSTRONG%3E%2B%20FullyQualifiedErrorId%20%3A%20System.OutOfMemoryException%2CMicrosoft.PowerShell.Commands.WhereObjectCommand%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EKindly%20suggest%20is%20there%20any%20other%20way%20we%20can%20do%20this%20comparison%2C%20so%20that%26nbsp%3Bwe%20can%20mitigate%20this%20exception%20and%20too%20much%20of%20time%20%3F%3F%3CBR%20%2F%3E%3CBR%20%2F%3EResponse%20would%20be%20appreciated.%3C%2FP%3E%3CP%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2433916%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EWindows%20PowerShell%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2434858%22%20slang%3D%22en-US%22%3ERe%3A%20csv%20file%20comparison%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2434858%22%20slang%3D%22en-US%22%3EI%20saw%20in%20old%20forum%20someting%20related%20to%20increase%20the%20memory%2C%3CBR%20%2F%3E%3CBR%20%2F%3Eexecute%20the%20command%C3%87%3CBR%20%2F%3EGet-Item%20WSMan%3A%5Clocalhost%5CShell%5CMaxMemoryPerShellMB%3CBR%20%2F%3Eby%20default%20it%20is%202147483647%3CBR%20%2F%3E%3CBR%20%2F%3Echeck%20the%20link%20bellow%2C%20it%20have%20a%20solution%20to%20increase%20it.%3CBR%20%2F%3E%3CA%20href%3D%22https%3A%2F%2Fstackoverflow.com%2Fquestions%2F28503765%2Fpowershell-gives-get-content-exception-of-type-system-outofmemoryexception-w%3Frq%3D1%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fstackoverflow.com%2Fquestions%2F28503765%2Fpowershell-gives-get-content-exception-of-type-system-outofmemoryexception-w%3Frq%3D1%3C%2FA%3E%3C%2FLINGO-BODY%3E
New Contributor

Hi All,
I am comparing two csv files based on USERNAME column of both files, kindly see my script below,

 

Compare-Object $SourceFile $TargetFile -Property "USERNAME" -PassThru | Where-Object {$_.SideIndicator -eq "=>"

this is working perfectly fine from the past few years.


Till last week both files had around 70,000 records in each, so this comparison took 2 hours to complete,
Recently these file(source, target) records has grown like 8,00,000, now this comparison is taking around 48 hours and some times i am getting  below error.

 

+ CategoryInfo : NotSpecified: (:) [Where-Object], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.WhereObjectCommand

 

 

Kindly suggest is there any other way we can do this comparison, so that we can mitigate this exception and too much of time ??

Response would be appreciated.



1 Reply

I saw in old forum something related to increase the memory,

https://social.technet.microsoft.com/Forums/en-US/ff516976-78a4-49b0-97e8-f443702eaf50/powershell-sc...

execute the command:
Get-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB
by default it is 2147483647

check the link bellow, it have a solution to increase it.
https://stackoverflow.com/questions/28503765/powershell-gives-get-content-exception-of-type-system-o...