csv file comparison

Occasional Visitor

Hi All,
I am comparing two csv files based on USERNAME column of both files, kindly see my script below,

 

Compare-Object $SourceFile $TargetFile -Property "USERNAME" -PassThru | Where-Object {$_.SideIndicator -eq "=>"

this is working perfectly fine from the past few years.


Till last week both files had around 70,000 records in each, so this comparison took 2 hours to complete,
Recently these file(source, target) records has grown like 8,00,000, now this comparison is taking around 48 hours and some times i am getting  below error.

 

+ CategoryInfo : NotSpecified: (:) [Where-Object], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.WhereObjectCommand

 

 

Kindly suggest is there any other way we can do this comparison, so that we can mitigate this exception and too much of time ??

Response would be appreciated.



1 Reply

I saw in old forum something related to increase the memory,

https://social.technet.microsoft.com/Forums/en-US/ff516976-78a4-49b0-97e8-f443702eaf50/powershell-sc...

execute the command:
Get-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB
by default it is 2147483647

check the link bellow, it have a solution to increase it.
https://stackoverflow.com/questions/28503765/powershell-gives-get-content-exception-of-type-system-o...