BizTalk process Host is not releasing memory until Hosts are manually restarted

[Akash_Chubb]

The BizTalk process Host is not releasing memory until Hosts are manually restarted (Ideally it should release the memory that is not in use)

• We tried terminating all running/suspended instances, still memory occupied by Host was stagnant.
• We tried processing small feed files, however still the memory was not changed.
• Memory keeps increasing during high load and is not getting freed up.

The memory slowly piles up and each of the service calls (WCF Web Service calls) starts getting delayed, taking around 10 mins to proceed (even when we clearly see on WCF Web Service Servers that response from service was sent on time within a minute)

Everything goes back to normal and speeds up as soon as we manually restart process host instance

[#2]

We have migrated Code from BizTalk Server 2010 to 2016 hosted on cloud and facing weird behavior.

On processing message of  16 MB of flat file simultaneously(2MB each), the memory consumption on task bar is around 2.5 GB.

Regarding Orchestration:

• Processing record on batch.
• Dependency on other orchestration
• Dependency on external web service

I don’t think the issue with the orchestration as the code was running good on BT2010.

Problem:

Memory Reading on Task Manager  against the host instance after terminating the all messages is not changing. where as Perfmon  showing flat zero when there were no messages processing.

Also after couple of mins when we tried to process a small file..  Reading came down to 1 or 3 MB. But again it went to constant reading.

Initially we thought it could be memory lead, but doesn’t appear to be. There is no unmanaged code,   but lot of file write operation using RollingFlatFileTraceListenerData Class

Additionally – Few observations:

1. File Server where log is being written : % Disk Write time  was way above 20% continuously reaching 100% frequently.  Expect value is less than Less than 15-20%.
2. Memory Utilization Processing Host Instance :

Discrepancy: Task bar showing 2.5 GB where as Perfmon  showing flat zero when there were no messages processing.

High Utilization:  Under huge load 16 MB  Flat File (2 MB file each)  memory utilization was 2.5 GB , after terminating messages memory utilization stuck- up  at 2.5 GB.

1. Degree of parallelism : SQL DB current value is Set as “0” ,  for BizTalk it should be set as “1”.

@Akash_Chubb 

[#3]

@Akash_Chubb 

Hi @Akash_Chubb Have you been able to resolve the problem? We are experiencing exact same symptoms and frankly have no idea how to approach the situatiion.  

[#4]

@sanoraya Hi

Yes

I found issue in our logging component, It was causing memory leak, I wrote a custom logging component & replaced and used it everywhere.

[#5]

Thank you @Akash_Chubb  for the update. I am dealing with the same issue, got to find the leaking component

[#6]

I have seen a few issues with logging frameworks leaking memory objects, e g config file watchers and timer objects. Since it was related to timers and events being fired, it was also using more and more resources. I could see millions of .NET objects in memory dumps. So it can be good to replace older logging frameworks and have more control over this yourself if you need logging. As a workaround until you replaced the logging, if not leaking too much memory you could e g on daily basis restart the BizTalk host or set recycling period of the IIS application pool if this is in isolated host.