BizTalk process Host is not releasing memory until Hosts are manually restarted

%3CLINGO-SUB%20id%3D%22lingo-sub-1564278%22%20slang%3D%22en-US%22%3EBizTalk%20process%20Host%20is%20not%20releasing%20memory%20until%20Hosts%20are%20manually%20restarted%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1564278%22%20slang%3D%22en-US%22%3E%3CP%3EThe%20BizTalk%20process%20Host%20is%20not%20releasing%20memory%20until%20Hosts%20are%20manually%20restarted%20(Ideally%20it%20should%20release%20the%20memory%20that%20is%20not%20in%20use)%3C%2FP%3E%3CUL%3E%3CUL%3E%3CLI%3EWe%20tried%20terminating%20all%20running%2Fsuspended%20instances%2C%20still%20memory%20occupied%20by%20Host%20was%20stagnant.%3C%2FLI%3E%3CLI%3EWe%20tried%20processing%20small%20feed%20files%2C%20however%20still%20the%20memory%20was%20not%20changed.%3C%2FLI%3E%3CLI%3EMemory%20keeps%20increasing%20during%20high%20load%20and%20is%20not%20getting%20freed%20up.%3C%2FLI%3E%3C%2FUL%3E%3C%2FUL%3E%3CP%3EThe%20memory%20slowly%20piles%20up%20and%20each%20of%20the%20service%20calls%20(WCF%20Web%20Service%20calls)%20starts%20getting%20delayed%2C%20taking%20around%2010%20mins%20to%20proceed%20(even%20when%20we%20clearly%20see%20on%20WCF%20Web%20Service%20Servers%20that%20response%20from%20service%20was%20sent%20on%20time%20within%20a%20minute)%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EEverything%20goes%20back%20to%20normal%20and%20speeds%20up%20as%20soon%20as%20we%20manually%20restart%20process%20host%20instance%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1564327%22%20slang%3D%22en-US%22%3ERe%3A%20BizTalk%20process%20Host%20is%20not%20releasing%20memory%20until%20Hosts%20are%20manually%20restarted%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1564327%22%20slang%3D%22en-US%22%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20have%20migrated%20Code%20from%20BizTalk%20Server%202010%20to%202016%20hosted%20on%20cloud%20and%20facing%20weird%20behavior.%3C%2FP%3E%3CP%3EOn%20processing%20message%20of%3CSPAN%3E%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E16%20MB%20of%20flat%20file%20simultaneously(2MB%20each)%2C%20the%20memory%20consumption%20on%20task%20bar%20is%20around%202.5%20GB.%3C%2FP%3E%3CP%3ERegarding%20Orchestration%3A%3C%2FP%3E%3CUL%3E%3CLI%3EProcessing%20record%20on%20batch.%3C%2FLI%3E%3CLI%3EDependency%20on%20other%20orchestration%3C%2FLI%3E%3CLI%3EDependency%20on%20external%20web%20service%3C%2FLI%3E%3C%2FUL%3E%3CP%3EI%20don%E2%80%99t%20think%20the%20issue%20with%20the%20orchestration%20as%20the%20code%20was%20running%20good%20on%20BT2010.%3C%2FP%3E%3CP%3EProblem%3A%3C%2FP%3E%3CP%3EMemory%20Reading%20on%20Task%20Manager%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eagainst%20the%20host%20instance%20after%20terminating%20the%20all%20messages%20is%20not%20changing.%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3Ewhere%20as%20Perfmon%26nbsp%3B%20showing%20flat%20zero%20when%20there%20were%20no%20messages%20processing%3C%2FSPAN%3E%3CSPAN%3E.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3EAlso%20after%20couple%20of%20mins%20when%20we%20tried%20to%20process%20a%20small%20file..%3CSPAN%3E%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3EReading%20came%20down%20to%201%20or%203%20MB.%20But%20again%20it%20went%20to%20constant%20reading.%3C%2FP%3E%3CP%3EInitially%20we%20thought%20it%20could%20be%20memory%20lead%2C%20but%20doesn%E2%80%99t%20appear%20to%20be.%20There%20is%20no%20unmanaged%20code%2C%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Ebut%20lot%20of%20file%20write%20operation%20using%20RollingFlatFileTraceListenerData%20Class%3C%2FP%3E%3CP%3EAdditionally%20%E2%80%93%20Few%20observations%3A%3C%2FP%3E%3COL%3E%3CLI%3E%3CSTRONG%3E%3CSPAN%3EFile%20Server%20where%20log%20is%20being%20written%3C%2FSPAN%3E%3C%2FSTRONG%3E%3CSPAN%3E%26nbsp%3B%3A%20%25%20Disk%20Write%20time%26nbsp%3B%20was%20way%20above%2020%25%20continuously%20reaching%20100%25%20frequently.%26nbsp%3B%20Expect%20value%20is%20less%20than%20Less%20than%2015-20%25.%3C%2FSPAN%3E%3C%2FLI%3E%3CLI%3E%3CSTRONG%3E%3CSPAN%3EMemory%20Utilization%3C%2FSPAN%3E%3C%2FSTRONG%3E%3CSPAN%3E%26nbsp%3B%3CSTRONG%3EProcessing%20Host%20Instance%26nbsp%3B%3C%2FSTRONG%3E%3A%3C%2FSPAN%3E%3C%2FLI%3E%3C%2FOL%3E%3CP%3E%3CSPAN%3EDiscrepancy%3A%26nbsp%3BTask%20bar%20showing%202.5%20GB%20where%20as%20Perfmon%26nbsp%3B%20showing%20flat%20zero%20when%20there%20were%20no%20messages%20processing.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EHigh%20Utilization%3A%20%26nbsp%3BUnder%20huge%20load%2016%20MB%20%26nbsp%3B%3CSTRONG%3EFlat%20File%3C%2FSTRONG%3E%26nbsp%3B(2%20MB%20file%20each)%26nbsp%3B%20memory%20utilization%20was%202.5%20GB%20%2C%20after%20terminating%20messages%20memory%20utilization%20stuck-%20up%20%26nbsp%3Bat%202.5%20GB.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3COL%3E%3CLI%3E%3CSTRONG%3E%3CSPAN%3EDegree%20of%20parallelism%20%3A%26nbsp%3B%3C%2FSPAN%3E%3C%2FSTRONG%3E%3CSPAN%3ESQL%20DB%20current%20value%20is%20Set%20as%20%E2%80%9C0%E2%80%9D%3CSTRONG%3E%26nbsp%3B%2C%26nbsp%3B%3C%2FSTRONG%3E%26nbsp%3Bfor%20BizTalk%20it%20should%20be%20set%20as%20%E2%80%9C1%E2%80%9D.%3C%2FSPAN%3E%3C%2FLI%3E%3C%2FOL%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F748553%22%20target%3D%22_blank%22%3E%40Akash_Chubb%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
New Contributor

The BizTalk process Host is not releasing memory until Hosts are manually restarted (Ideally it should release the memory that is not in use)

    • We tried terminating all running/suspended instances, still memory occupied by Host was stagnant.
    • We tried processing small feed files, however still the memory was not changed.
    • Memory keeps increasing during high load and is not getting freed up.

The memory slowly piles up and each of the service calls (WCF Web Service calls) starts getting delayed, taking around 10 mins to proceed (even when we clearly see on WCF Web Service Servers that response from service was sent on time within a minute)

 

Everything goes back to normal and speeds up as soon as we manually restart process host instance

1 Reply
Highlighted

 

We have migrated Code from BizTalk Server 2010 to 2016 hosted on cloud and facing weird behavior.

On processing message of  16 MB of flat file simultaneously(2MB each), the memory consumption on task bar is around 2.5 GB.

Regarding Orchestration:

  • Processing record on batch.
  • Dependency on other orchestration
  • Dependency on external web service

I don’t think the issue with the orchestration as the code was running good on BT2010.

Problem:

Memory Reading on Task Manager  against the host instance after terminating the all messages is not changing. where as Perfmon  showing flat zero when there were no messages processing.

Also after couple of mins when we tried to process a small file..  Reading came down to 1 or 3 MB. But again it went to constant reading.

Initially we thought it could be memory lead, but doesn’t appear to be. There is no unmanaged code,   but lot of file write operation using RollingFlatFileTraceListenerData Class

Additionally – Few observations:

  1. File Server where log is being written : % Disk Write time  was way above 20% continuously reaching 100% frequently.  Expect value is less than Less than 15-20%.
  2. Memory Utilization Processing Host Instance :

Discrepancy: Task bar showing 2.5 GB where as Perfmon  showing flat zero when there were no messages processing.

High Utilization:  Under huge load 16 MB  Flat File (2 MB file each)  memory utilization was 2.5 GB , after terminating messages memory utilization stuck- up  at 2.5 GB.

 

  1. Degree of parallelism : SQL DB current value is Set as “0” ,  for BizTalk it should be set as “1”.

@Akash_Chubb