2012r2 Direct Access non-paged memory leak

%3CLINGO-SUB%20id%3D%22lingo-sub-1404001%22%20slang%3D%22en-US%22%3E2012r2%20Direct%20Access%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1404001%22%20slang%3D%22en-US%22%3E%3CP%3EWe%20have%20a%20single%20site%202012r2%20Direct%20Access%20server%20running%20as%20a%20hyper-v%20guest%20in%20edge%20configuration.%26nbsp%3B%20The%20physical%20box%20is%20a%202012r2%20Dell%20R710.%26nbsp%3B%20The%20R710%20has%20a%20Qlogic%2FBCM5709C%20NIC%20card%20in%20it%2C%20we%20have%20turned%20off%20VMQ.%26nbsp%3B%20Direct%20Access%20is%20working%20but%20I%20am%20noticing%20a%20non-paged%20memory%20leak%20occurring%20in%20the%20NDnd%20tag%20on%20the%20Direct%20Access%20guest.%26nbsp%3B%20The%20leak%20occurs%20when%20UDP%20RDP%20packets%20traverse%20Direct%20Access.%26nbsp%3B%20The%20RAM%20can%20fill%20up%20in%20a%20day%20and%20the%20box%20blue%20screens%20and%20reboots.%26nbsp%3B%20To%20bypass%20this%20we%20have%20set%20up%20RDP%20traffic%20to%20go%20to%20a%20RDP%20gateway%20instead%20of%20across%20Direct%20Access%20for%20these%20users.%3C%2FP%3E%3CP%3EThe%202012r2%20boxes%20are%20fully%20patched%20and%20the%20clients%20are%20windows%2010%20enterprise%20fully%20patched.%26nbsp%3B%20I%20have%20removed%20all%203rd%20party%20software%20using%20the%20NDnd%20pool%20tag%20and%20a%20%22%3CSPAN%3Efindstr%20%2Fm%20%2Fl%20NDnd%20*.sys%22%20only%20returns%20ndis.sys.%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAny%20ideas%20would%20be%20appreciated.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1404001%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3Edirect%20access%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EHyper-V%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ENetworking%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EWindows%20Server%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1413107%22%20slang%3D%22en-US%22%3ERe%3A%202012r2%20Direct%20Access%20non-paged%20memory%20leak%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1413107%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F673132%22%20target%3D%22_blank%22%3E%40mattsutton1295%3C%2FA%3E%26nbsp%3B-%20this%20doesn't%20help%20you%2C%20but%20I%20have%20the%20exact%20same%20issue.%26nbsp%3B%20I%20have%20a%20set%20of%202012%20R2%20DA%20servers%20that%20are%20load%20balanced%2C%20and%20they%20started%20having%20a%20major%20memory%20leak%20during%20the%20transition%20to%20a%20new%202019%20RDS%20farm%20that%20utilizes%20UDP%20(old%20farm%20did%20not%2C%20and%20I%20had%20this%20DA%20farm%20up%20for%20years%20without%20issue).%26nbsp%3B%20I%20now%20have%20both%20servers%20set%20to%20a%20ridiculous%2032gb%20of%20RAM%20each%20so%20that%20they%20continue%20to%20function%20with%20only%20doing%20a%20weekly%20reboot%20-%20if%20you%20find%20anything%2C%20let%20me%20know!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1425015%22%20slang%3D%22en-US%22%3ERe%3A%202012r2%20Direct%20Access%20non-paged%20memory%20leak%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1425015%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F677516%22%20target%3D%22_blank%22%3E%40stopnik%3C%2FA%3E%26nbsp%3BInteresting.%26nbsp%3B%20Since%20moving%20the%20RDP%20traffic%20away%20from%20Direct%20Access%20the%20leak%20has%20stopped%20for%20us.%26nbsp%3B%20Something%20with%20RDP%20UDP%20packets%20only%2C%20otherwise%20we%20would%20see%20the%20leak%20grow%20with%20DNS%20queries%20and%20other%20UDP%20traffic.%26nbsp%3B%20Must%20be%20something%20internal%20in%20Direct%20Access%202012r2%20that%20is%20handling%20these%20packets%20differently%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E-Matt%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1901145%22%20slang%3D%22en-US%22%3ERe%3A%202012r2%20Direct%20Access%20non-paged%20memory%20leak%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1901145%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F673132%22%20target%3D%22_blank%22%3E%40mattsutton1295%3C%2FA%3E%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F677516%22%20target%3D%22_blank%22%3E%40stopnik%3C%2FA%3E%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHi%20guys%2C%20any%20luck%20with%20resolving%20the%20mentioned%20problem%3F%20It%20seems%20that%20I%20have%20faced%20the%20same%20one%20on%20HP%20physical%20server%20running%20Windows%202012r2%20and%20DA%20role%2C%20non-paged%20RAM%20leak%20on%20%22NDnd%22%20pointing%20to%20the%20%22ndis.sys%22%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1901236%22%20slang%3D%22en-US%22%3ERe%3A%202012r2%20Direct%20Access%20non-paged%20memory%20leak%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1901236%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F873476%22%20target%3D%22_blank%22%3E%40david0n%3C%2FA%3E%26nbsp%3Bunfortunately%20there's%20no%20fix%20as%20far%20as%20I'm%20aware%20of%20with%20Direct%20Access%20running%20on%202012%20R2%2C%20it's%20fixed%20in%202016%20and%202019.%26nbsp%3B%20I%20am%20not%20in%20the%20position%20to%20upgrade%20ours%20right%20now%2C%20so%20I%20have%20my%202%20servers%20on%20an%20automatic%20reboot%20schedule%20at%202am%20once%20a%20week%20(staggered%20as%20I%20have%20an%20HA%20pair)%2C%20this%20has%20managed%20the%20leak%20so%20that%20the%20servers%20don't%20fail.%26nbsp%3B%20It's%20not%20ideal%2C%20but%20it%20works%20for%20now%20until%20I%20can%20move%20all%20of%20our%20clients%20over%20to%20a%20newer%20version.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EGood%20luck!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1901283%22%20slang%3D%22en-US%22%3ERe%3A%202012r2%20Direct%20Access%20non-paged%20memory%20leak%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1901283%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F873476%22%20target%3D%22_blank%22%3E%40david0n%3C%2FA%3E%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F677516%22%20target%3D%22_blank%22%3E%40stopnik%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20ended%20up%20retiring%20our%202012r2%20Direct%20Access%20environment%20and%20moved%20to%202019%20Always%20on%20VPN.%26nbsp%3B%20We%20were%20rebooting%20every%20night%20for%20a%20while.%26nbsp%3B%20We%20finally%20just%20bit%20the%20bullet%20and%20migrated%20over%20to%20AOVPN.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EGood%20Luck!!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1901286%22%20slang%3D%22en-US%22%3ERe%3A%202012r2%20Direct%20Access%20non-paged%20memory%20leak%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1901286%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F673132%22%20target%3D%22_blank%22%3E%40mattsutton1295%3C%2FA%3E-%26nbsp%3Bthat's%20good%20to%20hear!%26nbsp%3B%20Unrelated%20to%20this%20post%20I%20realize%2C%20but%20how%20did%20you%20handle%20transitioning%20over%20your%20clients%3F%26nbsp%3B%20Were%20you%20able%20to%20get%20away%20with%20delivering%20the%20new%20certs%20%26amp%3B%20GPO%20settings%2Fetc%20over%20the%20old%20DA%20connection%20or%20did%20you%20have%20to%20bring%20all%20the%20clients%20back%20into%20the%20corporate%20office%20to%20get%20reconfigured%3F%26nbsp%3B%20That's%20the%20main%20reason%20I%20haven't%20done%20our%20upgrade%20-%20with%20this%20pandemic%2C%20having%20everybody%20bring%20in%20or%20ship%20their%20devices%20back%20to%20our%20office%20isn't%20feasible.%26nbsp%3B%20%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1901364%22%20slang%3D%22en-US%22%3ERe%3A%202012r2%20Direct%20Access%20non-paged%20memory%20leak%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1901364%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F677516%22%20target%3D%22_blank%22%3E%40stopnik%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20had%20employees%20bring%20devices%20back%20to%20the%20closest%20office%20to%20their%20residence%20to%20get%20them%20on%20the%20corporate%20network.%26nbsp%3B%20Not%20ideal%2C%20but%20I%20think%20it%20worked%20out%20that%20they%20scheduled%20the%20time%20vs%20something%20go%20wrong%20and%20then%20making%20them%20come%20in%20because%20their%20device%20can%20no%20longer%20connect.%3C%2FP%3E%3C%2FLINGO-BODY%3E
New Contributor

We have a single site 2012r2 Direct Access server running as a hyper-v guest in edge configuration.  The physical box is a 2012r2 Dell R710.  The R710 has a Qlogic/BCM5709C NIC card in it, we have turned off VMQ.  Direct Access is working but I am noticing a non-paged memory leak occurring in the NDnd tag on the Direct Access guest.  The leak occurs when UDP RDP packets traverse Direct Access.  The RAM can fill up in a day and the box blue screens and reboots.  To bypass this we have set up RDP traffic to go to a RDP gateway instead of across Direct Access for these users.

The 2012r2 boxes are fully patched and the clients are windows 10 enterprise fully patched.  I have removed all 3rd party software using the NDnd pool tag and a "findstr /m /l NDnd *.sys" only returns ndis.sys.  

 

Any ideas would be appreciated.

 

 

 

 

 

7 Replies

@mattsutton1295 - this doesn't help you, but I have the exact same issue.  I have a set of 2012 R2 DA servers that are load balanced, and they started having a major memory leak during the transition to a new 2019 RDS farm that utilizes UDP (old farm did not, and I had this DA farm up for years without issue).  I now have both servers set to a ridiculous 32gb of RAM each so that they continue to function with only doing a weekly reboot - if you find anything, let me know!

@stopnik Interesting.  Since moving the RDP traffic away from Direct Access the leak has stopped for us.  Something with RDP UDP packets only, otherwise we would see the leak grow with DNS queries and other UDP traffic.  Must be something internal in Direct Access 2012r2 that is handling these packets differently?

 

-Matt

@mattsutton1295 @stopnik  

 

Hi guys, any luck with resolving the mentioned problem? It seems that I have faced the same one on HP physical server running Windows 2012r2 and DA role, non-paged RAM leak on "NDnd" pointing to the "ndis.sys"

@david0n unfortunately there's no fix as far as I'm aware of with Direct Access running on 2012 R2, it's fixed in 2016 and 2019.  I am not in the position to upgrade ours right now, so I have my 2 servers on an automatic reboot schedule at 2am once a week (staggered as I have an HA pair), this has managed the leak so that the servers don't fail.  It's not ideal, but it works for now until I can move all of our clients over to a newer version.

 

Good luck!

@david0n @stopnik 

We ended up retiring our 2012r2 Direct Access environment and moved to 2019 Always on VPN.  We were rebooting every night for a while.  We finally just bit the bullet and migrated over to AOVPN.

 

Good Luck!!

@mattsutton1295- that's good to hear!  Unrelated to this post I realize, but how did you handle transitioning over your clients?  Were you able to get away with delivering the new certs & GPO settings/etc over the old DA connection or did you have to bring all the clients back into the corporate office to get reconfigured?  That's the main reason I haven't done our upgrade - with this pandemic, having everybody bring in or ship their devices back to our office isn't feasible.   

@stopnik 

We had employees bring devices back to the closest office to their residence to get them on the corporate network.  Not ideal, but I think it worked out that they scheduled the time vs something go wrong and then making them come in because their device can no longer connect.