Lookin At Some AD Dumpage

[Alex Simons (AZURE)]

First published on CloudBlogs on Dec, 14 2006

Understanding how things work is as much of a tool as anything included in the Adminpack or Support Tools.    If you know how things work, you know what to expect in normal behavior, and what questions to ask when the behavior your seeing is abnormal.

For example… J

We had an customer who was contacting us because they had something pretty unusual they were seeing in their Active Directory.  What they saw was a partially GUIDed , in the manner you see objects that have been tombstoned, entry appearing in the list of members of a security group.  This entry was recognizable as a particular user—let’s call him Bob—that had been removed many months prior.  There was a dim recollection that this user was one from another forest who had been added to this security group for awhile.

It looked like this:

CN=Clown Big BobADEL:6ca094b1-934d-407c-bed8-65b8e2949da1,CN=Deleted Objects,DC=silliness,DC=org

An even more unusual part of viewing this entry in the properties of a group in DSA.MSC was that the entry could not be removed from the list.  An error would be given when you select the members tab of the Administrators group “No superior reference has been configured for the directory service. The directory service is therefore unable to issue referrals to the objects outside the forest.”

If you clicked OK and then deleted the entry it would not give an error, but the entry (really a back link) would not delete.

Quite a puzzler.  We thought it could be database corruption, so we used NTDSUTIL to check the integrity of the database for the DCs but that didn’t help.

Before we thought things through we thought that maybe REPADMIN /REHOST would do the trick.  When you think about it, though, the user is not one of that forest, so he won’t be in the global catalog at all (those only host partial attribute sets of naming contexts in their own forest).

Finally we thought we should look at this object in the database itself.  To do that we used LDP.EXE to send a DUMPDATABASE control to the AD.  This exports the database to a .DMP file, which is viewable in Wordpad.exe.  Obviously, you have to be an admin to do that, so no worries there.

Here’s an article on how to do a dumpdatabase by the way:

How to Use the Online Dbdump Feature in Ldp.exe

http://support.microsoft.com/kb/315098

Why did we think this would be a good thing to do? Well, there are different types of objects in AD, and different properties of them, some of which are only viewable in this manner.  Even better is a method of tracing an object by a unique identifier (this will ring bells with those of you out there familiar with Exchange or other Jet databases) called a distinguished name tag, or DNT.   Each object also has a parent distinguished name tag, or PDNT.  That’s the child objectàparent container relationship, such as when you have a user object in an OU.

But I digress.

What we found when we looked at this object was how it looks here, with its parent containers:

DNT     PDNT  Ver        IsObject

10013  5224   2      -      false 2005-07-28 14:00:47 -     11     BigTop      BigTop             -      -      255ce266-8bbb-45b0-a5c6-109a44aead7f -

-

10014  10013  4      -      false 2005-07-28 14:00:47 -     11     Circus Clowns OU    Circus Clowns OU    -      -      38123a11-4b10-4b78-a11c-51c0161e58f8 -

-

10015  10014  1      -      false 2005-07-28 14:00:47 -     11     袈઎˜ ”“ç•·Ù઎袈઎˜ •·Ù઎袈à&o rdf;ŽÂ˜ -Ȇ ” àªŽæ‡ àªŽ ”੼  €í-í¥â£ªè‰‹î-€çƒ¿æ”˜Â¡ ’ˆà©¼î’ˆCNF:27fd187b-086f-4a18-b3f5-1525e9b06637 袈઎˜ ”“ç•·Ù઎袈઎˜ •·Ù઎袈઎˜ -Ȇ ” àªŽæ‡ àªŽ ”੼  €í-í¥â£ªè‰‹î-€çƒ¿æ”˜Â¡ ’ˆà©¼î’ˆCNF:27fd187b-086f-4a18-b3f5-1525e9b06637 -      -      27fd187b-086f-4a18-b3f5-1525e9b06637 -

-

10016  10015  1      -      false 2005-07-28 14:00:47 -     11     袈઎´ ”“ç•·Ù઎袈઎´ •·Ù઎袈઎´ -Ȇ  ઎ ”੼  €í-í¥â£ªè‰‹î-€çƒ¿æ”˜Â¡ ’ˆà©¼î’ˆ

CNF:f2644711-3177-4b05-afad-c17b0b835b0a 袈઎´ ”“ç•·Ù઎袈઎´ •·Ù઎袈઎´ -Ȇ  ઎ ”੼  €í-í¥â£ªè‰‹î-€çƒ¿æ”˜Â¡ ’ˆà©¼î’ˆ

CNF:f2644711-3177-4b05-afad-c17b0b835b0a -      -      f2644711-3177-4b05-afad-c17b0b835b0a -

-

10017  10016  2      -      false 2005-07-28 14:00:47 -     3      bigbob       bigbob            -      -      51d1b8a1-080f-4b11-93aa-6ea1d1dd36de -

-

I placed a few of the field names as hints.  When you look at the above there are several things which jump out at you.

-Two of the OUs which BigBob lived in are corrupt.

-The “isobject” value says False.

Why are these things important to this drawn out story?  Well, all objects in a domain and forest will show True as objects—even if they are deleted objects (tombstoned).  But phantom objects will show False.  Add to that the fact that two of the parent OUs were showing messed up and we have an interesting hypothesis.

What if BigBob from the BigTop domain (in a different forest) was added to Silliness.org security group a long time ago.   When that’s done the AD creates a phantom object and structure there to have some data about this principal onhand.  Then something bad happens in the database and sometime around there BigBob is deleted in his original forest.

Here’s a reference on scary phantoms:

248047 Phantoms, Tombstones and the Infrastructure Master

http://support.microsoft.com/?id=248047

We would expect, in a working scenario, for BigBob and his parent OU phantoms to be garbage collected after the tombstonelifetime.  But in his case, this doesn’t happen because the garbage collector encounters difficulty when it sees these munged parent phantoms which BigBob resides in.

In case you’re wondering we tried manually forcing garbage collection and didn’t see what we hoped for.  Here’s the step to do that:

1.>In Ldp.exe, when you click Browse on the Modify menu, leave the  Distinguished

name box empty.

2.>In the Edit Entry Attribute box, type  "DoGarbageCollection" (without the

quotation marks),

3.>In the Values box, type  "1" (with out the quotation marks).

4.>Set the Operation value set to Add and click the Enter button, and then click

Run.

But all we saw when we did so was this event and no removal of the offending back link and phantom

Event Type:       Information

Event Source:    NTDS General

Event Category: Garbage Collection

Event ID:           1132

Date:                7/28/2005

Time:                11:00:47 AM

User:                NT AUTHORITYANONYMOUS LOGON

Computer:         ClownDC1

Description:

Internal event: The Directory Service removed the expired, deleted object

CN=Clown Big BobADEL:6ca094b1-934d-407c-bed8-65b8e2949da1,CN=Deleted Objects,DC=silliness,DC=org

from the database.

The above was patently untrue.  Without drawing this out too long ultimately we (it was not my idea though I’d love to take credit for it) used LDP.EXE to delete the phantom using it’s objectGUID, and then our back link from the nether regions went away for good.

Here’s how to do that:

1) Open LDP .EXE.

2) Connect to the directory and BIND using administrative credentials.

3) Pull down the Modify menu and select Delete .

4) In the DN: area type in the objectGUID in the format of: "<GUID=objectGUID>"

(without the quotes).

For example:

<GUID=41019973-cc91-498b-8ffd-77152738b9a9>

The returned message in the LDP right hand pane should be similar to this if

succesful:

ldap_delete_s(ld, "<GUID=41019973-cc91-498b-8ffd-77152738b9a9>");

Deleted "<GUID=41019973-cc91-498b-8ffd-77152738b9a9>"

Am I saying go out and what some objects or phantoms using LDP as soon as possible?  No.  But we’ve gone over some less common stuff in a real world scenario.  This knowledge may save you some grief down the road.  My little gift to you, hopefully you never need it but it’s there if you do.

Happy Holidays everyone.  I hope your year ends on a wonderful note.