From crush to product documentation: The story of Squeaky Lobster
When customers first hear about being able to enable extra JET Blue or ESE Database performance counters via adding a "Squeaky Lobster" registry value, they often think it must be a joke or ask you to repeat it. And invariable the question comes up ... _why_ "Squeaky Lobster"? Various lore or conjecture has surfaced around it ... that it was a developer's child's toy or that is just thought up after lunch one day. Not exactly ... and retrieving the information was a little tricky as the initial checkin was 2 years before I started ... The story starts with a certain developer, let's call him, "Andrew Goodsell", for the time being, but I call (or called) him, "boss", and I can assure you those explanations are not quite right. If you were to get incriminating information about your boss, you might wonder what is the proper time to blog about it? Right after you stop working for him I think is the right answer to that ... and as of today I have transferred to the Exchange organization, while "Andrew" has not. I am as they say, unsupervised. ;-) Andrew, is a very sharp man who prides himself on his professional decorum, and a bit of a perfectionist to boot. In fact his professionalism is what makes this singular lapse in judgment so amusing and fairly uncharacteristic of any decision he would make (or have let me make ;-) today. Andrew has been involved in a fair share of the database performance work in ESE and as anyone decent at perf knows, one of most critical steps is having a way of accurately measuring performance, base-lining performance, at several levels in a software stack. Exchange 5.5 added ESE Database performance counters to facilitate this work. Most of these perf counters were intended for only internal ESE and Exchange developer usage (as you can probably deduce from one of the code snipits below, and looking at what these counters measure... things like B+ Tree Inserts/sec for example). Enter cute girl ... So that autumn while Andrew was working on adding perf counters and working on Exchange 5.5's performance, a cute girl was managing computers in the Exchange Performance lab. Andrew was working on performance and adding performance counters while this girl worked with the powerful computers in the performance lab. It isn't a wonder how they ran into each other. She was cute, it is no wonder Andrew got a crush. Some of the simplest ESE performance counters would also be helpful for advanced administrators in debugging Exchange server issues (Cache Size, File Operations/sec, etc). Andrew mentioned he thought if competing companies had access to some of the more detailed perf counters, they might be able to reverse engineer our implementation details and steal intellectual capital (now that's the professional Andrew I know, trying to protect our intellectual capital). After he says this, he rolls his eyes and says something to the effect of "I don't know what I was thinking!" (this is because the implementation details are more complex than what the counters show, but at the time this made sense). So there was a need to split the ESE Database perf counters into two kinds. ... internal (dev only) and external (admins). The internal ones would have to be hidden. Now, fairly smitten with this girl, ANYTHING she did was fascinating (to Andrew). Oh come on, we've all been there. And one such thing she did was participate in a Giving Campaign, basically an event where we raise money for charities and non-profits. In order to encourage people to give money, various prizes would be donated and a raffle sort of mechanism to win the prizes would be done. The girl of interest donated a toy Squeaky Lobster to the charity raffle. Andrew thought that was the funniest and most random thing to contribute as a prize for a giving campaign ever. Who would want a Squeaky Lobster? Most random prize in giving campaign I guess equals most uneasy to guess registry value ever. Anyway, a name was chosen that was about as random as you could get for 1997 (as there weren't very many words available in 1997, I think the Seattle was recovering from that awful grunge thing). Knowing the crowd Andrew hung out with back then, it would not surprise me if they decided it at or on the way back from lunch as they used to work 12 hour days and take extra long lunches. An excerpt of Andrew's fateful check in from SLM logs (note, I've deleted the developer actual email aliases), relevant information highlighted: #F eseperf.cxx v1 #K text #O in #P 1.00 #T Fri Sep 26 11:15:00 1997 #A <dev_alias_1_deleted>3 #C 47424, 41913 finalize perf ctrs #I 2 #D 1113 27a28,32 > #pragma const_seg( ".text" ) > const char szDisplayDevOnly[] = "Squeaky Lobster"; #pragma const_seg() As an aside, note the ".text" segment notation, I had forgotten about that, now that's old school. Man, 8 years is a LONG time in the computer industry. And so a little over 4 months later, the first Squeaky Lobster Enabled product shipped on Feb 3, 1998, with ESE97 in Exchange 5.5. In fact this is when the database engine under Exchange was renamed from JET Blue to ESE (to avoid confusion with JET Red, which has only vestigial relations to JET Blue / ESE). ESE97 shipped in 1998, just to confuse everyone. The above picture is of the original Exchange 5.5 packaging with the original Squeaky Lobster. But at this point in our story, the counters are secret so it's not a big deal. But there is no way to keep any sort of more extensive analysis mechanism a secret for long ... eventually someone will need the information. And eventually an Exchange performance case came along sometime in 1998 (according to the earliest record I can find in the PSS DB) that required the extra analysis these performance counters offered. The PSS engineer told a customer how to enable the counters so they could analyze the customer's issue. I mean, it comes down to the customer, you do what is necessary and possible to support them. Then another case came in, and another, and eventually someone from PSS thought to publish a KB on 4/17/2000, KB 259895, what counters are enabled by squeaky lobster, _official_ Microsoft documentation admitting the existence of Squeaky Lobster!!! Well, that didn't last long (about 6 months). In about 2000 someone public (Brian Sheaffer / Paul Thurrott) noticed and of course such a thing was far too silly for the serious professionals that control the web site of most big corporations. ;-) About as silly as Squeaky Lobster for a prize in a charity benefit! I mean we're a professional organization. Now there are about 300 PSS related investigations, service tickets and KBs referencing this phrase (including the above one). There is a Squeaky Lobster in Andrew's office as well, but do not be fooled, it is not the original. That Squeaky Lobster was given to him by a PSS engineer who thought it was funny, or maybe to thank him for being repeatedly asked by customers, "What was that? Can you spell it?". A mere 4 and a half years after the checkin, someone wised up ... and for Windows 2003 RTM, and Exchange 2003 SP1, code was checked in to try "Show Advanced Counters" first, and if that fails try "Squeaky Lobster". The comment in the code is: > // deprecated name (yes, we are ending the insanity) > err = DwPerfUtilRegQueryValueEx(hkeyPerf,(char*)"Squeaky Lobster",&Type,&lpbData); Note: we moved away from that archaic .text seg stuff, the compilers are now smart enough. And finally to come full circle, once this happened the name of the registry value appropriate for product documentation on our web site. Note that page specifies Exchange 2003 SP1. If you replace "Show Advanced Counters" with "Squeaky Lobster", the instructions should work for Exchange 5.5 through Exchange 2003 RTM as well as current products, though the registry key you use for step 2, varies. For Ex 5.5 it is "ESE97" instead of "ESE". In Ex2k you have to use "ESE98" before SP2, and finally for SP2 and later, just "ESE". The same process works for Windows, but use "ESENT" instead of "ESE", oh and you have to enable the performance counters for Windows first. I promised an Exchange MVP, Michael B. Smith, about a year ago and I apologize for the delay, some would even mock me for taking so long with this single post, but at that time, the fact that there was a Ms. Squeaky Lobster unknowingly influencing Microsoft product development was not known. You got to wait until you get all the details, Eric. It wasn't till Andrew, accidentally spilled the information at a dinner. After that I blurted out, "I'm SOOO BLOGGING THAT!", Andrew had an immediately look of regret on his face, and now he won't really talk about it anymore. ;-) So that is about all the Squeaky Lobster trivia I could collect. The next time you have to really dig into Exchange (or Active Directory) database performance, just remember how responsible professional ESE software engineers are adding easily discoverable (based upon the activities of their current crush at the time) mechanisms for diagnosing your top issues. So with that in mind, the next time you go to buy a software product, remember to check the box to see if this is one of the many Squeaky Lobster Enabled products. Oh just checked an Exchange 2003 box, not actually listed on the box, hmmm, I guess serious professionals are in control of the packaging too. ;-) So Sept 26th of this year is Squeaky Lobster Day, the 9 year anniversary of the checkin of the Squeaky Lobster registry value, be sure to Squeaky Lobster a server at 11:15, and then go to an extra long lunch, and make a silly decision you'll regret and feel embarrassed about 9 years down the road ... sorry, Andrew, you can't stop the insanity ... Cheers, Brett Shirley ESE (aka JET Blue) DeveloperOffice 365 Message Attribution
When a message arrives at Office 365, one of the first things we need to do is figure out which organization it belongs to. At first, this sounds simple – just look at the recipient, right? Well, it is more complicated than that, because of Hybrid and complex routing scenarios.The SDE career path at Microsoft
In Microsoft, there are a number of standard job titles. The most important one is SDE – Software Development Engineer. The rest can be considered the supporting cast (one requirement of being an SDE is to be so supremely confident that such a statement as the previous one can be made without actually laughing out loud while doing so). The life of an SDE at Microsoft is as close to heaven as a computer geek can get. Chocolate donuts for breakfast, a quick bite in the cafeteria for lunch, any fast food at all for dinner, and before, in between and after is glorious coding. Of course, other things intervene occasionally – like annoying testers who keep insisting the code actually works, or program managers who insist the code actually be useful to someone (honestly, I don’t know where they find these people), or bosses who insist on “talking”, sometimes even in the form of meetings. These are unfortunately necessary evils and an SDE will tolerate them and, on occasion, even indulge them from time to time. There is a cloud on the horizon in many SDE’s careers however, and it’s called management. At Microsoft, there aren’t a whole lot of real managers, they are mostly technical people who at one time or another did real work and then – probably due to chemical imbalances in the brain brought on by a diet of donuts, cafeteria and fast food – suddenly “stepped up” to the management plate. It’s pretty insidious. One day you’re a happy go lucky developer enjoying work to its legal limit and with masses of extremely marketable skills, and then your boss suggests that maybe you should take on a report or two. Just junior people, you understand, to see how you like it. Zap. Six months later you have a team of 15 and you can’t write a line of code to save your life anymore (but you can create a mean Powerpoint presentation). Microsoft does have an alternative career path for SDE’s (and the other, lesser, disciplines) which enables an IC (individual contributor) to remain an IC but continue to progress in that role. While the ultimate destination of a manager is a VP, the equivalent IC role for an SDE is a DE (Distinguished Engineer). DE is quite a bit harder to attain then VP, actually. An intermediate IC role is usually some kind of architect. I know some people who have followed this path successfully, but it is a bit of a harder road to hoe. In my case, I’ve taken the Bilbo Baggins approach (“There and Back Again”). Prior to 1998, I had spent 13 years as an IC (not all at Microsoft, although I wish I had joined the company in ’85). I then took on two reports. Then six. Then, after Exchange 2000 shipped, I took on a team of fifteen. Then suddenly I had seventy developers working for me. I even attained the exalted level of PUM (Product Unit Manager). Those were the dark times. I forgot how to code. I forgot how to debug. I pretty much forgot how to think. I finally snapped out of it and ever since I’ve been striving once more for IC status. I currently am down to a team of five, but I’m not likely to get much lower than that any time soon. No matter. I’m coding again and once more feel like a contributing member of the team rather than a roadblock on the way to success. And those chocolate donuts are yummy. - Jon AvnerLet's talk about Server Extension Objects (SEO)
I’ve worked for Microsoft for about 8 years, all in the Exchange group, and I really enjoy it. The people who work here are bright, self-motivated, and we have the resources we need to get our jobs done. It feels great to ship world-class products that our customers love. However, not everything is perfect, it never is. As my father-in-law once said, “It’s not all grapes and gravy.” For example, the engineering group must work well together with the user education group to produce documentation that lets customers easily get the most out of the software we’ve produced. The single most frustrating episode I’ve endured has to do with the documentation for SEO, or the relative lack thereof. It’s not like I’m free of blame here, I worked on SMTP when these Server Extension Objects (SEO) were first implemented many years ago. So without further adieu, let’s talk about what I think should have been documented about SEO. Please read on, if you’d like to learn a little bit more about a soon-to-be obsolete technology (it will remain in some SKUs of Windows SMTP, but will be replaced in E12), and help me get something off my mind at the same time. Let’s first describe SEO a bit. SEO is a COM-based technology that allows customization of the Windows SMTP service. In the SEO model, information flows from the source (the SMTP server) to the sink (the SEO COM object), whenever an interesting event in processing an SMTP connection is reached. After the SMTP server is finished calling all the sinks, it then proceeds with handling the SMTP session, guided by the actions of the sinks. The last piece of the puzzle is the event bindings database, where the ordered collection of sinks to call for each processing event is persisted. One piece of code that customizes the Windows SMTP service is Exchange 2000/2003. Without the customization and extension made possible by SEO, Exchange would not be the compelling mail server that it is. So let’s pick apart the parts of the IIS metabase that contains the SEO information, and we’ll discover an object model along the way. To get us started, let’s use the adsutil.vbs script that comes with an IIS install, to inspect the “eventmanager” part of the metabase. If you run “cscript.exe %systemdrive%\inetpub\adminscripts\adsutil.vbs /enum_all eventmanager” you’ll see a big bunch of unintelligible output. Let’s dissect just a fraction of it: [/eventmanager/SourceTypes/{FB65C4DC-E468-11D1-AA67-00C04FA345F6}/EventTypes/{F6628C8D-0D5E-11d2-AA68-00C04FA35B82}] What does this mean? The first GUID can be seen in the public SDK file SmtpGuid.h, where it is given the name g_szGuidSmtpSourceType. The second GUID, located in the same file, corresponds with g_szcatidSmtpOnInboundCommand. To understand the rest of the string, we need to take a quick peek at the SEO object model. The root object here is IEventManager, which has a programmatic ID of “Event.Manager”, and is reflected in the “eventmanager” string. This object is the one you create and use to get at all the rest of the SEO entities. The IEventManager object contains an IEventSourceTypes object (please notice the “s” at the end here, we’ll see other similarly-named collection classes later), and the IEventSourceTypes object just contains a collection of IEventSourceType objects. Each IEventSourceType object contains an IEventTypes object, which, of course, just contains a set of IEventType objects. So, if we re-parse the string above, we can read it as telling us that the IEventManager contains an IEventSourceType with the ID {FB65C4DC-E468-11D1-AA67-00C04FA345F6}, and that this source knows how to generate {F6628C8D-0D5E-11d2-AA68-00C04FA35B82}, or g_szcatidSmtpOnInboundCommand, events. The rest of the info in the eventmanager part of the IIS metabase serves the same purpose, to define the two sources of SEO events, the SMTP and NNTP server, and to list the types of events that each source may generate. The other SEO information, which describes which consumers are to be called for which SEO event types, and in what order, is located in other parts of the metabase. For example, the first SMTP virtual server has its SEO information listed under smtpsvc/1/eventmanager. So if you type “cscript.exe %systemdrive%\inetpub\adminscripts\adsutil.vbs enum_all smtpsvc/1/eventmanager”, you’ll see info like [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/DisplayName] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SinkClass] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SourceProperties] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SourceProperties/priority] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SourceProperties/rule] What could this mean? The first portion of the string -- /smtpsvc/1/eventmanager/ -- just lets us know we are looking at the IEventSource for the first SMTP virtual server instance. Note that there is a difference between an IEventSourceType and an IEventSource. The IEventSourceType is an abstract, like “SMTP server”, while the IEventSource is a literal instance, like “the first SMTP virtual server instance on this machine”. The next part of the string -- EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82} -- tells us we are looking at the g_szcatidSmtpOnInboundCommand event type. To understand the rest of the string, we need to hearken back to the SEO object model, which defines the IEventSource as containing an IEventBindingManager object. The IEventBindingManager object is used to read, write, create, and destroy IEventBinding objects. There is one IEventBinding object contained in the metabase for each point of customization between the SMTP server and external code. The event binding is how this customization is persisted when IIS is recycled or the system restarted. Finally, the IEventBinding object contains an IEventPropertyBag object, which allows the event binding to contain additional information about itself. Now that we know a little more, we can look at the second fragment of output and see that it’s referring to information regarding an SEO binding. The binding is attached to the first virtual SMTP server instance, and it’s for the SmtpOnInboundCommand event type. The binding has a few properties, too. But why can’t we see more? It turns out that the adsutil.vbs script makes use of the ADSI schema for the metabase to understand and display the data contained in the metabase. The SEO information is not contained in the ADSI schema, and so doesn’t display much interesting information at all. If we run the “displaybindings” script (see near the end of this posting for the full vbs script), we’ll see some more descriptive detail displayed: \Source Types\Sources\Bindings\{F6628C8D-0D5E-11d2-AA68-00C04FA35B82} Display Name = SMTP Protocol OnInboundCommand ID = {2EFF593B-5451-4D20-891C-1D32BFE95A1C} priority = 9000 rule = X-EXPS SinkClass = Exchange.PS-EXPS DisplayName = Exchange SMTP Protocol Security EXPS Sink The SEO information looks a little nicer. We have a bit more context given to the GUIDs, and we can see all of the properties and their values. All of the other information under /smtpsvc/1/eventmanager/ is there for the same purpose, to describe event bindings. Now we’ve looked at the SEO object model a bit, as well as pulled apart the SEO information in the metabase. And we’ve mentioned that SEO is not part of our future product shipments. I would describe ways you can use SEO to improve your system, but there really aren’t any. SEO information should just be left alone, there is nothing to be gained from tweaking it. So just as we’ve started, we’re already done. If you have more questions about SEO, or the customization that can be gained from working with it, take a look at MSDN, and let us know if you have any more questions. It was interesting, from my side, to get this technology out into the world, and it would be interesting to hear from you how you’re using it. Here is the content of the displaybindings vbs script: Option Explicit ' ' \nt\private\inet\iis\staxinc\export\smtpdisp.idl ' Const catidPtSourceType = "{FB65C4DC-E468-11D1-AA67-00C04FA345F6}" Const catidPtSmtpSvc = "{1B3C0666-E470-11D1-AA67-80C04FA345F6}" REM ======================================================================================================== REM Function: REM e -- keystroke saver for WScript.Echo REM REM Arguments: REM strMessage -- message to echo to screen REM REM ======================================================================================================== Sub e (strMessage) WScript.Echo strMessage End Sub REM ======================================================================================================== REM Function: REM WriteCollection -- echoes a property bag REM REM Arguments: REM objCollection -- object to write (must implement _NewEnum) REM REM ======================================================================================================== Sub WriteCollection(objCollection) Dim strProperty For Each strProperty In objCollection If IsObject(objCollection(strProperty)) Then WriteCollection objCollection(strProperty) Else e strProperty & " = " & objCollection(strProperty) End If Next End Sub Function GetEventTypeDesc(strEventType) On Error Resume Next Dim strRetVal Dim objComCat Set objComCat = CreateObject("Event.ComCat") strRetVal = objComCat.GetCategoryDescription(strEventType,0) If(0 = Len(strRetVal)) Then strRetVal = "Unknown (" & strEventType & ")" End If Set objComCat = Nothing GetEventTypeDesc = strRetVal End Function Dim objEventManager Dim objEventUtil Dim GUID_SourceGuid Dim strEventType Dim objBindingManager Dim objBindings Dim objBinding Dim strEventTypeDesc ' ' create server event objects ' Set objEventManager = CreateObject("Event.Manager") Set objEventUtil = CreateObject("Event.Util") ' ' make source guid for instance 1 ' GUID_SourceGuid = objEventUtil.GetIndexedGUID(catidPtSmtpSvc,CLng(1)) ' ' get binding manager ' Set objBindingManager = objEventManager.SourceTypes(catidPtSourceType).Sources(GUID_SourceGuid).GetBindingManager ' ' iterate through the event types ' For Each strEventType In objBindingManager strEventTypeDesc = GetEventTypeDesc(strEventType) e "Event Type : " & strEventTypeDesc 'strEventType e "====================================================" Set objBindings = objBindingManager.Bindings(strEventType) For Each objBinding In objBindings WriteCollection objBinding.EventBindingProperties e "" Next e "" Next ' ' cleanup ' Set objEventManager = Nothing Set objEventUtil = Nothing - Ed PrescottPublic Folder Replication Troubleshooting – Part 1: Troubleshooting the Replication of New Changes
Note: this blog post is the first in a Public Folder Troubleshooting series. Please also make sure to check out part 2, part 3 and part 4 of the series. Introduction The purpose of this blog post series is to serve as a guide to troubleshooting public folder replication problems. It will not tell you exactly how to fix every possible replication problem. However, it will show you how to isolate every possible replication problem so that you focus your troubleshooting on the point of failure. Put another way, this post is intended to take you from a problem description like “The content on my old server isn’t replicating to my new server” to a much narrower problem description like “My old server isn’t responding to the status requests from my new server, therefore the new server doesn’t know it’s missing data and isn’t trying to backfill. This means the problem is actually with the old server.” This post will also describe how to identify a few of the most common replication problems. Before I get into the details of troubleshooting, I want to give an overview of my general approach to these issues. The best troubleshooting tool for public folder replication is the application log. In order to isolate a replication problem, you must be able to follow the replication events in the log to see exactly where the process is breaking down. Typically, you should turn up Diagnostics Logging on Replication Incoming and Replication Outgoing to maximum as a starting point for troubleshooting. Each time a store sends or receives a replication message, it will log an event to that effect (assuming logging is turned up). The various kinds of replication messages can be differentiated by the 'Type' shown in the description of the event. I prefer to focus on the Type rather than the event ID for several reasons: - Event IDs change between versions of Exchange. For instance, in Exchange 5.5 an outbound backfill request was a 3014. In Exchange 2000 and 2003 it's a 3016. - Incoming and outgoing event IDs are different for each type. An outgoing hierarchy message is a 3018, while an incoming hierarchy message is 3028. - Status requests and status messages use the same event IDs, even though they are two different types of messages. Thus, you can't distinguish between them from event ID alone. Focusing on the Type is a little easier. You can easily correlate these with Event IDs by examining your application log. There are only 7 types, and you can see if the message is outgoing or incoming by looking at the Category of the event. If you focus on types instead of event IDs, all you need to remember is: Hierarchy - 0x2 Content - 0x4 Backfill Request - 0x8 Backfill Response - 0x80000002 (for hierarchy) or 0x80000004 (for content) Status - 0x10 Status Request - 0x20 Also note that Replication Errors logging is rarely helpful. Even when replication is working normally, most servers will generate lots of replication error events, such as event ID 3093 indicating there was an error reading a property. In most cases the property has no effect on replication and the error can be ignored. I recommend leaving Replication Errors logging at None unless you're looking for something specific, such as the problem described in this previous blog post. Troubleshooting the Replication of New Changes To troubleshoot public folder replication, you must first be familiar with the normal message flow that is expected when replication is working. Based on that knowledge, you can identify the point of failure when a problem arises. Before I discuss how to troubleshoot the replication of new changes, let's describe what the expected behavior is. Hierarchy Replication The replication of new hierarchy changes takes place whenever a folder is created or deleted, or a change is made to the properties of a public folder, such as the replica list, client permissions, description, administrative note, or storage limits. Note that this does not include the email addresses on a mail-enabled folder. The email addresses are stored on the directory object in the Active Directory, so changing them does not result in hierarchy replication. Only changing the properties stored in the public store itself will cause hierarchy replication Every 15 minutes (by default), the store broadcasts any changes that were made to the folder properties in one or more hierarchy replication messages. With logging turned up to Maximum on Replication Outgoing, you'll see an event like this on the server where the hierarchy was modified: Event Type: Information Event Source: MSExchangeIS Public Store Event Category: Replication Outgoing Messages Event ID: 3018 Description: An outgoing replication message was issued. Type: 0x2 Message ID: <91ACCD0758385549A56A10971798985572D5@bilongexch1.bilong.test> Database "First Storage Group\Public Folder Store (BILONGEXCH1)" CN min: 1-72CF, CN max: 1-72D3 RFIs: 1 1) FID: 1-6994, PFID: 1-1, Offset: 28 IPM_SUBTREE\NewFolder Notice the "Type: 0x2" at the beginning of the description, identifying this as a hierarchy replication message. A hierarchy replication message is sent from the originating server directly to all other public stores. There's no concept of topology for the replication of new changes. All changes to the hierarchy are sent directly from the server on which the changes were made to all other servers that have a public store associated with the same hierarchy. Every other server should log an incoming event showing a type 0x2 when they successfully process the incoming replication message. Content Replication Content replication takes place whenever a message is created or deleted, or the properties of a message are changed. The times at which the store broadcasts content changes for a folder can be modified by changing the replication schedule on that folder, but by default the changes will be broadcast every 15 minutes just like the hierarchy. A content replication message is identified by the type 0x4 in the event description. Once again, there is no concept of topology for the broadcast of new changes. When the content of a folder is modified on any given replica, that replica will send a 0x4 message directly to all other replicas of the folder. And again, every receiving server should log an incoming 0x4 event when they successfully process the incoming message. Troubleshooting Steps Those are the two most basic scenarios for replication. When new hierarchy changes or new content changes are not making it from one server to another, troubleshooting is very straightforward. 1. Did the server generate an outbound replication message? So you made a change to a folder or you added content to a folder on particular server, and that content didn't make it to some other server. The first question to answer is did the target server broadcast the changes? When troubleshooting, it's important to keep track of what server you're working against when you make changes. There are several ways to accomplish this. In ESM, you can right-click on the Public Folders node and choose "Connect To" to point to a particular store. For the most part ESM will make any changes on the specified store, but be aware of one exception - Client Permissions. Prior to Exchange 2003 Sp2, when you changed Client Permissions through ESM, ESM would attempt to make the change against a store that houses a replica of the folder, even though this isn't necessary since the permissions are stored as a property of the folder in the hierarchy. With the 2003 Sp2 version of ESM, this was changed so that it now does make the change on the hierarchy you’ve pointed it to. When you're testing hierarchy replication by making changes through ESM, you should avoid using the permissions for testing since it may be hard to predict which store the change will be made against, unless you’re running ESM from 2003 Sp2. If you're using Outlook and you want to verify which replica of a folder you are hitting, you can use MFCMAPI or a similar tool to view the PR_REPLICA_SERVER property of the folder. This will show you the name of the server that Outlook is using to access the content of that folder. If the replication schedule is Always for the folder in question (which is always true for the hierarchy), and you don't see an outbound 0x2 or 0x4 within 15 minutes, then you know something is wrong on the originating server. If the server is not generating any outbound hierarchy or content broadcasts, the replication agent may have failed to start. One of the common scenarios is described in KB272999. The important thing to note here is the 3079 event: Event ID: 3079 Source: MSExchangeIS Public Type: Error Category: Replication Errors Description: Unexpected replication thread error 0x3f0. EcGetReplMsg EcReplStartup FReplAgent This will be logged even with no additional logging turned up when you mount the public store. If the 3079 includes the "EcReplStartup" function, this means the replication agent failed to start and no new changes will be broadcast until the problem is corrected and the store is mounted again. Hierarchy replication is also vulnerable to certain permissions problems when Exchange 5.5 public stores are present in the organization. When an Exchange 2000 or 2003 server sends a hierarchy replication message to an Exchange 5.5 server, it must produce a ptagACLData property (the 5.5-style permissions based on legacyExchangeDN) from the ptagNTSD property (the 2000-style permissions based on SID). This means each SID must be converted to a legacyExchangeDN. This SID to legacyExchangeDN conversion can fail for several reasons. For instance, if a SID resolves to more than one user, an event like this may be generated: Event ID: 9528 Category: General Source: MSExchangeIS Type: Error Description: The SID S-1-5-21-408248388-469072634-37170099-1391 was found on 2 users in the DS, so the store cannot map this SID to a unique user. The users involved are: /DC=com/DC=company/CN=Users/CN=User1 /DC=com/DC=company/CN=Users/CN=User2 Since the SID can't be converted to a legacyExchangeDN, the store will fail to generate an outbound hierarchy broadcast message. 2. Was the message addressed to the server that didn't receive the change? If the originating server generated the outbound message, the next step is to be sure it was addressed to the server that didn't receive the data. The easiest way to verify this is to track the message. In message tracking, you can just track the message ID that was reported in the outbound replication event. In the Message History window, the To: line will be visible. If the store that didn't receive the changes is not listed here, then again the focus should be on the originating server. Does it see that server in the organization? Does that server have email addresses on its public store object? Does the originating server see that store in the replica list on the folder in question? 3. Did the message make it to the destination server? Once you've verified that the message was addressed to the destination server, the next question to answer is - did the message make it that far? You can determine this through message tracking. If message tracking says the message was delivered to the store, but you don't see an incoming replication event acknowledging the message, see the "Common Problems" section in the last post of this series. In next blog post: Troubleshooting the Replication of Existing Data. - Bill LongFindTime, your favorite scheduling add-in just got better!
We heard you and so we re-wrote the entire back-end infrastructure for FindTime, we built a new service within the Office 365 compliance boundary! More specifically, the organizer’s poll data is now stored in their mailbox and will not leave your tenant’s environment.
