Getting changes to Lookups to trigger a crawl

%3CLINGO-SUB%20id%3D%22lingo-sub-634574%22%20slang%3D%22en-US%22%3EGetting%20changes%20to%20Lookups%20to%20trigger%20a%20crawl%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-634574%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%3EWe're%20using%20a%20Lookup%20column%20to%20link%20to%20master%20data.%26nbsp%3B%20When%20the%20master%20data%20is%20updated%2C%20the%20changes%20are%20applied%20to%20the%20content%20via%20the%20Lookup.%26nbsp%3B%20While%20this%20works%20well%20for%20the%20browsing%20and%20filtering%20within%20the%20list%20view%2C%20the%20update%20to%20Lookup%20pull-through%20data%20does%20not%20appear%20to%20trigger%20a%20crawl.%26nbsp%3B%20We%20use%20these%20properties%20for%20search%20purposes%20(refiners)%20and%20need%20them%20to%20be%20updated.%26nbsp%3B%20Is%20there%20a%20way%20to%20trigger%20the%20crawl%20when%20an%20item%20is%20updated%20in%20the%20Lookup%20list%20for%20the%20libraries%20that%20use%20it%3F%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-634574%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EDocument%20Library%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ELists%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3Esearch%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-642516%22%20slang%3D%22en-US%22%3ERe%3A%20Getting%20changes%20to%20Lookups%20to%20trigger%20a%20crawl%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-642516%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F294901%22%20target%3D%22_blank%22%3E%40melkinsco%3C%2FA%3E%26nbsp%3BI'm%20going%20to%20guess%20here%2C%20but%20I%20think%20it%20might%20be%20because%20updating%20the%20Master%20Data%20doesn't%20actually%20update%20the%20records%20that%20lookup%20to%20it%20(because%20that%20is%20related%20by%20ID).%20To%20confirm%2C%20make%20a%20change%20to%20your%20master%20data%20and%20check%20the%20modified%20date%20of%20your%20related%20records...%20I%20do%20not%20think%20the%20modified%20date%20will%20change.%3CBR%20%2F%3E%3CBR%20%2F%3EI'm%20guessing...%20that%20because%20of%20this%2C%20SharePoint%20doesn't%20recognize%20it%20was%20a%20change%20and%20doesn't%20re-crawl%20your%20item%20during%20an%20incremental%20crawl.%20It%20would%20only%20be%20picked%20up%20during%20a%20full%20crawl%20which%20we%20have%20no%20control%20over%20in%26nbsp%3B%20SharePoint%20online.%3CBR%20%2F%3E%3CBR%20%2F%3ETo%20test%2C%20you%20could%20re-index%20the%20list%20and%20see%20if%20SharePoint%20picks%20it%20up%20during%20the%20next%20crawl.%20How%20often%26nbsp%3B%20is%20your%20Master%20Data%20changing%3F%20If%20it's%20not%20often%2C%20scheduling%20a%20re-index%20is%20likely%20fine...%20however%2C%26nbsp%3B%20if%20it's%20changing%20often%2C%20the%20re-index%20will%20put%20a%20heavy%20load%20on%20the%20system%20and%20should%20be%20avoided%2C%20and%20you%20may%20need%20to%20look%20into%20another%20approach%20for%20refining%20on%20master%20data.%3CBR%20%2F%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-644211%22%20slang%3D%22en-US%22%3ERe%3A%20Getting%20changes%20to%20Lookups%20to%20trigger%20a%20crawl%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-644211%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F39836%22%20target%3D%22_blank%22%3E%40Beau%20Cameron%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHi%20Beau!%20You%20are%20correct%20all%20around.%26nbsp%3B%20The%20change%20in%20the%20Lookup%20list%20is%20bound%20to%20the%20lookup%20key.%26nbsp%3B%20The%20key%20is%20what%20is%20used%20at%20the%20library%20level%20and%20the%20rest%20of%20the%20data%20is%20associated%20to%20the%20key.%26nbsp%3B%20If%20a%20change%20is%20made%20to%20the%20lookup%20list%20itself%20(not%20the%20key%2C%20but%20any%20value%20associated%20to%20it)%20it%20does%20get%20pulled%20through%20into%20the%20consuming%20library.%26nbsp%3B%20Also%2C%20as%20you%20noted%2C%20if%20we%20re-index%20the%20library%2C%20which%20we%20are%20forced%20to%20do%20now%2C%20the%20new%20value%20will%20be%20picked%20up%20for%20the%20search.%26nbsp%3B%20It's%20just%20not%20a%20very%20elegant%2C%20nor%20practical%20solution%20in%20the%20long%20run.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20reason%20for%20the%20Lookup%20is%20that%2C%20by%20having%20the%20data%20local%2C%20we%20do%20get%20crawled%20properties%20that%20we%20can%20use%20for%20search%20refinement.%26nbsp%3B%20Very%20hand%20in%20certain%20business%20cases%20and%20the%20access%20to%20the%20external%20data%20brings%20in%20guaranteed%20data%20quality.%26nbsp%3B%20Secondarily%2C%20when%20data%20changes%2C%20it%20typically%20changes%20outside%20of%20the%20content%20platform.%26nbsp%3B%20Example%3A%26nbsp%3B%20When%20an%20employee%20retires%2C%20their%20retirement%20data%20is%20set%20in%20the%20HRIS%20system.%26nbsp%3B%20That%20event%20should%20trigger%20a%20number%20of%20things%20in%20SharePoint%20automatically%2C%20especially%20from%20a%20compliance%20perspective.%26nbsp%3B%20Also%2C%20name%20changes%2C%20etc.%20to%20ensure%20that%20the%20information%20can%20be%20found%20both%20by%20the%20historical%20data%20as%20well%20as%20the%20current%20data.%20%26nbsp%3BThe%20goal%20is%20to%20eliminate%20stale%20content.%26nbsp%3B%20While%20the%20document%20itself%20may%20not%20change%2C%20the%20information%20about%20what%20the%20content%20relates%20to%20quite%20often%20does.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ELastly%2C%20we%20sync%20the%20master%20data%20with%20the%20SharePoint%20list%20on%20a%20periodic%20(nightly%2C%20weekly%2C%20etc.)%20basis.%26nbsp%3B%20This%20ensures%20that%20we're%20always%20in%20sync%20with%20the%20source%20data%2C%20now%20and%20into%20the%20future.%26nbsp%3B%20The%20sync%20process%20allows%20us%20to%20have%20local%20data%20which%20gives%20us%20the%20crawled%20properties%20and%20ability%20to%20provide%20strong%20sorting%20and%20filtering%20for%20the%20industries%20we%20serve.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-644399%22%20slang%3D%22en-US%22%3ERe%3A%20Getting%20changes%20to%20Lookups%20to%20trigger%20a%20crawl%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-644399%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F294901%22%20target%3D%22_blank%22%3E%40melkinsco%3C%2FA%3E%26nbsp%3BHi%20Michael!%20%3A)%3C%2Fimg%3E%20How%20is%20data%20normally%20presented%20to%20users%3F%20Do%20users%20open%20up%20the%20items%20and%20click%26nbsp%3B%20on%20the%20lookup%20columns%20to%20find%20more%20information%20on%20the%20master%20data...%20or%20are%20you%20possibly%20using%20term%20driven%20pages%20to%20present%20some%20of%20this%20content%20to%20your%20users%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-644420%22%20slang%3D%22en-US%22%3ERe%3A%20Getting%20changes%20to%20Lookups%20to%20trigger%20a%20crawl%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-644420%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F39836%22%20target%3D%22_blank%22%3E%40Beau%20Cameron%3C%2FA%3EThe%20users%20don't%20see%20the%20source%20list%2C%20only%20the%20library%20where%20the%20Lookup%20is%20applied%2C%20and%20any%20of%20the%20pull%20through%20values%20are%20available%20via%20the%20view%20in%20the%20library.%26nbsp%3B%20If%20a%20value%20changes%20in%20the%20list%2C%20it%20is%20automatically%20reflected%20in%20the%20library%20right%20away%2C%20and%20available%20for%20filtering%2C%20etc..%26nbsp%3B%20The%20other%20way%20we%20make%20the%20info%20is%20available%20is%20via%20search%20interfaces%20as%20any%20of%20the%20pulled%20values%20can%20also%20be%20used%20as%20refiners.%26nbsp%3B%20Of%20course%2C%20without%20the%20crawl%2C%20the%20search%20interface%20is%20behind%20a%20bit...%3C%2FP%3E%3C%2FLINGO-BODY%3E
Contributor

We're using a Lookup column to link to master data.  When the master data is updated, the changes are applied to the content via the Lookup.  While this works well for the browsing and filtering within the list view, the update to Lookup pull-through data does not appear to trigger a crawl.  We use these properties for search purposes (refiners) and need them to be updated.  Is there a way to trigger the crawl when an item is updated in the Lookup list for the libraries that use it?

5 Replies

@melkinsco I'm going to guess here, but I think it might be because updating the Master Data doesn't actually update the records that lookup to it (because that is related by ID). To confirm, make a change to your master data and check the modified date of your related records... I do not think the modified date will change.

I'm guessing... that because of this, SharePoint doesn't recognize it was a change and doesn't re-crawl your item during an incremental crawl. It would only be picked up during a full crawl which we have no control over in  SharePoint online.

To test, you could re-index the list and see if SharePoint picks it up during the next crawl. How often  is your Master Data changing? If it's not often, scheduling a re-index is likely fine... however,  if it's changing often, the re-index will put a heavy load on the system and should be avoided, and you may need to look into another approach for refining on master data.


@Beau Cameron 

Hi Beau! You are correct all around.  The change in the Lookup list is bound to the lookup key.  The key is what is used at the library level and the rest of the data is associated to the key.  If a change is made to the lookup list itself (not the key, but any value associated to it) it does get pulled through into the consuming library.  Also, as you noted, if we re-index the library, which we are forced to do now, the new value will be picked up for the search.  It's just not a very elegant, nor practical solution in the long run.

 

The reason for the Lookup is that, by having the data local, we do get crawled properties that we can use for search refinement.  Very hand in certain business cases and the access to the external data brings in guaranteed data quality.  Secondarily, when data changes, it typically changes outside of the content platform.  Example:  When an employee retires, their retirement data is set in the HRIS system.  That event should trigger a number of things in SharePoint automatically, especially from a compliance perspective.  Also, name changes, etc. to ensure that the information can be found both by the historical data as well as the current data.  The goal is to eliminate stale content.  While the document itself may not change, the information about what the content relates to quite often does.

 

Lastly, we sync the master data with the SharePoint list on a periodic (nightly, weekly, etc.) basis.  This ensures that we're always in sync with the source data, now and into the future.  The sync process allows us to have local data which gives us the crawled properties and ability to provide strong sorting and filtering for the industries we serve.

@melkinsco Hi Michael! :) How is data normally presented to users? Do users open up the items and click  on the lookup columns to find more information on the master data... or are you possibly using term driven pages to present some of this content to your users?

@Beau CameronThe users don't see the source list, only the library where the Lookup is applied, and any of the pull through values are available via the view in the library.  If a value changes in the list, it is automatically reflected in the library right away, and available for filtering, etc..  The other way we make the info is available is via search interfaces as any of the pulled values can also be used as refiners.  Of course, without the crawl, the search interface is behind a bit...

I think it is an obvious oversight in the architecture of Lookup columns. Microsoft should fix it that when a lookup column is configured on a target list the re-indexation is expanded to both source and target lists automatically when an item on the source list is changed. How to get this kind of 'niche' flaw noticed by the powers that be, is another question though