User Profile
dustintadam
Iron Contributor
Joined 8 years ago
User Widgets
Recent Discussions
Re: OneDrive Client, Files on Demand and Syncing large libraries
Joe McGowan Its likely the rate of change that's causing you the most problems. Every time something changes (especially in a file copy scenario), OneDrive has to re-scan everything that's on disk to ensure that it hasn't lost a file somewhere (i.e. "Was this thing renamed? or was it moved? or both?"). When multiple machines are moving a lot around, it exponentially increases the local demand on each individual machine. The only way to alleviate this is to break up the content across multiple libraries. Do all 100,000+ files need to all be in the same library? do they have to all be brought down locally? if you are lucky enough to answer no to both of those questions, the solution is simple: break up the content into smaller libraries and make stale data online-only (disable sync through in the library settings). Unfortunately, this isn't exactly OneDrive's fault, Windows still has no mechanism where an app could simply subscribe to a subsystem and say "tell me anytime something changes on disk, including all the context". That leaves it with no other alternative than to literally go searching the disk every time anything updates to make sure it hasn't lost anything... but if it gets overwhelmed with too many changes it simply cant keep up.Re: OneDrive Client, Files on Demand and Syncing large libraries
Joe McGowan Nothing major that will simply "solve" the problem unfortunately. Couple questions: What Client version are your users running? Does the library contain folders with broken inheritance? How many users are syncing the library simultaneously? Does the library have a high rate of change? (i.e. lots of files being modified in many different folders) or is it a lot of old static content?Re: OneDrive For Business Sync Client Custom Message
I dont think the notification itself can be modified, but you can help prevent low disk space by deploying policies that control the Storage Sense feature in 1903+ https://support.office.com/en-us/article/use-onedrive-and-storage-sense-in-windows-10-to-manage-disk-space-de5faa9a-6108-4be1-87a6-d90688d08a48?ui=en-US&rs=en-US&ad=US https://docs.microsoft.com/en-us/windows/client-management/mdm/policy-csp-storage not only can Storage Sense interact with OneDrive and make content Cloud Only when it becomes stale or inactive, it can also automatically clean up temporary files that build up over time.Re: OneDrive Client, Files on Demand and Syncing large libraries
dimitrispavl1400 JonnaP JonasBack Couple other things we've discovered / are aware of: depending on what article you're reading, you might see the words "file" or "objects" when discussing limits. In reality, the upper limits should always be discussed as objects. From the sync clients' perspective, a folder, as an "object", carries as much weight as a file when it comes to tracking things that can change. When calculating how much pressure is being placed on the client, its important to consider the count of folders in your tree as well as files, add those two together, and that's the object count you should be considering. For example, on a test VM I can successfully sync content that totals 98,000 files stored in a tree that contains 100,000 folders. It's not super-performant, but in this scenario that's because I'm not syncing 98,000 files, I'm syncing 200,000 'objects'. Permissions play a huge role in the sync clients performance as well, and often in ways that aren't obvious: One of the things the OneDrive development team has been working on (they may have fixed this already, but I tend to think not) is the notion of a "Sync Root"; essentially, at the moment, sync can only be initiated from the root of a document library. How this manifests is that even if you tell a user to sync a subfolder, if they still have permissions to content higher in the tree, the sync client will still download and track all the objects in the library the user has access to behind the scenes, even if it's not being presented on disk in Explorer. The overhead isn't as high, but it does add up. The goal is to allow SharePoint to set a per-user custom Sync Root at any level in a library so that the sync context is scoped to only the level at which the content is synced from. You can help improve that by using the "Restricted View" SharePoint permission. This is a special permission level that has been around since the old Groove client, and is designed to prevent content sync. By setting Restricted View on a parent folder, but granting Edit on child content, you can essentially create online-only content that can't be synced, and the kicker is that the Sync Client wont download anything under that parent folder to its local manifest, helping reduce load significantly. This helps keep content in the same library for business or process continuity. Its also helpful to enable the 'PermitDisablePermissionInheritance' GPO switch if you are using Read Only permissions and Sync, as without this the OneDrive client is exceptionally slow to sync content if Read Only is used anywhere in the tree being synced. https://docs.microsoft.com/en-us/onedrive/use-group-policy#PermitDisablePermissionInheritanceRe: OneDrive Client, Files on Demand and Syncing large libraries
I hear you, its a struggle, our own roll-out of OneDrive has been exceptionally limited because of the upper limits. I know that it's on their radar, and there are a lot of customers that are struggling. Through the grapevine I hear that the internal target they are going to try and work towards is getting the sync client to handle a million objects gracefully, but they aren't going to attempt to get it to handle more than that. Obviously, they wont provide a timeline for this officially, work on performance improvements isn't at all the same as building a new feature that's easier to hold yourself to. Could be this year? I don't have a sense of how much of a priority this is for them, and I would suspect that achieving that is going to require engineering changes beyond simply the sync client itself. All of that is to say, yeah, it'll get significantly better and likely open up most use cases for migrating file server content to SharePoint... but don't hold your breath, it'll happen when it happens.Re: OneDrive Client, Files on Demand and Syncing large libraries
I would suggest, at least for the individuals who have access to a sub-set of the content, enabling the "Permit Disable Permission Inheritance" Group Policy flag for the OneDrive client: https://docs.microsoft.com/en-us/onedrive/use-group-policy#PermitDisablePermissionInheritance You should also consider enabling the Storage Sense group policy option in Windows 10 (on computers running at least 1903) that can automatically make synced files Online Only if they haven't been accessed in a specified time period, as this helps keep the pressure off the sync client as well. https://support.office.com/en-us/article/use-onedrive-and-storage-sense-in-windows-10-to-manage-disk-space-de5faa9a-6108-4be1-87a6-d90688d08a48 You can enable that manually, but you can also set Storage Sense options through GPO or InTune, depending on how your client's PCs are managed. Lastly, I find it concerning that all the content is stored in a OneDrive account as opposed to a SharePoint Document Library. While, yes, a ODfB account is in essence a SP Library, and content can be shared from it, it wasn't designed to be used in the same manner. OneDrive libraries are not given the same processing overhead as a SP Library, which is intended to serve a large number of clients. all OneDrive requests place burden on the backend SQL infrastructure to serve back responses to clients, storing the organizations content in OneDrive instead of SharePoint is bound to introduce performance issues, as OneDrive is provisioned to serve a single client primarily.Re: OneDrive Client, Files on Demand and Syncing large libraries
JonnaP Are there a lot of broken inheritance permissions in the content that your client is syncing? or does everyone have access to everything? there is a performance switch that can be enabled in the Sync Client if it's handling read-only folders or folders with broken inheritance.Re: OneDrive Client, Files on Demand and Syncing large libraries
Ah, yeah, thats not how the client works. Whether you're using Files on Demand or Full Sync, the OneDrive client is integrated with the NTFS file system and actually uses a number of unique NTFS property flags to set the state of any given file. So in reality, something is always going to be written to disk in some fashion when you sync a library. What you're referring to is only possible in SharePoint and OneDrive if you use WebDAV, and in fact we do at my org so get people into libraries that the Sync Client cant handle yet, it's dirty and comes with it's own set of unique challenges but it does act largely the way you describe.Re: OneDrive Client, Files on Demand and Syncing large libraries
_Chris_G Hey Chris; I'm not sure if this is exactly what you are looking for, but through MDM or ADML templates you can enforce the OneDrive Client to use Files On Demand by default: https://docs.microsoft.com/en-us/onedrive/use-group-policy#FilesOnDemandEnabled If I misunderstood your question let me know.Re: OneDrive Client, Files on Demand and Syncing large libraries
jab365cloud, Innovation ; A couple things to look for and consider: One of the things we've discovered that isn't really documented anywhere is that the more content you shove into a single Document Library, the worse that library performs. Adding additional Indexes to the Library manually can help, but in general, the fewer items you put into a library the better. This becomes apparent even when browsing the library via the UI: a library with fewer total objects browses faster than one with hundreds of thousands. The Sync client will ultimately be affected by that increased overhead as well: when it makes API calls to detect or replicate changes, it's going to take longer to complete. We learned this the hard way ourselves and are actively working on breaking up our libraries. If you haven't migrated data yet, find a way to break up your content into as many libraries as possible to reduce the total volume. Also bear in mind that as is the nature of all file storage, it never gets smaller, nobody ever deletes anything, if you start with an overly large library, your experience will never get better from that point. I know that the eventual goal is to get the OD Client to gracefully handle syncing up to a million objects, but that hasn't been publicly communicated and there is no timeline for when that might be realized.Re: OneDrive Client, Files on Demand and Syncing large libraries
jab365cloud ; There are a lot of variables involved that can impact sync performance. One thing to note however: The OneDrive Team has been working on a method to sync individual folders that will be more performant, its related to the notion of "Sync Root". Essentially, SharePoint currently only allows the "Sync Root" to be the root of a Document Library. How this manifests in practice is that the Sync Client will download ALL the metadata regarding the entire library, even if you're only syncing a single subfolder. When they do manage to finish the work to allow SharePoint to set the Sync Root to a different level, you'll see improved performance when synchronizing a single subfolder. However, another thing to keep in mind (that my organization learned the hard way), is that large libraries place additional pressure on the SQL infrastructure behind the scenes. Even if everything is working, you'll notice that all operations against a larger library will be slower across the board. We are in the process of breaking up our larger libraries into smaller ones to ensure that we can maintain better performance in web browsing, sync, etc.Re: OneDrive Client, Files on Demand and Syncing large libraries
Diego Vasquez ; All the current technical documentation regarding the Sync Client and it's limits can be found here: https://support.office.com/en-us/article/invalid-file-names-and-file-types-in-onedrive-onedrive-for-business-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa?ui=en-US&rs=en-US&ad=US#synctoomany Interestingly, and I'm assuming this is based on feedback, the upper sync limit has been revised down from 300,000 files to 100,000. There isn't any formal documentation regarding the behavior of Restricted View and it's impact on the sync client, but in our testing it does honor the underlying permission limits defined by the permission level: The critical thing to understand with this is that this is a recursive permission, this permission will cascade down the rest of the directory tree underneath where this permission is defined, regardless of any other permission a user may have (Even if you have broken inheritance further down). In our use case, we applied this permission to our "Archive" folders inside our document libraries, effectively allowing our colleagues to access archived content in the browser, but restricting their ability to download it with the sync client. Not only does this help protect the archived content, it relieves stress on the client. I would recommend that anyone currently struggling with over-loaded clients take a second look at cloud content and determine if any of the data in a Document Library really NEEDS to be synced, if not, there is probably method to help relive client stress by implementing Restricted View on parent folders that hold content that can effectively remain cloud-only and then re-assign contribute permission below that level.Re: OneDrive Client, Files on Demand and Syncing large libraries
Matt_Ignite They are working towards higher limits. Part of the problem stems from the fact that they were trying to bring the Cloud Experience down to the desktop with the highest fidelity possible. By most measurements, they succeeded. For example, renaming or moving files can be completed without downloading the file to the desktop, it simply brokers the API calls that complete the action in OD or SharePoint. This is super cool, but it begins to fall over when the number of objects starts to increase. Windows itself was never designed to really support this sort of integration, so there isn't any robust or elegant way to poll the drive and ask "What Changed?". This leaves it up to the client itself to have to scan periodically for changes to Files on Demand content, even if it hasn't been downloaded, because action can still be taken on those objects. While I can't give you any specific number, I can tell you that the total number of objects their client is about to be able to handle without crashing is significantly higher. But that's really just the first step, while the client is getting more bullet-proof when handling large numbers of objects, there is still more work to do on performance. The more objects you bring down, the longer it will take for the client to detect changes and complete a replication either to or from the cloud. They are making significant headway, and I can assure you they care more than you might imagine about the limits the client can handle, hang in there, it's getting better ;)Re: OneDrive Client, Files on Demand and Syncing large libraries
Hey Nando; As it happens I managed to get in contact with the product development team and they’ve been working with us over the last month or so regarding sync and large libraries. Long story short: they are actively making changes to the sync client based on our feedback and use case that will see the upper limits raise fairly substantially. In addition, there is a neat little trick we learned from the SharePoint product team: If you don’t want to restrict sync for an entire library, but only a subset of content inside a library, you can set “Restricted View” permissions on a parent folder, yet still add higher permissions lower down in the tree. This causes the sync client to ignore all content below the Restricted View permission, effectively allowing you to segregate content inside a library that you want to allow the local sync client to bring down. We have an NDA with Microsoft so I can’t disclose any detail, but large library sync will be getting better within the next few months, with more to come later in the year.Re: OneDrive Client, Files on Demand and Syncing large libraries
Not at the moment no, though it’s on our roadmap. We are in early planning stages for an RMS rollout, part of the problem is that we are leveraging a wide number of other Microsoft technologies such as Cloud App Security, and we want to make sure that we take advantage of the full ecosystem across CAS, Azure RMS, SharePoint, Windows Information Protection, etc. I’m sure once we get that far I’ll have more to share :)Re: OneDrive Client, Files on Demand and Syncing large libraries
Update: I recently got to have a conversation with some people from the OneDrive engineering team regarding an issue we were having that stemmed from a recent code update to SharePoint that impacted the way OneDrive handles document library content (our tenant is on First Release). In short, our permissions model was specifically designed to take advantage of client behavior wherein, if a user had access to content, but NOT the parent folder, OneDrive would not try to represent that content in the local Files on Demand cache file and would only process the data that the user could navigate to. We had various reasons for doing this, from effectively hiding content that was stale or archived, reducing load on OneDrive iOS clients, as well as reducing workload on the OneDrive Windows clients. This behavior was driven by what Microsoft apparently saw as a problem that they referred to as Gap Folders, folders whose permissions had a gap from the parent, and because of this, the Sync Client couldn't synchronize content that had a gap in permissions. Well, they recently pushed a code update to Sharepoint that addressed this issue, allowing the Client to have an understanding of all content in a library that a user has permissions to, which undoubtedly solved sync issues for many customers. Unfortunately this introduced a huge issue to us, as over 80% of our content is considered Archived and should never be synced locally, even though it's routinely accessed for historical content and we had taken advantage of the SharePoint permissions model and the client to manage this is a way that to us was an elegant and easy to manage solution. If you're using gapped permissions in a similar manner with clients actively deployed, be aware that there are code changes coming down the pike that could cause clients to get overloaded and fail if they are suddenly presented with significantly more data that had been planned for. They are actively evaluating this issue now, but I don't have an update yet on what the path forward will look like, as there are legitimate use cases for both wanting the client to see gapped content and not.
