Inside the new Files On-Demand Experience on macOS
Published Jan 12 2022 10:28 AM 254K Views
Microsoft

2.24.22 UPDATE: We've been listening to your feedback, and we've made some design changes. We're releasing a new version that addresses the most common themes and makes it easier to achieve the previous experience. Please read the latest blog post for more details.

 

2.15.22 UPDATE: We're actively reviewing feedback and are aware of the difficulties some users are experiencing with the recent update. We working as quickly as possible to resolve these issues. We will share an update soon.

Additional information can be found in the FAQ updated on 2.1.22.

- Team OneDrive 

 

 

In June 2021, we announced several important updates for OneDrive on macOS, including an update to our Files On-Demand experience. This new experience is better integrated with macOS and will also help enable new features like Known Folder Move for macOS.

 

Today, we are excited to share that we have begun rolling out the new Files On-Demand experience to all our customers using macOS 12.1 or later. We also want to share some additional details about how the new Files On-Demand experience works, what changes you can expect, and when you can expect them.

We know that many of you are supporting organizations with lots of Macs that run OneDrive, and the more information we can provide you, the better you can serve your users.

 

Why we are building a new experience

 

In 2018, we shipped the first version of Files On-Demand for macOS. Since then, we’ve rolled it out for everyone using macOS 10.14 (Mojave) or later.

 

We are building a new experience for several reasons. One of the most important is that the new technology stack (based on Apple’s File Provider platform) is much better integrated with the operating system compared to the first version. This means a better user experience, better application compatibility, and better reliability. This technology stack also enables us to offer new features that we couldn’t offer before, like Known Folder Move, along with lots of other little improvements that you probably won’t notice right away!

 

Because the new experience is more integrated with macOS, it will have long-term support from Apple. The first version of Files On-Demand is built on several pieces of technology that are now deprecated. Moving to the new platform enables us to support this feature for years to come.

 

Supported macOS versions

 

For users who are not part of our Insiders program, the new Files On-Demand experience requires macOS 12.1 or later. Users on our Insiders program can continue to use macOS 12.0, but we strongly encourage them to update to the latest version.

 

macOS 12.2 will be the last version that supports the classic Files On-Demand experience. For macOS 12.3 or later, this means:

  • Files On-Demand will default to on for all users and cannot be disabled.
  • Devices will migrate automatically to the new Files On-Demand as soon as they receive a macOS update. You cannot delay this update without also delaying an update to macOS.
  • Both our Standalone and App Store versions of OneDrive will have the same behavior.
  • Users running a developer or beta version of macOS will have the same experience as a release version of macOS.

 

If you are concerned about application compatibility with this change, we strongly suggest that you install macOS 12.3 or later and test your workload. Depending on the results of that test, you may want to delay updating your Macs.

 

File system requirements

 

The new Files On-Demand experience requires a volume that is formatted with APFS. HFS+ volumes are not supported.

 

We are ending support for HFS+ after macOS 12.3 or later. A very small minority of customers are syncing OneDrive on HFS+ volumes today. As we roll out the new Files On-Demand, these users will first experience a warning in the OneDrive activity center telling them to upgrade to APFS. Once the new Files On-Demand experience is fully rolled out, OneDrive will not launch until the volume is upgraded to APFS.

 

You can use Disk Utility (built in to macOS) to determine which volumes are running APFS, and upgrade any HFS+ volumes to APFS.

 

Sync root location

 

When users set up OneDrive, they must choose a location where files will sync. This is called a sync root. Historically, we’ve allowed users to choose any location on any fixed volume mounted on their Mac.

 

With the new Files On-Demand experience, the sync root is always located within users’ home directory, in a path such as:

 

~/Library/CloudStorage/OneDrive-Personal

 

As part of the upgrade, the sync root will be moved to this location. This location cannot be moved or changed and is controlled by macOS.

 

This path is a little cumbersome for users to use, so they can access this directory in two other ways:

  • Under Locations on the Finder’s left navigation pane.
  • Via a symlink at the original location the user picked when setting up OneDrive. For example, if the user chose to sync OneDrive at ~/OneDrive, then a symlink will be created from here to ~/Library/CloudStorage/OneDrive-Personal.

 

Cache path

 

To support the new experience, OneDrive maintains a cache path in a hidden location. This path contains a replica of the file tree that the user is syncing. Most of the files in this location are usually dataless and don’t consume disk space, but occasionally files here can have data, such as if a user pins a file or if a change is being transferred to or from the cloud.

 

OneDrive tries to maintain as little data here as possible, and instead prefers to keep data in the sync root. As such, file data is not generally kept in both locations unless a file is marked as “Always Available on This Device.” In that case, the file’s data will sometimes be retained in both the sync root and the cache, but the files will be linked using a clone, so they do not occupy any additional space.

 

Using another volume

 

Sometimes, users choose a path on another volume to set up OneDrive. A typical use case for this happens when a user has a small internal drive on their Mac, but also has a larger external drive attached.

 

This configuration is still supported in the new Files On-Demand experience if an external drive is selected during the first-run experience. A few things change as a result:

  • The sync root remains in ~/Library/CloudStorage, on the user’s home volume. As noted above, this path cannot be moved from this location.
  • The cache path is on the volume that was selected during the first-run experience. This is located in a hidden folder that’s a sibling of the location that was chosen.
    • This folder begins with the name “.ODContainer”.
  • A symlink is created from the chosen location to ~/Library/CloudStorage.

 

For example, if the user selects /Volumes/BigDrive/OneDrive for their OneDrive path:

  • The sync root will remain in ~/Library/CloudStorage/OneDrive-Personal
  • The cache path will be set up at /Volumes/BigDrive/.ODContainer-OneDrive
  • A symlink will link from /Volumes/BigDrive/OneDrive to ~/Library/CloudStorage/OneDrive-Personal

 

Because the cache path is located on an external drive in this scenario, any pinned content will be stored there and not on the main drive.

 

The cache path folder is hidden by default. Users should not modify this folder or its contents.

 

User consent

 

For OneDrive to complete setup with the new File Provider platform, the user must consent to allow OneDrive to sync. This experience is like the experience of allowing an application access to the Documents folder or the user’s Contacts.

 

AnkitaKirti_0-1642011516079.png

 

Consent is not required in the following cases:

  • If the user previously opted-in to use the Finder Sync extension. This is set by default for the Standalone build, and the vast majority of our App Store users have opted-in as well.
  • If the OneDrive app was deployed and managed through an MDM tool. MDM-managed applications are considered to have implied consent by the administrator.

 

If consent is required, the user will be prompted to provide it during the first run experience when setting up OneDrive for the first time.

 

The user can withdraw consent from the System Preferences -> Extensions preference pane. If consent is withdrawn, OneDrive will display an error dialog, an error in the Activity Center, and an error icon, until the user provides consent again. OneDrive cannot run without consent.

 

Always Keep on This Device

 

A standard feature of Files On-Demand on all our platforms is the ability to mark files as “Always Keep on This Device.” Internally, we call this operation “pinning.”

 

AnkitaKirti_1-1642011516145.png

 

 

When a file is pinned, it is downloaded to disk and is always available offline, even if there is no network connection. The presence of the check mark icon indicates that a file is in this state. Folders can also be pinned, which means that all files and folders underneath the folder will inherit the state, and new files added to that folder will also inherit the state.

 

Pinning a file on the new Files On-Demand platform means that its contents will be downloaded into the OneDrive cache. Because is the file is in the OneDrive cache, it can always be provided to the sync root whenever it is needed, even if the machine is offline or the OneDrive app isn’t running. The presence of the gray check mark indicates a file that is in this state.

 

You may notice that pinned files sometimes have an icon next to them that indicates they aren’t downloaded. This icon just means that the file isn’t in the sync root. If a file has the gray check icon, it is still always available because OneDrive has the file in its cache and can always provide it.

 

Free Up Space

 

When you no longer need a file on your Mac, you can use the “Free Up Space” option to immediately evict its data. When you do this, data is evicted from both the sync root and OneDrive’s cache, ensuring it occupies no bytes on disk. However, it is still available in the cloud.

 

Disk space usage

 

Files that are kept in the sync root do not count against disk space usage, unless they are marked as “Always Keep on This Device.”

 

For example, imagine the sync root contains five files, each 20mb in size, for a total of 100mb. These files are fully in-sync with the cloud. Now imagine another application asks about the amount of free space on the drive. These five files do not count against the disk space used, so the size reported to the application will be 100mb larger than you might expect.

 

The reason for this behavior is that in low disk space situations, these files can be automatically evicted from the disk to make room for more data. For instance, imagine an application wants to write 50mb of data to disk, but there is no more disk space. However, because the five files in the sync root can be evicted as needed, the write can safely complete. To do that, three of these five files will be evicted to make 60mb of space, and so the 50mb write completes.

 

This behavior has several implications:

  • Files that have data in the sync root can be evicted at any time.
  • The system will automatically clean up files as disk space runs low.
  • Mark files as “Always Available on This Device” if you do not want them to be evicted.

 

We know that some organizations have scripts or something similar to automatically free up space for OneDrive content, usually at login or on a set schedule. Because macOS will automatically free space from OneDrive files as needed, such scripts are no longer necessary, unless you want to prevent users from keeping content in the “Always Keep on This Device” state.

 

File system feature support

 

The new Files On-Demand experience supports some existing features of APFS that were previously poorly supported by OneDrive. These include:

  • File tags
  • Last used date
  • File system flags
  • Extended attributes
  • Type and creator code
  • Symlinks

 

Note that changes to these properties do not sync to or from the cloud, but OneDrive will preserve them on the local file system. Previously, they might only have been preserved for a short while but overwritten by a change from the cloud.

 

Symlinks have special support in the new experience. They are preserved as a symlink in the sync root but do not sync to the cloud as a symlink, as the OneDrive cloud does not support symlinks. Instead, the symlink will sync to the cloud as a plain text file with the symlink target as its contents. Previously, OneDrive ignored symlinks.

 

Packages

 

OneDrive now supports syncing packages, or files that appear as a single file but are actually a directory with many files and folders underneath them. Some applications exclusively create packages. Additionally, most Mac applications are stored on disk as a package.

 

Traditionally, the problem with syncing packages has been that packages often contain file states that don’t sync well in the cloud. For example, some packages contain internal symlinks, extended attributes, or other file system quirks that can result in a corrupt package if these are not synced correctly. The OneDrive app itself is an example of such a package – previously, if you saved the OneDrive app in OneDrive and attempted to open it on another Mac, it would be corrupt.

 

With the new experience, packages are now synced as a single file with a hidden .pkgf extension appended automatically. For instance, if you create a file named “Foo.app” in your OneDrive, it will sync to the cloud as “Foo.app.pkgf”. OneDrive automatically strips the .pkgf extension on compatible Macs, and the file will appear as a valid package on all compatible Macs.

 

Note that Macs not running the new Files On-Demand experience cannot read files in the .pkgf format.

 

Unlink, unmount, and reset

 

When you unlink your Mac or unmount a syncing location, OneDrive will preserve the non-dataless contents of your sync root. This works by removing the symlink to ~/Library/CloudStorage, creating a new folder in its place, and moving the files in your sync root that are not dataless to that location. Files in the OneDrive cache path are removed.

 

OneDrive also ships with a reset script included in the application bundle. This script behaves in a similar way, except that the non-dataless files are always moved into a folder in the user’s home directory named something like “OneDrive (Archive).” Files in the cache path are removed, except if the cache path is located on a volume other than the home volume.

 

Over time, we expect to improve this experience.

Learning more

 

 To learn more about OneDrive,

 

Thanks for reading!

 

Jack Nichols

Principal Software Architect - OneDrive

 

Update 2/1/2022

 

Hi everyone - Jack from Microsoft here. Just to quickly introduce myself, I'm the author of the original blog post, and also the architect for the OneDrive sync client. I'm the engineer who led the teams that designed and built Files On-Demand for Windows, macOS, and now the new macOS experience, so I have the most context about how Files On-Demand works and the trade-offs involved in building something like this.

The entire OneDrive team has been reading your comments, concerns and feedback, and we really appreciate everyone taking the time to write them. The community clearly has a lot of passion for OneDrive and how it works. I've spent many hours in the last week or two reading comments here and elsewhere, to understand how we can improve our macOS experience further.

Although we can't respond to all of you directly, there are a couple of themes and frequently asked questions that I wanted to answer to help provide some more clarity.

 

Why are all my files redownloading with this update? Why are my always-available files displaying a "not downloaded" icon?

 

Let me first set you at ease: your files aren't actually redownloading. What you are seeing is a bit of an optical illusion.

 

When your OneDrive instance is upgraded to the new Files On-Demand, macOS creates a new folder for your OneDrive files and we move your old folder into our cache location. We do it this way for many reasons, but two of the most important are that we can preserve your settings around which files are always available, and we can prevent the sync client from performing a costly reindex of all of your content.

As your files are brought into our cache, we tell the macOS File Provider platform about them. That causes the operating system to create the files in the new OneDrive folder that you will actually use. As part of telling the File Provider platform about your files, we include metadata about them, so that the operating system knows how big they are, what icons to show, and so forth.

Unfortunately, the current implementation of File Provider does not allow us to tell the operating system that we already have the file's contents available – so they appear to be online-only, even though their contents are safe in our cache, ready for the first time you access them. The best that we can do is tell the system to show the always available icon (the checkmark), but we can't tell the system to hide the "not downloaded" icon. The "not downloaded" icon is shown automatically by the File Provider system when the file is dataless in the sync root, and there's no current way for OneDrive to override this. Please know that we are actively investigating ways to address this, as we understand that it is a top source (if not the top source) of user confusion with this update.

 

The key thing to remember here is that if you double-click the files that we already have in our cache (files that you pinned when you selected “Always Keep On This Device” and anything you had downloaded before we did the upgrade), they will be retrieved and opened as expected, without any network traffic. This will work even if OneDrive isn't running, is paused, and so forth.

 

Why were my Finder favorite folders removed?

 

During the upgrade to the File Provider platform, OneDrive removes these favorites as they no longer point to a valid location. Most users will have a "OneDrive" favorite that will be removed in this manner, but a few users have dragged other folders of interest to this sidebar, which will also be removed if they were pointing at a OneDrive folder.

After the upgrade, if you want these favorite folders back, you will need to add them again by dragging your favorite folders to the Finder's sidebar.

 

How can I make it so that all my files are synced on my Mac and made available for offline access?

 

If you want all files synced on your device, you should pin the OneDrive folder. The easiest way to do this is to browse to your OneDrive in the Finder, change the view to Icons view, and then right-click the blank space between icons. Then, select Always Keep on This Device.. We're actively looking at ways to make this easier to configure on both macOS and Windows.

 

macOS Pin the Root 2.gif

 

 

Is there a technical reason that explains why Files On-Demand must always be enabled?

 

OneDrive has taken a dependency on Apple's File Provider platform as part of this update, as we believe it is the right long-term path forward for the product. Files On-Demand functionality is a core part of Apple's File Provider platform, but File Provider offers a lot more than that, too. I'll touch on a few of those things here below.

  • For instance, the little icons you see next to your files in OneDrive for macOS are now handled by the File Provider platform. This seems like a small thing, but it has a big impact. Before File Provider, we used something called a Finder Sync extension to show these icons in the Finder, but the Finder Sync extension was one of the top sources of problems on the macOS sync client. For example, the icons sometimes mysteriously disappeared, or performance problems affected the system. Because we eliminated the Finder Sync extension, we also eliminated an entire class of problems as a result.
  • This has also improved the reliability of OneDrive running on macOS. As part of our normal sync process, the sync client occasionally runs checks to ensure everything is syncing correctly. The results of those checks are reported to us as telemetry which we use to help ensure there aren't emerging bugs, and most of the time, we find and fix bugs before anyone notices them. We've been very closely monitoring this telemetry as the new macOS Files On-Demand experience has rolled out, and we’ve noticed is that reliability is significantly better than what we had before. This translates to a much better sync experience for you.
  • Finally, it is important to note that beginning in macOS 12.3, File Provider is the only Files On-Demand solution that is supported on macOS. Our prior solution is no longer supported. 

Files On-Demand has been available on Windows since 2017, and on macOS since 2018. In that time, we've progressed from the feature being opt-in only to being on by default for all users and have closely monitored how many users turn off Files On-Demand. Only a very small number of users disable Files On-Demand on both platforms, and there are two main reasons for that.

  • Application compatibility: When Files On-Demand first shipped on Windows, some applications didn't work well with the way we stored files, or with anti-virus or other security software that was installed. Over time, we've fixed most of these problems. On macOS, we took a similarly cautious approach, but the application compatibility landscape is quite different and, in some ways, less complex. Still, there were a few cases where, due to the technology stack we were using on macOS, it made sense for certain users to disable Files On-Demand to preserve compatibility. With the File Provider platform, these problems have gone away, so application compatibility issues on macOS should be much less likely to occur. If you find something different with your setup, please reach out to your support contact so we can diagnose the issue.
  • Locally available files: We know that keeping all content locally on the device is an important scenario for a small set of users. The best way to do this is to select Always Keep on This Device from the right-click menu to mark content as “pinned”.

Note that this applies to folders too; if you pin a folder, all of the content that's currently in it and new content that is added to it will be kept on the device.

 

Why is it sometimes slow to browse folders in my OneDrive?

 

To save space and system resources, the File Provider platform doesn't actually create the files OneDrive is managing until the first time you need them. The first time you open a OneDrive folder, macOS will create them on-demand. This can sometimes take a moment.

To avoid this delay, you can force the system to pre-create all of these files and folders for you without downloading your content. To do this, open a Terminal window and type "ls -alR ~/OneDrive" (or the path to your OneDrive). This will ensure all of your files and folders are created, but not downloaded, before you browse.

 

Can OneDrive be stored on an external drive? How does pinning a file work when I use an external drive? Are there multiple copies of my data?

 

I've seen several threads on this topic but let me clarify with an emphatic yes: external drives are fully supported without any difference in the end-user experience.

That said, external drive support as it exists today is implemented differently than it was in the past because of how File Provider works. Very few users are running this configuration, but for them, it's an important scenario because often their content won't fit on the home drive. File Provider doesn't support creating the sync root on any drive except for the home drive. So, we had to find a way to support external drives within these constraints.

When you choose a path to sync your OneDrive, we use that path to derive where we put your OneDrive cache path. If that path is on an external drive, we'll put the cache path there. We wanted to honor this preference because the cache path is where your pinned content is stored, as I'll explain below.

When your cache path is placed on an external drive, OneDrive tries to minimize the number of copies of your data it makes, and in most cases, only one copy will exist, usually in the sync root. If your home drive runs into disk pressure, the operating system will evict (dataless) files from the sync root, but they can always be obtained again from the cloud if needed. In some cases a file might exist in both places for a short time, but over time we will ship fixes that will optimize this further.

Pinned files on an external drive have behavior quirks that are worth understanding. If you pin a file, it will download to the cache path only, and will show both the checkbox and "not downloaded" state icons. This is because the file is dataless in the sync root but exists as a full file in the cache path. However, if you pin a file and also double-click it to open it, we will bring it into your sync root, so there are two copies, one in each location. Note that files brought into your sync root in this manner can still be evicted by macOS when it encounters disk pressure, but when this happens, only the file in the sync root is evicted. We still keep the data in your cache path, so you can always get to the file's content, even if you are offline.

The table below depicts how this works when you set up sync on external drives:

 

User action

File in sync root

File in cache path

Default state

Dataless

Dataless

Right click -> "Always Keep on This Device"

Dataless

Has data

Double click the file

Has data

Has data

macOS runs into disk pressure

Dataless

Has data

Right click -> "Free Up Space"

Dataless

Dataless

Essentially, the table depicts the guarantee that OneDrive makes about pinned files, namely that as long as we have a pinned file, we'll always keep the data available to you locally. The only time we don't have that data is either in the default state, or if you tell us to free up space for the file.

 

How does disk space usage work in the sync root?

 

In the original blog post, I mentioned that files with data in the sync root do not count against your disk usage. Some people took this to mean that these files occupy zero bytes on the disk, but what actually happens is that these files don't count against your used disk space. That is, if an application asks, "How much space is free on this disk?" that answer will exclude these files.

There are a handful of special cases where these rules don't apply:

  • Pinned files, if your cache path is on your home drive. In this situation, the file in the cache path and the file in the sync root are Apple File System (APFS)clones of one another, and although there are two files, they share the same space on disk until one changes. File Provider won't evict files that have a clone, and such files will count against used disk space.
    • If your cache path is on an external drive, there is no clone, so pinned files can be evicted from your sync root and don't count against used space on your home drive.
  • OneDrive designates certain file types as non-evictable, and therefore these files count against used space. The most important of these file types are shortcuts to OneNote files, which only occupy a handful of bytes. This matches the behavior on Windows as well.

The system logic to decide what files count against used disk space and what files do not is provided by the File Provider platform. If you find behavior that works differently than I've described here, please reach out to support, or to Apple.

 

Will this work with local file indexing (e.g. Apple's Spotlight)?

Yes. Spotlight indexes everything that is in your sync root, but note that Spotlight will not fetch (or hydrate) files that are dataless. If you are looking for something in Spotlight that could only be read from the full file (such as image EXIF data), only fully hydrated files will be indexed.

Spotlight will not index our cache folder.

 

Why is my AutoSave not working after this upgrade?

 

We have been made aware of users experiencing issues with AutoSave when using the Store version of the OneDrive app. We are actively working to resolve this in the next few days. In the interim, if you want to get unblocked, you can move to the Standalone build.
To move to the Standalone build from the Store version, you can unlink your account, uninstall the App Store version, and reinstall the Standalone version from this link: https://go.microsoft.com/fwlink/?linkid=823060.
We will keep you updated about the fix on this thread.

Update 2/3: The fix for AutoSave for the App Store version was released in the Store today. It is fixed with 22.002.0201.0005.

 

If you still have more queries feel free to reach out to the team directly on this discussion thread.  Files On-Demand for macOS QA - Microsoft Tech Community

Thank you for your constant support and partnership!

Jack 

 

598 Comments
Iron Contributor

This is by far the most disadvantage that happened to a digital product that I have seen in ages.

 

Force users to use a function they don't want to use (Files On Demand)

Take away flexibility and options.

Try to sell this as a better user experience.

 

Do I get it right, that when I want to keep my files on my local drive, everytime I have a new file or directory I have to mark it as pinned?

Copper Contributor

Will this work with local file indexing (e.g. Apple's Spotlight)?

Will this work with low level file access via the UNIX shell, especially mass operations like a recursive search for file containing "example".

grep -lr example

Will these at least work if one marks all files as "Always Keep on This Device"?

The stuff with the root and the cache sound very complicated.

Copper Contributor

Is it possible for you guys to make Quickview work for files that we haven't already opened but are in folders that have been marked as Always On This Device?

Brass Contributor

So after happily using OneDrive on my Mac for 7 years, all of a sudden this morning it stops working. Just wasn't running any more.

 

I startup onedrive and what to my wondering eyes should appear? I get asked to re-sign in and setup onedrive???.....which is now redownloading everything across my 2 onedrive accounts.....AND even though I picked the SAME folders, all my pinned onedrive folders now point to orphaned folders and I have 2x the data on my drive......

 

Leaving my machine overnight....will see what mess I have in the morning.

 

No time in my schedule to deal with this. This happen to anyone else?

 

Pedro Cardoso

Brass Contributor

Sorry, but after reading this article, I decided to stop using OneDrive as soon as possible!

All this sound like a awful hack - the user will completly loose the control not knowing where his data really are, in what state they are and how the get them back...

Very sad story....

BTW: Google did the same lunatic move with their Google Drive by completly crippling their Sync function a while ago.

Best regards

Wolfgang

Iron Contributor

Yes, check this out. @CanadianTechie 

 

https://answers.microsoft.com/de-de/msoffice/forum/all/option-dateien-bei-bedarf-in-einstellungen/77...

 

Only a rollback was helping.

 

I really dont understand the better user experience they talk about. Maybe someone could explain? @Ankita Kirti 

Copper Contributor

It seems very interesting but... what version that will start? How we will detect our system has been updated? Too much words but the real interesting thing, what versions do that, are not shown.

Copper Contributor

No news on Known Folder Move? What a joke.

Brass Contributor

What a nonsense are these news:/ Microsoft, are you thinking at all? Please, let an option to disable this messed up on demand feature. My situation. I have Onedrive sync root on the external drive. I do not want to have anything related to Onedrive on my internal drive. All of my synced data is stored in onedrive cloud and locally. And after this "nice" change my sync root gets redirected to internal drive, my local files are moved to hidden cache folder. What a heck is this? What to do with new additional files which I will want to have synced and locally available? I'll be forced to copy them to new sync root (which is on internal drive) and they will end up there and not on my external drive. Microsoft, is your plan to move your customers away from onedrive? Please, let users to disable these your innovative changes completely.

 

Microsoft, be a little different from Apple - do what your users want, not only what you want:/ Looking at the comments I see that majority of commenters do not like this your new experience.

Brass Contributor

I just wanted to follow up these comments here, because I just spent all morning speaking to Microsoft support, who had no idea what I was talking about.

 

I first encountered this problem when I did exactly as I was told by the onedrive app, upgrade my external drive to APFS.

 

Now, all my files are sync'd to my internal drive, which is way too small to host them all.

 

I'm about to re-format the drive to HFS in the hope that this disables this new 'feature', but I wish I'd known what this 'upgraded experience' was going to do before I started all this.

 

**EDIT** - Re-formatting my drive to HFS was not successful. Now, Onedrive blocks users from setting external drives as the onedrive location if they're not formatted to APFS. And if it's APFS, we're stuck with this on-demand nonsense. Great work, thanks.

 

A terrible idea and a massive waste of time!

Brass Contributor

This is a disastrous update. I used to spend the whole weekend on dealing with it in December 2021 and solving it by opting out Insider and re-installation. The new Files On-Demand experience is broken, syncing is broken, speed is slow, ... where is any advantage?

 

Claim:

This means a better user experience, better application compatibility, and better reliability.

 

Reality:

This means a worse user experience, worse application compatibility, and worse reliability.

Copper Contributor

This update clearly isn't working properly.  When my team creates files now I see a .pkgf extension (which this article says should be removed by OneDrive - see below) so they can't be opened by me. However that doesn't happen when I create files.  Even if I remove the pkgf extension the file can't be opened by me. Our workflow was working fine until recently.

 

Screenshot 2022-01-20 at 06.43.02.png

 

We've also noticed parts of synched package files just going missing in the last couple of days. For example pieces of audio synchronised as part of a video package are gone. We are APFS but all on MacOS 12.0.1. and we all use 'Files on Demand'. This update has already cost me hours of troubleshooting and rerecording missing content.

 

UPDATE - TLDR - Make sure all Macs have the same version of the OneDrive app and are on Monterey 12.1. Recreate any Mac login profiles that you can't get to sync properly. 

I updated the Macs to Monterey 12.1. I noticed that there was a mix of standalone and app stored versions of OneDrive. I switched everyone to the app store version 21.245.1128 of OneDrive. One Mac that had the standalone version needed a new profile, because trying to resync files after reinstalling OneDrive didn't work. It would let me choose a location to sync to. I deleted all the OneDrive related files I could find in /library, removed OneDrive and restarted. When I tried to sync OneDrive or SharePoint folders, I couldn't choose a sync location, and was redirected to a 'group containers' folder and couldn't use the users profile folder.

 

I had to workaround this by creating a new profile, logging in and setting up OneDrive from scratch. Working sweetly now and that Mac is not synching .pkgf files any more either. 

Brass Contributor

@MattMunslow

Based on your screenshot and version (supposed to be 21.245.1128), yours is not the future version like what this post describes.

Copper Contributor

@Bustycat  You're right it is 21.245.1128 - thanks. I've corrected that version number in my original post.

I downloaded that version from the app store today, so MS obvs haven't updated the app store with the 'future' version. I was wondering what was going on because I noticed 'files on demand' could still be switch off in 21.245.1128. That explains it.

 

One of my Macs had the standalone version with the Insider updates ticked (version was 22.xxx.blah). That was the one that was causing all the trouble. Now they are all on 21.245.1128 life is good again.

Brass Contributor

@MattMunslow Your post saved a lot of problems, actually. I was trying to downgrade my version of OneDrive to keep the option of turning off 'files on demand', but all the versions I got from the MS website already had this option removed.

 

As of today, the version available on the Mac App Store still has the option of disabling 'files on demand'. So that's the problem fixed for me, for now. 

 

Am I right in thinking that an App Store-installed app doesn't update automatically, but rather through the App Store's updating system? This would be a great way to make sure I'm not accidentally updated!

Copper Contributor

@danielgrieff 

Yes it'll updated be updated by the app store. So I guess you could switch automatic app store updates off to manually control what version of you had.

Brass Contributor

What are the implications on local backups?

How can we ensure we are keeping a local backup fully updated at all times?

Brass Contributor
  • Devices will migrate automatically to the new Files On-Demand as soon as they receive a macOS update. You cannot delay this update without also delaying an update to macOS.

 

The way I am reading this is that you will no longer able to deactivate Files on Demand  once you update MacOS to 12.3 regardless of whether you have the latest version of OneDrive or not. I suspect the code is already on our Macs.

Brass Contributor

@Wando64 I'm not sure this is the case, I'm using MacOs 12.1, not 12.3 and it was the latest version of OneDrive (as downloaded from the office 365 website) that enforced files on demand.

 

Version 21.245.1128, from the mac app store, allows disabling of files on demand. 

Brass Contributor

Maybe I am overthinking it, but as I understand it, there will be two key locations:
- the Sync Root, which contains the links (links, not actual files) to the files stored on OneDrive, and
- the Cache Path, which will contain the actual files, but only if currently downloaded or if Pinned.

The Sync Root can only be on the main drive.
The Cache Path can be on any drive, however "The cache path folder is hidden by default. Users should not modify this folder or its contents".

So, what happens exactly when I move a new file into the OneDrive folder in the Sync Root?
Yes, it will get uploaded to One Drive, but...
Will the data be immediately moved from the Synch Root to the Cache Path?
Indeed, will it even pass via the Sync Root in the first place?

Also, is the data contained in the Cache Path usable (i.e. are the files accessible) without the OneDrive client?
e.g. Should there be a problem with the OneDrive client app and I need to turn it off, OR If I decide I don't want to use OneDrive anymore
At that point would I still be able to access the data? How do I convert it back to normal "files in a folder" (so to speak?) from the current situation of an index pointing to a hidden cache?
How do I back up my data?

...and many more questions

Iron Contributor

@Wando64 - I guess it will be linked with symlinks, so you can use it even without Onedrive IF (!!!) they are marked as pinned.

If not, then you need an Internet connection (and maybe onedrive running).

 

Worst change ever.

 

Really, I would love to hear from someone from Microsoft, why they made this decission. I would love to understand because for me it does not make ANY sense to take away that option. None at all. At least not from a user perspective.

 

 

Brass Contributor

This update just forced itself on my Mac.  I had "Turn on Files-On-Demand" disabled.  This update turned it back on, deleted my local files, and downloaded on-line files only. Now the new software does not have the ability to turn off Files-On-Demand.  Even worse, the new software has removed all direct access to files/folders.

 

This change just made it impossible to move files or folders out of OneDrive.  Even if files/folders are marked as "Always Keep on this Device" they are not accessible outside of an application.  There is no way in Finder to copy any of your folders or files and drag them out.  After receiving the new downgrade, I marked all files and folders as "Always keep on my device."  After several hours of downloading, OneDrive stated they were downloaded - however, if you click on any file or folder, it still shows 0mb in size and there is no direct file path to access a document.  There is now two icons - one that shows it is downloaded and one that shows it is in the cloud.  All of the downloaded files are in the Cache folder, that is hidden and inaccessible, and the files shown have no data associated with them and show 0kb with no path.

 

Microsoft has essentially locked up all files / folders in a private location and a user has to ask permission to retrieve them by an application MS approves.  

Even using the symlink option as noted above, there is only direct file access to the Sync root and no direct access to any file or folder itself.  AGAIN, there is NO direct access to any file or folder any more except for files that have been recently opened -- even with "Always keep on my device" is selected.  All users are LOCKED out of their own files.

 

There is ZERO user advantage to this upgrade.  This upgrade did one thing, and one thing only, make MS the owner of your documents.  As stated above, your files are cached in a "Hidden Location" and no user can have direct access any longer.  

 

BACK UP your files now.  OneDrive is no longer a backup location.  Any users forfeits the right to own a file using this tool.  

Copper Contributor

This is a terrible update. I'm a photographer who uses Adobe Lightroom libraries on my MacBook Air to keep my RAW photos organized. My Lightroom import folder is in my OneDrive volume which was convenient for when I do import images from my SD Cards it was automatically was backed up to the cloud. Key was that the file's home was on my local drive and I would be able to periodically also backup those files to my external SSD via Lightroom, copy my OneDrive versions to another folder in the cloud that did not sync to my computer, then delete photos on my local drive without risk of also deleting in the cloud. I'd then have two copies of my photos - one on my external SSD and one on OneDrive in the cloud.

 

After this update (I upgraded to MacOS 12.1) all my files seemed to be deleted off my local drive so now I have to redownload them from the cloud in their original folders on my local drive. But once redownloaded, Lightroom can't move the files over to my SSD for some reason. I'm stuck.

 

Microsoft, fix this and give me back control of my data and files. This is a dealbreaker for me.

Brass Contributor

I doubt if the team accepts any feedback from users. The developers just thought it is cool for users and even didn’t test properly. Only casual users feel it’s good and the team feels content with those results.

Copper Contributor

Any one knows how to activate KFM? I don't see any option to do so...

Thanks.

Copper Contributor

It is a catastrophe... had my entire logic Pro plugins library on a synced drive (290GB)... The result of this wonderful new customer experience is that OneDrive deleted all the local files leaving me just with links ... if I do not re-download everything and somehow protect them elsewhere, I suppose it would take about 2 hours for my Logic Pro environment to start..

 

and I am not talking about the GBs of pictures ... and the fact that I will need from now on to think of the files I will need to work on while flying somewhere ....

 

and icing on the cake ... if you un-select a folder to stop syncing it, it seems that it will hide the folder on your mac... Microsoft decided that the master copy of your files is on OneDrive and not on your local device anymore.

 

thank you very much OneDrive...

Copper Contributor

So I agree with all that is said, and I was very agitated by the fact all files were moved off to the Cloud without my permission.  That is a complete and utter failure to communicate by MS.  However, I upgraded to 12.1 on Sunday, turning off OneDrive. Then I restored my OneDrive files to ~/Library/Cloud Storage/OneDrive and turned on OneDrive. Next, I clicked on all of my top-level folders and chose "Always Keep on this Device." So far, everything looks good and is working as expected. I tried to do it without restoring, and it got all tied up, I guess because it was 50GB that had to be downloaded, and I was not patient.  

 

Regarding the Symbolic Link implementation with OneDrive, that should not be an issue. I used Symbolic Links for years with Unix and Mac in particular and Dropbox and never had a problem.  

Brass Contributor

@Mikedvzo if/when you back up your OneDrive files to a local backup solution, such as Time Machine, does everything work as it should? i.e. are the files restored from Time Machine usable?

Copper Contributor

I restored the files from TM and told it to keep the existing directory since the Link existed already. I moved the files, and everything was fine.  If you refer to backing them up now that they are a symbolic link, you will want to navigate to ~/Library/Cloud Storage/OneDrive when restoring.  

Copper Contributor

Moved back to DropBox today... will take me days since more than 400GB where moved offline to OneDrive without my permission.. i need to download back everything and then bye bye.

Brass Contributor

@Wando64 This update broke my Time Machine.

I also tried to move the files off manually and was unable to do so.

 

Essentially, even if "Always available on this device" is checked, the OS still goes through OneDrive to check for the file.  In my case, a few files in a couple directories would not download, so it would not allow OS to move the files to another location. 

I had to find the files by hand and copy everything, except those couple of files to copy them to another location.

 

The same thing happened to Time Machine, Time Machine would never complete because OneDrive would never say the file was completely downloaded.

Brass Contributor

@Mikedvzo The symbolic links are not the issue.  The issue is the links do not take you directly to the actual file.  OneDrive must interact with the OS to provide the information from the "hidden location" and when that fails, as it did with me, everything stops working.

 

If they used Symbolic links, but the location of the files were not hidden, this would work ok.  Hiding the "cache" files essentially takes your files away from you and you no long have direct access or control over them.  This should never, for any reason, be the case. 

Brass Contributor

Is there a user voice or bug/issue report published for this feature? 

This shouldn’t be allowed to continue in its current form. I don’t say this as someone who hates change, but this has ruined my use of OneDrive.

Copper Contributor

@crcochran, I just reread your original post, which is not my observed behavior. I have no issue moving, copying, or getting information for files from the OneDrive location. I tested it many times, and it works the same as it always did. Perhaps I a missing something, but I see no issues since restoring my files from TM and copying them to the new location.  

 

I read the explanation of the Cache Path, and it seems to me to be for record-keeping and supporting the new macOS 12.1 available features: "This path contains a replica of the file tree that the user is syncing. Most of the files in this location are usually dataless and don’t consume disk space."

Brass Contributor

@Mikedvzo You have not had issues because all of your files downloaded successfully.  A couple of mine did not, and once they failed, they would never download.  Any copy, move, download of the entire folder structure would fail because of one file.

 

Even with OneDrive turned off, it prevented the OS from copying files and folders if a bad file was in the folder.

 

I can download the individual file directly from OneDrive, but it is corrupted or otherwise has an issue locally.  The new change assumes everything is working -- when it fails, even for one file, you lose everything.  

 

 

Copper Contributor

Hi Ankita. 

It’s great that you improve your apps for better compatibility with macOS. 

Please, it’d be great if you and the Microsoft team dedicated to macOS, could talk to Apple and ask them to enable TPM2.0 on Intel Macs so the update to Windows 11 could be officially supported. 

It’d be very easy for them to enable it or release a tool to enable it. Thanks. 

Copper Contributor

It's almost like people didn't read the post where it explains WHY this change is happening. Microsoft doesn't have a choice, people! The current way of doing things will be blocked with macOS 12.2! Dropbox is having the same issue, but they haven't addressed it with a workable solution like Microsoft has with OneDrive now using Apple's File Provider API.

 

Also, OneDrive is NOT a backup solution!

Copper Contributor

and by the way it apparently also messes up with Time Machine.... fortunately I could catch up quickly and find the day before the all thing happened to restore from Time Machine instead of redownloading MY files. Lost a few updates I had made since that day but should be ok. After the day the new OneDrive kicked in Time Machine does not seem to show any files from the sync folders.. it keeps on searching forever but goes into a loop.

Another issue is if you work with large files.. if OneDrive does not have the time to upload, I am not sure where is the file anymore.. or at least I could not find it... that's how I lost a few large files that I had amended lately.

Copper Contributor

@crcochran Shell out to Unix and delete the corrupted file with "rm." That always works unless you have SIP-protected files on a managed computer, but those files would not be anyway.  

Copper Contributor

@pierre1410 What gets messed up with TM?  I just checked and I had no issues!  Are you navigating to ~/Library/Cloud Storage/OneDrive

Copper Contributor

@warpedgeoid They do have a choice on the implementation! Deleting all the local files during an upgrade is unacceptable. 

Copper Contributor

@MikedvzoHave you looked in the path stated in the article? It explicitly states that the files are relocated to a new path.

Copper Contributor

This is probably the worse upgrade.

Basically, Spotlight is now broken. Clicking on a document in a spotlight search nearly never work (doing only a search on file name... the file appears but isn't clickable)

I've got a lot of files (maybe around 100.000).

Memory/CPU use seems at least as high as before, if not more.

 

File on demand was already broken, and this is why I used manual synchronization, which was working OK.

Now, we no longer have choice...

Brass Contributor

@warpedgeoidYou haven’t read the article carefully. The old Files On-Demand remains available on macOS 12.2, and 12.3 hasn’t entered the beta stage yet.

Iron Contributor

@warpedgeoid there is always a choice. And if you know more insights, can you explain how this is related force users to "Files on Demand"? 

Brass Contributor

@Kwikk

Dropbox also warned about macOS 12.3 as well. The so-called better Files On-Demand experiences will be inevitable to all cloud storage services.

Copper Contributor

@Mikedvzo  

When I finally get to browsing files in TM.. i get a partial view of folders and files.. some folders are empty; others are simply missing... accessing them indeed needs be done via Library/ cloudstorage ... The sym link does not work from within TM.

 

and sometimes TM gets stuck to "Loading" until you go to the elder backups...

 

If I go back to a date prior to the update everything is fine .. with obviously files updated up to that specific date.

 

I am not really surprised that TM was messed up since the files were simply removed from the HD.. so what is left for TM to backup !!!

Copper Contributor

I want "All files pinned by default" option.

Copper Contributor

Worst change ever. The previous way of sync was far superior and simple to use. I had auto-update enabled and realized that all my files were deleted from my HDD. Selecting "always keep on this device" added a stupid icon NEXT to the download icon and didn't do squat. Reinstalled 2-3times. Removed the computer from my account, nothing fixed this. Thank god you can still download the old app and download all your files locally. This way you can at least make a local backup until I find another provider. Any suggestions to what else we can use on mac for file sync that doesn't use this stupid root/cache system?

Brass Contributor

It's good to have this info but I'm afraid this transition is going to be a problem for many people. I tried the insiders beta and it was such a mess.  My OneDrive  folder is stored on an external drive and this file was basically orphaned by all appearances.  

 

It seems that users will also have issues with anything that's been mapped to OneDrive- like the system Screenshots folders, application folders, etc, that will need to "just work" after any transition.  I don't think the upgrade process handles everything it needs to, and surely doesn't explain to the user what is changing like it needs to.  

 

I hope in the end the new subsystem works better, but dear Microsoft- you need to do better at the change over process.

Co-Authors
Version history
Last update:
‎Feb 24 2022 10:17 AM
Updated by: