BrowserMetrics

[alexfreu]

There is a folder %localappdata%\Microsoft\Edge Dev\User Data\BrowserMetrics with thousands of 4 MB files like BrowserMetrics-5D3A8F5A-1AF0.pma that fill up my precious SSD disk space.

Can I remove these or prevent or limit the creation? What is the purpose of these files?

I'm currently at Version 79.0.309.5 (Offizielles Build) dev (64-Bit)

I think it would be a good idea to make them sparse files on NTFS volumes or compress the folder by default. In my case compressing the folder reduced the size from 8.55 GB to 462 MB.

Solution:

I disabled the creation of the files by setting the following registry entry:

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Edge]
"MetricsReportingEnabled"=dword:00000000

I checked the policy with edge://policy

All existing files were deleted after a restart of Edge.

See https://docs.microsoft.com/en-us/DeployEdge/microsoft-edge-policies#metricsreportingenabled

[#2]

Happened again last night while leaving the browser open overnight, over 76GB of these files 

[#3]

Yes, I'm up to 14 GB since 3/20/21; almost 3 GB/day. Some of you seem to be having much higher usage by Browser Metrics. I'm hoping for progress by the Microsoft Edge team soon!

[#4]

Hey folks, 

Quick update here from the team - they're still investigating but have requested some additional information.  Even if you've submitted already, if you don't mind submitting again with the new info we'd really appreciate it! 

1. On edge://support, do you see a Client Id? What is it?
2. The list of PMA files, with time stamps. Mainly the count of files and oldest date would be interesting.
3. Run these from a cmd prompt and share the results:
1. reg query "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion" /v CurrentBuildNumber
2. reg query HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection
3. reg query HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\EdgeUpdate\ClientState /s
4. reg query HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\EdgeUpdate\ClientStateMedium /s
4. edge://local-state/
5. edge://version/?show-variations-cmd

If you could head to edge://policy/ and submit feedback (Alt + Shift + I), check to include the screenshot as well as diagnostic data, attach the information requested above, and add the string "TC0317AR-1010LUO-TMEP908" so we can link it to this issue. If you'd like the team to be able to reach out directly, please check to include your email as well.

[#5]

@lexcyn (and @dave260) Have you tried turning on automatic compression for the BrowserMetrics folder, as I detailed above? (If so, do you have anything notable to report?)

Anyone else?

(I haven’t personally seen any ill effects yet from enabling automatic compression.)

PSA: if you can fulfill the data request from @Alexandra-R, please do that first! The sooner the team gets data the sooner they can figure this out.

 - Jim

----

Aside: With the explosion (in both usage and number) of plain-text configuration, log, and analytics files in recent years, I wonder if NTFS compression should be used by default in many more places...

[Even with medium-speed SSDs, accessing a compressed file might be faster (fewer IOPS as well), let alone for those poor souls still using HDDs for their boot disk.

I’m not well-versed to know for sure, but memory-mapping files that a program needs extensive access to may contribute to the current disk performance challenges across various ‘modern’ operating systems (in theory and benchmarking, maybe this shouldn’t be a problem due to something like ‘intelligent caching’ in disk and VM systems, but in real-world multi-threaded multi-tasking usage, might not even the most ‘intelligent’ cache prediction system fail to determine user intent in order to optimize performance?).

It shouldn’t be that surprising that browsers such as Chromium-based Edge, in some ways nearly mini-OSs themselves, might end up being a canary for this sort of systemic issue. They are very complex and usage can vary widely by user and time - this isn’t just ‘winword.exe accessing hugereport.doc’ anymore!]

... and all that is aside from the performance impacts of having thousands or more files per process - each one places load on systems and filesystems upon reading (e.g. for each and every file opened, even read-only: verifying access rights, recording ‘atime’ access time, and performing other security/audit logging), writing (most of those same activites, plus updating modification times and file system journals, and perhaps SSD ‘write amplification’), and for purposes of indexing, backup, and security (Windows Defender Real-Time Protection). Many are likely to be left behind when new builds are installed. Sure, some of those things can be disabled on a per-file, directory, or process basis, but... without giving admin rights to the application?

Even if developers of most programs know how to do this, even if allowed to by system security and domain group policy, why would they remember to do this on every release? If using a cross-platform framework, they may not even _be aware_ of these issues, or have an API to address them.

[#6]

Thanks - done as requested.

I just cleared out the files a couple of days ago and already up to over 12k new files, all spaced 1 day apart. Added all the details into a txt file in the upload!

[#7]

Jim, yes I did try setting the BrowserMetrics folder to auto compress. That seemed to help initially, but today it is up to 2400 files, 9.6 GB (4K/file). Date-range is 3/28-3/31. Interesting to note that uncompressed files were same size in properties - Size of File and Size on disk (4.00 MB). As compressed, there is a slight difference in Size of File (4.00 MB) vs Size on Disk (3.94 MB). So compression is doing very little it seems. Initially, compression was very good, but I'm wondering if upgrading to latest version of Edge changed compression effectiveness?
Latest Edge version updated after the first compression trial. Now 89.0.774.63 (Official build) (64-bit). Previous version was 89.0.731.xx as I recall..
@Alexandra-R My apologies for not submitting the info for your request. I will hopefully get that to you today!
- Dave

[#8]

@dave260,

Thanks for the details - interesting. Based on those comments and my own observations, it appears that:

• a new empty 4MB file is created as often as about 10 minutes (but not as a ‘sparse’ file) in the BrowserMetrics folder
  • this _could_ happen more often if you have multiple tabs or windows open
• (depending on browser usage? sleeping tabs?) the file fills up during that time with data that is either pre-compressed or otherwise not well-compressed by the LZX algorithm
• the fact that my files were mostly empty / highly compressible _could_ be related to my having optional diagnostics disabled ... maybe the files are allocated but the policy correctly prevents writing much data to them
• my files may have been mostly empty if my browser was open but not being interacted with during those times (and/or because most/all tabs were ‘sleeping’)
• in some cases those files aren’t deleted after a period (48 hours? after upload?)

Of note, I checked the BrowserMetrics folders for my installations of the Dev and Canary branches and didn’t notice excess files (but I haven’t used those branches much in the past week).

• Have you tried the Beta, Dev, or Canary branch/ring versions of Edge*?
  • If so, did you notice this issue? (if you answer either of those, please include the version number and branch/ring name)
• If you have any non-standard networking situation (network firewall or custom DNS) that could be blocking analytics uploads, that’s probably worth mentioning.
• Have you noticed BrowserMetrics files being created more often than about every 10 minutes?
• Since Edge runs some/all tabs separately (i.e. thread or process per a set of tabs and/or windows, and _maybe_ also for certain javascript / 3rd party website tracking code), perhaps some of those processes are crashing (even invisibly to the user) and in that case never getting the chance to clean up their mess.
  • Have you noticed any sites you have been using frequently crash / auto-reload?
  • Are you using the 'sleeping tabs' feature of Edge?

- Jim

* (if you haven’t, they seem to work nicely with synced Edge profiles, so it’s fairly easy to give them a try!)

[P.S. Even if this is an intermittent problem, or it is identified and fixed soon, perhaps to proactively identify _future_ issues, something as complex and resource-intensive as a modern web browser should be programmed to occasionally police its entire profile/temporary/etc. directories to identify abnormally large or old data sets and either automatically clean them up or (at the very least) report the problem to the user and developer.]

[#9]

@lexcynHow far apart are the dates/times on your files? [Unless my quick math is wrong, 12,000 files over 2 days (48 hours) would be 250/hour, or a new file every 15 seconds or less. Even if over 4 days (96 hours), that is 125 files/hour, still over 2 new files _per minute_.]

@dave260 Here's probably why the NTFS file compression isn't working for you:

• When NTFS compression is enabled on a file:
  • The file is broken up into small sections
  • Every time one of those sections is written to,
    • The data for that section (including any changes being saved) are compressed in RAM.
      • If the compressed size is smaller than the size of the uncompressed data, the compressed section is written to disk.
      • If the compressed size is about the same or larger than the uncompressed data, that section is written uncompressed
• What I think Edge is doing:
  • Creates a new empty 4MB file
  • 'Memory-maps' that file to an area of memory so that Edge can write to it directly
  • Each time it has a drop of 'metrics' data, it stores it in RAM
    • Since the file is 'memory-mapped', the system (Windows' virtual memory subsystem and/or the CPU's internal Memory Management Unit) automatically writes the changed data to disk, some small number of bytes at a time
  • Since only a tiny bit of data is changing at any one moment, NTFS compression doesn't bother with it (guesses: because it's either too small, because by itself it doesn't compress well, or because it never really works for memory-mapped files)

If you were to turn off compression for your entire BrowserMetrics folder and its contents, and them immediately turned it back on, I think you'd find that all existing files would be greatly compressed, but new files would not. So much for that idea!

[#10]

Usually around 24h apart from first to last. I just checked and there are another 12004 files in there that I just removed. Can't wait until they fix this :)

[#11]

Okay, here’s a temporary fix* to try if you need the disk space before this gets fixed (I’m not at my computer and so haven’t tested it yet):

1. Close Edge
2. Delete existing files from the “BrowserMetrics” folder
3. Open the “Properties” window for the “BrowserMetrics” folder, and remove all permissions for your username (optionally except for “Read” permission)
4. Reopen Edge, and check for the results (after say, 10 minutes and then a few hours later).

If that doesn’t work, instead of step 3 above, try:

3a. Delete the BrowserMetrics folder itself
3b. Create a new empty text file called “BrowserMetrics” in the place of the deleted directory
3c. Make sure the new file does not have a file extension (e.g. “.txt”)

Hopefully that will block Edge from creating new metrics files there.

If you don’t need the disk space right now, though, please don’t do that yet so you can continue helping with the troubleshooting.

- Jim

* Background: the base code from the Chromium project is set to fail gracefully and discard the temporary data if it can’t write to that directory

[#12]

@Vovan / Edge team:

The site (pomotodo) that you mention uses something called 'web workers' (the Chromium 'Blink' rendering engine apparently calls them 'Blink Workers').

In my (very limited) understanding, this causes a piece of JavaScript to run on a different thread than the one controlling the display of the page to which it belongs, allowing it to run on a different actual (or virtual) CPU.

If each of these new threads (as many as one or more every time you switch pages?) causes Edge to create a new file in BrowserMetrics (whether intentionally or not), maybe they are crashing or quitting before it can clean up the excess files.

Metrics files from different files are supposed to be automatically collected into a single file, but perhaps that part of the code (e.g. subprocess_metrics_provider.cc, or something that calls / is called by a function or method there or elsewhere) is malfunctioning.

Many (most?) sites these days use 'web workers', but the inconsistent (across the Edge user base) nature of this problem suggests that if they are indeed part of what causes this, something in the JavaScript on certain sites is triggering this (the browser should behave even in the presence of horrible / buggy JavaScript, of course).

- Jim

[#13]

Well Jim, I just wrote a really good response to your really good post and deleted it accidentally! So briefly, great explanation on how the Browser Metric files work as far as size and frequency. I do use sleep tabs (Tab Session Master). I often have a lot of tabs open. Edge has done pretty well until I ran into high CP usage with Windows Explorer after a few days (25-40%) and Spot Verifier (svsvcs) - 25-30% and some others that are not easily stopped, so I end up rebooting Edge or the computer.
No unusual firewall or DNS setup.
I haven't closely analyzed how often BM files are created. Basically I had 2400/72 hours, so that is about 24/hour or 1 per 2 minutes? I will try some of your ideas about the BM folder and let you know if anything really worked to control the rapid buildup. Actually the BM files, for me, are not nearly the problem that Windows Explorer and other system services are causing with High CPU. Once today I had 100% CPU and every program was closed except System Services that could not be manually stopped. I like W10 and Edge, but these challenges cause me to occasionally look my previous standby, Firefox!
Thanks for your input. If I missed responding to a specific question you had, just let me know.
-- Dave

[#14]

Thanks Jim for the update and details! I will attempt to try your ideas, including the other versions of Edge in your other post above. I like your reasoning about why the auto compress may not compress as expected. Brilliant! I didn't know that. Since I had good results with compression at first, I was surprised when I saw almost no compression today. Other comments above in your later post.....
-- Dave

[#15]

Note: this reply likely doesn’t directly relate to the BrowserMetrics issue discussed in the rest of this thread

@dave260 : To troubleshoot your CPU usage issue, especially if the ‘Details’ and ‘Services’ tabs of ‘Task Manager’ don’t lead you to the solution, you can always try Microsoft’s free ‘Process Explorer’.

(It’s got a bit of a learning curve, but one should be able to pick up the basics in 5-10 minutes - I’m sure there are gobs of tutorial videos and articles online in addition to the official docs.)

- Jim

P.S. If you’re at all like me, it’s easy to confuse that tool with the similarity named ‘Process Monitor’ - this later one instead monitors and records access by processes to the registry/files/network/etc. It can filter by process or access type (e.g. read vs write vs checking if a file exists), either in real-time or after stopping a recording. I normally use this to figure out why something is slamming the disk and/or network. It could potentially help in your case, too, by showing exactly what a busy process is doing to eat up all that time. (Expect to spend up to maybe 20 minutes figuring out how to use this tool - hint: <CTRL-e>, <CTRL-a>, and <CTRL-x> will be your friends here.) If there’s not much shown in Process Monitor, your app is probably just doing tons of math! ;)

[#16]

@Alexandra-R
Finally got it done, with the Project String this time!
-- Dave

[#17]

Thanks for the info on Process Explorer. I used that a long time ago, so I'll give that a shot again. I also downloaded and installed Edge Beta. I'll try the more buggy development versions later. So far Browser Metric has one file in it after 15 minutes!
-- Dave

[#18]

@Alexandra-R (and Edge team devs):

Other things of note that I've recently noticed (or heard from others):

Spoiler

1. Edge seems to clean up these files upon launch. It knows how to do that.
2. Edge behaves differently depending on version
   1. For me, it occurs in:
      1. Edge Stable: Version 89.0.774.63 (Official build) (64-bit)
      2. Edge Canary: Version 91.0.837.0 (Official build) (64-bit)
   2. It does not occur in:
      1. Edge Beta: Version 90.0.818.22 (Official build) beta (64-bit)
   3. Therefore, it seems there is either
      1. a regression between 90.0.918 and 91.0.837, or
      2. a difference in 'feature flag' or other configuration
   4. I did notice that the Beta version ran with the following command line argument that was absent from the Stable version
      1. "--monitor-self"
3. At first I thought Edge might be losing a 'race condition' with Windows Defender Real Time Protection (RTP); that is, Defender locks the file and Edge can't delete it  (see detail [1] below)
4. That seems unlikely, though, since the Beta ring works fine with Defender
5. Instead, it appears that the Stable version (and possibly the Canary one as well) aren't even _trying_ to delete the old files until after the next time Edge is completely closed and re-opened (... which for some people can be hours to weeks!) (see detail [2] below)
6. Perhaps unrelated, there was a mention of this in one of the file access traces - perhaps it is another lead for the devs:

   1. msedge.dll - RelaunchChromeBrowserWithNewCommandLineIfNeeded

 

 

To recap this entire discussion, it seems there are two (separate but related) issues that need to be fixed:

1. Edge is keeping more than one BrowserMetrics file around, and for far too long
   1. As I understand it, having more than one file in the BrowserMetrics directory is designed to only ever be a short-term (i.e. 30-60 minutes max) occurrence
2. For some combination of builds/users/conditions, Edge is creating these new files at a very high rate (6-300 files / hour).
   1. Even if the previous issue is corrected, this one is arguably more important and will have more impact on the overall UX of Edge.
   2. Whether or not they are full of data, that can result in a ton of small disk accesses.
      1. Pity those unfortunate students using low-end laptops* with spinning HDDs!
         1. Memory-mapped files like this sometimes get written to very often, since the data is saved in very tiny chunks.
         2. Based on some of the reports from above, one can imagine the poor little hard drive heads bouncing back-and-forth nearly constantly, impacting system responsiveness and the user's overall opinion of Edge
            1. An average laptop HDD may only get 60 operations, likely fewer, each second, for the entire operating system and all applications
            2. The fastest server HDDs still only allow a maximum of about 200 operations / second
         3. As I have previously mentioned (ok, mildly ranted), each new file can drive activity from filesystem, antivirus/Defender, Windows Auditing, disk indexing, and potentially other processes. To the extent that most of these are reads, their biggest negative impact relates to HDDs.
         4. * some customers may even have Windows Virtual Desktop instances backed by HDDs, driven by inertia or Azure budgets
      2. This causes unnecessary wear on consumer SSDs as well.
         1. The primary concern here is with the frequent small disk writes; read cost / wear is fairly insignificant
         2. (I believe Chromium on Android turns this feature off altogether, choosing instead to just keep the data in RAM until it needs to be uploaded.)

Thank you for all the attention you and your team are giving to this! 

Happy hunting,

  - Jim

---

Detail [1] from item (3) in above list:

Spoiler

1. Edge creates the 4.0 MiB PMA (Persistent Memory Access) file named "BrowserMetrics-spare.pma" in the "%LocalAppData%\Microsoft\Edge\User Data" directory.
2. Edge moves that file to the "%LocalAppData%\Microsoft\Edge\User Data\BrowserMetrics" directory and renames it.
3. Edge loads that re-named file as a memory-mapped file
   
   1. After some time, the 'spare' file from step 1 is re-created
4. If Windows Defender is running,
   1. writes to this file are sometimes / always accessed filtered through the Defender (MsMpEng.exe) process (Synchronous Paging I/O)
5. If Windows Defender ATP (Advanced Threat Protection) is running,
   1. that process (MsSense.exe) requests an 'opportunistic lock (oplock)' [FSCTL_REQUEST_OPLOCK IOCTL (winioctl.h)] while it is scanning
   2. Before that scan is complete (and therefore the oplock is released), Edge (msedge.exe) closes the file

Detail [2] from item (5) in above list:

Spoiler

Beta - Version 90.0.818.22 (Official build) beta (64-bit)

  --> Normal (working) process, that prevents build-up of "BrowserMetrics-hhhhhhhh-hhhhh.pma" files:

  CreateFile                                              Desired Access: Generic Read/Write, Disposition: OpenIf, 
                                                            Options: Synchronous IO Non-Alert, Non-Directory File, 
                                                            Attributes: n/a, ShareMode: Read, Write, Delete, 
                                                            AllocationSize: 0, OpenResult: Created
  CreateFileMapping                                       SyncType: SyncTypeCreateSection, 
                                                            PageProtection: PAGE_EXECUTE_READWRITE
                    Result: FILE LOCKED WITH WRITERS
  QueryStandardInformationFile                            AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, 
                                                            DeletePending: False, Directory: False
  SetEndOfFileInformationFile                             EndOfFile: 4,194,304
  CreateFileMapping                                       SyncType: SyncTypeOther
  ReadFile                                                Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, 
                                                            Synchronous Paging I/O, Priority: Normal
	...
  CreateFile                    IRP_MJ_CREATE
  QueryAttributeTagFile         IRP_MJ_QUERY_INFORMATION  Type: QueryAttributeTagFile, Attributes: AC, ReparseTag: 0x0
  SetDispositionInformationEx   IRP_MJ_SET_INFORMATION    FILE_DISPOSITION_DELETE, FILE_DISPOSITION_POSIX_SEMANTICS, 
                                                            FILE_DISPOSITION_FORCE_IMAGE_SECTION_CHECK
                                IRP_MJ_CLEANUP
  CloseFile                     IRP_MJ_CLOSE

When running the Stable ring version of Edge, "SetDispositionInformationEx" never appears and the files persist.

[#19]

@JimGrisham I haven't enabled compression because I am on a corporate device where I do not have that ability - besides, the root cause should be fixed so I will leave as is.

[#20]

Thanks Jim for all the updates and spoiler alerts! Good information.
I will quickly add at this point, I am using Edge Beta now. It seems to work well, but I haven't tested every possible issue. Interesting that this is the third day I have used Beta, and there is only ONE file in Browser Metrics. This is the third day I have been using it. On the other hand, Stable Edge has 854 BrowserMetric entries for 3/31-4/1 minimal use.
-- Dave