NTFS Beta Chat Transcript (July 12, 2006)

First published on TECHNET on Sep 20, 2006

Chat Topic: Join the NTFS file system team to ask questions about NTFS in Windows Vista
Date: Wednesday, July 12, 2006

Wendy [MSFT] (Moderator):
Welcome to today’s chat with the NTFS Team.  We will try to answer as many questions as we can today. Participants should type their questions, select the “Ask the experts” check box, and click “Send.” Those posts will go into a private queue, from which our experts will draft answers and repost questions in the upper window with their answers. (To confirm: if you selected the “Ask the experts” check box when you posted, you don’t need to resubmit.

Wendy [MSFT] (Moderator):
We’ll tell you all about what’s new (and what’s the same) and answer questions relating to Transactional File System (TxF), symbolic links, NTFS interop between Windows XP and Windows Vista, and much more!

Surendra[Msft] (Expert):
Hi I'm Surendra Verma. I'm the development manager for the file-systems group.

Senthil [MSFT] (Expert):
I'm the PM for NTFS. I have been with the NTFS team for just about 6 months now and have been working in the file systems area for the last 4 years.

Wendy [MSFT] (Moderator):
Hi Everyone!  My name is Wendy and I just recently joined the Longhorn/Windows Vista team.  Previously I was part of the R2 and MSN 9.0 Beta Teams.  My job is to see that you BTs get what you need to have the best beta experience ever!  We’ve got lots of great things in the works for you BTs!

Dana [MSFT] (Expert):
Quick bio: I am the program manager for NTFS' transactional technologies in Vista.

Malcolm [MSFT] (Expert):
Hi, my name is Malcolm, and I'm working as a developer on NTFS.

Jill [MSFT] (Expert):
Hi, I'm Jill Zoeller and I'm the Community PM for the Core File Services team.

Dan [MSFT] (Expert):
Hi, my name is Dan.  I am one of the NTFS developers.

Senthil [MSFT] (Expert):
Q: If you guys own this feautre, any work being planned to clean up the NTFS permissions editing UI?
A: What kind of problems are you seeing with this UI?

Andy [msft] (Expert):
Greetings.  I'm Andy Herron and I'm a developer on the NTFS local file system.  I work on both core NTFS and TXF.  Have worked on NT for roughly 14 years.

Jill [MSFT] (Expert):
Q: Will we get a more reliable defrag?
A: Defrag is still a work in progress. We posted an FAQ recently on our blog at https://blogs.technet.com/filecab/articles/440717.aspx to answer some of the common questions.

Malcolm [MSFT] (Expert):
Q: Has there been a change in how ACLs are handled in Vista?  (e.g., in XP - an admin could change permissions even if they didn't have the appropriate access privilages)
A: No.  The specific issue you're asking about is by design - owners have reserved rights on objects in addition to those provided explicitly in the ACL.  That's a requirement of C2 certification, and has been in NT since the beginning.

Malcolm [MSFT] (Expert):
Q: With development of WinFS stopped, is it now up to the NTFS team to come up with solutions for all file requirements for the future?
A: NTFS will continue to develop new storage solutions over time, such as the great work we've done with Vista.  I don't think that means we'll be the only innovators in the storage space, however :)

Malcolm [MSFT] (Expert):
Q: Will Vista make any specific uses of NTFS file streams, file streams seem to be overlooked by most people and to the average user serve little purpose
A: Although not directly related to NTFS - NTFS simply stores stream information as requested by applications - XPSP2 and newer have used streams to store metadata for untrusted downloads, and similar things.  Explorer and IE in Vista continue to use streams for metadata of user files.  WS03 introduced new APIs for enumerating streams.

Wendy [MSFT] (Moderator):
For those just joining us, today’s chat is about NTFS.  To post a question, please type your question, select the “Ask the experts” check box, and click “Send.” That way, we can track which questions we still need to answer.

Dana [MSFT] (Expert):
Q: What do each of you find are the most exciting and groundbreaking changes in NTFS Vista?
A: Some highlights include self-healing, symbolic links and transactions.  I am biased of course but IMHO NTFS supporting transactions is really the most groundbreaking change.  This feature allows you to coordinate file system operations with each other, the registry (also transactional), SQL, MSMQ and any other transactional resource that works with DTC.  Think what this does to your coding style!  You no longer have to figure out what changes were made to a file and back them out if there was an error!

Jill [MSFT] (Expert):
Q: If a user has a laptop and has his My Docements redirected to a USERS share on the server and available offline, and sets encrypt the offline files will they also be encrypted on the share on the server.
A: This is the NTFS team here today. The Offline Files team would love to see your question in the beta newsgroup though! Try asking in the microsoft.beta.longhorn.networking.filesystems newsgroup.

Jill [MSFT] (Expert):
Q: running the defrag when the computer next switches on - is USELESS. what if this is a lap top and i am now on battery. the defrag should NEVER run when on battery. low priority I/O is useless if is still causes seeks to ruin high priorityI/O
A: Thisi s all good feedback for the defrag team, but we're here with the NTFS team today. Please log your feature request for having defrag not run on battery.

Malcolm [MSFT] (Expert):
Q: Are there plans to have NTFS work on external USB devices, or is that the purpose of exFAT?
A: NTFS is supported on removable media currently, and will be supported on media that can be surprise removed in Vista (eg., flash devices.)  It may not be the wisest thing to do, since NTFS will rewrite the same sectors frequently, which is not optimal for flash devices.  The primary intent for NTFS is USB disks.

Surendra[Msft] (Expert):
Q: If a USB external NTSF disk is used with a laptop and a folder is encrypted, can the same user open and read the contents of the folder if it is plugged into his desktop?
A: No, not unless they've exported the encryption key (certificate) along with the data.

Andy [msft] (Expert):
Q: Is there any movement towards placing files more intelligently so they dont get fragged so much?
A: Yes, we've got background defragmentation going on now as a service.  We've also got a prefetcher that based on usage patterns will attempt to fill the cache with what it expects will be asked for next.  From the NTFS side, this all goes on above us.  The only thing we do is try to minimize the number of disk runs allocated to any given file up front at file extention time.  That logic has been refined but not majorly expanded in Vista.

Senthil [MSFT] (Expert):
Q: Should I be able to access a symlinked folder via the network?
A: Yes, remember that the client and the server needs to be Vista. By default only local to local links are turned on. You would need to enable the Local to Remote symlinks through the policy editor.

Surendra[Msft] (Expert):
Q: Will we get a more reliable defrag?
A: Here's some info I got from the defrag team for this:

UX improvements:

Defrag scheduled by default on all volumes – no need to worry about defrag anymore

New streamlined and simplified UI

Defrag engine improvements

More meaningful metrics for free space fragmentation – free scpace fragmentation was rarely > 0% before

By default only do partial defrag which consolidates files only if they are in extents smaller that 64 MB. This allows for defrag of large files even when there isn’t much free space available. It also prevents moving large extents around when it is not necessary.

Improved and more efficient free space consolidation pass

Improved cancellation support -- defrag can exit even if in the middle of defragmenting a really large file (not possible before – caused machines to fail to reboot for patches etc)

Perf/Footprint improvements

Using Low priority I/O and low priority CPU to minimize impact on other activity while running (I can defrag while watching video off of the same spindle)

Dan [MSFT] (Expert):
Q: Are there any specific changes in NTFS that might affect things such as file recovery (using Winhex, Encase or the like?)
A: NTFS in Vista has a self healing feature that may affect file recovery.  Basically upon detection of certain types of corruption, NTFS will repair the corruption in place.  The goal is to restore consistency of the metadata much like chkdsk will except it does not require a lock of the entire volume.  If user does not want it enable by default, it can be disabled on a per volume basis using fsutil.exe.

Dana [MSFT] (Expert):
Q: will ntfs be slowly evolving towards an easy winfs implementation from now on are are the two mutually exclusive?
A: WinFs was initially built on top of Transactional NTFS (TxF) for storing files.  WinFs did a lot for you -- including providing a rich new programming environment.  NTFS will always support existing users and today we are evolving our APIs not providing a whole new environment.  As we go forward I expect that there will be enhancements but to what extent has not yet been planned.  WinFs and NTFS/TxF were never mutually exclusive but complementary.

Malcolm [MSFT] (Expert):
Q: Why are Administrators not given permission to edit security on disk drives by default?
A: This question is probably better to take up with the security folks.  Basically the fear is that a large amount of the Windows userbase runs as Administrators, and providing too much access allows for viruses, worms, and badly behaved applications to cause problems.  UAC is intended to overcome this and require elevation before writing to certain areas in the filesystem.  The restriction on Administrators on the root of a volume comes from that.  You can, however, take ownership of the root and manipulate ACLs afterwards.

Senthil [MSFT] (Expert):
Q: Are the deadly slow problems, accessing files and drive in Explorer, NTFS-related or Explorer-related?
A: I think the performance related issues are being looked at from all angles. The NTFS team is aggresively working to identify and solve the performance related issues.

Surendra[Msft] (Expert):
Q: With NTFS supporting transactions, does that mean no more registry corruption? =)
A: Registry doesn't use the NTFS transaction implementation. It does however expose transactional behavior like NTFS does.

More directly to the point, transactions can help with incosistent updates, but if you're talking about corruption due to bad hardware etc, transactions can't help with that.

Jill [MSFT] (Expert):
Q: My VISTA build 5456 has a default scheduled task defrag.exe -c that runs once a week. I have a C; D: and G: NTFS disk and a E: DVD-RW. Is that only defragging the c drive and if so do I need to change the line to defrag.exe -cdg or add three entries.
A: Great question--I believe that defrag covers all volumes by default (not counting your DVD drive). This is a good question for the newsgroups since the NTFS team here today doesn't own defrag (it's a separate team).

Surendra[Msft] (Expert):
Q: Are there any changes to the NTFS Change Journal?
A: The change journal is now on for all machines in Vista. It's turned on by the services (like search indexer).

Dan [MSFT] (Expert):
Q: What does self-healing mean? Can you tell us (me) more about that? :)
A: Self healing is a new feature in Vista.  Basically upon detection of certain types of on disk corruption, NTFS will repair the corruption in place.  The goal is to restore consistency of the metadata much like chkdsk will except it does not require a lock of the entire volume.  If user does not want it enable by default, it can be disabled on a per volume basis using fsutil.exe.

Surendra[Msft] (Expert):
Q: Does the NTFS from Vista influenze an NTFS formatted disk example : "NTFS formatted by XP data drive on local machine", or will there be no change to the NTFS system be inflicted on that drive?
A: NTFS format is unchanged from XP.

Andy [msft] (Expert):
Q: I'm really interested in how NTFS works. Is there any (technical) documentation available?
A: "Inside Windows 2000" has a pretty good section on NTFS.  Haven't checked Inside Windows Server 2003 but I believe it may provide an updated section.  Rajeev Nagar, Windows NT File System Internals, OSR Press is also good, though a bit dated.

Malcolm [MSFT] (Expert):
Q: Question for the Queue: Will each FSO object still take up approximately 1K in MFT usage? Meaning will an NTFS volume with 1 million objects still be 1GB in size?
A: Yes.  The size per file in the MFT is 1k under most conditions, so each object will require a minimum of 1k of MFT space.

Dana [MSFT] (Expert):
Q: Is the transactional file system something that developers can utilize via an API, or is this an internal  system service?
A: Via APIs.  There are enhancements to the filter manager for filters; new Win32 api's for applications.  We are working on managed code api's as well.  TxF (Transactional NTFS) is NTFS -- not a new service.  It is built on top of new kernel technology (Kernel Transaction Manager) there are no services started for TxF.  (TxF does its recovery at boot before any services even start.)

Malcolm [MSFT] (Expert):
Q: Is there a really easy way to make a symlink?
A: Open CMD, and type 'mklink'.  This tool will allow you to create junctions, hardlinks and symbolic links.

Wendy [MSFT] (Moderator):
For those just joining us, today’s chat is about NTFS.  To post a question, please type your question, select the “Ask the experts” check box, and click “Send.” That way, we can track which questions we still need to answer.

Jill [MSFT] (Expert):
Q: Is there a help option with defrag for command line switched I have tried -? /? -help and /help and it appears to just close.
A: Please log a bug on this. I think there should be command-line help for all command-line tools.

Andy [msft] (Expert):
Q: What is exFAT?  What is DTC?
A: exFAT is a new file system based on FAT that allows for larger volumes.  I