On 29 Jul 2015 at 18:09, Gavin Lambert wrote:
On 29/07/2015 14:06, Niall Douglas wrote:
NTFS compressed files act exactly like normal files. Reparse point files do not and require significant additional processing to figure out what kind they are. That's the difference.
From AFIO's perspective, when it does NtQueryDirectoryFile() to fetch metadata about a file entry, it can zero cost learn if an entry is a reparse point by examining FileAttributes for the FILE_ATTRIBUTE_REPARSE_POINT flag. It cannot tell what kind of reparse point file it is without opening the file and asking.
Windows' CreateFile() API is astonishingly slow. To require calling that, then an additional NtQueryDirectoryFile() to fetch the FILE_REPARSE_POINT_INFORMATION metadata and close the handle - which is the fastest way I know of to fetch the reparse point tag code - would impose an enormous performance penalty for all file entries marked with FILE_ATTRIBUTE_REPARSE_POINT.
If it helps, https://msdn.microsoft.com/en-us/library/windows/desktop/aa365511.aspx seems to specify that reparse points provide their tag id in the dwReserved0 field of the WIN32_FIND_DATA structure (I'm not sure how that maps to the native API, but I assume it's somewhere). That should be sufficient to identify the reparse point type.
That does help greatly in fact. I know FindXXXFile doesn't open each file, so somehow or other the Win32 layer is able to fetch the reparse tag type for directory entries purely from the directory handle.
Granted, a single NtQueryDirectoryFile on the whole directory is not enough to get both sets of data, but you should still be able to do it in just two calls per directory (times however many calls are required to fully enumerate the directory, of course).
Presumably you're currently using one of FileBothDirectoryInformation or FileFullDirectoryInformation. You should be able to switch to the "Id" variants (FileIdBothDirectoryInformation or FileIdFullDirectoryInformation) instead (if you're not already using them). This gives you a FileId for each file, along with the other information.
After you've enumerated the entire directory, you can go back and get FileReparsePointInformation for the whole directory, and then match up the FileId against the FileReference to merge the data and get the reparse tag for each file.
(I haven't tested this, so I'm not sure if it gives you an empty tag for files that aren't reparse points, or only lists reparse points. The latter would be nice, as it would be close to zero overhead for directories that do not contain reparse points.)
Unfortunately getting FileReparsePointInformation returns just a single record which is the reparse point for the directory handle being enumerated. It doesn't return reparse tags for directory contents. There is an index of all reparse points on a NTFS volume in a magic NTFS file stream, but that's NTFS specific code, and it requires a file handle to be opened. I'm thinking that as reparse points are really just an overload on EA, maybe the returned EaSize field is magically set to the reparse tag when attributes specify it's a reparse point file? I'd have to experiment to find out. I can't see any other obvious field which would return the reparse tag. EDIT: What a guess I just made!: https://www.osronline.com/showthread.cfm?link=171655. Thanks Gavin, you just solved the problem for AFIO at least.
I appreciate you're saying the cost is worth it, but we're thinking all Boost users here, not just the small minority on Windows Server 2012 with dedup turned on.
I'm not on Server 2012, but this thread caught my attention because I remember encountering a bug that prevented all WinXP clients from accessing deduped files on CIFS shares provided by Server 2012. I think in the end this was a server-side bug related to McAfee and the different protocols used by WinXP vs. Win7, and so clients shouldn't normally be able to see whether files are deduped or not remotely, but I haven't explicitly verified that. If CIFS shares do expose files as dedup reparse points instead of concealing that then it might affect quite a lot of users.
I had understood from the OP that CIFS is exporting the reparse point tag to clients, hence the breakage. The reason, I suspect, that CIFS is being so braindead here is that opening a deduped file is more expensive than usual and clients ought to know. Which is exactly why I am opposed to treating these things as a regular file. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/