On 05/03/2021 10:45, Andrey Semashev via Boost wrote:
On Windows, files have an intermediate state of being-deleted. So what you can do is:
1. Have an async thread keep a pool of file handles full. You open these handles to randomly named files, and once opened, you mark them for deletion.
2. When you need to "create" a new file, you unmark a handle for deletion, and rename it to what you need. All this can be done with SetFileInformationByHandle.
3. If there is sudden process exit, or sudden power loss, all files marked for deletion are actually deleted. So no need to clean caches on a timer or anything like that.
If you are on a new enough Windows 10, you can make the entire file entry vanish when you mark it for deletion, rather like O_TMPFILE on Linux. You can then make it reappear on an as-needed basis. On older Windows, the file entry will still appear in directory listings, but will be unopenable.
An interesting idea, but I'd prefer if the backend behaved roughly the same on POSIX and Windows. Besides, I'd prefer to avoid spawning threads in the backend, as Boost.Log currently only creates threads for asynchronous logging and never else. Even then the user can provide his own threads.
I appreciate that, but the two systems are very different. Using the same design patterns for both, imposing one design pattern on a platform on which it performs very badly, personally speaking I don't find that a high Quality of Implementation. i.e. When in Rome, do as the Romans do.
Keeping open files is not only a concern from resource usage perspective. It may affect user's experience, as e.g. on Windows he won't be able to remove the open files or the containing directory.
You can delete or rename open files on Windows just fine. It's just std::ofstream opens files in a way where you can't.
But the containing directory remains locked, doesn't it?
Locked in the sense that it cannot be renamed or deleted, yes. The simple workaround is to rename the file out of its directory before you delete it, then the containing directory behaves like on POSIX. Recent Windows 10 implement "POSIX delete" by renaming the file into a magic hidden NTFS folder in the root called \$Extend\$Deleted, and marking it as in-process-of-being-deleted. That same technique works right back to NT 3.5.
Another suggestion is that any decent Windows AV will not scan files incapable of holding executable code for binary viruses. std::ofstream opens files capable of holding executable code, so everything written to them must be scanned deeply. If you instead create a file without executability privileges, the AV should only scan for a small subset of potential malcontent.
This might be useful to do. How do you specify that a file is not executable on Windows? There are dwDesiredAccess flags for CreateFile[1], but my understanding is that they describe the access granted through the returned handle and not permissions of the actual file.
[1]: https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-create...
A file can be capable of execution according to its ACL. A file handle can be capable of executing code according to its privileges (dwDesiredAccess contains FILE_GENERIC_EXECUTE). You can create a file with an ACL not permitting execution using lpSecurityAttributes, or use hTemplateFile to clone a security descriptor. Or, you can temporarily adjust the current thread's impersonation token such that specifying NULL to CreateFile means files created by that thread aren't executable. The former is best when you can adjust source code, the latter is best when you are calling third party code and you want it to create different permissioned files. Note that I don't actually know if the Windows default AV is clever enough to not deep scan files without Execute ability. I do know it specially skips files signed by a whitelist of Microsoft approved signing authorities, and it was written by Microsoft themselves who suffer the most from it, so your chances are good. (Incidentally, to anyone not aware yet, you can exclude whole directory trees from Microsoft Defender realtime analysis. Just add your MSVC directory, your WSL directory (this makes a HUGE difference to WSL), all your build directories, all tooling and source directories, and so on. Build times approximately halve) Niall