On 3/5/21 12:36 PM, Niall Douglas via Boost wrote:
On 04/03/2021 19:34, Andrey Semashev via Boost wrote:
Unfortunately, text_multifile_backend is supposed to open and close file on every log record, as the file name is generated from the log record.
If you maintain a small LRU cache, will it help?
That's what I had in mind, but it will have its own problems.
One of the main use cases for text_multifile_backend is when you want to maintain a separate log for each of the user's business process entity (e.g. a session or a connection of some sort). There can be any number of such entities, and I can't come up with a reasonable limit for a cache in the sink backend. Naturally, the cache will be ineffective if the number of actively logging entities exceeds its capacity. At the same time I don't want to keep any stale open files in the cache.
It seems, I'll have to maintain some sort of a timeout for closing files and cleaning up the cache. But I will only be able to perform the cleanup on log records, which means I'll still have stale open files in cache if there's no logging happening.
You're thinking in terms of POSIX.
Indeed. I'm not using Windows very often, so I might be biased. Frankly, I'm writing code with POSIX in mind by default and add workarounds for Windows, when necessary.
On Windows, files have an intermediate state of being-deleted. So what you can do is:
1. Have an async thread keep a pool of file handles full. You open these handles to randomly named files, and once opened, you mark them for deletion.
2. When you need to "create" a new file, you unmark a handle for deletion, and rename it to what you need. All this can be done with SetFileInformationByHandle.
3. If there is sudden process exit, or sudden power loss, all files marked for deletion are actually deleted. So no need to clean caches on a timer or anything like that.
If you are on a new enough Windows 10, you can make the entire file entry vanish when you mark it for deletion, rather like O_TMPFILE on Linux. You can then make it reappear on an as-needed basis. On older Windows, the file entry will still appear in directory listings, but will be unopenable.
An interesting idea, but I'd prefer if the backend behaved roughly the same on POSIX and Windows. Besides, I'd prefer to avoid spawning threads in the backend, as Boost.Log currently only creates threads for asynchronous logging and never else. Even then the user can provide his own threads.
Keeping open files is not only a concern from resource usage perspective. It may affect user's experience, as e.g. on Windows he won't be able to remove the open files or the containing directory.
You can delete or rename open files on Windows just fine. It's just std::ofstream opens files in a way where you can't.
But the containing directory remains locked, doesn't it?
Another suggestion is that any decent Windows AV will not scan files incapable of holding executable code for binary viruses. std::ofstream opens files capable of holding executable code, so everything written to them must be scanned deeply. If you instead create a file without executability privileges, the AV should only scan for a small subset of potential malcontent.
This might be useful to do. How do you specify that a file is not executable on Windows? There are dwDesiredAccess flags for CreateFile[1], but my understanding is that they describe the access granted through the returned handle and not permissions of the actual file. [1]: https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-create...