The right thing to do would probably be to implement a deferred close where a file lingers around for a second or so, then is closed by a background
Peter Dimov via Boost wrote: thread. If
it's used again within this second, it doesn't need to be closed and reopened.
I'd also be happy with just an option on the backend that requires
opting-in to use an unbounded cache. It would be off by default but could
be turned on just like flushing on the regular file stream backend. Like
Andrey pointed out, if your use case involves an unbounded number of log
files, using a cache is going to have complications and shortfalls. But if
your use case is like mine - using the multifile backend instead of having
to declare a bunch of regular streams and filters - just keeping all of the
files open until program termination is exactly what you want.
Thinking a little bit differently, the multifile backend could allow for a
client-supplied callback, run after each record consume, which tells the
backend whether to close the file or leave it open in an unbounded cache.
Then cache management is completely customizable but also not the backend's
problem. This probably makes a lot of sense for the different use cases too:
- By default, there is no callback. The file is always closed.
- For my use case, the callback always returns don't close the file.
- For the use case with a log file for each connection/business process
entity/etc, the client probably knows when they are done with the log file
for that entity (e.g., the socket is closed). They are then responsible for
the callback saying it is time to close the file when the last log entry
for it is issued
- For some unimagined use case, the client can customize however they want.
The callback could even provide info about the open files, last time used,
etc, if the client wanted to do something super customized.
Thoughts?
On Thu, Mar 4, 2021 at 1:56 PM Peter Dimov via Boost
On 3/4/21 10:04 PM, Peter Dimov via Boost wrote:
Andrey Semashev wrote:
Unfortunately, text_multifile_backend is supposed to open and close file on every log record, as the file name is generated from the log record.
If you maintain a small LRU cache, will it help?
That's what I had in mind, but it will have its own problems.
One of the main use cases for text_multifile_backend is when you want to maintain a separate log for each of the user's business process entity (e.g. a session or a connection of some sort). There can be any number of such entities, and I can't come up with a reasonable limit for a cache in the sink backend. Naturally, the cache will be ineffective if the number of actively logging entities exceeds its capacity. At the same time I don't want to keep any stale open files in the cache.
It seems, I'll have to maintain some sort of a timeout for closing files and cleaning up the cache. But I will only be able to perform the cleanup on log records, which means I'll still have stale open files in cache if
happening.
Keeping open files is not only a concern from resource usage
Andrey Semashev wrote: there's no logging perspective. It
may affect user's experience, as e.g. on Windows he won't be able to remove the open files or the containing directory. Deleting an open file on POSIX systems will succeed, but it will also mean some subsequent log records can be lost as they will be written to the deleted file instead of the new one.
It's definitely not trivial.
The right thing to do would probably be to implement a deferred close where a file lingers around for a second or so, then is closed by a background thread. If it's used again within this second, it doesn't need to be closed and reopened.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost