On Sun, May 10, 2020 at 6:02 PM Niall Douglas
So this turned into a weekend long thing to figure out, and I thank you for it, as we have a whole bunch of code here which assumes things which it turns out are out of date on recent Windows. FYI I had to break out the Windows Performance Recorder to figure out what the hell was going on in the Windows kernel, more on that shortly.
Quick results for enumerating all of C:\ on my machine, warm cache:
``` G:\boostish\cdglove>x64\Release\llfio.exe test-llfio found 1485232 files. 13.872090s wall, 1.734375s user + 12.109375s system = 13.843750s CPU (99.8%)
G:\boostish\cdglove>x64\Release\win32.exe test-win32 found 1491222 files. 14.713440s wall, 2.328125s user + 12.375000s system = 14.703125s CPU (99.9%)
G:\boostish\cdglove>x64\Release\boost.exe Exception thrown: filesystem::recursive_directory_iterator increment error: The system cannot find the path specified Up until fatal error test-boost_filesystem found 868657 files. 66.506680s wall, 5.187500s user + 61.234375s system = 66.421875s CPU (99.9%)
G:\boostish\cdglove>x64\Release\std.exe Exception thrown: recursive_directory_iterator::operator++: The system cannot find the path specified. Up until fatal error test-std_filesystem found 870341 files. 63.407547s wall, 2.921875s user + 60.375000s system = 63.296875s CPU (99.8%) ```
Can confirm I am seeing the same results as you now. After updating llfio, performance improved massively such that I was within 2x of FindNextFile. Then after making your recommended changes to how I was using llfio, they roughly converged. I'm not convinced that this could replace a high level directory iterator though. llfio is easy to use incorrectly as evidenced by the annotations you needed to add to my code. That's not a criticism of the design, only a commentary that it's sufficiently low level to cause some grief if one isn't very careful. I have a few minor improvement ideas on that which we can discuss off list. I'll likely play with the unfinished work to directory_iterator to see if any improvements can be made. I pushed the updated changes to https://github.com/cdglove/fs-test. I think I also fixed the access error exception you were getting in the boost test. (added pop_on_error to the directory options). Thanks for taking the time to look into all of this! -- chris