On 15:45 Mon 31 Aug , Niall Douglas wrote:
On 31 Aug 2015 at 2:33, Andreas Schäfer wrote:
1. Should Boost.AFIO be accepted into Boost? Please state all conditions for acceptance explicity.
I vote no. For acceptance I'd expect a library that outperforms "naive" OpenMP code.
I will place a wager with you for €100 right now that nobody will EVER implement an asynchronous file i/o library on Linux < 4.0 which beats OpenMP.
I run Linux 4.1.5.
AFIO is a simple design which keeps close to the bare metal in order to expose the underlying systems file system concurrency guarantees.
No, it's not. As I and others have explained, if does way more than it needs, in a strive to be generic. The benefit of this genericness remains to be proven.
I tried to assess the library's performance as IMHO (and the library's documentation) performance is the major motivation for using AFIO. The code failed to compile with Intel's C++ compiler icpc[2]. With g++ 4.9.3 the code compiled, but segfaulted when traversing a Linux 4.1.6 source tree[3].
The library's documentation clearly specifies that the compilers tested are GCC >= 4.8, clang >= 3.3 and VS2015. Any other compiler is not supported.
That clearly doesn't make it any better. I'd expect a Boost library to adhere to the language standard (which it clearly doesn't if it's not compatible with ICC).
As to why you saw a segfault when traversing a Linux source tree (was it with find regex in files?), I am sorry this occurred.
No need to be sorry, bugs are just part of the development cycle, right? :-)
I would suggest to use the find regex in files binary compiled using Boost.Build rather than any custom build and see if it happens there too. If it does, open an issue on github please.
How would Boost.Build have an influence here? If my (documented, simple) script compiles and links, why should it not run?
The code (and the naive OpenMP implementation provided in the documentation) completed successfully on a Boost.AFIO source tree, but the OpenMP code (~600 MB/s) was significantly faster than the AFIO code (~270 MB/s), see [4]. Test system: Intel Core i7-3610QM, Samsung EVO 1TB SSD, Linux 4.1.5. (I can provide further details if necessary.) All of this does not positively impress me.
You have highly unrealistic expectations. On Linux there is no useful async backend, so AFIO emulates async using a thread pool and the normal synchronous APIs. It goes without saying that will always be slower than OpenMP using the same APIs. It can never be faster.
I like the fallback to a reasonably fast alternative Kernel API. However: - Just because both examples use the same (synchronous) APIs and a thread pool internally, it doesn't mean that one code's performance is confined to be less or equal to the other code's performance. Also, if this was true, we could infer by symmetry, that this holds true the other way around, which would give us that both performances have to be equal. Obviously not true. - The naive OpenMP example is using a thread pool, AFIO is using these elaborate, efficient futures, continuations etc. I'd have expected this to be more efficient, especially given the nature of the OpenMP example (which does a sequential directory walk before spawning threads). - Even if we assume that AFIO can't be faster than an OpenMP code in this setting: why is the overhead so high? What is making it >2x slower?
What AFIO provides is a *portable* API for async file system and file i/o. It provides the least benefit on Linux of any of the platforms, but that is not AFIO's fault.
Sorry, but that's not true. It remains AFIO's fault as long as you can't explain where this slowdown comes from. Why don't you, say, default to a naive, simple OpenMP implementation on Linux? Cheers -Andreas -- ========================================================== Andreas Schäfer HPC and Grid Computing Department of Computer Science 3 Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany +49 9131 85-27910 PGP/GPG key via keyserver http://www.libgeodecomp.org ========================================================== (\___/) (+'.'+) (")_(") This is Bunny. Copy and paste Bunny into your signature to help him gain world domination!