On 18 Mar 2015 at 9:41, Pete Bartlett wrote:

...

This release took far longer than I originally intended mainly due to the extensive debugging of weird filing system races, and for six weeks now I've been working till 5am every night after my day job. I cannot tell you how glad I am to see this release out!

You are obviously very proud of this library and I believe rightly so. You've worked hard on it and in terms of things like soak testing it probably sets a new high watermark for Boost libraries.

I'm proud of what I've been able to do with the very limited resources available to me, but I feel not especially proud of shipping a library which isn't well tested in absolute terms. For example, almost all the testing is functional rather than unit testing (lack of time to write more), plus the soak tests run for minutes instead of the 6 or 24 hours I'd much prefer and on only quad core hardware with local drives rather than the 16-32 core distributed over Samba/NFS network shares that it should be. I'm also not happy with the testing of the (too) many build options, the lack of testing of std::bad_alloc correctness (right now I claim exception safety except for bad_alloc), and there is no testing of localised resource leaks, only global testing. Ok, so like most Boost maintainers I see all that I failed to do before any of the achievements. Still, it could be a lot better than it is, and a bug in AFIO means data loss for users. In the end, we all have day jobs and families.

You're also a highly active member of this community. So why is the library always "proposed".... When is there going to be a review? :)

When a review manager volunteers themselves! Antony nearly did a few months ago, but this release was in progress and at that time I had no ABI nor API versioning in AFIO, so I could not propose an even semi-stable API at that time. In the v1.3 release, both ABI and API are versioned and there is a unit test which compiles two intentionally dimorphic and ABI incompatible copies of AFIO into the same translation unit and runs both sets of unit tests for each to make sure they don't collide. They don't, so I am now sure v1.3 of AFIO can coexist without issue with v1.4 of AFIO or any other version. So I now have no objections, and furthermore no major work will be done to AFIO until after CppCon in September. Still, asynchronous file i/o and race free filesystem is an extremely niche use case. I am not aware of anyone who uses AFIO in their code. That has surely to count against the wisdom of including it into Boost official as without a user base, who can really say if a design or implementation is any good during review? Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 19 Mar 2015 at 19:58, Mathias Gaunard wrote:

Hi, this is my first time actually looking at the documentation of the library.

I think it would be interesting to develop a couple more things in the design or FAQ sections.

Suggested topics are welcome.

Why use a specific asynchronous model with its own when_all etc.? There is a standard way of defining asynchronous tasks, which is getting extended by compiler support using await/yield_value, so why not do something using that?

It's not too obvious from first glance, but an async_io_op is a struct with a shared_future inside it which is publicly accessible called 'h'. So it uses the standard STL11 asynchronous model, just slightly hidden. The when_all etc is purely due to legacy reasons. Indeed the reason I couldn't use shared_future directly is purely legacy, back in the day I had to use shared_ptr due to STL11 bugs, and it was easier to wrap that into an async_io_op, plus back in 2013 there was no fully functioning when_any/when_all implementation so I had to write my own and I needed type sugar to stop my when_any/all conflicting with the ones in Thread. I think the v1.2 engine replaced the shared_ptr with shared_future, so that was the major refactor for v1.2. Hence, especially as the Concurrent TS continuations API has finally firmed up, async_io_op can go away next release now, and an afio::future<T> with .then() etc replaces it. I've tried to bridge the APIs with a get() and a few other future like member functions to make porting the code easier, but it's going to be a major break, not least that the current central hash table is going away too in favour of a lock free linked list. So basically AFIO currently looks the way it does mostly due to legacy baggage and problems with C++ 11 back in 2013. Each release I refactor some of the baggage out. v1.3 was the first to drop < VS2013 support for example, it was a ton of work purging the VS2010 custom code paths. v1.4 will purge async_io_op + when_any/all and replace those with a proper C++ 17 compliant custom continuable future type which understands resumable functions. A big part of why that didn't enter v1.3 is because VS2015 and its resumable functions support I can test against isn't released yet, plus I was waiting for the Concurrency TS to firm up, it's only really stabilised these past six months.

What about the relationship with ASIO? ASIO has its own event-driven main-loop system for asynchronous tasks, can AFIO use that and can ASIO use AFIO's future method for other types of I/O too?

AFIO uses ASIO's event loop. It doesn't use the async_result idiom of ASIO because for file i/o you want strong graph ordering, and that's a pain to do with async_result. A more functional idiom, for which I chose future-promise, makes strong ordering much easier to program. Once I get Fibers/resumable function support in there next release, writing strongly graph ordered filing system algorithms ought to become optimally easy, and then I can get down to writing race free tree visitors etc which right now are very messy to write and debug. The AFIO-ASIO integration isn't deep right now, though of course ASIO's async_result can work with any arbitrary sync object, and that includes an async_io_op. Writing code (mostly completion lambdas) which arbitrates between the two is straightforward, if verbose and tedious. What is needed is zero copy kernel side support, and AFIO's design intentionally allows that (i.e. async write this gather list from this file to this socket all in the kernel using DMA). I'll be honest it's not high on my priority list given no one is using AFIO anyway, but potential deep AFIO-ASIO integration is very high on the radar. I will say though I'll need changes to ASIO to make that work, so it's not a trivial feature add. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 19 Mar 2015 at 21:26, Mathias Gaunard wrote:

So v1.4 will be the interface that will be submitted for inclusion into Boost?

v1.0 was the interface submitted for inclusion into Boost. It reflected the state of C++ in 2013 where XP compatibility was still very important, and therefore so was VS2010 compatibility. Every version thence is also submitted for inclusion until a review happens, which I assume will come back with a long list of changes needed anyway. If you're really asking when would I expect to finish the major refactoring, currently I expect v1.4 to be the last. However, I thought that about v1.3 during v1.2. A lot of this is outside my control of course. Had WG21 gone for Chris's concurrency proposal for example, I'd be refactoring in a very different direction. This is a big consequence of being the first C++ 11 only Boost library to enter the community review queue - you build in not just your own technical debt, but also the technical debt of the C++ language itself, *plus* that of the Microsoft compiler team, the libstdc++ team, the mingw team and so on. In fact the oldest compiler from v1.0 still supported identically is clang, because they had the most complete C++ 11 support the earliest, so clang 3.2 still works perfectly. As the ecosystem collectively reaches closer to conformance to the standard and therefore to each other, I can keep ripping out workaround patterns which used to make sense before but don't anymore. I'll put this another way. Back in v1.0 future-promise looked fast and buggy, but then the engine could only push 150k ops/sec. By v1.2 future-promise looked slow because it could now push 1.5m ops/sec. So by me ripping out the shared_ptr workarounds, I made the performance spotlight fall on other code. As I fix that, spotlights shine elsewhere and more fixes are needed. At some point both the C++ language ecosystem changes will settle down, and nails will stop sticking up to be hammered down. I'd expect API maturity to start then, same as any other Boost library. At that point I'll start building the embedded graph database AFIO is intended for.

...

AFIO uses ASIO's event loop. It doesn't use the async_result idiom of ASIO because for file i/o you want strong graph ordering, and that's a pain to do with async_result. A more functional idiom, for which I chose future-promise, makes strong ordering much easier to program. Once I get Fibers/resumable function support in there next release, writing strongly graph ordered filing system algorithms ought to become optimally easy, and then I can get down to writing race free tree visitors etc which right now are very messy to write and debug.

I think it would be useful to be able to make use of the future-promise idiom for other ASIO code as well.

ASIO already provides future-promise support via asio::use_future.

Is it possible to use your futures for any ASIO async_result?

Absolutely. Mine has just standard shared_futures, same as the ones ASIO uses. The custom afio::future in v1.4 will also work perfectly in ASIO, indeed I'll be providing the async_result specialisations so it's just as seamless to use. ASIO has Fiber support too, so Fiber should seamlessly multiplex ASIO and AFIO i/o for you. You just glue together the logic with resumable functions, and let the runtime figure it out for you. Which will be neat. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/