On 15 Apr 2015 at 22:55, TONGARI J wrote:

...

Would anyone else like to comment on this library based instead of compiler based resumable execution? I'm not against some compiler support for resumable execution, but I am finding the await based proposal highly problematic, and Chris does an excellent job in summarising my own thoughts on what's wrong with it.

IIUC, this proposal requires more compiler magic and is definitely compiler based.

This surprises me. He provides a reference implementation library which appears functionally complete except for the error checking which does require a compiler pass. The only compiler support needed is for reporting error messages, or at least that's what I read?

Await/yield from the "Resumable Functions" proposal is not that mysterious as one may think, at least I can emulate it in some degree using preprocessor: (https://github.com/jamboree/co2)

By "compiler magic" I mean a new compiler keyword which returns a STL type, and assumes a great deal about how that STL type is implemented. I find that possibly okay for brand new STL types newly added. I find that deeply worrying for existing STL types. Even with their proposed generic all types of future traits support. I also feel that resumable functions need to not require allocating memory. I know they hacked MSVC to magically optimise out the memory allocation in the future<T>, but that doesn't appeal to me.

The main point of this proposal is the implicit propagation of the resumability, that is, if a function calls a resumable function, then the function itself becomes a resumable function, which makes the existing template-based algorithms also usable to resumable functions.

Does this paragraph refer to the Microsoft resumable functions or to Chris' resumable expressions? The latter makes resumability like constexpr, and therefore resumability is caller determined instead of callee determined like the Microsoft proposal. Caller determined resumability has the enormous advantage that you only need to write implementations once, instead of with resumable and non-resumable variants.

A major benefit of this approach is that it provides a unified syntax for both stackful and stackless coroutine, and IMHO is better than the approach proposed in N4398.

In general, I like the idea of N4453 and would love to see a working language implementation.

That said, I do have some questions, for example, it's not clear to me that whether a ctor/dtor could be a resumable function or not, and there may be some other issues that I can't foresee without playing with it.

It seems to me that a constructor is an expression, and therefore capable of being resumed under Chris' proposal. The object being constructed would, I assume, remain partially constructed. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 16 Apr 2015 at 0:25, TONGARI J wrote:

...

By "compiler magic" I mean a new compiler keyword which returns a STL type, and assumes a great deal about how that STL type is implemented.

Both the types that you can await on and the types that you can return from a resumable function are extensible and not restricted to STL.

I think therein lies exactly the problem. await is far too limited in usefulness - it has a very specific use case, and is hardcoded to that. What I'd far prefer is the ability to ask the compiler for: 1. An instantiation of a call tree which uses a guaranteed immutable amount of storage at the beginning of the call. "Fixed stacks" is a good description, though it includes any operator new which is always deleted - under fixedstacks they auto convert into fixedstack allocation. This is essentially the same thing as stackless python and how it implements tasklets - fixedstack call trees are like tasklets. 2. The ability to portably ask the compiler for a dump of the current fixedstack (fast) or C stack (slow) into some STL type - let's call it std::execution_context. And that is actually all I want. Because that is all I need to implement coroutines with ideal performance characteristics. What I do NOT want is: 1. A profusion of additional traits and extensions of STL threading primitives where the design is by committee with very little experience of real world use before it is written into stone with many potentially unexpected consequences.

I also feel that resumable functions need to not require allocating

...

memory. I know they hacked MSVC to magically optimise out the memory allocation in the future<T>, but that doesn't appeal to me.

The resumable functions in Chris' proposal have to be visible to the compiler in the same TU, if you want them to be in another TU, you need to abstract the resumability away and thus need allocation.

Fixedstacks makes that go away because the ABI for fixedstack functions exposes the relevant internal memory allocation state changes for consuming call trees. For call trees which are not fixedstack, you are correct.

Now that I think of it, the resumable function in Chris' proposal is also not a first-class object, so I guess you can't take its type as it's completely a compile-time construct.

Fixedstacks make enough of a call tree entirely known to the top most point of call stack construction, and I believe should make them a first class object. No dynamic memory allocation required. The consequence, of course, is that fixedstack call trees can't do some things if they are to remain fixedstack (same as constexpr where you can call a constexpr function in a non-constexpr fashion). The most obvious is you can't use malloc and cast the allocation into some void *, as that would defeat the whole point of fixed stack. I am sure the compiler could be made to error out if you try that though. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 15 Apr 2015 at 22:26, Avi Kivity wrote:

On 04/15/2015 04:27 PM, Niall Douglas wrote:

...

I'm finding Chris' paper on resumable expressions instead of compiler magic created resumable functions awfully compelling:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4453.pdf

I essentially proposed the same with my functionally composed constexpr basic_future design some months ago.

Regarding that, you may be interested in a similar effort, seatar [1], with very lightweight thread-unsafe futures and continuations (future<> size is 16 bytes) with multi-core utilization provided by explicit dispatching.

Thanks for that. There has been a profusion of lightweight future object alternatives in recent months. I think yours is like the fourth I've seen now, each different but all similar in that they are much lighter weight than anything being proposed by the Concurrency TS. I fear, unfortunately, that WG21 is not as aware of these trends in real world use cases as they should be. Certainly the presently proposed continuable future is much fatter than the C++ 14 future, and I think that is not welcome. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 15 Apr 2015 at 15:40, Rob Stewart wrote:

...

I'm finding Chris' paper on resumable expressions instead of compiler magic created resumable functions awfully compelling:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4453.pdf

I essentially proposed the same with my functionally composed constexpr basic_future design some months ago.

Would anyone else like to comment on this library based instead of compiler based resumable execution?

This is off-topic for this list.

With respect to you as list moderator, I disagree. An already conditionally accepted library, Fiber, is fundamentally affected by what WG21 are up to here. An existing Boost library, ASIO, is fundamentally affected, hence its maintainer leading out the opposition to present developments. The Thread v5 rewrite is fundamentally affected, and shortly so will be AFIO which fair enough, is not a Boost library. Most WG21 papers aren't as important to Boost libraries, and hence I would ordinarily agree such discussion is off topic. The feedback already received here about resumable functions I have already found valuable - I hope so have many other library developers here. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 16 Apr 2015 at 14:05, Rob Stewart wrote:

...

An already conditionally accepted library, Fiber, is fundamentally affected by what WG21 are up to here. An existing Boost library, ASIO, is fundamentally affected, hence its maintainer leading out the opposition to present developments. The Thread v5 rewrite is fundamentally affected, and shortly so will be AFIO which fair enough, is not a Boost library.

There was no mention of any connection to Boost libraries. Now that you've created that connection, I agree with you.

Sorry. As I discovered in off list discussion about coroutines and resumable functions as well, the connections which are obvious to me need to be explicitly stated much more frequently. Especially as it would appear that the connections in my head are quite different to anyone else involved in the discussion. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 14/10/2015 08:32, Vinícius dos Santos Oliveira wrote:

2015-04-15 10:27 GMT-03:00 Niall Douglas :

...

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4453.pdf

revision 1 (p0114r0) is available: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/p0114r0.pdf

Let me just say up front that I'm definitely not an expert in this and I've only just recently read these and the await paper (N4286; not sure if there's a newer one). So please let me know if I've missed something. It bothers me a bit that it imposes limitations on use (given in 7.6, and presumably present so that the compiler can unambiguously know the required stack size of the call chain) that would render it nearly useless for practical purposes (since most code is not header-only templates, except in certain library contexts). It goes on to admit this in section 12 and offers some library-based solutions, but makes a point of stating that these aren't part of the proposal itself. There are some things I like about this -- namely that it potentially allows for different library implementations for different requirements, and theoretically even the potential for application writers to completely customise it for certain needs (eg. enforcing certain mutex types or memory patterns). But this is also a potential complication -- look at all the entertainment we're already having with interop between different pointer/future/thread/mutex types. I must admit to not being entirely happy with await (although most of my experience has been with C# code), due to some of the points mentioned in this paper (viral keywords, magic use of special library types, and duplication of implementation if desired to provide both async and non-async methods). However this proposal shares the latter limitation *unless* implemented as header-only templates (see 7.10 -- non-templates cannot be both resumable and non-resumable, even if header-only). And where async and non-async methods are required to be separate, then viral keywords are not a bad thing as it makes it more obvious where the potential suspension points are. But one advantage of await is that it does not have those limitations, and is intended to be used directly by application code with minimal library support.

On 10/13/15 12:32 PM, Vinícius dos Santos Oliveira wrote:

2015-04-15 10:27 GMT-03:00 Niall Douglas :

...

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4453.pdf

revision 1 (p0114r0) is available: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/p0114r0.pdf

I looked at this briefly and found it intriguing. But I have a couple of questions. Suppose I use something like Boost.Coroutine and declare the function which calls it as constexpr. Now this means that the constexpr stack is implemented at compile time and not at all at runtime. So how is not a "stackless" co-routine? How is it different than this proposal. Given this, why is this proposal even necessary? Robert Ramey