*Niall*: I'm afraid using HPX is out of scope in context of a Boost library. If all else fail, I'll fall back to a simple implementation (e.g: 1). IMO, depending on this design decision will be this library a great tool or just something nice.
*Adam*: I'm not sure how this smart queue would solve the problem of reentrancy. Currently, if a thread returns from a transformation, it can never be resumed. This makes yielding hard/not possible.
I was suggesting that the thread would grab a single piece of data from a queue, do a single transformation, and then return where it would make a scheduling decision about where to go next. Then it would once again grab one piece of data and do one transformation before returning to scheduling.
I'm not suggesting that we would interrupt a thread in the middle of a stage... That would require writing most of an operating system kernel :)
The tricky part is knowing where there's work to be done. The "smart queues" would have to update some stats in the scheduler on every push, and potentially wake up threads. It's certainly a non-trivial task.
There's also the issue of the multi-in (N:1 & N:M) transformations, i.e. that take input queues. They would either have to take a whole thread, provide some kind of yield, or something else. I'd probably ignore this for now (just consume a thread) and focus on the simple 1:1 and 1:N cases as they naturally return after each transformation.
All of that functionality is available from HPX. It gives you work-queue based scheduling (with work stealing) of suspend-able/resume-able threads (i.e. supporting yield) with very little overhead. It also manages the 1:N or N:M threading for you. You could try building all of your functionality on top of HPX first. This could allow to figure out the actual underlying mechanisms your library would rely on. Later on you can move it to Boost after all of the required functionality has been accepted there. Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu
Let me know if that makes sense. Adam
If we make reentrancy mandatory, then we face the next challenge, which you
described above IIUC. The think the key points are here: (1) A task can yield, and can point to an other task which is waited upon. (2) A task (but not the thread) can "block" on the upstream queue, if it's empty. If it's not clear, I'll describe these in greater length.
*Oliver*: I considered using coroutines, however I don't clearly see how this would solve this problem exactly. IIUC, Coroutines provide a way for a single thread to execute multiple methods. I'd need multiples threads executing multiple methods in a similar fashion. I'll continue thinking about this.
I couldn't found it in the docs. Does Coroutine have any drawbacks?
Thanks, Benedek
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost