On 1 Jul 2014 at 23:24, Benedek Thaler wrote:
It's not deterministic when the first segment will end, the pipeline will hang and won't provide any output. Also, in a slightly less strict scenario, when the end of input is specified, the latency could be just too high.
To summarize, the following options are on the table:
1. Dedicate a thread to each segment (what to do with a fixed size threadpool?) 2. Constrain the transformations to be reentrant. 3. Run each transformations until there is input to be processed, from beginning to the end.
We are kindly asking the recipients to share their ideas or opinions, ideally with a matching usecse.
Mmm, you're getting into needing to implement a demand-driven transformation based data stream framework, something I suspected would always happen with this proposal. In short, either you make all your processors coroutineable and resumable - which is a ton of work. Or you implement a full data stream implementation, which is also a ton of work. If you go with your approach 2 you'll murder reusability, with approach 3 you'll use a ton of RAM. Given yours is a GSoC and needs to be done sooner rather than later, I'd actually implement a dynamic thread pool which scales up the worker threads to whatever is the minimum set of blocking processors. Or borrow the threading facilities in HPX and port your code over to HPX completely, then your threading and blocking problems go away because HPX has implemented the data stream framework for you. You'll obviously lose C++ genericity if you do this though. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/