On Thu, Jul 3, 2014 at 2:27 AM, Adam Berkan
I was suggesting that the thread would grab a single piece of data from a queue, do a single transformation, and then return where it would make a scheduling decision about where to go next. Then it would once again grab one piece of data and do one transformation before returning to scheduling.
In case of 1-1 transformations, I think this is done in the `development` branch, altough the really interesting issues are not taken care of... ->
I'm not suggesting that we would interrupt a thread in the middle of a stage... That would require writing most of an operating system kernel :)
The tricky part is knowing where there's work to be done. The "smart queues" would have to update some stats in the scheduler on every push, and potentially wake up threads. It's certainly a non-trivial task.
-> like this. Now I think we are on the same page regarding such queues or scheduling facility..
There's also the issue of the multi-in (N:1 & N:M) transformations, i.e. that take input queues. They would either have to take a whole thread, provide some kind of yield, or something else. I'd probably ignore this for now (just consume a thread) and focus on the simple 1:1 and 1:N cases as they naturally return after each transformation.
I think only 1:1 is an easy case. Even 1:N has an issue: consider a transformation which duplicates its input. What if the second element cannot be pushed to the downstream queue because it's full? The transformation should yield (and not spin). Thanks, Benedek