On 22 Jan 2014 at 18:37, Bjorn Reese wrote:
Niall, would you be able to propose a more universal model? Please read this as a simple invitation rather than a challenge. The goal of
A good place to start is to understand the limitations of the Asio model.
A great post Bjorn. You definitely explained it better than I. And my public thanks to you for all your work with me on improving AFIO (I'll be replying to your email soon, I ought to submit my maths coursework tomorrow). I should elaborate on the importance of easily chaining operations into patterns as it's probably non-obvious. Storage i/o, unlike fifo i/o, does not scale linearly to queue depth due to highly non-linear variance in callback latencies due to OS caching effects overlaid on mechanical motors, so ease for the programmer to fiddle with operation chain patterns is paramount for maximum performance, especially as it's mainly a trial and error thing given the complexities of the systems which make up filing systems etc. One basically designs an access pattern you think ought to be performant under both cold and warm cache scenarios, try testing it and find yourself a bit wide from the mark, so now you hunt around for the goldilocks zone through repeated testing cycles. I haven't personally found any better way than this yet sadly, storage is so very non-linear at the micro-level. Doing this using ASIO style callbacks involves a ton load of cutting and pasting code around, several iterations of compilation to remedy the unenforced syntax errors, more iterations of debugging because now you've broken the storage etc. Alternatively, doing this using AFIO style chained ops is far easier and quicker because you simply tweak the op dependency graph you're sending to the async closure engine to execute, and you let the engine figure out how best to send it to the OS and ASIO. There is also an additional debugging option made available because with a formal op dependency graph one could have code check that your sequence of reads and writes is power-loss safe and race condition free with other processes reading and writing the same files. Using an ASIO style callback model that would be hard without additional metadata being supplied. As much as AFIO style async code is a bit heavy due to the use of futures, for the kind of latencies you see with big chunks of data being moved around it's an affordable overhead given the convenience. Niall -- Currently unemployed and looking for work in Ireland. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/