Hi Niall,
On Fri, Apr 22, 2016 at 8:18 AM, Niall Douglas
On 21 Apr 2016 at 20:47, Aaron Boxer wrote:
The read thread will spend a lot of time waiting for file IO to complete. Would it be faster to us the ASIO classes to do the reading?
No, it won't.
Memory mapped file i/o is almost guaranteed to be the correct technique to use here. Simply map the entire file into memory, and fire threads at processing it. Let the kernel figure out how best to do the i/o.
Thanks. I do have a memory mapped file interface in my library for the compressed file, so I can certainly try that. My impression is that memory mapping is best when reading a file more than once, because the first read gets cached in virtual memory system, so subsequent reads don't have to go to disk. Also, it eliminates system calls, using simple buffer access instead Since memory mapping acts as a cache, it can create memory pressure on the virtual memory system, as pages need to be recycled for the next usage. And this can slow things down, particularly when reading files whose total size meets are exceeds current physical memory. In my case, I am reading the file only once, so I think the normal file IO methods will be better. Don't know until I benchmark. Thanks again, Aaron
Niall
-- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost