[Interest] [Semi OT] Concurrent (multi-threaded) read/write disk IO?

[Rainer Wiesenfarth]

Am 05.02.2015 um 20:21 schrieb Till Oliver Knoll:
> [...]
> Other links that I found seem to support that, that the underlying
> "scheduler" figures out the best read/write strategy, and any attempt by
> the application to implement that by itself would be counter-productive
> [...]

One addition: If using the OS without an own scheduler, it makes sense 
to give the OS an idea what you are about to do with the file. Reading 
chunks of 256 bytes vs. reading chunks of 1 MB vs. mapping the whole 
file into memory probably has a higher impact on performance than the 
number of threads used.

For your application: Memory mapping the files, limiting the total 
mapped size and thus the number of simultaneously mapped files should 
give the best performance. If you have eight worker threads, each may 
map its input file into memory and simply access it. The writer part 
then could be a single thread.

Personally, I would give QtConcurrent::mappedReduced() the first try, 
with all reading (on memory mapped files) and resampling in the map 
function and the writing in the reduce function.

Best Regards / Mit freundlichen Grüßen
Rainer Wiesenfarth

-- 
Software Engineer | Trimble Imaging Division
Rotebühlstraße 81 | 70178 Stuttgart | Germany
Office +49 711 22881 0 | Fax +49 711 22881 11
http://www.trimble.com/imaging/ | http://www.inpho.de/

Trimble Germany GmbH, Am Prime Parc 11, 65479 Raunheim
Eingetragen beim Amtsgericht Darmstadt unter HRB 83893,
Geschäftsführer: Dr. Frank Heimberg, Hans-Jürgen Gebauer

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4255 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.qt-project.org/pipermail/interest/attachments/20150206/05e1bd7d/attachment.bin>