[Interest] [Semi OT] Concurrent (multi-threaded) read/write disk IO?

Keith Gardner kreios4004 at gmail.com
Thu Feb 5 14:44:17 CET 2015


>
> Does it make sense to guarantee/enforce "sequential (exclusive) access to
> the harddisk" on application level, or would I re-invent functionality
> already present in the underlying OS/disk driver (and maybe even sacrifice
> performance)?
>

It depends on the task at hand.  If you know you are going to perform
significant IO, it makes sense to limit the number of reads/writes to the
storage device.  This is especially true when the storage medium is slow.


> Specifically I have the following scenario in mind: "batch image
> conversion". Let's assume we have a "Work Queue" (original images) and a
> "Result Queue" (processed images). Some worker pool would dequeue ("take")
> work items (images) from the Work Queue, process it and enqueue ("put") it
> into the Result Queue.
>

Have you looked at ThreadWeaver (
http://api.kde.org/frameworks-api/frameworks5-apidocs/threadweaver/html/index.html)?
They have an example application that does the exact scenario you are
describing.  In this case, you would have a thread pool and you would just
issue jobs instead of managing your threads directly.

You can have a job that performs the IO and put restrictions on how many of
those jobs can be run in concurrently.

If the answer is "don't bother in your application! Just read and write at
> the same time and the OS/driver will figure out the best access pattern for
> you already!", then I guess consideration of SSDs probably become moot.
>

I am using ThreadWeaver on an embedded device that has to write to an
SD-Card.  File IO is painfully slow but using ThreadWeaver allows for me to
rate limit complex events appropriately.  When moving my application to an
SSD, I just change how many concurrent File IO events run in parallel so
that it can scale with the system.

Keith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/interest/attachments/20150205/ded80b50/attachment.html>


More information about the Interest mailing list