[Interest] [Semi OT] Concurrent (multi-threaded) read/write disk IO?

[Till Oliver Knoll]

Hi all,

This is somewhat unrelated to Qt, but I hope one or another has stumbled over this and can share some thoughts.

Does it make sense to guarantee/enforce "sequential (exclusive) access to the harddisk" on application level, or would I re-invent functionality already present in the underlying OS/disk driver (and maybe even sacrifice performance)?

I first want to focus on "slow physically spinning harddisks" before we come to SSD (in case that makes a significant difference anyway).

Specifically I have the following scenario in mind: "batch image conversion". Let's assume we have a "Work Queue" (original images) and a "Result Queue" (processed images). Some worker pool would dequeue ("take") work items (images) from the Work Queue, process it and enqueue ("put") it into the Result Queue.

The Work Queue size would be limited, but as soon as the count would drop below some threshold some Reader thread would fill it again (by reading images from disk).

Likewise the Result Queue would be emptied by a Writer thread.

Now my question is about whether to fill the Work Queue ("read from disk") and empty the Result Queue ("write to disk") concurrently (2 threads: Reader and Writer), or have the disk access go through some kind of "IO Manager", such that disk access is sequential (be it read or write operations, while making sure not to "starve" either operation, e.g. some "fair access pattern").

To clarify: I do not want to have multiple threads to all /read/ different files at the same time (or write, for that matter - at least for now). My concern is not to "trash" the hardisk by reading and writing at the same time. Or rather: whether I should be concerned at all on an application level.

If the answer is "don't bother in your application! Just read and write at the same time and the OS/driver will figure out the best access pattern for you already!", then I guess consideration of SSDs probably become moot.

If the answer would be "avoid concurrent read/write access on spinning harddisks!", would it be different for SSDs then?


Thanks for sharing your thoughts!
  Oliver