[Interest] [External]Re: How to get QtConcurrent to do what I want?

[Murphy, Sean]

> Not knowing if a partial value makes any sense to your system.
> Qt::Concurrent::mappedReduced might make more sense, if its purely a
> speedup you are looking for, and not a "keep the GUI alive during it" possibly
> blockingMappedReduced.

I don't think mappedReduced would help me until after the remapping step 
when I want to assemble the individual tiles into a QImage - although since the 
ultimate destination for the image is inside a QGraphicsView, I'm tempted to just 
leave it as individual tiles, but I'm not there yet as far as testing.

Regarding the keep GUI alive portion, my tileManager is already running in a 
separate thread from the UI thread, so I do have the option of making blocking calls
in the tileManager class with the exception that I do need to fire progress signals out 
of tileManager back to MainWindow to provide progress to the user

> 
> If you need the gui, setting up a qfuturewatcher on the results of the
> mapped call, would be my approach
> 
> QFutureWatcher< XXX > watcher;
> connect( &watcher, &finished, manager, &handleFinished);
> 
> auto future = QtConcurrent::mapReduced(tiles, processTile, mergeFunction)
> watcher.setFuture( future );

I actually spent yesterday refactoring the tileManager class as you've just described, 
as well as changing out the tile class to no longer inherit from QObject. I've got a couple 
more things I want to try/clean up today but I still seem to be having trouble speeding 
up the allocation & assignment of the tiles themselves.

As my code currently stands, I now have to vectors, each of which will have 60,000 items 
in them once they're populated:
  QVector<int> mTileIndices;
  QList<tile> mTiles;

The mTileIndices vector is implementing Andrei's idea of quickly generating a list of unique tile 
indices, which can then be fed to a QtConcurrent::map() call to create & uniquely assign the 
tile items in parallel. The " mTiles " vector is obviously the tiles that will do the 
work.

As I build the vectors up, these are the timings I get:
  1. resizing tile index vector to 60000 took 0.1514 ms
      a. This is just calling QVector<int>::resize(60000) on the ID. This takes less than a 
           millisecond, so no complaints here.
  2. allocated 60000 indices in 0.0207 ms
      a. This is calling std::iota(mTileIndices.begin(), mTileIndices.end(), 0). Also takes less than 
          a millisecond, still no complaints
  3. assigning 60000 tiles took 18087.1 ms
      a. This timing is the result of QtConcurrent::mapped(mTiles, initTile) where the initTile 
          function takes in an integer from mTileIndices, and calls the tile constructor using the 
          combination of the tile index and tile size to do the assignment. The assigned tiles end 
          up in the mTiles
      b. This step takes 18 seconds, which seems excessive to me and I'd love to continue to 
           reduce that time.
  4. load finished in 37507.1 ms
      a. this is calling tile::load() on each tile. Right now that is just a dummy function that calls 
          msleep for a random amount milliseconds to simulate doing the actual work
  5. remapping 60000 tiles took 48519.3 ms
      b. this is calling tile::remap() on each tile. Right now that is just a dummy function that 
          calls msleep for a random amount milliseconds to simulate doing the actual work

So the only step in this process that still bothers me is step 3 - creating and assigning each tile 
object takes 18 seconds. I log the total time by each tile spent in steps 4 & 5 and compare how 
long steps 4 & 5 actually take vs. the sum of how long each tile spent sleep and I routinely get
a speedup factor of about 7.5. QThread::idealThreadCount() reports 8 on my machine, so I think
I'm getting what I should expect from those steps on this machine.

I'm not sure what else to try at this point. One thing I was thinking about measuring is that even 
though my tile class no longer inherits from QObject, it still is a class with a constructor and some 
getter functions. And when I look at the combination of steps 1 & 2 where I both resize a vector 
of integers AND assign each one a unique ID in less than a millisecond total, but then it takes 
me 18 seconds to create and assign each tile, I keep wondering if there isn't room for 
improvement there still?

And as I was typing this whole thing up, Konstantin gave me a different approach that I'll probably 
pursue, but I would like to better understand how to solve the question of "if you absolutely need 
to have a lot of items (whatever type an "item" needs to be), what's the right design approach to 
be able to create and populate them quickly..."

Sean