[Interest] [External]Re: How to get QtConcurrent to do what I want?

[Scott Bloom]

Something seems off.  

But without looking at the actual code that is allocation 60k tiles and the constructor itself, it just seems like a very expensive construction if the "new" + "moving a pointer" is taking 3ms each.

Scott

-----Original Message-----
From: Interest <interest-bounces at qt-project.org> On Behalf Of Murphy, Sean
Sent: Tuesday, February 1, 2022 7:32 AM
To: interest at qt-project.org
Subject: Re: [Interest] [External]Re: How to get QtConcurrent to do what I want?

> Not knowing if a partial value makes any sense to your system.
> Qt::Concurrent::mappedReduced might make more sense, if its purely a 
> speedup you are looking for, and not a "keep the GUI alive during it" 
> possibly blockingMappedReduced.

I don't think mappedReduced would help me until after the remapping step when I want to assemble the individual tiles into a QImage - although since the ultimate destination for the image is inside a QGraphicsView, I'm tempted to just leave it as individual tiles, but I'm not there yet as far as testing.

Regarding the keep GUI alive portion, my tileManager is already running in a separate thread from the UI thread, so I do have the option of making blocking calls in the tileManager class with the exception that I do need to fire progress signals out of tileManager back to MainWindow to provide progress to the user

> 
> If you need the gui, setting up a qfuturewatcher on the results of the 
> mapped call, would be my approach
> 
> QFutureWatcher< XXX > watcher;
> connect( &watcher, &finished, manager, &handleFinished);
> 
> auto future = QtConcurrent::mapReduced(tiles, processTile, 
> mergeFunction) watcher.setFuture( future );

I actually spent yesterday refactoring the tileManager class as you've just described, as well as changing out the tile class to no longer inherit from QObject. I've got a couple more things I want to try/clean up today but I still seem to be having trouble speeding up the allocation & assignment of the tiles themselves.

As my code currently stands, I now have to vectors, each of which will have 60,000 items in them once they're populated:
  QVector<int> mTileIndices;
  QList<tile> mTiles;

The mTileIndices vector is implementing Andrei's idea of quickly generating a list of unique tile indices, which can then be fed to a QtConcurrent::map() call to create & uniquely assign the tile items in parallel. The " mTiles " vector is obviously the tiles that will do the work.

As I build the vectors up, these are the timings I get:
  1. resizing tile index vector to 60000 took 0.1514 ms
      a. This is just calling QVector<int>::resize(60000) on the ID. This takes less than a 
           millisecond, so no complaints here.
  2. allocated 60000 indices in 0.0207 ms
      a. This is calling std::iota(mTileIndices.begin(), mTileIndices.end(), 0). Also takes less than 
          a millisecond, still no complaints
  3. assigning 60000 tiles took 18087.1 ms
      a. This timing is the result of QtConcurrent::mapped(mTiles, initTile) where the initTile 
          function takes in an integer from mTileIndices, and calls the tile constructor using the 
          combination of the tile index and tile size to do the assignment. The assigned tiles end 
          up in the mTiles
      b. This step takes 18 seconds, which seems excessive to me and I'd love to continue to 
           reduce that time.
  4. load finished in 37507.1 ms
      a. this is calling tile::load() on each tile. Right now that is just a dummy function that calls 
          msleep for a random amount milliseconds to simulate doing the actual work
  5. remapping 60000 tiles took 48519.3 ms
      b. this is calling tile::remap() on each tile. Right now that is just a dummy function that 
          calls msleep for a random amount milliseconds to simulate doing the actual work

So the only step in this process that still bothers me is step 3 - creating and assigning each tile object takes 18 seconds. I log the total time by each tile spent in steps 4 & 5 and compare how long steps 4 & 5 actually take vs. the sum of how long each tile spent sleep and I routinely get a speedup factor of about 7.5. QThread::idealThreadCount() reports 8 on my machine, so I think I'm getting what I should expect from those steps on this machine.

I'm not sure what else to try at this point. One thing I was thinking about measuring is that even though my tile class no longer inherits from QObject, it still is a class with a constructor and some getter functions. And when I look at the combination of steps 1 & 2 where I both resize a vector of integers AND assign each one a unique ID in less than a millisecond total, but then it takes me 18 seconds to create and assign each tile, I keep wondering if there isn't room for improvement there still?

And as I was typing this whole thing up, Konstantin gave me a different approach that I'll probably pursue, but I would like to better understand how to solve the question of "if you absolutely need to have a lot of items (whatever type an "item" needs to be), what's the right design approach to be able to create and populate them quickly..."

Sean

_______________________________________________
Interest mailing list
Interest at qt-project.org
https://lists.qt-project.org/listinfo/interest