[Qt-interest] End thread when work is done

[Jason H]

I think you are missing the point of QtConcurrent. It is an easy way to add multi-processor scalability to your software. That being said, you don't want to use QtConcurrent if you want to create a thousand threads. Max threads are IIRC, 2*CPU cores. Note that since you're dealing with a file system, you will probably be I/O bound if your processTile algorithm is trivial. In general, it is bad to start so many threads because of context switching and memory usage (you'll run to disk thrashing from swap which will destroy any attempted performance gains)

If we compare your processTile function to say a modern compiler (read/process/write) we find that the recommended make -j parameter value is 2x number of cores. And indeed there seems to be limited or negative results when going over that number. 

I would also change it so that one process would read, compute and write rather than breaking it up as you have. 




----- Original Message -----
From: "Cole, Derek" <dcole at integrity-apps.com>
To: Andreas Pakulat <apaku at gmx.de>; "qt-interest at qt.nokia.com" <qt-interest at qt.nokia.com>
Cc: 
Sent: Wednesday, August 17, 2011 3:12 PM
Subject: Re: [Qt-interest] End thread when work is done

This may be the case, indeed, but I must have some way to kill the active threads. It doesnt seem like QtConcurrent will let me to create more runnables than I have maxThreadCount() - so how would I know how many threads I can make? My intent is to max out the number of threads running processing a file, when that file is finished, load another and so-on.

Here is the complete code

    QStringList files = dir.entryList();
    for(int i = 0; i < files.size(); i++)
    {

        filename = dir.absolutePath() +"/" + files.at(i);

        QFutureWatcher<void> *copywatcher;
        copywatcher = new QFutureWatcher<void>();
        QFuture<void> copyfuture = QtConcurrent::run(this, &EraserBatch::copyFile,filename);
        copywatcher->setFuture(copyfuture);

        QFutureWatcher<void> *loadwatcher;
        loadwatcher = new QFutureWatcher<void>();
        QFuture<void> loadfuture = QtConcurrent::run(this, &EraserBatch::loadTiles,filename);
        loadwatcher->setFuture(loadfuture);

        std::vector<QFutureWatcher<bool> * > procWatchers;
        std::vector<QFuture<bool> > procFutures;
        int maxThreads = QThreadPool::globalInstance()->maxThreadCount();
        int activeThreads =  QThreadPool::globalInstance()->activeThreadCount();

        for(int j = 0; j < maxThreads - activeThreads; j++ )
        {
            QFutureWatcher<bool> *procwatcher;
            procwatcher = new QFutureWatcher<bool>();
            procWatchers.push_back(procwatcher);
            QFuture<bool> procfuture = QtConcurrent::run(this, &EraserBatch::processTile);
            procFutures.push_back(procfuture);
            procwatcher->setFuture(procfuture);

        }

    //   // only do one file at once, so wait until copy/write is done
        qDebug()<<"Waiting for finish";
        copywatcher->waitForFinished();
    }

so as you can see, I have one thread loading the file, one thread creating a copy that will be over-written with results, and then I want to spawn MAX-2 threads to do the actual processing.

This seems to not be working the way I have it written here. copyWatcher does the write-out of the results, so presumably when it is done, I could do all the work over again with the same number of threads on the next file in the directory

Derek
________________________________________
From: qt-interest-bounces+dcole=integrity-apps.com at qt.nokia.com [qt-interest-bounces+dcole=integrity-apps.com at qt.nokia.com] on behalf of Andreas Pakulat [apaku at gmx.de]
Sent: Wednesday, August 17, 2011 3:00 PM
To: qt-interest at qt.nokia.com
Subject: Re: [Qt-interest] End thread when work is done

On 17.08.11 18:31:10, Cole, Derek wrote:
> Hello, I have the following:
>
>         int maxThreads = QThreadPool::globalInstance()->maxThreadCount();
>         int activeThreads =  QThreadPool::globalInstance()->activeThreadCount();
>
>         for(int j = 0; j < maxThreads - activeThreads; j++ )
>         {
>             QFutureWatcher<bool> *procwatcher;
>             procwatcher = new QFutureWatcher<bool>();
>             procWatchers.push_back(procwatcher);
>
>
>             QFuture<bool> procfuture = QtConcurrent::run(this, &EraserBatch::processTile);
>             procFutures.push_back(procfuture);
>             procwatcher->setFuture(procfuture);
>             qDebug()<<"Active threadcount after proc "<<QThreadPool::globalInstance()->activeThreadCount();
>
>         }
>
> when "processTile" is finished running for each of these, the method
> returns fine, but when I check the activeThreadCount, it shows all of
> them still running. What can I put to make sure that each thread ends
> when the method to run has ended?

I think you may mis-interpret what activeThreadCount tells you. I can't
see any indication that QThreadPool kills threads when the runnable is
done. A thread started by QThreadPool probably simply waits until more
work is to be done, otherwise the whole pool-idea would not make much
sense.

Andreas

_______________________________________________
Qt-interest mailing list
Qt-interest at qt.nokia.com
http://lists.qt.nokia.com/mailman/listinfo/qt-interest
_______________________________________________
Qt-interest mailing list
Qt-interest at qt.nokia.com
http://lists.qt.nokia.com/mailman/listinfo/qt-interest