[PySide] Keeping GUI responsive
Israel Brewster
ijbrewster at alaska.edu
Fri Dec 6 18:25:53 CET 2019
> On Dec 6, 2019, at 7:43 AM, Marian Beermann <public at enkore.de> wrote:
>
> You are observing Python's GIL which prevents the interpreter from
> executing in parallel, because the actual workload you are performing is
> written in Python and does not unlock the GIL.
>
> Using a separate thread should keep the GUI more or less responsive, but
> that doesn't always work, because Python still has issues with
> starvation when it is handling mixed workloads (CPU and I/O).
>
> -Marian
Yeah, that’s what I thought. In my case, the task is 100% CPU bound, no I/O at all. It’s simply performing a bunch of calculations on potentially large Pandas data frames, and plotting the results in matplotlib (to an offscreen “page” of a tab widget - the user isn’t waiting for these results, they’ll go look at them later). The goal here isn’t so much to do stuff in parallel - pausing the calculations to update the GUI is completely acceptable - but rather to simply keep the GUI responsive so the user doesn’t get frustrated.
I’m afraid it sounds like the only “good” solution at this time would be to figure out some way to share the pandas data frames between processes and use multiprocessing (if that’s even possible). Otherwise, I think I’m stuck with the “bad” solution of calling QApplication.processEvents() in the main thread periodically, and figuring out where in the calculation code to stick that call in order to keep things responsive. Oh well, it works. Thanks though!
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell: 907-328-9145
>
> Am 06.12.19 um 16:33 schrieb Jason H:
>> I can confirm your experience on Mojave.
>>
>> If you increase your dataset 100-fold, then I get a beach ball. Based on
>> what I am seeing here, QThread doesn't actually quite work. To test
>> this, I changed it to:
>> self.threads = [ LongRunningThread(self.large_dataset, x) for x in
>> range(24)]
>>
>> and
>> [thread.start() for thread in self.threads]
>>
>> Which should saturate my 16 core CPU, at least briefly. At no time did
>> it go over 15%. OSX's activity Monitor showed Python maxed out at 100.25%
>> The threads increase from 4 at idle to 28. But they were not
>> sistricnuted across cores as one would expect.
>> it took ~69s per thread at 32 threads
>> ~34 for 16
>> 16 for 8
>> 8 for 4
>> 4 for 2
>>
>> Which means that at no time did any of my other cores get used. It is
>> like PySide is only running on the main thread, or only running one
>> thread itself.
>>
>>
>> *Sent:* Thursday, December 05, 2019 at 3:19 PM
>> *From:* "Israel Brewster" <ijbrewster at alaska.edu>
>> *To:* "Jason H" <jhihn at gmx.com>
>> *Cc:* "Kerby Geffrard" <kerby.geffrard at autodesk.com>,
>> "pyside at qt-project.org" <pyside at qt-project.org>
>> *Subject:* Re: [PySide] Keeping GUI responsive
>>
>> On Dec 5, 2019, at 10:40 AM, Jason H <jhihn at gmx.com
>> <mailto:jhihn at gmx.com <mailto:jhihn at gmx.com>>> wrote:
>>
>> That doesn't make any sense. Can you post a minimal example?
>>
>>
>> Sure, see the end of this message (only 58 lines, so I figured inline
>> was probably fine). It’s not *quite* as extreme an effect as what I am
>> seeing in my application (probably due to over-simplification), but it
>> still illustrates the problem: click the button, and try typing quickly
>> in the the line edit. While the thread is running, typing speed is
>> noticeably reduced over when the thread is not running. The thread
>> apparently doesn’t *completely* block the GUI (I’m not sure why - maybe
>> the GIL is released between loop iterations?), but it does introduce
>> noticeable lags. It’s those lags I’m trying to get rid of.
>>
>> Simplified example:
>>
>> import time
>> from PySide2.QtWidgets import (QApplication,
>> QMainWindow,
>> QPushButton,
>> QLineEdit,
>> QVBoxLayout,
>> QWidget)
>>
>> from PySide2.QtCore import (QThread)
>>
>>
>> class LongRunningThread(QThread):
>> """A process that takes several seconds to complete, involving
>> manipulation of data
>> contained in large data structures that are members of the
>> MainWindow instance."""
>> def __init__(self, dataset):
>> super().__init__()
>> self.dataset = dataset
>>
>> def run(self):
>> print("Thread running")
>> start_time = time.time()
>> for idx, val in enumerate(self.dataset):
>> self.dataset[idx] = val * 50
>> print(f"Thread complete after: {time.time()-start_time}")
>>
>>
>> class MainWindow(QMainWindow):
>> """Main application class, contains large data structures that can
>> not/should
>> not be easily copied to another process for various reasons such as
>> memory usage."""
>>
>> def __init__(self):
>> super().__init__()
>> central_widget = QWidget(self)
>> layout = QVBoxLayout()
>> central_widget.setLayout(layout)
>> self.setCentralWidget(central_widget)
>>
>> self.text_entry = QLineEdit(self)
>> layout.addWidget(self.text_entry)
>>
>> self.button = QPushButton("Run Thread")
>> self.button.clicked.connect(self._run_blocking_thread)
>> layout.addWidget(self.button)
>>
>> # Generate a random dataset. Adjust size so operations take a while.
>> self.large_dataset = list(range(20000000))
>>
>> def _run_blocking_thread(self):
>> self.thread = LongRunningThread(self.large_dataset)
>> self.thread.start()
>> print("Thread started!")
>>
>>
>> if __name__ == "__main__":
>> app = QApplication()
>> win = MainWindow()
>> win.show()
>> app.exec_()
>>
>>
>>
>> ---
>> Israel Brewster
>> Software Engineer
>> Alaska Volcano Observatory
>> Geophysical Institute - UAF
>> 2156 Koyukuk Drive
>> Fairbanks AK 99775-7320
>> Work: 907-474-5172
>> cell: 907-328-9145
>>
>>
>>
>>
>> Sent: Thursday, December 05, 2019 at 7:16 PM
>> From: "Israel Brewster" <ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>
>> <mailto:ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>>>
>> To: "Jason H" <jhihn at gmx.com <mailto:jhihn at gmx.com> <mailto:jhihn at gmx.com <mailto:jhihn at gmx.com>>>
>> Cc: "Kerby Geffrard" <kerby.geffrard at autodesk.com <mailto:kerby.geffrard at autodesk.com>
>> <mailto:kerby.geffrard at autodesk.com <mailto:kerby.geffrard at autodesk.com>>>, "pyside at qt-project.org <mailto:pyside at qt-project.org>
>> <mailto:pyside at qt-project.org <mailto:pyside at qt-project.org>>" <pyside at qt-project.org <mailto:pyside at qt-project.org>
>> <mailto:pyside at qt-project.org <mailto:pyside at qt-project.org>>>
>> Subject: Re: [PySide] Keeping GUI responsive
>>
>>
>>
>> On Dec 5, 2019, at 7:18 AM, Jason H <jhihn at gmx.com <mailto:jhihn at gmx.com>
>> <mailto:jhihn at gmx.com <mailto:jhihn at gmx.com>>> wrote:
>>
>>
>> I don't thinkso. The Python multiprocessing module uses
>> multiple processes, so you have to copy data between them.
>> QThreads are in the same process so no copy is needed.
>> GIL only matters when you're in interpreted code.
>>
>>
>> Yes, but we’re talking about threads vs QThread - I already
>> ruled out the python multiprocessing module due to the data copy
>> issue. You are correct that the GIL only matters when you’re in
>> interpreted code, but given the behavior I’m seeing, I have to
>> assume my function *is* interpreted code - that is, the time
>> intensive portions are not calling C libraries. Otherwise
>> running the function in a separate thread would work to keep the
>> GUI responsive.
>>
>> I did try using a QThread - with the same result as using a
>> python thread. The GUI freezes while the function is running. As
>> I stated before, I can only assume this is due to the GIL. And
>> in case there is any question about if I used QThreads correctly
>> or not, here is the code I used to start the thread:
>>
>> thread = ResultsThread(self)
>> thread.resultsReady.connect(self._on_results_ready)
>> thread.start()
>>
>> ResultsThread is, of course, a QThread subclass in which I
>> re-implemented the run() function to do my calculations.
>>
>> As a further test, I started another thread that just spit out
>> the “current” time (as per time.time()) every .2 seconds. What
>> I’m seeing is that under normal operation, this other thread
>> outputs as expected every (approximately) .2 seconds. However,
>> while the QThread (or python thread) is running, this other
>> thread “stutters”. This indicates that while the QThread
>> operation doesn’t block other threads completely, there are a
>> number of operations taking place in it that *DO* block other
>> threads for a noticeable length of time, causing the next
>> iteration of my timer loop to be delayed. Such blocking would
>> obviously apply to the main GUI thread as well.
>>
>> So yes, running in a QThread blocks the main GUI thread the same
>> as running in a python thread does.
>> ---
>> Israel Brewster
>> Software Engineer
>> Alaska Volcano Observatory
>> Geophysical Institute - UAF
>> 2156 Koyukuk Drive
>> Fairbanks AK 99775-7320
>> Work: 907-474-5172
>> cell: 907-328-9145
>>
>>
>>
>>
>> Sent: Thursday, December 05, 2019 at 9:42 AM
>> From: "Israel Brewster" <ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>
>> <mailto:ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>>>
>> To: "Kerby Geffrard" <kerby.geffrard at autodesk.com <mailto:kerby.geffrard at autodesk.com>
>> <mailto:kerby.geffrard at autodesk.com <mailto:kerby.geffrard at autodesk.com>>>
>> Cc: "pyside at qt-project.org <mailto:pyside at qt-project.org> <mailto:pyside at qt-project.org <mailto:pyside at qt-project.org>>"
>> <pyside at qt-project.org <mailto:pyside at qt-project.org> <mailto:pyside at qt-project.org <mailto:pyside at qt-project.org>>>
>> Subject: Re: [PySide] Keeping GUI responsive
>>
>> On Dec 4, 2019, at 4:53 PM, Kerby Geffrard
>> <kerby.geffrard at autodesk.com <mailto:kerby.geffrard at autodesk.com>
>> <mailto:kerby.geffrard at autodesk.com <mailto:kerby.geffrard at autodesk.com>> <mailto:kerby.geffrard at autodesk.com <mailto:kerby.geffrard at autodesk.com>>>
>> wrote:
>>
>> I think you can try to use a QThread for this.
>>
>> I was under the impression that QThreads had the same
>> limitations that regular python threads had, but I’ll give
>> it a shot. The need to subclass QThread rather than simply
>> being able to execute a function complicates things however
>> - I have to make sure that said subclass has access to all
>> the original class members it needs.
>> ---
>> Israel Brewster
>> Software Engineer
>> Alaska Volcano Observatory
>> Geophysical Institute - UAF
>> 2156 Koyukuk Drive
>> Fairbanks AK 99775-7320
>> Work: 907-474-5172
>> cell: 907-328-9145
>>
>>
>> Le 4 déc. 2019 à 18:41, Israel Brewster
>> <ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>
>> <mailto:ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>> <mailto:ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>>>
>> a écrit :
>>
>> I know this is a FAQ, however I haven’t been able to make
>> any of the standard answers work for me. Here’s the situation:
>>
>> - using PySide2 5.12.2
>> - I have an object (QMainWindow subclass) that contains most
>> of the code for my application
>> - One of the functions that runs in response to user input
>> takes around 2 seconds to run. The GUI obviously freezes
>> during this time (BAD!)
>> - Said function needs to access and modify several large
>> variables (pandas data frames) from the main object
>>
>> So here’s the problem: If I run this function as a separate
>> (python) thread, that doesn’t help - the GUI is still
>> frozen. I’m thinking this is due to the GIL, but I could be
>> wrong about that. Running under the multiprocessing module,
>> however, doesn’t appear to be an option due to the number
>> and size of the data structures that the function needs to
>> modify, and if I try just to see what happens, the process
>> actually crashes.
>>
>> So what are my options here? How can I keep the GUI
>> responsive while this function runs, without being able to
>> spin it off as a separate process? Or is the only option
>> going to be to completely rip apart the function and try to
>> re-build it in such a way that it can, somehow, still access
>> the memory from the main thread, while doing the processing
>> in a separate function?
>> ---
>> Israel Brewster
>> Software Engineer
>> Alaska Volcano Observatory
>> Geophysical Institute - UAF
>> 2156 Koyukuk Drive
>> Fairbanks AK 99775-7320
>> Work: 907-474-5172
>> cell: 907-328-9145
>> _______________________________________________
>> PySide mailing list
>> PySide at qt-project.org <mailto:PySide at qt-project.org>
>> <mailto:PySide at qt-project.org <mailto:PySide at qt-project.org>> <mailto:PySide at qt-project.org <mailto:PySide at qt-project.org>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.qt-2Dproject.org_listinfo_pyside&d=DwIGaQ&c=76Q6Tcqc-t2x0ciWn7KFdCiqt6IQ7a_IF9uzNzd_2pA&r=Sh-EhW_r32FEgrsAfK3Hc0du2ebOZ7PrbchABO_xtMo&m=r4Ron7tNrQN9BNhzXVlsKvYoyWrnjjOzLwALVV_CgnA&s=T2UjSQWTnYFAfFbfdTlMgLZnfC1dlXhNy1SmPijkWbs&e=<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.qt-2Dproject.org_listinfo_pyside&d=DwIGaQ&c=76Q6Tcqc-t2x0ciWn7KFdCiqt6IQ7a_IF9uzNzd_2pA&r=Sh-EhW_r32FEgrsAfK3Hc0du2ebOZ7PrbchABO_xtMo&m=r4Ron7tNrQN9BNhzXVlsKvYoyWrnjjOzLwALVV_CgnA&s=T2UjSQWTnYFAfFbfdTlMgLZnfC1dlXhNy1SmPijkWbs&e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.qt-2Dproject.org_listinfo_pyside&d=DwIGaQ&c=76Q6Tcqc-t2x0ciWn7KFdCiqt6IQ7a_IF9uzNzd_2pA&r=Sh-EhW_r32FEgrsAfK3Hc0du2ebOZ7PrbchABO_xtMo&m=r4Ron7tNrQN9BNhzXVlsKvYoyWrnjjOzLwALVV_CgnA&s=T2UjSQWTnYFAfFbfdTlMgLZnfC1dlXhNy1SmPijkWbs&e=%3Chttps://urldefense.proofpoint.com/v2/url?u=https-3A__lists.qt-2Dproject.org_listinfo_pyside&d=DwIGaQ&c=76Q6Tcqc-t2x0ciWn7KFdCiqt6IQ7a_IF9uzNzd_2pA&r=Sh-EhW_r32FEgrsAfK3Hc0du2ebOZ7PrbchABO_xtMo&m=r4Ron7tNrQN9BNhzXVlsKvYoyWrnjjOzLwALVV_CgnA&s=T2UjSQWTnYFAfFbfdTlMgLZnfC1dlXhNy1SmPijkWbs&e=%3E>
>> _______________________________________________ PySide
>> mailing list PySide at qt-project.org <mailto:PySide at qt-project.org>
>> <mailto:PySide at qt-project.org <mailto:PySide at qt-project.org>> https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside> <https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside>>
>>
>>
>> _______________________________________________
>> PySide mailing list
>> PySide at qt-project.org <mailto:PySide at qt-project.org>
>> https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/pyside/attachments/20191206/eb95536b/attachment-0001.html>
More information about the PySide
mailing list