[PySide] PySide Digest, Vol 83, Issue 13

Tue Dec 10 00:14:06 CET 2019

In my case, I'm trying to do very rapid spectrogram calculations, ....lots
of them, they each take ~40-100ms but the fact that I do these calculations
very rapidly makes it problematic for them occurring in the GUI thread.
These calculations are triggered by GUI events in pyqtgraph (user adjusts a
plot window in some way); and I attempt to redraw a spectrogram calculation
in as close to real-time as possible.  Now, I can make it so that the
function running on a separate thread _never_ has to access some reference
in the main thread.  Clearly there must be some way to spawn up a longish
numpy task that does not effect the GUI.

On Mon, Dec 9, 2019 at 3:07 PM Jason H <jhihn at gmx.com> wrote:

> Before anyone points it out, I'm not suggesting Qt is the problem ;-) Just
> that if it is a data-transform-heavy task shuffeling between Qt and Python
> might be ok for display and let the server handle the processing.
>
>
>
> *Sent:* Monday, December 09, 2019 at 3:47 PM
> *From:* "Israel Brewster" <ijbrewster at alaska.edu>
> *To:* "Jason H" <jhihn at gmx.com>
> *Cc:* "Ognyan Moore" <ognyan.moore at gmail.com>, pyside at qt-project.org
> *Subject:* Re: [PySide] PySide Digest, Vol 83, Issue 13
>
> On Dec 9, 2019, at 11:34 AM, Jason H <jhihn at gmx.com> wrote:
>
> "The Pandas data frames are pure Python classes, so they are not easy to
> construct from C/C++, but the underlying data storage of the individual
> columns (see class Series source)
> <https://github.com/pydata/pandas/blob/master/pandas/core/series.py#L88> is
> numpy 1D array. Numpy has a nice C API
> <http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html> and you can
> construct the numpy arrays from C and then pass it to your Python code." -
> https://stackoverflow.com/questions/11607387/is-there-a-c-c-api-for-python-pandas
>
> https://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html
>
> Ugh, it's not pretty. But I bet you could wrap the C api or target the
> Python API directly.
>
> If it were me though, having no idea what you are actually doing, I would
> problably write a json-RPC server in python and just relegate Qt to a
> pretty UI that makes json-RPC requests.
>
>
> Interesting thought. Separate out the UI and the backend in completely
> separate processes that don’t interfere with each other. Definitely
> something to consider, although given that I need to graph the data, it
> might still result in a lot of data needing to be shuffled around.
>
> As far as what I’m actually doing, it’s a data processing app that pulls
> in data from gas data collection flights, allows the user to select a bunch
> of different areas of interest, processes the data, and creates pretty
> graphs.
>
> ---
> Israel Brewster
> Software Engineer
> Alaska Volcano Observatory
> Geophysical Institute - UAF
> 2156 Koyukuk Drive
> Fairbanks AK 99775-7320
> Work: 907-474-5172
> cell:  907-328-9145
>
>
> The other me would just figure out a way to bypass pandas and work out how
> to do it entirely in C++ (Eigen?) except for the storage format. IIRC,
> Numpy arrays are just c arrays at the lowest level (don't quote me on that)
>
>
>
>
> *Sent:* Monday, December 09, 2019 at 1:56 PM
> *From:* "Israel Brewster" <ijbrewster at alaska.edu>
> *To:* "Jason H" <jhihn at gmx.com>
> *Cc:* "Ognyan Moore" <ognyan.moore at gmail.com>, pyside at qt-project.org
> *Subject:* Re: [PySide] PySide Digest, Vol 83, Issue 13
>
> On Dec 9, 2019, at 9:37 AM, Jason H <jhihn at gmx.com> wrote:
>
> That article does not take into account that the QThread logic is in
> python. That is the problem
> The problem is the processing function is in Python. Moving it to be C++
> only would work fine. Python would invoke a compiled function that would
> start the multithreaded processing and complete immediately, then the Qt
> event loop could be used via signals/slots to inform completion.
>
>
> Exactly. If the function could be moved to be C++ only, there wouldn’t be
> an issue. If the function could be executed in a different process rather
> than a thread, there wouldn’t be an issue. Unfortunately, the sheer volume
> of data involved makes moving the data between python and anything else a
> non-starter - the function *has* to be able to modify pandas data frames in
> the calling class directly - unless there is some shortcut that could make
> it work based on the fact that the data is in pandas data frame, and at its
> core pandas/numpy *is* C++ already.
>
> ---
> Israel Brewster
> Software Engineer
> Alaska Volcano Observatory
> Geophysical Institute - UAF
> 2156 Koyukuk Drive
> Fairbanks AK 99775-7320
> Work: 907-474-5172
> cell:  907-328-9145
>
>
>
>
>
> *Sent:* Monday, December 09, 2019 at 1:05 PM
> *From:* "Ognyan Moore" <ognyan.moore at gmail.com>
> *To:* pyside at qt-project.org
> *Subject:* Re: [PySide] PySide Digest, Vol 83, Issue 13
> I think it would be great if we can have some guidance regarding
> multi-threading with CPU heavy tasks, not just IO bound tasks.  Say I
> wanted to multi-thread a function
>
>
> def cpu_intensive_function(a: int, b: int, c: int) -> np.ndarray:
>     return some_numpy_function_calls(a, b, c)
>
>
> How would I run this function in such a way that the computation does not
> hang the GUI thread?  I have attempted to follow the guidance on this blog
> post here:
> https://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/ .
> Where I utilize QObject.moveToThread(some_thread) as that article suggests,
> the function still executes in the GUI thread.
>
> Some caveats here being that I am making an attempt to have no references
> to mutable objects in the GUI thread and the thread doing the computation.
>
> Thanks!
> Ogi
>
>
> On Mon, Dec 9, 2019 at 3:00 AM <pyside-request at qt-project.org> wrote:
>
>> Send PySide mailing list submissions to
>>         pyside at qt-project.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://lists.qt-project.org/listinfo/pyside
>> or, via email, send a message with subject or body 'help' to
>>         pyside-request at qt-project.org
>>
>> You can reach the person managing the list at
>>         pyside-owner at qt-project.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of PySide digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: Keeping GUI responsive (Sivan Greenberg)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sun, 8 Dec 2019 17:54:10 +0200
>> From: Sivan Greenberg <sivan at omniqueue.com>
>> To: Israel Brewster <ijbrewster at alaska.edu>
>> Cc: pyside at qt-project.org
>> Subject: Re: [PySide] Keeping GUI responsive
>> Message-ID:
>>         <
>> CAJNcf-aV+Xygpo6GV+Kr-neifnEdTiWg3GYyLYg16nJQJbRhhA at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi Israel,
>>
>>  If your non main function is mostly IO bound, you could try implement
>> your
>> processing function with an interface that allows for polling of results
>> using co-routines which is very slim and very robust (I used that in the
>> past for several PyQt & PySide small apps but also for a full blown
>> project
>> for Ubuntu back in 2006[0] , though with PyGTK - but principle is same)
>> and
>> will allow your GUI main loop to be in charge, even if you ever need to
>> use
>> a spawned shell command or program! (see links below)
>>
>>  It seems that nowadays, there are even ready made libs for that (this one
>> seems very interesting): https://github.com/reclosedev/async_gui and also
>> supports abstraction over multiprocessing which would support you even if
>> the the subtask is indeed CPU intensive (like data cleansing , validation
>> etc).
>>
>>  If you're using python 3 , the whole coroutines interface has been
>> greatly
>> simplified, and could be easier to use, however the first guide will
>> explain in deep and show how to achieve cooperative multitasking using
>> plain Python 2.x interface:
>> -
>>
>> http://masnun.com/2015/11/13/python-generators-coroutines-native-coroutines-and-async-await.html
>> -
>>
>> https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb
>>
>> HTH,
>>
>> -Sivan
>>
>> [0]:
>>
>> http://www.ubuntugeek.com/hubackup-backup-application-for-ubuntu-home-users.html
>>
>> On Thu, Dec 5, 2019 at 1:41 AM Israel Brewster <ijbrewster at alaska.edu>
>> wrote:
>>
>> > I know this is a FAQ, however I haven’t been able to make any of the
>> > standard answers work for me. Here’s the situation:
>> >
>> > - using PySide2 5.12.2
>> > - I have an object (QMainWindow subclass) that contains most of the code
>> > for my application
>> > - One of the functions that runs in response to user input takes around
>> 2
>> > seconds to run. The GUI obviously freezes during this time (BAD!)
>> > - Said function needs to access and modify several large variables
>> (pandas
>> > data frames) from the main object
>> >
>> > So here’s the problem: If I run this function as a separate (python)
>> > thread, that doesn’t help - the GUI is still frozen. I’m thinking this
>> is
>> > due to the GIL, but I could be wrong about that. Running under the
>> > multiprocessing module, however, doesn’t appear to be an option due to
>> the
>> > number and size of the data structures that the function needs to
>> modify,
>> > and if I try just to see what happens, the process actually crashes.
>> >
>> > So what are my options here? How can I keep the GUI responsive while
>> this
>> > function runs, without being able to spin it off as a separate process?
>> Or
>> > is the only option going to be to completely rip apart the function and
>> try
>> > to re-build it in such a way that it can, somehow, still access the
>> memory
>> > from the main thread, while doing the processing in a separate function?
>> > ---
>> > Israel Brewster
>> > Software Engineer
>> > Alaska Volcano Observatory
>> > Geophysical Institute - UAF
>> > 2156 Koyukuk Drive
>> > Fairbanks AK 99775-7320
>> > Work: 907-474-5172
>> > cell:  907-328-9145
>> >
>> > _______________________________________________
>> > PySide mailing list
>> > PySide at qt-project.org
>> > https://lists.qt-project.org/listinfo/pyside
>> >
>>
>>
>> --
>> -Sivan
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.qt-project.org/pipermail/pyside/attachments/20191208/fc69df57/attachment-0001.html
>> >
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> PySide mailing list
>> PySide at qt-project.org
>> https://lists.qt-project.org/listinfo/pyside
>>
>>
>> ------------------------------
>>
>> End of PySide Digest, Vol 83, Issue 13
>> **************************************
>
> _______________________________________________ PySide mailing list
> PySide at qt-project.org https://lists.qt-project.org/listinfo/pyside
> _______________________________________________
> PySide mailing list
> PySide at qt-project.org
> https://lists.qt-project.org/listinfo/pyside
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/pyside/attachments/20191209/2e7c4818/attachment-0001.html>