[PySide] PySide Digest, Vol 83, Issue 13

Tue Dec 10 17:26:48 CET 2019

Then I suppose a natural follow up question would be, what is the point of
QThread being exposed in PySide?  Qt already offers QtConcurrent to handy
asynchronous tasks.

On Tue, Dec 10, 2019 at 7:16 AM Jason H <jhihn at gmx.com> wrote:

> Not really.
>
> Maybe you need to flip your application in-side out? Make it a C++ program
> with all the threads it needs and just embed a python interpreter for your
> processing functions? Not sure how you'll efficiently move data between the
> two sides though. For OpenCV (c++), it's lucky that Qt and OpenCV support a
> few same memory formats so that you can just take the data as an opaque
> pointer.
>
> As long as you want to code your processing of data in Python, you'll have
> the GIL, with will have this problem. C++ Qt can deal with it with no
> problem.
>
> Maybe you could use some Async python (Python 3 only) and integrate Qt
> event loop with the python aync event loop and maybe do it that way? But
> I'm not a fan of Python async (Qt does it better, and the event loop
> integration isn't offically supported. Python's event loop  is nvery new
> and API isn't stable.)
>
> *Sent:* Monday, December 09, 2019 at 6:14 PM
> *From:* "Ognyan Moore" <ognyan.moore at gmail.com>
> *To:* "Jason H" <jhihn at gmx.com>
> *Cc:* "Israel Brewster" <ijbrewster at alaska.edu>, pyside at qt-project.org
> *Subject:* Re: [PySide] PySide Digest, Vol 83, Issue 13
> In my case, I'm trying to do very rapid spectrogram calculations, ....lots
> of them, they each take ~40-100ms but the fact that I do these calculations
> very rapidly makes it problematic for them occurring in the GUI thread.
> These calculations are triggered by GUI events in pyqtgraph (user adjusts a
> plot window in some way); and I attempt to redraw a spectrogram calculation
> in as close to real-time as possible.  Now, I can make it so that the
> function running on a separate thread _never_ has to access some reference
> in the main thread.  Clearly there must be some way to spawn up a longish
> numpy task that does not effect the GUI.
>
>
>
> On Mon, Dec 9, 2019 at 3:07 PM Jason H <jhihn at gmx.com> wrote:
>
>> Before anyone points it out, I'm not suggesting Qt is the problem ;-)
>> Just that if it is a data-transform-heavy task shuffeling between Qt and
>> Python might be ok for display and let the server handle the processing.
>>
>>
>>
>> *Sent:* Monday, December 09, 2019 at 3:47 PM
>> *From:* "Israel Brewster" <ijbrewster at alaska.edu>
>> *To:* "Jason H" <jhihn at gmx.com>
>> *Cc:* "Ognyan Moore" <ognyan.moore at gmail.com>, pyside at qt-project.org
>> *Subject:* Re: [PySide] PySide Digest, Vol 83, Issue 13
>>
>> On Dec 9, 2019, at 11:34 AM, Jason H <jhihn at gmx.com> wrote:
>>
>> "The Pandas data frames are pure Python classes, so they are not easy to
>> construct from C/C++, but the underlying data storage of the individual
>> columns (see class Series source)
>> <https://github.com/pydata/pandas/blob/master/pandas/core/series.py#L88> is
>> numpy 1D array. Numpy has a nice C API
>> <http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html> and you
>> can construct the numpy arrays from C and then pass it to your Python
>> code." -
>> https://stackoverflow.com/questions/11607387/is-there-a-c-c-api-for-python-pandas
>>
>> https://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html
>>
>> Ugh, it's not pretty. But I bet you could wrap the C api or target the
>> Python API directly.
>>
>> If it were me though, having no idea what you are actually doing, I would
>> problably write a json-RPC server in python and just relegate Qt to a
>> pretty UI that makes json-RPC requests.
>>
>>
>> Interesting thought. Separate out the UI and the backend in completely
>> separate processes that don’t interfere with each other. Definitely
>> something to consider, although given that I need to graph the data, it
>> might still result in a lot of data needing to be shuffled around.
>>
>> As far as what I’m actually doing, it’s a data processing app that pulls
>> in data from gas data collection flights, allows the user to select a bunch
>> of different areas of interest, processes the data, and creates pretty
>> graphs.
>>
>> ---
>> Israel Brewster
>> Software Engineer
>> Alaska Volcano Observatory
>> Geophysical Institute - UAF
>> 2156 Koyukuk Drive
>> Fairbanks AK 99775-7320
>> Work: 907-474-5172
>> cell:  907-328-9145
>>
>>
>> The other me would just figure out a way to bypass pandas and work out
>> how to do it entirely in C++ (Eigen?) except for the storage format. IIRC,
>> Numpy arrays are just c arrays at the lowest level (don't quote me on that)
>>
>>
>>
>>
>> *Sent:* Monday, December 09, 2019 at 1:56 PM
>> *From:* "Israel Brewster" <ijbrewster at alaska.edu>
>> *To:* "Jason H" <jhihn at gmx.com>
>> *Cc:* "Ognyan Moore" <ognyan.moore at gmail.com>, pyside at qt-project.org
>> *Subject:* Re: [PySide] PySide Digest, Vol 83, Issue 13
>>
>> On Dec 9, 2019, at 9:37 AM, Jason H <jhihn at gmx.com> wrote:
>>
>> That article does not take into account that the QThread logic is in
>> python. That is the problem
>> The problem is the processing function is in Python. Moving it to be C++
>> only would work fine. Python would invoke a compiled function that would
>> start the multithreaded processing and complete immediately, then the Qt
>> event loop could be used via signals/slots to inform completion.
>>
>>
>> Exactly. If the function could be moved to be C++ only, there wouldn’t be
>> an issue. If the function could be executed in a different process rather
>> than a thread, there wouldn’t be an issue. Unfortunately, the sheer volume
>> of data involved makes moving the data between python and anything else a
>> non-starter - the function *has* to be able to modify pandas data frames in
>> the calling class directly - unless there is some shortcut that could make
>> it work based on the fact that the data is in pandas data frame, and at its
>> core pandas/numpy *is* C++ already.
>>
>> ---
>> Israel Brewster
>> Software Engineer
>> Alaska Volcano Observatory
>> Geophysical Institute - UAF
>> 2156 Koyukuk Drive
>> Fairbanks AK 99775-7320
>> Work: 907-474-5172
>> cell:  907-328-9145
>>
>>
>>
>>
>>
>> *Sent:* Monday, December 09, 2019 at 1:05 PM
>> *From:* "Ognyan Moore" <ognyan.moore at gmail.com>
>> *To:* pyside at qt-project.org
>> *Subject:* Re: [PySide] PySide Digest, Vol 83, Issue 13
>> I think it would be great if we can have some guidance regarding
>> multi-threading with CPU heavy tasks, not just IO bound tasks.  Say I
>> wanted to multi-thread a function
>>
>>
>> def cpu_intensive_function(a: int, b: int, c: int) -> np.ndarray:
>>     return some_numpy_function_calls(a, b, c)
>>
>>
>> How would I run this function in such a way that the computation does not
>> hang the GUI thread?  I have attempted to follow the guidance on this blog
>> post here:
>> https://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/ .
>> Where I utilize QObject.moveToThread(some_thread) as that article suggests,
>> the function still executes in the GUI thread.
>>
>> Some caveats here being that I am making an attempt to have no references
>> to mutable objects in the GUI thread and the thread doing the computation.
>>
>> Thanks!
>> Ogi
>>
>>
>> On Mon, Dec 9, 2019 at 3:00 AM <pyside-request at qt-project.org> wrote:
>>
>>> Send PySide mailing list submissions to
>>>         pyside at qt-project.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>         https://lists.qt-project.org/listinfo/pyside
>>> or, via email, send a message with subject or body 'help' to
>>>         pyside-request at qt-project.org
>>>
>>> You can reach the person managing the list at
>>>         pyside-owner at qt-project.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of PySide digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>    1. Re: Keeping GUI responsive (Sivan Greenberg)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Sun, 8 Dec 2019 17:54:10 +0200
>>> From: Sivan Greenberg <sivan at omniqueue.com>
>>> To: Israel Brewster <ijbrewster at alaska.edu>
>>> Cc: pyside at qt-project.org
>>> Subject: Re: [PySide] Keeping GUI responsive
>>> Message-ID:
>>>         <
>>> CAJNcf-aV+Xygpo6GV+Kr-neifnEdTiWg3GYyLYg16nJQJbRhhA at mail.gmail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> Hi Israel,
>>>
>>>  If your non main function is mostly IO bound, you could try implement
>>> your
>>> processing function with an interface that allows for polling of results
>>> using co-routines which is very slim and very robust (I used that in the
>>> past for several PyQt & PySide small apps but also for a full blown
>>> project
>>> for Ubuntu back in 2006[0] , though with PyGTK - but principle is same)
>>> and
>>> will allow your GUI main loop to be in charge, even if you ever need to
>>> use
>>> a spawned shell command or program! (see links below)
>>>
>>>  It seems that nowadays, there are even ready made libs for that (this
>>> one
>>> seems very interesting): https://github.com/reclosedev/async_gui and
>>> also
>>> supports abstraction over multiprocessing which would support you even if
>>> the the subtask is indeed CPU intensive (like data cleansing , validation
>>> etc).
>>>
>>>  If you're using python 3 , the whole coroutines interface has been
>>> greatly
>>> simplified, and could be easier to use, however the first guide will
>>> explain in deep and show how to achieve cooperative multitasking using
>>> plain Python 2.x interface:
>>> -
>>>
>>> http://masnun.com/2015/11/13/python-generators-coroutines-native-coroutines-and-async-await.html
>>> -
>>>
>>> https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb
>>>
>>> HTH,
>>>
>>> -Sivan
>>>
>>> [0]:
>>>
>>> http://www.ubuntugeek.com/hubackup-backup-application-for-ubuntu-home-users.html
>>>
>>> On Thu, Dec 5, 2019 at 1:41 AM Israel Brewster <ijbrewster at alaska.edu>
>>> wrote:
>>>
>>> > I know this is a FAQ, however I haven’t been able to make any of the
>>> > standard answers work for me. Here’s the situation:
>>> >
>>> > - using PySide2 5.12.2
>>> > - I have an object (QMainWindow subclass) that contains most of the
>>> code
>>> > for my application
>>> > - One of the functions that runs in response to user input takes
>>> around 2
>>> > seconds to run. The GUI obviously freezes during this time (BAD!)
>>> > - Said function needs to access and modify several large variables
>>> (pandas
>>> > data frames) from the main object
>>> >
>>> > So here’s the problem: If I run this function as a separate (python)
>>> > thread, that doesn’t help - the GUI is still frozen. I’m thinking this
>>> is
>>> > due to the GIL, but I could be wrong about that. Running under the
>>> > multiprocessing module, however, doesn’t appear to be an option due to
>>> the
>>> > number and size of the data structures that the function needs to
>>> modify,
>>> > and if I try just to see what happens, the process actually crashes.
>>> >
>>> > So what are my options here? How can I keep the GUI responsive while
>>> this
>>> > function runs, without being able to spin it off as a separate
>>> process? Or
>>> > is the only option going to be to completely rip apart the function
>>> and try
>>> > to re-build it in such a way that it can, somehow, still access the
>>> memory
>>> > from the main thread, while doing the processing in a separate
>>> function?
>>> > ---
>>> > Israel Brewster
>>> > Software Engineer
>>> > Alaska Volcano Observatory
>>> > Geophysical Institute - UAF
>>> > 2156 Koyukuk Drive
>>> > Fairbanks AK 99775-7320
>>> > Work: 907-474-5172
>>> > cell:  907-328-9145
>>> >
>>> > _______________________________________________
>>> > PySide mailing list
>>> > PySide at qt-project.org
>>> > https://lists.qt-project.org/listinfo/pyside
>>> >
>>>
>>>
>>> --
>>> -Sivan
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <
>>> http://lists.qt-project.org/pipermail/pyside/attachments/20191208/fc69df57/attachment-0001.html
>>> >
>>>
>>> ------------------------------
>>>
>>> Subject: Digest Footer
>>>
>>> _______________________________________________
>>> PySide mailing list
>>> PySide at qt-project.org
>>> https://lists.qt-project.org/listinfo/pyside
>>>
>>>
>>> ------------------------------
>>>
>>> End of PySide Digest, Vol 83, Issue 13
>>> **************************************
>>
>> _______________________________________________ PySide mailing list
>> PySide at qt-project.org https://lists.qt-project.org/listinfo/pyside
>> _______________________________________________
>> PySide mailing list
>> PySide at qt-project.org
>> https://lists.qt-project.org/listinfo/pyside
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/pyside/attachments/20191210/f1fcfb46/attachment-0001.html>