[PySide] PySide Digest, Vol 83, Issue 13
Israel Brewster
ijbrewster at alaska.edu
Mon Dec 9 21:47:10 CET 2019
> On Dec 9, 2019, at 11:34 AM, Jason H <jhihn at gmx.com> wrote:
>
> "The Pandas data frames are pure Python classes, so they are not easy to construct from C/C++, but the underlying data storage of the individual columns (see class Series source) <https://github.com/pydata/pandas/blob/master/pandas/core/series.py#L88> is numpy 1D array. Numpy has a nice C API <http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html> and you can construct the numpy arrays from C and then pass it to your Python code." - https://stackoverflow.com/questions/11607387/is-there-a-c-c-api-for-python-pandas <https://stackoverflow.com/questions/11607387/is-there-a-c-c-api-for-python-pandas>
>
> https://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html <https://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html>
>
> Ugh, it's not pretty. But I bet you could wrap the C api or target the Python API directly.
>
> If it were me though, having no idea what you are actually doing, I would problably write a json-RPC server in python and just relegate Qt to a pretty UI that makes json-RPC requests.
Interesting thought. Separate out the UI and the backend in completely separate processes that don’t interfere with each other. Definitely something to consider, although given that I need to graph the data, it might still result in a lot of data needing to be shuffled around.
As far as what I’m actually doing, it’s a data processing app that pulls in data from gas data collection flights, allows the user to select a bunch of different areas of interest, processes the data, and creates pretty graphs.
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell: 907-328-9145
> The other me would just figure out a way to bypass pandas and work out how to do it entirely in C++ (Eigen?) except for the storage format. IIRC, Numpy arrays are just c arrays at the lowest level (don't quote me on that)
>
>
>
>
> Sent: Monday, December 09, 2019 at 1:56 PM
> From: "Israel Brewster" <ijbrewster at alaska.edu>
> To: "Jason H" <jhihn at gmx.com>
> Cc: "Ognyan Moore" <ognyan.moore at gmail.com>, pyside at qt-project.org
> Subject: Re: [PySide] PySide Digest, Vol 83, Issue 13
> On Dec 9, 2019, at 9:37 AM, Jason H <jhihn at gmx.com <mailto:jhihn at gmx.com>> wrote:
>
> That article does not take into account that the QThread logic is in python. That is the problem
> The problem is the processing function is in Python. Moving it to be C++ only would work fine. Python would invoke a compiled function that would start the multithreaded processing and complete immediately, then the Qt event loop could be used via signals/slots to inform completion.
>
> Exactly. If the function could be moved to be C++ only, there wouldn’t be an issue. If the function could be executed in a different process rather than a thread, there wouldn’t be an issue. Unfortunately, the sheer volume of data involved makes moving the data between python and anything else a non-starter - the function *has* to be able to modify pandas data frames in the calling class directly - unless there is some shortcut that could make it work based on the fact that the data is in pandas data frame, and at its core pandas/numpy *is* C++ already.
>
> ---
> Israel Brewster
> Software Engineer
> Alaska Volcano Observatory
> Geophysical Institute - UAF
> 2156 Koyukuk Drive
> Fairbanks AK 99775-7320
> Work: 907-474-5172
> cell: 907-328-9145
>
>
>
>
> Sent: Monday, December 09, 2019 at 1:05 PM
> From: "Ognyan Moore" <ognyan.moore at gmail.com <mailto:ognyan.moore at gmail.com>>
> To: pyside at qt-project.org <mailto:pyside at qt-project.org>
> Subject: Re: [PySide] PySide Digest, Vol 83, Issue 13
> I think it would be great if we can have some guidance regarding multi-threading with CPU heavy tasks, not just IO bound tasks. Say I wanted to multi-thread a function
>
> def cpu_intensive_function(a: int, b: int, c: int) -> np.ndarray:
> return some_numpy_function_calls(a, b, c)
>
> How would I run this function in such a way that the computation does not hang the GUI thread? I have attempted to follow the guidance on this blog post here: https://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/ <https://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/> . Where I utilize QObject.moveToThread(some_thread) as that article suggests, the function still executes in the GUI thread.
>
> Some caveats here being that I am making an attempt to have no references to mutable objects in the GUI thread and the thread doing the computation.
>
> Thanks!
> Ogi
>
>
> On Mon, Dec 9, 2019 at 3:00 AM <pyside-request at qt-project.org <mailto:pyside-request at qt-project.org>> wrote:
> Send PySide mailing list submissions to
> pyside at qt-project.org <mailto:pyside at qt-project.org>
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside>
> or, via email, send a message with subject or body 'help' to
> pyside-request at qt-project.org <mailto:pyside-request at qt-project.org>
>
> You can reach the person managing the list at
> pyside-owner at qt-project.org <mailto:pyside-owner at qt-project.org>
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of PySide digest..."
>
>
> Today's Topics:
>
> 1. Re: Keeping GUI responsive (Sivan Greenberg)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sun, 8 Dec 2019 17:54:10 +0200
> From: Sivan Greenberg <sivan at omniqueue.com <mailto:sivan at omniqueue.com>>
> To: Israel Brewster <ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>>
> Cc: pyside at qt-project.org <mailto:pyside at qt-project.org>
> Subject: Re: [PySide] Keeping GUI responsive
> Message-ID:
> <CAJNcf-aV+Xygpo6GV+Kr-neifnEdTiWg3GYyLYg16nJQJbRhhA at mail.gmail.com <mailto:CAJNcf-aV%2BXygpo6GV%2BKr-neifnEdTiWg3GYyLYg16nJQJbRhhA at mail.gmail.com>>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Israel,
>
> If your non main function is mostly IO bound, you could try implement your
> processing function with an interface that allows for polling of results
> using co-routines which is very slim and very robust (I used that in the
> past for several PyQt & PySide small apps but also for a full blown project
> for Ubuntu back in 2006[0] , though with PyGTK - but principle is same) and
> will allow your GUI main loop to be in charge, even if you ever need to use
> a spawned shell command or program! (see links below)
>
> It seems that nowadays, there are even ready made libs for that (this one
> seems very interesting): https://github.com/reclosedev/async_gui <https://github.com/reclosedev/async_gui> and also
> supports abstraction over multiprocessing which would support you even if
> the the subtask is indeed CPU intensive (like data cleansing , validation
> etc).
>
> If you're using python 3 , the whole coroutines interface has been greatly
> simplified, and could be easier to use, however the first guide will
> explain in deep and show how to achieve cooperative multitasking using
> plain Python 2.x interface:
> -
> http://masnun.com/2015/11/13/python-generators-coroutines-native-coroutines-and-async-await.html <http://masnun.com/2015/11/13/python-generators-coroutines-native-coroutines-and-async-await.html>
> -
> https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb <https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb>
>
> HTH,
>
> -Sivan
>
> [0]:
> http://www.ubuntugeek.com/hubackup-backup-application-for-ubuntu-home-users.html <http://www.ubuntugeek.com/hubackup-backup-application-for-ubuntu-home-users.html>
>
> On Thu, Dec 5, 2019 at 1:41 AM Israel Brewster <ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>>
> wrote:
>
> > I know this is a FAQ, however I haven’t been able to make any of the
> > standard answers work for me. Here’s the situation:
> >
> > - using PySide2 5.12.2
> > - I have an object (QMainWindow subclass) that contains most of the code
> > for my application
> > - One of the functions that runs in response to user input takes around 2
> > seconds to run. The GUI obviously freezes during this time (BAD!)
> > - Said function needs to access and modify several large variables (pandas
> > data frames) from the main object
> >
> > So here’s the problem: If I run this function as a separate (python)
> > thread, that doesn’t help - the GUI is still frozen. I’m thinking this is
> > due to the GIL, but I could be wrong about that. Running under the
> > multiprocessing module, however, doesn’t appear to be an option due to the
> > number and size of the data structures that the function needs to modify,
> > and if I try just to see what happens, the process actually crashes.
> >
> > So what are my options here? How can I keep the GUI responsive while this
> > function runs, without being able to spin it off as a separate process? Or
> > is the only option going to be to completely rip apart the function and try
> > to re-build it in such a way that it can, somehow, still access the memory
> > from the main thread, while doing the processing in a separate function?
> > ---
> > Israel Brewster
> > Software Engineer
> > Alaska Volcano Observatory
> > Geophysical Institute - UAF
> > 2156 Koyukuk Drive
> > Fairbanks AK 99775-7320
> > Work: 907-474-5172
> > cell: 907-328-9145
> >
> > _______________________________________________
> > PySide mailing list
> > PySide at qt-project.org <mailto:PySide at qt-project.org>
> > https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside>
> >
>
>
> --
> -Sivan
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.qt-project.org/pipermail/pyside/attachments/20191208/fc69df57/attachment-0001.html <http://lists.qt-project.org/pipermail/pyside/attachments/20191208/fc69df57/attachment-0001.html>>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> PySide mailing list
> PySide at qt-project.org <mailto:PySide at qt-project.org>
> https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside>
>
>
> ------------------------------
>
> End of PySide Digest, Vol 83, Issue 13
> **************************************
> _______________________________________________ PySide mailing list PySide at qt-project.org <mailto:PySide at qt-project.org> https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside>_______________________________________________
> PySide mailing list
> PySide at qt-project.org <mailto:PySide at qt-project.org>
> https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/pyside/attachments/20191209/49d74347/attachment-0001.html>
More information about the PySide
mailing list