[PySide] PySide2 + multiprocessing + requests = crash?
ijbrewster at alaska.edu
Wed Dec 18 18:56:36 CET 2019
> On Dec 18, 2019, at 2:56 AM, Jason H <jhihn at gmx.com> wrote:
> I've done something similar in there past. I would generate the images on the main thread, but then send it to a QThread/QRunnable for saving because I was using PNG and the save took far longer than anything else. All my cores were saturated with saves and the QImage generation was 100s of frames ahead. Fortunately, I had enough RAM to render the sequence.
> Maybe something like that would work better? But I did it all in C++ so no GIL.
Unfortunately not - my time is spent in the generation/data load, not in the saving. The final images are only 800x1000 pixels, so they save quite quickly. It’s the generation I need to parallelize.
Using threads may work - most of the time is spent in either I/O or numpy array operations (which should be C, and so GIL shouldn’t be too much of an issue), but it seems like processes are simply a better option here.
That said, I did figure out a workaround - spin any pre-qt web work off to a child process. A bit kludgy, but by doing that it only “breaks” the child process, not the main process, allowing me to spin up as many child processes as I want to run the Qt code without issue.
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
>> Sent: Tuesday, December 17, 2019 at 8:06 PM
>> From: "Israel Brewster" <ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>>
>> To: "Jason H" <jhihn at gmx.com <mailto:jhihn at gmx.com>>
>> Cc: pyside at qt-project.org <mailto:pyside at qt-project.org>
>> Subject: Re: [PySide] PySide2 + multiprocessing + requests = crash?
>>> On Dec 17, 2019, at 9:56 AM, Jason H <jhihn at gmx.com> wrote:
>>> QApplication has to be on the main thread.
>>> So I would guess (without seeing the crash) that yes, it's bad form.
>> It *is* on the main thread. Of a child process, true, but the main thread of that process. Multiprocessing, not multithreading. Presumably that’s why it generally works - as long as I don’t try to make a web request :-)
>>> Generally when I want to do something like ehat you are doing, I set a singlshot timer to fire (interval: 0) start the event loop (app.exec() or app.exec_() in python) then the event loop wth execute the timer code.
>> No event loop here. I’m just creating QWidgets, setting contents, and using .grab() to save the contents as an image, after which the widgets are discarded. The only reason I have a QApplication at all is that you can’t create QWidgets without one.
>> There is no user interaction, no displayed GUI at any point. In fact, I’m initializing QApplication with a platform of offscreen. I just want to work with multiple QWidgets simultaneously to generate multiple images in parallel - thus the need for multiprocessing.
>>> Sent: Tuesday, December 17, 2019 at 1:13 PM
>>> From: "Israel Brewster" <ijbrewster at alaska.edu>
>>> To: pyside at qt-project.org
>>> Subject: [PySide] PySide2 + multiprocessing + requests = crash?
>>> I’ve found what seems to be an odd crash to me. Using PySide2 and multiprocessing works fine to create a QAppliction in a separate process (unless this is considered bad form?) - unless I call requests.get at some point before starting said process, in which case the process crashes (hard crash, not python exception). How might I fix this?
>>> A little background: I’m using Qt classes to generate a bunch of images from data files. To speed things along, I launch a process (using multiprocessing.Process) for each image I want to generate, and in each process I create a QApplication, create and fill the widgets I need for the image, and then save the image to a file. This appears to work fine, creating the images in parallel. Today I tried to do a requests call before doing any of the processing (simple get, doesn’t interact with QT in any way), and suddenly my code started crashing.
>>> Some simplified sample code that illustrates the issue:
>>> import time
>>> import multiprocessing
>>> import sys
>>> from PySide2.QtWidgets import QApplication
>>> import requests
>>> def test_wait():
>>> print("Starting proc")
>>> QApplication(sys.argv + ['-platform', 'offscreen'])
>>> except Exception as err:
>>> print("oops", err)
>>> print("Sleeping now...")
>>> if __name__ == "__main__":
>>> proc = multiprocessing.Process(target=test_wait)
>>> print("Awake again. Let's try another:")
>>> res = requests.get('https://www.google.com <https://www.google.com/> <https://www.google.com/ <https://www.google.com/>>') # or whatever your favorite is
>>> res.text # Get the response, close it out
>>> proc = multiprocessing.Process(target=test_wait)
>>> start = time.time()
>>> # The process should sleep for 10 seconds, so if we got here in less than 9, we died.
>>> if time.time() - start < 9:
>>> print("Never slept :(")
>>> print("We lived!")
>>> Israel Brewster
>>> Software Engineer
>>> Alaska Volcano Observatory
>>> Geophysical Institute - UAF
>>> 2156 Koyukuk Drive
>>> Fairbanks AK 99775-7320
>>> Work: 907-474-5172
>>> cell: 907-328-9145
>>> _______________________________________________ PySide mailing list PySide at qt-project.org <mailto:PySide at qt-project.org> https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside> <https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside>>
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the PySide