[PySide] PySide2 + multiprocessing + requests = crash?

Wed Dec 18 18:56:36 CET 2019

> On Dec 18, 2019, at 2:56 AM, Jason H <jhihn at gmx.com> wrote:
> 
> I've done something similar in there past. I would generate the images on the main thread, but then send it to a QThread/QRunnable for saving because I was using PNG and the save took far longer than anything else. All my cores were saturated with saves and the QImage generation was 100s of frames ahead. Fortunately, I had enough RAM to render the sequence. 
> 
> Maybe something like that would work better? But I did it all in C++ so no GIL. 

Unfortunately not - my time is spent in the generation/data load, not in the saving. The final images are only 800x1000 pixels, so they save quite quickly. It’s the generation I need to parallelize.

Using threads may work - most of the time is spent in either I/O or numpy array operations (which should be C, and so GIL shouldn’t be too much of an issue), but it seems like processes are simply a better option here.

That said, I did figure out a workaround - spin any pre-qt web work off to a child process. A bit kludgy, but by doing that it only “breaks” the child process, not the main process, allowing me to spin up as many child processes as I want to run the Qt code without issue.
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory 
Geophysical Institute - UAF 
2156 Koyukuk Drive 
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145

> 
>> Sent: Tuesday, December 17, 2019 at 8:06 PM
>> From: "Israel Brewster" <ijbrewster at alaska.edu <mailto:ijbrewster at alaska.edu>>
>> To: "Jason H" <jhihn at gmx.com <mailto:jhihn at gmx.com>>
>> Cc: pyside at qt-project.org <mailto:pyside at qt-project.org>
>> Subject: Re: [PySide] PySide2 + multiprocessing + requests = crash?
>> 
>> 
>>> On Dec 17, 2019, at 9:56 AM, Jason H <jhihn at gmx.com> wrote:
>>> 
>>> QApplication has to be on the main thread.
>>> So I would guess (without seeing the crash) that yes, it's bad form. 
>> 
>> It *is* on the main thread. Of a child process, true, but the main thread of that process. Multiprocessing, not multithreading. Presumably that’s why it generally works - as long as I don’t try to make a web request :-)
>> 
>>> 
>>> Generally when I want to do something like ehat you are doing, I set a singlshot timer to fire (interval: 0) start the event loop (app.exec() or app.exec_() in python) then the event loop wth execute the timer code.
>> 
>> No event loop here. I’m just creating QWidgets, setting contents, and using .grab() to save the contents as an image, after which the widgets are discarded. The only reason I have a QApplication at all is that you can’t create QWidgets without one.
>> 
>> There is no user interaction, no displayed GUI at any point. In fact, I’m initializing QApplication with a platform of offscreen. I just want to work with multiple QWidgets simultaneously to generate multiple images in parallel - thus the need for multiprocessing.
>> 
>>> 
>>> 
>>> 
>>> Sent: Tuesday, December 17, 2019 at 1:13 PM
>>> From: "Israel Brewster" <ijbrewster at alaska.edu>
>>> To: pyside at qt-project.org
>>> Subject: [PySide] PySide2 + multiprocessing + requests = crash?
>>> I’ve found what seems to be an odd crash to me. Using PySide2 and multiprocessing works fine to create a QAppliction in a separate process (unless this is considered bad form?) - unless I call requests.get at some point before starting said process, in which case the process crashes (hard crash, not python exception). How might I fix this?
>>> 
>>> A little background: I’m using Qt classes to generate a bunch of images from data files. To speed things along, I launch a process (using multiprocessing.Process) for each image I want to generate, and in each process I create a QApplication, create and fill the widgets I need for the image, and then save the image to a file. This appears to work fine, creating the images in parallel. Today I tried to do a requests call before doing any of the processing (simple get, doesn’t interact with QT in any way), and suddenly my code started crashing.
>>> 
>>> Some simplified sample code that illustrates the issue:
>>> 
>>> import time
>>> import multiprocessing
>>> import sys
>>> from PySide2.QtWidgets import QApplication
>>> import requests
>>> 
>>> 
>>> def test_wait():
>>>    print("Starting proc")
>>>    try:
>>>        QApplication(sys.argv + ['-platform', 'offscreen'])
>>>    except Exception as err:
>>>        print("oops", err)
>>> 
>>>    print("Sleeping now...")
>>>    time.sleep(10)
>>> 
>>> if __name__ == "__main__":
>>>    proc = multiprocessing.Process(target=test_wait)
>>>    proc.start()
>>>    proc.join()
>>>    print("Awake again. Let's try another:")
>>> 
>>>    res = requests.get('https://www.google.com <https://www.google.com/> <https://www.google.com/ <https://www.google.com/>>')  # or whatever your favorite is
>>>    res.text  # Get the response, close it out
>>> 
>>>    proc = multiprocessing.Process(target=test_wait)
>>>    start = time.time()
>>>    proc.start()
>>>    proc.join()
>>> 
>>>    # The process should sleep for 10 seconds, so if we got here in less than 9, we died.
>>>    if time.time() - start < 9:
>>>        print("Never slept :(")
>>>    else:
>>>        print("We lived!")
>>> 
>>> ---
>>> Israel Brewster
>>> Software Engineer
>>> Alaska Volcano Observatory 
>>> Geophysical Institute - UAF 
>>> 2156 Koyukuk Drive 
>>> Fairbanks AK 99775-7320
>>> Work: 907-474-5172
>>> cell:  907-328-9145
>>> _______________________________________________ PySide mailing list PySide at qt-project.org <mailto:PySide at qt-project.org> https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside> <https://lists.qt-project.org/listinfo/pyside <https://lists.qt-project.org/listinfo/pyside>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/pyside/attachments/20191218/01021dc2/attachment.html>