[Interest] Windows 10 + native QFileDialog + QPrinter + some magic = application hangs

[Rainer Wiesenfarth]

Seems the issue is solved:

The whole stuff is related to some COM library we (have to) use. During my
tests I came along this code snippet in my main.cpp:

  CoInitializeEx( NULL, COINIT_APARTMENTTHREADED );  // (a)

  InitialUseOfCOMObject();  // (b)

  CoUninitialize();  // (c)

Using the call sequence (1)-(3) below, I can comment whatever part of (a)
to (c), but (3) will never succeed. That might be what you would expect,
either.

However, if I uncomment (a) to (c), even (1) did not succeed!

Taking a closer look at the return value of CoInitializeEx() showed that it
returned RPC_E_CHANGED_MODE, i.e. that some startup code in the COM library
already initialized it in a way incompatible with Qt. The CoUnitialize then
made the CoInitialize inside Qt to succeed. Apparently some code behind
QPrinter then again did a CoInitialize in the wrong mode, causing the
second call to fail. For a workaround, I ended up in re-initializing COM
myself and keeping this initialization until the end of my app.

So, as often: Qt is clear from guilt... :-)

Thanks to Henry and Harri for their assistance!

On Wed, Dec 20, 2017 at 10:42 PM, Rainer Wiesenfarth <
rainer_wiesenfarth at trimble.com> wrote:

> Hi all,
>
> I have a strange problem that is bugging me for a couple of days now.
> Although I think the chances are low to find someone who experienced the
> same issue, I would like to give it a try.
>
> I was able to strip the interesting part of my application down to these
> lines:
>
>   QString sProject = QFileDialog::getOpenFileName( this, tr( "Select
> Project File" ),
>     m_sProjectFile, tr( "Project File (*.prj);;" "All Files (*.*)" ) );
>  // (1)
>
>   QPrinter p;   // (2)
>
>   sProject = QFileDialog::getOpenFileName( this, tr( "Select Project File"
> ),
>       m_sProjectFile, tr( "Project File (*.prj);;" "All Files (*.*)" ) );
>  // (3)
>
> Case 1: Running my application on Windows 10 (four different machines):
> - (1) succeeds
> - (2) succeeds
> - (3) hangs the application in QWindowsNativeFileDialogBase::doExec()
> at m_fileDialog->Show(owner)
>
> Case 2: Embedding the above snippet into a small sample application on
> Windows 10:
> - (1) to (3) all succeed
>
> Case 3: Setting the QFileDialog::DontUseNativeDialog option on Windows 10:
> - (1) to (3) all succeed
>
> Case 4: Running my application on Windows 7 (several machines):
> - (1) to (3) all succeed
>
> I took a close look at the m_fileDialog in QWindowsNativeFileDialogBase
> for the Windows 10 case that gets stuck:
> - at (1), m_fileDialog references a vtable in comdlg32.dll!const
> CFileOpenSave
> - at (3), m_fileDialog references a vtable in
> OneCoreUAPCommonProxyStub.dll!0x...
>
> So it seems to be related to Windows "umbrella" libraries, but I have no
> idea how. I looked even deeper into where m_fileDialog is getting assigned
> and found the call to CoCreateInstance(), where CLSID_FileOpenDialog and
> IID_IFileOpenDialog is passed in both (1) and (3), but in (1) the comdlg32
> object is returned while in (3) it is the OneCore... object.
>
> My guess so far:
> 1. The constructor or destructor of QPrinter could lead to the replacement
> of the object behind CLSID_FileOpenDialog, but
> 2. this replacement not done always, but is triggered by the "some magic"
> part in my application that is executed before (1)
>
> ​Any ideas what the "​some magic" part could be? Unfortunately my
> application is quite complex and can not be stripped down easily...
>
> ​Cheers, Rainer
>
> --
> Software Engineer | Trimble Imaging Division
> Rotebühlstraße 81 | 70178 Stuttgart | Germany
> Office +49 711 22881 0 | Fax +49 711 22881 11 <+49%20711%202288111>
> http://www.trimble.com/imaging/ | http://www.inpho.de/
>
> Trimble Germany GmbH, Am Prime Parc 11, 65479 Raunheim
> Eingetragen beim Amtsgericht Darmstadt unter HRB 83893,
> Geschäftsführer: Dr. Frank Heimberg, Jürgen Kesper
>



-- 
Software Engineer | Trimble Imaging Division
Rotebühlstraße 81 | 70178 Stuttgart | Germany
Office +49 711 22881 0 | Fax +49 711 22881 11
http://www.trimble.com/imaging/ | http://www.inpho.de/

Trimble Germany GmbH, Am Prime Parc 11, 65479 Raunheim
Eingetragen beim Amtsgericht Darmstadt unter HRB 83893,
Geschäftsführer: Dr. Frank Heimberg, Jürgen Kesper
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/interest/attachments/20171222/2002343c/attachment.html>