[Interest] QProcess unbuffered

[Scott Bloom]

I really want to thank Björn

With his starting point, I was able to tweak his changes, and seem to have it working 😊 I have more testing to do. But it was actually pretty straight forward.

* I use VS (2022) so the packing was different
	__attribute__((packed)) became  __pragma( pack( push, 1 ) ) struct HackedHandlePasser
* The handles are 64 bit in the existing Startup Info, so I kept them as such, which changed the definition of HANDLE32
         using HANDLE32 = std::int32_t; was removed, and the standard HANDLE defined in windows.h was used
* The static assert was updated to reflect this
* I copy all 3 handles from the existing startupinfo
       std::memcpy(&handles.Handles[0], &invalid, 4);
       std::memcpy(&handles.Handles[1], &pipe.Write, 4);
       std::memcpy(&handles.Handles[2], &pipe.Write, 4);

       became

        std::memcpy( &handles.Handles[ 0 ], &args->startupInfo->hStdInput, sizeof( HANDLE ) );
        std::memcpy( &handles.Handles[ 1 ], &args->startupInfo->hStdOutput, sizeof( HANDLE ) );
        std::memcpy( &handles.Handles[ 2 ], &args->startupInfo->hStdError, sizeof( HANDLE ) );

Then I wrapped this into a function that returns a QProcess::QCreateProcessArgumentModifier.

And finally in my process, I set the modifier using process-> setCreateProcessArgumentsModifier( func );

You can find the code
https://github.com/towel42-com/SABUtils/blob/trunk/ForceUnbufferedProcessModifier.h
and
https://github.com/towel42-com/SABUtils/blob/trunk/ForceUnbufferedProcessModifier.cpp

and its usage at
https://github.com/towel42-com/MediaManager/blob/trunk/Models/DirModel.cpp

Thanks again.
Scott



-----Original Message-----
From: Scott Bloom <scott at towel42.com> 
Sent: Tuesday, March 7, 2023 5:34 PM
To: Scott Bloom <scott at towel42.com>; Björn Schäpers <qt-maillist at hazardy.de>; interest at qt-project.org
Subject: RE: [Interest] QProcess unbuffered

Yeah, Im being dense.. its from the existing startupinfo.. I read that paragraph a dozen times and missed it each time

-----Original Message-----
From: Interest <interest-bounces at qt-project.org> On Behalf Of Scott Bloom
Sent: Tuesday, March 7, 2023 5:28 PM
To: Björn Schäpers <qt-maillist at hazardy.de>; interest at qt-project.org
Subject: Re: [Interest] QProcess unbuffered

Maybe Im being dense (its been a long day)

But where is pipe defined ?

       std::memcpy(&handles.Handles[1], &pipe.Write, 4);
       std::memcpy(&handles.Handles[2], &pipe.Write, 4);

-- Scott
-----Original Message-----
From: Björn Schäpers <qt-maillist at hazardy.de> 
Sent: Tuesday, March 7, 2023 12:58 PM
To: Scott Bloom <scott at towel42.com>; interest at qt-project.org
Subject: Re: [Interest] QProcess unbuffered

So here is what I have:

struct __attribute__((packed)) HackedHandlePasser {
          using HANDLE32 = std::int32_t;

          DWORD    NumberOfHandles = 3; // 4 Byte
          BYTE     FlagsPerHandle[3];   // 3 * 1 Byte
          HANDLE32 Handles[3];          // 3 * 4 Byte
       };

       static_assert(sizeof(HackedHandlePasser) == 19);
       HackedHandlePasser handles;

       #ifndef FOPEN
#define FOPEN 0x01
       #endif
       #ifndef FDEV
#define FDEV 0x40
       #endif

       handles.FlagsPerHandle[0] = 0;
       std::memset(&handles.FlagsPerHandle[1], FOPEN | FDEV, 2);

       const HANDLE invalid = INVALID_HANDLE_VALUE;
       std::memcpy(&handles.Handles[0], &invalid, 4);
       std::memcpy(&handles.Handles[1], &pipe.Write, 4);
       std::memcpy(&handles.Handles[2], &pipe.Write, 4);

       startInf.cbReserved2 = sizeof(HackedHandlePasser);
       startInf.lpReserved2 = reinterpret_cast<LPBYTE>(&handles);

Some explanations:
* I have the HANDLE32 because I start a 32 bit application from within a 64 bit, as far as I understood the structure needs the right HANDLE size, thus for a 64 bit application it should be std:int64_t, but since I don't need that I never tested it.
* In cbReserved2 there is the size of the data in lpReserved2.
* The data in lpReserved2 starts with 2 bytes which indicate how many handles are passed to the spawned process (they may be INVALID_HANDLE_VALUE). Followed by one byte per handle with the opened flags - and here comes the trick: If it has the flag FDEV the ms runtime printf assumes it writes to a console and thus flushes. And then followed by the handles.
* My use case needs only stdout (and I also open stderr, in my case to the same handle), these are the handles number 1 and 2. Number 0 would be stdin.

So when using it with QProcess I think one should copy the handles from the STARTUPINFOW structure into this struct, then I think the normal QIODevice interface should keep working. If one should clear the STARTF_USESTDHANDLES flag I also don't know. I stopped my experiments when I achieved success for my "simple" use case.

Let me know what works for you.

And last but not least some sources:
https://stackoverflow.com/questions/40965496/windows-how-to-stop-buffering-of-redirected-stdout-using-createprocess
which apparently now also has a C/C++ answer, when I needed it there was only the python code.
http://www.catch22.net/tuts/undocumented-createprocess does use the fields for some other stuff, but explains a bit of background.
https://github.com/cansou/msvcrt/blob/master/src/isatty.c One if the involved files to handle the input.

Kind Regards,
Björn.
_______________________________________________
Interest mailing list
Interest at qt-project.org
https://lists.qt-project.org/listinfo/interest