[Development] Supporting helper functions in auto tests by providing throwing Qt Test macros

[Mitch Curtis]

Resending to the list...

> -----Original Message-----
> From: Mitch Curtis
> Sent: Wednesday, 3 April 2019 1:58 PM
> To: 'Giuseppe D'Angelo' <giuseppe.dangelo at kdab.com>
> Subject: RE: [Development] Supporting helper functions in auto tests by
> providing throwing Qt Test macros
> 
> > -----Original Message-----
> > From: Development <development-bounces at qt-project.org> On Behalf
> Of
> > Giuseppe D'Angelo via Development
> > Sent: Wednesday, 3 April 2019 1:21 PM
> > To: development at qt-project.org
> > Subject: Re: [Development] Supporting helper functions in auto tests
> > by providing throwing Qt Test macros
> >
> > Hi,
> >
> > Il 02/04/19 17:14, Mitch Curtis ha scritto:
> > > Quoting the commit message here:
> > >
> > >      WIP: Add _THROW variants to testlib macros
> > >
> > >      This allows using Qt Test macros in helper functions, avoiding the need
> > >      to write a lot of boilerplate code as seen with alternative approaches.
> >
> > I may like the overall approach. However, this would imply that such
> > tests require exceptions / RTTI turned on inside QtTestLib / user
> > code; is it "OK" in the grand scheme of things?
> 
> From my testing, user tests already support exceptions by default, so I guess
> it only affects cases where they have explicitly disabled them?
> 
> For Qt auto tests, exceptions are already enabled by default. The exception
> is testlib selftests, where we need to use CONFIG += exceptions. Quoting
> the comment that Robin found:
> 
> "note that we now load qt_build_config.prf instead of just qmodule.pri,
> which means that exceptions_off is set everywhere. we forcibly re-enable
> them for testcases to minimize the deviation from default 3rd party usage.
> testlib selftests are not qt testcases, so the one that needs exceptions needs
> to enable them explicitly."
> 
> https://code.qt.io/cgit/qt/qtbase.git/commit/?id=6226fcdc
> 
> Whether or not it's acceptable to force users to turn on exception support in
> their tests... I don't know.
> 
> > I have a question regarding this particular implementation, that is,
> > why adding dedicated testing functions? Can't you "simply" change the
> > currently existing QVERIFY/QCOMPARE to throw exceptions in case of
> > failure, instead of returning from the current test function? This
> > would achieve major consistency -- simply use the same functions
> everywhere, no matter what.
> 
> To me, a change like this (making the existing macros throw) seems more like
> a Qt 6 thing, but I could be wrong. I only went with new macros because I
> was afraid of breaking stuff, but yeah, just changing the existing ones would
> be ideal.
> 
> > This idea isn't failproof, as:
> >
> > 1) it requires the exception to propagate through a direct
> > QMetaObject::invoke (that is, whatever mechanism testlib uses to
> > invoke the test functions); but that should be already supported and
> tested, even.
> 
> Hmm, I hadn't thought about invoke()... hopefully that's not an (unsolvable)
> issue, otherwise it kinda ruins this whole idea.
> 
> > 2) it causes a minor SC in case the user was doing it wrong --
> > "catching all", without rethrowing an unknown exception type. Before,
> > a failing test would simply return. With this idea, it would throw,
> > and the exception could be caught by the catch all and not used to
> > block the execution of the current test function.
> >
> > The last can be solved in a very simple way (e.g. make this mechanism
> > opt-in; and also provide something to the user to test if the current
> > catch-all is a Qt test failure, via std::current_exception()).
> 
> What kind of opt-in mechanism did you have in mind?
> 
> >
> > My 2 c,
> > --
> > Giuseppe D'Angelo | giuseppe.dangelo at kdab.com | Senior Software
> > Engineer KDAB (France) S.A.S., a KDAB Group company Tel. France +33
> > (0)4
> > 90 84 08 53, http://www.kdab.com KDAB - The Qt, C++ and OpenGL Experts