[Development] Supporting helper functions in auto tests by providing throwing Qt Test macros

[Mitch Curtis]

> -----Original Message-----
> From: Giuseppe D'Angelo <giuseppe.dangelo at kdab.com>
> Sent: Friday, 5 April 2019 12:39 AM
> To: Mitch Curtis <mitch.curtis at qt.io>; development at qt-project.org
> Subject: Re: [Development] Supporting helper functions in auto tests by
> providing throwing Qt Test macros
> 
> Il 03/04/19 13:58, Mitch Curtis ha scritto:
> >>> I have a question regarding this particular implementation, that is,
> >>> why adding dedicated testing functions? Can't you "simply" change the
> >>> currently existing QVERIFY/QCOMPARE to throw exceptions in case of
> >>> failure, instead of returning from the current test function? This
> >>> would achieve major consistency -- simply use the same functions
> >> everywhere, no matter what.
> >>
> >> To me, a change like this (making the existing macros throw) seems more
> like
> >> a Qt 6 thing, but I could be wrong. I only went with new macros because I
> >> was afraid of breaking stuff, but yeah, just changing the existing ones
> would
> >> be ideal.
> 
> If you expect Qt 6 to be as much source compatible with Qt 5 as
> possible, then you can make this change right now as much you can make
> it in Qt 6.
> 
> So, let's discuss: what breaks if you make today's macros throw instead
> of return?

To take the obvious one first: anyone using their own custom macros in helpers shouldn't be affected (that's me, currently).

Anyone using Qt macros as they're supposed to be used shouldn't be affected so long as we correctly catch the exceptions that we throw. :)

Anyone (incorrectly) using Qt macros in helpers shouldn't be affected so long as exceptions are enabled; their tests will just correctly terminate early upon failures in those helpers now.

Anyone who is throwing exceptions as part of their tests could be affected, I suppose, though your point about handling exceptions correctly and/or the opt-in suggestion (e.g. a QTEST_THROWING_MAIN) seems like it would cover it.

Any tests with exceptions explicitly disabled would fail to compile. That might be the biggest issue?

> 
> >>
> >>> This idea isn't failproof, as:
> >>>
> >>> 1) it requires the exception to propagate through a direct
> >>> QMetaObject::invoke (that is, whatever mechanism testlib uses to
> >>> invoke the test functions); but that should be already supported and
> >> tested, even.
> >>
> >> Hmm, I hadn't thought about invoke()... hopefully that's not an
> (unsolvable)
> >> issue, otherwise it kinda ruins this whole idea.
> 
> As I said, I believe these codepaths support exceptions (but it's not
> exactly documented). If they don't we can should fix them to do so.
> 
> 
> >>> 2) it causes a minor SC in case the user was doing it wrong --
> >>> "catching all", without rethrowing an unknown exception type. Before,
> >>> a failing test would simply return. With this idea, it would throw,
> >>> and the exception could be caught by the catch all and not used to
> >>> block the execution of the current test function.
> >>>
> >>> The last can be solved in a very simple way (e.g. make this mechanism
> >>> opt-in; and also provide something to the user to test if the current
> >>> catch-all is a Qt test failure, via std::current_exception()).
> >>
> >> What kind of opt-in mechanism did you have in mind?
> 
> Anything that is suitable -- env variables, a different macro to create
> the test's main, something to set in your initTestCase(), a CONFIG
> switch in your .pro, etc.
> 
> Cheers,
> --
> Giuseppe D'Angelo | giuseppe.dangelo at kdab.com | Senior Software
> Engineer
> KDAB (France) S.A.S., a KDAB Group company
> Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
> KDAB - The Qt, C++ and OpenGL Experts