[Development] Supporting helper functions in auto tests by providing throwing Qt Test macros

[Mitch Curtis]

> -----Original Message-----
> From: Edward Welbourne
> Sent: Wednesday, 3 April 2019 5:07 PM
> To: Mitch Curtis <mitch.curtis at qt.io>
> Cc: development at qt-project.org; Joerg Bornemann
> <Joerg.Bornemann at qt.io>
> Subject: Re: [Development] Supporting helper functions in auto tests by
> providing throwing Qt Test macros
> 
> On 4/2/19 5:14 PM, Mitch Curtis wrote:
> > >> As described in https://bugreports.qt.io/browse/QTBUG-66320,
> > >> currently Qt users are on their own if they want to call helper
> > >> functions that can fail a test. The reason is documented:
> > >>
> > >>      Note: This macro can only be used in a test function that is
> > >>      invoked by the test framework.
> 
> Edward Welbourne (3 April 2019 12:07 PM)
> > This note is not strictly true.  You can use it in any function that returns void.
> > However, the *caller* won't return in response to it, which means it
> > doesn't work *fully* in a helper.
> 
> Mitch Curtis (3 April 2019 12:27) replied:
> > It's true enough that we would never recommend users to use it in a
> > helper function, hence the note.
> 
> There is a whole world of difference between "can only be used in" and "is
> not recommended for use outside".  The note is incorrect: if you use these
> macros inside a helper function, they do cause the test system to know the
> test has failed.  They merely don't cause the caller of the helper to return
> prematurely, unless the helper checks for whether they failed.

It doesn't really matter if they've always been documented that way. Users should follow the documentation, not the implementation.

> It's probably worth having a macro with which to wrap code, that'll catch any
> exception it raises, add the current location to what'll be output to log the
> failure and then rethrows.  This would give us the extra file/line information
> that your proposed macro gives, compared to a naive throw/catch solution
> (where the failure throws and TestLib's driver system catches, reporting the
> helper but not its caller).

What would the effective difference be from QCHECK_EXCEPTION in the proposed patch, in terms of behaviour and output?

> I think the idea of rewriting TestLib (including the existing QFAIL, QCOMPARE,
> QVERIFY and friends) to use exceptions in Qt6 is a good way forward; adding
> some convenience macro for currentTestFailed() checks is probably sufficient
> to make helper functions easier to use until then.

The currentTestFailed() macro is a nice idea, but it's not sufficient if your helpers call other functions, because then you have to wrap every call site of those functions with the macro.

> 	Eddy.