[Development] Supporting helper functions in auto tests by providing throwing Qt Test macros
mitch.curtis at qt.io
Thu Apr 4 08:43:23 CEST 2019
> -----Original Message-----
> From: Edward Welbourne
> Sent: Wednesday, 3 April 2019 5:07 PM
> To: Mitch Curtis <mitch.curtis at qt.io>
> Cc: development at qt-project.org; Joerg Bornemann
> <Joerg.Bornemann at qt.io>
> Subject: Re: [Development] Supporting helper functions in auto tests by
> providing throwing Qt Test macros
> On 4/2/19 5:14 PM, Mitch Curtis wrote:
> > >> As described in https://bugreports.qt.io/browse/QTBUG-66320,
> > >> currently Qt users are on their own if they want to call helper
> > >> functions that can fail a test. The reason is documented:
> > >>
> > >> Note: This macro can only be used in a test function that is
> > >> invoked by the test framework.
> Edward Welbourne (3 April 2019 12:07 PM)
> > This note is not strictly true. You can use it in any function that returns void.
> > However, the *caller* won't return in response to it, which means it
> > doesn't work *fully* in a helper.
> Mitch Curtis (3 April 2019 12:27) replied:
> > It's true enough that we would never recommend users to use it in a
> > helper function, hence the note.
> There is a whole world of difference between "can only be used in" and "is
> not recommended for use outside". The note is incorrect: if you use these
> macros inside a helper function, they do cause the test system to know the
> test has failed. They merely don't cause the caller of the helper to return
> prematurely, unless the helper checks for whether they failed.
It doesn't really matter if they've always been documented that way. Users should follow the documentation, not the implementation.
> It's probably worth having a macro with which to wrap code, that'll catch any
> exception it raises, add the current location to what'll be output to log the
> failure and then rethrows. This would give us the extra file/line information
> that your proposed macro gives, compared to a naive throw/catch solution
> (where the failure throws and TestLib's driver system catches, reporting the
> helper but not its caller).
What would the effective difference be from QCHECK_EXCEPTION in the proposed patch, in terms of behaviour and output?
> I think the idea of rewriting TestLib (including the existing QFAIL, QCOMPARE,
> QVERIFY and friends) to use exceptions in Qt6 is a good way forward; adding
> some convenience macro for currentTestFailed() checks is probably sufficient
> to make helper functions easier to use until then.
The currentTestFailed() macro is a nice idea, but it's not sufficient if your helpers call other functions, because then you have to wrap every call site of those functions with the macro.
More information about the Development