[Development] Supporting helper functions in auto tests by providing throwing Qt Test macros

[Edward Welbourne]

On 4/2/19 5:14 PM, Mitch Curtis wrote:
> >> As described in https://bugreports.qt.io/browse/QTBUG-66320,
> >> currently Qt users are on their own if they want to call helper
> >> functions that can fail a test. The reason is documented:
> >>
> >>      Note: This macro can only be used in a test function that is
> >>      invoked by the test framework.

Edward Welbourne (3 April 2019 12:07 PM)
> This note is not strictly true.  You can use it in any function that returns void.
> However, the *caller* won't return in response to it, which means it doesn't
> work *fully* in a helper.

Mitch Curtis (3 April 2019 12:27) replied:
> It's true enough that we would never recommend users to use it in a
> helper function, hence the note.

There is a whole world of difference between "can only be used in" and
"is not recommended for use outside".  The note is incorrect: if you use
these macros inside a helper function, they do cause the test system to
know the test has failed.  They merely don't cause the caller of the
helper to return prematurely, unless the helper checks for whether they
failed.

> Tests should be able to assume that any code that comes after a call
> to a helper means that the helper succeeded. That assumption fails if
> you were to use e.g. QVERIFY in that helper and it failed.

I can quite understand why those using these macros might naively
suppose that it does magic.  (I did when I first used them, before I
tripped over the problem of helper functions, which forced me to think
about it.)  However, if you call a void function, by what magic do you
expect it to cause your present function to have returned ?

Which is, roughly speaking, exactly why we should rewrite the whole
system to use signals - because they're the magic that *can* make that
naive assumption work.  It remains that the present system's macros
don't live up to that naive assumption and, indeed, document that they
don't work other than from the top-level test (albeit only to the extent
that they doesn't live up to this naive assumption).

However, a signal-based solution does have to work in all the places
where the present system does - including the ones where that involves
checking QTest::currentTestFailed() in the actual test slot.  As it
happens, we have uses of QVERIFY and friends in code that runs in helper
classes, via signal/slot connections; such code may be run off the
test-system's event-loop.  I suppose that *does* happen with the actual
test function still on stack, so that it can catch exceptions, but
you'll need to check that carefully.

It's probably worth having a macro with which to wrap code, that'll
catch any exception it raises, add the current location to what'll be
output to log the failure and then rethrows.  This would give us the
extra file/line information that your proposed macro gives, compared to
a naive throw/catch solution (where the failure throws and TestLib's
driver system catches, reporting the helper but not its caller).

I think the idea of rewriting TestLib (including the existing QFAIL,
QCOMPARE, QVERIFY and friends) to use exceptions in Qt6 is a good way
forward; adding some convenience macro for currentTestFailed() checks is
probably sufficient to make helper functions easier to use until then.

	Eddy.