[Development] Supporting helper functions in auto tests by providing throwing Qt Test macros

[Matthew Woehlke]

On 02/04/2019 11.14, Mitch Curtis wrote:
> As described in https://bugreports.qt.io/browse/QTBUG-66320, 
> currently Qt users are on their own if they want to call helper
> functions that can fail a test. The reason is documented:
> 
> Note: This macro can only be used in a test function that is invoked
> by the test framework.
> 
> [...]
> 
> I'm curious what other people think about [throwing exceptions
> instead]?

IMNSHO, I think Qt would do well to take a close and careful look at
Google Test, which IMO has a much better design, at least as far as how
test assertions are handled.

In particular, I miss having a distinction between fatal and non-fatal
assertions, which, aside from their own inherent value, I think would be
a better solution to this problem. These could be combined with a macro
to invoke a helper function which would a) add additional context if the
helper function produces *any* test output (I'm thinking this should
include use of QDebug), and b) optionally¹ abort the caller² if a fatal
assertion has occurred.

(¹ Why optional? I've found it extremely useful to be able to use
"fatal" assertions to abort a sub-block of a unit test, such as a helper
function. So useful, in fact, that I've adopted a regular pattern of
immediately-invoked lambdas for the sole purpose of controlling "how
much" of a test case a fatal assertion aborts.)

(² How? By recording somewhere that a fatal exception has occurred, and
having the invocation wrapper check that and abort the caller. Note that
this needs to distinguish between "fatal exception(s) happened
previously" and "new fatal exception(s) happened in what I called".)

-- 
Matthew