[Development] Supporting helper functions in auto tests by providing throwing Qt Test macros

[Matthew Woehlke]

On 05/04/2019 10.49, Matthew Woehlke wrote:
> IMNSHO, I think Qt would do well to take a close and careful look at
> Google Test, which IMO has a much better design, at least as far as how
> test assertions are handled.
> 
> In particular, I miss having a distinction between fatal and non-fatal
> assertions, which, aside from their own inherent value, I think would be
> a better solution to this problem. These could be combined with a macro
> to invoke a helper function which would a) add additional context if the
> helper function produces *any* test output (I'm thinking this should
> include use of QDebug), and b) optionally¹ abort the caller² if a fatal
> assertion has occurred.

To elaborate on that, I would suggest having at least three assertions:

1. Evaluate an expression's truthiness (expecting true).
2. Compare a computed value to an expected value (expecting equality).
3. Execute a statement (expecting no failed assertions in the process).

Each of these should *additionally* come in both fatal and non-fatal
forms. A fatal form triggers a `return` in the containing scope. A
non-fatal form merely logs the failure and continues on. A test fails if
*any* assertion failed during its execution.

The fatal forms of (1) and (2) are equivalent to QVERIFY and QCOMPARE,
respectively. (3) can and should be used to invoke "helper functions"
and obviates the need to use exceptions. Note however that it doesn't
*need* to be used, since the helper can still contain assertions which
will still cause the test to fail. However, using (3) will also log the
call site if the helper has failing assertions, and is required for
assertions in the helper to abort execution in the caller.

For example:

   1 void helper()
   2 {
   3   QVERIFY(false);
   4   abort(); // won't get here
   5 }
   6
   7 void Test::testcase()
   8 {
   9   QTRY_CALL(helper());
  10   QCALL(helper());
  11   abort(); // won't get here
  12 }

...would produce something like:

  FAIL! ...
    Loc: sample.cpp(3)
    Via: sample.cpp(9)
  FAIL! ...
    Loc: sample.cpp(3)
    Via: sample.cpp(10)

-- 
Matthew