[Development] Supporting helper functions in auto tests by providing throwing Qt Test macros

[Mitch Curtis]

> -----Original Message-----
> From: Joerg Bornemann
> Sent: Wednesday, 3 April 2019 9:55 AM
> To: Mitch Curtis <mitch.curtis at qt.io>; development at qt-project.org
> Subject: Re: [Development] Supporting helper functions in auto tests by
> providing throwing Qt Test macros
> 
> On 4/2/19 5:14 PM, Mitch Curtis wrote:
> 
> > As described in https://bugreports.qt.io/browse/QTBUG-66320, currently
> Qt users are on their own if they want to call helper functions that can fail a
> test. The reason is documented:
> >
> >      Note: This macro can only be used in a test function that is invoked by
> the test framework.
> >
> > A common workaround for this is to make the helper function return a bool
> indicating success or failure, and pass in a QString reference which is set to
> the failure message (if any).
> >
> > I don't know how many people reading this have written comprehensive
> auto tests for an application, but not having helper functions is just not an
> option if you want maintainable code.
> 
> This is massively annoying, and also a reason for
>    - either writing longish macros that should be functions instead
>    - duplicated code in tests.
> 
> > I looked into this briefly during the last hackathon we had, and from what I
> found, throwing an exception was the best approach:
> >
> > https://codereview.qt-project.org/#/c/248490/
> 
> To me this looks promising. I wonder if qtestlib could catch a dedicated
> QTestException instead to avoid the need of the QCHECK_EXCEPTION macro.

That's definitely an option, it just means that you no longer get the additional, more specific warning at the location of the call site of the helper, which can make finding the exact location of the failure a bit confusing if the test calls the helper several times.

For example, this:

   FAIL!  : tst_App::openClose(ImageType) 'animationPanel->isVisible()' returned FALSE. ()
      Loc: [/home/mitch/dev/slate/tests/shared/testhelper.cpp(1990)]
   WARNING: tst_App::openClose(ImageType) createNewProject(projectType)
      Loc: [/home/mitch/dev/slate/tests/auto/tst_app.cpp(258)]
      
might become this:

   FAIL!  : tst_App::openClose(ImageType) 'animationPanel->isVisible()' returned FALSE. ()
      Loc: [/home/mitch/dev/slate/tests/shared/testhelper.cpp(1990)]
      
On the other hand... we could keep the QCHECK_EXCEPTION macro but make it optional to use it, so those who wanted the more exact failure location could get it, and those who don't don't need to bother with it. 

> 
> Cheers,
> 
> Joerg