[Releasing] RFC: Test Reports for source/binary packages

[marius.storm-olsen at nokia.com]

Hi,

Back in the days, when we were approaching a new release of Qt, we would use a black team <http://www.t3y.com/tangledwebs/07/tw0706.html> which would be responsible for driving the testing of a new package. Their main focus would be to test every new package generated, both source and binary packages, and try to break them with the most used configuration (every configuration is not possible due to the vast number of configurable options, and system configurations).

When doing this they would have a common list of issues to look out for, to make sure that the reporting had some structure (same report fields for each reporter/platform/package) and that they wouldn't skip something for one package, etc. The list contained things like:

  *   General:
     *   Report date
     *   Testers name
     *   Package name
     *   Package type
     *   Build date
     *   Makespec used
  *   Source packages:
     *   Configuration line for main testing
        *   Configure asks about license when run with no license options
     *   Compiling the package
        *   Compiles with minimal options (e.g. -opensource -confirm-license)
        *   Compiles as a static build, where supported
        *   Compiles in namespace, where supported
        *   Compiles with shadow build
        *   Cross-compiles, where supported
     *   Installing the package works
        *   "make install" to system directory as root works
        *   "make install" to local prefix as regular user works
        *   "make install" distributes files correctly on Mac
        *   "make clean" and "make distclean" works
     *   Text files have the correct EOL for the type of package?
     *   Files/directories in the package have sane file permissions and timestamps?
     *   Tags (%VERSION%, %SHORTVERSION%) have been replaced properly
     *   README has valid information about how to compile the package on the platform tested
  *   Binary packages:
     *   Fresh install
        *   Installer is correctly signed, e.g. Windows shows correct vendor/certificate data and no warnings about untrusted vendor.
        *   Installer displays appropriate graphics, strings, version numbers
        *   Installer offers the correct license(s)
        *   Installer offers sane default install directory (including version number)
        *   Installer correctly installs to default directory
        *   installer correctly installs to non-default drive/directory
        *   Installer sanely reports progress and completion
        *   Installer installs only selected components
        *   Component selection works sanely
        *   Shortcuts from last page of installer work (e.g. shortcuts to README, demos, etc)
        *   Installer correctly creates desktop shortcuts and they all work
        *   Installer correctly creates Start Menu shortcuts and they all work
        *   Environment settings are correct in Qt MSVC Command Prompt
        *   Package shows up in Control Panel/Package Manager with correct description/version
        *   Any patching of files has been done correctly, e.g. patching of library paths into .exe files and the paths hard-coded in the QtCore library.
     *   Upgrade install, if supported
  - As above
     *   Parallel install, if supported
  - As above
  - Can't install over the top of an existing package without being warned
  - Shortcuts point to the right package
  - Previously installed packages still work
     *   Uninstall
  - Removes all installed files,
  - Removes all empty dirs,
  - Removes registry keys,
  - Reverses any other changes made by installer
  - What to do with data files created by installed files, e.g. saved settings and files created by demo apps?
  - What to do with data shared between more than one package instance?
     *   Aborting installation, if supported
  - Cancel button is available
  - Installer cleans up, as for full uninstallation
     *   Failed installation
  - insufficient disk space (can be faked by installing onto a small USB key/SD card)
  - network problems while using online installer
  *   Both package types:
     *   Verify the license
     *   Assistant works
     *   Designer works
     *   "Showcase" demo/example apps launch without crashing
     *   "Showcase" demo/example apps function acceptably.
     *   Demo/example apps can be rebuilt
     *   External Qt apps can be built against the package (e.g. Qt Creator)
     *   Did "DLL Swapping" on a Qt app (e.g. KDE, Google Earth, Qt Creator) work?
     *   Click around various examples/demos for a while (GUI stress-testing)
     *   Audio/Video w/phonon
     *   Audio/Video w/QtMultimedia
     *   Raster paint engine works
     *   Image formats work
     *   Graphicsview
     *   OpenGL
     *   OpenVG
     *   Printing
     *   QML
     *   QtNetworking
     *   QtScript
     *   QtSql
     *   QtSvg
     *   WebKit works?
        *   Logging into a popular site (facebook, hotmail, gmail etc) using the demo-browser
        *   Uploading an image to imgur.com using the demo-browser
        *   Watch a cat video on youtube using the demo-browser
     *   Xml
     *   qmlviewer (outside of qtdemo)

Thanks Jason for a thorough list from the Qt 4 days!
This testing should be distributed on as many people as possible to lower the load on individuals, and to allow us to get as much feedback as quickly as possible. This in turn allows for shorter test periods, and that way we can get the Qt releases out according to our schedule(s).

The question is now, how do we organize this? We need a form, and a place to collect them and allowing us to easily browse the reports and see the status for any given release.

I don't think using SurveyMonkey would cut it with this scope. :)

How about maybe installing the "JIRA Issue Collector<https://plugins.atlassian.com/plugins/com.atlassian.jira.collector.plugin.jira-issue-collector-plugin>" plugin (You are not required to have a Jira account to use the issue collector, as it can use a fall-back account for general reporting!), allowing us to report against a "Fix Version", and perhaps use "Labels" to distinguish between build dates? Maybe these reports should have their own types (currently we only have Bug/Task), like "Test Report"? Then on a no-go report, Jira "Bug" issues could be created and linked to the "Test Report"?

Opinions?

--
.marius


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/releasing/attachments/20120501/4397719e/attachment.html>