[Releasing] [Development] brown paper bag issue in Qt 5.6.1 packages

[Johanna Äijälä]

Hi Thiago & all,

In addition what Tuukka wrote about RTA, currently I'm mainly responsible of the RTA tests for offline and online installers. The goal of the RTA is to minimize the need of manual testing, but everything can't be tested automatically, manual tests are always needed. 

RTA tests installer functionality and the packages that are installed, it does not test the content in git repositories. 

There are two level of tests, RTA smoke and RTA full tests. 
- Smoke test set takes couple of hours to run and analyze. RTA smoke is done before manual test requests are send out, it does a sanity check for the installer by verifying that installer installs correct content with default settings and builds all examples to verify that the content works, also light QtCreator testing is included. 
- Full test round takes about 48 hours. It is done parallel with manual testing. Tests include installing with different options, license checking, source package tests (content and licenses check and building Qt with different configure options) and some iOS and Android deployment tests.

Test execution times depends on the number of available machines, RTA uses same build machines as coin.

Br,
Johanna

-----Original Message-----
From: Releasing [mailto:releasing-bounces+johanna.aijala=qt.io at qt-project.org] On Behalf Of Tuukka Turunen
Sent: 23. kesäkuuta 2016 12:38
To: Sergio Ahumada; releasing at qt-project.org
Subject: Re: [Releasing] [Development] brown paper bag issue in Qt 5.6.1 packages



> -----Original Message-----
> From: Releasing [mailto:releasing-bounces+tuukka.turunen=qt.io at qt-
> project.org] On Behalf Of Sergio Ahumada
> Sent: torstaina 23. kesäkuuta 2016 8.22
> To: releasing at qt-project.org
> Subject: Re: [Releasing] [Development] brown paper bag issue in Qt 
> 5.6.1 packages
> 
> On 23.06.2016 01:07, Thiago Macieira wrote:
> > On quarta-feira, 22 de junho de 2016 10:20:15 PDT Jani Heikkinen wrote:
> >> 10. Test the release. If we update existing release with one change 
> >> it is much easier to test. We need just test that the fix is in and 
> >> fixes the issue + don't break anything else. And of course test 
> >> that packaging went ok. So we can rely on RTA in this case pretty 
> >> much but with totally new release we need significantly more manual 
> >> test effort in addition and that takes time (testing + collecting results).
> >
> > What (or who) is RTA?
> >
> > Rapid Testing Automation?
> >
> > Release Testing Automation?
> >
> > Release Testing Authority?
> >
> 
> http://lists.qt-project.org/pipermail/releasing/2013-November/001502.h
> tml
> 

Release Test Automation (RTA) is a system that automatically installs binary packages and runs a series of tests in multiple platforms using Froglogic Squish and various Qt applications. It is currently a separate system, commanded by Jenkins. The system can run a quick smoke test set before manual testing of a snapshot starts, and it can also run a more complete set, which is especially beneficial for patch releases. 

It of course does not have to be separate, it could be part of the same CI system as other tests are. But for the while being it is not connected to the CI. 

People operating the RTA are working in the release & qa team, it is not a separate organization.

Yours,

	Tuukka

> --
> Sergio Ahumada
> sahumada at texla.cl
> 
> _______________________________________________
> Releasing mailing list
> Releasing at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/releasing
_______________________________________________
Releasing mailing list
Releasing at qt-project.org
http://lists.qt-project.org/mailman/listinfo/releasing