[Development] Need advise on acceptable timeouts for autotests

[Simon Hausmann]

Hi,


(1) We don't know where the slowdowns come from. There is no VM migration involved in the CI (the license does not include this feature). There is a SAN under every single virtual machine disk (something we are getting rid of in the not-for-qt-5.8.1 time).


(2) I don't know what the correct choice of a timeout here is. The current logic for dealing with timeouts for tests in qtbase is quite simple:


    (a) The total time "make check" is allowed to take when it is run in each sub-directory is 15 minutes (but I think that there are exceptions).

    (b) While the test is running, if no output appears in stderr or stdout within the course of 20 minutes, a timeout error is raised. Any output appearing resets this 20-minute timer back to zero.




Simon

________________________________
From: marc at kdab.com <marc at kdab.com> on behalf of Marc Mutz <marc.mutz at kdab.com>
Sent: Thursday, March 16, 2017 10:00:55 AM
To: development at qt-project.org
Cc: Qt CI
Subject: Need advise on acceptable timeouts for autotests

Hi,

We repeatedly have the problem that timeouts that developers think are ample
(because they exceed typical runtime by, say, two orders of magnitude) are
found to be insufficient on the CI.

Latest example:
http://testresults.qt.io/coin/integration/qt/qtbase/tasks/1489618366

The timeout to run update-mime-database was recently increased to 2mins. But
that still does not seem to be enough. For a call that hardly takes a second
to run on unloaded machines.

We can of course crank up timeouts to insane amounts like 1h, but that means
that a test will just sit there idling for an hour in the worst case.

I have two questions:

1. Where do these huge slowdowns come from? Is the VM live-migrated? Is the
   SAN, if any, down? At this point it lools like no overcommitting of CPU/RAM
   could ever explain how update-mime-database can take 2mins to run.

2. What should we choose as timeouts? I understand that tests which are stuck
   are killed after some time (how long?). Maybe timeouts should be set to the
   same value?

Thanks,
Marc

--
Marc Mutz <marc.mutz at kdab.com> | Senior Software Engineer
KDAB (Deutschland) GmbH & Co.KG, a KDAB Group Company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20170316/75f8454f/attachment.html>