[Development] New Coin feature: automatic provisioning

[Jędrzej Nowacki]

On Wednesday 27 of July 2016 13:58:44 Dominik Holland wrote:
> Hi,
> 
> Am 07/27/2016 um 01:47 PM schrieb Jędrzej Nowacki:
> > Hi,
> > 
> >   Good news everyone! Last month we managed to deploy a new automatic
> > 
> > provisioning system to Coin. What does it mean to you? Now you can modify
> > existing VM templates used by Coin. Installing additional packages should
> > be trivial and more or less safe from now.
> > 
> >   I. The goal:
> >   - Coin should use VMs with vanilla OS installations (aka base template)
> >   - Anything that needs to be installed in addition should be documented,
> > 
> > reproducible in an automatic way (create a provisioned template)
> > 
> >   - templates may be only modified in such way that the _whole_ Qt still
> >   works> 
> > on a VM create from it
> > 
> >   - It should not be required to log in to a VM to modify it
> >   
> >   II. So how it works now:
> >   Qt5 is our product repository. The product repository contains directory
> > 
> > called "coin/provisioning", which contains provisioning scripts used
> > against base templates to produce provisioned templates. When an
> > integration starts on a certain platform, it checks for the directory. If
> > base platform template name matches one of the subdirectories then all
> > scripts in the subdirectory are used to provision the new template. Of
> > course the result is cached and re- used for all integrations that needs
> > it.
> > 
> >   III. Qt Developer oriented example:
> >   We need to install FooBar on Ubuntu 14.04
> >   
> >   1. Prepare bash script that installs it:
> >      #!/bin/env bash
> >      # Package FooBar is need to ... and can be safely removed if ...
> >      apt install FooBar
> >   
> >   2. Place it in the product repository (Qt5) in
> >   coin/provisioning/qtci-linux-> 
> > Ubuntu-14.04-x86_64/ under foobar.sh name.
> > 
> >   3. Commit it to the right branch (5.6..dev) depending where the package
> >   is
> > 
> > needed
> > 
> >   4. Get through the review, by default we do __not__ want any additional
> > 
> > software on VMs, so you need to have a good reason to install something
> > 
> >   5. Stage the change together with a recent submodule update
> >   6. If it pass then you can enjoy FooBar in your code
> >   7. Follow-up on Qt5 merges, to ensure that FooBar is installed in right
> > 
> > branches and only in them
> > 
> >   Be aware that provisioning scripts are tested against Qt5 state while
> >   newly
> > 
> > provisioned templates are used for all integrations. So technically it is
> > possible to break stuff if Qt5 is old or if a breakage was integrated
> > during Qt5 testing. In reality I do not think it would affect anyone, but
> > that is why in point 5 I recommend to staging together with a recent
> > submodule update.> 
> >   Currently only bash and powershell are supported.
> >   
> >   If you want to share a script between different platforms (for example
> > 
> > Ubuntu 14.04, and Ubuntu 16.04) you place it in
> > "coin/provisioning/common".
> > Then you can call it from any other "coin/provisioning" subdirectory.
> > 
> >   The current naming convention for base templates is suboptimal and it
> >   will
> > 
> > be changed in a reasonable future, but you should not worry about that,
> > the
> > process will be transparent (TM).
> > 
> > Cheers,
> > 
> >   Jędrek
> > 
> > ps. The feature consists of a few dozen patches and the probability says
> > that there is at least one bug, so keep your eyes open.
> 
> great news !
> 
> Does the provisioning only work for the qt5 repository or does it also
> work for each of the modules ?
> 
> E.g. Could there be a provisioning script in qtmultimedia, installing
> some of the dependencies e.g. gstreamer1.0-dev ?
> 
> Dominik
> 

Qt5 repository contains definition for all modules. All modules shares the same 
set of machines(*). There is no direct way to install something just for one 
module. Technically you could install something to a custom place and export a 
environment variable, which would be used only by one module. I advice against 
that. 

The feature is useful for installing software that is already shipped by a 
distribution but just not installed by default. Regarding gstreamer1.0, the 
problem is that in RHEL 6 it doesn't exist at all therefore it would need to 
be compiled from sources and then we would need to ship gstreamer binaries... 
So, no the feature do not solve qstreamer case :(

Cheers,
 Jędrek

*there are tricks to opt-out and opt-in, they but should be avoided