[QBS] RFC: Deployment support for qbs

[Christian Kandeler]

Hi,

in the last couple of days, I've thought a bit about what deployment 
support could look like in the Qt Build Suite. Here's what I've come up 
with. The following description is meant to be conceptually complete and 
provides concrete examples. It might therefore appear a bit lengthy. 
Bear with me.


1) Defining "Deployment"

Deployment can be viewed as
     a) the last part of the build process or
     b) as a dedicated process that follows after building.
The distinction is not academic, as we will see below.
In the simplest model, deployment is just a "make install" that puts 
binaries and/or other resources from your project into the local file 
system. However, while this might cover your needs for some project that 
is supposed to be executed on your local machine, it is not enough if 
files are to be deployed to other machines, e.g. for distributing your 
software or, in the case of cross-compilation, even just testing.
Therefore, we split the concept of "Deployment" into two sub-concepts, 
which we will call "Local Installation" and "Remote Deployment".


2) The example project

Let's assume our project is a simple library consisting of one source 
and one header file. There is also a small test application that uses 
the library's services. Without any deployment support, the project 
could look like this:

import qbs.base 1.0

Project {
     Product {
         type: "dynamiclibrary"
         name: "mylib"
         Depends { name: "cpp" }
         files: [ "mylib.h", "mylib.cpp" ]
     }

     Product {
         type: "application"
         name: "mylibTestApp"
         Depends { name: "cpp" }
         files: [ "main.cpp" ]
     }
}


3) "Local Installation" in qbs

This concept exists already: Products can have the "installed_content" 
property, which enables them to install files. This happens as part of 
the build process, i.e. qbs currently implements definition a) of the 
deployment term.
In our example above, you would want to install the header file and the 
shared object, because they are needed to compile and link dependent 
applications, respectively. You would not want to install the test app, 
because it is not needed globally - you can just run it from the build 
directory (or, in the case of cross-compilation, you cannot even run it 
locally at all). So the library product now looks like the following, 
while the rest of the project stays the same:

Product {
     type: [ "dynamiclibrary", "installed_content" ]
     name: "mylib"
     Depends { name: "cpp" }
     files: "mylib.cpp"

     Group {
         qbs.installDir: "/usr/lib"
         files: target
         fileTags: "install"
     }

     Group {
         qbs.installDir: "/usr/include"
         files: "mylib.h"
         fileTags: [ "hpp", "install" ]
     }
}

Note that in the case of cross-compilation, "qbs.installDir" will be 
interpreted as being relative to a sysroot, which is a directory 
specified in the platform description. The same directory is also used 
when building the project, e.g. for gcc's '--sysroot' option.

Issues with the above approach:
     - The "target" keyword used above does not yet exist. Currently 
there is no way to refer to the build result of a product.
     - The fact that the installation is part of the build process means 
that if files are supposed to be installed to "global" locations in the 
host file system, the user is forced to build the project as root. If 
the install step were separate, the user could build normally and 
afterwards do "sudo qbs install".


4) "Remote Deployment" in qbs

There is an important difference between local installation and remote 
deployment: While it is rather unambiguous what the former means, the 
latter can have very different variations. For our example project, 
remote deployment could mean:
     - Building a Debian package, copying the package to a device and 
installing it there (e.g. the Harmattan use case).
     - Copying the files into a directory tree that has been mounted via 
a network file system, thereby copying it to a device.
     - Copying (parts of) the files to a device via rsync.
     - Building a tarball and uploading it to a web server for distribution.
     - Putting the files into a self-extracting archive or some other 
form of installer.
     - ...
It seems hopeless to cover all these cases in qbs itself. However, what 
qbs can do is to prepare the actual deployment in a sensible way.

4a) Specifying what to deploy

Just as for local installation, this can be done by introducing a 
corresponding file tag. Let's assume you have cross-compiled the example 
project and you now want to deploy it to some device for testing:

Product {
     type: [ "dynamiclibrary", "installed_content" ]
     name: "mylib"
     Depends { name: "cpp" }
     files: "mylib.cpp"

     Group {
         qbs.installDir: "/usr/lib"
         files: target
         fileTags: [ "install", "deploy" ]
     }

     Group {
         qbs.installDir: "/usr/include"
         files: "mylib.h"
         fileTags: [ "hpp", "install" ]
     }
}

Product {
     type: "application"
     name: "mylibTestApp"
     Depends { name: "cpp" }
     files: [ "main.cpp" ]

     Group {
         qbs.installDir: "/usr/bin"
         files: target
         fileTags: "deploy"
     }
}

Note that in contrast to local installation, deployment does not include 
the header file, since it is not needed on the device itself. 
Conversely, it does include the test application, because it is run there.
However, this is not the only deployment case. When you are done with 
testing, you want to distribute your library. If you intend to build a 
"devel"-like package, you actually deploy the same files that you 
install locally:

Product {
     type: [ "dynamiclibrary", "installed_content" ]
     name: "mylib"
     Depends { name: "cpp" }
     files: "mylib.cpp"

     Group {
         qbs.installDir: "/usr/lib"
         files: target
         fileTags: [ "install", "deploy" ]
     }

     Group {
         qbs.installDir: "/usr/include"
         files: "mylib.h"
         fileTags: [ "hpp", "install", "deploy" ]
     }
}

Product {
     type: "application"
     name: "mylibTestApp"
     Depends { name: "cpp" }
     files: [ "main.cpp" ]
}

You can support both cases via a property (does not work like this 
currently, but should):

Project {
     id: myProject
     property bool testing: true

     Product {
         type: [ "dynamiclibrary", "installed_content" ]
         qbs.installDir: "/tmp"
         name: "mylib"
         Depends { name: "cpp" }
         files: "mylib.cpp"

         Group {
             qbs.installDir: "/usr/lib"
             files: target
             fileTags: [ "install", "deploy" ]
         }

         Group {
             qbs.installDir: "/usr/include"
             files: "mylib.h"
             baseFileTags: [ "hpp", "install" ]
             fileTags: myProject.testing ? baseFileTags : 
baseFileTags.concat("deploy")
         }
     }

     Product {
         type: "application"
         name: "mylibTestApp"
         Depends { name: "cpp" }
         files: [ "main.cpp" ]

         Group {
             qbs.installDir: "/usr/bin"
             files: target
             fileTags: myProject.testing ? "deploy" : []
         }
     }
}


4b) Deploying

As explained above, qbs cannot be responsible for the complete 
deployment process, but it can do some preparation for common use cases. 
For instance, packaging often involves copying the files to be packaged 
into a directory structure mirroring the one on the target system. For 
instance, the abovementioned "devel package" case for our example 
project could look like this, if you are targetting a dpkg-based system:

$ qbs deploy myProject.testing:false project.deployRoot:debian/mylib

This would remove the contents of the given directory if it exists 
already and then copy all files that have a "deploy" tag to their 
respective "installDir"s, prepending the value of the "deployRoot" 
property. In this example, you would then call dpkg-buildpackage in the 
same directory. This is outside the scope of qbs, but it builds nicely 
on qbs' work. (Alternatively, you would perhaps want to integrate qbs 
into the dpkg-buildpackage flow via the rules file.)
Now let's assume that for testing, you do not want the overhead of 
package creation. Instead, you just want to copy file by file to the 
device using scp. In such a case, you don't need "qbs deploy", since 
that would just be an unnecessary indirection. What you do need, 
however, is the list of files to copy over, preferably in a simple, 
easily parsable format:

$ qbs dump-deploy-info myProject.testing:true

For our example project, this would result in output similar to the 
following:
       /home/icke/myproject/build/debug/mylib.so|/usr/lib/mylib.so
       /home/icke/myproject/build/debug/mylibTestApp|/usr/bin/mylibTestApp


That's it. Opinions?


Christian