[Qbs] Improving qbs resolve performance

[Christian Kandeler]

On Thu, 11 Jul 2019 12:18:30 +0000
Maximilian Hrabowski <hrabowski at clausmark.com> wrote:

> I have a fairly large project with a root projects that pulls in several SubProjects, altogether with unit tests its about 313 qbs files that are pulled in.

The number of project files is not all that important by itself. Qt Creator, for instance, has a comparable number, and resolves much faster than your project.

> Let root.qbs be the root qbs file of my project and depend.qbs a subproject that many other subprojects depend on.
> 
> With a single profile and single config I encounter the following duration for qbs resolve on my macbook pro with 12 logical CPUs:
> 
> 1) qbs resolve (clean, new build directory, so includes creating a build graph):  ~1m 25s

qbs --log-time might give some hints as to where that time is spent.

> 2) qbs resolve (no changes): ~0.5 s
> 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
> 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
> 
> From the durations i would expect that there is some room for improvement. To me it seems (by looking at 3 and 4) that qbs rebuilds the whole build graph if it detects any change to any qbs file in the project. 

Yes, there is no partial re-resolve.

> If this is the case i wonder how difficult it would be to improve this and what the right approach would be. Maybe looking at “Depends {}” would be enough to determine the “dirty path”.

I suspect this is hopeless. In general, there are too many possible interdependencies for a statement like "if file x has changed, we know what only product y can be affected". At the very least, much more now-temporary data from the resolve stage would have to be stored for such logic to work.

> Fortunately QtC 4.9.x seems to cache something now since opening a qbs project will no longer cause a full resolve.

I don't think anything has changed there recently. qbs re-resolved when it thinks that's necessary, i.e. a project file or a property or the environment has changed.

> Any ideas or maybe even someone looking at that already?

The only remote possibility that I see is somehow making use of concurrency when resolving, e.g. have one thread per product. It would not be trivial, though, due to inter-product dependencies etc; also, you'd need a dedicated script engine per thread. I don't remember the details, but whenever I though about this topic, I quickly stopped again after examining what we do in the ModuleLoader, which is by far the most expensive stage of project resolving these days, and thus the only one worth optimizing.
(I certainly don't want to to dissuade anyone from trying, it's just that you need to be prepared for a disappointing outcome. For a much lower-hanging fruit in the area of performance improvement, take a look at https://bugreports.qt.io/browse/QBS-1448.)


Christian