[Qbs] Improving qbs resolve performance

[Christian Gagneraud]

On Fri, 12 Jul 2019 at 00:45, Christian Kandeler
<Christian.Kandeler at qt.io> wrote:
>
> On Thu, 11 Jul 2019 12:18:30 +0000
> Maximilian Hrabowski <hrabowski at clausmark.com> wrote:
>
> > I have a fairly large project with a root projects that pulls in several SubProjects, altogether with unit tests its about 313 qbs files that are pulled in.
>
> The number of project files is not all that important by itself. Qt Creator, for instance, has a comparable number, and resolves much faster than your project.
>
> > Let root.qbs be the root qbs file of my project and depend.qbs a subproject that many other subprojects depend on.
> >
> > With a single profile and single config I encounter the following duration for qbs resolve on my macbook pro with 12 logical CPUs:
> >
> > 1) qbs resolve (clean, new build directory, so includes creating a build graph):  ~1m 25s
>
> qbs --log-time might give some hints as to where that time is spent.

I checked out my old experimental branch and compared resolving qtc
and this project:

QtC:
Qt was set up successfully.
        Project file loading and parsing took 161ms.
        Preparing products took 1ms.
        Setting up product dependencies took 4s, 826ms.
                Setting up transitive product dependencies took 930ms.
        Handling products took 5s, 160ms.
                Running Probes took 2s, 190ms.
                1916 probes encountered, 8 configure scripts executed,
1901 re-used from current run, 0 re-used from earlier run.
        Property checking took 689ms.
Activity 'ModuleLoader' took 11s, 433ms.

My project:
Qt was set up successfully.
        Project file loading and parsing took 100ms.
        Preparing products took 0ms.
        Setting up product dependencies took 1m, 37s, 54ms.
                Setting up transitive product dependencies took 1s, 74ms.
        Handling products took 17s, 925ms.
                Running Probes took 1s, 799ms.
                804 probes encountered, 5 configure scripts executed,
797 re-used from current run, 0 re-used from earlier run.
        Property checking took 194ms.
Activity 'ModuleLoader' took 1m, 55s, 496ms.

So the obvious difference is "Setting up product dependencies".

Looking at valgrind/callgrind data, it seems that the hot path is a
loop, here is a simplified tree with the final loop
- resolveProjectFromScratch()
- loadProject()
- handleTopLevelProject()
- setupProductDependencies()
- 0 - resolveDependencies()
- 1 - resolveDependsItem()
- 2 - loadModule()
- 3 - instanciateModule()
- 4 - goto 0

This match with "my gut feeling" expressed previously, projects with
complex dependency tree are penalised.
In this particular case, a cache system would greatly help, or maybe
there is one in place, and the project files do something that
invalidate this cache...

Chris



>
> > 2) qbs resolve (no changes): ~0.5 s
> > 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
> > 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
> >
> > From the durations i would expect that there is some room for improvement. To me it seems (by looking at 3 and 4) that qbs rebuilds the whole build graph if it detects any change to any qbs file in the project.
>
> Yes, there is no partial re-resolve.
>
> > If this is the case i wonder how difficult it would be to improve this and what the right approach would be. Maybe looking at “Depends {}” would be enough to determine the “dirty path”.
>
> I suspect this is hopeless. In general, there are too many possible interdependencies for a statement like "if file x has changed, we know what only product y can be affected". At the very least, much more now-temporary data from the resolve stage would have to be stored for such logic to work.
>
> > Fortunately QtC 4.9.x seems to cache something now since opening a qbs project will no longer cause a full resolve.
>
> I don't think anything has changed there recently. qbs re-resolved when it thinks that's necessary, i.e. a project file or a property or the environment has changed.
>
> > Any ideas or maybe even someone looking at that already?
>
> The only remote possibility that I see is somehow making use of concurrency when resolving, e.g. have one thread per product. It would not be trivial, though, due to inter-product dependencies etc; also, you'd need a dedicated script engine per thread. I don't remember the details, but whenever I though about this topic, I quickly stopped again after examining what we do in the ModuleLoader, which is by far the most expensive stage of project resolving these days, and thus the only one worth optimizing.
> (I certainly don't want to to dissuade anyone from trying, it's just that you need to be prepared for a disappointing outcome. For a much lower-hanging fruit in the area of performance improvement, take a look at https://bugreports.qt.io/browse/QBS-1448.)
>
>
> Christian
> _______________________________________________
> Qbs mailing list
> Qbs at qt-project.org
> https://lists.qt-project.org/listinfo/qbs