[Qbs] Improving qbs resolve performance

[Vincent Hui]

Perhaps QThreadPooler
<https://code.woboq.org/qt5/qt3d/src/core/jobs/qthreadpooler_p.h.html> can
help accelerating building products with inter-product dependencies.


On Thu, 11 Jul 2019 at 20:46, Christian Kandeler <Christian.Kandeler at qt.io>
wrote:

> On Thu, 11 Jul 2019 12:18:30 +0000
> Maximilian Hrabowski <hrabowski at clausmark.com> wrote:
>
> > I have a fairly large project with a root projects that pulls in several
> SubProjects, altogether with unit tests its about 313 qbs files that are
> pulled in.
>
> The number of project files is not all that important by itself. Qt
> Creator, for instance, has a comparable number, and resolves much faster
> than your project.
>
> > Let root.qbs be the root qbs file of my project and depend.qbs a
> subproject that many other subprojects depend on.
> >
> > With a single profile and single config I encounter the following
> duration for qbs resolve on my macbook pro with 12 logical CPUs:
> >
> > 1) qbs resolve (clean, new build directory, so includes creating a build
> graph):  ~1m 25s
>
> qbs --log-time might give some hints as to where that time is spent.
>
> > 2) qbs resolve (no changes): ~0.5 s
> > 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
> > 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
> >
> > From the durations i would expect that there is some room for
> improvement. To me it seems (by looking at 3 and 4) that qbs rebuilds the
> whole build graph if it detects any change to any qbs file in the project.
>
> Yes, there is no partial re-resolve.
>
> > If this is the case i wonder how difficult it would be to improve this
> and what the right approach would be. Maybe looking at “Depends {}” would
> be enough to determine the “dirty path”.
>
> I suspect this is hopeless. In general, there are too many possible
> interdependencies for a statement like "if file x has changed, we know what
> only product y can be affected". At the very least, much more now-temporary
> data from the resolve stage would have to be stored for such logic to work.
>
> > Fortunately QtC 4.9.x seems to cache something now since opening a qbs
> project will no longer cause a full resolve.
>
> I don't think anything has changed there recently. qbs re-resolved when it
> thinks that's necessary, i.e. a project file or a property or the
> environment has changed.
>
> > Any ideas or maybe even someone looking at that already?
>
> The only remote possibility that I see is somehow making use of
> concurrency when resolving, e.g. have one thread per product. It would not
> be trivial, though, due to inter-product dependencies etc; also, you'd need
> a dedicated script engine per thread. I don't remember the details, but
> whenever I though about this topic, I quickly stopped again after examining
> what we do in the ModuleLoader, which is by far the most expensive stage of
> project resolving these days, and thus the only one worth optimizing.
> (I certainly don't want to to dissuade anyone from trying, it's just that
> you need to be prepared for a disappointing outcome. For a much
> lower-hanging fruit in the area of performance improvement, take a look at
> https://bugreports.qt.io/browse/QBS-1448.)
>
>
> Christian
> _______________________________________________
> Qbs mailing list
> Qbs at qt-project.org
> https://lists.qt-project.org/listinfo/qbs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/qbs/attachments/20190712/96fc92c7/attachment-0001.html>