[Development] Build Hotspots in the Qt build process

[Shane McIntosh]

Hi Simon,

On Feb 5, 2014, at 11:58 AM, Simon Hausmann <simon.hausmann at digia.com> wrote:

> On Monday 3. February 2014 22.39.27 Shane McIntosh wrote:
>> Hi Qt developers!
>> 
>> My name is Shane. I’m a PhD student at Queen’s University in Canada.
>> 
>> I’ve been working on an approach for detecting build hotspots, i.e., files
>> that not only take a long time to rebuild, but also change often. We think
>> that these files are ideal candidates for refactoring that could shave time
>> off of incremental builds that are really impacting software teams.
>> 
>> We came up with an approach that I presented last weekend at FOSDEM ( slides
>> are available here:
>> http://www.slideshare.net/shanemcintosh/identifying-hotspots-in-software-bu
>> ild-process ). One of the projects that we analyzed was Qt.
>> 
>> I bumped into Tobias at FOSDEM and he suggested that I post the list of Qt
>> hotspots here. So, I’ve made the hotspot list available here:
>> http://sailhome.cs.queensu.ca/~shane/content/qt_hotspots.txt
>> 
>> I’m happy to provide a more detailed Qt dataset when I return to my lab next
>> week.
> 
> Interesting stuff :)

Thanks! :-)

> How do you determine that a particular header file is the "hot spot"? (I didn't 
> understand that part from the slide set) Is that header file commonly included 
> in other files that result in long-building object files?

Specifically, we have a tool for extracting the build dependency graph from a concrete run of the build system. We then annotate the graph with timing information from each build command. Using this graph, we can figure out how long a rebuild will take if a developer changes any source file (add up the times of the edges triggered by changing a source file). We combine this rebuild time info with change frequency data we extract from the version control system (e.g., Git). The hotspots are then the files that change frequently (info from Git) and also rebuild slowly (info from the build dependency graph).

Of course, we need to pick thresholds to identify the files that are too slow and change too much. For this experiment, we used the median number of changes and 90 second rebuild time. These thresholds might need some tweaking based on your development culture. For example, maybe 90 seconds is too low for your system?

Kind regards,
-Shane