[Development] ChangeLogs

[Jason McDonald]

On Fri, Jan 18, 2013 at 1:05 AM, Oswald Buddenhagen
<oswald.buddenhagen at digia.com> wrote:
> as some certainly noted, the completeness of dist/changes-* severely
> deteriorated over the last few minor releases, and in particular 5.0.0
> is one of the poorest qt changelogs seen in a while.
>
> also, sergio is now attempting to make 5.0.1 changelogs in a heroic
> one-man-effort - not entirely surprisingly, this turns out to be a tad
> more tedious than planned.

I was quietly wondering to myself how long it would take for someone
to notice that I stopped acting as editor for the changelogs when I
left Nokia.

As Sergio has no doubt discovered, manually generating the changes
file is a tedious and time-consuming process, particularly when the
level of co-operation from developers is low to non-existent.

Back in the Trolltech and Nokia days, I made a few (largely
unsuccessful) attempts to get people to describe their important
changes in the changes-file, but mostly I only succeeded in getting
myself flamed (and maybe this email will have the same effect), to the
point where one senior developer tried to wash his hands of this
particular duty by stating on an open mailing list that "the release
branch is the Release Manager's hostile fork of Qt for which the
developers have no responsibility".  While that gave the folks in Qt's
former QA team a good laugh, I would hope that attitudes have shifted
a little since then.

During my tenure as Qt's Release Manager, I came to the following
conclusions about the changes file:
(a) There is no feasible way to automatically produce a high-quality
changes file,
(b) The changes file is an important resource for Qt users,
(c) Jira is not an acceptable substitute for the changes file, and
(d) The git commit history is not an acceptable substitute for the changes file.

Therefore, I basically gave up chasing developers to update the
changes file and did it myself by examining both the bugtracker's list
of closed tasks for the release and the git log, making decisions
about what items were worthy of a mention in the changes file, what
kind of change they were (feature, fix, doc, approved BC/SC break,
etc) and trying to turn the task and commit descriptions into a brief,
high-level, human-readable description of what changed.  This could
take anything from one to five days work for each release depending on
how many commits were in the release.  The quality of the changes file
for each release was largely a factor of whether I could actually
spare the time to do it properly in addition to all the other things a
release manager has to do.

In my opinion, as Qt's most experienced past release manager, this
should have gotten a lot better with modularization, as the ultimate
responsibility for documenting the important changes for each module
is now very clear -- it belongs to the module maintainer.

Now, let me explain my reasons for the conclusions above:

(a) The changes file can't be automatically generated, partly because
the Qt Project's development processes are too flexible (i.e.
inconsistent, poorly-defined and, in some cases, highly resistant to
change) and partly because writing the changelog is a complex
decision-making process for which suitable AI will probably not exist
within our lifetimes.  You need to decide what's important to
end-users and what's not. You need to filter out stuff that user's
never saw in an official release (i.e. stuff that was broken and fixed
again between releases, bugs fixed before a new feature was first
released, fixes that got reverted, patches that were effectively empty
because of later changes and/or resolution of merge conflicts). You
need to figure out when multiple commits equal one fix from the user's
viewpoint.  You need to figure out when one commit equals multiple
fixes from the user's viewpoint.  You need to filter out duplicate
bug-tracker tasks.  You need to filter out changes to code that was
later completely removed or rewritten.  You need to figure out where
in the changes file a particular change belongs.  You need to turn one
or more commit descriptions into a user-understandable description.
Sometimes you even need to read diffs to figure out what really
changed.

That said, I believe that it is possible to create a tool to
assist/manage the process of manually writing the changes file.  The
old changelog tool we had in Trolltech basically showed you a list of
the commits in a release and allowed you to tick off the ones you had
already dealt with, so that you knew which ones you still had to make
decisions about.  Unfortunately, the tool was written for the
source-control system we used before moving to git and fell into
disuse thereafter because writing the changes file became less the
developers' problem and more my problem.

(b) End users do care about the changes file. I know this because I
always received emails and IRC messages each time Nokia's web team
forgot to post the changes file on the web when a new release went
out.  The changes file was the first place many customers looked when
they were trying to decide whether there was any value in them
upgrading to a new version.  Many Qt users, and particularly
commercial customers, don't automatically upgrade every time a new
release is made, because that carries a significant cost and a
significant risk of their code being broken by regressions in Qt (CI
doesn't fix that -- last time I looked our test coverage was barely
50% in most cases).  Those people look at what's changed and only
attempt to upgrade if a new release gives them features or fixes that
they actually need.

(c) We can't drop the changes file and tell people to look at Jira.
There are many commits that have no corresponding Jira task.  Even if
there was a task for every commit, the Jira tasks usually describe
symptoms, not the root cause or what was changed as a result of
diagnosing those symptoms.  There is also considerable reluctance
among developers to modify Jira task summaries and descriptions and to
delete irrelevant comments.  I don't agree with that reluctance, but I
also don't think I'm ever going to change it.

(d) We can't drop the changes file and tell people to look in the git
history.  For one thing, the changes file has a very different
audience to the git history.  Many of the people who pinged me when
the changes file wasn't published weren't Qt developers.  Some were
developers using Qt to write their own apps.  Others were managers and
project-leads who were trying to decide whether taking a new release
was worth the potential disruption to their projects.  Those aren't
people who rarely have the desire to grovel through the thousands of
commit messages in each release, nor the detailed knowledge of Qt
internals to draw conclusions about whether those commit messages
means that the release includes changes that their project needs.


So, based on my bitter experience, here are my recommendations:

1. Let's acknowledge that writing the changes file is going to have to
be a manual process due to the complexity of the decision-making and
the lack of a strict development process.

2. Let's agree that module maintainers should be responsible for their
module's changes file, and clearly document that responsibility on the
part of the wiki that lists a maintainer's duties.

3. Let's call for volunteers to prototype a new changelog tool along
similar lines to the old one, but this time with a better UI, the
ability to share data between users via a git repo, and some kind of
checklist to assist the decision-making process.  I'm happy to help
with setting requirements, but I regret that I won't be able to do
much more than that as my new full-time job does not involve Qt.

4. Let's try to make the job of our maintainers that little bit easier
by writing good commit summaries and diligently reviewing the commit
summaries of our peers.


Ok, I'm wearing my asbestos jumpsuit, so you may now flame me at will.....

--
Jason