[Development] Proposal: let's use Markdown for the dist/changes changelog

Mon Nov 2 11:26:31 CET 2020

It has been frustrating for maintainers to generate changelogs in the venerable wizened format that we use.  One hope was that if everyone would write ChangeLog messages in their git commits, the changelogs could be generated automatically; and we have two competing scripts to do that (more below about that).  But in practice, it’s often neglected.  I often forget too, unless I get reminded somehow.  Getting nagged about it on codereview is not much fun for either party.

So in practice, changelogs still require manual editing.  Probably there is someone who knows just what settings to use in just what editor to make it effortless, but for me it always involves a lot of tedious manual spacing, because paragraph indentation is so damn weird (5 spaces usually), etc.  Creator doesn’t know how to wrap a paragraph in a changelog, for example.  There is no tool to center the text inside the star-box headings except for the generator scripts themselves, AFAIK.

Markdown is much more standardized, and there are many editors and other tools for working with it.  Even Qt has built-in support in QTextDocument since 5.14.  So for comparison purposes, I’ll attach the 5.15.2 changelog for qtdeclarative, and the proposal for how it could look in Markdown, which I was editing just now in my QTextBrowser-based editor.  In read-only mode in QTextBrowser, it looks like this:

[cid:95df007b-9427-4811-b8f2-493a6799bead at eurprd02.prod.outlook.com]

Perhaps later we could use the markdown changelogs to render html for the web site, or something.  That’s not my concern yet; I just want to make editing easier, and I think that users who are accustomed to viewing plain-text changelogs are not losing anything either, because markdown also looks quite fine in the terminal, IMO.  There are even terminal formatting tools that render it a bit more fancy (I wrote one version of mdcat in go, but there are multiple versions of that out there).  Vi can syntax-highlight markdown, and there are hacks to make less use vi syntax highlighting, and stuff like that.

One possible bikeshed is whether we should use the easier format for headings like

# QtQuick

## Animations

or one that perhaps looks a bit better in the terminal, but only allows 2 levels of headings (which is enough in practice, so far):

QtQuick
=======

Animations
----------

The advantage of the hash format is QTextDocument already writes markdown that way; but I think I could (and probably should) somehow add a flag to configure it to write either way.  It’s able to read it either way, already.

Now, more about the tooling.  A perl script has been used for some years to generate the dist/changes files from git ChangeLog entries.  At some point Simon wrote a go script which did more or less the same thing (because it’s a minority of us who are happy working with perl, I guess; I also prefer go because it’s easier to read, can be compiled, etc.  But it was my first time using go for anything real.)

But I didn’t like it when we generate a point-release changelog that says there are only minor changes, when I know for a fact that some significant bugs got fixed, and should be mentioned, even though the author forgot to add a ChangeLog to the git commit.  So a few years ago, before each release I was actually scanning the git log manually to look for such bug fixes that ought to be mentioned.  I got tired of that because it always requires several windows at the same time: a terminal to view the git log, an editor for the changes file, and a browser with a couple of tabs open to look up bugs and code changes.  So I modified Simon’s go script to also find the git log entries that mention bugs, and append the git commit message plus extra text taken from the jira subject line.  And I just quietly used that script to overwrite the generated changelogs, and then edited by hand to format it nicely, in qtdeclarative only.  So probably qtdeclarative has had more-complete change logs than most other modules during the last few years.  I’d selectively use it to add a few qtbase changes that I wanted to highlight too though, now and then.  Other people were using it in other modules sometimes.

Then it was proposed that we might as well use the go script all the time.  And the result is what you can see now: https://codereview.qt-project.org/c/qt/qtbase/+/319071  Nobody wants to edit such a long changelog, instead they are just complaining that it’s a mess.  I was able to deal with this in qtdeclarative; I don’t mind the mess, because it means I get all the information I need in the editor I’m using, and I take it for granted that the changes file needs manual editing anyway.  But in qtbase it’s a lot of changes, and it will be a lot, each time.  Some people say if the author didn’t write a ChangeLog, tough luck: the change will not be mentioned.

So what I’d like to do is work on the go changelog generator a bit, before Qt 6.0.  It should do a better job categorizing the entries under the right headings: the Jira component should help to do that in case there is no ChangeLog entry.  It should have a command-line option to control whether to generate entries at all when the git commit does not have a ChangeLog but does have a Task-number or Fixes line.  There are various small bugs to fix.  And I’d really like to generate markdown now, because I don’t want to do any more manual editing of the old format, but manual editing is always needed.  That will help all the maintainers, because they’ll have a choice of tools to use (markdown editors and re-formatters) instead of having to look up obscure vi or emacs settings for how to wrap a paragraph that is indented 5 spaces and starts with a bullet character.

We could also make the sanity bot complain when a change that includes a bug-fix does not include a ChangeLog entry.  But maybe that’s going too far?  Many non-native speakers would write badly worded changelog messages anyway, so just getting more developers to write “something” there does not remove the need for manual editing of the changes file after it’s generated.  But at least you’d get less-messy output from the changelog generator, and reliably categorized.

Maybe the commit message template should include commented-out example changelog lines for every possible category, because otherwise you have to scan the git log and look for examples each time you try to write one in a commit message.  At least I can’t remember all the categories that have been used.  So maybe that’s one of the obstacles, why people are not writing them.  But the commit message template could get really long that way.  Or we could write some tool for editing commit messages, but… meh, who would use it?

We can discuss whether we want to automatically generate changes files with or without all the bug fixes on a module-by-module basis.  IMO it’s too much for qtbase, but OK for other modules that have active maintainers who are willing to rewrite the markdown each time.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20201102/6d90d1b8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: changelog.png
Type: image/png
Size: 153596 bytes
Desc: changelog.png
URL: <http://lists.qt-project.org/pipermail/development/attachments/20201102/6d90d1b8/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: changes-5.15.2
Type: application/octet-stream
Size: 3977 bytes
Desc: changes-5.15.2
URL: <http://lists.qt-project.org/pipermail/development/attachments/20201102/6d90d1b8/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: changes-5.15.2.md
Type: text/markdown
Size: 2876 bytes
Desc: changes-5.15.2.md
URL: <http://lists.qt-project.org/pipermail/development/attachments/20201102/6d90d1b8/attachment-0001.bin>