[Development] Introducing API Change Review Bot

[Daniel Smith]

Hi all,

It's likely that you've already been introduced to QtAPIReviewBot, but in case you haven't, I think the Proof-of-Concept has settled enough to start a discussion about our API Review process which now includes this bot.

In addition to the new bot commenting on changes, we still carry out the normal API review process before release, which consists of running a handful of scripts that generate gerrit changes that contain "interesting" header file changes for each module. This process has been very successful in preventing unwanted API changes in a release, so it's sensible to keep it around.

Historically, API change reviews have caused delays in the planned release schedule for .0 releases for various reasons, but most can be traced back to changes earlier in the development cycle may not have been thoroughly evaluated for their effect on the public API. To help combat this, we wanted to find a way to kick off the API review process as early as possible, and thus QtAPIReviewBot was born.

The bot listens for Code-Review +2 events on changes in dev, and then feeds the diff for public header files and commit message into GPT-4 to generate a summary of changes, along with the significance of the change. If the change is deemed as "having significant changes to the public API", then the hashtag Needs API-Review_6.8<https://codereview.qt-project.org/q/hashtag:%22Needs+API-Review_6.8%22> is added to the change ( _[version] increments when the branching event occurs from dev -> new feature branch). Further. If the change is being cherry-picked with the use of the Pick-to footer, the hashtag API change cherry-picked<https://codereview.qt-project.org/q/hashtag:%22API+change+cherry-picked%22> is added.

Module maintainers should investigate changes with these hashtags to validate as soon as possible (preferably before merging the change) that the parts of the change which would affect the public API are desirable and correct for the intended release. Care should also be taken to ensure that such changes which affect the public API are appropriate to backport to older feature branches when tagged with "API change cherry-picked", since avoiding breakage within a feature branch is vital.

  *
When a change has been reviewed and the reviewer/maintainer is satisfied, the "Needs API-Review_x.x" hashtag should be removed.

Current limitations and the future:

  *
New LLMs are regularly under investigation to improve comprehension and accuracy of the bot.
  *
The bot currently evaluates changes atomically. Evaluating entire Topics or Relation Chains could be possible (Feedback wanted)
     *
A JIRA ticket could be auto-generated when all changes of a topic (preferred) or relation chain which include public API changes receive a Code-Review +2, requesting API review of said linked changes.
  *
While the new API Review Bot should mean less work during the formal API Review, creating tickets for issues discovered during the formal API review would help to improve response time during this critical phase before release. See QUIP-10<https://contribute.qt-project.org/quips/10>.

What you can do today:

  *
Join the discussion. How can we improve the API review process?
  *
Comment on QTQAINFRA-5781<https://bugreports.qt.io/browse/QTQAINFRA-5781> if the bot has made a mistake or hallucinated, or if a path should be excluded.
  *
Ping for API reviews early when you've made a change that will affect the usage or behaviour of the public API.
  *
Subscribe to changes with the Needs API-Review_x.x hashtag:
     *
Codereview -> Settings -> Notifications
     *
Fill in the repo you want to receive notifications for, and use the search expression prefixhashtag:"Needs API-Review_"

Best regards,
-Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20240422/bff13b6e/attachment.htm>