[Development] New Qt Multimedia

[Lars Knoll]

Hi all,

After 5 months of work, I am now preparing to merge the new API and implementation for Qt Multimedia back into the development branch.

You can find the first iteration of a merge commit here: https://codereview.qt-project.org/c/qt/qtmultimedia/+/351108

With this, Qt Multimedia is probably the module that is changing most between Qt 5 and Qt 6. The reason for that is that we had large issues maintaining Qt Multimedia during the Qt 5 lifetime, and never really got to a point where it offered a consistent experience across all platforms.

The hope is that we can change that for Qt 6. To make this possible, we have changed not only parts of the public API, but completely redone its internal architecture, especially how multimedia connects to the platform specific backends. Apart from cleaning up the backend API and greatly simplifying it, I also chose to make it private and remove the plugin architecture around it. The backend is now selected at compile time, and we’re now only supporting one backend per platform.

The architectural cleanup lead to huge simplifications in the code base. The module went from 140k LOC in 5.15 to 73k LOC in Qt 6, while keeping 90% of the functionality we had in 5.15 and adding a few things that were missing there. This should make it significantly easier to maintain and further develop the module over the lifetime of Qt 6.

Here’s the list of largest changes (copy from the commit log of the merge commit above):

* New QMediaDevices API to discover available input/output devices
  and get notifications about changes
* New QMediaFormat API to query for supported codecs and file formats
* Refactored playback API, QMediaPlayer and playlist handling are
  fully separated (and can be connected on the app level)
* Simplified QMediaPlayList API to use QUrl instead of the
  QMediaContent/QMediaResource pair.
* Support for multiple audio/video/subtitle streams when playing back
  media
* Better meta data support for audio/video files and individual streams
  inside the file
* New QMediaCaptureSession API as a central class to handle media capture
* QCamera, QMediaEncoder and QMediaImageCapture are to be used together
  with the capture session to define a capturing pipeline
* Removed QCameraFocus, QCameraExposure and QCameraImageProcessing
  and merged those features into QCamera itself
* New QMediaEncoder class to connect to a capture session for encoding
  captured audio/video.
* new QMediaEncoderSettings class to define the audio/video codecs and
  file format to be used when encoding.
* Larger changes to the QML API, it is not symmetric to the C++ API,
  and to the largest part simply uses the public C++ classes
* Rewritten video output pipeline. The new QVideoSink class is the
  main interface between the media player/capture session and an
  output surface.
* Built in support for rendering more uncompressed video formats
  (RGB and YUV), both through QPainter and shaders
* Support for conversion of video frames in all formats to a QImage
* Support for HW accelerated rendering through RHI for the QML
  VideoOutput element
* Support for HW accelerated decoding in many cases
* Cross platform support for audio decoding through QAudioDecoder
* The backend API is now private, so we can adjust it as needed to
  implement new functionality
* Removed the QMediaService/QMediaObject abstraction
* Removed the plugin architecture for backends, backends are now
  chosen at compile time
* Removed the QAudioProve/QVideoProbe APIs
* Removed QSound. Use QSoundEffect instead.
* Removed the Radio API
* Removed QAudioRecorder. Use QMediaCaptureSession/QMediaEncoder
  or QMediaRecorder instead.
* Removed QAbstractVideoFilter. For filtering QML video, use
  shader effects instead.
* Removed QAbstractVideoSurface. QVideoSink replaces it.
* QVideoSurfaceFormat has been renamed to QVideoFrameFormat.
* Significantly reduced the API surface of the backend API (from
  ~40 to 13 classes)
* Fully rewritten gstreamer backend that only relies on base and good
  plugins plus codecs.
* Windows uses WMF exclusively, added camera support through WMF

There are still open issues and gaps in the implementation that need fixing, but the code is now in a decent enough shape to merge it back to dev and continue on that branch. We will however now have everything ready in Qt Multimedia for the 6.2 feature freeze, and will be working with an exception on this module. Especially the support for camera and media capture on Windows and the Android backend still need more work. The gstreamer backend for Linux and AVFoundation for iOS and macOS should be in a pretty decent shape. QNX support is missing right now, and is planned after 6.2.

Volker has done a smaller API review round already, but I’d now appreciate more eyes on the code base, both on the API side and implementation. Let me know what you think, and also what doesn’t work.

Cheers,
Lars