[Development] The dark side of QtMultimedia

[Nichols Andy]

Hi Gianluca,

> But, what would be the solutions ?!?! 

That is still up for discussion.  The ideal solution would be more contributors and a clear mandate to fix the existing issues.  Right now we have neither.

> The problem is that there is no clear evidence of development planning and improvements. As Callegari said in this email, he filed various bugs long time ago and they still there without any evidence of planning. From my side, I’m stuck on an limitation of VideoOutput on iOS that it’s still there from one year and I don’t see any planning of resolving the issue.
> So, from my point of view, the QtMultimedia seems abandoned (I hope it’s a wrong deduction from a superficial view of the project). The indication of that deduction comes, for example, from the wiki page about backend supports (http://qt-project.org/wiki/Qt_Multimedia_Backends). That page was not updated for a long time giving the impression that no work has been done to improve the QtMultimedia. 
> In that table, there are a lot of “holes" and for each “hole” there is no indication if the “hole” has been planned to be resolved and in which Qt version, or if it is an “hole” impossibile to solve. So, it seems that better than this it’s not planned.
> 
> For example, no one told me (even in this mailing list) if the VideoOutput limitation on iOS it’s impossible to solve or it’s planned to be solved in some Qt versione (i.e. Qt 5.5)

This is a bit of a side issue from the topic at hand, but I am the correct person to answer this question so I will.  Regarding VideoOutput on iOS there is currently a serious limitation that we have been unable to over come.  If you are using the QWidget based API on iOS (and you probably shouldn’t be) then everything should work fine.  That is because it is easy to embed “native” controls in QWidget hierarchies, and thats what we do for QtMultimedia video output (overlay a AVPlayerLayer where the video output should go).  However when we would like to render video in a Qt Quick 2 scene then we need to be able to render the video to a texture.  On OS X it is possible to render video to a OpenGL texture from a hidden AVPlayerLayer window, and then render that in the Qt Quick 2 scene, but that API is not available on iOS.  There is AFAIK currently no high-level API to render video from an AVPlayer to an OpenGL texture on iOS.  The work around to provide any video at all in Qt Quick 2 is to to instead render to a “Window” control which instead falls back to the overlaying the native video window surface on top of the QQuickWindow.  Yes it is less than ideal, but we are not the only framework with this limitation.

There are a couple of alternative approaches on iOS.   It is actually possible using the AVFoundation API to decode encoded multimedia files that can be played and synced manually.  For example you can feed AVFoundation a video file to be decoded and you will get access to: each image buffers (per frame), raw pcm audio data, and the timing information for synced playback, but then it is up to the end user to put that together  as what gets seen and heard.  I did study this route when I was working on the AVFoundation backend for iOS, but it was going to be too much work for the amount of time I had to get something working.  Also as a comparison if you look at the Unity 3D game engine they have a similar limitation; it is possible to playback video’s to textures on iOS, but there is no support for audio simultaneously.  Their workaround is to say that multimedia playback on iOS is only supported fullscreen (to a native window surface).

Another approach that is less than ideal is to use another API than AVFoundation.  We could use another framework like libVLC which uses FFMpeg under the hood to render the video assets.  This will generally be a less efficient path on iOS than using the native AVFoundation API, and also will potentially cause deployment headaches.

So unfortunately it is not being working on, their still isn’t a way to overcome this limitation that I’m aware of in the latest release of iOS, and the alternatives require quite a bit of effort and no one is volunteering to do this work.

> 
> Since Qt 5.1, at each new release I’ll look into new features and improvements and what you can see is a very different pace of some modules respect to others and the QtMultimedia is always one of the slowest one in improvements and fixes.
> 
> So, why there is so few resources and/or interest on developing the QtMultimedia module ? For me, it seems a very fundamental module that I’m really surprised that Digia’s do not commit strong resources on developing it.
> 
> QtMultimedia has an very ambitious objective, and I think it cannot be good improvement just only hoping from community contributions, and there will be no good improvements if Digia (or some other big company) commit resources on it.

I think this comes back to the issue of resources.  From the community side there has not been much interested in larger contributions to QtMultimedia.  Many issues in QtMultimedia require more serious development effort than the average bug fix, so it’s harder to attract the interest of developers who would work on it in their spare time.  I think the best approach here is to be open about what the situation is (as I have in this mail series), and to be as helpful as possible to anyone who would like to contribute to the project.

>From the commercial side resource allocation is influenced by customer demands.  The is true for us at The Qt Company(Digia) and it is in our interest to prioritise the issues our customers are having on the platforms they are using.  The more interest paying customers have in the QtMultimedia module, the more incentive we have to put more resources into it (vs other areas of Qt).

It is important that we are having this discussion to raise awareness of the situation in the QtMultimedia module but I don’t see a clear path forward to resolving these issues unless there are more developers out there genuinely able to contribute.

Regards,
Andy Nichols