[Interest] determining graphics frame swap time when rendering in Qt 6

Be be.0 at gmx.com
Sun Feb 7 21:57:51 CET 2021

Stepping back from the specifics of the application, the general
question is: How to render a smooth animation representing the value of
a continuously changing variable (the position in the audio track) that
can move in either direction at any time at any velocity and
acceleration? Perhaps we need a PID controller?

On 12/16/20 12:10 PM, Be wrote:
> Hi, I am a maintainer of Mixxx DJ software and I'm trying to come up
> with a roadmap for the future of our GUI. The current GUI uses a
> home-baked XML layout system for QWidgets that predates Qt's UIC. One
> of the main features of the application is waveform visualizations of
> the audio tracks being played. For smooth scrolling of the waveforms
> synchronized with the audio going to the audio interface, the waveform
> renderers need to determine at the time of rendering when the next
> buffer swap onto the screen will occur. This is currently implemented
> with QGLWidget by disabling auto buffer swapping and calling
> QGLWidget::swap from a thread which synchronizes with VSync. If you
> want to see the current implementation, look at the VisualPlayPosition
> and VSyncThread classes:
> https://github.com/mixxxdj/mixxx/blob/main/src/waveform/visualplayposition.h
> https://github.com/mixxxdj/mixxx/blob/main/src/waveform/vsyncthread.cpp
> Unfortunately, this is not a viable solution going forward. QGLWidget
> has been removed in Qt 6. Even before that, the current implementation
> causes severe performance problems that make the application unusable
> when built with a macOS SDK >= 10.14. From what I understand, this is
> because of layer backed rendering which can only be disabled by
> building with the macOS 10.13 SDK, which we have to do now with a hack
> downloading the SDK separately so it can be used with up to date
> versions of XCode which do not officially support the 10.13 SDK.
> We attempted another solution (
> https://github.com/mixxxdj/mixxx/pull/1974 ) with QOpenGLWidget using
> the QOpenGLWidget::frameSwapped signal to estimate when the rendered
> frame would be shown on screen. This performed much better on macOS
> with SDKs > 10.13. However, we could not accurately predict the timing
> of the frame swap onto the screen because:
> > Unlike QGLWidget, triggering a buffer swap just for the
> QOpenGLWidget is not possible since there is no real, onscreen native
> surface for it. Instead, it is up to the widget stack to manage
> composition and buffer swaps on the gui thread. When a thread is done
> updating the framebuffer, call update() on the GUI/main thread to
> schedule composition.
> https://doc.qt.io/qt-5/qopenglwidget.html#threading
> The result was that the waveforms did not scroll smoothly. Instead,
> they jerked back and forth.
> This might be overcome with the approach recommended in the
> QOpenGLWidget documentation:
> > This involves drawing using an alternative render target, that is
> fully controlled by the thread, e.g. an additional framebuffer object,
> and blitting to the QOpenGLWidget's framebuffer at a suitable time.
> However, we would like to move away from our hacky old QWidget XML GUI
> layout system and use QML. Just the other day I decoupled the backend
> application logic from QWidgets and got a simple Qt Quick
> proof-of-concept GUI working.
> We could adapt our legacy OpenGL waveform rendering code to work in a
> Qt Quick GUI, however I'm not sure we'd really solve our performance
> problems on macOS because the Qt Quick scene graph rendering in a
> separate thread is not supported with the OpenGL backend on macOS.
> So it seems the most maintainable and performant way forward would be
> adapting our existing OpenGL GLSL waveform renderer to
> Vulkan-compatible GLSL to use the new Qt Shader Baker. This still
> leaves open the question of how to determine when the frame being
> rendered will be swapped to the screen. If I understand correctly,
> either the application needs to be able to control the swap time or we
> need a guarantee from Qt that swaps will occur at a steady,
> predictable time. QQuickRenderControl allows the application to
> control the render time, but I have not found anything in the Qt 6
> documentation about how to control or determine the swap time. From my
> quick glance at the Qt 6 source code, it seems this is controlled by
> RHI in private APIs. Is this possible? Are there any other solutions
> you can suggest?
> Potential solutions I have thought about but I don't know if they'd work:
> 1. If we control the rendering time of the entire window with
> QQuickRenderControl, could we measure the difference between the start
> of rendering and the QQuickWindow::frameSwapped signal to estimate the
> next swap time? I don't know if this would work though because I think
> the rendering time would have to be roughly constant. With all the
> optimizations the Qt Quick scene graph does, I'm guessing the
> rendering time can vary dramatically depending on what needs updating.
> 2. Another approach might be using QQuickRenderControl and
> QQuickRenderTarget to render a QQuickWindow to an offscreen buffer so
> the application can determine when to swap that buffer on screen. I'm
> not sure sure how feasible this is. We still need an on screen QWindow
> to receive input, resize, and hide/show events which would need to be
> redirected to the offscreen QQuickWindow. As far as I can tell, the
> only way we might be able to do that with thie existing Qt 6 APIs is
> with a QOpenGLWindow which shares a QOpenGLContext with the
> QQuickWindow, so we'd be bound to OpenGL and couldn't take advantage
> of the other RHI backends.
> By the way I never got the registration confirmation email when I
> tried to sign up for this mailing list with my mixxx.org email address.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/interest/attachments/20210207/98ca919a/attachment.html>

More information about the Interest mailing list